Closing quotation mark is removed from title about newspaper HOT 4 CLOSED

Good point @igor-shevchenko , this shouldn't be a hard fix.

BUT, I'm starting to feel that we are doing things gimmicky if this were to be added. If we keep going down this path, and keep adding rules like this for every language's exotic delimiters, then we'd be left with a really confused logic to handle titles.

At this rate, I'd prefer to remove delimiting a title by "»" entirely!

What are your thoughts?

from newspaper.

You are right, title processing rules should be as simple as possible. Yet, it is important not to corrupt the text while cleaning.

The main issue with "»" is not even with arrow splitter. TITLE_REPLACEMENTS remove quotes no matter what splitter is used. I'm not sure if I have ever seen a page that requires such a handling. So, I support the removing of this logic. On the other hand, there could be a reason to add it in the first place.

Still, it leaves us with a problem of splitting a title with "»" when the character is used not as a delimiter. It's not specific to quote. Any other delimiting character can be a part of the actual title. I think this may be solved with a change of get_title function logic. It should try to split string with all of the existing splitters and then select the one that gives more pieces.

What do you think about it? If you agree, I'll send both pull requests (remove TITLE_REPLACEMENTS and modify get_title).

from newspaper.

I agree with you. But, let's see those PRs first and make a decision from there when we can make tests.

from newspaper.

I noticed that when "»" is used as a separator it's usually surrounded by spaces, but not when used as a quote. So changing the line to ARROWS_SPLITTER = StringSplitter(" » ") should probably fix this issue, while preserving the title parsing.

The TITLE_REPLACEMENTS does seem redundant, as it does the same thing as ARROWS_SPLITTER. I don't see the need for it there...

from newspaper.