Comments (4)
Good point @igor-shevchenko , this shouldn't be a hard fix.
BUT, I'm starting to feel that we are doing things gimmicky if this were to be added. If we keep going down this path, and keep adding rules like this for every language's exotic delimiters, then we'd be left with a really confused logic to handle titles.
At this rate, I'd prefer to remove delimiting a title by "»"
entirely!
What are your thoughts?
from newspaper.
You are right, title processing rules should be as simple as possible. Yet, it is important not to corrupt the text while cleaning.
The main issue with "»"
is not even with arrow splitter. TITLE_REPLACEMENTS
remove quotes no matter what splitter is used. I'm not sure if I have ever seen a page that requires such a handling. So, I support the removing of this logic. On the other hand, there could be a reason to add it in the first place.
Still, it leaves us with a problem of splitting a title with "»"
when the character is used not as a delimiter. It's not specific to quote. Any other delimiting character can be a part of the actual title. I think this may be solved with a change of get_title
function logic. It should try to split string with all of the existing splitters and then select the one that gives more pieces.
What do you think about it? If you agree, I'll send both pull requests (remove TITLE_REPLACEMENTS
and modify get_title
).
from newspaper.
I agree with you. But, let's see those PRs first and make a decision from there when we can make tests.
from newspaper.
I noticed that when "»" is used as a separator it's usually surrounded by spaces, but not when used as a quote. So changing the line to ARROWS_SPLITTER = StringSplitter(" » ")
should probably fix this issue, while preserving the title parsing.
The TITLE_REPLACEMENTS
does seem redundant, as it does the same thing as ARROWS_SPLITTER. I don't see the need for it there...
from newspaper.
Related Issues (20)
- 三国历史
- There seem to be complaints related to the user agent scraping permission issue HOT 3
- Downloaded article from Google RSS News only return Google Images
- can't start new thread HOT 2
- the API doesn't work HOT 10
- Is there any optimization in the configuration of newspaper?
- download() halts/stuck forever with a specific URL
- download() halts/stuck forever with a specific URL
- suggestions for calculate_best_node
- Date extraction is faulty HOT 2
- not working for gnews.org
- TIPS FOR FAST IMPROVEMENT
- Project status HOT 4
- Parsing Problem HOT 6
- Consider switching from lxml's clean_html for enhanced security (and possibly performance) HOT 2
- Getting Older News Articles HOT 4
- gnews with user agent returns empty text HOT 1
- Can I get the CSS Selectors ? HOT 2
- TIPS FOR IMPROVEMENT HOT 5
- Questions about Copilot + Open Source Software Hierarchy
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from newspaper.