functional design about twint-ng HOT 5 OPEN

My thoughts below:

1) The package & CLI options are a good way of exposing Twint's APIs
Agreed - the combination is enough to allow the majority of use cases
2) Propose to remove databases, ES, translations
Agreed - we should keep this package lean and purpose specific
3) Do we really need all the async stuff?
Synchronous requests should suffice
4) Python3 only
Definitely :)

Output should be a file or stdout
The module should expose a generator interface which can iterate between requests - this will allow "streaming" of results

The CLI options all make sense

I will setup project scaffold this weekend to get us started

from twint-ng.

I think you forgot to push the branch.
We can port all the logic concerning which URL's to use and the HTML elements to look at. But since we're going to use (synchronous) Requests there is no use in porting the entire scraper I think.

During tests I did this weekend, I did not find any reason to do things like rotate user agents. So we should keep the code very simple, only adding bells and whistles when we really need them.

from twint-ng.

Scaffolding done in a separate branch - let me know what you guys think of the tooling choices

How much is portable from the current Twint package? I would assume the scraper can be moved across

from twint-ng.

The module should expose a generator interface which can iterate between requests - this will allow "streaming" of results

Yeah, second that, that would be really neat.

from twint-ng.

Totally agree 🎉

from twint-ng.