hayesall / ffscraper Goto Github PK

The design of ffscraper/__main__.py is starting to show some flaws, and its current state will become more difficult to maintain as more features are added.

• Crash-proofing: pickle intermediate outputs, trigger certain actions based on the contents of the log file.
• Outputs need to be rethought; currently there are special functions (schemaString and Utils.PredicateLogicBuilder) for converting values to a representation that is written to files. A more powerful design would involve a standard internal representation of the data that would be converted to the necessary format before being written to disk, regardless of whether that output format was BoostSRL, Cytoscape, JSON, or SQL.
• Replace the progress module with something like tqdm, perhaps with a flag for silencing the progress bar (if the process is sent to the background, for instance).
• Three "phases" exist for scraping stories, ordering them with respect to time, and scraping users. These phases should be split into functions, or written in a way that makes it more obvious what information is needed in which phase. For instance: Phase I needs a list of story-ids, Phase III needs a list of user-ids, etc.

Currently an exception is raised when parsing the contents of /s/12242141, resulting in remaining sids being dumped to a file.

Produced with:

$ echo "12242141" > sid.txt
$ python scraper/Scraper.py -f sid.txt