alanswenson / amz-review-scraper Goto Github PK

add a model to store users

https://github.com/AlanSwenson/amz-review-scraper/blob/master/amz_review_scraper/models.py#L3

move the code that creates the DB from scratch out of the model_fuctions.py file and into its own file in the top level so it can be executed when necessary.

make one central layout.html in the static folder that all the blueprints templates extend

UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.

You might also might want to think about the shared functionality of each model, and how to refactor.

language: python python: '2.7' script: echo "all tests passing" after_success: - pip install -r requirements.txt - zappa update dev

tests for model_functions

add tests for the actual scraper

this is the last scrape that still doesn't use find_attribute()

attrs is not really needed and it's covered with **kwargs

scraper should check for new reviews on ASINs already in the DB periodically

http://exploreflask.com/en/latest/static.html

right now everything is #

uses db.create_all()
Need to figure out where to put this and the best was to implement it

Right now with the Giant HTML file, it is hard to look through and understand what the code is looking for. And also makes the project look like it's HTML. It being real is a great advantage, But I would think of an easy way to a run a test against real html, without saving it. And the regular test should just use an artificially trimmed chunk

https://hackernoon.com/how-to-change-repo-language-in-github-c3e07819c5bb

Useful for checking when the last time the particular ASIN was scraped

Originally posted by @beginbot in #60

clear instructions on how to startup a new DB on initial launch in the readme

tests for run.py

It would be cool to deploy this as a Lambda function with Zappa. https://github.com/Miserlou/Zappa

Shouldn't be too hard.

right now default is /track
change to /login

check the current number of reviews against last time checked and update if it is different

https://flask-s3.readthedocs.io/en/latest/

An Error Occured While Scraping
Traceback (most recent call last):
File "app.py", line 44, in
main()
File "app.py", line 35, in main
amazon.scrape(url, asin)
File "/Users/alanswenson/projects/amz-review-scraper/amz-review-scraper/amzscraper.py", line 26, in scrape
raw_html = requests.get(url, headers=header, proxies=proxies)
File "/Users/alanswenson/.local/share/virtualenvs/amz-review-scraper-SIwdd7Z0/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/Users/alanswenson/.local/share/virtualenvs/amz-review-scraper-SIwdd7Z0/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/alanswenson/.local/share/virtualenvs/amz-review-scraper-SIwdd7Z0/lib/python3.6/site-packages/requests/sessions.py", line 515, in request
prep.url, proxies, stream, verify, cert
File "/Users/alanswenson/.local/share/virtualenvs/amz-review-scraper-SIwdd7Z0/lib/python3.6/site-packages/requests/sessions.py", line 690, in merge_environment_settings
no_proxy = proxies.get('no_proxy') if proxies is not None else None
AttributeError: 'str' object has no attribute 'get'

@davidbegin

Running with just python app.py assumes you are in the amz-review-scraper folder. But the pipenev instructions are the root. The directions should be from the root. Also you should include some info on any options and what you expect to happen the output. I suggest https://github.com/asciinema/asciinema if you want a reallllll slick looking readme.

Currently runs in terminal, should eventually run as a web app

So you have some tests now, and you want always make sure you are maintaining and hopefully increasing coverage. I would figure out a code coverage tool, and enforce it in travis and add a badge to readme.

Modify DB to account for different types of users