This code is written on Python 3.6 and uses scrapy 1.4.0.

You should install scrapy with pip: pip install --user scrapy

Note: on my system, python points to python3.6 and pip points to pip3, you should check yours before running.

This spider starts from http://www.doctoralia.com.br/medicos, and goes to the next pages until reach 500 occurrences.

To run this spider, open the terminal on this repository, then type:

cd climatempo
scrapy crawl climatologia

Your csv file with the scrapped data should be on climatologia-clima-tempo/climatempo/climatologia.csv

• This spider was designed for getting only the 100 first occurences, thus it will not scrap the entire web site;

• I have only attached instructions for running this spider localy but we could also use scrapinghub's cloud, which has excellent tools for debugging, and data checking;

• The csv structure should be:

          <City>, <Minimum temperature>,  <Maximum temperature>, <Accumulated precipitation>
  

where fields are:

• This website relies on java script for changing cities and we cannot navigate on it entirely with scrapy.
• I usualy use (splash)[https://splash.readthedocs.io/en/stable/] along with scrapy for rendering the java script. However this particularly website makes a lot of requests for its api which makes it is dificult to countour with (scrapy-splash)[https://github.com/scrapy-plugins/scrapy-splash].
• One option was to use (Selenium)[www.seleniumhq.org] which I didn't because it does not work properlly with the scrapy's parallel processing;
• Since I could get acces to climatempo's (api)[http://apiadvisor.climatempo.com.br/], I downloaded a list of cities for the State of São Paulo, which are in the climatempo/cities.json file;
• I did not attach the code for requesting the list of cities because it requires an api token, wich would complicate the reproduction of this code. Hence I lefted the list of cities hardcoded in the climatempo/cities.json file;