acrosman / sc_salary_data Goto Github PK

The raw_data directory has gotten to the point that the file names are inconsistent. They should be updated to remove %20 and replace with a [space].

Currently attempts to merge files into the processed.json file using combined_files.py triggers the following:

$ python combine_files.py 
/path/to/project/raw_data/FOIA Transparency Salary Data 1.2021.csv
Traceback (most recent call last):
  File "combine_files.py", line 34, in <module>
    file_date = re.match('.*([0-9]{8}).*csv$', listing).group(1)
AttributeError: 'NoneType' object has no attribute 'group'

The process in get_salary_data.py is doing a nice job of pulling data from the state's servers on a regular basis. But #8 is caused by the fact that it's not handling file names well. File names should be consistent when saved to disk. Please update the process to make sure the files are consistently named, and artifacts like %20 from the URL are properly decoded.

This project should conform to Flake8 but doesn't in some places.

The readme file could use to have a good copy edit done.