This is our first project status update. We hope to make more of these during the year.
A new home
We've join the Code 4 Puerto Rico community! A very important part of our effort is community. What better community to be a part of than the one we are trying to help and impact.
We are happy to join this awesome community and hope we can help it grow.
Sunset of ContratosPR on Spectrum
At some point we will be closing out the space on Spectrum.chat in favor of our Slack channel in the Code 4 Puerto Rico Slack workspace. You can join here.
A new repo
We've added this repo, yes this one 😄, to serve as the central repository for ContratosPR.
Our goal is to create a place where issues, ideas, and community conversation can happen without being tied to either of our existing projects. We will also be posting our transparency efforts in this repo.
With this addition ContratosPR is now:
New Issue Templates
We've added some new issue templates to better engage with all of you. We hope you all find them helpful.
Talks
José and I (froi) are going to be speaking at PyCascades on Feb. 08, 2020 in Portland, OR!
Data
- We've downloaded and extracted data from 200k+ contract documents (PDFs)
- There were 3k documents where we couldn't extract data from. This was because the content was in images.
- We are working to get these processed using Tesseract
Tooling
We've add new tools to our data harvesting process, Poppler and Tesseract.
Poppler
- pdftotext for text extraction
- pdfinfo to get file metadata like amount of pages
- pdftoppm to convert PDF pages to images
Tesseract
Tesseract is an open source OCR engine by Google. We are using this to process PDF pages that we convert into images or have its content in images.
Data
We've harvested data from our last harvest up to what is available in Jan. 2020. A Django manage command was developed to help us do atomic data harvests.
Our data grew a bit out of control for a moment and we had to move back to a Heroku hosted Postgres instance.
We've also started to play with our data a bit. We're thinking on what to do next with it and will definitely be asking for help in figuring out what we can do and what is needed.
Ideas up to now
-
Contractor grouping / normalization: since this data is supplied from a number of government entities, there are instances of contractors having the same name spelled differently or very similar names. We've been thinking on how to group these contractors to improve search and our UX.
-
We've gotten feedback that this data should be a part of a bigger knowledge graph. We agree! Although it's not in our near future, we have started to think about what this means and how to get to that goal.
-
Data experiments: we want to see what we can build to further demonstrate the uses for this data. We'd love to see what suggestions you all have.
We'll I think this is long enough. Thanks for reading and hopefully we will have more updates soon. ❤️