opengovernment / opengovernment-local Goto Github PK

View Code? Open in Web Editor NEW

6.0 6.0 1.0 13.67 MB

Legislative information from local governments in the United States

License: MIT License

Python 95.68% Shell 4.32%

opengovernment-local's Introduction

OpenGovernment is a Ruby on Rails application for aggregating and presenting open government data.

Overview

This project powers OpenGovernment.org and was started by the Participatory Politics Foundation.

We hope you'll get involved! Read our Contributors' Guide for details.

Mailing list: Join our developer list.
IRC: Find us in chat.freenode.net channel #opengovernment.
Development Roadmap: December 2012.
Project management & bug tracker: to come, will be updated Dec. 2012. Previously on Pivotal Tracker & Lighthouse.

Visit our Wiki for full installation instructions.

opengovernment-local's People

Contributors

Stargazers

Watchers

Forkers

janbenson

opengovernment-local's Issues

Deployment procedure

Need a deployment procedure to push this repo to a staging or live server.

pa-philadelphia María Quiñones-Sánchez page contains unicode chars

Error when Billy scrapes this page:

02:05:18 INFO scrapelib: GET -
http://philadelphiacitycouncil.net/council-members/councilwoman-maria-d-quinones-sanchez-7th-district/councilwoman-maria-d-quinones-sanchez-contact/
Traceback (most recent call last):
File "/u/apps/virtualenvs/billy/src/billy/billy/ext/ansistrm.py", line 56, in emit
stream.write(message)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 45:
ordinal not in range(128)
Logged from file init.py, line 177

Here are the obvious non-ascii chars I can see, indeed the first one is Unicode point F1:

Not certain, but I'm wondering if Billy might be constrained to the ascii character set:

billy$ find . -type f -print | xargs grep ascii | egrep -v git
./billy/importers/bills.py:            r.encode('ascii', 'replace') for r in remaining]))
./billy/importers/committees.py:            logger.debug("No matches for %s" % member['name'].encode('ascii',
./billy/web/api/emitters.py:                           ensure_ascii=False)
./billy/web/api/emitters.py:        return obj.encode("ascii", "replace")
./billy/scrape/bills.py:        return filename.encode('ascii', 'replace')
./billy/scrape/legislators.py:        return filename.encode('ascii', 'replace')
./billy/scrape/legislators.py:        return filename.encode('ascii', 'replace')
./billy/utils/fulltext.py:        text = text.encode('ascii', 'ignore')
./billy/utils/fulltext.py:        text = text.decode('utf8', 'ignore').encode('ascii', 'ignore')

pa-philadelphia Brian J. O'Neill contact page contains multiple paragraphs

http://philadelphiacitycouncil.net/council-members/councilman-brian-j-oneill-10th-district-minority-leader/councilman-brian-j-oneill-contact/

Scraper results:

18:39:07 INFO scrapelib: GET - http://philadelphiacitycouncil.net/council-members/councilman-brian-j-oneill-10th-district-minority-leader/councilman-brian-j-oneill-contact/
18:39:09 WARNING billy: Skipped paragraphs:
<p><strong>City Hall, Room 562</strong><br>
Philadelphia, PA 19107<br>
T:(215) 686-3422<br>
F:(215) 686-1939<br>
Hours: 8:30AM-5:00PM</p>

<p><strong>Neighborhood Offices</strong></p>

<p><strong>Northeast Office</strong><br>
Bustleton &amp; Bowler, 2nd FL.<br>
Philadelphia, PA 19115<br>
T:(215) 685-0432<br>
F:(215) 685-0436<br>
Hours: 8:30AM-5:00PM</p>

<p><strong>FOP Heroes Hall</strong><br>
11630 Caroline Road<br>
Philadelphia, PA 19154<br>
T:(215) 437-9167<br>
F:(215) 437-9350<br>
Hours: 8:30AM-5:00PM</p>

<p><strong>Email Councilman O&#8217;Neill:</strong></p>

<p><a href="http://[email protected]" target="_blank">Brian.O&#8217;[email protected]</a></p>
                                        &#13;

18:39:09 INFO billy: PART City Hall Office
18:39:09 WARNING billy: Skipped: City Hall Office
18:39:09 INFO billy: save person Brian J. O'Neill

ca-san-jose session_list scrapes outdated page

http://www3.sanjoseca.gov/clerk/agenda.asp contains this text:
THIS WEBSITE IS NO LONGER BEING UPDATED - PLEASE VISIT THE CITY CLERK'S ACTIVE WEBSITE AT: http://sanjoseca.gov/index.aspx?NID=145

pa-philadelphia term '2013-2017' may be missing from metadata

However, adding this obvious naive line:
dict(name='2013-2017', start_year=2013, end_year=2017, sessions=['2013']),

fails with error:
19:02:41 CRITICAL billy: Error: no session_details for session(s): ['2013']

ca-san-jose scraper hangs then fails

Running ca-san-jose scraper:

19:37:14 INFO billy: save person Chuck Reed
19:37:14 INFO scrapelib: GET - http://www.sjdistrict1.com/
19:37:14 INFO billy: save person Pete Constant
19:37:14 INFO scrapelib: GET - http://sjdistrict2.com
Traceback (most recent call last):
  File "/u/apps/virtualenvs/billy/bin/billy-update", line 9, in <module>
    load_entry_point('billy==1.7.0', 'console_scripts', 'billy-update')()
  File "/u/apps/virtualenvs/billy/src/billy/billy/bin/update.py", line 393, in main
    run_record += _run_scraper(stype, args, metadata)
  File "/u/apps/virtualenvs/billy/src/billy/billy/bin/update.py", line 104, in _run_scraper
    scraper.scrape(time, chambers=chambers)
  File "/u/apps/opengovernment-local-staging/scrapers/ca-san-jose/legislators.py", line 92, in scrape
    councilmember_doc = lxml.html.fromstring(self.urlopen(url))
  File "/u/apps/virtualenvs/billy/lib/python2.7/site-packages/scrapelib/__init__.py", line 393, in urlopen
    raise HTTPError(resp)
scrapelib.HTTPError: 403 while retrieving http://sjdistrict2.com/

Billy 1.7.0 incompatible with current scrapelib

We have not yet updated our scrapers to use new Billy module loading per this commit:
openstates/billy@ddcdb91#billy/scrape/__init__.py

As a workaround, we have downgraded to Billy 1.7.0, however Billy 1.7.0 is not compatible with current scrapelib, producing these errors:

TypeError: __init__() got an unexpected keyword argument 'timeout'
TypeError: __init__() got an unexpected keyword argument 'cache_obj'
TypeError: __init__() got an unexpected keyword argument 'cache_write_only'

Best fix: update scrapers to conform to edge Billy.

Short-term workaround: fork billy 1.7.0 and remove these lines from /src/billy/billy/scrape/init.py:

        kwargs['cache_obj'] = scrapelib.FileCache(settings.BILLY_CACHE_DIR)
        kwargs['timeout'] = settings.SCRAPELIB_TIMEOUT
            kwargs['cache_write_only'] = False

pa-philadelphia David Oh email is on different page

David Oh page does not contain his email:
http://philadelphiacitycouncil.net/council-members/councilman-david-oh-minority-whip/councilman-david-oh-minority-whip-contact/

However, it does link to an alternate external page:
http://www.davidohphilly.com/

David Oh email can be scraped from a page on this domain:
http://www.davidohphilly.com/councilman-ohs-office/contact-david/

pa-philadelphia Curtis Jones Jr. contact page idiosyncracies

http://philadelphiacitycouncil.net/council-members/councilman-curtis-jones-jr-4th-district/contact-councilman-curtis-jones-jr/

Scraper results:

18:39:17 INFO billy: PART City Hall, Room 404
18:39:17 INFO billy: PART , Philadelphia, PA
18:39:17 INFO billy: PART Email: [email protected]
18:39:17 INFO billy: PART (215) 686-3416, (215) 686-3417
18:39:17 INFO billy: PART Fax: (215) 686-1934
18:39:17 INFO billy: PART Local Office:
18:39:17 WARNING billy: Skipped: Local Office:
18:39:17 INFO billy: PART 5398 Wynnefield Avenue, Philadelphia PA
18:39:17 WARNING billy: Skipped: 5398 Wynnefield Avenue, Philadelphia PA
18:39:17 INFO billy: PART Phone: (215) 685-0293, (215) 685-0295
18:39:17 WARNING billy: Already have phone numbers for one office: Phone: (215) 685-0293, (215) 685-0295
18:39:17 INFO billy: save person Curtis Jones, Jr.

Billy framework is deprecated

Note from https://github.com/jamesturk
We're moving away from billy in favor of a new scraping framework (opencivicdata/pupa) more suited to municipal work, so I can't recommend using it for municipal work or really guarantee much of anything.

represent-boundaries v1.0.0 no longer available

Building a the Python environment specified by requirements.txt now returns this error:

Obtaining represent-boundaries from git+http://github.com/rhymeswithcycle/[email protected]#egg=represent-boundaries (from -r /u/apps/opengovernment-local-staging/requirements.txt (line 13))
Cloning http://github.com/rhymeswithcycle/represent-boundaries.git (to v1.0.0) to ./src/represent-boundaries
Could not find a tag or branch 'v1.0.0', assuming commit.
error: pathspec 'v1.0.0' did not match any file(s) known to git.
Complete output from command /usr/bin/git checkout -q v1.0.0:

Indeed, this repo shows no branch or tag by that name anymore; I suspect requirements.txt needs to be updated:
https://github.com/rhymeswithcycle/represent-boundaries