opencivicdata/ocd-division-ids#10

Still on ScraperWiki

Found a PDF - to determine whether easy enough to parse: http://www.community.gov.yk.ca/pdf/loc_govdir.pdf

Found a source: http://web5.gov.mb.ca/Public/municipalities.aspx

Scraperwiki:

• Alberta municipal officials
• Québec municipal officials (check commented-out code)
• British Columbia municipal officials (we will get API access, add regional districts)

New:

• ca_mb_municipalities (see @todo)
• ca_nb_municipalities
• ca_nl_municipalities
• ca_ns_municipalities (see @todo)
• ca_nt_municipalities
• ca_pe_municipalities
• ca_sk_municipalities
• ca_yt_municipalities

There's a broken scraper at https://classic.scraperwiki.com/scrapers/quebec_council/ I don't know if it ever worked, to be honest, so feel free to start from scratch.

https://github.com/opencivicdata/municipal-scrapers-us/blob/master/denver/utils.py

All representatives should have a district_name for CSV exports to work nicely. This will automatically be completed once all Scraperwiki scrapers are converted to Pupa.

https://travis-ci.org/opennorth/represent-canada-quality

Use Excel from http://www.municipalaffairs.alberta.ca/am_municipal_codes.cfm

In other words, export it as JSON that looks like the JSON produced by ScraperWiki.

http://www.toponymie.gouv.qc.ca/ct/toponymie-municipale/municipalites-arrondissements/arrondissement.aspx

A LaSalle scraper was written for LaSalle, Ontario by mistake. Write another for the borough of Montreal.

http://opendata.peelregion.ca/data-categories/regional-geography/ward-boundaries-%282010-2014%29.aspx

• ca_ab_edmonton (4811061)
• ca_on_windsor (3537039)
• ca_ab_wood_buffalo (4816037)
• ca_qc_cote_saint_luc (2466058)

The scraper is broken, so may as well port it as part of the fix:

https://classic.scraperwiki.com/scrapers/saskatchewan_mlas/

See update_from_data_source in models.py. We will need to move this code to transition to Imago.

https://github.com/rhymeswithcycle/represent-reps/blob/master/representatives/models.py#L144

See opencivicdata/pupa#31

See opencivicdata/docs.opencivicdata.org#1

May replace https://github.com/opennorth/scraperreports

http://www.beaconsfield.ca/en/your-council.html

treat the 'note' field for contact_details as the 'type' field for offices, eg: 'constituency', 'office', or 'home'

capture position of representative

Anything to do before merging?

• ca_sk_saskatoon http://apps2.saskatoon.ca/app/aForms/councillor.aspx
• ca_qc_saint_jean_sur_richelieu (on councillor's individual page)
• ca_on_milton (some have emails, some use webforms)
• ca_on_peterborough

Shapefile was recently added to Represent

Shapefile was recently added to Represent

Still on Scraperwiki. Might be fixed by #42.

Work through the broken scrapers at http://scrapers.herokuapp.com/

• ca_qc_mercier
• ca_qc_montreal_est

Ignore:

• jurisdictions ending in _municipalities
• ca_on_toronto (website frequently fails)

Previous issue content

Reserved for Matthew Leon:

• ca_mb

Priority (already in Represent):

• ca_nb_frederiction
• ca_nb_moncton
• ca_on_richmond_hill
• ca_qc_brossard
• ca_qc_senneville

https://github.com/opencivicdata/municipal-scrapers-ca

Shapefile was recently added to Represent

It's not raising an error, but it is only scraping the mayor currently:

https://classic.scraperwiki.com/scrapers/edmonton_city_councillors/

https://classic.scraperwiki.com/scrapers/british-columbia-mlas_1/

The government website simply links to municipal websites. AMO simply lists members on its website.

AMO charges $1800 for the contact information, and AMTCO charges $300 top-level contact information.

http://www.ville.levis.qc.ca/Fr/Conseil/

The scraper is broken, so may as well port it as part of the fix:

https://classic.scraperwiki.com/scrapers/windsor_on_council/

https://classic.scraperwiki.com/scrapers/sherbrooke/

http://www.toponymie.gouv.qc.ca/ct/toponymie-municipale/municipalites-arrondissements/toponymie_municipale.aspx

Compare the QC status to the census subdivision type to see if they match.

• ca
• ca_ab
• ca_mb
• ca_nb (see also https://gist.github.com/nickspacek/9070874)
• ca_ns
• ca_nl
• ca_on
• ca_pe
• ca_qc
• Waterloo Region
• ca_ab_calgary
• ca_ab_edmonton
• ca_ab_grande_prairie
• ca_ab_strathcona_county
• ca_bc_victoria
• ca_mb_winnipeg
• ca_nl_st_johns
• ca_ns_halifax
• ca_on_brampton
• ca_on_hamilton
• ca_on_kingston
• ca_on_kitchener
• ca_on_london
• ca_on_mississauga
• ca_on_ottawa
• ca_on_sault_ste_marie
• ca_on_waterloo
• ca_on_windsor
• ca_pe_charlottetown
• ca_sk_regina

Once all the scrapers are in Pupa:

• Delete scraperwiki_spec.rb from represent-canada-dashboard
• Go through the @todo in represent-reps

http://www.ville.saint-jean-sur-richelieu.qc.ca/conseil-municipal/membres-conseil/Pages/membres-conseil.aspx

Add a mapping. http://donnees.ville.montreal.qc.ca/fiche/arros-liste/

https://github.com/opencivicdata/imago

See also https://github.com/opencivicdata/opencivicdata-api that brings all API components together.

Probably just requires a quick fix. Up to you whether it's worth the effort to switch to Pupa now.

https://classic.scraperwiki.com/scrapers/ontario_mpps_1/

See http://scrapers.herokuapp.com/warnings/

Valid warnings have no checkbox:

1. ca_nb_moncton (1) voice (same type and note)
2. ca_on_markham (3) multiple urls
3. ca_on_milton (7) some use webforms
4. ca_on_oakville (12) multiple emails
5. ca_on_peel (28) post (handled with boundary_url)
6. ca_on_pickering (2) multiple urls
7. ca_on_sault_ste_marie (1) email (Frank Manzo, Ward 6 has none)
8. ca_on_toronto (1) voice (same type and note)
9. ca_pe_charlottetown (1) email (Edward Rice, Ward 1 has none)
10. ca_qc (26) email (many people do not yet have emails)
11. ca_qc_brossard (1) email (Francine Raymond, District 3 has a typo in her name)

Resolved:

• ca_mb_winnipeg (local part of email is in email webform URL)
• ca_ns (emails are JavaScript-encoded)
• ca_on_london (emails are in JavaScript)
• ca_on_milton (some have emails, some use webforms)
• ca_on_newmarket (loads of warnings, check the scraper)
• ca_on_peterborough (emails are on same page)
• ca_qc_saint_jean_sur_richelieu (emails are on councillor's individual page)
• ca_sk_saskatoon (emails are on http://apps2.saskatoon.ca/app/aForms/councillor.aspx)
• ca_sk_saskatoon: email, Ward 3, Ward 6

I have a very complex script that scrapes URLs of municipalities across Canada from the FCM's website, and matches them to a census subdivision according to their name and province. It produces this file.

There are other sources of URLs, such as CivicInfoBC. I wouldn't base myself off the script I'd written, and instead just start by trying to do straight name matching against the list of census subdivisions for BC (just limit yourself to those whose code starts with "59" to get only BC). There are only a few subdivisions with the same name in BC, in which case you need to match on name plus type.

• Langley
• North Vancouver
• Esquimalt
• Sechelt
• Okanagan 1
• Alert Bay

If this works, we can add others for other provinces. You can write the script in a fork of ocd-division-ids.

Shapefile was recently added to Represent