Address data does not appear to be normalized. For instance, Virginia Engineers PAC's 09/07/2012 contribution says that their primary place of business is "Richmon, VA." (To be fair, that's not address data.) It appears that the software used by many committees is providing normalization, but they normalize differently. For instance, some software normalizes on long street suffixes ("Court," "Boulevard," "Road," etc.), while some software normalizes on short street suffixes ("Ct.," "Blvd.," "Rd.," etc.) So the good news is that reports often have internal consistency that should make it easy to join all of the reports in collective consistency.

Implement the an address normalization system (presumably the USPS's API) to deal with this problem.

The only question is at what point this should be done. Is it appropriate to do this prior to saving the data and generating the JSON? Or is it wrong to alter the SBE's data? Wouldn't this mean making tens of thousands of API calls every time that the parser is run?

This might be an argument for standardizing addresses via a cruder, local function at the time of input, and save the USPS API calls to be used beyond the Saberva pipeline.

It's too difficult to pick out errors from among the noise of the reports of successes. Support varying levels of verbosity, via a logging class.

Per the to-do at the top of the file.

In the employment of this on opeva.com, create a cron job to run this nightly, after the 5 PM SBE file publication time.

Hardcoding terms like "expenses" and "contributions" demands that the terminology's accuracy be compared to SBE practice. Do so, and correct anything that's out of line.

We've almost got this problem fixed. The remaining problem is that this code:

$tmp = $report->ScheduleA->LiA;
$report->ScheduleA->LiA = array();
$report->ScheduleA->LiA[] = $tmp;

generates this error:

It is not yet possible to assign complex types to properties

I'm pretty confident that this is a SimpleXML error, since $report is a SimpleXML object. The solution is to stop trying to just append to that object, and instead use SimpleXML's own functionality (presumably addChild) to make this change to the data.

For example, all of these contribution records are null. Are there improper XML records? Or is this an artifact of Saberva? Opening up the PDF, I see six contributions, compared to the three "null" contributions recorded in the JSON. (Notably, none of those six contributions are from individuals.)

Saberva does not yet do anything with the campaign finance XML files. It should:

[x] Download all of these XML files
[x] Turn them into JSON
[ ] Load them into a database

e.g.:

ScheduleA: {
LiA: [
{
Contributor: {
@attributes: {
IsIndividual: "true"
},
FirstName: "John",
LastName: "Whitbeck",
Address: {
Line1: "116E EDWARDS FERRY RD",
City: "Leesburg",
State: "VA",
ZipCode: "20176"
},
NameOfEmployer: "Whitbeck Cisneros",
OccupationOrTypeOfBusiness: "Attorney",
PrimaryCityAndStateOfEmploymentOrBusiness: "LEESBURG VA"
},
TransactionDate: "2014-03-06",
Amount: "1000.00",
TotalToDate: "1000.00"
},
{
Contributor: {
Address: {
Line1: { }
}
}
}
]
},

Figure out why this is happening and make it stop—it's creating a series of messes.

Add docblocks etc.

This just happened:

Honest Reform PAC reports retrieved.
House Democratic Caucus reports retrieved.
House Republican Campaign Committee reports retrieved.
Operation timed out after 5001 milliseconds with 0 bytes received
830 committees
10th District Republican Congressional Committee: PP-12-00458 saved to committees/PP-12-00458.json
11th Congressional District Democratic Committee: PP-12-00366 saved to committees/PP-12-00366.json

It looks like the timeout was cause to abandon the entire process of fetching the list of reports, causing the parser to move onto fetching individual reports. The result was it stopped at "House Republican Campaign Committee," when there were hundreds more reports remaining (they're retrieved alphabetically).

I suspect that we need to have cURL retry after a failure or, if not that, just move onto the next committee.

Sometimes we're having empty strings saved as objects, which wreaks havoc downstream. For instance, this:

{
    "Contributor": {
        "@attributes": {
            "IsIndividual": "false"
        },
        "LastName": "McGuire Woods PAC",
        "Address": {
            "Line1": "901 E CARY ST",
            "Line2": {},
            "City": "Richmond",
            "State": "VA",
            "ZipCode": "23219"
        },
        "OccupationOrTypeOfBusiness": "Political Action Committee",
        "PrimaryCityAndStateOfEmploymentOrBusiness": "RICHMOND VA"
    },
    "TransactionDate": "2013-11-03",
    "Amount": "250.00",
    "TotalToDate": "448.80"
}

That "Line2": {},? It's big trouble. Line2 should be empty, but instead it's an object with nothing in it. Downstream, we expect that to be a string. Casting it as a string yields a fatal error:

PHP Catchable fatal error:  Object of class stdClass could not be converted to string

We need to prevent these bad data from being encoded in the first place. That means modifying class.Saberva.inc.php to check the output and make sure that we don't have any objects where we shouldn't. If we do, then we should replace them with empty strings.

Right now the JSON is a straight translation of the SBE XML. But it would be useful to provide the URL for the detailed reports etc. within that JSON. Figure out where to best add that data, and start including it.

Both expenses.csv and contributions.csv lack header rows. Add them.

Iterate through every contribution and build up a huge JSON file that lists every contributor, the amount contributed, and the campaign to which it was contributed.

(This is the first step towards producing individual donor records.)

Right now it's coming up null, even for individuals. I doubt it's being accessed properly via $contribution->Contributor->{@attributes}->IsIndividual.

I'm seeing 1969-12-31 as a date sometimes. Obviously, something is going wrong—fix it.

The only progress graph available right now is for parsing through the master committees XML file. From there on out, a verbose progress log is displayed. Duplicate the graphing functionality for additional loops.

They're in the source XML, but not in our output CSV/JSON. Figure out what's happening.

Appending -h or --help when running the program should display a list of options.

It's non-obvious that the reload option is required. Do three things:

• infer reload if the cache is more than X hours old
• make the max cache age a configuration option
• change parameter format from reload to --reload
• add a --from-cache option that will ignore the cache age
• document this feature

We've got header rows appearing over and over in contributions.csv, like such:

address_1,address_2,address_city,address_state,address_zip
address_1,address_2,address_city,address_state,address_zip
address_1,address_2,address_city,address_state,address_zip
address_1,address_2,address_city,address_state,address_zip

Figure out why this is happening and stop it.

On the one hand, I want to do this as a series of JSON files. On the other hand, I'm not sure that's the right approach.

I suspect that this is the line at which it starts to make sense to use a database. We don't have unique identifiers for each donor, but we'll need them. That means that each time that the parser is run, we'll be generating new random identifiers. (Or we make them something weird, like MD5s of names & addresses, which is bad for a bunch of reasons.)

We're getting a bunch of contributions and expenses CSV files that have nothing but a header row and JSON files that are nothing but []. Figure out why these are being created and stop it.

There are a few good tools for this:

https://github.com/cjdd3b/fec-standardizer
https://github.com/huffpostdata/campfin-linker
https://github.com/open-city/dedupe