influence-usa / lobbying_federal_domestic Goto Github PK

If you compare the original copy of this document to the transformed data, it seems that there is a mistake in how the various fields related to expense are calculated. Specifically, the expense_less_than_five_thousand field is false, implying that the expense is greater than 5000, while the expense_amount field is null. In the original document, neither field was check and expenses was left blank, which seems to imply that there were no expenses (rather than implying there was an unknown expense that was greater than 5000).

I was under the assumption that the json itself would have a key corresponding to some sort of ocd object.

not sure the source of the file, need to look into it

Unsure how the quarter didn't transfer over.

The scraper seems to be missing out on any other line besides the first when scraping the specific issuses from lobbying reports. View this form and then check the json you have locally to see that the line HR 4310, National Defense Authorization Act, provisions relating to tanks and communications just doesn't show up. My local file path for this file is /home/zmaril/data/sopr_html/2013/Q4/f7c1c383-cbf0-4233-b094-e1324b78d042.json.

There are several fields that are common to both the reports and registrations forms that are expressed in different ways in each. Calling the fields the same thing in both instances would make working with the files much easier to understand and program against. Besides the following three fields, everything else matches up as well as it can and is easy to work with:

This html is garbage and produces a nulled out json file. What's happening here?

SOPR LD forms have a "Filing ID"

<Filing ID="B0584458-99B9-4BDC-89B9-0C2098B0E28C" Year="2000" Received="2000-11-01T00:00:00" Amount="40000" Type="MID-YEAR REPORT" Period="Mid-Year (Jan 1 - Jun 30)">

things to investigate:

• are they unique at any scope?
• what scope (yearly, quarterly, by filer)?
• do they appear to have any semantic information?

Write a simple parser for the houses_and_agencies field on lobbying activities in reports. The fields come prepopulated and so we know all the possible leaves of the parse tree that could exist. Should be straightforward to do.

What seems to be the source? What are the possible values? Why does it seem to be okay to have LobbyistCoveredGovPositionIndicator = "COVERED" while also reporting OfficialPosition="N/A"

eg:

<Filing ID="B0584458-99B9-4BDC-89B9-0C2098B0E28C" Year="2000" Received="2000-11-01T00:00:00" Amount="40000" Type="MID-YEAR REPORT" Period="Mid-Year (Jan 1 - Jun 30)">

(snip)

  <Lobbyists>
    <Lobbyist xmlns="" LobbyistName="EPPENBERGER, RYAN A" LobbyistCoveredGovPositionIndicator="COVERED" OfficialPosition="N/A"/>
    <Lobbyist xmlns="" LobbyistName="EPPENBERGER, RYAN A" LobbyistCoveredGovPositionIndicator="COVERED" OfficialPosition="N/A"/>
    <Lobbyist xmlns="" LobbyistName="DELMONTAGNE, REGIS J" LobbyistCoveredGovPositionIndicator="COVERED" OfficialPosition="N/A"/>
    <Lobbyist xmlns="" LobbyistName="NUZZACO, MARK J" LobbyistCoveredGovPositionIndicator="COVERED" OfficialPosition="N/A"/>
  </Lobbyists>

Registrations for client-registrant relationships prior to 1999 are not available through SOPR.

Below is the firm and client we were looking to find a filing for.

{
    "_id" : ObjectId("5359516b6e955232f6d25b32"),
    "Received" : ISODate("2000-08-14T00:00:00.000Z"),
    "Period" : "Mid-Year (Jan 1 - Jun 30)",
    "Registrant" : {
        "RegistrantPPBCountry" : "USA",
        "RegistrantID" : "41454",
        "RegistrantCountry" : "USA",
        "RegistrantName" : "Williams and Jensen, PLLC",
        "GeneralDescription" : "Lobbying law firm",
        "Address" : "701 8th Street, NW\r\nSuite 500\r\nWashington, DC 20001"
    },

a la http://soprweb.senate.gov/index.cfm?event=getFilingDetails&filingID=7b2332f3-d438-4d0a-a9f9-741081eab317&filingTypeID=78

example:

xmlParseCharRef: invalid xmlChar value 16, line 277, column 44
/home/blannon/dev/influence-usa/lobbying-federal-domestic/data/original/house_xml/LD2/2013/Q4/300616796.xml

this was a  in the LOBBYINGDISCLOSURE2/alis/ali_info/specific_issues/description field, which was either meant to be a linefeed or a DLE char.

Need to have a full test set, finding a set of documents that, together, offer at least one example of a non-null value for every field.

Are their general search options--or do searches have to be assigned specific fields? (For example, "issue area" etc.)