Return objects instead of JSON about propublica-congress HOT 10 OPEN

Yeah, I'd be open to returning native types in the API.

from propublica-congress.

Let me know if there's something we can help with!

from propublica-congress.

I'm curious about your experience with Marshmallow. It looks like there's a lot of schema defining to be done, and I worry about having to recreate all the underlying data models in the API, and then being behind when it changes.

I could also use some help thinking through what the interface should look like for a response. If I get a member, what does that class look like? What can I do with it? What does it do with the incoming data? All of that.

Very much appreciate more hands and eyes on this.

from propublica-congress.

You are definitely right about the schema defining. It can feel redundant writing the classes and the mallow schema. I haven't seen anything that lets you do it as succinctly as Django + Django REST. Marshmallow is just nicely agnostic to framework with the same serialization power.

As for whether to model objects or not, I suppose getting behind the API when it changes is a risk, but frankly it's worth it to me to have native types. And really I kinda don't mind the idea that changes to the API have to be explicitly reflected in its wrapper.

Re: interface, I haven't started mucking with class methods yet, but that's the promised land, I think. Basically you'd get a fully fledged python library out of their API.

There is a world where these things could develop in parallel. You've got all the API methods returning JSON. I've got (a couple) serializers to turn that JSON into native types. Would be nicer if those two functions lived in the same lib, tho, but sure that's why we're chatting!

from propublica-congress.

Looking through the API docs, it looks like everything is coming back as a string, even in places where it could be returning numbers (like years and counts). Might be worth talking to @dwillis about solving some of that upstream, which would make our lives easier.

I'm going to try to tackle #11 today, which should also make a lot of this easier to manage, since everything won't be piled in one file.

from propublica-congress.

OK #11 is done. @dwillis should I open an issue on congress-api-docs for native types?

from propublica-congress.

@eyeseast Yes, please do.

from propublica-congress.

I've been thinking a little about what I want out of class objects (separate from what anybody else might want). Here are a couple use cases I have in mind:

Get members, then get one member, then find bills they sponsored:

# get and filter members
>>> mass = congress.members.filter('senate', state='MA') # Warren, Markey
>>> warren = next(m for m in mass.members if m.last_name == 'Warren')

# get basic attributes on demand, with types converted
>>> warren.in_office
True
>>> warren.date_of_birth
datetime.datetime(1949, 6, 22, 0, 0)
>>> warren.twitter_account
'SenWarren' # still a string

I'm most interested in getting related data:

>>> warren.bills # all Warren's bills, or a bill client?
>>> warren.votes # etc

The subqueries can get harry pretty quickly here, so this is where I really want to be careful and think through the interface. I'll add more examples here as I think of them.

from propublica-congress.

Here's another piece of the response object puzzle, sticking with member endpoints: Different API calls give back different amounts of data per member.

Get a list of members returns metadata about the query (num_results, congress, chamber) plus a list of members.

Get a specific member returns the member, plus member roles.

Members by state/district returns a list of members, but with less data.

In all cases, each object in the list comes with an API URL, so a response object could have a method to fetch the remaining data.

from propublica-congress.

How's the progress with pulling related data?

from propublica-congress.