Conflict error for NYC action on bil about scrapers-us-municipal HOT 3 CLOSED

Okay, this is what I think is going on.

It takes multiple hours to scrape nyc bills and in that time there may be some bill activity added to the legistar system. This can change the sort order and cause the same bill to be iterated through. This only sometimes happens, of course.

Not sure what the best way to handle this is. Thoughts @mileswwatkins @rshorey

from scrapers-us-municipal.

This may actually be a deeper question. Who's responsibility is it to make sure that every bill object in a scrape is unique. The scraper or pupa? cc @jamesturk @jpmckinney @paultag

from scrapers-us-municipal.

I think this needs to be done by the scraper. Pupa throws errors for non-unique stuff, but it has no way of knowing what to resolve to if two things are not unique.

For NYC, perhaps it'd make sense to pull the whole bill list first, store it, and then access the individual bills so there's less time for things to change. Would that make sense in this case?

from scrapers-us-municipal.