Bulk import about dataportals.org HOT 18 OPEN

@marks the underlying DB is basically a spreadsheet so if the data is in the right structure one can copy and paste into the spreadsheet.

from dataportals.org.

Right - it's just a bit unclear without digging into the code where the underlying db lives. The only clear path is through the google spreadsheet :-\

For example, I can probably get you a description, lat/long, etc. for all these data portals. A lot of the ones on the map are currently out of date

http://api.us.socrata.com/api/catalog/v1/domains

from dataportals.org.

@marks this is a create idea and much appreciated. Let me know if you need any further help.

from dataportals.org.

What's the best way to get this started? Provide the content that's in the google form as a spreadsheet?

from dataportals.org.

@marks I'll create a Google sheet and share it with you. There are some differences between the online form and what's published that I'm discovering as a new Editor of DataPortals.org. I'll try and make things efficient for both of us. (May take a couple of days).

from dataportals.org.

@rgrp @mattfullerton please review this cleaned up copy of the database - personal information removed. If you think this is OK to ask @marks to fill out, say so here, then Mark can take a copy and share back here.

Validation rules appear to have changed over time. I've added validation based on the current form.

from dataportals.org.

@marks You'll note in the database, columns for api_endpoint, api_type, and full_metadata_download with descriptions at http://goo.gl/PfHswi

Is there a standard way to access Socrata to download the full metadata?

from dataportals.org.

Socrata portals have a data.json which include a description of each dataset and from which a count could be derived. Is that what you're asking?

Mark Silverberg
(m) 512 826 7004
http://twitter.com/skram

On Aug 26, 2015, at 1:34 AM, Stephen Gates [email protected] wrote:

@marks You'll note in the database, columns for api_endpoint, api_type, and full_metadata_download with descriptions at http://goo.gl/PfHswi

Is there a standard way to access Socrata to download the full metadata?

—
Reply to this email directly or view it on GitHub.

from dataportals.org.

The ideal case for the last one is a dump of all the metadata for all datasets, i.e. the same as what you would get if you grabbed the metadata from each one individually. If it is that or nearly that I would put it in. If it isn't possible, leave blank.

from dataportals.org.

@mattfullerton I might have misspoke. Wont be able to get all of this information out of one single API but I am committed to working with the right people to compile this and get it contributed. Do you literally want a CSV with the different fields that are marked as "In Production DB" or what?

from dataportals.org.

My take is a row per portal in a CSV file or equivalent will be fine. I didn't expect to go to one place and get catalog contents for many portals. Just complete as much as you can.

from dataportals.org.

Sorry if I was confusing things here - I was only explaining what goes in the full_metadata_download field. The list of catalogs can be on whatever format you like. CSV would be the easiest.

from dataportals.org.

@marks @Stephen-Gates @mattfullerton are we good to go here? It would be great to get this imported.

from dataportals.org.

@rgrp - just to be clear, you want folks to submit PRs to the portals.csv file in this repo?

from dataportals.org.

@marks yes - or you can give us a CSV and we'll perform the merge for you :-) - obviously you doing the PR is probably preferable.

from dataportals.org.

@marks is this complete?

from dataportals.org.

I've made some corrections but run into other issues and time constraints. Feel free to close this issue if you want to keep the backlog clean

Thanks

Mark Silverberg
(m) 512 826 7004
http://twitter.com/skram

On Mar 14, 2016, at 8:20 AM, Rufus Pollock [email protected] wrote:

@marks is this complete?

—
Reply to this email directly or view it on GitHub.

from dataportals.org.

@marks we can keep it open for the time being - also if you are busy any chance you could you post the list of portals you wanted added somewhere as csv and someone else can help integrate?

from dataportals.org.