NOTE: requested by CGI, Chicago and other customers.

Command Line from Linux:

Error sending email to: [email protected] addressjava.lang.IllegalArgument

Relevant line from config.json:

"adminEmail": "[email protected]",

Currently porting data requires that the schemas of source and destination datasets are identical. We should enable porting to a destination dataset that has subset of columns of the source.

SmartUpdate will allow customers to efficiently do replace operations in DataSync on very large datasets (1 million+ rows). SmartUpdate works by DataSync sending a zipped CSV file with all data over FTP and then the Socrata platform determines which rows have been added/updated and only publishes those to the dataset. This leads to dramatic gains in efficiency/performance so customs can have uploads complete faster and data will also be geocoded and indexed for searching much more quickly. SmartUpdate will enable using the "replace" method for datasets with upwards of 1 million rows which should remove the need for customers to handle determining which rows have been added/updated/deleted and using upsert. Instead it is as simple as dump all the data into a CSV and upload the CSV through DataSync "SmartUpdate".

This can be remedied by re-generating the command to execute with scheduler upon opening an .sij file.

If the the header row of a CSV file has leading or trailing whitespace around any of the column names the upload will fail for those columns. This can be fixed by simply trimming off whitespace around each header column name automatically before publishing the CSV.

When running jobs to publish data to certain datasets (with specific sharing configurations) this error occurs when using 'upsert' in DataSync:

com.sun.jersey.api.client.ClientHandlerException: org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "userEmail" (Class com.socrata.model.importer.Grant), not marked as ignorable

This is a bug in the soda-java library which was fixed in version 0.9.4

This issue is related another issue: #14

There should be an upload method call append/upsert/delete which allows specifying a column in the CSV with header :deleted and then having any value of true in that column trigger a delete of that row. Be sure to notify user if there is no Row ID set in the dataset. Include a Help icon explaining how this works.

Somehow we should create a "How-to" guide for Port Jobs. This should be a lot like this article:
http://support.socrata.com/entries/24241271-Setting-up-a-basic-DataSync-job

This would allow for creating shortcuts that not only launched DataSync but also opened a specified job file, similar to what can do with many applications that open/edit files. I know you can specify a job file when running a job but there could be situations where someone wants to open it before running it interactively.

It is possible this can already be done and I am just not figuring out how to do it.

Thanks.

In my case, the field in the original dataset was mm/dd/yyyy but in the copy, time of day (midnight since the data had no times set) was shown.

Could be either as settable parameters or use the domain in the credentials.

Repro steps:
use upsert or replace method to upload a CSV file where one value in the Row Identifier column is the wrong datatype (i.e. text in a number column)

Expected: Unknown number format 'X' (line X of file)

Actual: Cannot find column X...

A dataset will allow, although grey out, an invalid URL but DataSync produces an error message. My preference would be that if the portal, itself, allows the value, DataSync allow it. Dropping invalid values might be ok if necessary.

I am not in a position to give the specific example to reproduce the error since it is a private dataset under development (happy to provide the link to Adrian or someone else at Socrata but cannot post it quite this publicly) and I am changing the field to Plain Text in order to be able to proceed for now. However, I suspect it would be easy to reproduce with a test dataset. Leaving off the top level domain and using a comma instead of a period both create this error, although they are not the only conditions that produce it.

Although it is unclear within the existing UI of DataSync the 'upsert' and 'append' methods behave in exactly the same way. When an upsert or append is performed and no Row ID is set for the dataset being published to, the rows are all appended. If a Row ID is set, whether you use 'upsert' or 'append' the rows being uploaded that match existing rows will be updated.

Proposed change: When the user selects append as the publish method DataSync should check if a row ID is set in dataset. If one it set have a message appear that warns the user about it. Have a help bubble explaining everything next to "Publish method" field in UI as well.

Also need to update knowledge base articles.

Relevant support ticket:
http://support.socrata.com/tickets/3342

Customers who requested this: City of Seattle, Hawaii

DataSync should give the user the option of uploading a CSV file without column headers. Have a checkbox to user checks to tell DataSync their CSV does not contain a column header row. If this box is checked DataSync should refer to the order of the columns in the CSV.

It might also be useful to give the user feedback in the UI if the column headers of the CSV file (or lack thereof) does not match that of the dataset.

Use Maven for package management. Move mail, soda-java, and org.json packages to be imported from Maven. Also update the README to reflect this change.

Presently DataSync returns an error saying 'Not found' when the authenticated user (i.e. user whos username and password are entered into bottom left fields in DataSync) does not have rights to publish data to the given dataset. Instead the error message should be '[email protected] does not have permission to publish data to the dataset with ID X'

If I select the option to publish the destination dataset, will it be public, private, or whichever the source dataset was? Thanks.

DataPort is a command line tool that allows you to transfer data from one Socrata data to another or duplicate an existing Socrata dataset. DataSync will have a new job type called a Port job where you can configure a job to do the things DataPort supports as part of DataSync.

Allow users to upload TSV (in addition to CSV files).

This was requested by City of Seattle.

Currently publishing very large files causes DataSync to crash due to running out of memory. NOTE: chunking will only be supported by 'upsert' and 'replace' for this release.

Enable normal DataSync jobs to be 'smart' about the data they publish by only publishing the rows of data that were updated or added in the CSV (File to publish). DataSync will maintain a record of the changes made to the CSV file to be published since the last publish operation was performed (i.e. last time job was run) in order to identify which rows in the CSV file were updated or changed since the last run of the job. Then when it runs the job it will only publish those newly updated or added rows (rather than all rows in the CSV as it does now). This feature will only work for append/upsert (not replace).

If you wanted to be really slick, maybe offer a set of standard options -- such as appending "Copy" or "Backup" to the source name -- in addition to the ability to enter one's own value.

This may help customers who have issues using a standard API connection due to firewall issues.

Customers who may benefit: Lombardia, CMS, Dept of Health (NY)

For larger datasets, we can update the progress of a job and report it to the user as it's running.

The line number of the problematic row is given relative to the chunk being uploaded so it does not reflect the line number in the CSV file making it difficult for the user to find the invalid row in their CSV.

NOTE: This bug only occurs when file chunking is enabled.

It's awesome that this is open source, but it would be great to have an explicit license attached.

GitHub has some great boilerplate options:
https://github.com/blog/1530-choosing-an-open-source-license

Currently the chunk size (in number of rows) as well as the filesize threshold that triggers chunking are both hard-coded into DataSync. Instead the user should be able to optionally configure these settings by going to Preferences.

The simplest way to achieve this would be to enable configuring a DataSync job by passing command line parameters as an alternative to running a saved job file. Then a Pentaho template could be created that calls DataSync in this way.

Mainly this means documenting that Maven is used for package management (rather than importing JARs as external libraries).

This will be useful to notify people if they accidentally moved or deleted a .sij file that a scheduling or other external tool was using. Currently an error is thrown but no information is recorded in log and thus auto-emailing would not occur.

Currently only append/upsert methods support uploading TSV files. Support for replace method should be added for TSV format.

This is a bad bug that somehow didn't get noticed before the last release (v0.3). I already fixed it and re-uploaded the JAR for the release.

In the case where none of the header row column names match any existing column in the dataset the user should be notified. Currently the job just dies with no error or success message when this is the case.

To optimize the upsert API calls we should investigate using the upsert CSV method directly (but this may only work for files that contain headers). This may be challenging to support with chunking.

It makes a lot more sense to have chunk size be based on file size rather than number of rows.

To fix this the user should be notified that saving the job failed and be asked to Save the job to a new location. This fix should include error handling for any issues that occur when a job is saved to a file.

This would be great to have!

Our particular use case is shapefiles (.zip). The Java API client seems to support that, per this support thread:
http://support.socrata.com/entries/28307593-Is-it-possible-to-programmatically-upload-or-update-shapefiles-

With the GUI interface, one can set certain settings and preferences once and have them applied to all jobs (unless specifically overridden by a command-line parameter, I assume). They are saved in the Windows registry or other OS-specific location.

However, this only works for a machine capable of running the GUI. In some cases, such as a Linux server accessed through terminal emulator, that is not possible. It would be great if there were a way to save these settings through the command line. Maybe some parameter that effectively said "Save all these other parameters as if entered through the GUI"? Would that allow for substantial reuse of the code the GUI uses to save these settings?

I realize this does not add a lot of security vs. using command-line parameters or a configuration JSON file but it adds a little, even if only security through obscurity and adds convenience vs. the command-line.

Thank you.

Also allow the user to select the Port publish method that would be used to load data into the other dataset (upsert, append or replace).
NOTE: this feature has been requested by customer: CGI.

Suggested by Matthew VB (from unknown client)

Explain Dataset ID, Publish Method, etc within the UI by putting help bubbles that popup explanations upon hovering.

I ran a job (interactively) and it fails silently -- meaning no alert, no apparent change to the dataset, no entry in the Log Dataset. The only signal is that the Run Job Now button becomes clickable again.

It is a Replace job and the source CSV is 76 MB. I have the chunking threshold at 64 MB. I initially had the chunk size at the default of 25,000 rows but lowered it as low as 10,000, with no success.

What other diagnostic information can I provide?

Thank you.

Currently chunking (which enables uploading very large files) is only supported by 'append' and 'upsert'.

This should be implemented by creating a dataset working copy via .copySchema method and then pushing rows to the resulting working copy in chunks. This is to prevent the dataset being in a bad/inconsistent state if a job fails part-way through.