acdha / restful-bag-server Goto Github PK

Can commit return a 200 iff a bag is complete and a 4XX code if not all files have been uploaded.

In addition to the PUT method to start a manual validity check it might be worthwhile to take a hint from cijoe and have a uri for last full validity check, independent of specific job. So it would return:

• 200 - OK (last check was successful)
• 412 - PRECONDITION FAILED (like cijoe) or 500 error??

Could include last modified headers for when job was completed.

A payload could be retrieved with specific errors.

We need appropriate rel values for links - see http://tools.ietf.org/html/rfc5988#section-6.3.

The BagIt Spec is a little unclear on bag-info.txt, but the most liberal interpretation allows for repeated fields and ordering being significant. The json map may not be the optimal representation for a bag-info.txt.

Should a 400 return if manifests differ. What about re-ingesting manifests, or should that return a 409

Here's the commit testing I am putting into my sample server:

1. Ensure bagit and bag-info.txt exist and are parsable.
   • fail if any cannot be read
   • other tag files are not checked
2. Compare all manifests to ensure they cover the same files
   • fail if there are any differences
3. Test for existence of all files listed in manifests
   • fail on any missing
4. Test for extraneous files
   • fail on existense.

Build a test suite, possibly using the client library from #6, which validates expected behaviour particularly as regards the SHOULDs and MUSTs in the draft

How much detail should we keep for the validation history? Do we want to consider a standard form for a report?

Initial off-the-cuff suggestion: HTTP ranges on PUT to allow resuming transfers

Currently the spec defines certain elements as Atom, which introduces a fairly complex toolchain. Google's JSON-C seems like an attractive alternative:

http://code.google.com/apis/calendar/data/2.0/developers_guide_protocol.html#JSON-C

Simple Python prototype which serves a directory of bags, generating resources like manifest as needed.

Possible option so that any specific versioning scheme is unimportant:

GET

• always returns latest bag, dependable regardless what server supports
• could potentially have tag file for version history which clients can ignore. those URI's would serve a conflict or 301 to preserve history as below.

PUT

• 409 Conflict if versioning is not enable. Another possibility is 301 (Moved Permanently) which http spec states "If the server desires that the request be applied to a different URI, it MUST send a 301 (Moved Permanently) response" for PUT
• 200 OK if versioning is enabled and successfully accepted

This should be a simple directory which could be used for client testing with a simple python -mSimpleHTTPServer

Should the spec say anything about supporting DELETE to /contents/data prior to a commit? Mike Smorul raised this question at the 05-13-2011 telecon. The theory is that DELETE could be useful for interactive assembly of bags.

Much of the content at LC resides on an HSM. To get access to the content, it must be staged from tape to spinning disk. This will take an indeterminate amount of time. A mechanism should be specified for the server to indicate to client that content is not immediately available, but will be at some point in the future, and for the client to check when the content is available for retrieval.

DELETE is allowed for entire bags, what about individual tag files, data files, manifest files?

How will bags of small files perform. Some concerns/options:

1. keep-alive
2. Option to stream a whole/partial bag as tgz/zip

During the latest conference call, the question of endpoint discovery came up. We should consider exactly how people intend to use this protocol to determine whether we in fact do need a way to retrieve a list of endpoints and, if so, how that should work.

• Efficiently handling small files for retrieval or upload (keepalives?)
• Bulk downloads - convention for content.zip, Accept: application/x-tar, etc.