I do not think it is currently possible to set the grammaticality of a form to the empty string. The Pylons validators are filtering out the empty string and returning None instead. It may be desirable to change this.

Add a new morpheme delimiter to an OLD system settings. Now any forms that contained that delimiter before it was system-recognizes as such will have out-of-date morphemeBreakIDs and morphemeGlossIDs values that will cause an IndexError in the morpheme auto-linking functionality. Make sure this doesn't happen in the new system.

Allow users to export one or more files as an archive. The input to this request would be a JSON array containing a list of one or more file ids or perhaps even a JSON search filter.

The person model could then be used to reference elicitors who are not account holders, i.e., not users.

Use case: need some way to allow forms, say, to have been elicited by someone who is not an account holder on the system.

It is possible to create an overly complex phonology foma script and crash the system by requesting that its tests be run. Fix/prevent this somehow.

The parent_directory values of the morphemelanguagemodel, morphologicalparser, morphology, and phonology tables are absolute paths. They should be paths relative to the directory holding the Pylons config file. The absolute paths make it difficult to move the databases.

We need to re-consider the nature of OLD files resources/models/objects in view of the possibility of referring to subintervals of audio files in order to sidestep expensive manual editing.

Is programmatic editing of audio files feasible in a web app like this; if so, this might be a part of a solution to this.

The inconsistency of MySQL wrt datetime precision across OSes and engines is very frustrating, and there doesn't seem to be very clear documentation about it. If this bug is to be resolved, it may mean exploring the edge cases or switching over to MariaDB or PostgreSQL.

Shouldn't a lexicon corpus be required for a morphology? Why isn't it?

Make sure that the OLD web service setup procedure works on a clean system and update the install docs.

it should...

Fails on expecting h.restrict('GET') to return a 405 from POST /forms/history/1. Returns 404 instead.

We don't want to always return all of the data on every revision for a given form, since often the user will only be comparing a couple of pairs of versions and won't need to see everything. Cf. dativebase/dative#115

Decide what to log. All db queries, all requests, a subset of the union of those?

Involves changing the SQLAQueryBuilder script. Example: Session.query(model.File).filter(model.File.parentFile.has(model.File.filename.like('%_%')))

rst2odt is possible! I need to decide how much of this to do client-side vs server-side.

Filtering by a non-relational scalar's length can already be accomplished via regex, e.g., ['Form', 'transcription', 'regex', '^.{15}$'] will return all Forms whose transcription has 15 characters.

Filtering by the count of a relational collection, i.e., getting all forms with 2 glosses or all collections with more than 50 forms is a bit more complicated: you have to use subqueries and joins. Below I show how to do it in SQLAlchemy.

Implementing this in SQLAQueryBuilder in an open-ended way may be tricky. I was thinking of allowing attribute names to have a 'dot' syntax so that a filter expression like ['Collection', 'files.count', '>', '50'] would return all collections with more than 50 forms associated. The tricky part would be keeping track of subqueries & joins. Anyways, no time for this now ...

How to get all Forms with 2 glosses

stmt = Session.query(model.Gloss.form_id, func.count('*').
label('gloss_count')).
group_by(model.Gloss.form_id).subquery()
r = Session.query(model.Form).
outerjoin(stmt, model.Form.id==stmt.c.form_id).
filter(stmt.c.gloss_count==2).all()

How to get all Forms with 2 files

stmt = Session.query(model.FormFile.form_id, func.count('*').
label('file_count')).
group_by(model.FormFile.form_id).subquery()
r = Session.query(model.Form).
outerjoin(stmt, model.Form.id==stmt.c.form_id).
filter(stmt.c.file_count==2).all()

System should suggest morphological analyses based on similarities. I.e., if you enter a morpheme in your analysis that is similar to something already present, the system should recognize this an alert you to what you might be wanting to use.

Ogg Vorbis is a lossy format that's easy to convert to and has pretty good browser coverage. mp3 and webm might be good too but the former is hard to convert to (in linux at least) and ffmpeg doesn't have obvious support for webm...

For examples, GET /forms could use the Range header to specify pagination in resource requests.

Cf.

• http://www.jmarshall.com/easy/http/#headerlines
• http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
• http://en.wikipedia.org/wiki/List_of_HTTP_header_fields

Consider writing a controller action to test a phonology against a corpus. This could be done client-side with multiple requests or using the JS-foma thingy, but server-side would be more efficient than the former at least ...

When a user is trying to compile a phonology, for example, a compile request may fail yet the binary from a previously successful compile request will remain and will be available for applydown/runtests requests. When one of these tests is then made, it is confusing: what phonology is being used? what was its state (i.e., script) when it was compiled?

E.g., /forms/2/history, /phonologies/1/compile instead of E.g., /forms/history/2, /phonologies/compile/1.

Note this requires changes to:

• routing.py
• test scripts
• docs

Makes sense, I think ...

e.g., for putting lots of tests in there ...

Cf. https://code.google.com/p/mitlm/issues/detail?id=32

Request a foma object be compiled twice in succession. flookup will be invoked twice in near succession, spawning two subprocesses. This should probably be prevented, I.e., parallel composition of the same foma-based object.

Cf. https://github.com/jrwdunham/old/blob/master/onlinelinguisticdatabase/lib/schemata.py#L553

E.g., if collection.forms is eager loaded, does that mean that, say, collection.forms[0].enterer is eager-loaded in the same query. I hope so. Test this. If not, this should probably be done ...

Create a normal file from subinterval-referencing files using the wave module. I have written a simple script showing how this can be done which essentially copies the instructions at http://stackoverflow.com/questions/2881012/how-to-get-the-contents-of-the-wav-file-into-array-so-as-to-cut-the-required-seg?rq=1.

The challenge is the ui for this. Do it automatically? If so, do we make that a config setting? Allow users to create these sub-files via the interface?

How would this interact with the authorization scheme currently implemented. How does LingSync do this (wrt client-side security)?

python-magic depends on it and some of the tests (cf. test_files.py) depend on it.

Write tests to ensure that email_reset_password() in the login controller is able to send email from the server and not just via Gmail. This requires a server that can send email (which I haven't configured yet).

Allow files to be created without needing to first convert them to base64 encoding client-side and then send them as JSON, i.e., use multipart-form-data. This is certainly more efficient; the con is that it is contrary to the "always send JSON" way of doing things currently established in the OLD. Note that WebTest has a self.app.encode_multipart method.

Give phonology.applydown four of the same UFs (e.g., "nit-ihpiyi nit-ihpiyi nit-ihpiyi nit-ihpiyi") and you'll get four identical SFs returned (e.g., "nitsspiyi, nitsspiyi, nitsspiyi, nitsspiyi"). This should not be.

One issue to remember: since the new OLD stores without alteration the MIMEtype of uploaded files, data transfer from old OLD databases will require conversion that takes this into consideration.

Note that this will involve Pythonic data manipulation since with pure MySQL commands it is not possible (for me at least).

Clearly this is easier in a document store/nosql datastore, but it can be done in a a rdb.

The simplest solution would seem to be storing UDFs in a BLOB of the form table, e.g., form.udfs. Then, udfs could contain a pickled Python dict or a JSON object. Indexing attributes of form.udf poses some challenges, but there are methods for doing it, cf. the friendfeed approach in the link below.

• http://stackoverflow.com/questions/5106335/how-to-design-a-database-for-user-defined-fields
• http://backchannel.org/blog/friendfeed-schemaless-mysql

ACTUALLY, this isn't the case. lib/SQLAQueryBuilder.py uses a left (outer) join so forms without syntactic categories will show up in a search like that described below.

This means that if you search for forms that have category 'S' or whose transcription contains a space you will not get any of the space-containing form with a NULL value for syntacticCategory_id. This is probably not the desired behaviour and we should probably warn users about it, or provide a way of achieving the desired behaviour. Maybe a different type of join?...

System was calling delete_morphology_directory instead of the model class's delete_directory method. Strange I didn't have tests to catch that ...