markmuetz / omnium Goto Github PK

Currently this is done programmatically, which works but doesn't allow for e.g. showing a (network) graph of the workflow at a later stage. Will need to think about how to add this to the config and add/test it.

setup.py + pip + adding to docs (see #6).

Not just take the midpoint as it does now (although this is correct for the iUM levels at the mo.)

So as to stop it from being accidentally overwritten

After you've e.g. done a remote sync

Like a precip unit converter, as well as a few more simple outputters.

All the files that I can lay my hands on that are useful for working out the UM settings, including fcm revision numbers etc if I can get these.

Basic test suite. Ideally test each of the commands exposed to user plus a few of the various command line switches. Functional tests of various processes and checking that I'm handling config files right etc.

It inhibits other classes' use when running from a shell.

Make them proper python packages, with a defined list of analysis classes at the top level.

Simple first pass of docs. readme/installation/quick start as minimum. Sphinx for docs (stormtracks for setup).

Will build on #7, grabbing metadata and copying all relevant files to /nerc/... As in #7 comment, will need to be able to see /nerc, /work, so means I'll have to switch to PP nodes I think.

Currently threed and threed-monc are sep commands. Turn into one. N.B. linked to #26.

They should be smarter about which file types etc they accept, and the input/output they consume/create.

whereas they are OK on zerogravitas

Plus super simple way to read it (i.e. not using sqlalchemy).

Although not strictly a problem - can still create/access them with convoluted filenames like:
/nerc/n02/n02/mmuetz/um10.5_runs/20day/u-af095_64x64km2_1km_no_graupel/results/../work/20000101T0000Z/atmos/atmos.438.1.nc
not ideal.

will need to provide data to all processes that need it.

Quite a few moving parts. Need to generate a list of files on the remote computer, compare with the currently held local files, transfer files. Relies on being able to sync node dag across computers first.

a tests dir, accompanied by omnium test command.

It's not great to have e.g. 3 cylc tasks logging to the same file.
Could do it like <comp_name><task_name><task_id>.log or similar.
Extra functionality to do rotation perhaps too.

Version numbers (see stormtracks for ideas).

Would be useful for repeatability. In its simplest form could just by a custom logger that only saves command line args.

On comp A: batch0 produced, batch1 processed from batch0. On comp B: batch1 synced and process changed. Now you can't gen-nodes --force --regen because there are no init nodes. Hack for now is to make the init nodes (touch atmos.000.pp5, atmos.000.pp5.done). More generally, need to handle processes being changed in a more sensible way.

Create src/python/<proc_name>.py, fill in some of the boilerplate code for creating a process. Offer some options about which class to base it on (e.g. --baseclass=IrisProcess or similar).

e.g. gen-node-graph -> gen-nodes

Using inbuilt python logging. Needs to be smart about transferring these on sync of remote files too.

State of omnis/ dir, version of omnium etc.

Regenerate creates a new dag. All other classes are set up with a dag when they are created. Manifested itself as an OperationalError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (OperationalError) database is locked u'UPDATE groups SET status=? WHERE groups.id = ?' ('done', 3) when doing a self.dag.verify_status(update=True) whilst running a proc_eng.process_batch(batch) after doing a dag = NodeDAG.regenerate(config, process_classes).

omnis/ should just hold the config info needed to run omni commands, not its logs too. (will make them better candidates for version control)

There isn't much point in having analysis settings in the rose suite, move these into the classes. N.B. rose suite will still have run settings like e.g. order of analysis, whether particular analysis is required.

i.e. work out a way to stream the data without loading it all at once.

e.g. get by ID or just show info for all nodes with name.

• Add new omnium 3D command
• Handle multiple vars sensibly
• Allow config of all opts through interface
• Simple repr of 2D fields