tunnell / wax Goto Github PK

Basically we are always running the DAQ reader with compression on. However there is the option to turn it off. If that comes up for some reason it would be nice if it didn't crash wax.

The toggle is in the run document under run['reader']['compressed']. It is a bool (or an int, if you'd like, with 0=false and 1=true).

This should work similar to injector class.

In the 1.8.8 version some extra data seems to be collected in the sum waveform.

Attached are zoomed pictures from xenon100 data where:

• Event 0 has a higher baseline
• All events seem to have 'additional' peaks in their sum
• All real peaks are also in the sum
• Events 1 t/m 4 have no higher baseline
• No difference is seen between sum and pmt sum data.

Consult:

https://groups.google.com/forum/#!topic/bson/XvobOF8yQiA

Get Celery working again

There is a trigger mode field in the runs db. Looks like:

        run['trigger']['mode'] = string field

Wax should use this mode field to decide what type of data is being written by the run and adjust itself accordingly.

Right now we want three modes for testing:

• 'bern_test_daq' - Normal peak finding. Data is usually NaI pulses.
• 'bern_test_led' - No peak finding. Triggered data. Group occurrences that come at the same time and shovel them to the output. If that's difficult you don't even have to group occurrences at the moment (will talk to LED group later what they want data to look like).
• 'bern_test_off' - Don't look for a run in the buffer DB. There isn't one. We're testing something that doesn't need the event builder but will still put an entry in the runs DB.

This should be told to cito through a control document.

Find duplicate data payloads.

This could be useful if we wanted to, for example, trigger on some different signal. It could be an extension so people didn't have to modify the main code base.

Most likely overkill.

http://stevedore.readthedocs.org/

Maybe syncdelay will help with jumps in I/O speeds?

http://docs.mongodb.org/manual/reference/parameters/#param.syncdelay

Currently, the data is set to the output data base after processing. However, there is a file builder. The idea behind this is that it's easy to store the data for, e.g., tape and later reprocessing. Currently, the file builder just pickles the file, which is a Python specific serialization routine that can't be easily used in C++.

What is better for longer term storage? Maybe another file format. The options I see are:

• Output to HDF5Output file with file builder.
  • Pro:
    • Standardized, so nice...
    • C++ and Python bindings
    • Standardly used for data analysis
    • Can convert to ROOT file
  • Con:
    • Extra dependency
    • Seems jargony.... wasn't immediately obvious how to do this
• Have no file builder and just use mongodump
  • Pros:
    • Easy API in Python (C++ also?)
    • Fewer dependencies
    • https://groups.google.com/forum/#!topic/bson/XvobOF8yQiA
  • Cons:
    • Less standard and documented that normal file formats
  • Protocol buffers (by google)
  • Pros:
    • Easy API in Python and C++
  • Cons:
    • Yet another format
    • Not typically used for data analysis

The main thing of interest is not to create our own file format (such as on other experiments).

Combine? Avoid two numpy casts.

XeDB.get_min_time asks if something is mongomock due to limitations on mongmock. This is a bug.

Make this a 2xN array. This requires fiddling with PeakFinding and math. However, it should greatly simplify some logic where the index within the array is used.

http://docs.mongodb.org/manual/tutorial/use-capped-collections-for-fast-writes-and-reads/

Would be nice if the event builder .ini objects were stored in the same DB as the reader .ini objects. A reader .ini object would then link an event builder .ini file that should be used for that data by writing the name of the .ini doc to the run['trigger']['mode'] field. The same will hopefully be done for the data processor.

Right now reader .ini files are in online.run_modes. Trigger ones could go in online.trigger_modes or something like that.

Just a wishlist thing for unification of these parts, so no hurry on this.

Delete used time ranges after processing rather than dropping collection at end.

Explore just binning coarsely instead of filtering. This can be like a low-pass filter by having an inner window that must be small and an outer window (signal width) that must have some charge over threshold. Think moving average.

pip install git+https://github.com/tunnell/wax.git
$ wax-off --help

change in cito installation

notify user if end of run and also how many events have been processed, also how much time has been

This could speed things up. Need to check dependencies to see if this is possible.