This should eliminate some of the logging junk.

It would be interesting to deal with numeric formatting by setting an attribute on the Epics object.

Something like theta.numeric_format = '000.000' will always format theta with three numbers before and three numbers after the decimal. And sclr.numeric_format = 'int' will always format sclr as an integer. These are parameters that need to be decided by the beamline scientists, but this would make formatting the numbers super easy.

Most of this has already been salvaged, but at least one thing -- Tom's awesome fly scan plot -- has not. Once we get it all, we should nuke the notebook, which is way behind the current API and not as good a teaching tool as a the docs.

mumble mumble "consensus and working code"

lup theta -1 1 10 .1 ---> lup theta,-1,1,10,.1 ---> lup(theta, -1, 1, 10, .1)

Transposing a one-sided gchat conversation with @tacaswell that he probably hasn't actually seen yet:

There should be set-able, readable objects that wrap lmfit Models. On 'set' they can be passed x and y data; on 'read' they can return a dict of best-fit values. The scan can use these values to decide which x points to sample next.

Center already does this in its way. But by putting Model in a motor-like object, you can easy integration with subscriptions. Like so:

def scan(motor, det, model)
    model = ModelObject(model)  # ModelObject can read and set
    Msg('subscribe', None, LiveModelPlot(x='motor', y='det', model=model)
    ret_mot = Msg('read', motor)
    ret_det = Msg('read', det)
    x = ret_mot['motor']['values']
    y = ret_det['det']['values']
    Msg('set', model, {'x': x, 'y': y})
    best_fit = Msg('read', model)
    Msg('set', motor, best_fit['center'])

When an Event is generated, LiveModelPlot can update the x and y data points and redraw a smooth best-fit curve, sampled more densely in x than the data itself.

Since the Model and its best fit results are driving the scan, best_fit is Level 0 data that belongs in metadatastore. If that sounds like a stretch, consider that some Models and fitting procedures incorporate randomness that would be impossible to replicate after the scan unless output from the fitting were captured as data.

e.g., {'all': func, 'event': func}

moved from Nikea/historydict#3

Using IPython parallel, probably

This will probably supercede #15

Right now they will fall through to the default exception handling, we might want to handle one or the other in a way that drops the run into a paused state.

I think around all of the yield from calls at the top level we wrap those with try...except which will snarf and then either re-raise (to let the task really die) or drop it into a paused state to stop the loop and return control.

%reset_beamline
%clear or %new_sample
%panic

I'm not sure if I like it better or not, but it is less typing and maybe also more clear. Just making a note so I don't forget.

%logit and %grabit

This might be trivial, but I'm not 100% sure it doesn't use stuff in the session manager.

Dark frames and white fields could be taken adaptively: if the beam current changes too much, take a new white field.

Here's something that just occurred to me.. Every N events, it would nice to reprint the column names. I'm thinking on the order of every 50-100 events, so that you will most likely always have a row on the screen where the column names are present

Make sure sure adaptive_scan in docs actually works. @tacaswell will help me out here.

One way is a list subclass with methods on it. There are many other ways that seem about equally good.

Make one of the startup files explicitly check all of the connectivity at IPython start time, ex check olog, mds, fs, ...

related to #52

First-order: Make sure nothing raises DatumNotFound
Second-order: Validate and then hash contents and save attributes in new FS collection.

Delay RunStop until all detectors are done. Therefore, all data should be available to validate and we ensure that we don't start a new run and screw up filenames.

For now, act under the assumption that when RunStop is written detector is about done so we can start validating its data in chronological order.

using pyepics directly

@tacaswell

The queues are processed in permutation P_{4123} of the correct one.

A command that causes main thread to ask the user if they are sure they want to start the scan and wait for input before proceeding. This is maybe a minor variation on hard pause.

handling scan_id, sample, etc.

include a prompt if into is missing

lots of cool stuff that's easy to forget

a partial list to start:

• Runs can be paused, aborted, and resumed with controllable checkpoints.
• In addition to the built-in scans, users to specify custom scans and "macros" in a simple, procedural way. (You can use for loops!) There are many documented examples.
• You can register new commands with the RunEngine -- say, to control your robot.
• You can write your own high-level "motors" that control many PVs. Examples include coordinated motion and controllers like XPD's "gas switcher" that don't really map onto a continuous single-axis motor.
• You can run collection from a Python script, no IPython involved. This will provide easy integration with beamline-specific control GUIs.
• Optional automatic data export at the end of each run. This paves the way for prompt HPC.
• Live plotting
• Adaptive scans (sampling fast-changing regions more). There are ready-to-use functions and examples for writing your own.
• Automated tests ensure stability of the project going forward.
• Documentation, including:
  • how to configure new motors and detectors
  • how to write macros
  • how to start agents that pause or abort the run when danger or failure are detected
  • how to use a PV to trigger pause or abort
  • how set up customized live plots
  • how set up in situ analysis on a parallel process

Holding off on these until the algebra is ready

Various docstrings and docs claim that certain messages and message sequences are not allowed. We don't actually validate all of them yet, so the RunEngine is likely to fail in nasty ways if users break the rules.

A running list of things to validate:

• 'checkpoint' must not come between 'create' and 'save
• 'subscribe' cannot subscribe to 'start'
• 'save' must follow 'create'
• 'deconfigure' must follow a 'configure' with the same target

Anything you can do to the run engine needs to be a valid command (including creating run start/run stop documents).

One use case is queing up N scans into a mega-scan and still having pause work right and working with the robot to change the sample, so you need to change the sample, which means you need a new run start ect.

Imagine: RE(itertools.chain(ascan, dscan, ct))

To prevent issues like #70

Copy from xf23id config.