mooneyse / lb-loop-2 Goto Github PK

This should be a straight forward step, writing a parset in Python and calling NDPPP.

Do we use WEIGHT_SPECTRUM or WEIGHT? And does it make sense to include this while adding things (not averaging)? e.g. When combining weights, should I take their intersection or average? When adding data, what role should the weights play?

Some time axes come out with 1029 timesteps after interpolation and when matched this other time axes. Is this correct? It looks suspiciously like one was added by mistake but it might be to do with the fact that not all time axes in solution tables are the same. Check it out.

The relevant line in the code is here:

Now, evaluate_solutions will find the coherence metric using the first frequency axis, even if there are multiple axes. Change this so it loops through them all. Then do one of the following:

• Write the frequency to the master text file as well, and when building HDF5s, introduce the functionality to use different HDF5s for each frequency band. This requires reformatting the MTF, changing dir2phasesol. Also, update_list would need to be fixed. This would get messy if not all HDF5s have the same frequency bandwidth chunks.
• Have evaluate_solutions average the coherence values across the total bandwidth to get one number for the coherence, and write this to the MTF. This is preferred at the moment.

dir2phasesol copies across amplitude and phase solutions that are in one HDF5 file. In reality, the amplitude and phase solutions will be in different HDF5 files though. As it stands, to copy across amplitude solutions would require running this function again with the amplitude HDF5 file, but corresponding to the phase validity as written to the MTF. Is this the desired behaviour? That will not be the file that was evaluated by the evaluate_solutions function, so some tinkering would be needed.

Perhaps it would be neatest if loop 3 wrote one HDF5 file containing two solution sets: sol000 which has phase000, and sol001 which has phase000 and amplitude000 corresponding to the amplitude solutions.

This line is a good starting point for this issue:

This is the relevant line in the code:

There is a copy of loop 3 in this repository. Importing it and trying to run it from within loop 2 causes an error like this:

    File "/data020/scratch/sean/run1/git/long_baseline_pipeline/bin/loop3B_v1.py", line 736, in main
    spec_info = casatb.table( vis + '::SPECTRAL_WINDOW')
    RuntimeError: Table SILTJ135044.06+544752.7_L693725_phasecal.MS-eef114.MS::SPECTRAL_WINDOW does not exist

This does exist though.

The reason is that I am not calling loop 2 from the same directory as the MS (even though the full filepath was given), and loop 3 involves making a directory and moving into it. It is something to do with changing directories and should be easy to track down.

As a workaround, I've been running loop 2 and then loop 3 from the command line.

update_list calls evaluate_solutions to evaluate the new and improved solutions (which are a combination of the initial solutions and incremental loop 3 solutions). evaluate_solutions will write to the MTF. However, now there are two lines in MTF for this one direction, where the other line is just for the incremental solutions. They may also be good. In order to make sure these improved solutions are used preferentially, one of two things has to happen:

• Get the MTF to be read from the bottom up. Then the solutions which are the combination of initial and incremental values will be used first. This is not great.
• If there already exists a set of solutions in the MTF for the same direction, evaluate_solutions could remove them, so there is always only one per direction. This assumes that the most recent HDF5 will have the best solutions. Although perhaps they could be compared and the one with the better solutions could be kept.
• Write an additional parameter to the MTF, like an increasing counter, and have dir2phasesol use the HDF5 with the highest value in the case of multiple HDF5s for one direction. This is the neatest option.

The relevant line is here:

It is critical that all HDF5s use the same reference station. Otherwise, taking solutions for different stations from different HDF5s would be incorrect.

So a check needs to be introduced for this (by seeing which station has solutions of zero), where it raises an error if it is not the case. Better still, write a function which converts the solutions from one reference station to another, and do this if necessary. (First, make sure the new station we are using as the reference is present in all HDF5s.)

Use the logging module to write plenty of information to a log file and to terminal.

This is the line where a new HDF5 solution set is created:

It would be nice if using LoSoTo, it wrote to the history of the MS where the solutions from each station came from. This essentially is produced in a working text file:

It would just be a case of writing this information to the HDF5.

The direction for the new HDF5 is passed to the command line at the moment. I envisaged there being one MS for the field, so passing a list of positions for the sources made sense, and then it produces an MS for each direction as it applies the relevant solutions in those directions. However, in the case of the current testing, we have the MS centred on the source of interest and averaged. Make it so that, if no directions are passed to loop 2, it just reads the pointing from the MS. Here is the TODO in the file.