lmorabit / lofar-vlbi Goto Github PK

Not a big deal, just opening this for future reference.

Technically the kMS solutions should be applied as fulljones.

To finalise the pipeline we need to offer an option to apply both ddf and delay cal solutions to take DATA to CORRECTED_DATA for the dpppconcat files. At the moment it only applies delay cal solutions.

This was mentioned during the busyweek I think, but lest we forget I'll open this here. The following line is problematic:
https://github.com/lmorabit/long_baseline_pipeline/blob/225aadfb9e44d87b8ab2191b95dcee43cd06dcdf/bin/loop3B_v1.py#L164
as ant is an ndarray and not a list. Because of this the try/except block fails and it always writes out "incoherent" metrics (i.e. values of 2.0).

Either

j = np.where(ant == antenna_list[i])

or casting ant to a list should fix it (the former is more elegant I think).

Line 436 should be:

addIS.control.executable = {{ long_baseline_pipeline_dir }}/bin/addIS_to_h5.py

not {{long_baseline_dir}}

It looks to me like the substeps "h5imp_ddf_map" and "h5imp_ddf" are missing from the apply_ddf list of steps (at least, fixing this stops it crashing for me!) My current pull request updates this too.

If there are bad channels, difmap copes with this by a select command e.g. select I,1,3,5,10 takes channels 1-10 except channel 4 using two ranges. If there are more than 19 ranges, an error will be generated because difmap has a hard limit (see chlist.c and chlist.h in the difmap software tree). There are two possible solutions: (i) make a second change in the difmap software tree (in addition to the change in corplt.c), or (ii) modify selfcal_difmap.py to only allow the first 19 ranges. Both are hopefully straightforward changes, but we should discuss which approach is better.

I have an issue when running a dataset through the Pipeline. It seems to be not able to load the mapfile? Here is the last part of the Logfile:

2018-09-03 16:48:09 DEBUG genericpipeline.executable_args: *********** finished ionosphere predictions ***************
2018-09-03 16:48:09 INFO node.lonode001.python_plugin: Total time 55.1168s; user time: 13.7200s; system time: 2.0040s
2018-09-03 16:48:09 DEBUG node.lonode001.python_plugin: Start time was 1535986034.0748s; end time was 1535986089.1923s
2018-09-03 16:48:09 DEBUG genericpipeline.executable_args:
2018-09-03 16:48:09 WARNING genericpipeline.executable_args:
2018-09-03 16:48:09 INFO genericpipeline.executable_args: Subprocess completed with exit status 0: /bin/sh -c python /opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 198 127.0.1.1 51121
2018-09-03 16:48:09 DEBUG genericpipeline.executable_args: compute.dispatch results job 198: job_duration: 58.7669501305, returncode: 0
2018-09-03 16:48:09 DEBUG genericpipeline.executable_args:
2018-09-03 16:48:09 WARNING genericpipeline.executable_args:
2018-09-03 16:48:09 INFO genericpipeline.executable_args: Subprocess completed with exit status 0: /bin/sh -c python /opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 199 127.0.1.1 51121
2018-09-03 16:48:09 DEBUG genericpipeline.executable_args: compute.dispatch results job 199: job_duration: 55.7309069633, returncode: 0
2018-09-03 16:48:10 DEBUG genericpipeline.executable_args: Adding node_logging_information
2018-09-03 16:48:10 DEBUG genericpipeline.executable_args: Writing data map file: /scratch/WORK/tmp//long_baseline_pipeline/mapfiles/transfer_amp_clock_sols.mapfile
2018-09-03 16:48:10 INFO genericpipeline.executable_args: recipe executable_args completed
2018-09-03 16:48:10 INFO genericpipeline: Beginning step is_amp_gains
2018-09-03 16:48:10 INFO genericpipeline: Running task: pythonplugin
2018-09-03 16:48:10 INFO genericpipeline.executable_args: recipe executable_args started
2018-09-03 16:48:10 INFO genericpipeline.executable_args: Starting /scratch/WORK/long_baseline_pipeline/bin/addISgains.py run
Reading configuration file: /scratch/WORK/pipeline_lb.cfg
Reading task definition file(s): /opt/lofar/lofar/share/pipeline/tasks.cfg
2018-09-03 16:48:10 DEBUG genericpipeline.executable_args: Pipeline start time: 2018-09-03T14:14:26
2018-09-03 16:48:10 ERROR genericpipeline.executable_args: Could not load input Mapfile ['/scratch/WORK/tmp//long_baseline_pipeline/mapfiles/transfer_amp_clock_sols.transfer_parmDB.mapfile', '/scratch/WORK/tmp//long_baseline_pipeline/mapfiles/ndppp_prep_target.mapfile']
2018-09-03 16:48:10 WARNING genericpipeline.executable_args: Note: recipe outputs are not complete
2018-09-03 16:48:10 WARNING genericpipeline.executable_args: recipe executable_args completed with errors
2018-09-03 16:48:10 WARNING genericpipeline: pythonplugin reports failure (using executable_args recipe)
2018-09-03 16:48:10 ERROR genericpipeline: *******************************************
2018-09-03 16:48:10 ERROR genericpipeline: Failed pipeline run: long_baseline_pipeline
2018-09-03 16:48:10 ERROR genericpipeline: Detailed exception information:
2018-09-03 16:48:10 ERROR genericpipeline: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'>
2018-09-03 16:48:10 ERROR genericpipeline: pythonplugin failed
2018-09-03 16:48:10 ERROR genericpipeline: *******************************************
2018-09-03 16:48:10 ERROR genericpipeline: LOFAR Pipeline finished unsuccesfully.
2018-09-03 16:48:10 WARNING genericpipeline: recipe genericpipeline completed with errors

I tested it with the master branch and the branch "new" leading to the same issue. If anyone has an idea, where this could be coming from, this would be much appreciated.

Cheers,
Alex

After converting the DDF solutions to H5parm format the concatenation at the h5imp_ddf appears to go wrong. The concatenated ddf_solutions.h5 has a wrong and strange internal structure (checked plotting with both h5plot and LoSoTo for sanity). Each time slot contains a subset of frequency slots.

Inspecting the individual bands shows there are valid solutions there, so the concatenate must be getting confused somehow. It's probably something in losoto, as that's what's called, but I'm raising it here as well so people are aware.

The plots below show the strange transition between two adjacent timeslots and that e.g. time slot 2 contains normal valid solutions in the missing frequency spans.

Time slot 2	Time slot 3

150 MHz block timeslot 2	164 MHz block timeslot 2

since the dpppconcat step moved to earlier in the pipeline, do_parallel needs to be fed a mapfile_all_to_one from dpppconcat

Plotting amplitude against time for the core stations, from the in-field calibrator's instrument table (after the copyST_gains step) gives a strange result. The amplitudes for the core stations look quite different from the ST001 amplitudes.
I did not run AOFlagger on the data before the pipeline (re-running it now to see if that makes a difference), but the overall calibration seems ok and since it copies solutions it should not affect anything, right? Running the flagger afterwards flags a large fraction of the data, but produces a decent image so at least the phases are alright.
Here's what I get from parmdbplot (same reference antenna for both):

ST001:

Core Stations: (this is CS006HBA0)

Ran into this because newer containers run a more recent Python 2 version than the "e75" container. This only affects gains_toCS_h5parm.py at the moment. For some reason this does not show up in Python 2.7.5 (e75...), but does become an issue on Python 2.7.15 (recent containers). In the case of gains_toCS_h5parm.py

overrides the logging variable, after which it is no longer the module but a Logger instance from LoSoTo (I believe). This leads to the following obscure crash:

Traceback (most recent call last):
  File "./gains_toCS_h5parm.py", line 238, in <module>
    format_stream = logging.Formatter("%(asctime)s\033[1m %(levelname)s:\033[0m %(message)s", "%Y-%m-%d %H:%M:%S")
AttributeError: 'Logger' object has no attribute 'Formatter'

From my tests it's solved by simply importing the required LoSoTo functions explicitely (from what I can see it's only reorderAxes, but maybe I've missed one):

from losoto.lib_operations import reorderAxes

Hi Leah,

I have downloaded the recent version of the prefactor and long baseline pipeline yesterday. The prefactor V3 runs successfully on the calibrator and target data but the long baseline pipeline crashes with error:

2019-03-01 15:00:18 DEBUG node.lof005.control.lofar.executable_args.L401323_SB247_uv.dppp.MS: /opt/cep/lofim/daily/Tue/lofar_build/install/gnucxx11_opt/bin/NDPPP stdout: MSReader selecting baselines ...
DEBUG - SolTab TGSSphase does not exist in solset combinedsols

And finishes with ERROR:
lofar_versions/LOFAR-Release-3_2_0/lofar_build/install/gnucxx11_opt/lib64:/opt/cep/lofim/daily/Mon/lofar_build/install/gnucxx11_opt/lib64:/opt/cep/lofim/daily/Wed/lofar_build/install/gnucxx11_opt/lib64', 'MODULEPATH': '/etc/modulefiles:/opt/cep/modulefiles'}] failed on localhost
2019-03-01 15:06:57 DEBUG genericpipeline.executable_args: compute.dispatch results job 27: job_duration: 20.3579678535, returncode: 1
2019-03-01 15:06:58 DEBUG genericpipeline.executable_args: Adding node_logging_information
2019-03-01 15:06:58 ERROR genericpipeline.executable_args: A job has failed with returncode 1 and error_tolerance is not set. Bailing out!
2019-03-01 15:06:58 WARNING genericpipeline.executable_args: Note: recipe outputs are not complete
2019-03-01 15:06:58 WARNING genericpipeline.executable_args: recipe executable_args completed with errors
2019-03-01 15:06:58 WARNING genericpipeline: dppp reports failure (using executable_args recipe)
2019-03-01 15:06:58 ERROR genericpipeline: *******************************************
2019-03-01 15:06:58 ERROR genericpipeline: Failed pipeline run: LB-Delay-Calibration
2019-03-01 15:06:58 ERROR genericpipeline: Detailed exception information:
2019-03-01 15:06:58 ERROR genericpipeline: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'>
2019-03-01 15:06:58 ERROR genericpipeline: dppp failed
2019-03-01 15:06:58 ERROR genericpipeline: *******************************************
2019-03-01 15:06:58 ERROR genericpipeline: LOFAR Pipeline finished unsuccesfully.
2019-03-01 15:06:58 WARNING genericpipeline: recipe genericpipeline completed with errors

Please let me know how to fix this error. Thanks

The pipeline crash
The do_parallel step, which calls long_baseline_pipeline/bin/evaluate_potential_delay_calibrators.py, crashes:

ERROR   node.lof020.control.lofar.python_plugin: sequence item 2: expected string, Column found

delay_calibrators.csv looks like this:

'LOTSS_RA','LOTSS_DEC','Source_id','Total_flux'
187.277915,2.052388,cal1,9.8

I ran it outside the pipeline and it threw this error:

Traceback (most recent call last):
  File "./evaluate_potential_delay_calibrators.py", line 314, in <module>
    main( MS_input, args.lotss_file, ncpu=args.ncpu, datacol=args.datacol, nsbs=args.nsbs )
  File "./evaluate_potential_delay_calibrators.py", line 294, in main
    coords = lotss2coords( lotss_file )
  File "./evaluate_potential_delay_calibrators.py", line 238, in lotss2coords
    a = ','.join([str(a['LOTSS_RA']),str(a['LOTSS_DEC']),a['Source_id']])
TypeError: sequence item 2: expected string, Column found

The fix
This line has to change:
https://github.com/lmorabit/long_baseline_pipeline/blob/ac54f61c85466d016be60595d052e0d28bc63639/bin/evaluate_potential_delay_calibrators.py#L238
It should be this:

src = a['Source_id']
if type(src) != str:
    src = 'S' + str(src)
    a = ','.join([str(a['LOTSS_RA']), a(tmp['LOTSS_DEC']), src])

This is already done for the case where there is more than one source in delay_calibrators.csv.

After running the Delay-Calibration parset, the pipeline runs successfully, but fails to produce an image of the calibrator. In the directory after the pipeline has finished running, I have the files:

SL440512_L590163_SB244_uv_12A06A5EDt_121MHz_12A06A5F0t_144MHz.apply_delay_autoI.p.ps
SL440512_L590163_SB244_uv_12A06A5EDt_121MHz_12A06A5F0t_144MHz.apply_delay_autoI.ps
SL440512_L590163_SB244_uv_12A06A5EDt_121MHz_12A06A5F0t_144MHz.apply_delay_autoI_uvcut.ps

But SL440512_L590163_SB244_uv_12A06A5EDt_121MHz_12A06A5F0t_144MHz.apply_delay_autoI.ps is empty when opened. I attach below the diffmap log, showing clear errors in most steps of making the image.
SL440512_difmap.log

The delay selfcal script, current default settings in the pipeline (docircular=True), obtains solutions in circular correlations. That means the measurement sets need to be converted first to circular before this applycal can be run. Then after the applycal they need to be converted back to linear. (if docircular=False then this is not needed, so there also needs to be a switch for that)

For example, in loop3:
https://github.com/lmorabit/long_baseline_pipeline/blob/407813eb5b481f0a354c6b143fba3e928cb34ac6/bin/loop3B_v1.py#L714
Opening a subtable with / is not reliable, as mentioned in the TaQL note. :: shoud be used instead, i.e. instead of

vis + '/SPECTRAL_WINDOW'

open it with

vis + '::SPECTRAL_WINDOW'

This should fix e.g. the case where loop3 crashes with

RuntimeError: Table <something>/SPECTRAL_WINDOW does not exist

Please attach your images.

The pipeline should have a license

If do_download is set to False in LB-Delay-Calibration.parset, then the DownloadCats step does not write any output delay_cat, subtract_cat and image_cat csv files. I'm unsure if there is a set of circumstances in which this is a useful option. It would seem intuitive that do_download should be set to False in a situation where both the lotss catalogue and the lbcs catalogue are already on disk rather than needing to be downloaded from the VO (e.g. because the data is from a bit of sky not in DR1, and calibrators from a bit of LBCS not yet on the VO). But actually at the moment do_download needs to be set to true in that situation, otherwise the catalogues needed for the LB-Split-Calibrators step are not written.

If there is an intended scenario where users don't want the catalogues written out then it might be useful to rename the do_download option to (e.g.) do_make_cats or similar, and/or make sure the various options around input and output from the Download_Cats step are explained in the documentation.

If users will always want the csv catalogues written out, then the do_download parameter is unnecessary, since if do_download is set to True, DownloadCats decides whether or not to download from the VO depending on whether the files already exist on disk, which works fine.

Regarding lofar-astron/prefactor#214
Is there a reason we have updateweights=False when applying the beam?

In loop3B_v1.py, there are tabs thoughout and it was written with a tab being 8 spaces. Many text editors interpret a tab as 4 spaces which messes up the indentation.

e.g. The image below. If I open and save the script in nano/Gedit/Atom the tabs are converted to 4 spaces and it fails when I run it.

To remove the ambiguity maybe next week one of us can convert all tabs to 8 spaces. It is fairly trivial but useful imo.

Also, separate issue really but it is small: montage isn't installed in the Singularity image so I comment it out every time, otherwise I get errors like this:

montage -tile 6x0 -geometry 600x600 direction_133.305_19.515.MS_A_output.png
sh: montage: command not found

Given that it isn't a standard package and it isn't essential to the functioning of loop 3, I suggest adding a flag to the make_montage function to optionally skip this plotting step. e.g. changing

def montage_plot( filepattern, imscale=0.65, nup='4x2', plot_resid=True):
    if filepattern.split('.')[-1] == 'fits':

to something like

def montage_plot( filepattern, imscale=0.65, nup='4x2', plot_resid=True, dontplot=False):
    if dontplot:
        print('Skipping the montage plotting.')
        return
    if filepattern.split('.')[-1] == 'fits':

But it's no big deal either way, just a suggestion.

skynet.py expects 'RA_LOTSS' in the delay catalogue. However, parallel_split.py expects 'LOTSS_RA' (and likewise for declination).

https://github.com/lmorabit/long_baseline_pipeline/blob/afd3df8ecb26b72bf96c35f65a58b55ce47aa7d3/bin/parallel_split.py#L101

I won't change either as I'm not sure which is the desired label, so I'm just flagging it for others.

The pipeline crashes when there is no LoTSS coverage for the data. I tried 3949173 (the latest) on 4C43.15 which isn't in the surveys pointings (yet), causing problems as find_close_objs is run directly after querying the catalogues.

For now I've tried changing

https://github.com/lmorabit/long_baseline_pipeline/blob/39491737de1ecc34df9b85bde2f5774624440be5/plugins/PipelineStep_DownloadCats.py#L290-L292

to

lotss_catalogue = my_lotss_catalogue( MSname, Radius=lotss_radius, bright_limit_Jy=bright_limit_Jy ) 
lbcs_catalogue = my_lbcs_catalogue( MSname, Radius=lbcs_radius ) 
if len(lotss_catalogue) == 0:
	print('Target field not in LoTSS coverage yet! Only writing {:s}'.format(delay_cals_file))
	lbcs_catalogue.write(delay_cals_file, format='csv')
	return
result = find_close_objs( lotss_catalogue, lbcs_catalogue, tolerance=match_tolerance ) 

which seems to work for now (will see if it crashes later on perhaps, because of this).

Hi, when running the evaluate_potential_delay_calibrators.py script the pipeline crashes.

What I find in the log is in short:

2018-10-04 12:58:17 INFO genericpipeline: Beginning step do_parallel
2018-10-04 12:58:17 INFO genericpipeline: Running task: pythonplugin
2018-10-04 12:58:17 INFO genericpipeline.executable_args: recipe executable_args started
2018-10-04 12:58:17 INFO genericpipeline.executable_args: Starting /home/iacobelli/long_baseline_pipeline/bin/evaluate_potential_delay_calibrators.py run
2018-10-04 12:58:17 INFO genericpipeline.executable_args: Limiting to 8 simultaneous jobs/node
2018-10-04 12:58:17 INFO genericpipeline.executable_args: ********************** Remote method is local
2018-10-04 12:58:17 INFO genericpipeline.executable_args: Subprocess starting: /bin/sh -c python /opt/cep/lofim/daily/Mon/lofar_build/install/gnucxx11_opt/lib64/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 10.144.5.111 45297 (['/bin/sh', '-c', 'python /opt/cep/lofim/daily/Mon/lofar_build/install/gnucxx11_opt/lib64/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 10.144.5.111 45297'])
2018-10-04 12:58:17 WARNING genericpipeline.executable_args: /opt/cep/lofar/lofar_versions/LOFAR-Release-3_2_0/lofar_build/install/gnucxx11_opt/lib64/python2.7/site-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
2018-10-04 12:58:17 INFO node.lof009.control.lofar.python_plugin: Processing ['/data/scratch/iacobelli/long_baseline_pipeline/L665012_SAP000_SB000_uv.ndppp_prep_target', . . . . . , '/data/scratch/iacobelli/long_baseline_pipeline/L665012_SAP000_SB225_uv.ndppp_prep_target', '/data/scratch/iacobelli/long_baseline_pipeline/L665012_SAP000_SB226_uv.ndppp_prep_target']
2018-10-04 12:58:18 INFO genericpipeline.executable_args: Waiting for compute threads...
2018-10-04 12:58:30 WARNING genericpipeline.executable_args:
2018-10-04 13:07:14 WARNING genericpipeline.executable_args: 0%....10....20....30....40....50....60....70....80....90....100%
2018-10-04 13:07:15 ERROR node.lof009.control.lofar.python_plugin: list indices must be integers, not tuple

The list of stations I specify is the default of the parset, e.g.
! closure_phase_stations = 'DE601;DE605;ST001'
Strange enough it worked up to the LOFAR school . . any hint ?

The recent update to the DownloadCats plugin seems to have introduced a bug in the call to my_lbcs_catalogue (line 310). This looks like a copy of the call to my_lotss_catalogue, whereas it shouldn't have the bright_limit_Jy argument, and the outfile should be lbcs_catalogue, not lotss_catalogue. Fix coming shortly I hope.

Some datasets have gaps between recorded subbands bigger than the frequency width of the concatenated ILTJ....msdpppconcat files. Normally data gaps are not a problem because dummy subfrequencies are written into the msdpppconcat files, but if the gap is this big, then there is an entire missing ILTJ...msdpppconcat file, instead of it being present and containing entirely dummy data. This becomes a problem later on when the file is combined, the tec is applied and a conversion to FITS is attempted, since the conversion can't be done because the resultant file doesn't have equal frequency widths across the band.

For me
https://github.com/lmorabit/long_baseline_pipeline/blob/51f2c3c49d1e350c9210bb32ddc546e53f48168d/plugins/PipelineStep_DownloadCats.py#L94
in lines 94 and 133 fail. I had to change them to just t.to_table(). This mis probably just a PyVO version difference.

Hi, just following on from the chat on Slack - while the LBCS VO isn't up to date, I have a short script that looks up the LBCS webpage and writes a delay catalogue for sources around a given position like this for the LB pipeline:

Source_id,LOTSS_RA,LOTSS_DEC,Separation_deg,Goodness
L618324,133.703625,20.108528,0.0,PPPPPPPXPPXX-
L618058,133.305833,19.514722,0.7019,PPPPPPPXPPXX-
L618344,134.238375,21.195639,1.1967,PPPPPPPXPPXX-

I know the VO will be updated sometime so it eventually won't matter, but if this script has any value for the pipeline, let me know and I can make it pipeline-friendly. (If not, no hassle, this issue can be closed.) It's linked below in its current form, but would need some tweaks (Python 3 to 2, read the pointing centre from an MS, and making the script a plugin).

https://github.com/mooneyse/scripts/blob/master/find_best_lbcs.py

Hi,
I'm running the lb pipeline on the Herts cluster and I've run into a cannot open shared object file error in the delay_cal_aoflag step - it can't find libgtkmm-3.0.so.1. It's on /usr/lib64 in the LD_LIBRARY_PATH and the log file shows this path. Any suggestions would be much appreciated!
Thanks,
Deepika

I'm opening this issue for those of us testing the eht imager. So far I've tried testing it on two types of data:

1. a single subband of a calibrator source (3C 48) which has already been through NDPPP once to get the amplitudes on approximately the right scale
   

2. the full bandwidth (but heavily averaged, post delay calibration) of an in-field calibrator source which is about 1.95 Jy

What I've learned so far:

• using cphase and camp separately gives slightly better results than using 'bs' mode
• flagging is important. If you have catastrophic outliers this will give you chi^2 values of several million and the fitting won't go anywhere
• if the prior is too big, you can end up pushing flux into the first sidelobes of the psf, which drives the model in completely the wrong direction
• the convergence critieria will be important for any automated pipeline -- at the moment, I can see where the model becomes good and then it keeps going to the max niter and becomes worse

all of this has been accomplished with https://github.com/lmorabit/long_baseline_pipeline/blob/master/eht_imager_scripts/eht_imaging.py

it's also possible to use another image as a prior, e.g. download something from the VLA imaging archive, but in this case I found that if imaging was done with AIPS the eht imager doesn't find the beam parameters so there's also a script https://github.com/lmorabit/long_baseline_pipeline/blob/master/eht_imager_scripts/update_fits_with_beam_parameters.py to fix that

The closure phase script crashes when ran without an X environment. For consistency with the other scripts and to avoid these crashes we should use the Agg backend here as well.

I had to change the order of some variables in the parset, moving
https://github.com/lmorabit/long_baseline_pipeline/blob/39491737de1ecc34df9b85bde2f5774624440be5/long_baseline_pipeline.parset#L39-L43

below

https://github.com/lmorabit/long_baseline_pipeline/blob/39491737de1ecc34df9b85bde2f5774624440be5/long_baseline_pipeline.parset#L46

or {{ job_directory }} wouldn't properly expand for the catalogues.

I've left some detailed comments on the selfcal channel of the slack workspace about this issue. There is a problem with the dif_script on some data I am processing, namely the part where the weighting becomes natural. Has anyone encountered the same thing? (Alternatively, if the dif_script as-is works for you, does it do so if you cut out the loop beginning with uvw 0,-1?)

Hi Leah,
I see the makeDirectory plugin is not yet in the plugins directory. Just moving it from prefactor/plugins solve the issue.
Marco

A polarization axis is assumed to exist currently for amplitude or phase tables. Solving for e.g. scalarphase results in phase solutions without a polarization axis, crashing the copy script.

On the "new" branch.
https://github.com/lmorabit/long_baseline_pipeline/blob/7100c23363907dd7e1ee25fca18f811899b4f022/LB-Delay-Calibration.parset#L268-L271

ndppp_prep_target crashes when it tries to apply the RMextract solutions:

std exception detected: SolTab RMextract does not exist in solset combinedsols

For me, either solset_name needs to be target if it should only have it for the Dutch stations, or add_IS_h5parm needs to be run before the RMextract steps (it extracts the present stations by looking at the stations in the existing solset).

Loop 3 crashes, giving the following error:

File "/data020/scratch/sean/run1/git/long_baseline_pipeline/bin/loop3B_v1.py", line 78, in selfcal
coh[i] = loop3_service.coherence_metric (outcal)
ValueError: could not broadcast input array from shape (63) into shape (74)

The reason for this is that len(coh[i]) is 74 but the right hand side has 63 elements. The len(coh[i]) is defined by the MS antenna table:

import casacore.tables
ms, antennas = 'test.ms', []
for antenna in casacore.tables.table(ms + '/ANTENNA').select('NAME'):
    antennas.append(antenna)
print(len(antennas))  # 74

However, in loop3_service.coherence_metric, the number of antennas is taken from the H5parm:

https://github.com/lmorabit/long_baseline_pipeline/blob/7610d53f209b389bf03e6949508259dccb51280c/bin/loop3_serviceB.py#L144

Here, len(ant) is 63 because this excludes antennas for which 100% of data have been flagged for some stations (during Prefactor).

I'll write a fix for this later in the week. I'm documenting it here for future reference. This is before phasing the stations up. After using an NDPPP filter step, with filter.remove = True, antennas not used are removed. This solves the issue. Since loop 3 will be used after this has been done, it shouldn't be an issue for the pipeline.

addIS_to_h5.py sets the source table to the phase center of the MS that is passed. This resets any directions that were present in the solset to which the CS are added.

Currently there is a mismatch with the name of the argument --continue_no_lotss: in the main function (line 297) the variable is named fail_lotss_ok = kwargs['continue_no_lotss'].lower().capitalize() instead of continue_without_lotss as expected in lines 319. As a consequence the plugin (and the pipeline) can fail, but a simple renaming of the variable can fix it.

Mostly a small cosmetic issue. The pipeline will crash if there is no LoTSS coverage and no manual catalog is provided. I think this is proper behaviour, but there is no output from download_cats in the logs making the problem a bit vague to track down.

I guess genericpipeline eats the prints and they should be changed to logging.info or logging.warn? (I see some logging statements in there already)

The pipeline will crash at prep_delay_dir with

Beginning step prep_delay_dir
*******************************************
Failed pipeline run: Delay-Calibration
Detailed exception information:
<type 'exceptions.IOError'>
[Errno 2] No such file or directory: '/net/nieuwerijn/data2/rtimmerman/A1795_HBA/A1795/Delay-Calibration/results/best_delay_calibrators.csv'
*******************************************
LOFAR Pipeline finished unsuccesfully.
recipe genericpipeline completed with errors

Further up at the offending step there is only

2020-07-15 15:06:38 INFO    genericpipeline: Beginning step download_cats
2020-07-15 15:06:40 INFO    genericpipeline: Beginning step ndppp_prep_target

while if I run the script separately I see the actual output

Attempting to find or download LBCS catalogue.
lbcs_catalogue.csv
DOWNLOADING LBCS Skymodel for the target field
6
Attempting to find or download LoTSS catalogue.
DOWNLOADING LOTSS Skymodel for the target field
Target field not in LoTSS coverage yet! Only writing delay_calibrators.csv

It seems that no dysco is used, leading to enormous disk space usage.

We should use msout.storagemanager=dysco
for all DPPP write steps

@tikk3r bit surprised (wondering how you managed to fit it on our nodes) since you mentioned it was used, but I only see it used once in predict_ateam.argument.msout.storagemanager = "Dysco"

The pipeline crashes at this point:
https://github.com/lmorabit/long_baseline_pipeline/blob/45914063edee7fc155f2902476d7f10bf6fa0484/bin/copySTgains_toCS.py#L85
It states that 'parmdb' object has no attribute 'addValuesGrid'.

I had a look at the parmdb help file:

from lofar import parmdb
help(parmdb)

There is no addValuesGrid function, but there is a getValuesGrid function. Is this what it should be? I'm not 100% sure. (There is also an addValues function.)

It seems to fail at the killMS2H5parm.py step. Not sure why though....

2019-10-24 07:22:54 INFO genericpipeline: Beginning step createmap_ddf
2019-10-24 07:22:54 INFO genericpipeline: Beginning step ddf_solutions
2019-10-24 07:22:54 INFO genericpipeline: Beginning step ddf_h5parms
2019-10-24 07:22:54 INFO genericpipeline: Beginning step convert_to_h5
2019-10-24 07:22:54 INFO genericpipeline: Running task: executable_args
2019-10-24 07:22:54 INFO genericpipeline.executable_args: recipe executable_args started
2019-10-24 07:22:54 INFO genericpipeline.executable_args: Starting /opt/lofar/losoto//bin/killMS2H5parm.py run
Reading configuration file: pipeline.cfg
Reading task definition file(s): /opt/lofar/lofar/share/pipeline/tasks.cfg
2019-10-24 07:22:54 DEBUG genericpipeline.executable_args: Pipeline start time: 2019-10-22T12:50:19
2019-10-24 07:22:54 ERROR genericpipeline.executable_args: Exception caught: list index out of range
Traceback (most recent call last):
File "/opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/cuisine/WSRTrecipe.py", line 132, in run
status = self.go()
File "/opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/recipes/master/executable_args.py", line 357, in go
arglist_copy[ind] = arglist_copy[ind].replace(name, value[i])
IndexError: list index out of range
2019-10-24 07:22:54 WARNING genericpipeline: executable_args reports failure (using executable_args recipe)
2019-10-24 07:22:54 ERROR genericpipeline: *******************************************
2019-10-24 07:22:54 ERROR genericpipeline: Failed pipeline run: LB-Delay-Calibration
2019-10-24 07:22:54 ERROR genericpipeline: Detailed exception information:
2019-10-24 07:22:54 ERROR genericpipeline: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'>
2019-10-24 07:22:54 ERROR genericpipeline: executable_args failed
2019-10-24 07:22:54 ERROR genericpipeline: *******************************************
2019-10-24 07:22:54 ERROR genericpipeline: LOFAR Pipeline finished unsuccesfully.
2019-10-24 07:22:55 WARNING genericpipeline: recipe genericpipeline completed with errors
Results:

Currently the split out infield calibrator(s) do not seem to be corrected for being away from the phase center further down the primary beam. This could cause beam effects to be absorbed in the amplitude solutions obtained with selfcal, more so for the ones further away, which can then no longer be transferred across the field.

The pipeline crashes with this error.

2020-01-20 08:50:05 WARNING node.LOFAR.executable_args.L751472_SB227_uv.MS: /opt/lofar/lofar/bin/NDPPP stderr:
std exception detected: Antenna Expression: Parse error at or near '{'
(near char. 0 in string "{{ filter_baselines }}")

filter_baselines isn't defined in LB-Delay-Calibration.parset but appears in the filter step.

https://github.com/lmorabit/long_baseline_pipeline/blob/afd3df8ecb26b72bf96c35f65a58b55ce47aa7d3/LB-Delay-Calibration.parset#L208

I removed it from my run for now but it should be defined or the correct parameter put in its place.

The pipeline crashes with the following error:

2019-02-25 11:14:25 INFO    genericpipeline: Beginning step download_cats
WARNING: AstropyDeprecationWarning: The astropy.vo.samp module has now been moved to astropy.samp [astropy.vo.samp]
DOWNLOADING LOTSS Skymodel for the target field
2019-02-25 11:14:30 ERROR   genericpipeline: *******************************************
2019-02-25 11:14:30 ERROR   genericpipeline: Failed pipeline run: LB-Delay-Calibration
2019-02-25 11:14:30 ERROR   genericpipeline: Detailed exception information:
2019-02-25 11:14:30 ERROR   genericpipeline: <class 'pyvo.dal.exceptions.DALQueryError'>
2019-02-25 11:14:30 ERROR   genericpipeline: Field query: column \"major\" does not exist LINE 1: ...flux AS Total_flux, E_Total_flux AS E_Total_flux, Major AS M...                                               
               ^ HINT:  Perhaps you meant to reference the column \"main.maj\" or the column \"main.minor\".
2019-02-25 11:14:30 ERROR   genericpipeline: *******************************************
2019-02-25 11:14:30 ERROR   genericpipeline: LOFAR Pipeline finished unsuccesfully.
2019-02-25 11:14:30 WARNING genericpipeline: recipe genericpipeline completed with errors

It occurs at the query.execute() step of

https://github.com/lmorabit/long_baseline_pipeline/blob/23d44f86d68a4f1c8bbfeae63c6f6a0c70788ef6/plugins/PipelineStep_DownloadCats.py#L84-L91

These issues aren't relevant if preparing the data for LB-Split-Calibrators.parset but I thought I'd document them here in case it is useful for someone.

Going off what is in PipelineStep_TargetListToMapfile.py, the pipeline step requires one edit, and a few of the arguments are unused. I think it can be changed from

# Initialise file with direction of delay calibrator
prep_dirs.control.kind                    = plugin
prep_dirs.control.type                    = TargetListToMapfile
prep_dirs.control.mapfile_dir             = input.output.mapfile_dir
prep_dirs.control.infile                  = dpppconcat.output.mapfile
prep_dirs.control.filename                = prep_dirs.mapfile
prep_dirs.control.wd                      = {{ job_directory }}
prep_dirs.control.nP                      = 3   # this is the default setting
prep_dirs.control.counter                 = 0
prep_dirs.control.manual                  = True ## 
prep_dirs.control.target_file             = find_delay_cal.output.calfile

to

# Initialise file with direction of delay calibrator
prep_dirs.control.kind        = plugin
prep_dirs.control.type        = TargetListToMapfile
prep_dirs.control.mapfile_dir = input.output.mapfile_dir
prep_dirs.control.mapfile_in  = dpppconcat.output.mapfile  # infile -> mapfile_in, I think
prep_dirs.control.filename    = prep_dirs.mapfile
prep_dirs.control.target_file = find_delay_cal.output.calfile

Also PipelineStep_TargetListToMapfile.py itself reads in primary_delay_calibrator.csv and expects a header on that CSV:

t = Table.read(target_file, format='csv')
RA_val = t['RA_LOTSS'].data[0]
DEC_val = t['DEC_LOTSS'].data[0]
Source_id = t['Source_id'].data[0]

But there is none in the file. I added one manually as a workaround.

Lastly, the mapfile written out by the prep_dirs step causes the next step, dppp_phaseup, to crash with a mapfile mismatch error. I manually edited it from

[{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_122MHz.msdpppconcat', 'skip': False}]

to

[{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_122MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_124MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_126MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_128MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_130MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_132MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_134MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_136MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_138MHz.msdpppconcat', 'skip': False},{'host': 'localhost', 'file': '/data5/sean/LC12_022/working//LB-Delay-Calibration-run2-no-ddf/S6850_L751472_SB010_uv_12EDF7FBDt_140MHz.msdpppconcat', 'skip': False}]

to get it working, but I don't know enough genericpipeline mapfile kung fu to patch it properly.

In the new branch, if DDF solutions are written to the DATA_DI_CORRECTED column

https://github.com/lmorabit/long_baseline_pipeline/blob/f6433c8cf56974786a5006614d645d3507e8f134/LB-Delay-Calibration.parset#L47

Then this should be reflected in the comment in LB-Split-Calibrators.parset, changing

https://github.com/lmorabit/long_baseline_pipeline/blob/f6433c8cf56974786a5006614d645d3507e8f134/LB-Split-Calibrators.parset#L28

to this, I believe:

! data_col = DATA ## set to DATA_DI_CORRECTED to use DDF solutions

Found after investigating #70. For fulljones solutions the dummy entries should have 0 amplitude on the off-diagonal terms instead of 1.

The script uses an old version of the LBCS catalog on the Astron VO server (resulting in zero sources found and an error)

https://github.com/lmorabit/long_baseline_pipeline/blob/e29d33b2c8b8252125dfe07827801d95bb4b5950/plugins/PipelineStep_DownloadCats.py#L127

Good sources - 0
Target field not in LoTSS coverage yet! Only writing /net/rijn/data2/rvweeren/A2256_LB/prefactor_output//LB-Delay-Calibration/delay_calibrators.csv
2019-10-07 20:19:06 ERROR genericpipeline: *******************************************
2019-10-07 20:19:06 ERROR genericpipeline: Failed pipeline run: LB-Delay-Calibration
2019-10-07 20:19:06 ERROR genericpipeline: Detailed exception information:
2019-10-07 20:19:06 ERROR genericpipeline: <type 'exceptions.AttributeError'>
2019-10-07 20:19:06 ERROR genericpipeline: 'NoneType' object has no attribute 'write'
2019-10-07 20:19:06 ERROR genericpipeline: *******************************************
2019-10-07 20:19:06 ERROR genericpipeline: LOFAR Pipeline finished unsuccesfully.
2019-10-07 20:19:06 WARNING genericpipeline: recipe genericpipeline completed with errors
Results: