Code Monkey home page Code Monkey logo

copper_preprocessing_config's People

Contributors

alanloh avatar cedricdviou avatar

Watchers

avatar

Forkers

cedricdviou

copper_preprocessing_config's Issues

Missing quote in toml file

Command: (on copper1)
parset2preprocconfig --parset testCygX3.parset --directory .

(empty parameter field)

Result file:

tasks = ['process', 'rsync_quality', 'rsync']  #<<<====== missing 'quality' step before 'rsync'

log_email = [email protected]    #<<<====== missing quote around email

[worker]
env_file = env_default.sh   #<<<====== missing absolute path & quote around value

[process]
avg_timestep = 8
avg_freqstep = 6
startchan = 0
nchan = 64
compress = False
flag_strategy = NenuFAR-64C1S.rfis   #<<<====== missing absolute path & quote around value

# missing 'quality' step details derived from existing Subbands list in parset (PhaseCenter[0].subbandList=[192..435])

Add an option to automatically delete L0 data

From Florent/Cedric :

On Thursday, Florent and I discussed a bit the pre-processing tasks running on copper. In 2 words, it seems that this process is reliable enough to automate the suppression of L0 and L1 data immediately after transfert on databf.
We could however offer an option for users to keep the data. One use case would be testing some setups on a set of data, so you need to keep L0 for a little while to re-run the pre-proc several times. The options would be keep_L0 and keep_L1, (default = False).
I think the right place for this is in
@alan Loh
copper_preprocessing_config module. By default the tasks list could include rm_l0 and rm_l1. If keep_LX is present and set to True in the parameter of the observation parset then the corresponding rm_lX is removed from the task list.
There is mechanism in place to not run the rm_lX task if any previous tasks failed, so that should prevent any issue with failed processing or failed rsync, but we can think as well of an extra check tasks TBD.
A last thing is that by default the rm_lX tasks are interactive, they ask confirmation before deleting the data. For the automation, there is a --force option that bypass the confirmation. So in runPreProcessing.py, nenuprep need to be called with the --force option.

Error with Slack webhook from copper1

Command:
parset2preprocconfig --parset testCasA.parset --directory .

result:

nenufarobs@copper1:~/parset2toml$ parset2preprocconfig --parset testCasA.parset --directory .
2022-06-01 20:48:41 | INFO: Parset '/home/nenufarobs/parset2toml/testCasA.parset' loaded.
2022-06-01 20:48:41 | INFO: Initializing the writing of ./testCasA.toml...
2022-06-01 20:48:41 | WARNING: No 'parameters' found for PhaseCenter[0], or no PhaseCenter found at all.
2022-06-01 20:48:41 | INFO: Applying default pre-processing...
2022-06-01 20:48:41 | INFO: No quality step will be applied.
2022-06-01 20:48:41 | INFO: Only the first email is considered: '[email protected]'.
2022-06-01 20:48:41 | INFO: './testCasA.toml' written.
Traceback (most recent call last):
  File "/home/nenufarobs/.local/lib/python3.6/site-packages/copperconfig/nenufar_parset.py", line 562, in to_config_toml
    success=True
  File "/home/nenufarobs/.local/lib/python3.6/site-packages/copperconfig/nenufar_parset.py", line 662, in _slack_message
    f"Request to slack returned an error {response.status_code}, the response is:\n{response.text}"
ValueError: Request to slack returned an error 404, the response is:
no_service

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nenufarobs/.local/bin/parset2preprocconfig", line 35, in <module>
    parset.to_config_toml(directory=args.directory)
  File "/home/nenufarobs/.local/lib/python3.6/site-packages/copperconfig/nenufar_parset.py", line 571, in to_config_toml
    success=False
  File "/home/nenufarobs/.local/lib/python3.6/site-packages/copperconfig/nenufar_parset.py", line 662, in _slack_message
    f"Request to slack returned an error {response.status_code}, the response is:\n{response.text}"
ValueError: Request to slack returned an error 404, the response is:
no_service

Documentation issue with command line example

Context: package was git pulled and pip installed locally on copper1

Command format:
python parset2preprocconfig --parset testCasA.parset --directory .

does not work, but

parset2preprocconfig --parset testCasA.parset --directory .

parset2preprocconfig is the bin located in ~/.local

Add two other parameters

Add two parameters to the toml config file: process.flag_rfi (default=true) and process.flag_memoryperc (default=30).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.