avijeetprasad / pipmag Goto Github PK

Most of the errors in the implementation are related to changes in the format of the dataframe. Adding a test on that should help us to identify better those problems.

Describe the bug
The movie links for the 2023-08 campaign does not show up in VideoSelector2

To Reproduce
Steps to reproduce the behaviour:

1. Go to la_palma_quicklook.ipynb
2. Run the VideoSelector2 widget
3. Set the date to 2023-08-28:T08:21:18
4. No links show up, and if you click the Show button, it displays an old movie from 2023-07-20

Expected behaviour
It should load up the correct videos

Screenshots

Additional context
It's possible that the movie link format from the last campaign has changed and needs to be parsed differently.

Before reading any data frame saved as a .csv file, we have to convert the types, including:

• Read the date_time column as datetime
• String to list ['links', 'video_links', 'image_links', 'instruments']
• NaN to None ['comments', 'polarimetry', 'target']
• Boolian to string ['polarimetry']

This should be done in a function.

Describe the bug
The la palma quicklook links the wrong video for one date (maybe more?).

To Reproduce
Steps to reproduce the behavior:

1. Go to 'la_palma_quicklook.ipynb'
2. Run the notebook.
3. Set the date as 23.07.2023 and time 10:53:13
4. The links to the videos are for CHROMIS videos from 2022-05-27?

Expected behavior
The link should be for the same date that was selected. Otherwise, the 23rd links to all correct dates and times.

Desktop (please complete the following information):

• OS: Ubuntu
• Version 22.04 LTS

Additional context
I will update here if I find more such errors for 2023.

Add some pages to the wiki to describe the workflow for this project.

Currently we are saving two dataframes in pkl format:

• data/all_media_links_20230706_081644.pkl
• data/la_palma_obs_data_20230706_081731.pkl

Issues with time stamps:

• We end up getting multiple versions of these files with different timestamps. This adds up in memory when pushed to the repository under version control.
• The timestamps can be inferred from git commits. So they don't add any more info.

Issues with pkl file format:

• It become python version dependent.
• It is harder to read externally. So would be difficult if the update is made by a user and we want to review it as a pull request.
• Having it as csv format will allow git to track the changes incrementally instead of saving the full version every time.

Describe the bug
Movies are not playing in the quicklook notebooks when opened in Codespaces. The reason is that ffmpeg is not installed by default in these images.

To Reproduce
Steps to reproduce the behavior:

1. Go to the PipMag repository
2. Click on Create new codespace.
3. Open la_palma_quicklook.ipynb and look the widget
4. The movies don't play

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. mac OS]
• Version [e.g. 13.4]

Additional context
Add any other context about the problem here.

Summary

Create a well-defined outline for the data structure in data/la_palma_obs_data.csv. This will assist in devising specific tests for tests/test_gen_la_palma_df.py.

Motivation

• Establish a clear data structure for better data manipulation and retrieval.
• Simplify the creation of unit tests by offering a well-defined schema.

Tasks

• Update the pipmag/gen_la_palma_df.py to reflect the specified data structure.
• Update the project wiki with the new data structure.
• Implement user hints for available keywords when manual input is required.
• Create unit tests to validate the data structure.
• Update the wiki with the changes.

Data Structure Outline

Data Type Handling

• Lists: Separated by semi-colons ; without spaces.
• Booleans: 'True' or 'False' as strings.
• Empty Values: Represented as empty strings ''.

Data Type Conversion: Python vs CSV

Available Keywords

• Instruments Keywords: ['CRISP', 'CHROMIS', 'IRIS']
• Targets Keywords: ['Active Region', 'Sunspot', 'Light Bridge', 'Flare', 'Burst', 'UV Burst', 'Ellerman Bombs', 'Penumbral Ellerman Bombs', 'Surges', 'Flux Emergence', 'Pores', 'Faculae', 'Plage', 'Filament', 'Quiet Sun (QS)', 'Coronal Bright Point', 'Coronal Hole', 'Magnetic Bright Points', 'Spicules', 'Prominences']

• Update the text in the README file.

Add pages to the documentation folder regarding

• Outline the structure of the La Palma data frame
• Keyword schema for the data frame
• Naming convention for quick-look movies

When updating information in the data-frame using VideoSelector2, said information is not being saved in the pickle file.

Use the Add to GitHub projects workflow
https://github.com/marketplace/actions/add-to-github-projects

Describe the bug
The data frame should be able to be updated and then added to the existing data frame without losing any information already added to the data frame. The pickle file with the data frame should be updated without losing the old information.

To Reproduce
Steps to reproduce the behavior:
N/A

Expected behavior
When an update is made to the pickle file, this should be saved together with the old information.

Desktop (please complete the following information):

• OS: Ubuntu LTS
• Version 22.04

Additional context

Describe the bug
VideoSelector2 does not seem to work when the data frame only contains one year.

To Reproduce
Steps to reproduce the behavior:

1. Go to la_palma_query.ipynb
2. Make a filtered data frame with the dates 1 July 2023 to 29 July 2023.
3. Save the data frame and run the rest of the notebook.
4. See error

Expected behavior
Even if only one year, month and day are present in the data frame, it should show up and show the videos.

Desktop (please complete the following information):

• OS: Ubuntu
• Version 22.04 LTS

Multiple entries in the quick-look movies correspond to the same observations with time stamps a few minutes apart. These need to be grouped.

Query implementation to extract dataset based on tags/requirements

Describe the bug
ADS API key is not found, despite being set in the config.ini file.

To Reproduce
Steps to reproduce the behavior:

1. Go to ads_query.ipynb
2. Run the notebook
3. The third bracket does not run, but instead raises a ValueError.

Expected behavior
The code should find the key from the config.ini file when using ADSSearch().

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu LTS
• Version: 22.04

Additional context

Motivation
Run a python script which produce a movie with subplots:

• if only data (one input file):
  |Red|Core|Blue|
• if also inversion results (3 files: data, Blos, Vlos)
  |Red|Core|Blue|Blos|Vlos|

Save everything following the same format we have defined in the wiki. A file per line will be generated.

Create a branch protection rule so that new changes are merged through branches upon review.

Summary

Write a script to generate the quicklook movies for SST mosaics.

Motivation

• Generate a quicklook movies after taking the mosaics to estimate the data quality.

Detailed Description

• Would need to load the mosaic grid
• Would need the data from the turret log files

Implementation

• interactively generate mosaic sequence
• load the turret file
• load the mosaic images
• stitch the mosaic to generate the final mosaic

I can create a Jupyter book interface for the quick look notebook and host the HTML file on GitHub. That way, anyone would be able to access it.

Find all the papers related to any given observational dataset, and add it as a property in the database.

Describe the bug
There are a few duplicates in the keywords after grouping the times together (CRISP showing twice etc). This is probably due to the use of 'groupby: sum'.
For the polarimetry keyword: All should be False by default and use an operation to combine the groupby and take always if there is at least one True.

To Reproduce

1. Go to 'la_palma_quicklook.ipynb'
2. Run the notebook
3. Set the date to 6th of June
4. CRISP will appear twice

Expected behavior
CRISP and other keywords should only show up once

Desktop (please complete the following information):

• OS: Ubuntu
• Version 22.04

Describe the bug
The query in la_palma_query.ipynb is not behaving as expected. It is missing many entries.

To Reproduce
Steps to reproduce the behavior:

1. Go to la_palma_query.ipynb
2. Run

query = su.Query(df)
query.create_widget()

3. Select 'Instrument': 'CHROMIS'
4. Keep the default start and end date, and set 'Target': 'Active Region'
5. You will see that it shows only two entries.
   

Now, repeat the same with 'Instrument': 'CRISP', you will find many entries which has both 'CRISP' and 'CHROMIS' as instruments. These entries were not capture before.

Expected behavior
The query should find all the entries with 'CHROMIS' all the time!

Describe the bug
When running the VideoSelector2, long names cannot be displayed.

After the first version of the implementation of the query by @semyaat, she suggested some ideas:

• Currently, the user must hold 'shift' to select more than one instrument. It would be better to have separate buttons for each instrument or to let the user click a button twice to de-select it.

• For the target, the user can only select one. Implementing the same dropdown as was used for the instruments is not straightforward, but should be done.

By using the metadata of the observations of the internal archive we can add the following:

• Duration of the observations (new keyword).
• Wavelength information (spectral lines and wavelength sampling of the dataset).
• Check if the data is fully-reduced (new boolean keyword: [True, False]), so we can quickly identify those that need to be processed.
• Location of the dataset (new keyword).
• Quality of the seeing (extracting info about the seeing plot).
• Module to generate new movies from the science-ready datasets under the new convention.

• Check for files having the structure Blos.6302.fcube for each folder in the La Palma database
• Run potential field extrapolations
• Use Vapor Python API to save the output as a png file

Use dictionary with keywords to automatically populate the polarimetry column

Fill the dataframe with keywords for the query.

Add a check that edits keywords to fit a certain convention?

Target: Sunspot, Pore, Quiet Sun, Active Region, Facula/Plage, Filament/Prominence, Flare, Emerging Flux Region, Light Bridge, Burst/UV Burst, Ellerman Bomb, Surges, Penumbral ellerman bombs (PEB), Flux Emergence, Coronal Bright Point, Coronal Hole,
Magnetic Bright Points, Spicules, Prominences (off-limb filaments), Polarity Inversion Line (PIL)

With the stabilization of the database format, we have reached a point where the usage of git Large File Storage (LFS) for handling the La Palma CSV files is no longer necessary. This change will bring about the following benefits:

1. Simplification of Setup: By removing the git LFS dependency and directly adding the CSV files to the repository, we can streamline the setup process for new users. This change reduces complexity and potential obstacles during the initial configuration.

2. Efficient Use of Repository Space: Since we anticipate that future modifications will primarily consist of adding new entries, the differences (diff) between versions of the CSV files are expected to remain small. Consequently, this change should not significantly increase the repository's overall size.

3. Enhanced Accessibility: This update will foster a more user-friendly experience, especially for those who might not be familiar with git LFS.

Given these considerations, I propose that we proceed with the removal of the git LFS dependency and the integration of the CSV files directly into the repository. This adjustment aligns with our goal of maintaining an accessible and efficient codebase.