Release

Release number: v1.2.1

• Major - Structural/User interface change
• Minor - New feature or Deprecated feature
• Patch - bug fixes

Scope

This patch release is purely for the numpy depreciation of np.str as it breaks both pydarnio and pydarn on import.

Propose Deadline

Date: ASAP

Pull Requests

Current develop branch.

Pull Request To Be Included

PLEASE LET ME KNOW IF THERE ARE ANY OTHER MINOR FIXES TO BE INCLUDED ASAS

The cFit format (described here https://superdarn.github.io/rst/superdarn/src.doc/rfc/0007.html) is not supported by the current pydarnio. cFit is used quite widely as it contains most of the same stuff as fitACF but is typically 10x smaller.

I noticed that pydarn will incorrectly classify an input bz2 file as having an issue with the first record.
If it's not the intention of pydarn to handle compressed files, then it should produce an error message saying so. If it's the intention of pydarn to handle compressed files, it should uncompress it first.

import pydarn
in_fn = '/home/kevin/PycharmProjects/pydarn/20090101.1201.00.sas.rawacf.bz2'
sd = pydarn.SDarnRead(in_fn)
r = sd.read_rawacf()
Error: /home/kevin/PycharmProjects/pydarn/20090101.1201.00.sas.rawacf.bz2 contains an block size 643383601 > remaining bytes 13381653 at record = 0.
Traceback (most recent call last):
File "", line 1, in
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/superdarn.py", line 421, in read_rawacf
self._read_darn_records(file_struct_list)
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/superdarn.py", line 372, in _read_darn_records
self._read_darn_record(format_fields)
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/superdarn.py", line 347, in _read_darn_record
record = self.read_record()
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/dmap.py", line 449, in read_record
remaining_bytes, "remaining bytes")
File "/home/kevin/PycharmProjects/pydarn/pydarn/io/dmap.py", line 287, in bytes_check
self.rec_num)
pydarn.exceptions.dmap_exceptions.MismatchByteError: Error: /home/kevin/PycharmProjects/pydarn/20090101.1201.00.sas.rawacf.bz2 contains an block size 643383601 > remaining bytes 13381653 at record = 0.

the original issue was in pyDARN SuperDARN/pydarn#58

Writing of SuperDARN data files

Name: Writing of SuperDARN data files

module: fitacf_convert.py

package: utils

Scope

Create a check list using markdowns checklist syntax on the scope of the feature

What will the feature do?

• Convert fitacf files to data formats such as hdf5, netCDF, or csv which have been commonly used in the space science community.

Description

*PyDARNio is supposed to incorporate the features of reading and writing of SuperDARN data. The writing part is mostly missing except for the borealis radars. Writing the fitacf (grid and map) files into data formats that are commonly used by our community would significantly improve the usage of SuperDARN data. And it is convenient for users to combine SuperDARN data with data from other instruments. In addition, it would be great if the geolocation information of each backscatter echo could be included in the output files instead of range gates for the fitacf files. *

pyDARNio Checklist

• Does this fit within pyDARNio's scope?
• Is this a minor change i.e., new small utility function?
• Is this a major change i.e., new plotting function?
• Do you need help developing it?
• Have you created a pyDARNio GitHub project to show the tasks needed to be done?

Development help

pyDARNio does not have a dedicated development team on hand for full feature development.
However, if you are able to provide some time and help, the community will try to aid you in the best way they can to make your feature possible.

If you need help please indicate what expertise you are looking for:

• developer help with design and integration of code
• scientific help with ensuring what you develop is scientifically correct/accurate
• project management to help organize what steps need to be taken to make this possible

Please also provide any other information you may need help with, not including testing/reviewers of code

User Interface

Please provide pseudocode on how you see your new feature interacting with the code or user-interface

Extra Notes

Please provide any other details about this feature

• https://github.com/vtsuperdarn/SD_exodus

Add utilities to do commonly needed data manipulations, including:

• Build datetime objects from dmap time variables
• Extract time series of specified variables from record lists
• Extract scans from record lists

I believe this is only useful for data of the fitacf and earlier level.

BUG

Restructuring Borealis v0.6+ antennas_iq files from array to site format

Priority

• Minor - typo, incorrect naming, fails on specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation

Example of the bug

>>> import pydarnio
>>> infile = '20221102.2146.29.sas.0.antennas_iq.hdf5'
>>> reader = pydarnio.BorealisRead(infile, 'antennas_iq', 'array')
>>> records = reader.records
The file cannot be restructured due to the  following error: Arrays from 20221102.2146.29.sas.0.antennas_iq.hdf5: Error restructuring BorealisAntennasIq from array to site style: only integer scalar arrays can be converted to a scalar index
Traceback (most recent call last):
  File "/home/remington/pyDARNio/pydarnio/borealis/borealis_array.py", line 178, in records
    records = self.format._array_to_site(self.arrays)
  File "/home/remington/pyDARNio/pydarnio/borealis/base_format.py", line 1154, in _array_to_site
    timestamp_dict[key][field] = data_dict[field][index_slice]
TypeError: only integer scalar arrays can be converted to a scalar index

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/remington/pyDARNio/pydarnio/borealis/borealis.py", line 186, in records
    return self._reader.records
  File "/home/remington/pyDARNio/pydarnio/borealis/borealis_array.py", line 184, in records
    raise borealis_exceptions.BorealisRestructureError(
pydarnio.exceptions.borealis_exceptions.BorealisRestructureError: The file cannot be restructured due to the  following error: Arrays from 20221102.2146.29.sas.0.antennas_iq.hdf5: Error restructuring BorealisAntennasIq from array to site style: only integer scalar arrays can be converted to a scalar index

Attempts

Print statements in the code led me to find that pulse_phase_offset is the culprit, and for the file in question this field was filled with empty arrays.

I was able to correctly deal with the file by changing borealis_formats.py: Line 2314 to 'pulse_phase_offset': [],. This may need to be done for bfiq files as well, on Line 2200 of the same file.

Data Location

It's a big file (324 MB), reply back if you need it and I can get it to you.

Potential Bug Location

Line 2314 of borealis_formats.py

Potential Solution(s)

Change line 2314 of borealis_formats.py to 'pulse_phase_offset': [],

Extra Notes

Please provide other pertinent details about this feature:

• OS: openSUSE Leap 15.3
• Python version 3.9.14

An extra check for map file reading sees if the station ids are for the correct hemisphere.

Purposed solution:
Using the hardware files and SuperDARN.Radars class you can determine which stid belongs to which hemisphere then cross-checking that will stid present in the map file and the hemisphere field in the map file.

BUG

When trying to convert Borealis HDF5 rawacf.hdf5.site files, I get the following error:

The file cannot be converted to SDARN DMap rawacf due to the following error: Increased complexity: Borealis rawacf file record 1633013160047 blanked_samples [ 0 72 96 160 176 208 216] does not equate to pulses array converted to sample number [ 0 9 12 20 22 26 27] * 8

On inspection of those arrays, they should be the same (i.e. each number in the first list is 8x the number in the second list). Therefore, the error message is incorrect.

On further inspection, the cause of the error message is around line 424 of borealis_convert.py:
...
sample_spacing = int(record['tau_spacing'] /
record['tx_pulse_len'])
normal_blanked_1 = record['pulses'] * sample_spacing
normal_blanked_2 = normal_blanked_1 + 1
blanked = np.concatenate((normal_blanked_1, normal_blanked_2))
blanked = np.sort(blanked)

            if not np.array_equal(record['blanked_samples'], blanked):

...

I don't understand why "normal_blanked_2" exists, or why it is concatenated into "blanked"
If we just did blanked = normal_blanked_1 instead, then it seems everything would work fine.

Priority

• Major - misrepresentation of data, failure in a common situation
  Fails each time we try to convert the rawacf.hdf5.site

Example of the bug

from pydarnio import BorealisConvert
filename= '20210930.1446.00.wal.0.rawacf.hdf5.site'
borealis_filetype = 'rawacf'
dmap_filename = '/borealis_nfs/borealis_data/daily/20210930.1446.00.wal.a.rawacf.site'
slice_id = 0
borealis_converter = BorealisConvert(filename, borealis_filetype,
dmap_filename, slice_id, borealis_file_structure='site')

pydarnio.exceptions.borealis_exceptions.BorealisConvert2RawacfError: The file cannot be converted to SDARN DMap rawacf due to the following error: Increased complexity: Borealis rawacf file record 1633013160047 blanked_samples [ 0 72 96 160 176 208 216] does not equate to pulses array converted to sample number [ 0 9 12 20 22 26 27] * 8

Attempts

What have you tried already to fix it?

Tried commenting out the normal_blanked_2 stuff, got a different error:
pydarnio.exceptions.borealis_exceptions.BorealisConvert2RawacfError: The file cannot be converted to SDARN DMap rawacf due to the following error: 'agc_status_word'

What have you tried to get around it?

Data Location

to be uploaded here

Please provide other pertinent details about this feature:
Ubuntu linux

Discussion

Topic: when a file is read in using read_rawacf (and others) the code does a few checks to check the integrity of the dmap files, however when the checks fail there is no reason given, we should return more verbose and if required give a reason as to why this file has failed. We could also maybe made a new 'check_rawacf' in which this is done so it doesn't effect the read time if you're not interested in the checks.

This is to find a solution following discussion at the DDWG working group who use pyDARNio to check the integrity of dmap files on the mirrors.

Category

• User-friendly
• Software Design
• Data related
• Capabilities
• Clarity
• Workflow

Hi all,

I've renamed the master branch to be called main in-keeping with updates across the org.
In your local envs you can switch the upstream by:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

before making any changes. But most of you shouldn't be making changes to the main branch anyway :P
This should have minimal impact on regular users, and only the admins will have to amend some things when we do the next release.

If you have any questions or need help with this move comment here or give me an email on [email protected]

Thanks,
Carley

Note that the new default branch is also develop, this means that any new PR made will automatically assume it wants to be merged into develop not main which is the general workflow anyway.

Hi

I'm currently having an issue where using read_fitacf() with certain files - seems to be those from the Hankasalmi radar for years up to the late 2000s - returns the SuperDARNFieldMissingError, saying that the origin.command, origin.code, origin.time, and bmazm fields in record 0 are missing.

`
import pydarnio
import bz2
file = "/lustre/ahome3/s/sktr1/data/20060916.0000.28.han.fitacf.bz2"

with bz2.open(file) as fp:
fitacf_stream = fp.read()

reader = pydarnio.SDarnRead(fitacf_stream, True)
records = reader.read_fitacf()
`

I haven't encountered the problem when using more recent files or files from different radars.

Discussion

Topic: Decreasing memory usage of file I/O

Description

Conversion of HDF5 files from 'site' to 'array' structured is done regularly on remote site computers. Increasingly often, these site computers do not have enough memory available to do this restructuring.

Category

• User-friendly
• Software Design
• Data related
• Capabilities
• Clarity
• Workflow

Details

Our largest regularly generated filetype (antennas_iq) usually is around 3-4 GB in size, but with some experiments we have generated files of 5.5 GB. When restructuring these files, the entire file is loaded into memory, then a copy of the data is made, but in the other structure format. This means up to 11 GB (6-8 typically) of memory is needed to restructure one file. Libraries like h5py and deepdish which facilitate I/O of HDF5 files support partial loading, so it would be great to figure out a way to reduce the memory footprint when converting.

Naively, it makes sense to me to have the following workflow when restructuring:

• From 'site' to 'array' format:
  1. Open the file with h5py to get a list of the keys for the file
  2. Load the record for the first key into memory
  3. Allocate arrays based on the number of keys and the size of data in the first record
  4. Record by record, load the record into memory and copy its data into array format, updating the size of the arrays if necessary
• From 'array' to 'site' format:
  1. Load the file into memory (unavoidable since all arrays are needed to write a single record)
  2. Copy the data corresponding to the first record into a new dictionary
  3. Write that dictionary to a new file
  4. Repeat steps 2 and 3 for each record, but add each record onto the same file that was created the first time around

Example

Code

Current process:

import pydarnio

filename = 'path/to/file.antennas_iq.hdf5.site'
outfile = 'path/to/file.antennas_iq.hdf5'

reader = pydarnio.BorealisRead(filename, 'antennas_iq', 'site')  # Entire file read into memory
arrays = reader.arrays   # Now have two copies of all data in memory
writer = pydarnio.BorealisWrite(outfile, arrays, 'antennas_iq', 'array')

I'm currently working on a bug fix branch for this. It currently only places one character of the string field in the array files, which ends up being '{'.

When reading in older RAWACF, FITACF, MAP files. Sometimes dates are all over the place in bouncing between years, months, days. This would be a nice check to have in the io library that DDWG could use and give warning to users the file is bad.

Simplify unit tests by:

• using one unit test package (unittest, after some discussion with @mts299)
• use subTest to reduce duplication in tests
• use discover and remove main declaration on bottom of current tests

New Feature

Name: to_xarray

Scope

• Create a to_xarray method for each of the data classes

Description

Custom data structures are useful, but can be difficult to use in standard data analysis. Comparisons between SuperDARN and satellites would be made easier of SuperDARN data was in a standard analysis format. Based on the multi-dimensional nature of the data, I suggest an xarray Dataset.

pyDARNio Checklist

• Does this fit within pyDARNio's scope?
• Is this a minor change i.e., new small utility function?
• Is this a major change i.e., new plotting function?
• Do you need help developing it?
• Have you created a pyDARNio GitHub project to show the tasks needed to be done?

Development help

pyDARNio does not have a dedicated development team on hand for full feature development.
However, if you are able to provide some time and help, the community will try to aid you in the best way they can to make your feature possible.

If you need help please indicate what expertise you are looking for:

• developer help with design and integration of code
• scientific help with ensuring what you develop is scientifically correct/accurate
• project management to help organize what steps need to be taken to make this possible

Please also provide any other information you may need help with, not including testing/reviewers of code

This is something I am willing to test and code review, but I have no funding for development in this project.

User Interface

# Load a fitacf file's data here into something named `fitdata`
xdata = fitdata.to_xarray()

Extra Notes

Please provide any other details about this feature

• https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html

This is a low priority feature request.

It might be nice to include some cursor control methods in the DmapRead class so that one can jump to different records in a file without reading out all of the other records in the file.

For example:

import pydarn
filename='20171231.0001.00.sas.fitacf' #any dmap file will do
dmap_handler = pydarn.DmapRead(filename)
print(dmap_handler) # prints a 0
record = dmap_handler.read_record()
print(dmap_handler) # prints something larger than 0.

begin new methods

dmap_handler.rewind() # sets cursor to beginning of the file
dmap_handler.next() # jumps to next record in the file
dmap_handler.jump(record_number) # sets the cursor to an offset where record "record_number" resides.
I've used this functionality in davitpy before, but it isn't critical to typical use cases.

Originally raised here: SuperDARN/pydarn#6

BUG

setup.py install is being depreciated, need to move to using a toml file to future proof the repo

Priority

• Minor - typo, incorrect naming, fails on specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation

Example of the bug

On installation from branch you will get the setuptools depreciation warning on newer versions of setuptools.
python3 setup.py install will no longer work but you will still be able to pip install .

Attempts

We can do the same as in the pyDARN develop branch which now uses a toml file for installation.

New Feature

Name: read_netcdf

module: in superdarn.py

package: superdarn

Scope

Will read netcdf files produced at APL into dictionary to use with pyDARN.

Code to work from:

"""
Demo read SuperDARN netCDF files available from https://superdarn.jhuapl.edu/download/
nc_utils available from github.com/alexchartier/nc_utils
Alex Chartier, 2023
"""
try:
    import nc_utils
except:
    print('Get nc_utils from github.com/alexchartier/nc_utils')
import sys

def main(fit_fn, grid_fn):
    # Read the whole file using load_nc() (provides access to header info)
    grid_file = nc_utils.load_nc(grid_fn)
    fit_file = nc_utils.load_nc(fit_fn)

    # Read the variables (a little easier to use)
    grid_vars = nc_utils.ncread_vars(grid_fn)
    fit_vars = nc_utils.ncread_vars(fit_fn)

    """ Print the details of the fit file """
    print('\n*****')
    print(fit_fn)
    print(fit_file.description)  # Note the grid_file (and other python objects) can be interrogated on the commandline using dir()
    print('Lat: %s' % fit_file.lat)
    print('Lon: %s' % fit_file.lon)
    print('Boresight: %s\n' % fit_file.boresight)
    for k, v in fit_vars.items():
        print('%s: %i entries, e.g. %1.1f' % (k, len(v), v[0]) )
    print('*****\n')

    """ Print the details of the grid file """
    print('\n*****')
    print(grid_fn)
    print(grid_file.description)  # Note the grid_file (and other python objects) can be interrogated on the commandline using dir()
    print('Lat: %s' % grid_file.lat)
    print('Lon: %s' % grid_file.lon)
    print('Boresight: %s\n' % grid_file.boresight)
    for k, v in grid_vars.items():
        print('%s: %i entries, e.g. %1.1f' % (k, len(v), v[0]) )
    print('*****\n')


if __name__ == '__main__':
    args = sys.argv
    demo_fit_fn = "~/Downloads/19940701.gbr.v3.0.despeckled.nc"
    demo_grid_fn = "~/Downloads/20150317.cly.v3.0.grid.nc"
    assert len(args) == 3, 'Should have 2 args, e.g.:\n python3 read_nc.py %s %s' % (demo_fit_fn, demo_grid_fn)
    fit_fn = sys.argv[1]
    grid_fn = sys.argv[2]
    main(fit_fn, grid_fn)

issue is related to pytables and deepdish.

I am trying to convert a bfiq file to an iqdat file with the following code:

import pydarn
import pydarnio

borealis_file = '20210113.2115.56.839612.inv.0.bfiq.hdf5'
sdarn_file = '20210113.2115.converted'
converter = pydarnio.BorealisConvert(borealis_file, 'bfiq', sdarn_file, 0, borealis_file_structure='array')

I successfully installed both pytables and deepdish on my machine and the error is related to .dll files.

The error is as follows:

'Could not load any of %s, please ensure that it %s.' % (hdf5_dlls, dll_loc))
ImportError: Could not load any of ['hdf5.dll', 'hdf5dll.dll', 'pytables_hdf5.dll'], please ensure that it can be found in the system path.

Discussion

I am working on an update to the fitacf file format to accommodate changes to the elevation angle determination in FitACF 3.0. The reason for the update is to provide more descriptive names for the elevation angle fields and their errors, while not changing the behaviour for FitACF 2.5.

Specifically, I am proposing the following changes:

1. Add two new arrays to the fitacf file format: elv_fitted and elv_error.
   This change was approved by the PIEC in November 2021.

2. When FitACF 3.0 is used, the unused XCF arrays will not be written to file (x_qflg, elv_low, elv_high, x_v, x_p_l, etc.).
   This change is not essential.

Both of these changes will break pyDARNio due to the checks for extra fields and missing fields. I would like some advice on how to implement this seamlessly across pyDARNio and RST. The update will not be included in the next RST release, so we have an open time frame.

Category

• User-friendly
• Software Design
• Data related
• Capabilities
• Clarity
• Workflow

Sample file

Here is a file with the new format, which can be used for testing: modified_format.lyr.fitacf3.zip

If you have RST installed, you can view the contents of this file using dmapdump -d [filename]

Documentation

• February 2021 DAWG report
• Draft update to the fitacf file format description in RST

Discussion

Topic: Reformatting borealis_formats.py and base_format.py to reduce the redundancy between different file types of the same version.

Description

The current implementation of borealis formats shares a lot of data fields between the different types of Borealis file (antennas_iq, bfiq, rawacf). All of these shared data fields are expected to have the same type. This ends up being a lot of boilerplate copy-paste code.

I propose that it would make more sense to define the type of each data field once per version, rather than once per file type per version. We would still probably need a class for each file type, but these classes could all share the list of data fields and types rather than redefining them. This would make it easier to define a new data format as Borealis file types evolve, as you would only need to update the list in one spot as opposed to 4.

The big offenders here in terms of copy-paste code are single_element_types() and array_dtypes() methods for each version.

Category

• User-friendly
• Software Design
• Data related
• Capabilities
• Clarity
• Workflow

It would be nice if the data write options for either BorealisWrite or SDarnWrite could allow a keyword for implicit casting or type conversion of the dictionary elements. It's quite tedious and time consuming to write basic files in testing or prototyping since the inputs need to match an exact type.

BUG

Copied from jpriestad's issue on pydarn repo SuperDARN/pydarn#209

When opening the map_2.5 file '20010817.n.map' downloaded using globus (November 12, 2021), I get the following error when opening:

SDarn_read = pydarn.SuperDARNRead(file)
data = SDarn_read.read_map()

SuperDARNFieldMissingError: Error: The following fields in record 653 are missing: {'vector.index', 'vector.mlat', 'vector.mlon', 'vector.kvect', 'vector.vel.sd', 'vector.stid', 'vector.vel.median', 'vector.channel'}

I am on main branch, installed with pip install pydarn on May 11, 2021.

To me this seems like a bug. I would have expected pydarn to just thrash the record at that particular time if the above fields are missing, and proceed to the next record. Instead, reading is interrupted. I suppose this does not mean that the rest of the datafile I have is corrupted? When commenting out lines 177-178 in '/dmap/superdarn.py' , I can see that this situation occur 4 times in my file.

In the meantime I will keep these lines commented, and assume that the rest of the file is ok to use. How to properly handle this situation, I am not sure of.

Priority

• Minor - typo, incorrect naming, fails on specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation

Attempts

read_dmap works to read in the file, so assuming it's the read_map method and not the file

BUG

Need to change np.str to str for all occurences, the newest version of numpy has depreciated this method.
Was done in pyDARN but not here.

Priority

• Minor - typo, incorrect naming, fails on specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation

Example of the bug

Occurs on import.

Potential Bug Location

in utils/conversions.py
could be elsewhere

BUG

Conversion of the Borealis rawacf hdf5 site files failed due to a NaN in the file. Tested using @RemingtonRohel's standard Borealis file and one of mine.

Priority

• Major - misrepresentation of data, failure in a common situation

Fails in Borealis data_flow's file conversion necessary for use of Borealis radar data by the wider SuperDARN community (e.g. RST etc.).

Example of the bug

from pydarnio import BorealisRead
filename = '/home/radar/Downloads/20211006.2132.00.sas.0.rawacf.hdf5.site'
borealis_filetype = 'rawacf'
borealis_reader = BorealisRead(filename, borealis_filetype, borealis_file_structure='site')
arrays = borealis_reader.arrays

The file cannot be restructured due to the following error: Records from /home/radar/Downloads/20211006.2132.00.sas.0.rawacf.hdf5.site: Error restructuring BorealisRawacfv0_5 from site to array style: cannot convert float NaN to integer
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pydarnio-1.1.0-py3.8.egg/pydarnio/borealis/borealis_site.py", line 200, in arrays
File "/usr/local/lib/python3.8/dist-packages/pydarnio-1.1.0-py3.8.egg/pydarnio/borealis/base_format.py", line 939, in _site_to_array
ValueError: cannot convert float NaN to integer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "test_pdio.py", line 6, in
arrays = borealis_reader.arrays
File "/usr/local/lib/python3.8/dist-packages/pydarnio-1.1.0-py3.8.egg/pydarnio/borealis/borealis.py", line 194, in arrays
File "/usr/local/lib/python3.8/dist-packages/pydarnio-1.1.0-py3.8.egg/pydarnio/borealis/borealis_site.py", line 207, in arrays
pydarnio.exceptions.borealis_exceptions.BorealisRestructureError: The file cannot be restructured due to the following error: Records from /home/radar/Downloads/20211006.2132.00.sas.0.rawacf.hdf5.site: Error restructuring BorealisRawacfv0_5 from site to array style: cannot convert float NaN to integer

Attempts

Tried a couple of different files, same error each time.

Data Location

https://drive.google.com/drive/folders/1j37Ez6JY_IlNAYXx2kG6IGR3aWzhMNMi

Potential Bug Location

Not sure - either borealis.py, borealis_site.py or an error in SuperDARN Borealis radar format

Potential Solution(s)

Not sure - either fix file write (outside pydarn scope) or Borealis rawacf site file read here in pyDARNio

Extra Notes

Please provide other pertinent details about this feature:
Ubuntu Linux. Python 3.8

BUG

When converting rawacf Borealis files to dmap, the last 10 ranges for all lag0 correlations are replaced by correlations from the alternate lag0 which is uncluttered at those ranges. The choice of the last ten ranges works for normalscan with 75 range gates, but not for experiments with different pulse sequences or more range gates (for example, the RKN radar with default 100 ranges).

Priority

• Minor - typo, incorrect naming, fails on specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation

Example of the bug

Hard to give an example for this in code.

Attempts

Haven't attempted a fix yet, I'm making an issue first and in discussions with Dr. Pasha Ponomarenko about the best way to manage this moving forward. It may be best handled in the borealis code, but will probably still need to be addressed here for files generated before any fix is implemented.

Data Location

Any RKN, INV, or CLY rawacf file since borealis has been installed at those sites. RKN and INV from October 2021 onwards.

Potential Bug Location

Line 858 of borealis_convert

Potential Solution(s)

Use the parameters from the borealis rawacf file to calculate the index of the second pulse in the sequence, and replace all samples after it.

Extra Notes

BUG

From pyDARN repo
author: @Yas979

Please can I check if anyone can help with the following error message? I am attempting to create map plots of the convection electric field, using pydarn.SuperDARNRead to extract the map data from the .map file. This has worked in earlier versions of pydarn. However, the latest version of PyDarn results in the following error:

mapfile='/data/sol-ionosphere/sd_map_potential/fitacf3_allrad/north/map/2011/20110101.fit.map'
SDarn_read = pydarn.SuperDARNRead(mapfile)
map_data = SDarn_read.read_map()

Error: /data/sol-ionosphere/sd_map_potential/fitacf3_allrad/north/map/2011/20110101.fit.map contains an block size 540160288 > remaining bytes 69250442 at record = 0.
Traceback (most recent call last):
File "", line 1, in
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/superdarn.py", line 517, in read_map
self._read_darn_records(file_struct_list)
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/superdarn.py", line 393, in _read_darn_records
self._read_darn_record(format_fields, optional_list)
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/superdarn.py", line 359, in _read_darn_record
record = self.read_record()
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/dmap.py", line 449, in read_record
self.bytes_check(block_size, "block size",
File "/alice-home/1/y/ys378/alice3_python_env/env/lib64/python3.9/site-packages/pydarnio/dmap/dmap.py", line 286, in bytes_check
raise dmap_exceptions.MismatchByteError(self.dmap_file,
pydarnio.exceptions.dmap_exceptions.MismatchByteError: Error: /data/sol-ionosphere/sd_map_potential/fitacf3_allrad/north/map/2011/20110101.fit.map contains an block size 540160288 > remaining bytes 69250442 at record = 0.

Notes

Follow up:
From @carleyjmartin
If you could email/transfer me the file you are trying to open at [email protected]? It looks like the file itself doesn't fit the format of what pydarnio is expecting in read_map for a map file, have you tried to dmapdump using RST to check the contents of the file (don't worry if not it'll just be the first thing I do when I have the file)?

You could also try read_dmap() or read_records() instead of read_map() but I suspect you might still get the same error - let me know if that is the case if you give that a go!

Also could you let me know what version of pydarn and pydarnio you have installed, you can find it using pip3 list. Updating pydarn does not always update the pydarnio version, but it should be >1.1.0, we haven't really updated the lines causing the error for quite a few years, so if you can read this file using an older version of either pydarn/pydarnio, is it possible to let me know which version so we can track back and find out?

Thanks, Carley

SuperDARN/pydarn#354

Reported on pyDARN: SuperDARN/pydarn#122
While accessing some of the map2 files I'm getting this error

Error: The following fields in record 462 are missing: {'vector.mlat', 'vector.vel.median', 'vector.mlon', 'vector.kvect', 'vector.vel.sd', 'vector.channel', 'vector.index', 'vector.stid'}
Traceback (most recent call last):
File "convert_map_to_csv.py", line 22, in read_map_files
_md = reader.read_map()
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 493, in read_map
self._read_darn_records(file_struct_list)
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 374, in _read_darn_records
self._read_darn_record(format_fields)
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 350, in _read_darn_record
SDarnUtilities.missing_field_check(format_fields, record, self.rec_num)
File "/home/shibaji/anaconda3/envs/collb/lib/python3.7/site-packages/pydarnio/dmap/superdarn.py", line 178, in missing_field_check
SuperDARNFieldMissingError(rec_num, missing_fields)
pydarnio.exceptions.superdarn_exceptions.SuperDARNFieldMissingError: Error: The following fields in record 462 are missing: {'vector.mlat', 'vector.vel.median', 'vector.mlon', 'vector.kvect', 'vector.vel.sd', 'vector.channel', 'vector.index', 'vector.stid'}

File - /sd-data/2012/map2/north/20120309.north.map2.bz2

I assume the issue is once again partial records

The license badge in master branch README lists the license as LGPL v3.0, whereas the actual license in the repo is GPL v3.0. Which is the intended license for this repo?

BUG

Unable to restructure Borealis 0.6.1 antennas_iq file using the BorealisRestructure class

Priority

• Minor - typo, incorrect naming, fails on specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation

Example of the bug

Converting antennas_iq file produced from Borealis v0.6.1 on INV:

python3 borealis_convert_file.py --low_memory 20221104.1400.00.inv.0.antennas_iq.hdf5.site
Traceback (most recent call last):
  File "/home/transfer/data_flow/site-linux/borealis_convert_file.py", line 264, in <module>
    main()
  File "/home/transfer/data_flow/site-linux/borealis_convert_file.py", line 233, in main
    low_memory)
  File "/home/transfer/data_flow/site-linux/borealis_convert_file.py", line 191, in borealis_site_to_array_file
    outfile_structure='array', hdf5_compression='zlib')
  File "/home/transfer/pydarnio-env/lib/python3.6/site-packages/pydarnio/borealis/borealis_restructure.py", line 136, in __init__
    self.software_version][self.borealis_filetype]
KeyError: 'v0.6.1'

Data Location

Antennas_iq files available on INV NAS.

Shall we set up a release as is?

The README.md file contains the following link to the pydarnio readthedocs documentation:

https://pydarnio.readthedocs.io/en/master

However, that leads to a broken page and should probably be updated to this:

https://pydarnio.readthedocs.io/en/latest/

BUG

When converting from HDF5 to DMAP, files with multiple beams in a record only have the last beam converted for each record. I'm not very familiar with the DMAP file standards, so I'm not sure how would be best to store multiple beams.

Priority

• Minor - typo, incorrect naming, fails on specific uncommon situation
• Major - misrepresentation of data, failure in a common situation
• Urgent - failure in usage, large misrepresentation

Example of the bug

Hard to show this in code, but when testing out a multiple-beam file with Dr. Pasha Ponomarenko, only the last beam made it from the HDF5 file to the DMAP file (rawacf files).

Attempts

Nothing yet

Potential Bug Location

Lines 590-757 and 855-1004 of borealis_convert.py (converting records: bfiq->iqdat, rawacf->dmap)

Potential Solution(s)

Any beam-specific values will need to be extended for multiple beams, i.e. make it a list instead of just a number. I don't know what the implications will be downstream.

Extra Notes

• OpenSuse Leap 15.3
• Python version 3.6

DEPRECATION

package: deepdish

Reason

See this PR and this issue on the deepdish github. Newer numpy installs are going to have problems, so we can update now to avoid any issue.

Replacement

The h5py library is already used in places within pyDARNio, and is supported by the HDF5 group. It is slightly less convenient to use, but much faster than deepdish.

Details

This is a convenient time to change the structure of some Borealis files - namely, fields such as data_descriptors and correlation_descriptors which are lists of utf-32 strings stored as one large array of uint8 type. It's a headache to write and read these fields with h5py, so this is a prime opportunity to refactor the data format to something more user-friendly. There may be some headache in dealing with scan_start_marker. Trying to read in the field with h5py yields:

>>> rec.attrs['scan_start_marker']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/remington/pydarnio-env/lib64/python3.6/site-packages/h5py/_hl/attrs.py", line 61, in __getitem__
    dtype = attr.dtype
  File "h5py/h5a.pyx", line 334, in h5py.h5a.AttrID.dtype.__get__
  File "h5py/h5a.pyx", line 336, in h5py.h5a.AttrID.dtype.__get__
  File "h5py/h5t.pyx", line 422, in h5py.h5t.TypeID.py_dtype
TypeError: No NumPy equivalent for TypeBitfieldID exists