Ali and Darrelle noted that when manual ticks (tickValues) are used that excessive and inconsistent resolution is used. For example, for angles we have 0,50,100,150 by default, but then when +30 is requested, then 0.0,30.0,60.0,90.0,120.0,150.0, and 180.0 are printed. Ali had to go through and manually block out the extraneous ".0" on each number.

When data is to the right of the plot window, the error message is "dataset contains no plottable data" but when data is to the left it is "data ends before range". This needs to be corrected to match the left. See https://github.com/autoplot/dev/blob/master/bugs/2022/20220809/demoBugError.jy

The createEvents code would accept a n by two-element dataset of start times and stop times and return a canonical bundle dataset of events, but the stop time was used for the label. This would confuse Autoplot's autoranging, and is inconsistent with the behavior of other branches of the same code which use gray as the default color and "_" as the default message.

This is fixed.

Quite a few places use the layout strings like "50%+3em". There should be a specialized editor for these strings, like there is for tick spacing.

Chris asked if Autoplot supported bold text in its rendered labels, and I was a little surprised I hadn't put that in. I've added support for <b> for bold (</b> to turn off) and <i> for italics.

I have a time series which shows two issues with the rendering. First, the cadence is dependendent on what's shown. I thought this was fixed but this data seems to tweak a bug. Second, if an outlier is off screen, connectors aren't drawn, and there is no way to know that the point is there.

I just removed a warning from QDataSet where a dataset unbundled from a bundle is itself a bundle. I don't think this is actually a problem, and QDataSet allows this.

However, there are other limits like this and this might be a good one to verify. For example, slicing any dataset (slice0) will always produce a qube, and this allows nice assumptions to be made.

All of these limits need to be explicit and well-documented.

The GrannyTextRenderer has a means for allowing custom drawing codes in. For example, there's an Autoplot example which imports a LaTeX library and uses it to render LaTeX within a string. This would be a great feature, but unfortunately, the code that calculates the size of the draw region has an issue, I think.

The Autoplot script is here: https://github.com/autoplot/dev/blob/master/rfe/sf/625/latexPainter.jy

Keith pointed out issues with the FFTPower routine with Cluster data, where transitions and waveform cadence changes were not handled properly. These were resolved and released in Autoplot v2023a_2, but some issues still remain.

The output of FFTPower will be modified with a kludge when the waveform units are mV/m to make the output be (V/m)^2/Hz. This is the one special case where the FFTPower output has units, usually the output is dimensionless. See https://github.com/das-developers/das2java/blob/main/QDataSet/src/org/das2/qds/util/FFTUtil.java#L207
It might be better to report a unit of "(wavUnits)^2 * cadenceUnits" so that there is always something, but note that there is an SIUnits class which does the math. I've always wanted to use this, but I know it will cause all sorts of new problems.

The FFTPower library says the window weight is not accounted for and then three lines later a comment says it is. I believe it is, but this should be verified.

There is a check for fixed cadence within a packet, checking that the entire packet length (in time) is consistent with the length (of time) between the first ten measuments. This continues for translation, but drops the non-fixed-cadence packet when there is no translation.

Last, there are three FFTUtil codes! Two within QDataSet, one in DasCore. I'm pretty sure these all have similar static routines, and one should be used and the others deprecated. (The extra one in QDataSet can probably be removed, since I don't think any third parties are using it.)

• units should always be reported
• window weight normalization
• check for one cadence in packet which continues for translation but not without translation
• multiple FFTUtil codes

In Autoplot, you can see where a ticks logic doesn't consider the height of the Y-axis
when picking ticks for a log axis:

1. plot 48.0 to 970.0, log, for the yaxis
2. tear off the canvas
3. make the plot shorter or taller, and note that the number of ticks doesn't change.

James at APL observed that a relatively trivial png-walk creating code would spin up many threads and warnings the DasUpdateListener count. There's code that prints a warning when the count is greater than 100, and this is reporting for him. (DasDevicePosition.java, addUpdateListener(DasUpdateListener) )

The PNG output includes an extra timestamp tag ("*tEXtCreation Time"), which makes it so the same png file written twice cannot be compared using file comparisons. It would be nice to be able to disable this tag.

Ali pointed out about a month ago that when the annotations are told to point at something, the drag renderer which shows the arrow during the drag doesn't draw. This is also the case for the move annotation mouse module. I was looking into this now, and I found that it's because getMouseDragEvent in ArrowDragRenderer returns null, when others return something here. Experimenting with this, I found that simply returning "new MouseDragEvent(source)" will allow it to work as I remember.

I found that v2020a_1 works fine, and v2021a_1 shows the bug. I will improve the resolution here shortly to isolate the change.

I just noticed as Bill was navigating around during a zoom meeting that he gets progress wheels
where the x axis labels aren't visible, hinting that it's busy loading data even though it's not needed.

James and I were looking at where the plot title is clipped by the yaxis, but then the yaxis doesn't really use the real estate. We think there should be an option to allow components to be more willing to overlap one another.

James also pointed out that while the DasAxis has a nice DEBUG_GRAPHICS mode, the DasPlot doesn't have this feature. Other components, like the DasAnnotation, would be well-served by this too.

• 1. shared real estate
• 2. DEBUG_GRAPHICS for DasPlot

Keith nicely isolated a case with Cluster WBD data where a change in the data cadence results in inconsistent results, as if a constant cadence throughout the operation is assumed.

I just noticed the DataPointRecorder GUI sorts by column using an alpha sort, not numeric. I thought I could use the digitizer, then sort by the "sigmas" column, then delete the low ones, but I can't because of this limitation (and it would be easy to think it was a numeric sort, leading to mistakes). The units of the column should determine the sort type.

A noisy test of Autoplot (https://cottagesystems.com/jenkins/job/autoplot-test005/18804/, 5th bookmark) showed where there's an instability in the Das2 code. The Units class builds up a table of converters, and then these converters are automatically connected such that new conversions work automatically. (cm→m and cm→ft can derive m→ft.) This has a problem though that the order that units things are added matters, as different paths can be derived. Logging would trigger a different path and the bug would show in test005. Change the log levels or logger format, and the bug would go away.

Note too that though there was a direct route from cdfEpoch to "hr since 2001-01-01T00:00:00.000Z" the conversion from ms to hr might not be. (Something doesn't jibe there, since ms to hr should be quite stable, so it might be a different offset conversion, but this should be considered as well.)

To resolve this I'm going to take out the conversion part of the code which decomposes times and make sure this is stable.

For years the logic for the log axis ticks when only about one cycle is visible is to use linear ticks. This isn't quite right, where really it should be linear ticks within the log cycles: 1,2,3,4,5,6,7,8,9,10,20,30,40,50,60,70,80,etc. I'm noticing other systems are getting this right and Das2 plots look sloppy here. Further, the logic here has multiple paths, so if you drag the axis it will flicker as different algorithms are used. This should be corrected with one, consistent logic.

I'm leaving a note here to remind us that the repository should probably have a github workflow to automatically build and run a few unit tests on the main branch after a push or a pull-request merge.

I can kick this off and turn it over to others to maintain, but to get started I'd need a reference or two to the current unit test main functions or some other information about tests that can be run without a full MIDL or Autoplot build.

Also instructions for building via Ant without the netbeans frontend would be handy.

Ivar's events file had "mms-ev1_01" and the dash was interpreted to mean "through." Now the through operator will only work
with numbers (\d+-\d+).

A few people have asked for a way to offset the dasAxis from the plot, so a Z-stack of plots axes can be used to control parameters. Right now you can have a left axis and a right axis, but this will allow for any number of parameters and the axis can always be on the left side.

DasAnnotation has always had three modes to anchor (position) the annotation: canvas, plot, and data. I think the original idea was that "data" was going to allow anchoring to data, where the bounds of the data were going to be used to position the data. This never happened, and a simple bounding box (xrange = 20 to 25; yrange = 40 to 45) is allowed and Autoplot scripts are used to identify the data bounds. The "plot" mode was going to anchor to axes, but is never used because this is what the data position does. (Canvas just anchors to a row and column, or null values are just the entire canvas.) Further complicating things, we later added a verticalAnchorType which allows the two dimensions to be independently controlled.

I'd like for the "plot" to be made useful, where inside anchorPositions will clip using the plot clip, and outside anchorPositions will clip using reasonable boundaries so that adjacent plots are not affected. (Item 1)

There is additional cleanup which is needed, such as an improved GUI to control it, which Autoplot could use. (It has a quick one I threw together, but I don't think it's very effective.) (Item 2)

One still cannot rotate annotations, and this is an obvious feature that is overdue (Item 3). The properties "url" and "scale" should be deprecated since painters can be used for images (Item 4). "<img>" tags were supposed to be supported as well, but that's a different ticket (Item 5).

Last, there needs to be a clean document describing annotation use (Item 6).

See #21, and Autoplot ticket https://sourceforge.net/p/autoplot/bugs/1696/ .

• 1. Precisely define "PLOT" anchorType to mean anchor to a plot row and column, with clipping.
• 2. Improve GUI for controls
• 3. Allow for annotation rotation (90 degree, then arbitrary)
• 4. deprecate "url" and "scale", removing these from GUIs
• 5. support <img> in GrannyTextRenderer (#21)
• 6. new, accessible, and correct documentation
• 7. correct "point at" which moves the annotation.
• 8. "move annotation" adjusts anchorOffsets, but does not consider splitAnchorType.
• 9. "move annotation" sign errors.
• 10. add "underscore" border type.
• 11. adjustment for tightness of margins.
• 12. "data" anchors to the pointAt position, not the xrange, yrange box. This is very confusing and it should be the box. (Is this the same as item 7?)
• 13. AnchorType canvas should support pointAt when a plot is identified.
• 14. The anchor box controls should be similar to the series renderer controls.
• 15. The editor can have multiple tabs to simplify controls
• 16. The anchor rectangle can be marked with ellipse

Back in June I added to QDataSet the new metadata AVERAGE_TYPE, which can specify that data should be averaged with geometric averages or the data exists in a "mod360" realm and (259+1)/2 = 0. This could have a nice application with the series renderer when it draws a line from 259 to 1 degree, or averages data in spectrograms.

Kristoff and I have needed a way to control the color along a curve. You can use the "color scatter" mode and bundle(X,Y, Z) where Z is an RGB color, but this is just with the points and there's no way to do this with connected points. I needed it to show s/c mode along a trajectory, and Kristoff needed it to do colored contours.

Chris has a case where he would like to use an ASCII file to label an axis with additional ephemeris ticks (TCA). This is showing a number of shortcomings of the mechanism, which will be enumerated here.

The Das2 Server has always had the problem that you could pass in a parameters string to a data source, but the client code would have to know how to interact with the source. Sometimes this string is simple, with "LFRL" meaning just send over the LFRL mode, and sometimes very complex like for the Cassini data readers. Chris has developed a specification for these reader params which should support all Das2 readers, and has a website that uses the specification to build HTML GUIs, https://das2.org/browse. This site, the federated data catalog, also addresses other issues like describing redundancy, but this ticket focuses on the problem of the reader parameters.

Autoplot provides a space to insert this parameter string, and Bill and the other scientists are able to manage with it, inserting the magic strings which they know will provide the data they need. The reader DSDF is visible to them as a pop-up, and the name/value pairs PARAM_00, PARAM_01, etc are used to convey what they should enter.

I'd like to create a GUI using the federated data catalog's specification and provide it as a means for creating the magic parameter string. I imagine the scientist would have two tabs in the parameters area, one as before to enter an arbitrary string and the other the GUI in a scroll pane. This would be done for Autoplot, but would be a Java code which is useful in custom Das2 applications as well.

Some references:
https://das2.org/browse/uiowa/juno/wav/survey
http://das2.org/catalog/das/site/uiowa/juno/wav/survey/das2.json
https://das2.org/browse/uiowa/cassini/rpws/survey

I've been thinking about adding QDataSet.CADENCE_MIN and QDataSet.CADENCE_MAX, which would be alternatives to QDataSet.CADENCE. This would fix a problem were sometimes you want to know an upper or a lower limit, and this would allow codes and datasets to support this. I've always imagined that QDataSet.CADENCE might be allowed to be a rank 1 dataset, and this might be a better solution here as well.

As with any QDataSet property, any property can be missing, and the three must be considered as group now, CADENCE_MIN=CADENCE_MAX -> CADENCE=CADENCE_MIN, etc.

James requested that the new horizontal interval interval average have a couple more changes:

1. Recall you added the “Horizontal Interval Average” tool that operates on a spectrogram plot. Two small issues with this: one is that the horizontal axis is blank -- would it be possible for it to pull the horizontal axis label from the parent plot, the way all the other slicers/averages do?
2. The other issue with the “Horizontal Interval Average” is that the window title is “Horizontal Spectrogram Average”. These should presumably match, and probably the title “Horizontal Interval Average” is the better choice -- more generic -- OK?
3. Finally, (again, when will you learn that your good deeds never go unpunished?) would you be able to add the options to the Vertical Interval Average that you recently added to the Horizontal version of the tool? The horizontal one has the choice of Average or Sum, but currently the vertical version doesn’t.

Up until now it's been allowed for DatumRanges to contain fill values. While this might cause many codes to break I think we need to consider disallowing this. These are typically something you need to deal with anyway, and the maxim fail early so that bugs don't reveal themselves far away from their source says this should be done.

https://github.com/autoplot/dev/blob/master/bugs/2023/20230210/demoDatumRangeFill.jy

If you list a directory and then list it again within 10 seconds, you will get a cached listing. This is also true of files, where it will always return a file which has been quite recently written to the cache. This is not something that can be controlled, and it would be nice if the application could control it. For example, sometimes it's nice to know that no web activity will occur during a batch run. See WebFileSystem.MEMORY_LISTING_TIMEOUT_MS.

Right now an annotation can be a granny text string or an image. When it is an image, it is the URL of the image. This allows somewhat arbitrary images to be inserted onto the plot. However, since granny text strings allow <b>, <i>, and <u> it would make sense to add support for images within the strings as well, and then the strange image mode for an annotation could be dropped. This would also allow a figure with a caption to be inserted using a granny text string.

In Autoplot, the annotation command

annotation(text='<img src="https://das2.org/das2new-300.png"><br>Das2 Logo') could be used.

A special editor for tick locations is needed. This will be used to create strings like "+10/5" which means a tick every two units with five divisions per major tick. These specs have been effective, used with the axes and contour plots, but potentially with any place intervals need to be enumerated.

This editor could have an axis showing the proposed tick locations as feedback, and have examples in a droplist, and a description of the specification.

On Autoplot there's been a ticket about manual positioning of the ticks (https://sourceforge.net/p/autoplot/feature-requests/469/), and this should really be in the das2java project. There's a consistent method for specifying tick locations with the tickValues property of the axes. This can be:

• "" default is automatic
• "+15deg" make a tick every degrees
• "0,15,30,45,60,75,90" explicit tick positions
• "+15minute" make a tick every 15 minutes
• "2015-001T00:00,2015-001T04:00"
• *10 ticks at 1,10,100,1000 etc
• "+15deg/5" every 15 degrees with 5 minor tick intervals.

The DasAxis should provide a "scan next" button when the "step next" action would advance to a frame with no data. This feature has a few problems:

• "scan" action doesn't show initially, must step first
• scanning to the next frame steps an integer number of times, when it would be more appropriate to recenter.
• sometimes the "scan" or "step" button doesn't appear, though clicking there would take the action.

Someone (Kris?) recently asked if the data point recorder could automatically set its table column widths based on the data loaded. This should be pretty easy to do, and I wouldn't be surprised if there's a method for automatically setting the widths. Also, I don't believe it scrolls, and it should be put within a JScrollPane when the columns are too wide.

Jon N at UNH ran into a problem where his system had created a file with the date 20221131 (2022-11-30 is the last valid day), but Autoplot treats it as if it is 2022-12-01, and would continue without complaint. This creates confusing and difficult-to-diagnose problems, and some more validation should be done.

In Autoplot:

1. load vap+hapi:http://amda.irap.omp.eu/service/hapi?id=cass-caps-snghr&parameters=Time,cass_caps_sng_sum&timerange=2012-05-31+22:29+to+2012-06-01+05:30
2. take horizontal slice

Over the years I've noticed where there are breaks in a spectrogram at low frequencies, often when looking at the result of fftPower in Autoplot. Corentin pointed this out with his spectrogram, too. This is a bit of his spectogram:

A review of the guess cadence routine needs to be done.

Corentin had a case where the QubeDataSetIterator couldn't iterate over the data, because it contained empty datasets. Quite a bit of care was needed to check for these cases and to advance properly, and I think the problem is fixed now. However I'm running across things that don't work, and there should be a ticket to record impacts.

A low-priority and fun project would be to clean up the contours plot. With recent changes to the tick specification, I realized I ought to support major and minor contours. +20/4 would draw a major contour every 20 units, with lighter and unlabelled minor contours at 5,10, and 15 units past. (This is refered to as (1) below.)

(2) Also, the labels for each contour need to be oriented so that the length of the name will be aligned with the contour. See the attached plot to see the arbitrary orientation of each contour because of noise which is much higher frequency(<1px) than the label (30px).

Sadie showed where a das2stream could be produced containing times like "2017-11-27T12:20:12.-00". This is when the milliseconds are nearly 0. To demo the bug:

• plot vap+cdaweb:ds=RBSPA_EFW-L2_FBK&filter=filterba&id=fbk7_scmw_pk_lowoverlay&timerange=2017-11-28
• zoom in to 2017-11-28 11:48:40 to 11:49:20
• horizontal interval average (which does sum over frequency)
• right-click on the DasPlot and "dump data to file"
• grep for "-00" in the output.

Sadie pointed out a data set that performs horribly with the SeriesRenderer, where it was about 40,000 points, plotted on a log Y axis, and there are many zeros which end up plotting at -10000. It appears that the graphics engine is painting a bunch of stuff that's off screen, because when I limit the distance to just off screen performance is dramatically improved. The data is fed into a newer class (2017) called the DataGeneralPathBuilder, which manages the pen location and up/down motions.

This does point out that it might be a good time to start introducing some optimizations, however. For example, the three co-linear points could be drawn as one line segment, and reasonable approximations for co-linear would make this happen often. The "Huge Scatter" renderer in Autoplot would keep track of the min and max within a column of pixels and draw one segment. This has the virtue of scaling to any size data set, because the complexity is limited by screen resolution, not the number of data points.

Likewise, nice optimizations can be made for tiny and indistinguishable movements within a pixel which can be collapsed.

When working to clean up my HAPI server, I ran across an input which causes a misformatted timerange:

time.min=2014-01-09T00:00:00.001Z&time.max=2014-01-09T00:00:00.002Z

This will be formatted as "2014-01-09 through 2014-01-08". The following code does not work to demo the bug, but something similar probably would:

tr= datumRange('2014-01-09T00:00:00.001Z/2014-01-09T00:00:00.002Z')
print tr

We were asked to verify our fftPower routine, and after finding the script
https://github.com/autoplot/dev/blob/master/demos/2023/20230209/checkPowerFFT.jy at http://www-pw.physics.uiowa.edu/~jbf/rbsp/script/chris/powerfft/checkPowerFFT.jy, I could see that it was off by a factor of 2. The code contained a comment about the factor 4 being incorrect and a factor of 2 should be used.

I reviewed other functionally similar codes (there's a more general version which is used widely in Autoplot, but it handles more cases and is harder to follow), and found these use the correct factor of 2.

I've made the code used in checkPowerFFT.jy consistent, all are using 2 now.

Ivar and I ran into a difficult bug to diagnose when we has the expression:

ds.lt(100).or(valid(ds).eq(False))

The problem is that where ds is not valid, the "or" result would also be invalid, making the logic be incorrect.

We should consider whether (True.or(Fill)) should be True, since the or has a True argument. The impact on all operators, making sure that rules like commutativity follow.

For years I've dealt with a problem where adding TCAs (ephemeris) to an image can result in slightly unpredictable images when scripting and generating PNGWalks. There's a mechanism which should block the "waitUntilIdle" calls that is failing as TCAs are loaded.

This was originally created as an Autoplot bug, but the bug is really in Das2. See https://sourceforge.net/p/autoplot/bugs/2442/.

There's a bug that the FileSystem keychain is misimplemented for HttpFileSystems. This is because it was originally added to support FTPFileSystems, which are based on directory, and HTTP directories are not secured this way. The credentials should be paired with the "basic realm" specified in the 401 response. Any two directories using the same basic realm use the same credentials.

This has caused a number of problems. For one, the keychain file used to store the credentials does so based on the directory. So you might end up with credentials attached to http://autoplot.org/autoplot/data/agg/efi/1999/ and then not have credentials for http://autoplot.org/autoplot/data/agg/efi/2000/. (Note this folder doesn't require credentials.) This also means kludgy logic ascends through the folders looking for passwords in the keychain.

The das2server uses the same 401 auth mechanism, but it has it implemented properly. Perhaps the keychain for the das2server can also serve as the keychain for filesystems.

See https://datatracker.ietf.org/doc/html/rfc7617.

Further, there's a bug where it fails to realize that it could ask for credentials to get more data. See the procedure below which shows an error similar to what Kristoff is seeing.

Autoplot initializes and for one paint you can see it paints the roughly 100-by-100 pixels of the canvas before properly drawing the entire canvas. This is because its size must be set on a different thread, so the axis is drawn at full height though the clip region is small. Other inconsistencies are found in the DasAxis, and as the component is resized, the axis bounds will jump around a bit. Note that everything is drawn properly after the transition, but the jumpiness and partial draws are sloppy and distracting.

Ivar has a case where the data has a VALID_MAX of 1.0, and most of the data is at 0.999 or 1.0. When plotting with the series renderer, the data triggers the reduce branch, since there are many points. The reduce branch looks at the axes, and then reduces the data. However, this has the effect of slightly changing the reduced values, to a point where the bin is above 1.0 and they are all marked as invalid.