Reference: opensatelliteproject/OpenSatelliteProject#8

https://www.dropbox.com/s/2q0r2f2swlx247c/demuxdump-1492400420%20-%20Copy.bin?dl=0

7:16:13 AM/INFO  I don't have a demuxer for VCID 56. Creating...
7:16:31 AM/INFO  New HIMAWARI8 ABI - Channel 3 (IMG_DK01IR1_201704171350_007.lrit)
7:17:07 AM/ERROR Lost -12247583 frames. Last Frame #12247582 - Current Frame #0 on VCID 0
7:17:08 AM/ERROR Lost -9342745 frames. Last Frame #9342744 - Current Frame #0 on VCID 1
7:17:08 AM/ERROR Lost some frames on MSDU, the file will be corrupted. CRC Match: False - Size Match: False
7:17:17 AM/ERROR Lost -883619 frames. Last Frame #883618 - Current Frame #0 on VCID 61
7:17:30 AM/INFO  New HIMAWARI8 ABI - Channel 7 (IMG_DK01IR3_201704171350_002.lrit)
7:18:01 AM/ERROR Lost -1454799 frames. Last Frame #1454798 - Current Frame #0 on VCID 13
7:18:09 AM/ERROR Lost -16171254 frames. Last Frame #16171253 - Current Frame #0 on VCID 23
7:18:17 AM/ERROR Lost -9842067 frames. Last Frame #9842066 - Current Frame #0 on VCID 33
7:18:23 AM/INFO  New GOES 15 ABI - Infrared United States (gos15chnIR04rgnUSseg001res04dat107141753521.lrit)
7:18:24 AM/INFO  New GOES 15 ABI - Visible United States (gos15chnVS01rgnUSseg001res04dat107141753334.lrit)
7:18:25 AM/ERROR Lost -1057254 frames. Last Frame #1057253 - Current Frame #0 on VCID 60
7:18:32 AM/INFO  New GOES 15 ABI - Water Vapour United States (gos15chnWV03rgnUSseg001res04dat107141753540.lrit)
7:18:38 AM/INFO  New HIMAWARI8 ABI - Channel 7 (IMG_DK01IR3_201704171350_003.lrit)
7:19:43 AM/INFO  New GOES 15 ABI - Infrared United States (gos15chnIR04rgnUSseg002res04dat107141908643.lrit)
7:19:47 AM/INFO  New GOES 15 ABI - Visible United States (gos15chnVS01rgnUSseg002res04dat107141908419.lrit)
7:19:53 AM/INFO  New GOES 15 ABI - Water Vapour United States (gos15chnWV03rgnUSseg002res04dat107141908662.lrit)
7:19:53 AM/ERROR Lost -13815201 frames. Last Frame #13815200 - Current Frame #0 on VCID 41
7:20:20 AM/INFO  New GOES 13 ABI - Infrared Northern Hemisphere (gos13chnIR04rgnNHseg001res04dat107141919028.lrit)
7:20:20 AM/ERROR Lost -1195249 frames. Last Frame #1195248 - Current Frame #0 on VCID 59
7:20:34 AM/INFO  New HIMAWARI8 ABI - Channel 7 (IMG_DK01IR3_201704171350_004.lrit)
7:20:34 AM/ERROR Lost -1420740 frames. Last Frame #1420739 - Current Frame #0 on VCID 58
7:21:16 AM/INFO  New GOES 15 ABI - Infrared United States (gos15chnIR04rgnUSseg003res04dat107142028754.lrit)
7:21:21 AM/INFO  New GOES 15 ABI - Visible United States (gos15chnVS01rgnUSseg003res04dat107142028514.lrit)

Goesdump stops generating images for channel 15 soon after starting. It generates images for channel 14 for a time but this channel sometimes begins to lag behind too. Occasional restarts of goesdump force the images to generate.

The current text output on HRIT for many products is way too wide. Incoming file name along with output file name but with full path easily fills and wraps. Perhaps look at LRIT console logging and make HRIT logging like that. Or add a knob to allow user to turn down width verbosity.

https://www.dropbox.com/s/0elpei8dtbiutro/demuxdump-1492397472.bin?dl=0

https://www.dropbox.com/s/771ko1szews8d0m/xrit_errors.log?dl=0

20:46:09 ERROR   Error processing image (SysExcpt) 1492486837-GOES 15-Special interest or unspecified: System.IO.FileNotFoundException: Could not find file 'F:\sw\goes_xrit_viewer\goes_xrit_viewer\bin\Debug\channels\Images\Area of Interest\gos15chnWV03rgnXXseg001res04dat107034209587.lrit'.
File name: 'F:\sw\goes_xrit_viewer\goes_xrit_viewer\bin\Debug\channels\Images\Area of Interest\gos15chnWV03rgnXXseg001res04dat107034209587.lrit'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(String path)
   at OpenSatelliteProject.Tools.FileParser.GetHeaderFromFile(String filename) in F:\sw\goes_xrit_viewer\goes_xrit_viewer\XRIT\Tools\FileParser.cs:line 13
   at OpenSatelliteProject.ImageTools.GenerateFullImage(OrganizerData data, Boolean crop) in F:\sw\goes_xrit_viewer\goes_xrit_viewer\XRIT\Tools\ImageTools.cs:line 105
   at OpenSatelliteProject.ImageManager.ThreadLoop() in F:\sw\goes_xrit_viewer\goes_xrit_viewer\GoesDecoder\ImageManager.cs:line 180

20:46:09 DEBUG   Starting Generation of Water Vapour for GOES 15-Special interest or unspecified-WV-1492486837.png.

20:46:09 ERROR   Error processing image (SysExcpt) 1492486837-GOES 15-Special interest or unspecified: System.IO.FileNotFoundException: Could not find file 'F:\sw\goes_xrit_viewer\goes_xrit_viewer\bin\Debug\channels\Images\Area of Interest\gos15chnWV03rgnXXseg001res04dat107034209587.lrit'.
File name: 'F:\sw\goes_xrit_viewer\goes_xrit_viewer\bin\Debug\channels\Images\Area of Interest\gos15chnWV03rgnXXseg001res04dat107034209587.lrit'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(String path)
   at OpenSatelliteProject.Tools.FileParser.GetHeaderFromFile(String filename) in F:\sw\goes_xrit_viewer\goes_xrit_viewer\XRIT\Tools\FileParser.cs:line 13
   at OpenSatelliteProject.ImageTools.GenerateFullImage(OrganizerData data, Boolean crop) in F:\sw\goes_xrit_viewer\goes_xrit_viewer\XRIT\Tools\ImageTools.cs:

When disabling weather data file generation the ingester is not precise. Charts, text etc. are still generated. Many users don't care about this low res stuff and the files add up quickly and are a pain to manually delete. Please add the appropriate product/sub-product ids to make weather data disablement more accurate.

Would it be possible to make timestamps in PNG generated images in UTC?

Wrong cropping:

Occasionally goesdump attempts to generate a false color mesoscale but color is not applied to the entire image: https://wx-star.s3.amazonaws.com/goes/hrit/2018/2018-01-17/CMI%20Products/Mesoscale/GOES-16/False%20Color/G16-Mesoscale-FSCLR-1516204796-2018-01-17T1559Z-watermark-size100.png

This should be fixed or disabled altogether.

"Can there be an option to remove any files in the channels directory over 7 (or 14, etc) days old? Images, lrit, etc. Anything older than X days. Basically to keep the directory size from growing massively large"

Running latest package from nuget. Example full disk with outlines:

https://www.dropbox.com/s/do4ts2jzc61jg94/G16-Full%20Disk-VIS-1537195531.png?dl=0

Here is recording so you can see good meso and bad full disk:

https://www.dropbox.com/s/7ayqzb6u20s7atq/demuxdump-1537197025.bin?dl=0

Reference: opensatelliteproject/OpenSatelliteProject#6

In the case of EMWIN on HRIT, there are errors (perhaps on uplink) that results in bad decompression. These are not currently recorded. Add a counter so it obvious to the user that over time there is a problem with decompression. I suppose you could also have one for RICE decompression failures too.

Reference to: opensatelliteproject/OpenSatelliteProject#13

Reference: opensatelliteproject/OpenSatelliteProject#3

Now we will have Informative Label support with XRIT 1.2. The "OpenSatelliteProject blalba" in the bottom is customizable for anything.

Setting RecordToFile to True, False, True periodically while running will result in following:

21:05:49 ERROR Error writting demuxdump file: System.ObjectDisposedException: Cannot access a closed file.
at System.IO.__Error.FileNotOpen()
at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
at OpenSatelliteProject.DemuxManager.parseBytes(Byte[] data) in F:\sw\goes_xrit_viewer\goes_xrit_viewer\DemuxManager.cs:line 117

21:05:49 ERROR Error writting demuxdump file: System.ObjectDisposedException: Cannot access a closed file.
at System.IO.__Error.FileNotOpen()
at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
at OpenSatelliteProject.DemuxManager.parseBytes(Byte[] data) in F:\sw\goes_xrit_viewer\goes_xrit_viewer\DemuxManager.cs:line 117

21:05:50 ERROR Error writting demuxdump file: System.ObjectDisposedException: Cannot access a closed file.
at System.IO.__Error.FileNotOpen()
at System.IO.FileStream.Write(Byte[] array, Int32 offset, Int32 count)
at OpenSatelliteProject.DemuxManager.parseBytes(Byte[] data) in F:\sw\goes_xrit_viewer\goes_xrit_viewer\DemuxManager.cs:line 117

21:05:50 ERROR Error writting demuxdump file: System.ObjectDisposedException: Cannot access

Many more errors are repeated like this. I am assigning RecordToFile from another Thread. Seems like it's not thread safe yet.

Decouple everything to XRIT Library to have goesdump just as an interface for manipulating it.

Reported here: opensatelliteproject/OpenSatelliteProject#2

Sometimes new products are sent. Add a counter for Unknown products so user knows to check that directory for content.

Example: GOES 16 is current sending product ID 7 LRIT files. These segment files are put into Unknown directory but there is no status indication otherwise like we have for ABI products etc.

In goesdump/master commit ef32c60, when I did a "Rebuild goesdump" in Monodevelop, got the following error (5 times) when set to "Release". There are no errors when rebuilding as "Release Headless"

/home/k4kdr/osp/goesdump/goesdump/Main.cs(82,82): Error CS0117: OpenSatelliteProject.PacketData.Enums.NOAAProductID' does not contain a definition for SCANNER_DATA_1' (CS0117) (GOES Dumper)

Link: opensatelliteproject/OpenSatelliteProject#5

The timestamp field on the false color filename is not same as the timestamp location in the filename for other images. Please match them so scripts can have same name matching.

ABI Channel 08 LRITs are still ingested but images are no longer being generated as of 2018-02-09. The LRITs are still readable by xritimg.

find /root/osp-build/bin/goesdump/output/Images/FM1/ -type f -name '*.png' | grep -v -e 'Channel 7' -e 'Channel 9' -e 'Channel 14' -e 'Channel 15' -e 'Mesoscale' -e 'IR' -e 'FSCLR' -e 'VIS'

returns no images.

Per Trango email with NOAA contact, their Dartcom receivers output LRITs that have segmentation sequence numbers between 1 and 16 and do not contain the Image ID in the filename:

OR_ABI-L2-CMIPM1-M3C02_G16_s20173500704305_e20173500704362_c20173500704430.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_001.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_002.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_003.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_004.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_005.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_006.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_007.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_008.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_009.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_010.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_011.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_012.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_013.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_014.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_015.lrit
OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274_016.lrit

Goesdump should follow this format for compatibility

look like Himawari images are broken with OSP organizaer
it's not generating the PNG because it thinks it already has done it. and then it erases everything.

i see this file was HIMAWARI8chnFull DiskrgnFull Diskseg000res00dat generated but with no suffix... add .png and its the final image. however since it has not timestamp no more himawari images will be generated since organizer things it already did that image.

bin file for reproducing

https://www.dropbox.com/s/tjnmvrx80isbulw/demuxdump-1495166529.bin?dl=0

As described in https://wx-star.com/post/169608535305/designing-false-colors-for-goes-r-hrit, hdoverobinson has created an improved false color look-up table (LUT) for GOES-R.

Modify XRIT to either replace the existing LUT or to provide an option allowing the user to select the LUT to be used in generating false color images.

Currently OSP can draw shapefiles in all images, but it takes lots of time for full disks ( > 10 seconds ), so we need to figure out a way to cache the overlays.

when you get a chance test the Reset() call into your DemuxManager()
it throws an exception
Collection was modified; enumeration operation may not execute.
foreach (var k in demuxers.Keys) {
demuxers[k] = new Demuxer(this);
in the reset() method

Current workaround in PacketManager.cs

try {
File.Delete(filename);
} catch (Exception) {
// Do nothing, file doesn't exists
}
return null;

The following was substituted in:
if (UseNOAAFileFormat) {
origName = origName != null ? Path.GetFileName (origName) : "";
string baseName = Path.GetFileNameWithoutExtension (origName);
return $"{baseName}.png";
}

This produces images with the same filename as the first LRIT in their respective segmentation sequences. False Color images are no longer produced - have to look into how to identify those images in the UseNOAAFileFormat if statement so they can retain their OSP filename

Something to check:

https://www.dropbox.com/s/mcusidqx83tu2cy/demuxdump-1495113348.bin?dl=0

Setting ImageManager.EraseFiles = true; does not seem to impact *.lrit files when False Color generation is set to false. Would be nice to assume a user may have false color complete off and just want IR, WV and VIS generated PNG/JPG.

https://www.dropbox.com/s/9v2h21cx0ooj7z0/demuxdump-1496790733.zip?dl=0

No errors on demod.

Allow segmented full disk sequences to be generated after a timeout even if not all segments are present-

Occasionally HRIT will deliver 15/16 segments of a GOES-16 full disk. Goesdump currently does not attempt to generate the image in this scenario. This incomplete image could still be generated with black or white bars filling in for the missing LRITs after a timeout. The timeout would be longer than the normal amount of time we expect to receive all 16 segments of a full disk.

Link: opensatelliteproject/OpenSatelliteProject#9

Currently, different types of images output from GOES-16 HRIT have very different filename schemes:

G16-Mesoscale-GOES 16 ABI_None
G16-Mesoscale-VIS
G16-Full Disk-39048_Channel 7
GOES 15-Full disk-IR

This is broken and does not provide more value than NOAA's LRIT naming scheme. I propose that goesdump use the filename of the LRIT the image was derived from by default. Segmented images would be named after the first LRIT in the sequence e.g. OR_ABI-L2-CMIPF-M3C15_G16_s20173500745418_e20173500756191_c20173500756274.png (with segment number removed)

For false color images, the image could be named after the first LRIT for the visible channel, but the channel number could be changed to C00 or CFC e.g. OR_ABI-L2-CMIPF-M3CFC_G16_

Add a global identification of the current processed file. That can be using the Product ID.

So just to keep track, here are the changes @hwboy4 suggested on OSP Rocket Chat:

• Ability to specify path for received files (?by type)
• Syslog on Linux/Unix/FreeBSD and to the console (as-is now) on Windows.
• Enable / Disable streams (LRIT, EMWIN, DCS, etc.)
• Make headless an option in the config file
• Specify output filename format (printf format) in config file to allow interfacing with other programs.
• Use of custom LUT files in false color generation (again in config file)