FDSNWS-Dataselect stuck on multi-gap/overlap file about main HOT 13 CLOSED

Thank you for providing the example data. I could reproduce the error and it is not fdsnws related. It is caused by the SDS archive recordstream implementation.

from main.

Actually it is the MiniSEED parser that causes an endless loop. The issue with your records is, that there is no blockette 1000 which is required to determine the record length. I have fixed the endless loop which causes 100% CPU but it will return no data. The records without blockette 1000 are being ignored currently. As there are some other issues with the parser we need to rework that.

from main.

Many thanks for your reply, if could be useful we have tested the our old seiscomp3 version and there are no problem.

from main.

Which version is your old SeisComP3?

from main.

@gempa-jabe: Thank you for having looked into this. We then also discovered that the blockette 1000 were missing.

Apparently the data is pre-V2.3 "data only" SEED (sometimes these also are dubbed miniSEED as I found out), but not valid miniSEED as specified in SEED V2.3. Which makes kind of sense considering the age of the data, i.e. 1995. (Okay the specs of V2.3 were published earlier, but it is not unlikely that this update was not adopted at production time yet).

There seems to be plenty of such data around and a lot of relevant software supports it, in particular libmseed & Co, Obspy, ... For the services delivering non-decoded data the lack of the information contained in Blockette 1000 should not even be that relevant as most decisions would probably be based on the fixed header only, right?

Anyway, we would need to understand if you plan to support such pre-V2.3 data as well, so we can choose some strategy on how to deal with the situation. Indeed, as a general strategy we would prefer not to touch the original data if anyhow possible.

from main.

Our old version: seiscomp3-jakarta-2018.327p23-debian-9-x86_64.tar.gz

from main.

Our old version: seiscomp3-jakarta-2018.327p23-debian-9-x86_64.tar.gz

That is strange, as the parser code has changed with 2018.327p19. Anyway, I know what code change causes the issue.

from main.

There seems to be plenty of such data around and a lot of relevant software supports it, in particular libmseed & Co, Obspy, ... For the services delivering non-decoded data the lack of the information contained in Blockette 1000 should not even be that relevant as most decisions would probably be based on the fixed header only, right?

Unfortunately not. Blockette 1000 contains the length of the record which is necessary for correct reading from non-seekable streams. Without blockette 1000 you also don't have information about the used encoding. There seeems to be a default anyway. Without blockette 1000 we need to search for the next valid MS header to estimate the record length. This is actually what libmseed does as well. But again, not simple and portable with streams where you can't seek.

Anyway, we would need to understand if you plan to support such pre-V2.3 data as well, so we can choose some strategy on how to deal with the situation. Indeed, as a general strategy we would prefer not to touch the original data if anyhow possible.

I understand your wish ;) We will look into a portable solution to this issue. As I said, there are other issues with the parser for some corner cases which need to be resolved as well.

from main.

@massimo1962 and @gempa-jabe

Our old version: seiscomp3-jakarta-2018.327p23-debian-9-x86_64.tar.gz

That is strange, as the parser code has changed with 2018.327p19. Anyway, I know what code change causes the issue.

I guess the explanation is that the p23 was downloaded to the server but actually never deployed, AFAIK. Note also that some of the changes in p23 were actually triggered by some other issue we reported on data with gaps.

So yes, there might be some conflicting targets around how to determine the end of a record.

from main.

I have hopefully fixed it. The change is in the common repository. You can give it a try by either installing a master build or patching the corresponding source file in your installation and just rebuilding it. But all tests succeeded and I can read your file.

@massimo1962, @petrrr: Am I allowed to use the first two records of your example file as test data set and include it in the SC test suite?

from main.

Hi @gempa-jabe:

I post the commit for reference: SeisComP/common@1853992

We well test the patch. And sure you can use the file (or some small portion of it) for the test suite.

BTW: We have also reviewed the archive, and actually I learned that that correcting the archived files is less impactful I thought, because there is space for the Blockette and no repackaging would be necessary. So we probably will clean this up as well.

Still "tolerating" such files might still be beneficial for SC.

from main.

Hi @gempa-jabe, Thanks again.

We have done various testing, including some tests for possible regression. Looks your patch resolves the problem with blockette-1000-less files. Also gappy data is handled correction in our tests. So this issue seems to be solved.

Are you planning for a patch release any soon?
Just to understand how to handle deployment in production.

from main.

Yes, a new 4.x release is on the way.

from main.