Hi,

When I extracted TIC by .getChromatograms() , all TIC including MS2's were exported. Do you have any idea to filter them?

The mzTab-m 2.0 specification for metabolomics and the corresponding library will soon be released:
https://github.com/HUPO-PSI/mzTab/blob/master/specification_document-developments/2_0-Metabolomics-Draft/mzTab_format_specification_2_0-M_draft.adoc

and
https://github.com/nilshoffmann/jmzTab-m

mzTab 1.0 is already included in msdk.
Integrating basic mzTab 2.0 for metabolomics support should therefor be rather easy.

Hey,

what is the recommended way to import the MSDK into eclipse? I did not find any .project file in the directory so I figured I'd just ask.
Do you just import it as a maven project or is there a better way?

Best regards
Steffen

When importing mzML files, running MzMLUnmarshaller() takes a significant amount of time compared to the rest of the import code (e.g. 6 seconds vs. 5 seconds). During this time, getFinishedPercentage() is not updated. Any suggestions on how to improve this?

Here is a link to MzMLFileImportMethod.java and MzMLUnmarshaller.java.

1. In the current RawDumpParser.java l.300, the precursorMZ mzRange is set to a singleton instead of the isolation range of the mass spec.
2. Q3 values are not handled properly which is needed for SRM data. Currently this information is available through the scan.getScanDefinition(). This data should however be available through the scan.getIsolations().

See also the comments under the following issue.

We should be able to retain all CV params and their values found in mzML files when files are imported to MSDK and then exported again to mzML.
Example:

<cvParam cvRef="MS" accession="MS:1000512" name="filter string" value="FTMS - p ESI Full ms [100.00-1000.00]"/>

This is not a high-priority issue, but good to keep in mind.

Here's the relevant stack trace from mvn compile:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.268 s
[INFO] Finished at: 2016-09-06T13:37:19-07:00
[INFO] Final Memory: 12M/113M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project msdk-io-mzml: Could not resolve dependencies for project io.github.msdk:msdk-io-mzml:jar:0.0.7-SNAPSHOT: Failed to collect dependencies at uk.ac.ebi.jmzml:jmzml:jar:1.7.5: Failed to read artifact descriptor for uk.ac.ebi.jmzml:jmzml:jar:1.7.5: Could not transfer artifact uk.ac.ebi.jmzml:jmzml:pom:1.7.5 from/to nexus-ebi-release-repo (http://www.ebi.ac.uk/Tools/maven/repos/content/groups/ebi-repo/): Failed to transfer file: http://www.ebi.ac.uk/Tools/maven/repos/content/groups/ebi-repo/uk/ac/ebi/jmzml/jmzml/1.7.5/jmzml-1.7.5.pom. Return code is: 500 , ReasonPhrase:Internal Server Error. -> [Help 1]

This is after a fresh update to master (i.e. commit 790eb96).

After a bit of searching, I see the jmzml documentation suggests the following maven dependency info, however, visiting their repo doesn't have the 1.7.5 version:

        <repository>
            <id>ebi-repo</id>
            <name>The EBI Maven 2 repository</name>
            <url>http://www.ebi.ac.uk/~maven/m2repo</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>

        <dependency>
            <groupId>uk.ac.ebi.jmzml</groupId>
            <artifactId>jmzml</artifactId>
            <version>x.y.z</version>
        </dependency>

Is there a more recent repo that contains 1.7.5? Or is the build supposed to happen with 1.7.4?

It appears that TxtExportAglorithm.exportSpectra() will concatenate the output from multiple spectra into a single file without any line break between spectra. The only way to tell there are multiple spectra is to look for discontinuities in the m/z values. Is that intentional? If so, note that will probably not work in the perhaps-unlikely general case (e.g. spectrum 1 is 200-500, spectrum 2 is 505-800, etc.).

In the current mzTab format (v. 1.0.0) there are mandatory columns for small molecules which are currently not supported in the IonAnnotation - see page 13 in the mzTab Specifications. These are:

• inchi_key
• taxid
• species
• database
• database version
• spectra_ref
• search_engine
• best_search_engine_score
• modifications
• reliability

I think this information will make it more transparent where identifications are coming from when using multiple identification methods. Furthermore, it will also provide some of the structure needed to implement MS/MS identification methods.

What do you think about adding support for this in MSDK?

File a request to the psi-ms dev mailing list ([email protected]), requesting a new term for MSDK:
[Term]
id: MS:100XXXX
name: MSDK
def: "Mass Spectrometry Development Kit (MSDK) is a Java library of algorithms for processing mass spectrometry data." [PSI:PI, http://msdk.github.io/]
is_a: MS:1001456 ! analysis software
is_a: MS:1001457 ! data processing software

Once this has been added, we should adapt the MZML output created by MSDK to contain the CV term.

java.lang.AssertionError: expected:<809.6484375> but was:<809.1484375>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:553)
at org.junit.Assert.assertEquals(Assert.java:683)
at io.github.msdk.featdet.msmsdetection.MsMsDetectionMethodTest.testOrbitrap(MsMsDetectionMethodTest.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

The calculation of m/z tolerance in the gap filler is fishy, needs fixing.
https://github.com/msdk/msdk/blob/master/msdk-features/msdk-features-gapfilling/src/main/java/io/github/msdk/features/gapfilling/GapFillingMethod.java#L172

I noticed that MgfExportAlgorithm isn't a MSDKMethod like some of the other I/O classes. Is this intentional? Or something that should be fixed?

This issue is for tracking the development and bugs of ADAP3D.

1. Any of these files produce exceptions or hang the algorithm: test_files.zip

2. The tiny.mzXML file that is included in the ADAP3D test suite contains a very obvious peak at 166.071 m/z, 2.9 min. that is not found by the algorithm:

3. The cancel() method does not work if the algorithm is stuck at the FWHM estimation step

Hi,
when I try to build MSDK from source, it hangs while building the msdk-spectra-splash module:
Downloading: http://gose.fiehnlab.ucdavis.edu:55000/content/groups/public/edu/ucdavis/fiehnlab/splash/core/1.8/core-1.8.pom

Trying to contact the server manually also fails. Any chance of getting the splash artifacts into Maven central or bintray?

For convenience of downstream consumers and contributors, it would be great to have git tags associated to the MSDK release versions that are published to Maven Central.

The filterMsSpectrum method has a bug due to the usage of a TreeMap<Float, Double> (Intensity, Mz). Values are compared for equality by the comparator, therefore signals with equal intensity are omitted.

See this screenshot:

The map should have 2399 values, but it only contains 437 due to same intensities.
I noticed this issue when testing Sirius single row identification in MZmine when processing a IMS file (TDF).

I am trying to parse a raw file using ThermoRawImportMethod.

The exception can be reproduced this way:

ThermoRawImportMethod th = new ThermoRawImportMethod(file);
th.execute(); // Exception occurs here

Apparently, it stems from the exe file being run behind the scenes. The error message says

ERROR: System.Runtime.InteropServices.COMException (0x800736B1): Retrieving the COM class factory for component with CLSID {1D23188D-53FE-4C25-B032-DC70ACDBDC02} failed due to the following error: 800736b1.

I already made sure that I have the newest .NET framework running.

When I try to directly execute the ThermoRawDump.exe with the raw file path as argument I get the same error.

Any help is much appreciated. Thanks in advance.

morning how can we easily get these kind of information out of a mzML file in MSDK?

<fileDescription>
  <fileContent>
    <cvParam cvRef="MS" accession="MS:1000579" name="MS1 spectrum" value=""/>
    <cvParam cvRef="MS" accession="MS:1000580" name="MSn spectrum" value=""/>
  </fileContent>
  <sourceFileList count="1">
    <sourceFile id="RAW1" name="BioRec01_MX612252_posLIPIDS_postLongevity_Female013.raw" location="file:///D:\studies\research\2022\612252\lipids\pos\pools">
      <cvParam cvRef="MS" accession="MS:1000768" name="Thermo nativeID format" value=""/>
      <cvParam cvRef="MS" accession="MS:1000563" name="Thermo RAW format" value=""/>
      <cvParam cvRef="MS" accession="MS:1000569" name="SHA-1" value="1643a51477b4bd575a31ff69fda032d2d85a1c63"/>
    </sourceFile>
  </sourceFileList>
</fileDescription>
<referenceableParamGroupList count="1">
  <referenceableParamGroup id="CommonInstrumentParams">
    <cvParam cvRef="MS" accession="MS:1002877" name="Q Exactive HF-X" value=""/>
    <cvParam cvRef="MS" accession="MS:1000529" name="instrument serial number" value="Exactive Series slot #6121"/>
  </referenceableParamGroup>
</referenceableParamGroupList>
<softwareList count="2">
  <software id="Xcalibur" version="2.12-212021/2.12.0.3134">
    <cvParam cvRef="MS" accession="MS:1000532" name="Xcalibur" value=""/>
  </software>
  <software id="pwiz" version="3.0.21209">
    <cvParam cvRef="MS" accession="MS:1000615" name="ProteoWizard software" value=""/>
  </software>
</softwareList>
<instrumentConfigurationList count="1">
  <instrumentConfiguration id="IC1">
    <referenceableParamGroupRef ref="CommonInstrumentParams"/>
    <componentList count="4">
      <source order="1">
        <cvParam cvRef="MS" accession="MS:1000073" name="electrospray ionization" value=""/>
        <cvParam cvRef="MS" accession="MS:1000057" name="electrospray inlet" value=""/>
      </source>
      <analyzer order="2">
        <cvParam cvRef="MS" accession="MS:1000081" name="quadrupole" value=""/>
      </analyzer>
      <analyzer order="3">
        <cvParam cvRef="MS" accession="MS:1000484" name="orbitrap" value=""/>
      </analyzer>
      <detector order="4">
        <cvParam cvRef="MS" accession="MS:1000624" name="inductive detector" value=""/>
      </detector>
    </componentList>
    <softwareRef ref="Xcalibur"/>
  </instrumentConfiguration>
</instrumentConfigurationList>
<dataProcessingList count="1">
  <dataProcessing id="pwiz_Reader_Thermo_conversion">
    <processingMethod order="0" softwareRef="pwiz">
      <cvParam cvRef="MS" accession="MS:1000544" name="Conversion to mzML" value=""/>
    </processingMethod>
    <processingMethod order="1" softwareRef="pwiz">
      <cvParam cvRef="MS" accession="MS:1000035" name="peak picking" value=""/>
      <userParam name="Thermo/Xcalibur peak picking"/>
    </processingMethod>
    <processingMethod order="2" softwareRef="pwiz">
      <userParam name="removed extra zero samples"/>
    </processingMethod>
  </dataProcessing>
</dataProcessingList>

I would assume this is possible, but not really have found the correct way

In a different framework, I posted a question regarding whether one can read Thermo or Sciex files directly (chhh/batmass#9 (comment)). Is it possible with msdk? If so, is there an example on how to do this, and ideally, how to do this efficiently (ie streaming) so that one could pull several GB worth of data (in order of the scan numbers) without having to requery the raw file for every single scan number? (In some old implementations, such as jmzReader, the reading of the mzXML files were slow since it had to first build the entire mzXML file's index, and then one would have to fetch data one index at a time; not only would we rather read the original Thermo .raw or Scies .wiff file directly, but we'd like to be able to just "suck all the info out" in a streaming fashion as quickly as possible -- so no need to query by scan number etc..)

Hi,
I am currently starting to work on a LECO ChromaTOF csv peak table import method and would like
to soon work on netCDF-based support for 2D GCxGC-MS data from ChromaTOF exports.
I do have some questions whether I need to watch out for some assumptions in the higher-level
layers of the software:

1. the csv peak table export format is freely configurable, but I have already developed a decent
   importer for Maltcms that copes with the various configuration options. I would like to port that
   part of the code and I am positive that the mapping to msdk datastructures will be manageable.
   However, the format consists of sparse MS1 EI spectra, so I am not sure, whether I should implement
   support for RawDataFile or for FeatureTable? Would be great, if you could point me to the right
   direction there.
2. The ChromaTOF netCDF export format for GCxGC-MS (at least with ChromaTOF from about
   two years ago), only supported 1D RT data in the netCDF file. To reconstruct the 2D
   retention time information, we implemented two strategies in Maltcms: a) user-provided parameters
   for second column modulation time (usually 1 - 6 s) and reciprocal scan frequency (usually between 50 and 200 Hz),
   aka scan rate or period, b) automatic estimation of second column modulation time by autocorrelation of portions of
   the TIC signal and estimation of the scan rate from the differences in scan acquisition time (in netCDF).
   Would you consider it practical to require the user to provide the parameters manually? Usually they know
   the two parameters but the second approach could work almost automatically except for cases, where the estimation
   values vary too much (the approach proved to be quite stable, but requires more computational resources).
   Should I implement this as a separate module? Unfortunately, the exported netCDF files are indistinguishable
   from ther 1D cousins based on the contained attributes alone.

Best regards,
Nils

Hi,
the current setup for testing has all relevant files local within the module providing support for that file format. Since we use Maven, these files are correctly placed in the src/test/resources folder. However, this means that other modules can not use these files for their own testing purposes with the standard configuration. It seems possible to do this by customizing the JAR plugin:

http://stackoverflow.com/questions/174560/sharing-test-code-in-maven

Additionally, the modules currently store all test files in the root of the class path. This may be a matter of taste, but would it seem sensible to use the module's artifactId as a subpath to the files? For example in the msdk-io-mztab module, the file Sample-2.3.mzTab would move to msdk-io-mztab/Sample-2.3.mzTab.

We could also think about adopting a mime-type scheme:

application/x-netcdf/testdata.cdf
text/x-mztab/Sample-2.3.mzTab

command line:

ffmpeg -init_hw_device qsv=qsv:hw -hwaccel qsv -filter_hw_device qsv -v verbose -f rawvideo -pix_fmt p010le -s:v 1920x1080 -r:v 30 -i ./yuv/1080P_1920x1080.P010 -vf 'format=p010le,hwupload=extra_hw_frames=120' -an -c:v hevc_qsv -profile:v main10 -g 30 -slices 2 -bf 2 -b:v 1000k -maxrate 1000k -low_power 0 -vframes 150 -y test.h265

msdk patch:

commit 710d7ad7a2f9b4d025b72d5276fcc16be1a01f2e
Author: Elizarov Aleksandr [email protected]
Date: Fri Dec 20 16:23:54 2019 +0300

[HEVCe]: Move setting some caps to HardcodeCaps()

Usage of HardcodeCaps is a temporal solution.
It will be removed when all driver caps issues are resolved

If revert this patch, the command line can pass, otherwise fail.

Please also export the inchiKey string to the mzTab File if its set in IonAnnotation-Object. The code is missing in MzTabFileExportMethod.writeSampleData, but should be straight forward to add by using SmallMolecule.setInchiKey(String inchiKeyLabel).

Hey MZmine community,

as far as I understand the structure of the MSDK, it is a maven project with multiple modules. Is there any smart way to import this project into eclipse? After importing it as a Maven project, the development does feel unnatural and the structure is very confusing (see below). (It is my first time working on a multi module maven project)

Is Java 9 and project Jigsaw changing any future plans for the final module structure of the MSDK?

Best regards,
Robin

Hi, is there a plan to implement the "ChromAlign" algorithm as described in the publication below. Recently, several papers on IonStar claimed excellent alignment performance using the Sieve program due to this algorithm. It'd be quite useful to add it to MSDK.

Thanks for the wonderful works in MSDK! I hope to contribute at some point.

Wen

https://pubs.acs.org/doi/abs/10.1021/ac060923y
ChromAlign:  A Two-Step Algorithmic Procedure for Time Alignment of Three-Dimensional LC−MS Chromatographic Surfaces
Rovshan G. Sadygov*, Fernando Martin Maroto, and Andreas F. R. Hühmer

The mzML export doesn't define the terms used like the softwareRef and defaultInstrumentConfigurationRef, causing Proteowizard (and mzR) to fail when reading the files.

This could be fixed by simply adding some dummies to the exported mzml.

<softwareList count="1">
    <software id="MSDK" version="1.1.0">
    </software>
</softwareList>
<dataProcessingList count="1">
    <dataProcessing id="MSDK_mzml_export">
        <processingMethod softwareRef="MSDK" order="0">
        </processingMethod>
    </dataProcessing>
</dataProcessingList>
<instrumentConfiguration id="unknown"/>
<run id="MZmine mzML export" defaultInstrumentConfigurationRef="unknown">

This was tested with mzR 2.28

Hi,
I've notice some strange behaviour with this object, maybe I'm doing something wrong? Here it is:

// I create the data point list and fill it
MsSpectrumDataPointList dataPoints = MSDKObjectBuilder.getMsSpectrumDataPointList();
scan.getDataPoints(dataPoints);

// I transform it into a List
List ionList = Lists.newArrayList(dataPoints.iterator());

System.out.println(dataPoints.getSize()); // It gives me 426
System.out.println(ionList.size()); //It gives me 425

Shouldn't they be equal? What is happening?

Hi all

I have been thinking about how to best implement the MS/MS feature list builder and suggest that we do it in a different way compared to MZmine 2.

This is the workflow I suggest:

1. Find all MS/MS precursor ions in the raw data file and generate a list of m/z, RT and total MS/MS spectra intensity.
2. Sort the list by total MS/MS spectra intensity.
3. Go through all the m/z values in the list and remove m/z values with lower intensity and with a m/z within x ppm.
4. Use the m/z and RT list as an input for the targeted detection method to generate the feature table.

What do you think?

Using: implementation("io.github.msdk:msdk-all:0.0.27")

Could not find uk.ac.ebi.pride:jmztab-modular-model:3.0.7.
edu.ucdavis.fiehnlab.splash:core:1.8…

Where are all of these?

MsSpectrumDataPointList newDataPoints = MSDKObjectBuilder.getMsSpectrumDataPointList(); newScan.getDataPoints(newDataPoints);

float sumInt = 0;
for(MsIon ion : newDataPoints){
sumInt+=ion.getIntensity();
}
Assert.assertEquals(sumInt, newScan.getTIC(),0.001);

It fails the test. The sum of intensities is different than TIC... why?

Hi,

Recent days, I tried to get TIC information form mzML file by msdk package, but I met an exception when I parsed it. My codes as follow:
`MzMLFileImportMethod mzMLFileImportMethod = new MzMLFileImportMethod(spectrumFilePath);

        try {
            mzMLFileImportMethod.execute();
        } catch (MSDKException e) {
            e.printStackTrace();
        }

        MzMLRawDataFile mzMLRawDataFile = (MzMLRawDataFile) mzMLFileImportMethod.getResult();
        for (Chromatogram chromatogram : mzMLRawDataFile.getChromatograms()){
               System.out.println(Arrays.toString(chromatogram.getIntensityValues()));
        }` 

And this is exception:
Exception in thread "main" io.github.msdk.MSDKRuntimeException: java.lang.ArrayIndexOutOfBoundsException: -1 at io.github.msdk.io.mzml.data.MzMLChromatogram.getIntensityValues(MzMLChromatogram.java:419) at io.github.msdk.datamodel.Chromatogram.getIntensityValues(Chromatogram.java:94) at PSMviewer.fileimport.MSOneImport.parseSpectrum(MSOneImport.java:106) at PSMviewer.fileimport.MSOneImport.<init>(MSOneImport.java:69) at PSMviewer.fileimport.MSOneImport.main(MSOneImport.java:56) Caused by: java.lang.ArrayIndexOutOfBoundsException: -1 at io.github.msdk.io.mzml.util.ByteBufferInputStream.byteBuffer(ByteBufferInputStream.java:137) at io.github.msdk.io.mzml.util.ByteBufferInputStream.position(ByteBufferInputStream.java:255) at io.github.msdk.io.mzml.util.ByteBufferInputStream.constrain(ByteBufferInputStream.java:274) at io.github.msdk.io.mzml.data.MzMLPeaksDecoder.decodeToFloat(MzMLPeaksDecoder.java:59) at io.github.msdk.io.mzml.data.MzMLChromatogram.getIntensityValues(MzMLChromatogram.java:417) ... 4 more