prosit prediction benchmark about prosit HOT 5 CLOSED

Thank you for the benchmarks.

Some of the points do not match our experience. For example, CPU should not have a huge influence as those mostly govern I/O. Using SSDs or holding as much as possible in RAM for example may speed up the process substantially. When we benchmark only the prediction we see a clear benefit from faster GPUs and higher batch sizes. The supplement from our paper has a benchmark on speed as well (Supplemental Figure 11).

In general, please keep the Github issues for issues or questions with Prosit (i.e. errors, requests for clarifications) and consider pull requests or wiki entries for general comments. Thank you for sharing your results with us!

from prosit.

Some of the points do not match our experience. For example, CPU should not have a huge influence as those mostly govern I/O. Using SSDs or holding as much as possible in RAM for example may speed up the process substantially. When we benchmark only the prediction we see a clear benefit from faster GPUs and higher batch sizes.

The prediction is clearly single threaded, once this runs on 100% on a thread there is no I/O bottleneck. Otherwise the CPU would run 50% or less, then its clearly I/O. The 2080TI is faster than the 1080TI, no substantial difference. Then the fact (or elephant in the room) that the GPU is only fully utilized during the first 5% of the run, that's really interesting. So maybe that's the limiting step, because export is single threaded, basically it can not export the data fast enough. If the prediction would be actually multi-threaded on the CPU level then you still would have to stitch the results together and then the export could be potentially I/O limited. Overall nothing to worry about, predictions are very fast.

from prosit.

The new version after July 3rd with MS/MS and RT prediction utilized CPU much better, 50-60% of all threads are active. GPU use is still not observed after initial spike. An input file with 800k tryptic digests and 20 Mbyte file size uses 20 Gbyte RAM during prediction and finishes in 1 hour. Output file size is 3.3 GByte.

So the functionality for multiple and effective GPU predictions is also currently missing. See:
https://stackoverflow.com/questions/42409884/keras-tensorflow-prediction-on-multiple-gpus?rq=1
and
https://stackoverflow.com/questions/44255362/tensorflow-simultaneous-prediction-on-gpu-and-cpu

from prosit.

The CPU load you are seeing, comes from transforming the csv file into suitable numpy arrays and back. The prediction utilizes the GPU.

Currently, prediction speed is not a bottleneck that is why multi-gpu support is not high on our priority list. Pull requests are very welcome though :)

from prosit.

@gessulat
"pull requests" I wish, but its not my domain. If I would I'd probably focus on:

1. maybe some native deployment of the prosit code so it could run under Windows. Basically without the Docker skeleton (which is great, but not working under WIN due to GPU pass-through)
2. maybe some code that would do the prosit predictions just on the CPU (n=44 or 56)
   without the requirement to install GPU dependencies, with tf.device('/cpu:0')?
3. maybe some general benchmarks so we can saturate the CPU with use of ramfs/tmpfs
   for large(er) memory machines 100 to 500 GByte
4. maybe some prediction benchmarks that show use of a single/multiple GPUs is actually faster,
   but as you mentioned the Keras/TF code is quite complicated

Overall I agree, for people working on a single genome or organism the calculations are quite fast,
running all uniprot with all major digestion methods under 2-3 CID voltages might cover all important experiments. Then one would need to calculate that once and store in a database. No need to recalculate. Basically a google for prosit CID MS/MS and RT predictions, much faster of course.

from prosit.