Code Monkey home page Code Monkey logo

2022-branchwater-benchmarking's People

Contributors

bluegenes avatar ctb avatar

Watchers

 avatar  avatar  avatar  avatar

2022-branchwater-benchmarking's Issues

example output for benchmarking

        Command being timed: "/home/ctbrown/scratch/magsearch/bin/searcher -k 31 --scaled=1000 -o output.csv siglist-a.txt data/wort-list-a.txt"
        User time (seconds): 46185.31
        System time (seconds): 1329.98
        Percent of CPU this job got: 2593%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 30:31.75
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 24259940
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 12
        Minor (reclaiming a frame) page faults: 428254673
        Voluntary context switches: 166157
        Involuntary context switches: 7161352
        Swaps: 0
        File system inputs: 196555504
        File system outputs: 2304
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

benchmarks with snakemake

from code added in 2b78b96, this:

srun -p high2 --time=5:00:00 --nodes=1 --cpus-per-task 32 --mem 50GB --pty /bin/bash
snakemake -j 32 --snakefile benchmarking.snakefile 
csvtk concat benchmarks/*.txt | csvtk cut -t -f "h:m:s",cpu_time,max_rss

yields:

h:m:s   cpu_time        max_rss
0:23:14 40626.79        16106.77
0:23:14 38643.81        17553.49
0:22:37 37936.37        18257.87
0:25:05 39209.39        16107.48
0:26:36 37791.57        14046.44

average and median file size

in ~ctbrown/scratch/2022-magsearch-benchmarking,

%  scripts/intersect-ls-and-catalog.py data/ls-wort.txt data/metagenomes-catalog.txt
loaded 767277 accessions from 'data/metagenomes-catalog.txt'
example acc: ERR1083538
Loaded 767277 file sizes from 'data/ls-wort.txt'
average file size: 9716.0
median file size: 570

output of full run (1000 x entire catalog)

rule a_vs_catalog:

        Command being timed: "snakemake -j 32 --snakefile benchmarking.snakefile"
        User time (seconds): 3609042.48
        System time (seconds): 98105.28
        Percent of CPU this job got: 2963%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 34:44:51
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 52518880
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 24
        Minor (reclaiming a frame) page faults: 30125462817
        Voluntary context switches: 9804070
        Involuntary context switches: 564183263
        Swaps: 0
        File system inputs: 14879893680
        File system outputs: 178544
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.