Recently, I altered the codebase to include multi-block headers. In the process of doing so, I needed to keep track of the number of metadata blocks in the first block of the metadata blocks, i.e. the first block of the file. It should include the version and metadata_blocks fields/properties in the first two lines of the first block of the file. This means that the files block should never occur before that. I recently created a kdb file with the current master branch and could not observe the desired behavior.

Thanks to @peterjc, he discovered a bug in my writer class that was traced back to some private methods (_write_block) for binary encoded data. Switching to mode="w" fixed the issue. Will close with the new commit.

We should add MatthewRalston/LocalConfigurations to our developer dependencies. No, seriously.

We spent all day hardly making any modifications to the peerj.latex file, we did rename it and it did eventually get fixed enough to the point where it can stably be built. But it still has the pandoc 2.11.2 (our version) bug with the CLSReferences environment requirement. If we downgrade we might be able to circumvent this temporarily. But we did also get the emacs theme changed today for a different contrast, and we also set our .Rmd files to 'autodetect' and set mode to poly-markdown+r-mode ( I think the answer is (add-to-list 'auto-mode-alist '("\\.Rmd" . poly-markdown+r-mode))) and we have a new shortcut in case we're pissed with RStudio : C + c r. This will evaluate the spa/rmd-render function defined by John Geddes.

We have to take a moment to acknowledge the role that Vitalie Spinu has played in my past success at UD and also what I'll currently be able to do thanks to emacs polymode. And the RStudio developers for making this possible as well. Thank you for the amazing software. I'd say so far it's been about 100% RStudio until today, I encountered some rendering problems, and wanted to switch back to RStudio. But I remember how good their software was and wanted to thank both groups for their work on the interactive R experience...

And then I have to thank Yihui in the first place for KnitR. Really, really cool concept and now the RStudio team has so much to do. I mean how can we not go down the rabbit hole now and thank the R community and then the GNU community. And then I guess I'd also thank the Numpy, Scipy, and SKLearn teams for making the other parts of this project so easy to implement in a short amount of time. And I'd like to also thank a number of previous mentors and a current one who I don't quite get along with.

But anyways yeah I'm gonna go back and work some more and thanks again for everything, goodnight.

Today we learned a lot about the biblatex style I want to implement for a clean citation style, and I'm still thinking about a different style, like the plain citation module.

rstudio/rmarkdown#1649
rstudio/rmarkdown#1649
mpark/wg21#54
https://stackoverflow.com/questions/59193797/pandocs-environment-cslreferences-undefined-when-knitting-rmarkdown-to-pdf-in-r
https://gist.github.com/MatthewRalston/68f1e4f262712b0f2a60b3b617a3c8cb#file-peerj-tex-L290

The two biggest issues at the moment are:

• the title and author information will not print at the top of the first page
• the bibliography is printed stupidly because of the wrong pandoc version.

Explain how the individual genomes need separate fits of the model, but the model selected remains the same regardless of genome.

• Cullen Frey analysis to determine discrete distribution shape and similarity to Poissonian behavior.
• Distribution similarities and different fits. Compare the skewness and kurtosis between each Cullen Frey plot.
• Reminder if the NB model is selected and DESeq2 normalization applied, the user accepts normalization by other factors except size.
• Now we could log-transform and then normalize and it would likely be fine, but we can continue to use the transformed space for clustering

Currently, the __init__.py file duplicates the array across parallelization pools, even when -p = 1.

Moving this to the bottom should fix some memory issues with larger choices of k.

The parallelization for fastq data should be implemented correctly: meaning the pool should be utilized properly, even when accumulating k-mer from .fastq(.gz) data files.

Average memory per processor was low before << 30Gb. Will scale supralinearly in memory with k. But I don't plan on analyzing many human genomes at k>20 right now. The performance tradeoff is too important right now to consider high-value k in a database framework. The goal of this project is to perform data science operations on microbial ecological projects, not to inspect the level of complexity of a sparse Nxm (m samples at 4**k = N k-mer profile depth) matrix for human genome comparisons. If you're interested in this, contact me or maybe submit a pull request. Heck, make your own fork! Just recall the original License and make an effort to contact me for academic collaborations!

The goal of this issue is to highlight a problem in the database strategy: It seems to be CPU bound, not IO bound with this level of adoption of high-speed NVMe protocols. Postgres 13.4 is not maximizing the throughput of my NVMe RAID0 array (max clocked at roughly 21Gbps).

For this reason, my calculations have been severely CPU bound and potentialy other bottlenecks in the <=0.0.10 versions of this software. For this reason, I am returning once more to in-memory comparisons, in this already ridiculously complex variable space.

It is my opinion that not even UMAP can address the complexity of point density in arbitrarily complex k-mer profile spaces. The distances are ridiculously complex in the high-dimensional space and the Pearson and Spearman correlations seem to be preferable in some cases. At this stage, I am getting stuck during dimensionality reduction. I would like to visualize clusters of organisms in 2D/3D space, and Python seems to be my best option.

My next strategy will be dimensionality reduction with UMAP on a spearman distance matrix. The Spearman + PCA approach seemed appropriate around this time, and produced good clustering results. However, there aren't enough sample points in the distance matrix yet. I'll need to construct some more complex artificial metagenomes if I want to succeed.

The generation of the zero-length blocks is likely restricted to kdb/fileutil.py only.

The k-mer count distribution should be produced as a summarization function.

Using a first order markov model as part of the null model, and a uniform model as the remainder. The relationship is they are multiplied together and thus added when log10 transformed. So it's going down in the log odds ratio right now but not the probability of sequence.

Backbone - Droeloe

Using 'CITATION': ['CITATION'] or whatever it was is the improper way to include extra data into the package.

This fully specifies the memory profile of the in memory version of the k-mer counter. Choices : ["uint32", "uint64", "float32", "float64"]

It's worth stating that after some changes in the Pearson correlation coefficient code, my Correlation coefficients showed up. The pearson noise depends on what you're describing, a depth and or error rate. The model can't be specified. Ratios. I can't justify the 0.999 level accuracy that I'm seeing, it seem less than optimal but the real order should be specified as a Spearman correlation. I'm just doing that now. Well, I reran the pearson and suddenly my Pearsons show up and I'm like what. The last column is

0.9986406358838387      0.99870468338501        0.9987507311026925      0.998772078581407       0.9987762730886994      0.9987902044527077      0.9987863177029056   0.998805286411466        0.9988063501684376      0.9984427576272187      1.0

Alright here go my spearmans:

... Still blank. spearmanr RTFM targeted.

I'm trying to identify a TypeError default issue or one or more

Traceback (most recent call last):
  File "/home/matt/.pyenv/versions/kdb/bin/kmerdb", line 33, in <module>
    sys.exit(load_entry_point('kmerdb==0.5.2', 'console_scripts', 'kmerdb')())
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.2-py3.10.egg/kmerdb/__init__.py", line 1613, in cli
    args.func(args)
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.2-py3.10.egg/kmerdb/__init__.py", line 649, in get_matrix
    raise TypeError("One or more files did not have the default dtype or the default dtype did not match one or more files...")
TypeError: One or more files did not have the default dtype or the default dtype did not match one or more files...

I want to refactor the code to use 'metadata' as the keyword when referring to the file through the UI. Instead of 'header', which is a little non-specific and ambiguous. Historically, I think it made sense to use the term header very often, but i would prefer to use the phrase metadata to refer to the context. And the first step towards this refactor is replacing the k-mer line-specific metadata or 'neighbors' array with kmer_metadata.

I want to see both the Pearson's and the Spearmans for each dataset but we need to prioritize getting the plan written. Communicated.

The kmerdb probability function needs to count the number of steps it has made(we know ahead of time it is just L-k+1, where L is the sequence length), and the number of unique ids it has seen (set).

If the input sequences are smaller than k, i.e. there can be no k-mers derived from them, which module handles this? Oh, it wasn't. It shoots through Kmer.shred without checking.

We had retained the databases previously and their location was printed to a debug-level logging statement, not the most obvious places to keep the retained sqlite databases under the option --keep-sqlite under the kmerdb profile optional flags. In this issue, I'll move the temporary databases to a more obviously retained location, and I'll print the retainment statement out to sys.stderr instead of to a debug level logging.

A modest name for a more substantial bug, prior to this, I had been adopting to my own naming conventions of plain nucleic acid fasta files as .fasta or .fa and not the more explicit .fna, but I had been assuming that everything that wasn't .fq or .fastq was going to be not only fasta formatted input (which would get caught by SeqIO), but also that the suffix would be .fa or .fasta. The proper fix for this of course is to capture all 3 suffixes of fasta, both for fastq, and to raise a ValueError (less harsh than a IOError and more clear I think) about the format issue if it does not belong to the 5. Prior to this, this was not a feature of the kmerdb.seqparser module.

Steer the code towards resolving the following

• standardize the report on the tsv filesnames you expect them to follow from the documentation.
• The k-mer report by kdb.
• dendrogram
• pca and/or tsne plus kmeans
• pca elbow + kmeans elbow

Distances function needs to be reworked to better parse .kdb files properly and not just input dataframe CSVs.

The dimensions of biological data that we are able to investigate are formed by some choice of k, and we can represent biological scenarios in terms of .kdb files. By selecting some choice of k, we are able to concentrate the problem solving capacities of the computer, the logical imperative to mesh with the grammatical, and produce readable, semantic, organic codebase.

I don't spend enough time in the README, but it's a source of great anxiety.

Anyways yeah I'm going to consider this uint space for a moment to really get a grasp of the potential feature space.

The except statement was removed a while back. Needs to be cleaned up and un-indented.

Even if the CITATION text isn't completely finished because we don't have URLs, publish dates etc from a preprint or an article, it is important to have the essential process for showing the citation information and silencing it if needed.

The milestone suitable for academic use means that the methodologies have all been documented in a preprint, it does not mean that the methods are necessarily sound. No warranties are made about how you use this software. There is no guarantee that my formula for the Markov probabilities is correct, there is no guarantee that a first order Markov model is a suitable null model, there is no guarantee that a uniform random distribution is suitable for the frequency (p(x)) of the 0th k-mer in the null model. The counts are accurate and the methods applied are honest, but I can't talk about when an dhow this should be used until I better understand my audience and my peers who will review this work.

On line 90 of kmerdb/__init__.py, the module prints a data structure to stdout with a print statement. It also prints a pandas data frame as tsv to STDOUT. We need to remove the print statement.

Currently there is no command-line option for specifying the integer size (NumPy's dtype parameter) when accumulating counts for the k-mer profile. This should be in both kmerdb/parse.py and kmerdb/__init__.py numpy array instantiations (np.zeros()).

It should be equal to the number of sequences shredded.

After the most recent pull request regarding PostgreSQL and parallelism, the regression test coverage has hit a new low of < 5%. I need better regression test coverage.

While reading the header in kmerdb.fileutil.KDBReader.__init__() we need to rstrip() from each block the tag, which should be defined in kmerdb/config.py, and be also used by the writer when writing the entire header to the file.

This can be used by kmerdb view to delineate the end of the header when reading the file. The real reason to use this is if the kdb input is being read from stdin, we need to gzip decompress it and iterate using with gzip.open(sys.stdin) as stdin and stdin.readline to append to a header string, until the following is encountered. Throw everything behind this into yaml.safe_parse, followed by schema validation. Then either construct the writer to write the profile to the new file or stdout

    kdb : can someone get me some sentries
========================

Some commands under the hood require the input command to be named .kdb, like the index command. We need to smooth this over so that the filename is caught in more places than just each function in __init__.py. I'm going to close this issue with a first patch and reopen.

Chase this tuple (seq_id, starting_position, strand)

For k-mer db to be a full graph database format, it would need to store the following data:

1. seq_id: the full qualified read ID from the fastq or fasta input
2. starting_position, offset from the most 5' position
3. reverse: forward strand assumed.

What is the desired behavior for the metadata index? What's really the next step here?

The user wants to traverse back to the corresponding sequences from within the graph API. This merely stores it in the database, and should be an addition of oooff... potentially a lot of text, making the database size much bigger than anticipated. Hopefully the annotations end with just the sequence id.

This is an ongoing issue in the issues...

Please stand by.

The code commenting level is poor, functions are too monolithic, np.array functionality is not easily substitutable,

KDB | writer | reader seems clunky.

In prior versions of the distance method in bin/kdb, the distance function reported singleton distances as single printed values, instead of as TSV distance matrix. If we want to print single distances as single distances instead of as distance matrices, then we can restore this feature even with the new scipy.spatial.distance.pdist support.

metadata of the file so that it contains a sorting order that we're just starting to work on. To prioritize the elements of a list. We have to refine the algorithm to specify whether via Pearson or via Spearman correlation coefficients, we can focus our search on high affinity targets or low affinity targets.

We have to prioritize the method to at times de-link, de-couple, the types of the numbers flowing through the program.

At line 494 of __init__.py, the elif conditional checking arguments.input input types does not anticipate the cases when STDIN or /dev/stdin are supplied, but still require valid sys.stdin parsing. The fix was one line.

I've removed the numba jit compiler from the codebase, this is a genuine bug.

This IndexError is coming up in fileutil.py

Traceback (most recent call last):
  File "/home/matt/.pyenv/versions/kdb/bin/kmerdb", line 33, in <module>
    sys.exit(load_entry_point('kmerdb==0.5.0', 'console_scripts', 'kmerdb')())
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.0-py3.10.egg/kmerdb/__init__.py", line 1625, in cli
    args.func(args)
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.0-py3.10.egg/kmerdb/__init__.py", line 1200, in view
    with fileutil.open(arguments.kdb_in, mode='r') as kdb_in:
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.0-py3.10.egg/kmerdb/fileutil.py", line 189, in open
    return KDBReader(filename=filepath, mode=mode)
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.0-py3.10.egg/kmerdb/fileutil.py", line 326, in __init__
    self.slurp(dtype=dtype)
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.0-py3.10.egg/kmerdb/fileutil.py", line 477, in slurp
    self._slurp(dtype=dtype)
  File "/home/matt/.pyenv/versions/kdb/lib/python3.10/site-packages/kmerdb-0.5.0-py3.10.egg/kmerdb/fileutil.py", line 452, in _slurp
    self.kmer_ids[i] = kmer_id
IndexError: index 65536 is out of bounds for axis 0 with size 65536`1

This issue can be solved by enabling neighbor information by default, but is a low priority bug.

The other goal that can be started is the experimentation of a get_profile function or similar in fileutils that would slurp the entire profile into memory. I'm sick of not being able to have direct access to the data in memory if I'm sure it will fit, and having a function like this would be helpful for experimentation with the similarity metrics.

After the most recent pull request, the feature for inserting data into the database has changed. I need to test this on a small value of k to see how it goes.

This issue will not be directly addressed, but will be left with some comments in the codebase.

kmerdb shuf

This command shuffles the lines of a kdb file, much like the command line tool shuf. It requires the file to be indexed, since it does the shuffling on disk through a single stream, behind the index. This makes it very memory efficient shuffling, which will be necessary as the size of the nodes/kmer_metadata gets larger.

View needs to be able to read and write to STDIN/STDOUT and to read and write bgzf (un)compressed input/output, except we won't actually to bgzf uncompressed output to a file. Just redirect from the stupid STDOUT.

Minor syntactical sugar on the command line.

Old usage (still supported)

kmerdb matrix Unnormalized *.kdb | kmerdb distance spearman STDIN #or '/dev/stdin'
kmerdb matrix Unnormalized *.kdb | kmerdb matrix Normalized STDIN

New usage should be

kmerdb matrix Unormalized *.kdb | kmerdb matrix Normalized

The block size assertion does not work with blocks that are less than the block size. This would fail at the end of most .fastq files and for .fastq files that have less than b reads total.

Logging, flow, and Sphinx documentation needs to be reworked some more, perhaps removal of the multi-core processing now that it's in-memory.

And Type Errors. Investigating.

I have no idea what I'm doing. I'm trying to count the index of kmer_ids as a uint64 array type. I don't know why I'm bound to uint64 array maxima, what are they?