rieck / harry Goto Github PK

It would be great, if the Python module for Harry could be extended to support Python version 3.

When computing non-square matrices, Harry always calculates all matrix cells. Some computations might be redundant due to the symmetry of similarity measures.

If possible Harry should support receiving zlib compressed input on stdin as well as output compressed data on stdout. This feature needs to be merged with the compress flag in the output configuration.

The Python module requires a thorough testing:

• Does the mapping of command-line options to keyword arguments work?
• What happens if an external configuration file is loaded?
• Is the module robust to incorrect usage, e.g. tampering with the formats?

Happy testing.

There is a bug in the new split computation if certain ranges are selected. Example:

./harry -x :10 -y -10: harry.c /dev/null
harry: hmatrix.c:159: hmatrix_inferspec: Assertion `spec->b_left + spec->a + spec->b_right == width' failed.
Aborted

./harry -x 0:2 -y 3:4 -v harry.c /dev/null
harry: hmatrix.c:159: hmatrix_inferspec: Assertion `spec->b_left + spec->a + spec->b_right == width' failed.
Aborted

There exist analysis tasks where the similarity between pairs of strings needs to be computed. In this setting, computing a similarity matrix over all strings is clearly an overkill and it would be great if Harry could support this setting, e.g. using a special command-line option.

http://arxiv.org/abs/1401.6596

./configure --prefix=/opt/harry --with-libarchive
make
error message
make[3]: Entering directory /home/harry-0.4.0/src' /bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -DNDEBUG -std=c99 -fgnu89-inline -Wall -fPIC -fopenmp -o harry harry.o libharry.la -lpthread -larchive -lz -lm -lconfig libtool: link: gcc -g -O2 -DNDEBUG -std=c99 -fgnu89-inline -Wall -fPIC -fopenmp -o harry harry.o ./.libs/libharry.a -lpthread -larchive -lz -lm -lconfig -fopenmp ./.libs/libharry.a(input_arc.o): In functioninput_arc_open':
/home/harry-0.4.0/src/input/input_arc.c:50: undefined reference to archive_read_support_filter_all' collect2: ld returned 1 exit status make[3]: *** [harry] Error 1 make[3]: Leaving directory/home/harry-0.4.0/src'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory /home/harry-0.4.0/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory/home/harry-0.4.0'
make: *** [all] Error 2

I have check the tools version of libarchive.
It's libarchive-devel-2.8.3-4.el6_2.x86_64
My system version is CentOS release 6.5

Please help me to fix this issue
Thanks!

Looking at this paper here I'm curious if the benchmarks against Jellyfish in section 3.2 were using the C or Python implementation. It's not clear from the paper.

During the computation of a similarity matrix, the number of running threads steadily decreases. The bug occurs on Linux and Mac OS X. It's unclear if it has been introduced in the last commits or has been around for a long time.

Reproduce with

src/harry -v -n 4 examples/dna/dna_hg16.txt /dev/null

At around 26% the first of the four threads stops running and idles.

http://www.archives.gov/research/census/soundex.html

Internally, Harry only stores the upper triangle of full square matrices. If these matrices are split row-wise using the command-line option -s, the first block will require far more computations than lower blocks.

Hi,
This is the first time that i tried to build Harry (with latest version 0.4.1). However, i ran into some front-end compilation errors while firing off "make" (the ./configure appeared to go well) such as
.../src/harry.c:357: undefined reference to 'config_init'
.../src/harry.c:360: undefined reference to 'config_read_file'
....

All the errors reported are related to missing config_* references (looks like for the C API). i'm using libconfig_1.5 that i recently built. Any idea what went astray?

Thanks for reading.

776a9ec introduces support for comparing strings on the level of bits. However, similarity measures that rely on the efficient extraction of substrings are not yet supported.

The current master contains a new feature: The command-line tool can be called with either one or two input sources. This feature needs to be tested:

• Does the new feature work with different input formats?
• What happens if x-ranges and y-ranges are provided?
• Are there cases where the new feature breaks functionality?

Happy testing. It might be possible to construct a test case and add it to tests

The following call crashed Harry

harry -x 2:3 -y 2:3 -v /tmp/la /tmp/lala

Either the allocation or the access to the matrix is broken. This is likely an issue of computing the correct indices in a triangular submatrix. ;)

This is related to the fix in ddcce3d

The current version of Harry supports computing a partial similarity matrix using index ranges. However, unused strings are not freed from memory and waste space.

Harry should support automatically splitting a large matrix into smaller pieces. In particular, the user should be able to specify the number of required splits and the specific split to compute. A possible command-line option could look like this

harry -s 5:2 ...

where Harry would compute the 2nd out of 5 splits. This feature can be used to compute a similarity matrix in a distributed fashion by concurrently running -s 5:1, -s 5:2, ... -s 5:5.

As the merging of the splits depends on the selected output format, merging should be postponed to a later version.

When splitting a string into words using delimiters, Harry represents each word by a hash value. Currently, this hash value has 16 bit and thus collisions are likely if many different words are present.

The hash size should be increased to 64 bit, such that the hash values till fit into a generic data type. This requires reviewing and updating all similarity measures for side effects.