jorensix / tarsoslsh Goto Github PK

In CommandLineInterface.java

if(radius == 0 && hashFamilyType.equalsIgnoreCase("l1")){
    measure = new EuclideanDistance();
    radius = LSH.determineRadius(dataset, measure, timeout);
} else if (radius == 0 && hashFamilyType.equalsIgnoreCase("l2")){
    measure = new CityBlockDistance();
    radius = LSH.determineRadius(dataset, measure, timeout);
}

The position of EuclideanDistance and CityBlockDistance should be swapped?

I use this lib in my homework for Image and Audio Search ,Thank U very much.
“Drop me a line if you use TarsosLSH in your project. Always nice to hear how this software is used.” but I do not know how to do it .I have to write it here.
By the way, I think as a lib, some class lost some method .
For example:
I add "public Vector(String key,double[] double){}" to create a vector immediately(this is very important , I think)
I add some method to get dataset from memory instead of disk file, like:
“public LSH createIndex(double radius,String hashFamilyType,int numberOfHashes,int numberOfHashTables,int numberOfNeighbours,List dataset){}” for more convenient.

As a naive learner I want to ask a naive question that how to use LSH in Hamming. I have tried use BinLSH, however it can not run successfully, the format is as below and the dataset is your test dataset.

String[] test = new String[]{"-d","D:/TarsosLSH/build/dataset.txt","-q","D:/TarsosLSH/build/queries.txt"};
BinCommandLineInterface cli = new BinCommandLineInterface(test);

the problem is:
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Could you tell me how to solve this problem? I will be appreciated it if you give me the answer as soon as possible.Thank you.

In families/CityBlockHash.java, you calculate the hash with the following code (line 51):

hash[d] = (int) (vector.get(d)-randomPartition.get(d) / Double.valueOf(w));

I could be wrong here, but would you want parenthesis around the numerator? Otherwise order of operations takes precedence, which I'm not sure is correct.

Thanks. I'd be happy to put in a pull request for you, if need be.

who can explain the meaning of the parameter m?

The command line runner help is incorrect regarding EuclideanDistance:

if(radius == 0 && hashFamilyType.equalsIgnoreCase("l1")){
  measure = new CityBlockDistance(); 
  ...
} else if (radius == 0 && hashFamilyType.equalsIgnoreCase("l2")){
  measure = new CityBlockDistance();
  ...
}

CityBlockDistance is used for both L1 and L2

Even after using 13g of memory I get "GC overhead limit exceeded" exception with following stack trace:
Picked up _JAVA_OPTIONS: -Xms2g -Xmx13g -Xss100M -XX:MaxPermSize=3g
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.(String.java:203)
at java.lang.String.substring(String.java:1913)
at java.lang.String.split(String.java:2288)
at java.lang.String.split(String.java:2355)
at be.tarsos.lsh.util.FileUtils.readCSVFile(Unknown Source)
at be.tarsos.lsh.LSH.readDataset(Unknown Source)
at be.tarsos.lsh.CommandLineInterface.parseArguments(Unknown Source)
at be.tarsos.lsh.LSH.main(Unknown Source)

My dataset size is 4GB, it contains 15201 vectors. I run with following options "java -jar TarsosLSH-0.7.jar -f l2 -h 8 -t 32 -n 15 -d tfVectorFor105G -q tfVectorQueriesOn105g"

I would like search for similar text documents using TarsosLSH. Maybe you could suggest some techniques, how to transform text to vector, suitable for this library?

Thank you!

I am contemplating using LSH in my application, but I am unsure how to deal with absent/missing data in a vector. The nearest neighbor imputation implies that this type of algorithm deals with this scenario, but how would I go about implementing it?

Is it as simple as not calling "set" for that dimension?