Large scale database about sourceafis-java HOT 9 CLOSED

That's a large topic. I cannot just answer it here. I will keep this issue open as a reminder to write an article about it.

from sourceafis-java.

would you please attache here documents link that can help?
Is there any solution to index these data in sql database or nosql
I need your hints ASAP :D

from sourceafis-java.

All templates must be loaded in RAM and deserialized into FingerprintTemplate objects for matching to be fast. I cannot say more here.

from sourceafis-java.

but I think this is not an efficient way!
because for 100,000 users we have to handle about 60MB data compare.
Do have know another idea or platform to store and compare?

from sourceafis-java.

I use LMDB for that. It's neat in the sense that all the fingerprints can be read quickly and the index is fast. But yeah, there is no escaping keeping the processed templates in RAM for fast matching performance. One could try to modify the codebase for having flat templates that can also be memory-mapped, but that is sure to be a pain in Java.

edit: I mentioned LMDB, but I learned the hard way that the db technology doesn't matter much. What matters is keepíng all the processed templates in RAM at all times. That should be no problem for a modern system, even with tens of thousands of entries. Figuring out how to update this cache as efficiently as possible is important as well. It can be done using SQL triggers or by adding hooks to the system that modifies the DB.

from sourceafis-java.

@lucaspcamargo Thanks for your response buddy
I try to load data in memory and that solved my problem.(using a simple array list and comparing method for 1:n)

from sourceafis-java.

because for 100,000 users we have to handle about 60MB data compare.

Did you mean - Mb or Gb? 60 Mb is not such a large amount of data? It looks more like 60Gb per 100,000 users to me.

from sourceafis-java.

because for 100,000 users we have to handle about 60MB data compare.

Did you mean - Mb or Gb? 60 Mb is not such a large amount of data? It looks more like 60Gb per 100,000 users to me.

MB

from sourceafis-java.

Memory usage benchmarks for templates are now available:

https://sourceafis.machinezoo.com/cli

The general topic of using SourceAFIS on large databases is too broad for one issue. Please open separate issues to request specific documentation.

from sourceafis-java.