trandoshan-io / persister Goto Github PK

With ScyllaDB

Currently persister prevent duplicate content by deleting any duplicate before adding new content.

To support the up-coming website snapshot feature this behavior has to change.

This check has to be completely removed as it's not the persister responsibility to determinate if something has to be saved but It's the scheduler one.

Instead of storing content in a MongoDB the persister will now store the raw resources directly on the filesystem.

The storage format will be like this:

resource-url/64bit-timestamp

For example the following resource:

http://login.google.com/secure/createAccount.html

Will be store like this on the disk:

login.google.com/secure/createAccount.html/1570788418

After that the API will be the only process to access the database and therefore will allow change of database technology easier.

Need to do trandoshan-io/api#1 first

Each time content is received by the persister it will be persisted on disk once #7 is merged, but that's not enough:

To increase performances when doing a text search the content will be structured by the persister and stored into an Elasticsearch instance.

This instance will be used by the API when performing a text search