Storing list of artifacts in database about laminar HOT 5 OPEN

The original idea supports moving archived artefacts to cheaper storage but assumed this would be achieved by mounting or symlinking the archive directory appropriately. It's simpler to just dynamically iterate the folder, but iterating is more expensive especially on slow storage or if there are many artefacts. I'm not opposed to your suggestion, just wanted to check why mounting/symlinking would not work for you since this proposal has some (admittedly low) added complexity versus the current situation

from laminar.

If I want to archive artifacts on S3, mounting them so Laminar finds them means FUSE -- already extra complexity. Furthermore,
scanning S3 to render a job page (list of artifacts on the top) is both slow and costly.

Technically, I can keep empty files in /laminar/archive to inform Laminar about what artifacts are associated with the job, yet configure the reverse proxy to go to S3 instead, but that means using the filesystem as a database. Huge pain to back up. readdir(3) won't be happy.

from laminar.

Fair enough

from laminar.

@mitya57 are you still offering a PR for this? I'm happy with the justification

from laminar.

Sorry for the late response.

@mitya57 ended up with postgres-only fork. https://github.com/mitya57/laminar/tree/wip/postgres

That was necessary to speed up the things by taking advantage of Postgres materialized views and other nice features. Patch to keep the artifact list in the database ended up tightly coupled to other changes, so, I guess we can close the issue.

from laminar.