Directory cleaner about warp10-platform HOT 7 CLOSED

You mention "This means that we cannot easily delete inactive series by executing a WarpScript.", could such deletions be performed using a call to the /api/v0/delete endpoint, possibly with the use of the 'metaonly' flag to simply remove the metadata?

from warp10-platform.

On the implementation side of things, what you describe looks a lot like a Directory plugin, why modify the Directory class instead of creating a Directory plugin?

[Updated] is it because the lastactivity is not available to plugins? This can be easily added to the plugin interface if need be.

from warp10-platform.

Indeed, it's not possible for us to do a delete with metalonly as selected a series subset to delete with a find is too costly.

from warp10-platform.

For the directory plugin, I didn't think about it at all. I will check but as long as we get the init series metadata, this cleaner could be a directory plugin indeed. However, for my use case, I would still need to provide, inside the directory, the source code enabling a clean stop once the directory has loaded HBase.

from warp10-platform.

With some second thought, I do not think it matches the directory plugin pattern, as I am adding context when initialising the directory with HBase (in case the lastactivity is too old send a delete message to Kafka). Otherwise I keep the same pattern, and the directory work as usual. It's only in the case where the parameter directory.cleaner.stop is set to true that I change the behaviour (storing no series in RAM and shutting down the directory when HBase is read). This parameter may have another name, as even if I use it for the directory cleaner it can be used globally for the directory.

from warp10-platform.

We discussed with @hbs and there could be a possibility as a trusted directory plugin which require Mathias to merge it :) Once done we will give it a look, but as I understood it, this case would have all necessary context to arbitrate a deletion while keeping the same usage pattern or running a job regularly.

from warp10-platform.

Hello @StevenLeRoux @aurrelhebert ,
2.11.0 is released with TrustedDirectoryPlugin. Waiting for your feedback...

from warp10-platform.