jarnorfb / incense Goto Github PK

A common pattern when loading experiments is to pick some data for each experiment (such as the accuracy) and process it for each experiment (for example, storing it into a dataframe).

With the current code, in order to do that we have to first load all experiments into memory and then iterate over them. This is memory-expensive, and I had not enough memory in my machine for doing that for a (rather large) collection of experiments.

However, a lazy way to iterate over the experiments in the query can be provided, so that only one experiment needs to be created at a time.

Hi,
thanks for maintaining this cool library!

One problem I have encountered is that loading hundreds of experiments, each with several metrics, takes a very long time when done over a network.

E.g. loading 300 runs takes around a minute for me.

I think the problem is that the library sends a request to retrieve metrics for each run separately.
With 300 runs, loading the runs takes only around 5 seconds (one request) and the next 55 seconds (300 requests) are spent loading the metrics.

Would it be possible to load the runs and the metrics jointly with one request?
MongoDB should be able to fill in the metrics by their object IDs automatically I believe.

Thanks,
Ondrej

Hi, first of all thanks for making incense.

It seems like there is some caching issue (maybe related to #16 ?). In a notebook:

exps = loader.find_by_key("status", "FAILED")
print(len(exps))
for e in exps:
    e.delete(confirmed=True)
exps = loader.find_by_key("status", "FAILED")
print(len(exps))

outputs

714
714

Only after restarting the kernel does the query return 0 results.

Maybe use IPython display instead of matplotlib

https://codecov.io/

The info dictionary is not loaded in FileSystemExperiment. It should be easy to add.

Great package! Would it be difficult to add a filter method to the QuerySet object? I'm imagining it would work similar to exps.project(on=["experiment.name", "config.optimizer", "config.epochs"]), except it takes a list of key:value pairs and return a filtered list of experiment objects. I often find myself loading all experiments of a particular name, then writing ugly for loops to further filter the list based on some criteria.

• tries to import pytest
• mongo can't find native libraries

Using https://github.com/asottile/future-fstrings

The last release is not in PyPI. I suggest you to use a Github action to upload automatically to PyPI on release.

Otherwise the caching mechanism could lead to unexpected side effects.

Sacred creates its own encoder to encode numpy arrays in experiments. Thus, if sacred is not imported before obtaining an experiment with numpy arrays in the info dict, those are not unpickled. I propose to import sacred, at least when unpickle is True.

• isort
• mypy
• black

Currently, the host and port should be passed as a single URL to ExperimentLoader. If you already have those, it should be possible to pass them instead of building the URL.

Hi，
I pip install incense. but I find the pip distributed version cannot support the FileSystemExperimentLoader. Will you release a new version to fix it?
Thanks a lot

Draft for the API:

loader.find(...).artifacts["my_artifact"].save()

Hi, thanks for the library! Very useful for working with sacred experiments.

Is there a way to specify multiple reducers for projection? Say, I want to get the min and argmin of a metric. Right now I can do one of them with query_set.project(on=[{'metrics.loss': np.min}]) (or np.argmin) but not both because dicts only allow unique keys. Looking at the code, it doesn't seem supported yet. Would be nice if it is!

Experiments and loaders for filesystem and mongo should have a similar interface. In particular, a method such as find_by_ids should be available also for the filesystem loader, even if in that case it is just a loop. This allows a user to deal with both in the same way, using the optimized implementation for Mongo if available.

https://docs.travis-ci.com/user/deployment/pypi/

Hello,
Thanks for developping this library. I use Sacred + MongoDB to store movie acquisitions of chemistry experiments and I use Incense to do post-treatment of the movies. Would there be an easy way to store the post-processing results as new artifacts ? something like exp.add_artifact("path_to_temp_file") ?
Thanks !

Make the sources of an experiment accessible similar to artifacts.