We need to update error handling with two items:

(1) Catch and skip videos that error so that a single problem does not tank the entire job.
(2) Save errors in a machine readable format (file, message, and traceback) so that it is consumable by zamba cloud.

Eg CNNEnsemble

[email protected]('--sample_model',
-              type=bool,
-              default=False)
[email protected]('--model_class',

pjbull: Add an issue to remove this eventually when we can test with sample model through mocks instead

Currently, our video loading has a number of limitations:

• Does not support passing a list of videos rather than an entire directory
• Assumes everything in a folder is a video
• Does not check file is a video before trying to load it

If a video or file is invalid, the entire process throws an error and ends. This is probably not desirable in the case where someone is batch processing a large number of videos.

Path objects are magical, let's keep em around. Convert to str only when necessary.

FFMPEG

• a little ffmeg install walkthrough
• include overview of commands, 15 sec resize example in getting started

The RUN conda install av -c conda-forge in the Dockerfile is crucial. Installing av from pip will cause CnnEnsemble to fail.

If this exists, we don't download zamba.zip, which is a problem.

For example, it seems that second_stage.py and second_stage_nn.py share some function signatures such as loading data. Would it be possible to refactor here, and/or any other places?

• Upload zipped weights to PUBLIC s3 bucket - include mirror links for europe and asia
• If not exists weight_dir: ask to download, download
  -- warn about size!
  -- unzip to weights dir
  -- goes in init of CnnEnsemble
  -- use requests, os.environ.get for weights url, default to ours
  -- zipfile, in standard lib

• no hard-coded data paths!
• pull some, known classes, from s3
• add test on single nano vid (to ship with package)

The Problem

Now that we have simple functionality, we need to hone in on specific features. To do this efficiently, we need

• a lightweight model pipeline
• a pipeline similar enough to the primatrix classifier so that core functionality can be developed
• a pipeline simple enough so that development can proceed at a reasonable pace

A Solution

Building a convolutional network to classify the cifar10 data fits the above criteria. The cifar 10 problem is to classify 32x32 pixel RBG images into one of 10 classes. It is a dataset often used in academics, but I think can be very useful for application development as well, at least in our case.

The idea is to use cifar10 as the context, and a simple convolutional net as the "default model" (eventually dmytro's) that we can use to develop the predict, train, and tune functionality. Since the inputs (images) and outputs (row of probs for one image) are similar, minimizing the code's bias towards cifar should be less complicated than if we keep using random number data and non-predictors as a development model.

The goal is to frame as many pri-martrix problems as we can as cifar problems. My claim is that there are a non-trivial amount of pri-matrix problems that allow this framing. For example,

1. Handling real input directories, pre-processing their contents into temporary storage, and sending them into a model for prediction.
2. Handling output that is similarly structured to the chimps output, i.e., csv files indexed by a filename with class probabilities as columns.
3. Training, tuning, saving, and loading functionality for classifiers, not just tensorflow graphs (although that was a good start).
4. Development of any** additional performance metrics**, such as loss over time, validation scoring, etc.

As the tensorflow advanced tutorial on cifar says,

The reason CIFAR-10 was selected was that it is complex enough to exercise much of TensorFlow's ability to scale to large models. At the same time, the model is small enough to train fast, which is ideal for trying out new ideas and experimenting with new techniques.

In short, this approach seems to me a good solution to asynchronously working on the codebase with dmytro before his models are ready for integration.

Advantages

• The above-referenced tutorial comes with a lot of code that can be edited and used to speed up development of our project, such as network architecture, training criteria, data-fetching scripts, pre-processing tasks, and various other "advanced" features that we can consciously decide to develop or not as we work on the tool.
• The fact that inputs to the dmytro model are images and these data are images will allow us to develop a lot of the preprocessing pipeline (except the parts that convert the videos into images).
• We can experiment with some of the ensembling that we'll eventually need to handle, such as using tensorflow and xgboost to generate predictions.

Disadvantages

• Will inevitably have to deal with some idiosyncrasies of the cifar10 data, but given its simplicity and the fact that we have lots of starter code for manipulating it, this shouldn't set us back too much.

Timeline

Since the code skeleton is now fairly operational, I think it's reasonable to shoot for integrating this by the end of the week. End of first rev at the latest.

i.e. if you have images, we’ll run MegaDetector instead and give you results
could also consider other ways to further implement MegaDetector for videos as part of Zamba backend

Convert testing configs to @pytest.fixtures. Remove the temporary tf testing of the workflow.

The eu mirror doesn't have the weight files:

• input.tar.gz
• output.tar.gz
• data_fast.zip

Make cli a thin wrapper around model object. cli should be focused on user io only. Use ModelManager object to mediate loading, configuration, and logic of model calls.

Eg,

class Model(object):
    def __init__(self):
        pass




class ModelManager(object):
    def __init__(self, modelpath, tempdir=None, predict_thresh=None):
        self.model = load_model(modelpath)
        self.predict_thresh = predict_thresh

    def predict(self, datapath, outputpath):
        data = load_data(datapath)

        if self.predict_thresh:
            preds = self.model.predict(data) >= self.predict_thresh
        else:
            preds = self.model.predict(data)

        preds.to_csv(outputpath)

    def train(self):
        pass

    def tune(self):
        pass

By convention, Python projects usually have the tool name as the folders rather than a generic folder named src.

Once we've got a name, we should update these.

• for weights in PUBLIC s3 bucket - include mirror links for europe and asia

• ffmpeg: include overview of commands, 15 sec resize example

Clean up comments while there aren't many to keep track of.

long term have ModelManager "load" the data (whatever that means) and consistently pass the same thing to any model class that is implemented.

Think about this.

Load and parse requirements.txt for use in setup.py. That way we only have to maintain dependencies in one place.

https://github.com/drivendataorg/chimps-tool/blob/master/setup.py#L8

I setup an automated test run with Codeship so that we have continuous integration. Tests on master are currently failing.

(@caseyfitz I know there's WIP to update those, so this is just to track that work.)

The file zamba.models.cnnensemble.src.config.py sets a lot of paths. Yet many files such as zamba.models.cnnensemble_model.py import Path and config but use Path objects local to the file instead of config.

Use config where possible throughout the cnnensemble model.

Do this after initial ModelManager is merged. Doc everything that exists so far.

This includes updating README.md post djamba.

• check all vids up front before handing off to dmytro code, we expect 15 sec
• use skvideo.io.ffprobe (add skvideo to req)

https://zamba.drivendata.org/docs/

Options for Running zamba could include Zamba Cloud web application (available for conservation researchers upon request) with a link to zambacloud.com

@pjbull I had a note on this from when this has come up a couple times in the past. I think it's worth making explicit but don't feel strongly. Assigning to you for now, feel free to reassign/remove

We use megadetector v3, but there are significant improvements in v4.1 so we should use that

We're not looking for full coverage, but there is a lot of commented out code throughout the src directory

We always output one file, correct? If that's the case, we should let users specify the full filename not just the path.

https://arxiv.org/abs/1912.03538 (Sara Beery)
https://colab.research.google.com/github/tensorflow/models/blob/master/research/object_detection/colab_tutorials/context_rcnn_tutorial.ipynb

Remove asserts that aren't in testing. Raise exceptions if it makes sense, or just let the default error message arise.

We should have a better message for users rather than failing in this case:

(1) Ask zamba to run on 1 file that is not a valid video
(2) zamba will say "skipping

Logs look like:

2019-01-11 02:05:00,757 - root - INFO - making predictions17
Predicting on 1 L1 models:   0%|          | 0/1 [00:00<?, ?it/s]18
Processing 1 videos:   0%|          | 0/1 [00:00<?, ?it/s]�[A19
Processing 1 videos: 100%|██████████| 1/1 [00:07<00:00,  7.82s/it]�[A20�[A
Predicting on 1 L1 models: 100%|██████████| 1/1 [00:38<00:00, 38.37s/it]21
Skipping file that is not a valid video:  /paperspace/videos/123d4d52-858c-49b2-82d1-ae8d490df66f_12090006.AVI
Traceback (most recent call last):
  File "script.py", line 30, in <module>
    manager.predict(str(download_dir), save=True, pred_path=output_path)
  File "/src/zamba/zamba/models/manager.py", line 95, in predict
    preds = self.model.predict(data)
  File "/src/zamba/zamba/models/cnnensemble_model.py", line 67, in predict
    l2_results = second_stage.predict(l1_results, profile=self.profile)
  File "/src/zamba/zamba/models/cnnensemble/src/second_stage.py", line 273, in predict
    xgboost_predictions = xgboost_model.predict(l1_model_results)
  File "/src/zamba/zamba/models/cnnensemble/src/second_stage.py", line 141, in predict
    return self._predict(X)
  File "/src/zamba/zamba/models/cnnensemble/src/second_stage.py", line 201, in _predict
    return self.model.predict_proba(X)
  File "/usr/local/lib/python3.6/dist-packages/xgboost/sklearn.py", line 575, in predict_proba
    ntree_limit=ntree_limit)
  File "/usr/local/lib/python3.6/dist-packages/xgboost/core.py", line 1050, in predict
    self._validate_features(data)
  File "/usr/local/lib/python3.6/dist-packages/xgboost/core.py", line 1308, in _validate_features
    data.feature_names))
ValueError: feature_names mismatch: ['f0', 'f1', 'f2 
...

Not all ML code is easily testable, but where functions with well defined input and output exists, we'd like test coverage.

In the original Cnnensemble implementation ensemble of L1 models have been used trained on 4 folds.

It provides very limited if any accuracy gain while 4x slower, it's better to re-train the models on the whole dataset, including test data.

Make model_path optional in the case that we are training a model and don't have a path to load from

We will be developing a simple command line tool entirely in python, so being able to call training, testing, predicting, and submission processes from within Python (rather than through bash scripts) makes that more reliable.

As mentioned in the README.md, cmd is a placeholder.

Usage at the command line will look like

cmd predict --datapath /path/to/my/new/cameratrap-footage

People will install the tool from PyPi using

pip install cmd

The issue will be resolved when we replace cmd with something... better 😉

Suggestions so far (rest in comments):

• cam2chimp
• snapchimp
• SpeciesSpotter
• camtrap.ai
• chimply

The demo is great, but with https://asciinema.org/ we can allow users to copy/paste code from the video!

This only has a model file with it right now, so will move into model dir...

Requires homebrew, otherwise ligghtgbm docs suffice (but pip does not lead to an importable install)

Remove or create issues from #TODOs in the code base.

We install tensorflow by default in setup.py. Better to install tensorflow-gpu if a gpu is available.

• Use dmytro's soln intro