We can initially hard-code a popper-templates repo and add basic functionality
for pulling from there.

For example, in the 'Popper in Data Science' guide:

This was very confusing for me. What is the goal of the CLI? Is it
someone looking at someone else's popper repo and wanting to re-run
an experiment? Is it me starting a new project?

What does popper experiment jupyter-bww myexperiment do? Pull
stuff from the internet, re-run stuff?

In general, the guides give the impression that in order to understand
it you need to already be familiar with Popper. Isn't the goal of a
guide to familiarize the reader with Popper?

Executes every experiment in experiments/, for paper/ folder, it builds the
manuscript (if exists).

Roadmap

Lowering the barrier to follow a sound experimentation protocol rooted in best open source software (OSS) practices.

Validating experimental results in research is time-consuming; it takes too long to recreate the environment where an experiment originally ran. Popper is an experimentation protocol and convention for organizing files in a version-control repository. The Popper CLI tool aids practitioners in following the convention, providing an easy way to generate experiments that are automatically re-executed and validated.

What do we need to do?

This issue is our roadmap. It's a place to start to investigate the myriad of issues that you can contribute to. Check out the different milestones listed below, but feel free to explore the issues by label too.

Please check out our contribution guidelines and code of conduct to help you get started, and the README for an overview if you haven't read it yet!

Upcoming milestones

• 0.6
• 0.7

Completed Milestones

• REPPAR '17
• 0.5

Instead of storing it the repo root

When --update-status is given, the status of the experiment is
updated at http://falsifiable.us . The <repo, experiment> pair that
the RESTful service expects can be inferred from .git/config (by
reading the value of the origin remote) and the experiment in
question being executed.

use the "everything-in-the-cloud" template

create a tutorial with questions for the audience so they can realize how easy was to know information (e.g. parameters, flags, etc.) that otherwise it would have been hard to get.

We currently assume that every run/validate dependency is dockerized
by the user. For example, if an experiment relies on ansible, the
run.sh command creates a container from an image of the
corresponding ansible version that the experiment depends on.

We can make this easier for users by providing a list of built-in
dependencies. For example, the user specifies in popper.yml:

run: ansible:2.1.2.0

And, as part of our project, maintain a list of images that we know
have the requested dependency. In the above, we would create a
container from an william/ansible:2.1-debian8 image, bind-mount the
experiment folder and inside the container, we'd invoke the run.sh
or validate.sh script. For dependencies that we don't know images
for, we can open new issues to create them as part of this project.

This will display the experiment's README

• Add link(s) to 'Intro to Popper' once it's done (see #6)
• Add link to 'Tool agnosticism' once it's done (see #6)
• Add link to '.popper.yml config' once it's done (see #11)
• Add link to 'Folder Structure' explanation once it's done (see
  #10)
• Add description of BWW notebook (what it consists off)
• Finish with 'Adding More Datasets' section
• Finish 'Generating Image Files for Later Usage' section
• Finish 'Documenting Experiment' section

Add capability for user to specify ansible as a run or validate
dependency.

Currently, status of experiments are kept in memory only. We can make
use of an pure go KV store to persist statuses.

A pre-push hook would download the popper binary and run popper check

This takes care of retrieving the status for all experiments in a repo

Check dependencies when a --dependencies flag is passed to the check subcommand. The convention for checking things is the following:

Folders named after a tool (e.g. docker or terraform) have special meaning. For each of these, tests are executed that check the integrity of the associated files. For example, if we have an experiment that is orchestrated with Ansible, the associated files are stored in an ansible folder. When checking the integrity of this experiment, the ansible folder is inspected and associated files are checked to see if they are healthy. The following is a list of currently supported folder names and their CI semantics (support for others is in the making):

• docker. An image is created for every Dockerfile.
• ansible. YAML syntax is checked.
• datapackages. Availability of every dataset is checked.
• vagrant. Definition of the VM is verified.
• terraform. Infrastructure configuration files are checked by running terraform validate.

All these will be added in a pluggable way (we need to define an API), so that other tools can be included.

Create a RESTful service that maintains the status of experiments for
a project. This should be added to the popper cli as a subcommand
(e.g. popper status-service).

Popperize
this
experiment.

After #70 , popperized examples are in the github/popperized organization. This is the default organization that gets registered in .popper.yml. This issue will track the work needed to add support for adding other orgs or repos containing popper pipelines

we could leverage poldek or something alike that is
platform-agnostic

Explain what does it mean that an experiment is popper-compliant. Add a diagram
of the workflow and how each step is implemented with current tools . Also
explain the concept of a "entrypoints" (visualize and run for experiments; build for papers).

Also explain why Popper is tool-agnostic and what are the basic properties that
make a tool popperizable.

For every experiment in popper-templates, we can have the convention
of having a .popper.yml file that has a list of tools and versions.
When popper experiment add ... is invoked, the CLI quickly checks
for the existence of these and reports if they're available or not.

We assume git for code dependencies

The output of popper check is one of the following three:

• 1: failed to run experiment.
• 0: ran experiment and validation successfully, but results are
  not valid.
• 2: same as in above but results are valid.

When an --update-status flag is given, the status of the experiment
is updated at http://falsifiable.us. The <repo, experiment> pair that the
RESTful service expects is obtained explicitly from the user (--repo) flag and
experiment in question being executed.

github is hard-coded in the CLI tool. We can abstract and add support for bitbucket and others, as well as for plain URLs

Add a popper experiment init subcommand that creates empty,
documented run.sh and validate.sh scripts.

Please explain the issue. The first line will become the title.

Trailing

Initially, project-wide status. In the future, we could add partial
statuses (e.g. 4 out of 7 experiments are good).