expfactory / expfactory-battery Goto Github PK

View Code? Open in Web Editor NEW

This project forked from bissettp/cognitive_control_ontology

0.0 0.0 11.0 4.06 MB

version 1.0 expfactory battery template used with psiturk, django, and expfactory.github.io

License: MIT License

HTML 1.93% CSS 19.55% JavaScript 78.52%

expfactory-v1

expfactory-battery's Introduction

The Experiment Factory

See our documentation for getting started. If you are new to containers, read our background or paper first. If you want a more guided entry, see the detailed start

The Experiment Factory is software to create a reproducible container that you can easily customize to deploy a set of web-based experiments.

Citation

If the Experiment Factory is useful to you, please cite the paper to support the software and open source development.

Sochat, (2018). The Experiment Factory: Reproducible Experiment Containers. Journal of Open Source Software, 3(22), 521, https://doi.org/10.21105/joss.00521

Contributing

We have many ways to contribute, and will briefly provide resources here to get you started.

How to Contribute

If you are a developer interested in working on the Experiment Factory software you should read out contributing guidelines for details. For contributing containers and experiments, see our user documentation. If you have any questions, please don't hesitate to ask a question. You'll need to lint your code using black:

$ pip install black
$ black expfactory --exclude template.py

Code of Conduct

It's important to treat one another with respect, and maintain a fun and respectful environment for the open source community. Toward this aim, we ask that you review our code of conduct

Background

It's predecessor at Expfactory.org was never able to open up to the public, and this went against the original goal of the software. Further, the badly needed functionality to serve a local battery was poorly met with expfactory-python as time progressed and dependencies changes.

This version is agnostic to the underlying driver of the experiments, and provides reproducible, instantly deployable "container" experiments. What does that mean?

You obtain (or build) one container, a battery of experiments.
You (optionally) customize it
- custom variables (e.g., a study identifier) and configurations go into the build recipe
- you can choose to use your own database (default output is flat files)
- other options are available at runtime
The container can be easily shared.
You run the container, optionally specifying a subset and ordering, and collect your results

If you build on Docker Hub anyone else can then pull and use your exact container to collect their own results. It is exact down to the file hash. Note that bases for expfactory were initially provided on Docker Hub and have moved to Quay.io. Dockerfiles in the repository that use the expfactory-builder are also updated. If you need a previous version, please see the tags on the original Docker Hub.

Experiment Library

The experiments themselves are now maintained under expfactory-experiments, official submissions to be found by expfactory can be added to the library (under development) to be tested that they meet minimum requirements.

expfactory-battery's People

Contributors

Watchers

Forkers

ianeisenberg zenkavi tmaldonado1821 tangi75 bryanxwu earcanal vinhton123 jkl071 tonytbui henrymj poldracklab

expfactory-battery's Issues

proposed changes to battery

The need for the "battery" arose because I was essentially removing Ian's hard coded experiments from their base, the skeleton that is called the "battery."

Battery

What is a battery?

A battery is (still) a collection of experiments, but it really corresponds with a set of experiments and a desired deployment environment. As we have it now, we are hard coding common scripts and templates into expfactory-battery, holding an expectation about the experiment format, running, etc. This is a bad strategy, because it means that the experiments are dependent on the battery, can't be run without it, and you run into trouble with experiments having different dependencies (and needing to be served together). Thus, my proposal is that this battery component is entirely removed. The dependencies for each experiment must live with the experiment to make it reproducible. Any "plugging in" of something to a template should live with singularity-python. Thus my proposal is:

work toward removing expfactory-battery dependency entirely

dynamic loading of scripts

the current method to "dynamically load scripts" seems to be error prone in that plugins and style sheets aren't actually loading. I need to make a new set of methods to do this, and this will resolve expfactory/expfactory-experiments#96

load_experiment.js not retrieving "[SUB_TOTALTIME_SUB]"

The experimentDraw() function doesnt seem to return a list of the experiments. I suspect that since the code is not properly retrieving the value from SUB_TOTALTIME_SUB because of the "" on line 16.

var time = time || "[SUB_TOTALTIME_SUB]"

Also where is load_experiment.js retrieving values from
[SUB_EXPERIMENTCONCAT_SUB]
[SUB_EXPERIMENTTIMES_SUB]?

expfactory fails after installing psiturk

I installed and set-up psiturk (using pip install psiturk, pip installed with conda) and now expfactory won't open. Even after I pip uninstall psiturk I get the below error when calling expfactory. Further, even if I uninstall and reinstall expfactory, the error remains. The only solution I've found is to complete remove and reinstall anaconda python. Is this a anaconda issue or an expfactory/psiturk issue.

psyc308-01:~ congalab$ expfactory
Traceback (most recent call last):
File "/opt/anaconda/anaconda2/bin/expfactory", line 11, in
load_entry_point('expfactory==2.5.46', 'console_scripts', 'expfactory')()
File "/opt/anaconda/anaconda2/lib/python2.7/site-packages/expfactory/scripts.py", line 141, in main
from expfactory.interface import start
File "/opt/anaconda/anaconda2/lib/python2.7/site-packages/expfactory/interface.py", line 44, in
app = EFServer(name)
File "/opt/anaconda/anaconda2/lib/python2.7/site-packages/expfactory/interface.py", line 23, in init
custom_battery_download(tmpdir=self.tmpdir)
File "/opt/anaconda/anaconda2/lib/python2.7/site-packages/expfactory/vm.py", line 37, in custom_battery_download
download_repo(repo,"%s/%s/" %(tmpdir,repo))
File "/opt/anaconda/anaconda2/lib/python2.7/site-packages/expfactory/vm.py", line 23, in download_repo
return Repo.clone_from("https://github.com/expfactory/expfactory-%s" %(repo_type), destination)
AttributeError: type object 'Repo' has no attribute 'clone_from'

Local battery template overrides experiment's style.css

I'm not sure how the templates are generated, but the installed localbattery.html (2.5.47) contains:

<head>
    ...
    <link rel='stylesheet' href='static/css/default_style.css' type="text/css">
</head>

This refers to default_style.css after an experiment's style.css making it awkward to override styles (e.g. experiment background colour). As the canonical config.json includes default_style.css before style.css it looks like the reference in the template should be removed.

jsPsych is not defined in experiments

When trying to run a battery test with (plus_minus, stop_signal, and stroop), the experiment doesnt load because of the following errors

Uncaught ReferenceError jsPsych is not defined experiment.js:35
Uncaught ReferenceError jsPsych is not defined experiment.js:167
Uncaught ReferenceError jsPsych is not defined experiment.js:147
Uncaught ReferenceError stroop_experiment is not defined load_experiments.js:48

Error 1,2,3 are referencing to the line calling jsPsych.randomization
var numbers = jsPsych.randomization.repeat(numbers, 1, false)

Error 4 references to
case "stroop":
experiments = experiments.concat(stroop_experiment)
break;