To test, we will use the hardcoded / manual battery file. However, this file should be generated on creation:

• fields specific to the cognitive atlas task / concept should be filled in automatically
• user can specify database / connections, and they should be tested before generating battery
• likely best to run functions in a flask web interface

It would be possible to write the code into a function and drop into the experiment template, but it seems easier to just use a different html template.

Advice from @rwblair

http://python-future.org/compatible_idioms.html

Write function into executable to preview an experiment folder

right now, the export has the answers and the unique ids of the questions, but this should obviously also include the question options and content, to be comprehensive. This should be fairly easy to add as hidden fields to the form, or just other attributes on the form data (need to test what I can do).

How do we validate the jspsych files that are outside of experiment folder? We currently check that files specified to be in the folder are there, but the validator currently doesn't check that files specified to be in the (most up to date) battery folder are actually there. Should we? What do you think @IanEisenberg ?

right now the user just generates a battery, and not the Vagrantfile to generate the VM. Make the Vagrantfile, then turn it into an Amazon AWS one!

it's not efficient to have empty fields - if a field is not required (value of 1) then it should be optional to have in the file.

I'm not sure if I want to do this, because it will slow down validating the experiments. However it would be good to have some way of validating the survey.tsv file itself. My thinking is that we should have a circle test that does it, but not have it done when validating the config.json.

We do not want to have "-" in tag names

In the case that a script is http, it should return invalid in the config.json. This is for surveys and other experiments that link to external style sheets and javascript scripts.

the current --runbat command should always be used to run a single experiment or battery, and so it should be changed to --run. The functions --test, --validate, and --preview (the current run) should be associated with development of experiments

user should be able to use executable to generate a blank experiment "template" directory

Right now, get_experiment_html assumes we are adding css, js, and a variable "run" that goes into the document ready function. With different templates (eg, a survey) we should also be able to add html, and set the "run" variable to empty, for example. This should be pretty easy to add.

Right now no page is defined. There should be a screen to take user to different functions, and possibly running the package should open the browser automatically

JsPsych is probably overkill for just a survey. I want to start developing another option for just survey stuffs that could be used to generate custom forms from a tsv file. I'm not sure if this will be integrated into current experiments given the time frame, but it would be good to have this option available for users regardless.

Did not work for Patrick on Mac, check experiment (folder) generation.

not necessary, adds slowness. also generate random port so bug goes away

I am running expfactory --validate and receiving an error like so:
WARNING: config.json is missing field name: psychological_refractory_period
WARNING: config.json is missing field reference: psychological_refractory_period

On circle it says:
config.json could not be found in /home/ubuntu/expfactory-experiments/web
/home/ubuntu/expfactory-experiments/web is not an experiment.
Found 79 valid experiments
Directory not copied. Error: [Errno 2] No such file or directory: '/tmp/tmp9Zb8Nv/experiments/local_global_shape'
config.json could not be found in /tmp/tmp9Zb8Nv/experiments/local_global_shape

The config file is correct (and existing) however. You can see the PR here: https://github.com/IanEisenberg/expfactory-experiments/blob/fixes/local_global_shape/config.json

Not sure what's going on

This can come a bit later, when things are up and running for others (likely over xmas break is a good time)

The command line tool expfactory is not easily extendable to those that don't know how to use a command line. We should consider making an electron / node.js app to make platform specific executables.

because, questions can be required. yeah.

should look something like:

  jspsych_init: {"variable_name":"value"}

• provide instructions in docs (need to clarify how to do spaces
• add validation functions for variables that must be certain way

Akin to the "preview" function, we should have a similar one that can be run to deploy one or more experiments locally, on the fly. This would mean inputting a user ID, and then running either a single experiment or a battery.

An experiment should minimally not return an error page to be included. Not sure about the rest of the validation to do beyond that.

right now, you can run a survey folder with expfactory --preview. It should be possible to deploy any number of surveys together, equivalent to what we do for experiments, with the run command. For example:

  expfactory --run --surveys bis_bas,bis11

Advanced functionality (not for this PR) would be to be able to put together a battery of experiments and surveys:

  expfactory --run --experiments stroop,n_back --surveys bis_bas,bis11

but this will be for future / another PR, because the data structures output would need to be consolidated. For now, it seems reasonable to have surveys be a different thing than experiments (because they kind of are :O) )

Currently, there are no pages (no next button) #55 and no submission button. The surveys should have a next button, likely at the bottom of the page, and likely we want data to be saved from the forms as the user progresses, and then finishing the survey should trigger the same chain of events:

• expfactory_finished is set to true
• data is submit again to the sync url

We will also need a function to save the data locally. I will figure this out when I get to it, lots of details to work on before then.

This is useful for developers to test their experiments before doing PRs to expfactory-experiments