expfactory / expfactory-deploy Goto Github PK

The task directory names can be noticeably different from the “official” name that e.g. shows up in a search - when collaborating with others I may have to view their code / config file to make sure I’m grabbing the correct task. It might be nice to standardize these directory names (e.g. task-taskname_group-groupname.js) and have a check for formatting a la BIDS. Somewhat niche, but this did happen while setting up Matilde’s battery, and standardized naming could just help organization in general.

At the same time, expfactory does not like if you have numbers, or capital letters and quietly prevents you from uploading tasks. Either allow these things OR have clear warnings about those, so ross doesn’t have to go into the guts and figure out why tasks aren’t uploading correctly

Maybe add a validator for checking naming and config info?

This would help with downloading data as well.

1. Could be useful to have separate tabs (webpages) for testing batteries vs production batteries so that it’s quite clear what is what, and an easy way to turn testing batteries to production batteries
2. Ideally, the production battery would be brand new, meaning no data from the testing battery would be included

Things like mean ITI, stop probability, number of trials total / per block, fixation present/absent between trials, etc.

Not necessarily expfactory-deploy issues but making an issue here since its from the wishlist.

• Because of the rigidity of the survey template, some of our surveys are coded as experiments, which leaves them open to more bugs, and an inconsistent user experience for our participants - i.e. instead of just clicking a radio button in a survey, you have to click and hit enter for responses in experiments
• This also is frustrating from a researcher perspective -1. Having to code them, and 2. Many times I’ve been looking for something I know is a survey, but it’s located in expfactory-experiments instead.
• The following are a list of behaviors that expfactory-surveys doesn’t support:
  1. If the presence of a question is logically dependent on the answer to a previous question, surveys don’t have a way of supporting that behavior. All questions will be presented.
  2. Allow different types of survey questions (i.e., slider), or have an easier way for researchers to create ‘plugins’ for different survey question types.
  3. If a question type == numeric, allow the researcher to be able to define ranges for that numeric value

Users can be assigned to a project [which can encompass 1 or more batteries], and download data from that project, without having been the one who created that batteries within that project.

Currently redirects to list of all subjects.

For once mturk functionality is implemented. View for active assignments should hit the aws api directly so that we don't have to try and keep fresh state in the database that mirrors the state in their api.

Potentially: https://psych-ds.github.io/

• Have options for download size - single task within battery, single subject, all subs of a task within a battery, whole batteries...etc
• Have options for download type - json, dict, csv, tsv...etc
• Have a BIDS-like structure for downloaded data
• Battery-name and task-name should be consistent with each download
• Naming structure of tasks should also be consistent with each download
• Would also be nice if each download outputted a dataset_description that has links to the experiment(s) used on Expfactory, date of download, participants in download

Take a url and clone the repository.

• There are different but equally valid instantiations of an experiment (e.g. using a stop signal with a tone instead of a star) across groups as well as within groups. These should be clumped under standardized names (e.g. “stop signal”), and you should be able to choose the instantiation of the task by group/tag (e.g. poldrack__fmri, poldrack__network, ABCD [although obviously not the ABCD group’s stop signal haha], etc.)
• If this happens, it would be cool to link the high level task (e.g. stop signal) with the cognitive ontology for a description of the task and the constructs it’s associated with
• On that note, improved integration with the cognitive ontology could be cool for task/construct exploration. This is vague. Previously this was attempted with the config jsons, but these have not been consistently followed. Maybe if expfactory detects blank sections in the config, it could offer the user a search window to look up cognitive constructs and tag their task.

This will provide some organization to the website; currently all tasks and surveys are presented together.

• Better logging and display of errors on update process.
• Batch updates should be possible

• Write documentation on docker compose file and settings locations
• add basic .envs/.local/* files to repo so compose can work out of the box.
• add admin user to .envs/.local/.django to be autocreated on first startup.

@Dev-Logan-Bennett feel free to add any other things that could of made setup better.

Maybe "deletion" will be more of an archiving. Or at the very least will result in an automatic file based dump of relevant database fields for posterity.

It’s very easy to change battery information, such as battery name / consent / instructions / advertisement pages of a battery that is already in production. In the service of consistency, perhaps include a tracking of these changes, especially when the battery is already in production.

• Ability to edit and deploy specific versions of instructions and advertisement text.

It would be nice to go to a battery, and be able to see how they have been deployed, either via local links or through mturk. There is a tab that contains info about mturk deployment, but many useful information is missing or is inaccurate. There is no info about local links deployment - could have a separate tab for this.

Local link info tab:

• What links were given out and when / Which links were started and when
• Add a range of time for which links could be active
• lists current workers, how many experiments they have done, which experiments they have done, what experiments they still have to do, which workers are finished
• Which workers have consented and date of consent (IRB purposes)

Mturk info and operations:

• Approve / reject / contact / list current working subs / blacklist
• Much of the information was located on mturk site (or not at all), there were some bugs with the info collected and presented by expfactory, info isn’t being dynamically updated (i.e., subs who have returned the HIT will still show up as in progress on expfactory.org)
• lists current workers, how many experiments they have done, which experiments they have done, what experiments they still have to do, which workers are finished
• Which workers have consented and date of consent (IRB purposes)

• If you search for an experiment, click on one, then go back a page, the query will stay in the search box, but the search will not have re-occurred (all experiments are present in alphabetical order)
• You cannot update multiple task orders at once when editing a battery. You have to update a single task's order and save the battery.
• Every time you edit a task’s order number, it takes you back to the first page of the experiment, even if you were on (e.g.) the 3rd. Lots of redundant movement is required; leads to a slow and frustrating battery creation process
• When you click on an experiment/survey, it logs you out! But if you click the top middle 'Experiment Factory', it logs you back in?
• When doing an action on expfactory.org (i.e., updating experiments, deleting batteries), the landing page is quite odd. For example, if I were to delete a battery from ‘my batteries’, I land back on to the ‘all batteries’ page. It would be quite helpful to have the landing page stay on ‘my batteries’. There are other actions with odd landing pages, but here are some odd landing pages I’ve (Jamie) encountered.
• When updating experiments, it brings you to this page where it lists all of experiments again, but doesn’t allow you to update another experiment until you click the ‘Experiments’ button on the header

• Have the ability to update jspsych from within site
• be able to have battery deployments that span jspsych versions

• Need to be able to add and order specific versions of experiments and their libraries.
• Be able to edit group ownership of battery itself.

Currently one column of subjects is a checkbox field. These could be used with above menu to mass assign battery or deactivate ids.