expfactory / experiments Goto Github PK

so the robots render it nicely in search results :) 🤖 🚀

The table should link to the experiments (repos and previews) that are in the library. Thus, this table should be derived from the library manifest.

we want the instructions to be programmatically accessible for any deployment application, so for example, on install into the container we can pipe them in upon init.

I made some changes to the existing experiments and I'm wondering if I would need to make another pull request. If so, could you provide me wit some instructions? I remember not doing the pull request correctly and I would like to get them right this time.

These are relatively simple to provide, so it should be reasonable.

Required

• exp_id: must not yet exist for all expfactory experiments, alphanumerical / _ and - allowed.
• url: the repository where the code is maintained
• description: tell us what it is
• time: estimated time to complete experiment, in minutes

Optional

• contributors: a list of contributor names
• cognitive_atlas_task_id: if relevant
• notes: any additional notes
• template: a html/framework template, if relevant
• reference: references, if this is important to you

These will be checked in the CI, and added to the CONTRIBUTING.md document.

I'm having some trouble saving the experiment data where (1) the data would fail to save and (2) the program will not automatically advance to the next experiment in the container. This occurs for both JsPsych and custom experiments. Here is the code that I'm using to save the data:

For JsPsych:

const settings = {
    fullscreen: true,
    on_finish: function (data) {
        // Serialize the data
        var promise = new Promise(function (resolve, reject) {
            var data = jsPsych.data.get().json();
            resolve(data);
        })

        promise.then(function (data) {
            sendResults(data);
        })
    }

}

const jsPsych = initJsPsych(settings);
async function sendResults(results) {
    function handleErrors(response) {
        if (!response.ok) {
            throw Error(response.statusText);
        }
        return response;
    }

    fetch("/save", {
        method: "POST",
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ data: results })
    })
        .then(handleErrors)
        .then(response => console.log("Request complete! response: ", response))
        .catch(error =>
            jsPsych.data.get().localSave('csv', 'cognitive_estimation_task_result.csv')
        );
}

For custom:

async function sendResults(results) {
    function handleErrors(response) {
          if (!response.ok) {
              throw Error(response.statusText);
          }
          return response;
      }

      fetch("/save", {
          method: "POST",
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ data: results })
      })
          .then(handleErrors)
          .then(response => console.log("Request complete! response: ", response))
          .catch(error => console.log("We got an error: ", error));
}

• now experiment manifest is automatically generated: https://expfactory.github.io/experiments/library.json
• and tests need to run via parsing the markdown files instead! temporary disabled :)

experiments need more useful main pages too, what else to add?
https://expfactory.github.io/experiments

This will only be an issue if the repo gets yuuuge.

We would want a change in an experiment repo to trigger another test, that will alert the library if it doesn't pass. I need to think about how to best do this, there are a few options (and I'd like to avoid sharing of any kind of credential between repos). Arguably, the test cases should be provided for the experiment submitted, and then asserted to exist in testing. Any change to the status of the downstream repo (not passing tests, etc.) should alert the library.

e.g., a user should be able to build an image, and then "register" it here for others to see.