trentocrowdai / crowdai Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 2.0 20.55 MB

A platform for crowd-based screening of literature reviews.

JavaScript 96.96% HTML 0.41% PLpgSQL 2.27% CSS 0.04% Dockerfile 0.12% Shell 0.21%

crowdsourcing koa2 react semantic-ui semantic-ui-react

crowdai's People

Contributors

Stargazers

Watchers

Forkers

nicobell mattiapv

crowdai's Issues

Enhance error handling when publishing a HIT

Check the following errors so that we can send the corresponding error message to the frontend.

AccessDeniedException

{ AccessDeniedException: User: arn:aws:iam::637298523374:user/crowdai is not authorized to perform: mechanicalturk:CreateHIT
2018-04-06T17:42:14.849350+00:00 app[web.1]:     at Request.extractError (/app/node_modules/aws-sdk/lib/protocol/json.js:48:27)
2018-04-06T17:42:14.849352+00:00 app[web.1]:     at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
2018-04-06T17:42:14.849353+00:00 app[web.1]:     at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
2018-04-06T17:42:14.849356+00:00 app[web.1]:     at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14)
2018-04-06T17:42:14.849357+00:00 app[web.1]:     at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)
2018-04-06T17:42:14.849359+00:00 app[web.1]:     at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
2018-04-06T17:42:14.849361+00:00 app[web.1]:     at /app/node_modules/aws-sdk/lib/state_machine.js:26:10
2018-04-06T17:42:14.849362+00:00 app[web.1]:     at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)
2018-04-06T17:42:14.849364+00:00 app[web.1]:     at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12)
2018-04-06T17:42:14.849366+00:00 app[web.1]:     at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
2018-04-06T17:42:14.849368+00:00 app[web.1]:   message: 'User: arn:aws:iam::637298523374:user/crowdai is not authorized to perform: mechanicalturk:CreateHIT',
2018-04-06T17:42:14.849370+00:00 app[web.1]:   code: 'AccessDeniedException',
2018-04-06T17:42:14.849372+00:00 app[web.1]:   time: 2018-04-06T17:42:14.845Z,
2018-04-06T17:42:14.849374+00:00 app[web.1]:   requestId: 'ac9b8278-9e99-4ded-96b5-98528c098810',
2018-04-06T17:42:14.849375+00:00 app[web.1]:   statusCode: 400,
2018-04-06T17:42:14.849377+00:00 app[web.1]:   retryable: false,
2018-04-06T17:42:14.849379+00:00 app[web.1]:   retryDelay: 52.280048917572095 }

RequestError

{ RequestError: To use the MTurk API, you will need an Amazon Web Services (AWS) Account. Your AWS account must be linked to your Amazon Mechanical Turk Account. Visit https://requester.mturk.com/developer to get started.
2018-04-06T18:53:58.182646+00:00 app[web.1]:     at Request.extractError (/app/node_modules/aws-sdk/lib/protocol/json.js:48:27)
2018-04-06T18:53:58.182648+00:00 app[web.1]:     at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
2018-04-06T18:53:58.182650+00:00 app[web.1]:     at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
2018-04-06T18:53:58.182652+00:00 app[web.1]:     at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14)
2018-04-06T18:53:58.182654+00:00 app[web.1]:     at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)
2018-04-06T18:53:58.182655+00:00 app[web.1]:     at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
2018-04-06T18:53:58.182657+00:00 app[web.1]:     at /app/node_modules/aws-sdk/lib/state_machine.js:26:10
2018-04-06T18:53:58.182659+00:00 app[web.1]:     at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)
2018-04-06T18:53:58.182661+00:00 app[web.1]:     at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12)
2018-04-06T18:53:58.182663+00:00 app[web.1]:     at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
2018-04-06T18:53:58.182664+00:00 app[web.1]:   message: 'To use the MTurk API, you will need an Amazon Web Services (AWS) Account. Your AWS account must be linked to your Amazon Mechanical Turk Account. Visit https://requester.mturk.com/developer to get started.',
2018-04-06T18:53:58.182666+00:00 app[web.1]:   code: 'RequestError',
2018-04-06T18:53:58.182668+00:00 app[web.1]:   time: 2018-04-06T18:53:58.179Z,
2018-04-06T18:53:58.182670+00:00 app[web.1]:   requestId: '0d01847e-e886-4d

RequestError

{ RequestError: This Requester has insufficient funds in their account to complete this transaction. Please visit https://requester.mturk.com/prepayments/new to purchase Prepaid HITs. (1523041500385)
2018-04-06T19:05:00.367668+00:00 app[web.1]:     at Request.extractError (/app/node_modules/aws-sdk/lib/protocol/json.js:48:27)
2018-04-06T19:05:00.367670+00:00 app[web.1]:     at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
2018-04-06T19:05:00.367671+00:00 app[web.1]:     at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
2018-04-06T19:05:00.367673+00:00 app[web.1]:     at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14)
2018-04-06T19:05:00.367675+00:00 app[web.1]:     at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)
2018-04-06T19:05:00.367677+00:00 app[web.1]:     at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
2018-04-06T19:05:00.367679+00:00 app[web.1]:     at /app/node_modules/aws-sdk/lib/state_machine.js:26:10
2018-04-06T19:05:00.367680+00:00 app[web.1]:     at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)
2018-04-06T19:05:00.367682+00:00 app[web.1]:     at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12)
2018-04-06T19:05:00.367684+00:00 app[web.1]:     at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
2018-04-06T19:05:00.367685+00:00 app[web.1]:   message: 'This Requester has insufficient funds in their account to complete this transaction. Please visit https://requester.mturk.com/prepayments/new to purchase Prepaid HITs. (1523041500385)',
2018-04-06T19:05:00.367687+00:00 app[web.1]:   code: 'RequestError',
2018-04-06T19:05:00.367689+00:00 app[web.1]:   time: 2018-04-06T19:05:00.366Z,
2018-04-06T19:05:00.367691+00:00 app[web.1]:   requestId: 'e32bd760-c71e-4e3b-9c20-04c18b68dfd4',
2018-04-06T19:05:00.367693+00:00 app[web.1]:   statusCode: 400,
2018-04-06T19:05:00.367694+00:00 app[web.1]:   retryable: false,
2018-04-06T19:05:00.367696+00:00 app[web.1]:   retryDelay: 58.36158143271288 }

JobForm: Implement Criteria Knowledge tab

next-task algorithm API design

Currently, the next-task flow (next-task.js file) deal with quality management and the next-task generation logic. We can break it down into:

quality management module that generates initial tests and honeypots.
next task service that generates the tasks (array of items + criteria)
the current next-task.js module will be responsible for gluing everything together: stop condition, quality management, and task generation.

A very quick draft of the next-task service can be:

nextTask(experimentId, workerId, batch=10)

RESPONSE
{ 
    items: [1, 2, 3, 4, 5, 6, ...]
    criteria: [1]
}

The above function needs the following:

A service that given the experiment id returns the available criteria. Available criteria are the ones that have not been applied to all items yet.

getAvailableCriteria(experimentId)

RESPONSE
[1, 2, 3, 4, 5, 6]

A service that given the experiment id, the worker and the assigned criteria, returns a list of items with the current number of votes computed (sorted by votes).

getAvailableItems(experimentId, workerId, criteria, sort) 

RESPONSE
[
    {id: 2, votes: 1},
    {id: 1, votes: 3},
    {id: 3, votes: 4}, 
    …
]

Then the next task service for baseline can be something like:

nextTask(experimentId, workerId, batch=10) {
    let availableCriteria = getAvailableCriteria(experimentId);
    let criteria = randomlyPickOne(availableCriteria);
    let availableItems = getAvailableItems(experimentId, workerId, criteria, ASC);
    let items = pickFirst(availableItems, batch)  // eg, first 10 items.
    return {
	items: items,
	criteria: criteria
    }
}

In this case, for the batch size, we put the max number of tasks we want a worker to solve.

What do you think about this? @mbaezpy @Evgeneus @sphoebs @fabiounitn

FilterTask: rearrange finish and next task buttons

Make it look like the following:

Add columns task start and end to task/test_task records

Filter records related to a requester

Only show records related to a requester. For instance, in the projects DataTable we should only show projects associated with the requester.

WelcomePage: add payment information

The text:

- Your compensation is 0.X$ cents per task, up to a maximum of xx$.
- Some of the tasks are test questions. you will be paid for these as well, unless you miss any of the first T test questions.

Implement text highlighting component

FilterTask: reward widget not refreshing after getting initial test failed response

When a worker fails initial test, the backend returns a response that indicates we finish. The reward widget does not refresh to show cero as a reward. If we refresh the page, it works as expected.

Create quality management module

Quality management module generates initial tests and honeypots.

Job: Implement batched run

The batched run will allow requesters to run the job in batches. In order to do this, the requester should fill in the batch size.

Add batch size text field in the JobForm component.
Modify the JobDashboard so that it displays the current batch number if we are running in batched mode.
Add new states to the JobStatus object: BATCH_PUBLISHED, BATCH_DONE.

JobDashboard: show the computed values based on the parameters

Add a section, after the parameters, that shows the following computed values:

the number of items in the project.
max reward for a worker.
min number of workers, which is based on the max tasks and votes per task parameters.
the min budget for the job (assuming the case where all the workers finished all the tasks assigned to them)

Define API for basic task functions

UI: change "criteria" to "filter"

Add a tooltip to actions in DataTable

Add a tooltip to help the user to know what the action does.

Add stopping logic to the job manager module

Implement stopping logic. This should unlist/stop the HIT on Mechanical Turk. Basically, the task generation service tells us when we have to stop.

WelcomePage: do not show submit button if worker failed the initial test

Define aggregation algorithm API

Define the API for aggregation strategy.

We need to:

define the input for the aggregation API
define the output of the aggregation API

Regarding the output, this will define how we will store the result of the screening job in the database. Using this table of results we will be able to report the outcome in the UI.

Implement incremental attempts: stop and resume buttons

Basically, it should allow forcing the stop logic that #5 proposes. The JobDashboard interface should allow for resuming a stopped job.

When monitoring the HIT in the JobDashboard, when the job is DONE we should make the stop button invisible.

Start new cronjobs if the application restarts

If the application restarts, start new cronjobs for each of the published jobs.

The API returns ID values as text

The solution to this is explained here

Check if we pay for failed honeypot

Make sure failed honeypot is not considered when computing reward

DataTable: implement pagination widget

UI enhancements to FilterTask

Load next task right after the worker submits their answer (remove next task button).
Change the font of the instructions and task headers.
Add a border to the task segment.
Make sure the collapse/expand state of the instructions holds between tasks.

Run the cleaning logic when a worker finishes

JobForm: add post-job survey textfield

Add a text field where the requester can fillin a link to a survey that a worker can complete after finishing the job. This link should be displayed in the FilterTask component together with the finish message.

ExperimentForm: Add checkbox for text highlighting feedback

Add a checkbox to allow requesters to specify whether they want workers to highlight the words/keywords/sentence that helped them to solve the task.

FilterTask: do not show instructions when finished

Add MTURK_MODE config to api

FilterTask: add support for text highlighting

If the requester selected text highlighting feedback option (issue #12), then the FilterTask component should render the text highlighting component after the user submits their answer.

Move hit configuration parameters to JobForm

Move the hit configuration parameters segment in the JobDashboard to the JobForm component.

Jobs: add copy action

The copy action will just show the job creation form with data prefilled based on the record that we copy from.

MaxAssignment should be computed

The MaxAssignments parameter that the createHIT method needs, should be computed instead of defined by the user.

Forms: Redirect to datatable view after save

In each of the form, after a user save their changes we should redirect the user to the DataTable view.

Include project name in Jobs list header

Implement job edit form

Only unpublished jobs can be edited.

Refactor next-task.js

Refactor this so that we can use quality management module and eventually next task service. Rename this module to job manager.

Implement the Project edition feature

The update project method on the delegate should check the files that changed and should delete the records accordingly before creating the new ones. For instance, if we only change the items file, then before reading and creating new items we should delete the existing ones.

We can only edit the csv urls of projects that have no jobs associated.