Split AudioSet ontology into groups for crowdsourcing validations about freesound-datasets HOT 7 CLOSED

I see different things we could do:

• organize the table of categories by groups according to the hierarchy.

• ask raters to choose several categories to rate (and then start rating on them one by one). We could have a user page with some pending tasks that he chosed to do.

• recommend categories according to their previous ones. Again a user page might be interesting for this.

from freesound-datasets.

Would this user page be different for each dataset? and for each task in a dataset?
I'm just pointing out these things to trigger discussion. Let's also think about a simpler version of this. I like the idea of users choosing in which parts of a taxonomy they feel comfortable contributing more.

from freesound-datasets.

These small groups are taxonomy/ontology and task dependent. Note that these are not dataset dependent, since we can have several datasets related to the same taxonomy/ontology. And also note that these groups are task dependent, however for most tasks these will be equal.

Therefore, in order to start, we can focus in a basic implementation where we assume using a single taxonomy/ontology where tasks are similar. I think this assumption is reasonable among the tasks we envision (#35 ).

from freesound-datasets.

We could add one step for "Choose a group of category" before "Choose a category".
When a user chooses a group, he is redirected to another page where he can choose for a specific category of the group.

This is making the user aware of the related categories, but it is not forcing him to stay in the group and validate only sounds from this group. He is free to go back to the "Choose a group of category" page and go for other groups.

Without any more "complicated" changes (e.g. adding a user page), I think we cannot really do better.

from freesound-datasets.

For now, it has been suggested to adopt a simpler strategy, based on assigning single categories (and not groups) to a rater. The motivation to create and propose groups of related categories was to gather consistent annotations across similar categories (typically siblings). But maybe we can get the rater to consider them without the need for groups generation, by putting more emphasis in showing the context. In this way, the user chooses one category and, in the training phase, the context (parent(s) and siblings) is shown.

Why so? It was discussed that creating groups has some issues:

• may be subject of criticism as it involves decisions with unclear criteria
• makes more sense with bigger groups that include most of the siblings. But this is too heavy for crowdsourcing. Smaller groups would not show much of the context.
• since groups would be small, there would be 100+ groups, ie, troublesome task to do

A possible procedure is as follows. After guidelines are over, a new page shows up with:

• an introductory question at the top, eg, “Select a category:”
• 2 tabs:
  1. Suggested Categories (default)
     • Sorted according to predefined numeric score (#36)
  2. Of your choice
     • Ask user to advance through the first 2 levels of ASO (3 if majority of siblings are also parents) based on his/her interest
     • Present categories sorted according to score (same as in i, but within the subset of categories after previous step)

At this point (by either way) we will have a list of sorted categories for the user to pick one. By selecting one, the user is redirected to a new page which is part of the training phase, when not only the selected category but also its siblings and parent(s) are listed in order to show the context.

from freesound-datasets.

Update on the procedure, @ffont :

After guidelines are over, a new page shows up, horizontally split in 2 parts:

• Left side: "What category would you like to validate?"

  • Ask user to advance through the first 2 levels of ASO (3 if majority of siblings are also parents) based on his/her interest
  • Present list with resulting subset of categories

• Right side: "Our priority list:"

  • Present list of categories that require validations the most

In both cases:

• the presented list is composed of categories are sorted according to predefined numeric score (#36)
• the first category is pre-selected but it is possible to choose a different one

By clicking Continue, the user navigates to a different page that shows the explicit part of the training phase.

from freesound-datasets.

This has been implemented in #40

from freesound-datasets.