qutecoacoustics / baw-server Goto Github PK

From Raini

I was so curious what is "..." means: it display "?" and popup text "required" only, which doesn't gives information that I expected.
When i about to type in my user name, i am not sure what are the format i need to enter. coz i thought at first this system use user name for the log in, but then i saw email, so next i assumed it must be email address as login as lately most of the login using email instead of name.
Suggestion:
i) remove the popup text or replace the "required" text with the more informational text about the input data
ii) tell user that the login id will be their email address, so they can decide what email address they want to use (like me i make sure it wont be the long email address like my uni email).
iii) if you put the email address as login id, then the password should be after the email address

if the tools have not finished after a set amount of time (maybe 10 - 20 sec), then kill the process. They can take up resources otherwise.

I've got a check for this, this error message will show in the client if the image received has the wrong height.

AnnotationEditor:calculateUnitConversions: The natural height (257px) for image http://staging.ecosounds.org/audio_recordings/40/media.png?end_offset=30&start_offset=0 is not a power of two. The image has been STRETCHED to 256px! ALL MEASUREMENTS ON THIS SPECTROGRAM WILL BE WRONG!

I could fix this, but the effort required is a little prohibitive. Maybe it is easier on the server.

@cofiem said he had code that was already doing this? is it not working?

Request URL:http://staging.ecosounds.org/audio_recordings/40/audio_events/8
Request Method:PUT

Request Payload

audio_recording_id: 40
created_at: "2013-11-07T02:53:23.000Z"
end_time_seconds: 24.98467120181406
high_frequency_hertz: 7192.08984375
id: 8
is_reference: true
low_frequency_hertz: 3488.37890625
start_time_seconds: 20.96761904761905
taggings: []
updated_at: "2013-11-17T13:21:40.000Z"

Response:

audio_recording_id: 40
created_at: "2013-11-07T02:53:23Z"
creator_id: 7
deleted_at: null
deleter_id: null
end_time_seconds: 24
high_frequency_hertz: 7192
id: 8
is_reference: true
low_frequency_hertz: 3488
start_time_seconds: 20
taggings: []
updated_at: "2013-11-17T13:23:35Z"
updater_id: 7

Note all values for start/end/low/high rounded... big badness

It helps to know the exact lat/lng and enter exact co-ordinates.

These fields should be bound to the map (which the existing method of input)

http://staging.ecosounds.org/projects/397/sites/398?page=4

I don't think that column is still necessary, it's been put there to 'debug' the harvester. Consider only displaying the ones that exist, log an error for ones that don't exist to keep track of orphaned audio_recordings.

Assigning the default timezone to the site as Brisbane (10+) as in (e2435cb)[https://github.com/QutBioacoustics/baw-server/blob/e2435cbe76e0088601fa202cab5ab9abb6c7b75d/config/application.rb#L38] is terrible practice.

It was done to meet time constraints and so that old legacy problems were not a problem.

Removing this assumption will affect #22 and #23

Rails needs to route to route to angular site for external links for BirdWalk

From @jwim1

When I select ‘Home’ from anywhere with the Home link in the breadcrumb trail, I see a list of 3 projects (Isabelle Dehaviland, Myrtle Rust and EVB102). First I thought that these were the projects that I owned, but I own other projects (e.g. SERF Acoustic Study), and they do not appear. What is the criteria for projects that appear when I select home? I think for the first iteration, all projects that I have access to (or which are public), should appear. When I select Projects from the menu at the top, I do see all projects. http://staging.ecosounds.org/projects

We probably need to add disclaimers and contact us pages.

• Disclaimers (Github)
  • stored in repo
• Contact Us (Github, Private configuration)
  • general contact us page stored in github repo
  • contact specific settings stored in private configuration
  • email method, no db store
• Open source contributions (Github)
  • Autogen'ed stored in github repo
• Feedback / bug submission page (Github)
  • general layout stored in github repo
  • email method, no db
  • email settings stored in private configuration
  • also provide a link to the Github repo/issues for people that know what they are looking for
• ethics statement (Private only, Github stub)
• relationship to QUT (Private version & Public version)

Anonymous access is read-only access to a project, so that the front page is not empty for people who aren't logged in.

Main site
http://staging.ecosounds.org/projects/2/sites/138
Interface displays a list of recordings with a hyperlink ‘edit annotation’ but there is no indication of time or date. Should have date and start time (ideally a graphical display showing dates with amount of data).

CC @jwim1

Currently it appears projects are sorted by their id.

Sort by name instead.

e.g. http://staging.ecosounds.org/user_accounts/xxx

A user should see this page first when they ask for information about their account - rather than the change account information page

The rails app uses the following format strings to encode a date into an audio file name (here ):

:date => a.recorded_date.strftime("%Y%m%d"),
:time => a.recorded_date.strftime("%H%M%S"),

The majority of the files we have are encoded with a different format string:

 "%y%m%d"
 "%H%M"

Suggested changes

• temporarily check for both formats
• save new files with new format
• at some point create a script to cleanup old format
• eventually remove redundant code
• all file name encoding and decoding should be refactored into a single testable unit

The expected time format for the time fields on the dataset page is currently not shown or hinted at.

Suggestion: add placeholder text to inputs to show time format. Also add help hints...

At the moment we encode the creation date of audio files in their filename like so (with the 10+ time zone):

original_audio/63/631ed7d5-ef52-4aa6-9062-ffdd41256930_100810-1200.mp3

There are 220K files named this way due to legacy reasons.

The associated record for this file stored in the database under recorded_date is

2010-08-10 02:00:00 (UTC)

While the config.timezone rails setting is on there is no problem. However, if we ever want to change the timezone we will no longer be able to reference the bulk of the files.

Suggested tasks:

• Encode new files with UTC dates (disambiguate with Z suffix on time)
• Cleanup old files (convert their datestamps to UTC)
• When checking for the presence of an audio file, check for both the config.timezone variant and the UTC variant

Help the poor tests pass...

From @jwim1

When you are in a Site e.g. http://staging.ecosounds.org/projects/884/sites/887 the context menu on the left displays the ‘Add New Site’ link. This is not intuitive. The Add new Site link should only be at the Project level i.e. http://staging.ecosounds.org/projects/884

@cofiem - that's pretty standard design...?

Needs to use mime type, as very few files have original_file_name set.

From @jwim1

New Dataset:

Tried to create a new dataset from Peregian Ground Parrots project, all sites, no Date and Time filtering, selected ‘At least one’ in Number of Tags and left the others as default. Page crashes.

Result:

Errno::ENOENT in DatasetsController#create

No such file or directory - /tmp/stringio20131205-14480-269ezw.txt.lock

Ruby string IO being used as memory buffer.

This is site is designed to have international visitors. Date formats throughout the site should be consistent and unambiguous.

Suggested format: yyyy-MMM-dd

Currently noticed cases:

• datasets page, Start date and End date

Ultra confusing... I thought I only had access to three... especially since there's a blank spot (blank spots usually indicate an exhaustive listing of what is available).

Get rid of this concept entirely

Or

Show a full list (e.g. 4 items) and show and ellipsis or 'more' link and some kind of explanatory text.

The docs for the Tag API (http://staging.ecosounds.org/doc/tags/create_as_writer__) say:
tag[type_of_tag] required choose from [:common_name, :species_name, :looks_like, :sounds_like]

Is a general tag type allowed? Or can null be passed? Can this be reflected in the docs?

GET http://staging.ecosounds.org/audio_recordings/1956/media.json?end_offset=60&start_offset=30

Response:

Status Code:422 Unknown

{"error":"end_offset parameter must be a smaller than the duration of the audio recording requested: 30.188 seconds"}

Expected:

Status Code: 406

From @jwim1

Tried to play a recording in the annotation editor – didn’t load the audio or spectrogram

http://staging.ecosounds.org/listen/8946

This resolves to 501 on the server

http://staging.ecosounds.org/audio_recordings/8946/media.png?end_offset=30&start_offset=0

It currently uses notes for human-displayed content. Notes is for machine-readable content only.

Browse to:
http://staging.ecosounds.org/projects/1001/datasets

Diagnosed as no HTML response specified for action.

Proposed fix: redirect.

An open question:

• Should we Obfuscate geographical co-ordinates?
• Who should see obfuscated values? (none, read, write, owner, admin)
• To what accuracy should obfuscated values be shown? http://en.wikipedia.org/wiki/Decimal_degrees

Additionally:

• Ensure all geographical client values are properly encoded both in API and on HTML pages
• Double check values in notes fields!

From Shufei,

Problem: this page is really boring. I prefer to see some pictures like birds, koalas……..

“We gather recordings from a wide range of environments, find practical ways to annotate the sounds with information about what's going on in them, and then enable that information to be put to use by ecologists and other parties interested in conservation.” --- too complex, maybe bullet points are better.

With accept: application/json
PUT staging.ecosounds.org/audio_recordings/40/audio_events

with body:

{
    "audio_recording_id": 40,
    "created_at": "2013-11-19T13:06:01.231Z",
    "end_time_seconds": 0.301859410430839,
    "high_frequency_hertz": 11025,
    "id": null, // <-- invalid request
    "is_reference": false,
    "low_frequency_hertz": 10508.203125,
    "start_time_seconds": 0,
    "taggings": [],
    "updated_at": "2013-11-19T13:06:01.231Z"
}

Returns (404):

<!DOCTYPE html>
<html lang='en'>
<head>
<meta charset='utf-8'>
<meta content='IE=Edge,chrome=1' http-equiv='X-UA-Compatible'>
<meta content='width=device-width, initial-scale=1.0' name='viewport'>
<title>AWB</title>
<meta content="authenticity_token" name="csrf-param" >
<meta content="DLVJgjnP5CoCUy00hKMro+i9i/+3XMIIwBgsRzrX2YQ=" name="csrf-token" >
<!-- Le HTML5 shim, for IE68 support of HTML elements -->
<!--[if lt IE 9]>
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.6.1/html5shiv.js" type="text/javascript"></script>
<![endif]-->
<link href="/assets/application.css" media="all" rel="stylesheet" type="text/css" >
<link href='images/apple-touch-icon-144x144.png' rel='apple-touch-icon-precomposed' sizes='144x144'>
<link href='images/apple-touch-icon-114x114.png' rel='apple-touch-icon-precomposed' sizes='114x114'>
<link href='images/apple-touch-icon-72x72.png' rel='apple-touch-icon-precomposed' sizes='72x72'>
<link href='images/apple-touch-icon.png' rel='apple-touch-icon-precomposed'>
<link href='/favicon.ico' rel='shortcut icon'>
<script src="/assets/application.js" type="text/javascript"></script>

Apart from the result being HTML, the session is destroyed... seems a little harsh for a 404. What is the expected behaviour?

I think /listen should route to the angular site even if no id is provided.

POST staging.ecosounds.org/tags

With body:

{
  "tag": {
    "is_taxanomic": false,
    "text": "ad6",
    "type_of_tag": "species_name",
    "retired": false
  }
}

Always fails with the response:

{
"text":[
"can't be blank"
]
}

Seems validation always fails or is obscuring another problem?

GET staging.ecosounds.org/audio_recordings/40/media.json?end_offset=30&start_offset=0

{
  "datetime":"2012-10-12T06:00:00Z",
  "original_format":".wav",
  "format":"json",
  "start_offset":0.0,
  "end_offset":30.0,
  "id":"183ecf8f-0bbb-4b09-a5c1-ca10f00b874f",
  "date":"20121012",
  "time":"060000",
  // truncated for brevity
}

Why is there a datetime and date & time fields?

• Where are temporary files stored?
• Are they cleaned up when a process crashes (i.e. like #31)
• Do we have a setting for this?
• Is it necessary to create some kind of cleanup option on the admin pages

GET http://staging.ecosounds.org/audio_recordings/40

returns: (truncated)

{
    "duration_seconds": "7199.0"
}

expected: (truncated)

{
    "duration_seconds": 7199.0
}

GET staging.ecosounds.org/audio_recordings/40/

{
 "id":40,
 "uuid":"183ecf8f-0bbb-4b09-a5c1-ca10f00b874f"
 // truncated for berevity
}

GET staging.ecosounds.org/audio_recordings/40/media.json?end_offset=30&start_offset=0

{
 "id":"183ecf8f-0bbb-4b09-a5c1-ca10f00b874f",
 // truncated for berevity
}

• Inconsistent naming of the id field
• Media request only includes one id but not the other?

POST staging.ecosounds.org/audio_recordings/40/audio_events

with body:

{
    "audio_recording_id": 40,
    "created_at": "2013-11-19T13:06:01.231Z",
    "end_time_seconds": 0.301859410430839,
    "high_frequency_hertz": 11025,
    "id": null,
    "is_reference": false,
    "low_frequency_hertz": 10508.203125,
    "start_time_seconds": 0,
    "taggings": [],
    "updated_at": "2013-11-19T13:06:01.231Z"
}

Returns:

{
"start_time_seconds":[
"can't be blank",
"is not a number"
],
"low_frequency_hertz":[
"can't be blank",
"is not a number"
]
}

Both validation values are present and numbers... what gives?

Use nicer styles to render the pagination through http://getbootstrap.com/2.3.2/components.html#pagination, e.g. here http://staging.ecosounds.org/projects/397/sites/398?page=4

Pretty obviously the number of tags we get is going to explode once we seed the db... There will be at least a 1000 unique tags.

At the moment a request for 13 tags is 2.75KB. A 1000 of them will be 211KB.

At the moment that estimated 211KB is a pretty small part of our 2.8MB payload.

However, in the future, we might want to support partial searches against the Tag API.

This will tie into the tagging control, so that tags are only downloaded as a user types in the tagging control's autocomplete.

From Liang

can't match the sites with the locations in map

Proposed solution: add labels, tool tips, or change the marker (or all of the above)

Need support for bounding range queries for audio events.

GET staging.ecosounds.org/audio_recordings/40/audio_events??start_offset=90&end_offset=120

Returns (note how none of the end offsets are within the bounds specified by the querystring):

[
{
"audio_recording_id":40,
"created_at":"2013-11-06T03:54:35Z",
"creator_id":7,
"deleted_at":null,
"deleter_id":null,
"end_time_seconds":4,
"high_frequency_hertz":9173,
"id":1,
"is_reference":false,
"low_frequency_hertz":4435,
"start_time_seconds":0,
"updated_at":"2013-11-16T17:06:58Z",
"updater_id":7,
"taggings":[
]
},
{
"audio_recording_id":40,
"created_at":"2013-11-06T03:57:26Z",
"creator_id":7,
"deleted_at":null,
"deleter_id":null,
"end_time_seconds":8,
"high_frequency_hertz":9302,
"id":3,
"is_reference":false,
"low_frequency_hertz":5943,
"start_time_seconds":5,
"updated_at":"2013-11-16T17:06:56Z",
"updater_id":7,
"taggings":[
]
},
{
"audio_recording_id":40,
"created_at":"2013-11-06T04:00:52Z",
"creator_id":7,
"deleted_at":null,
"deleter_id":null,
"end_time_seconds":10,
"high_frequency_hertz":4651,
"id":4,
"is_reference":false,
"low_frequency_hertz":904,
"start_time_seconds":6,
"updated_at":"2013-11-17T13:12:52Z",
"updater_id":7,
"taggings":[
]
},
{
"audio_recording_id":40,
"created_at":"2013-11-06T04:01:41Z",
"creator_id":7,
"deleted_at":null,
"deleter_id":null,
"end_time_seconds":4,
"high_frequency_hertz":4780,
"id":5,
"is_reference":false,
"low_frequency_hertz":1421,
"start_time_seconds":1,
"updated_at":"2013-11-17T12:49:49Z",
"updater_id":7,
"taggings":[
]
},
 // truncated...
]

Adding seconds to a date in SQL is hard to do in a generic way.

Error from postgres:

ActiveRecord::StatementInvalid: PG::UndefinedFunction: ERROR:  operator does not exist: timestamp without time zone + numeric
LINE 1: ...".* FROM "audio_recordings"  WHERE (recorded_date + duration...
                                                             ^
HINT:  No operator matches the given name and argument type(s). You might need to add explicit type casts.
: SELECT "audio_recordings".* FROM "audio_recordings"  WHERE (recorded_date + duration_seconds >= '2013-11-06T06:30:00') AND (recorded_date <= '2013-11-09T11:45:00')

From @jwim1

In sites, there is no link to upload audio:

http://staging.ecosounds.org/projects/884/sites/885

We don't support this feature yet

For example, where do I go from here? I'm at t projects pages... point out what needs to happen next.

From Raini

maybe you can put an information to guide user what they can and need to do next or maybe give an information how to use it. Coz for me i just clicked on any button i saw.

It uses date and time manipulation, and that's not very compatible between databases.

On http://staging.ecosounds.org/my_account/edit, move the cancel my account link at the bottom to the left sidebar. Also consider moving the 'Back' Link into a breadcrumb

From Liang,

For sites that do not have a location, do not show a google map.

E.g. http://staging.ecosounds.org/projects/430 shows a map with the lat/lng set to 0.0, 0.0. Instead the map should be hidden and some kind of same-size place holder should read "No position given" (or something similar).

GET http://staging.ecosounds.org/audio_recordings/40/media.json?end_offset=30&start_offset=0

Returns

available_audio_formats: {mp3:{channel:0, sample_rate:22050, mime_type:audio/mp3,…},…}
available_image_formats: {png:{channel:0, sample_rate:22050, window:512, colour:g, mime_type:image/png,…}}
datetime: "2012-10-12T06:00:00Z"
end_offset: "30"
format: "json"
id: "183ecf8f-0bbb-4b09-a5c1-ca10f00b874f"
original_format: ".wav"
start_offset: "0"

Note of the offset fields are strings and not numbers.

Context duplicate insertion error.

Repo:
GET staging.ecosounds.org/audio_recordings/40/audio_events/10/taggings

Headers:
Accept:application/json
Content-Length:21
Content-Type:application/json

Payload:

{
  "tag_id": 13
}

Response:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8" >
  <title>Action Controller: Exception caught</title>
  <style>
    body { background-color: #fff; color: #333; }

    body, p, ol, ul, td {
      font-family: helvetica, verdana, arial, sans-serif;
      font-size:   13px;
      line-height: 18px;
    }

    pre {
      background-color: #eee;
      padding: 10px;
      font-size: 11px;
      white-space: pre-wrap;
    }

    a { color: #000; }
    a:visited { color: #666; }
    a:hover { color: #fff; background-color:#000; }
  </style>
</head>
<body>

<h1>
  ActiveRecord::RecordNotUnique
    in TaggingsController#create
</h1>
<pre>Mysql2::Error: Duplicate entry &#x27;10-13&#x27; for key &#x27;index_audio_events_tags_on_audio_event_id_and_tag_id&#x27;: INSERT INTO `audio_events_tags` (`audio_event_id`, `created_at`, `creator_id`, `tag_id`, `updated_at`, `updater_id`) VALUES (10, &#x27;2013-11-28 17:06:55&#x27;, 7, 13, &#x27;2013-11-28 17:06:55&#x27;, 7)</pre>

Expected response: json error message