Revision of how multi-sentence messages are handled when more than one decoder instance is configured for the same stream

Currently more than one decoder can pick up and process encoded messages from the same stream. Redis was being used to store the fragments of a message before processing them all as a unit. There is however a problem that might arise if the stream is too fast: One decoder instance might be still setting the fragment on the redis instance for the message while another decoder already started to work on the next fragment, asking for the rest of the fragments while they're still not there and throwing a PreviousFragmentsNotPresentForMessageId or NumberOfRecordedFragmentsDoesNotMatchMessageFragmentCount error.

As such both the decoder and the station applications need to be reworked if we're ever going to allow more than one decoder instance to process the same encoded messages stream:

Station changes

• Logic to accumulate all the sentences of a message before sending them into the worker queue for processing will need to be created, that logic will have to have into consideration the source's ip address/host and the source id when storing them;
• When the first sentence for a message id on a specific source is encountered and it's not starting from the beginning, all sentences related to that message must be discarded/ignored;
• If encountering the first sentence for a message id on a specific source that had already previous (and incomplete) storage that storage must be cleared for the new fragments to accumulate.

Decoder changes

• The decoder application will now need to be rewritten to support receiving multiple newline-separated sentences from the worker queue at once.
• With this change in place it might make sense to remove the redis code and revert back to the hashtable mechanism that was used before.

Allow the user to select the map type they prefer.

List of aids to navigation view

As a user, I want to be able to view the list of atons available, so that I can get an overview of all the atons stored in the database.

Sections

Filters section

Fields

• Search input

A search input that should search freely on the following fields:

• Name
• MMSI

• Search by country
• Search by aton type

Actions

• Search
• Clear filters

Grid section

A grid with the following fields:

• Country/Flag
• Name
• Aton type
• Position

The grid should be paginated.

Other specs

• Add an action on the navbar pointing to this view.
• Filters and pagination information should be kept in query parameters as they are changed.
• A loading indicator should be present while data is being fetched.
• Objects should be filtered when querying the database to make sure only atons are coming (using the MMSI internally, maybe).

To make a decoder instance truly stateless it might make sense to store accumulated payloads outside of the decoder process, for instance in a redis or memcached server.

Care should be taken when a source id is not supplied in an AIS sentence. Maybe it might be a good idea to use the queue name has the hash key.

Create a dockerfile for decoder

• A view to edit the account information should be created, to allow the user to update their information.

Privacy policy view

• With the possibility of user registration, there should be some sort of privacy policy view specifying things like what happens to the user data.
• This view should be linked somewhere in the user registration view.
• This view should be linked somewhere in the navbar/footer.

Create a view talking about the project with a link to its repository and a form of communication.

Stations should be associated to a list of source addresses.

• This application should query the table with the current information for all shipcraft;
• It should allow to filter for a timespan, so that older entries can be excluded.

When the decoder has an exception mid-message decoding the received message is kept in the work queue stream, being picked up over and over.

Coordinates, zoom level, etc. Maybe in url query parameters.

User accounts

As a user, I want to be able to create an account on the website, so I can access protected features and save personal information.

• Registration view
• Login view
• User roles view (there should be support for administrators, moderators, or other roles)

Actions in the navbar:

• Register: takes the user to the registration view
• Login: takes the user to the login view

List of vessels view

As a user, I want to be able to view the list of vessels available, so that I can get an overview of all the vessels stored in the database.

Sections

Filters section

Fields

• Search input

A search input that should search freely on the following fields:

• Name
• MMSI
• IMO
• Callsign

• Search by country
• Search by ship type

Actions

• Search
• Clear filters

Grid section

A grid with the following fields:

• Country/Flag
• Name
• Type
• Destination
• Position

The grid should be paginated.

Other specs

• Add an action on the navbar pointing to this view.
• Filters and pagination information should be kept in query parameters as they are changed.
• A loading indicator should be present while data is being fetched.
• Objects should be filtered when querying the database to make sure only vessels are coming (using the MMSI internally, maybe).

• This application should read decoded messages data that comes from the decoder outgoing exchange and persist them in a database.
• There should be a single table with a primary key by MMSI that stores all the fields every message can contain.
• For history logging purposes maybe a copy of that table with a timestamp can be used as a sort of ledger, this time without a PK by MMSI.

Add support for receiving data via-UDP.

• Create mechanisms to recover when a connection fails on the decoder. These can be inside the decoder proper, or on the docker compose file.
• Consider the possibility that one of the connections (incoming or outgoing) can fail while the other is still alive.

Vessel view

As a user, I want to be able to view the information of a vessel, so that I can get an overview of the data related to a vessel.

Information that should be included:

Static information

• Name
• Type
• MMSI
• IMO
• Call sign
• Flag
• Dimensions

Position/Status information

• Latitude
• Longitude
• Time when the position was received
• Navigation status
• Speed/Course
• Draught
• Destination
• AIS source
• Map component with the ship in focus

• One possibility could be to use the TAG block, but it would break if an ipv6 address were to be used (':' is used in the TAG block syntax). That character would need to be replaced or maybe the whole address would need to be hashed somehow.

As a user, I want to be able to delete my account thereby removing all data related to the account.

There should be an action to do this in the account edit view.

Allow for decoder configuration of these parameters through environment variables:

• Amqp connection url for incoming AIVDM packets;
• Exchange name for incoming AIVDM packets;
• Amqp connection url for outgoing decoded messages;
• Exchange name for outgoing decoded messages;

Terms and conditions view

• With the possibility of user registration, there should be some sort of terms and conditions view specifying things like photo usage or more.
• This view should be linked somewhere in the user registration view.
• This view should be linked somewhere in the navbar/footer.

The station application currently supports reading or receiving data via TCP or UDP. To allow for more flexible testing procedures a third supported protocol should be introduced to allow the messages to be read from a file.

• In order to be GDPR compliant, a mechanism to allow users to request all data collected by the website should be introduced.

• This could mean having something in the backend to consolidate sources or have something in the frontend where various sources could be added.

Aid to navigation view

As a user, I want to be able to view the information of an aton, so that I can get an overview of the data related to an aton.

Information that should be included:

Static information

• Name
• Type
• MMSI
• Dimensions

Position information

• Latitude
• Longitude
• Time when the position was received
• AIS source
• Map component with the object in focus

As a user, I want to be able to view photos that are associated to an object on the map view and the vessel view, so I can have a visual indication of what an object looks like.

• Vessel view should include the main photo next to the name
• Vessel view should be modified to include a carousel of photos
• Aton view should include the main photo next to the name
• Aton view should be modified to include a carousel of photos

• Number of transmitted objects per second;

User photos view

As a user, I want to be able to see all photos uploaded to the website.

As an administrator, I want to be able to see unmoderated photos and allow them to be shown for the general public.

Filters section

• Free search input by name, mmsi, imo, callsign
• Username/actual name search input
• Type of object (vessel, aton, etc.)

If the logged-in user has a role that allows it, the following filters should be present:

• Photos not yet moderated

Grid section

• Photo
• MMSI
• Object name
• Username/actual name
• Date

Actions

If the logged-in user has a role that allows it, the following actions should be present:

• Accept (not visible if photo has already been accepted)
• Deny (not visibile if photo has already been denied)
• Set as main picture (should automatically accept the photo)

This grid should be paginated.

Other specs

• Add an action on the navbar pointing to this view.

• This application should be listening for incoming decoded messages through the amqp connection and forward them through websockets so that it can be consumed in a frontend.
• The possibility of this being the same application that offers the REST endpoint should be considered.

Photo upload

As a user, I want to be able to upload photos for an AIS object, so there can be a visual representation of an object in its specific page.

• There should be a button in the AIS object (ship, aton, station, etc.) that allows for this upload.
• There should be a limitation on the size of the photo sent.
• Only registered users should be able to upload photos.
• Photos should be sent for moderation before being visible to the rest of the community, and only users with a specific type of role should be able to approve or reject the photos.

• A static account profile view should be created containing the basic information of a user.
• This view should be linked whenever there's user created content on the site (example: photos).

As a user, I want to be able to search objects directly from the navigation bar.

• The search input should allow to freely search objects by name, mmsi, imo, callsign.
• An icon for a looking glass should be inside the input.
• An icon with an 'x' should be present to delete the filter.
• Searching should start a few milliseconds after the user has stopped typing.
• A list of filtered items should appear below.
• Each item should include a name, type, flag/country as appropriate.
• Clicking on the item should take the user to the page for the object.

Live map filters

As a user, I want to be able to filter the live map, so that I can fine tune my view of the map.

Filters

• Vessel type
• Country
• Draught
• Length
• Beam
• Status
• Speed
• AIS station
• Type of object

These filters should be grouped in sections.

• It might be more effective to save messages in bulk instead of the current method where messages are saved one by one.
  In order to do that it might also make sense to employ some reactive extensions and collect messages in a certain parameterized number of milliseconds before proceeding with inserting them on the database.
• Change methods to async

As a user, I want to be able to see a ship travel history through time.

• Use standard logging methods in decoder;
• Should errors simply be printed out to stderr and/or persisted somewhere (to debug messages that weren't correctly decoded, for instance)?
• Should backend applications return an error code when fatal errors happen?
• More debug/info logging should be done;

Introduce a timeout in the connection and the data reception in the station application