leonhard-s / ps2-map-listener Goto Github PK

The current version of this component has been running for about a week now. During this time, several network stability issues arose.

These are more likely to be issues on Daybreak games' end, but we still must handle them one way or another. If successful, these mitigations will be pushed to the Auraxium wrapper itself, keeping the listener app as clean as possible.

Exceptions found

• ConnectionAbortedError raised upon attempting to reconnect after a dropped connection.

  ConnectionAbortedError: SSL handshake is taking longer than 60.0 seconds: aborting the connection
  

  This is our websocket giving up on trying to establish the SSL connection to the API due to the API not responding.

• getaddrinfo failed while trying to reestablish a dropped connection.

  socket.gaierror: [Errno 11001] getaddrinfo failed
  

  This is the websocket erroring out trying to find out if SSL/websocket connections to the API endpoint are even possible.

• Semaphore timeout period expired

  OSError: [WinError 121] The semaphore timeout period has expired
  

  This exception happens within asyncio, but it still occurs during the websocket reconnect block. I will have to look into how this exception could come about; this is likely a symptom of an issue on Auraxium's side, though it may be sparked by the same stability issues as the other errors.

Workaround is to just list all required servers manually.

See cooltrain7/Planetside-2-API-Tracker#131 for details.

The backend occasionally attempts to insert duplicate blips into the database.

This occurs in bursts and exclusively with BaseControl blips, with long stretches of normal operation inbetween.

I am unable to investigate this further as I am out of office, but here is the related traceback:

This could have multiple causes:

• The PS2 websocket API is sending duplicate events
• More than one event listener is active at the same time (unlikely)
• There exist combinations of FacilityControl events which are unique on the PS2 side but result in the same blip being generated
• There are map states in which bases switch ownership more than once instantaneously

In the last case, we would need to find a mitigation strategy to find which of the two events is correct.

The listener does not control what data is processed. It receives the list of servers to track from the database, then it just throws out blips until it breaks or is shut off.
Crucially, it is not supposed to hard-code ID filters locally.

This leaves two options - either we check in with the database every ~5 minutes to see which IDs are valid (thus filtering on the listener's side), or we just throw them at the database, catch the foreign key violations, and handle those.

The latter is what the code is currently doing. If we stick with this, logging should be improved so we can get a list of what unexpected IDs we've been seeing, how often, and so on.

Dynamic zone IDs and untracked continents are currently filtered out when attempting to fetch them from the DB. This is not necessary as we can determine such zones from the zone ID.

The listener should fetch the whitelisted zone IDs upon launch and discard any events not in that list as part of the event handling logic, not by throwing PK violations at the DB.