hubmakerlabs / replicatr Goto Github PK

currently relay is not being sent or not accepting follow lists and relay lists from clients, this needs to definitely be working

the existing encoder for storage in the local storage database (ie badger) is extremely silly and is to be replaced with simple gob encoding of the raw event data and eliminating the intermediate type

it is in draft still but NIP-66, relay metadata and monitoring is relevant to the ACL system:

https://github.com/dskvr/nips/blob/this-is-the-last-time/66.md

event kinds 30066 and 30166 seem like they can be self-published automatically by relays and contain useful data to clients implementing and publishing reports on this protocol feature

not high priority but would be good to implement this feature by end of MS3

requires the creation of a relay identity which represents automated reports and potentially later on could be part of a chat based interface

there is a bug in the eventstore query function that seems to stall the websocket and cause the client sending events to time out the socket context (7 seconds)

making a note for future reference of a possible feature addition

the central idea is adding a new index that adds a reference count to events so that it can be seen how many events are referring to another event... this would probably need to be some kind of weighted and complex score, so that it not only evaluates the direct references, which will be replies, but also, the number of indirect references, to provide a weighting to evaluate whether an event should be evicted preferentially to others of similar age in terms of how recently they are accessed

currently the prune strategy purely is based on Least Recently Used LRU style, but a relevance score like this could help avoid pruning out data that has high relevance

a candid specification for the (envelope) datatypes is required

in order to facilitate immediate propagation of new events related to existing subscriptions, the IC canister needs to have a short term storage table containing all active filters currently in open subscriptions, and when new events are stored, the filters are applied to them and if they match, logged in relays should be sent notifications of a match

this will also require some kind of filter identification scheme, probably a truncated hash of the receiver will suffice, and this key needs to be linked in to the relay side so that when the notifications are received the relevant subscriber is sent the matching event

by accident due to the debugging process for the race conditions in the relay interface and eventstore for replaceable events code that used to find and delete it has been commented out, as this extra step often triggered a race condition

a simple test shows that a kind 0 event being updated currently now adds the event but the order of the results and the number that come back from a query without a limit set is undefined behaviour

in order to make this work as clients expect while exposing the ability to return the history of replaceable events, as most replaceable events are intended to be one per pubkey, the following changes are needed:

• before applying the limit filter, sort the results by timestamp in reverse chronological order so newest is the first item and the one usually returned - this is already how the priority queue heap works in the query process, but it is being refactored into a struct instead of slice type
• default any filter with a request for replaceable events to return 1 if no limit is specified
• sort query results in descending created_at timestamp order, newest first, there isn't any way to specify this and clients usually will sort it but why make work for clients that are usually less efficient
• investigate NIP-66, which has some relation to returning replaceable event history

create indexes for search acceleration

• keyword index
• url index
• hashtag index

these will require new data sets in the IC to store tables of references between search keys and events

for now these should be sufficient to improve search capability and effectively implement some level of DVM capability

creating relay accounts for access to a canister

• signatures and verification for relay identities
• challenge/signature auth flow
• resource use accounting (cycles/data storage)
• master relay/client relays relations

an important element of the required features for this relay is to enable it to be used in a purely private environment - given a client that actually respects users relay choices...

for this, it is necessary that the relay has a private internal events storage and a class of access permission of administrator and user

the administrator will be able to push administration access control events to the relay which will alter the stored ACL that also needs to be implemented

so, a checklist:

• NIP-42 access control to privileged events (mainly private messages and application specific data)
• ACL data structure format and role definitions
• create configuration file and load/save
• administrative CRUD event types
• implement actions in response to CRUD event types
• add this functionality to the CLI client
  • merge nak features into postr for this
• add enforcement code to the default filtering code for the relay to auth-wall the access to the relay according to the role definitions

currently, queries repeatedly decode and decode again events that are hot in the cache, especially recent ones that are sent out in open subscriptions of current websocket connections

it would make a lot of sense to eliminate the duplication of both memory and processing required for this data by changing the event decoding library into a freelist that allows avoiding unbounded spikes of memory use and unnecessary re-decoding of the binary data in the database for events that are being heavily accessed concurrently

may not be so good for its lack of range scanning... investigate

code compiles but needs to be tested to work

using new client to send messages and get debugging info back

with the candid specs created for the data types, a simple library to encode and decode them into Go runtime variables is required

design several data use management strategies, pruning by age, keeping track of last access times of events, and setting low- and high-water marks

with a working replica, can now implement the IC backend

the storage of events on the badger side should remain less than the IC but for reasons of efficiency in both time and cost of access to the IC the badger event store needs to have cache management strategies to contain the age and size of the database, as well as to immediately store everything in parallel with the IC - IC's main purpose is propagating events to multiple relays

a candid specification for the (filter and event) datatypes is required

write a test suite to make sure that the agent and canister access is working correctly

there is the beginnings of an ACL implementation, and a chat based control interface with workaround for clients without NIP-42 - this needs to be completed

it is already quite simple to feed a collection of events to a relay, half of the code already exists in the export/import and ingestr but to cleanly pull the entire database over the network while hot will be an important disaster prevention feature

the owners prescribed in the relay configuration will be able to send a request with an empty filter that will return every event in the database

a small tool that sends such requests and receives the results and optionally writes to .jsonl file, and can also read a .jsonl file and push all of the events in it to the relay while it is live, in both directions using the owner authentication to permit this backup/restore functionality

this is not high priority but should be completed for MS3

the relay will monitor events to catch kind 4 messages with the relay's public key as the recipient, and respond to them

this is a less complicated way to implement user access control, owners and administrators can just issue commands, when you first message the relay or message it with /help it will print out the current command options