ok alright so

UserData:

• entirely abstract

methods related to "files"

methods related to raw "data" files

methods about user info?

• string GetUID(string key);
• string GetKey(string uid);
• string Create(string uid, bool forceNewKey);
• void RevokeKey(string key);
• int GetRegisteredCount();
• int GetAllCount();
• string[] GetRegistered();
• string[] GetAll();
• T[] LoadRegistered<T>() where T : new();
• T[] LoadAll<T>() where T : new();

FileSystemUserData:

Basics

• all files are under Server.Settings.UserDataRoot, e.g. ./UserData/
  • string UserRoot => ... = ./UserData/User
  • string GlobalPath => ...= ./UserData/Global.yaml
• Global.yaml only contains a Yaml-serialized FileSystemUserData.Global
  • which is a class with a Dictionary<string, string> UIDs
  • maps users' "key" => "UID"

Basic path/file related stuff

Examples for UID 12345:

• string GetUserDir(string uid) => ./UserData/User/12345
• static string GetDataFileName(Type type) => Celeste.Mod.CelesteNet.Server.BanInfo.yaml
• string GetUserDataFilePath(string uid, Type type)
  • => ./UserData/User/12345/Celeste.Mod.CelesteNet.Server.BanInfo.yaml (?)
• string GetUserDataFilePath(string uid, string name) => ./UserData/User/12345/name.yaml
• string GetUserFilePath(string uid, string name) => ./UserData/User/12345/data/name

implementations of user info related stuff:

• string GetUID(string key)
  • => tries to get UID for key from Global.yaml's UIDs dict, otherwise empty string
• string GetKey(string uid)
  • => Load<PrivateUserInfo>(uid).Key;
• string Create(string uid, bool forceNewKey)
  • load Global
  • check GetKey(uid) and if no forceNew, just return this key (loading global was pointless then)
  • generate new GUID as keyFull with no dashes, take first 16 chars as key
  • repeat while generated key already exists in Global.UIDs
  • Save(uid, new PrivateUserInfo { Key = key, KeyFull = keyFull });
  • save Global back to yaml
• void RevokeKey(string key)
  • load Global
  • if key is in UIDs, remove it, save Global again, Delete<PrivateUserInfo>(uid);
• int GetRegisteredCount()
  • => LoadRaw<Global>(GlobalPath).UIDs.Count;
• int GetAllCount()
  • => Directory.GetDirectories(UserRoot).Length;
• string[] GetRegistered()
  • => LoadRaw<Global>(GlobalPath).UIDs.Values.ToArray();
• string[] GetAll()
  • => Directory.GetDirectories(UserRoot).Select(name => Path.GetFileName(name)).ToArray();
• T[] LoadRegistered<T>()
  • => LoadRaw<Global>(GlobalPath).UIDs.Values.Select(uid => Load<T>(uid)).ToArray();
• T[] LoadAll<T>()
  • => Directory.GetDirectories(UserRoot).Select(dir => LoadRaw<T>(Path.Combine(dir, GetDataFileName(typeof(T))))).ToArray();
• void Insert(string uid, string key, string keyFull, bool registered)
  • only used by CopyTo in each UserData implementation /shrug

Raw (aren't overrides)

• T LoadRaw<T>(string path)
  • => TryLoadRaw(path, out T value) ? value : value;
• bool TryLoadRaw<T>(string path, out T value)
  • out new T() and return false if path no exist
  • otherwise open file and deserialize to T with YamlHelper
  • out that or new() if was null, return true
• void SaveRaw<T>(string path, T data)
  • create path if it don't exist
  • serialize T into path+".tmp"
  • afterwards delete actual path and move tmp to path
• void DeleteRaw(string path)
  • delete file at path if exists
• void DeleteRawAll(string path)
  • delete directory at path if exists

UserDataFile (i.e. yamls...) methods:

• bool TryLoad<T>(string uid, out T value)
• void Save<T>(string uid, T value)
  • => SaveRaw(GetUserDataFilePath(uid, typeof(T)), value);
• void Delete<T>(string uid)
• void InsertData(string uid, string name, Type? type, Stream stream)
  • only used by CopyTo in each UserData implementation /shrug
  • uses private void InsertFileRaw

UserFile (i.e. raw under "data"...) methods:

• bool HasFile(string uid, string name)
  • the implementation suggests this should be a "UserDataFile" thing but I'm 95% sure the implementation of this is simply wrong broken and should be using GetUserFilePath instead of GetUserDataFilePath
• Stream? ReadFile(string uid, string name)
• Stream WriteFile(string uid, string name)
• void DeleteFile(string uid, string name)
• void InsertFile(string uid, string name, Stream stream)
  • only used by CopyTo in each UserData implementation /shrug
  • uses private void InsertFileRaw

other:

• private void CheckCleanup(string uid)
  • used to remove empty dirs after deleting files
• void Wipe(string uid)
  • literally just DeleteRawAll(GetUserDir(uid));
• void CopyTo(UserData other)
  • /shrug
• private void InsertFileRaw(string path, Stream stream)
  • only used by InsertData and InsertFile which are used by CopyTo(UserData)

Observations:

• Just about any UID can have a user directory for itself
• Only registered users set up with a key by Create() will be in Global.UIDs and have a PrivateUserInfo

Just gonna use this for taking notes.

[Edit:] Initial conclusions after looking through most of the non-cmds ChatModule code, actually ChatModule isn't all that big? Tbh I wonder if it would be possible to pull "Chat" and "Commands" apart into two Modules and then rewrite those separately :widegladeline:

ideas/plans

• I still have that really old chat module rewrite I could revive
  • 0x0ade/CelesteNet@main...RedFlames:CelesteNet:chat-module-rewrite
  • 2023-07-12: Now has PRs of new commands rewrite
  • 0x0ade#108
  • 0x0ade#109
• Snip want to put an external lock around BroadcastRawCMD calls to prevent ws message race conditions
• include message types to stop classifying by colors
• might as well plan for client-local chatting with this rewrite too
• introduce more fine-grained spam-controls, like to allow rapid "/tp" and "/e" usage
• Make a /tp that doesn't send any session data and just goes to same room in same map (...could possibly be local??)
• make a /del moderation-command and prefix the last N (30ish?) messages in client chat log with numbers that cycle through; actually make range of numbers 2N so that if you want to delete oldest deletable you don't accidentally delete a brand new message? -> And only show these numbers when /del is typed in; actually just use make completion prompts
• should it be possible to fully hide global chat? Must be blocked from participating in it too. Implementation could be linked with a "forced" mode for moderation mutes or something.
• (Fix code duplication in 0x0ade#113) Maybe ForceSend should be renamed to just be Send and have a boolean parameter with a default value for "forcing", whatever that means (Version increment and Action invoke thingy)

Thoughts about breaking changes:

I'm wondering if we'll need to keep some minimal implementations of "legacy" (i.e. current) chat message structures around. For when new clients might be connected to old servers and still need to at least accept old chat messages?
And new server chat should probably still at the very least detect old style clients and send them motd or an upgrade notice 🤷
Idk how a chat "protocol" upgrade would go down

Existing ChatModule etc.

related DataTypes

DataChat

• fields:
  • DataPlayerInfo? Player
  • uint ID = 0xffffff
  • byte Version = 0
  • string Tag = ""
  • string Text = ""
  • Color Color = Color.White
  • DateTime Date = DateTime.UtcNow
  • DateTime ReceivedDate = DateTime.UtcNow
• Written to transport:
  • (OptRef) Player
  • (uint) Version & ID packed into one value
  • (string) Tag
  • (string) Text
  • (Color, 3 bytes) Color
  • (Date in binary = 64-bit integer) Date
• Read of the DataType sets CreatedByServer = false and ReceivedDate = DateTime.UtcNow
• Filters on Handle & Send for dropping when Text.IsNullOrEmpty()
• Server-only fields:
  • bool CreatedByServer
  • DataPlayerInfo[]? Targets
  • DataPlayerInfo? Target (is a property based on Targets)

Where does DataChat show up

• ChatModule:
  • Handle( ... DataEmote) creates a DataChat purely for the PrepareAndLog, logging emotes
  • Broadcast(...) as called by /bc in-game and chat/chatx wscmd's, plus join/leave messages if enabled
  • SendTo(...) as called by
    • ChatCMDEnv.Send used to send all the command server responses
    • MOTD, Spam-error, WSCMDDissolve
  • passed into PrepareAndLog by running /cc, /gc and /w (=/r)
• FrontendModule:
  • ...
• Client:
  • ...

DataCommandList

• ...

ChatModule

Pretty small besides the Command stuff

ChatModule methods

• Init()
  • instantiate SpamContext
  • instantiate ChatCommands
  • add OnSessionStart to Server's OnSessionStart delegate/event
  • (add OnSessionEnd to existing sessions, I guess on the mythical hot-reload)
• Dispose() -- basically opposite of Init()
• OnSessionStart(...)
  • broadcast "join" message if enabled
  • send MOTD to joining session
  • send CommandList
  • set up SpamContext stuff with error-sender
  • add OnSessionEnd to the session.OnEnd
• OnSessionEnd(...)
  • broadcast "leave" message if enabled
  • undo spam setup stuff
• PrepareAndLog(...)
  • TODO: does a bunch of funny stuff lol
  • called from:
    • DataChat handler here in the module
    • for logging DataEmote as if they were chat if enabled
    • SendTo, see above under DataChat section
    • /gc, /cc and /w to create the corresponding chat messages
• Handle(... DataChat)
  • TODO: Does a bunch of stuff lol
• Handle(... DataEmote)
  • just handled here for the logging part 🤷
• Broadcast(...)
  • see also above under DataChat sections
  • just creates a "server" DataChat message and shoves it into Handle(DataChat)
• SendTo(...)
  • creates a "server" DataChat to send to specific session (passed through PrepareAndLog)
• event Action<ChatModule, DataChat>? OnReceive
• event Action<ChatModule, DataChat>? OnForceSend
• ForceSend(DataChat)
  • uhh something

I always wonder how we even test this whole thing besides manually playing around in it.

I want something that can semi-automatically try and break things. Like maybe change data in packets randomly, see if anything unexpected ever happens.

I made EverestAPI/Everest#506 so that I could run multiple Itch.io clients with separate logs.

Goals/ideas:

• set some flag that puts clients into testing mode
• determine behaviour in some way, maybe simple "script" for what client will do?
  • can I control the game to auto-enter a level?
  • teleport players back and forth between each other
  • send messages into different chat modes
  • random disconnect/reconnect/simulate time-outs?
  • Randomly change bytes in packets either with something like mitmproxy or within our low-level networking code

The last Client release is dated 2022-01-22 on Gamebanana et al?

Latest Server update was after the commits on 2022-02-19 afaict

Server:
0x0ade/CelesteNet@e8e9091...main

Client:
0x0ade/CelesteNet@d5be441...main

Open PRs to be potentially merged:

• Features:
  • 0x0ade#33
    _{- opened 2022-08-21} -- x _{(server-side changes only)}
  • 0x0ade#46
    _{- opened 2022-10-09} -- 💟 _{(server-side changes only)}
  • 0x0ade#26
    _{- opened 2022-02-06} -- 💠 _{(would become client-side changes only after #46)}
• Fixes:
  • 0x0ade#29
    _{- opened 2022-05-24} -- 💠 _{(client-side changes only)}

Closed PRs that have been merged since:

• Features:

  • 0x0ade#22
    _{- merged on 2022-02-14 -- (client-side changes only)}

  • 0x0ade#27
    _{- merged on 2022-04-10}

• Fixes:

  • 0x0ade#23
    _{- merged on 2022-01-22 -- (client-side changes only)}

  • 0x0ade#24
    _{- merged on 2022-01-29 -- (client-side changes only)}

  • 0x0ade#25
    _{- merged on 2022-02-14 --} // is actually live, but janky?...

  • 0x0ade#28
    _{- opened 2022-04-18} -- 💠 _{(client-side changes only)}

  • 0x0ade#30
    _{- opened 2022-06-07} -- 💠 _{(client-side changes only)}

  • 0x0ade#31
    _{- opened 2022-06-11} -- 💠 _{(client-side changes only)}

  • 0x0ade#34
    _{- opened 2022-08-24} -- 💟 _{(Server.FrontendModule only)}

  • 0x0ade#35
    _{- opened 2022-08-26} -- 💠 _{(client-side changes only)}

  • 0x0ade#38
    _{- opened 2022-09-04} -- 💟 _{(Server.FrontendModule only)}

  • 0x0ade#39
    _{- opened 2022-09-04} -- 💠 _{(client-side changes only)}

  • 0x0ade#40
    _{- opened 2022-09-10} -- 💟 _{(server-side changes only)}

  • 0x0ade#41
    _{- opened 2022-09-15} -- 💠 _{(client-side changes only)}

  • 0x0ade#43
    _{- opened 2022-09-22} -- 💠 _{(client-side changes only)}

Messed around with Github API and git-log to find all commits that don't come from these merged branches?...

(Side note: Should we rebase before merges so that the commits don't get spliced into the existing history?)

The stuff I did, just for reference

for pull in 22 23 24 25 27; do 
     curl -s -H "Accept: application/vnd.github+json" -H "Authorization: token $PAT" "https://api.github.com/repos/0x0ade/CelesteNet/pulls/$pull/commits > pr_$pull.txt
done

for pull in 22 23 24 25 27; do
     commits="$(jq -r '.[]|.sha' < pr_$pull.txt)"
     echo "$commits"
done | tr -d '\r' > pr_commits.txt

git log --no-merges --since=2022-01-19 --oneline --no-abbrev-commit | grep -v -f pr_commits.txt

non-merge commits

2022-01-22 18:15:22     2d4bb16         Popax21         Clean up/Fix legacy code
2022-01-22 18:47:41     ea2913e         Popax21         Cleanup context core dispose
2022-01-22 18:58:47     d5be441         Popax21         Fix client disconnecting
2022-01-22 19:05:43     72742a7         Jade Macho      Cleanup
2022-01-22 19:18:16     62e3d7a         Jade Macho      Fix disconnect reason handler
2022-01-30 17:17:47     0328e1e         Popax21         Fix connection race condition
--- the above are probably already in server but not in client?...
2022-03-19 15:02:19     c55d6b3         Popax21         Improved connection disposal race condition handling
2022-07-03 01:05:06     9c1c404         Popax21         Fix UDP socket having incorrect address family