mellowagain / shiro Goto Github PK
View Code? Open in Web Editor NEWHigh performance, high quality osu!Bancho C++ re-implementation
License: GNU Affero General Public License v3.0
High performance, high quality osu!Bancho C++ re-implementation
License: GNU Affero General Public License v3.0
MSVC has a max. amount of macro arguments set to 127, which is only half of requirement set in place by the C++ standard.
BetterEnum's enum_macros.hh
contains macros with up to 258 macro arguments, which causes it to no longer being able to compile under MSVC. These macros are used by country_ids.hh
for the declaration of the enum.
We'll have to workaround MSVC's non-standard conform-ness somehow.
Currently, Shiro does not respect all options sent in the login body.
Login body:
Username
MD5 hash of the password
Version|UTC offset|Display full location|Colon seperated list of MAC addresses which are MD5 hashed|Block non-friend PMs
Current options which are not respected:
Display full location
Block non-friend PMs
Shiro should respect these options.
Side thing to implement since !clear is not for everyday kiddles but !localclear could be
Would locally clear chat of everyone in the current chat the user is tabbed into (i.e. #osu, #english, etc). PM's would be kept.
There is no permission system in place currently for Shiro. This means that everyone, including normal players, are able to execute staff-only commands such as !rtx
or !clear
.
Now that user punishments like silences, restrictions and bans have been implemented in #3, a permission system is high priority to ensure no unauthorized action will be made.
The permission system is built with roles, which each have permissions. Users with multiple roles have their permissions add up.
[Flags]
public enum Permissions
{
None = 0,
Normal = 1,
BAT = 2,
Supporter = 4,
Friend = 8,
Peppy = 16,
//Note: Tournament bit is not sent to or from Bancho in bUserPresences
Tournament = 32
}
Currently, the windows_setup_build.bat
references the following repositories owned by @emily33901:
The plan is to replace these references with the original repository and to apply patches onto the source code of the original repository, thus no longer relying on @emily33901 to host these repositories.
The SQL query we run the first time on Shiro's program startup is malformed and results in a
sqlpp::exception
being thrown as soon as a value is inserted.
MySQL debug: Executing: 'SELECT users.id,users.username,users.safe_username,users.password,users.salt,users.email,users.ip,users.registration_date,users.last_seen,users.followers,users.groups,users.user_page,users.pp_std,users.pp_taiko,users.pp_ctb,users.pp_mania,users.rank_std,users.rank_taiko,users.rank_ctb,users.rank_mania,users.score_std,users.score_taiko,users.score_ctb,users.score_mania,users.ranked_score_std,users.ranked_score_taiko,users.ranked_score_ctb,users.ranked_score_mania,users.accuracy_std,users.accuracy_taiko,users.accuracy_ctb,users.accuracy_mania,users.play_count_std,users.play_count_taiko,users.play_count_ctb,users.play_count_mania,users.country FROM users WHERE (users.id=1)'
MySQL debug: Constructing result, using handle at 0x1aef270
MySQL debug: Accessing next row of handle at 0x1aef270
MySQL debug: Executing: 'INSERT INTO users (id,username,safe_username,password,salt,email,ip,registration_date,last_seen,followers,groups,user_page,pp_std,pp_taiko,pp_ctb,pp_mania,score_std,score_taiko,score_ctb,score_mania,ranked_score_std,ranked_score_taiko,ranked_score_ctb,ranked_score_mania,play_count_std,play_count_taiko,play_count_ctb,play_count_mania,country) VALUES(1,'Shiro','shiro','f52fbd32b2b3b86ff88ef6c490628285f482af15ddcb29541f94bcf526a3f6c7','shiro','[email protected]','127.0.0.1',0,0,0,0,'beep boop',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,'JP')'
terminate called after throwing an instance of 'sqlpp::exception'
what(): MySQL error: Could not execute MySQL-statement: Field 'rank_std' doesn't have a default value (statement was >>INSERT INTO users (id,username,safe_username,password,salt,email,ip,registration_date,last_seen,followers,groups,user_page,pp_std,pp_taiko,pp_ctb,pp_mania,score_std,score_taiko,score_ctb,score_mania,ranked_score_std,ranked_score_taiko,ranked_score_ctb,ranked_score_mania,play_count_std,play_count_taiko,play_count_ctb,play_count_mania,country) VALUES(1,'Shiro','shiro','f52fbd32b2b3b86ff88ef6c490628285f482af15ddcb29541f94bcf526a3f6c7','shiro','[email protected]','127.0.0.1',0,0,0,0,'beep boop',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,'JP')<<
Shiro itself has no documentation apart from the README.md
file, which briefly describes how one would build and start Shiro.
There is no in-depth documentation on how to setup Shiro and its modules, how to get osu! clients to connect successfully and how to contribute. If we want Shiro to be adapted widespread, documentation is key.
I thought about implementing some sort of documentation using Sphinx and https://rtfd.org.
The problem with Read the Docs is that shiro.rtfd.org
is used by some other project already. However, osushiro.rtfd.org
or shiroosu.rtfd.org
is still available. I'm rooting for the first suggestion.
osu! has a built-in tournament client which is used for broadcasting osu! tournaments. It allows viewing of every player in a multiplayer room at once.
To allow for tournaments to be broadcasted on Shiro-based servers, support for this client is required.
Detecting a tournament client is easy. On user login, the client build string will end with tourney
when the osu! client is a tournament client, e.g b20160403.6tourney
. The client build version parser will not throw a exception for the non-numeric string in the version string, I have already accounted for this case.
A tournament client will always login in with all clients using the same username and password combo. It'll also send a special chat join / leave packet indicating the joining of the #multiplayer
channel, without joining the actual multiplayer room.
The tournament client should not be displayed in the global user presence list, however it should be displayed in the spectator list for the clients.
There are some design decision that may conflict with the further implementation of this feature. It may require some massive code base changes:
When requesting leaderboards and beatmap status for a map which hasn't been submitted to Bancho/Shiro beatmap mirror, the map header will return the ranked status ranked
and display a leaderboard.
Trying to submit a score on it will then fail with a JSON exception in the score submission route, because osu!Bancho returns non-JSON for the unsubmitted beatmap meta data.
When the score is submitted and performance is being calculated, the map file has to be downloaded. However due to Unix systems nature, the file cannot be written because the ./maps
folder doesn't exist, hence the performance cannot be calculated. No exception nor error message is shown up. Has to be fixed asap.
Make users able to click on the "Watch replay", to watch the replay associated with a score.
Shiro is nearing the completion of it's last major aspect. We should start versioning Shiro to keep track of updates easier and to allow the implementation of a changelog to be displayed on the frontend, Kuro.
There are multiple version schemes to choose from:
Most widespread in software development, SemVer proposes the usage of incremental version parts. It is suggesting the usage of major.minor.fix
layout. With a major breaking change, the major
part of the version gets upgraded and symbolizes no backwards compatibility. The same applies to minor
and fix
but does not symbolize breaking backwards compatibility.
Resulting versions would look like this: v1.33.7
Simple and straight forward date versioning. It displays the date of the release right in the version number. It suggests the layout of vyearmonthdate.fix
. With each release, the date is updated to the day the new version gets released. If another update is released on the same day, the fix
part gets incremented.
Resulting versions would look like this: v20181009.2
Build numbers are straight forward incremental changes to the versioning system. Instead of having well defined parts that indicate something, the number just gets incremented every time a new update is released. It suggests the usage of buildnumber
as single integer, without any other parts.
Resulting versions would look like this: b5427
A mix between Date and the continues integration build number. This versioning scheme is used by osu!lazer and newer, open-source ppy projects. In principle, it is the same as the Date versioning system but uses the CI build additionally instead of the month and day. It suggests the following layout: vyear.cibuild.fix
Resulting versions would look like this: v2018.205.3
Probably the most straight forward way to name a version. No thinking about if this is a minor or fix, no dates to organize or some other stuff - at the expense of being cryptic and ununderstandable to people who do not use Git as version control system.
Resulting versions would just be a git short commit hash and look like this: 22df1fe
Got another suggestion or even a mix between two suggested (e.g v1.33.7-22df1fe
), feel free to reply to discussion with it and it will be considered as well.
Discussion is open and I'm looking forward to input of any collaborator as well as other people.
osu!catch and osu!mania scores do not award pp at the moment. I suggest implementing these two game modes either using libakame
or their original pp calculators, catch-the-pp
and wifipiano
.
osu_buffer.cc:126: this->bytes.clear();
Exception thrown: read access violation.
**_Pnext** was 0xFFFFFFFFFFFFFFFF.
Inside of stl std::vector
Called from:
bot.cc:143: bot_user->queue.clear();
Allow implementation of Discord Webhooks, which will be triggered on various events such as a new #1
on a beatmap, new #1
on global leaderboards or a new punishment such as silence, restriction or ban.
osu!Bancho displays a image according to a current event, such as the Summer Fanart voting (as seen below):
Adding support for this is easy by sending a out_title_update
(id 76) packet to the newly logged in osu! client. The packet contains a string as payload.
If the payload string is null
, the currently displayed menu image will be hidden.
If the payload string is in valid format (image url|image href, optionally null
), osu! will display the image in the main menu as seen above.
Docker is a great way to ship whole of Shiro and its modules.
I propose the creation of the marc3842h/shiro:stable
docker container which will include Shiro and its modules, to allow for a quick and painless setup.
Sentry.io is a nice platform for error collecting. I propose the implementation of automatic error reporting to Sentry.io in Shiro.
Shutting down Shiro gracefully (by pressing CTRL-C
or sending a SIGINT
to it) will cause a massive delay because the database thread does not exit. Before the shutdown is fully completed, the following message can be seen:
Error in my_thread_global_end(): 1 threads didn't exit
After further investigation it seems that the MySQL thread doesn't exit correctly.
Edit: This only occurs when the application is built in debug mode.
Multiplayer is the only major part of the Bancho server which is still not implemented by any means on Shiro.
Implementation has begun with 7f7a0be. Here is the list of things to do for tracking the progress:
#lobby
) and receive a list of open multiplayer rooms#multiplayer
chat, based on per-room basis.#lobby
, #multiplayer
)On the beatmap rankings, the beatmap ranking timestamp isn't considered together with the score timestamp.
This could create the following scenario:
User a
submits a play on the unranked beatmap. It gets saved in the database because the server owner set save_unranked_scores
to true
in the score_submission.toml
config.User a
(which was submitted 4 days prior to the beatmap ranking).The primitive and simple fix for this would be to check if the score timestamp is smaller than the beatmap's last update timestamp, if that is the case then skip the score.
Allow translation for all messages displayed to the users, e.g using a translation engine such as GNU gettext
. This allows customization for all messages as well as translated messages to be shown to users depending on their location, which we already resolve using IP geo location.
I am opening this issue to discuss what this code style should be.
I propose a style that is based upon what is already used in the code base, along with some additions (that become much easier to carry out after using clang-format).
if(x) return y;
)// From beatmap.cc
this->id = row.id;
this->beatmap_id = row.beatmap_id;
this->beatmapset_id = row.beatmapset_id;
this->play_mode = row.game_mode;
this->beatmap_md5 = row.beatmap_md5;
this->song_name = row.song_name;
this->ar = row.ar;
this->od = row.od;
this->diff_std = row.diff_std;
this->diff_taiko = row.diff_taiko;
this->diff_ctb = row.diff_ctb;
this->diff_mania = row.diff_mania;
this->max_combo = row.max_combo;
this->hit_length = row.hit_length;
this->bpm = row.bpm;
this->ranked_status = row.ranked_status;
this->ranked_status_freezed = row.ranked_status_freezed;
this->last_update = row.last_update;
this->play_count = row.play_count;
this->pass_count = row.pass_count;
becomes:
// from beatmap.cc
this->id = row.id;
this->beatmap_id = row.beatmap_id;
this->beatmapset_id = row.beatmapset_id;
this->play_mode = row.game_mode;
this->beatmap_md5 = row.beatmap_md5;
this->song_name = row.song_name;
this->ar = row.ar;
this->od = row.od;
this->diff_std = row.diff_std;
this->diff_taiko = row.diff_taiko;
this->diff_ctb = row.diff_ctb;
this->diff_mania = row.diff_mania;
this->max_combo = row.max_combo;
this->hit_length = row.hit_length;
this->bpm = row.bpm;
this->ranked_status = row.ranked_status;
this->ranked_status_freezed = row.ranked_status_freezed;
this->last_update = row.last_update;
this->play_count = row.play_count;
this->pass_count = row.pass_count;
Macros like:
// from common_tables.hh
#define object_detailed_struct(name, var, type) \
struct var { \
struct _alias_t { \
static constexpr const char _literal[] = name; \
using _name_t = sqlpp::make_char_sequence<sizeof(_literal), _literal>; \
template <typename T> \
struct _member_t { \
T var; \
T &operator()() { \
return var; \
} \
const T &operator()() const { \
return var; \
} \
}; \
}; \
using _traits = sqlpp::make_traits<type>; \
} \
become:
// from common_tables.hh
#define object_detailed_struct(name, var, type) \
struct var { \
struct _alias_t { \
static constexpr const char _literal[] = name; \
using _name_t = sqlpp::make_char_sequence<sizeof(_literal), _literal>; \
template <typename T> \
struct _member_t { \
T var; \
T &operator()() { \
return var; \
} \
const T &operator()() const { \
return var; \
} \
}; \
}; \
using _traits = sqlpp::make_traits<type>; \
}
clang-format is not great at acknowledging where the user has taken care to indent a section of code in a specfic way. For example:
db(insert_into(punishments_table).set(punishments_table.user_id = user_id,
punishments_table.type = (uint16_t) utils::punishment_type::ban,
punishments_table.time = seconds.count(),
punishments_table.active = true,
punishments_table.reason = reason));
Will get formated to a single line.
There are some solutions to this.
//clang-format off
and //clang-format on
around areas of code that still wont reflow properly.I would suggest that the column limit is a more rigid, less "intrusive" approach.
Overall, usage of clang-format will make the code base more consisent and easier to read, update and expand. Whilst there may be some initial growing pains, this should help out greatly in the longer run.
Further use of githooks / integration with git would allow for automatic use of clang-format on commits - should people not have it installed.
My resulting clang-format file for reference
---
ColumnLimit: '150'
AllowShortCaseLabelsOnASingleLine: 'false'
AllowShortIfStatementsOnASingleLine: 'false'
AllowShortLoopsOnASingleLine: 'false'
AllowShortFunctionsOnASingleLine: 'false'
AlwaysBreakTemplateDeclarations: 'true'
AccessModifierOffset: '-4'
IndentCaseLabels: 'true'
IndentWidth: '4'
Standard: Cpp11
AlignOperands: 'true'
AlignConsecutiveAssignments: 'true'
AlignEscapedNewlinesLeft: 'true'
AlignConsecutiveDeclarations: 'true'
AlignAfterOpenBracket : 'Align'
BinPackParameters: 'false'
BinPackArguments: 'false'
NamespaceIndentation: All
FixNamespaceComments: 'false'
SortIncludes: 'false'
SpaceAfterCStyleCast: 'true'
...
Right now, Shiro is using ip.zxq.co for resolving the geolocation of IP addresses. While this is working good right now, in the long term I want to implement this feature without relying on a web service, using MaxMind's geolocation databases (which are free to use).
System statistics have been implemented on every platform except Darwin, where every method will throw a std::runtime_error
when invoked. As Shiro invokes the native::system_statistics::init
method on startup, this renders Shiro unable to start on Mac.
The file that requires implementation can be found here: https://github.com/Marc3842h/shiro/blob/master/src/native/mac/system_statistics_mac.cc
As of right now, Shiro and Kuro have both no way of communicating with each other except throught the SQL database.
I suggest implementing inter-process communication using a shared memory region, using sockets or with a new database engine like Redis.
This allows the web frontend (Kuro) to communicate reliably with the Bancho (Shiro) to sync up various actions such as username change, restriction, rollbacks etc.
Recently, @cyanidee and I discovered on our development server that some beatmaps have duplicate entries in the beatmaps
table.
While all having a different id
primary key, the beatmap would exist with the same metadata as all other entries, thus being duplicates.
The example we found was with the beatmap MOMOIRO CLOVER Z - SANTA SAN [X-Mas] with md5sum f9918c819e6d5dc65c206efdaddc9285
:
In total, there were 33 entries in the database all pointing to the same beatmap. This may be caused by a race condition and requires further investigation.
These are both technically keywords and mysql outright rejects them in queries, whilst mariadb accepts them.
consolidates a very common code pattern along with allowing me to more easily insert curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
for testing so on so forth.
Relay osu! client updates from bancho thru shiro without having to connect to bancho directly.
I can't really explain it better, I am going to attach two examples, how the file should look like, and how it looks like on Windows:
Windows:
osu file format v14
[General]
AudioFilename: audio.mp3
AudioLeadIn: 0
PreviewTime: 90062
Countdown: 0
SampleSet: Soft
StackLeniency: 0.4
Mode: 0
LetterboxInBreaks: 0
WidescreenStoryboard: 1
Correct one:
osu file format v14
[General]
AudioFilename: audio.mp3
AudioLeadIn: 0
PreviewTime: 4664
Countdown: 0
SampleSet: Soft
StackLeniency: 0.7
Mode: 0
LetterboxInBreaks: 0
WidescreenStoryboard: 0
It doesn't happen on our development server, only happened to me on my Windows machine. Assigning @emily33901, because it seems its only Windows' issue.
Even PHP deprecated it, why don't we?
@cyanidee and me have decided to stress test Shiro. We have used the following:
Shiro was able to correctly process all 102 login tries for about 20 seconds before resulting in a segmentation fault.
This means that Shiro was able to process (102 * (20 * (1000 / 2))) = 1'020'000 login retries within 20 seconds perfectly without any hiccups. The CPU and RAM usage was not measured as my system monitor didn't update fast enough to notice any major difference to before hitting.
Now this is already a great number considering Ripple, the most widespread cho-protocol implementation server, crashes with just 5 successful tries every second within 20 seconds (tested by @cyanidee).
To prevent further crashes by running out of memory, implementation of rate limit is suggested.
Issue for this exception. Needs further investigation.
2018-11-12 22:56:57.977 ( 183.605s) [ ] routes.cc:51 ERR| A exception occurred in root::handle: Dynamic exception type: class std::out_of_range std::exception::what: invalid vector<T> subscript
The host of a spectator match doesn't get the #spectator
chat removed after the last spectator stopped spectating the host.
@mempler's Banana client has added Shiro in its server list.
The Banana client routes its traffic through Cloudflare because it uses the actual domain, *.shiro.host
, to connect - unlike hosts redirection with the osu!Stable client.
This causes the X-Forwarded-For
header to be in Cloudflare's format, which directly conflicts with the way we use it for geo location resolving.
In the error logs, the following can be seen:
2018-10-19 10:55:09.281 (5262.213s) [ AACAE700] root_route.cc:58 WARN| Received POST from xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx, xxx.xxx.xxx.xxx without osu! user agent.
2018-10-19 11:01:48.369 (5661.301s) [ AACAE700] geoloc.cc:66 ERR| Received invalid response from Gitea API: Please provide a valid IP
2018-10-19 11:01:48.371 (5661.304s) [ AACAE700] user_manager.cc:35 0| User marc3842h logged in successfully.
There are various beatmap mirrors out there. I propose that the beatmap mirror used for osu!direct can be chosen from a config file, e.g direct.toml
.
The following mirrors should be available to choose from:
There shouldn't be any permission problems as Shiro itself is using it's own beatmap mirror. The server owners themselves are liable for what beatmap mirror they use and if they have the permission to do so.
The multipart/form-data
parser parses file content incorrectly and causes the resulting LZMA data stream of replays to be corrupt. This causes the created replay files to be corrupt and unreadable by the osu! client.
lol
Currently, Shiro only supports MySQL. This is not as customizeable as I want it to be.
I propose that we add support for the following two database systems, in addition to current MySQL support:
Thanks to the flexibility of sqlpp, only a few parts (namely connection init) of the code need to be changed up for this to work. There is also the ability to use odbc, which I still need to evaluate if it's of value or not.
When requesting a leaderboard for any beatmap which has existing overwritten / failed scores, the beatmap will display a invalid count.
A image says a thousand words so let me show the issue:
This most likely has to do with the way we're filtering out leaderboard scores. It probably hasn't been updated with the ability to overwrite scores or accept failed scores.
As mentioned in #40, I am opening this ongoing issue to investigate and track the state of thread safety within Shiro.
It would be helpful for my sake that issues that are potentially caused by thread safety or lack thereof could mention this issue so that I can keep everything in 1 place.
When #19 is resolved, we are most of the way there to having full support for windows. However there are still some problems that are outside of shiro that I need to look into
I propose the implementation of a impersonation mode that will allow a staff member (to be exact, a admin) to impersonate a user both in Bancho and on the frontend.
This will allow a admins to do any actions on behalf of a user, including submission of scores as well as sending chat messages.
The privacy concern is not really valid regarding this suggestion as administrators can already edit everything all over the place, anyways.
There are some situations in the life of a server where impersonation of another user may be the key to getting a rule breaker banned.
This is inspired by Gitlab's impersonation mode:
Shiro is unable to be compiled correctly using MSVC or Mingw, thus is unable to run natively on Windows.
This has to do with usage of reserved keywords as well as major third-party library dependency issues on Windows non-existing library management.
When resolving this, it is suggested to additionally implemented a CI to ensure the correct building of Shiro on Windows going forward, I propose using AppVeyor or VSTS.
Hidden players have the ability to spectate players without showing up in the osu! Spectator list for the host.
This causes the host to receive a #spectator
channel without having any spectator visible on his client.
I suggest implementing that hidden users receive the #spectator
chat while the host doesn't (as long as the all spectators are hidden players).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.