googleforgames / open-match Goto Github PK
View Code? Open in Web Editor NEWFlexible, extensible, and scalable video game matchmaking.
Home Page: http://open-match.dev
License: Apache License 2.0
Flexible, extensible, and scalable video game matchmaking.
Home Page: http://open-match.dev
License: Apache License 2.0
Wondering?
Supporting cockroachdb would allow clustering.
Go modules was launched in Golang 1.11 and have matured in 1.12 and is being pushed as the new way to manage dependencies within golang.
What's great about it is it's integrated into the golang tool and incorporates all the learnings of the many dependency management tools go has had over the years.
I think the main benefit here will be improved reproducibility of builds since all dependencies are pulled at predefined versions.
It also looks like Agones is moving towards Go modules as well. googleforgames/agones#625
The watcher functions in internal/statestorage/redis/redispb/matchobject.go and internal/statestorage/redis/redispb/player.go are both currently 'query every 2 seconds' and are going to generate a lot of the overall redis load on OM. It would be better to have them set up with (configurable) exponential backoff & jitter. There are a few implementations of exp bo + jitter as golang modules, we need someone to evaluate them and then test the one we pick.
I noticed that the implementions of (*frontendAPI).DeleteRequest()
and (*frontendAPI).DeleteAssignment()
in cmd\frontendapi\apisrv\apisrv.go
are identical. I am wondering what is the difference of these two grpc methods:
service API {
rpc DeleteRequest(Group) returns (Result) {}
rpc DeleteAssignment(PlayerId) returns (Result) {}
}
message Group{
string id = 1; // By convention, string of space-delimited playerIDs
string properties = 2; // By convention, a JSON-encoded string
}
message PlayerId {
string id = 1; // By convention, a UUID
}
Group.properties
is not used in DeleteRequest()
.
This should probably be added in deletePlayer
as another async call after the call to playerindices.DeleteMeta
on line 180. Otherwise could get into a situation where the player has deleted their object via the frontend API but still can't get in a game (as something put them on an ignorelist that never got cleaned up).
If someone wants to submit a PR it can go in as a bugfix and make 0.3.5, otherwise I'll put it in 0.4.0 and backport it later.
Have to establish if every scenario we currently want to call backend.DeleteMatch()
in can tolerate deleting all players in rosters of that match from all ignorelists, and if so, implement that.
The matchmaker_config.yaml has grown to be a very large file and Open Match is a set up different servers with different context. We should consider splitting up the configuration so that each server is only provided the context necessary to run properly.
This should improve reliability because it'll force users to have multiple ConfigMaps for each configuration so if there's a bad config change it'll only impact a subset of server types instead of all of them.
Ran into some issues building Dockerfile.mmforc. Looks like: grpc/grpc-go#2181
Step 16/17 : RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo .
---> Running in 209353b544c8
# google.golang.org/grpc/internal/channelz
/go/src/google.golang.org/grpc/internal/channelz/types_linux.go:41:15: undefined: unix.GetsockoptLinger
/go/src/google.golang.org/grpc/internal/channelz/types_linux.go:44:15: undefined: unix.GetsockoptTimeval
/go/src/google.golang.org/grpc/internal/channelz/types_linux.go:47:15: undefined: unix.GetsockoptTimeval
We currently don’t do vendor-locking. We could update the dockerfile to checkout the particular commit for golang.org/x/sys/unix
like it does for k8s.io, but I think we should go ahead and bite the bullet and investigate doing dependency locking.
We should have an executable binary that can basically run Open Match as a minified version for testing in other languages. It should be something simple that users can shell out to and kill it via HTTP.
$ ./minimatch -grpc_port 10000 -http_port 10001
Running Mini Match on 10000 and http://localhost:10001
$ curl -O http://localhost:10001/kill
Mini Match is closed.
Mini Match will basically use real Open Match code but use an in-memory Redis, all servers are hosted on the same port from the same binary, and configuration is calculated based on context rather than reading it from a file.
It's in the examples directory now, but we should work towards having the golang gRPC MMF harness in the pkg
directory. This will likely depend on #122
In prep for the glorious serverless function futuretimes, we'd like to get a prototype working that runs MMFs as Knative functions instead of k8s jobs (as mentioned in the roadmap). Plan is currently to get this in for 0.4.0 RC in late Feb 2019.
Goals:
Current implementation thoughts:
Things to evaluate:
I'm sure we'll add to this as we go and lots more thoughts will come up once 0.4.0 RC comes out with an implementation we can all mess around with, so please add your comments as you encounter things we should consider.
This is a benign error (no effect on running OM) but it can be misleading and is pretty spammy :/
msg="invalid character '}' looking for beginning of value" app=openmatch caller=internal/statestorage/redis/redispb/redispb.go cmd=HGETALL component=statestorage key=a2be9dbd0cc04367a06c0f4bea46baea.testprofilev1
Ran into this today
Step 13/14 : RUN composer install
---> Running in 6d98c9c5c406
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Composer could not find a composer.json file in /usr/src/open-match/examples/functions/php/mmlogic-simple
To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section
The command '/bin/sh -c composer install' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1
Probably need a little tweak to the Dockerfile
to sort it out.
Can cause you to get one fewer player than is actually in the pool, found by Ilya and he's got a fix he'll commit soon.
Currently the Open Match Dockerfiles are pulling code from multiple sources.
Let's see backendapi as an example.
# Pulls code from internal and config from 040wip from upstream.
RUN svn export https://github.com/GoogleCloudPlatform/open-match/branches/040wip/internal
RUN svn export https://github.com/GoogleCloudPlatform/open-match/branches/040wip/config
WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi
# Copies the main file from the local repository.
COPY . .
# Pulls dependencies from head.
RUN go get -d -v
The problem with this is the code for Open Match is pulled from multiple places, a bit from the local checkout and most from the upstream branch of Open Match. This build is problematic since it makes development more difficult and the build output is not consistent.
I'd like to propose 2 ways to build docker images.
Example: https://github.com/GoogleCloudPlatform/open-match/pull/98/files
With this approach all the building happens within a docker image. The current approach in the example would be to do the following:
FROM golang:1.12
ENV GO111MODULE=on
WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match
COPY . .
RUN go mod init github.com/GoogleCloudPlatform/open-match
RUN go mod tidy
RUN go mod edit -require k8s.io/[email protected]
RUN go mod vendor
Baseline image that has the codebase and dependencies.
FROM open-match-base-build as builder
WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo .
FROM gcr.io/distroless/static
COPY --from=builder /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/backendapi .
ENTRYPOINT ["/backendapi"]
Creates a minimal image that just has the output binary. Everything is built within docker.
Advantages
An alternative approach is to build outside of docker and simply copy the binary inside.
Example: https://github.com/jeremyje/open-match/blob/buildsys/cmd/backendapi/Dockerfile
# Golang application builder steps
FROM alpine:3.8
ARG APP_ROOT=/go/src/github.com/GoogleCloudPlatform/open-match
ARG APP_NAME=backendapi
RUN apk --update add ca-certificates \
&& adduser -D openmatch \
&& mkdir -p ${APP_ROOT}/config \
&& mkdir -p ${APP_ROOT}/cmd/${APP_NAME} \
&& chown -R openmatch:openmatch /go
COPY --chown=openmatch:root ./cmd/${APP_NAME}/${APP_NAME} ${APP_ROOT}/cmd/${APP_NAME}/${APP_NAME}
USER openmatch
WORKDIR ${APP_ROOT}
ENTRYPOINT ["/go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/backendapi"]
This example does 2 things, create a user which we should do anyways but assumes the binary is built outside already.
Advantages
We'll need a formalized process that details how contributors can officially be onboarded as Open Match approvers or have read-write access to commit PRs.
Agones has a good starting point policy: https://github.com/GoogleCloudPlatform/agones/blob/master/docs/governance/community_membership.md.
For e2e/load testing it'll be easier if all servers had a unified initialization and construction code path that can be injected with fake backends.
Golang is now mid stage at rolling out go modules. It'll be the default in the next version ~6 months away. An important requirement that go mod enforces is the use of semantic version tags for releases. This means if a game studio wants to depend on Open Match 0.4.0 the recommended way in Go 1.13+ will be run go get -u github.com/GoogleCloudPlatform/[email protected]
. Using branches and tag formats that are not vX.Y.Z will fail. There is a fall back for repositories that do not follow this standard which involves using the git commit hash instead which is ugly. See https://github.com/golang/go/wiki/Modules#version-selection for more information.
I'd like to propose we should start using semver starting with the 0.4.0 release. This change is tied with proposal, #97 which allows Open Match to confirm with typical Github workflows.
This is an internal representation thing - shouldn't have any impact outside of the core components, or in any custom functions/evaluators/backend client/frontend clients. Will drop the string size for the unique IDs from 36 characters to 20 characters, a pretty significant memory savings in Redis. We'll also be passing around less data this way, and it should help with the 64-character Job name limit in Kubernetes when kicking off evaluators and MMFs as well.
Since we want public images in 0.3.0, we'd like to start the work to un-jank-ify the Dockerfile
s and cloudbuild.yaml
files. This should be doable by anyone with a decent understanding of docker, cloudbuild, and kubernetes (to test that the results work correctly). Basically:
cloudbuild.yaml
and Dockerfile
that builds off the golang base image and adds the internal
directory. After making this new 'open match base image', the Dockerfile
and cloudbuild.yaml
file to build it should be the only ones left in the repo root, I think.cloudbuild.yaml
files and Dockerfile
s in the repo root to be in the directories of the associated components, examples, and test clients.Quick Example of the Pattern
message CreateMatchRequest {
messages.MatchObject match = 1;
}
message CreateMatchResponse {
messages.MatchObject match = 1;
}
service Backend {
rpc CreateMatch(CreateMatchRequest) returns (CreateMatchResponse) {}
}
Evaluator needs a pass either way; it's crufty and hasn't gotten much of an update since 0.1.0. If 0.4.0 turns it into a long-lived process then I'll take care of this then. Otherwise, just leaving this as a reminder so I don't forget to fix this. Belongs around this line - that rejected proposed ID needs to be removed from the proposed
ignorelist so it will be seen by later MMFs.
Hi. How to connect unity3d matchmaker to GCP open-match? Thanks)
Retrieving pool 'defaultPool'.
'defaultPool': count 000000 | elapsed 0.002
Retrieving pool 'supportPool'.
'supportPool': count 000000 | elapsed 0.002
Traceback (most recent call last):
File "/usr/local/lib/python3.5/random.py", line 262, in choice
i = self._randbelow(len(seq))
File "/usr/local/lib/python3.5/random.py", line 239, in _randbelow
r = getrandbits(k) # 0 <= r < 2**k
ValueError: number of bits must be greater than zero
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./harness.py", line 80, in <module>
results = mmf.makeMatches(profile_dict, player_pools)
File "/usr/src/open-match/examples/functions/python3/mmlogic-simple/mmf.py", line 34, in makeMatches
player['id'] = random.choice(list(player_pools[player['pool']]))
File "/usr/local/lib/python3.5/random.py", line 264, in choice
raise IndexError('Cannot choose from an empty sequence')
IndexError: Cannot choose from an empty sequence
Should catch and exit cleanly.
Min-spec k8s service account for creating k8s jobs for the MMFOrc would be nice for folks running in a cluster managed by another team or otherwise modified from the default SA permissions. Along with this would need to be a change to the manifests for the MMFOrc deployments to use that SA. Would avoid this MMFOrc error message that is caused by a default k8s SA account without job creation permission:
{"app":"openmatch","caller":"mmforc/main.go","component":"mmforc","error":"jobs.batch is forbidden: User \"system:serviceaccount:default:default\" cannot create jobs.batch in the namespace \"default\"","level":"error","msg":"Couldn't create k8s job!","time":"2018-12-21T00:13:28Z"}
As part of the testing enhancements for Open Match we should have a harness that allows for in-process hosting and connection to Open Match servers to do e2e testing.
Reference Document: https://docs.google.com/document/d/13ligVygwQ__1liFZdHpMLwGCXb35T-5QzQ6daf6r_JM/edit
Proposal Document: https://docs.google.com/document/d/1BOhIKP3N9TjOEUSn28jZvLSTFWrE7DKg66LANNerLdg
Agones Issue: googleforgames/agones#649
frontendclient panic:
open city.percent: no such file or directory
Currently, Open Match can be configured with MMFs as containers that get run as K8s jobs. This mechanism does not scale well and will be deprecated. The recommended mechanism would be to host the MMF as a gRPC service and to pass along, the connection details of this service to Open Match.
Open Match will prescribe the service proto and will create clients to this service and call into this service to execute MMFs.
Open Match will also provide a gRPC harness that handles the server setup bit, with instructions to stand up a gRPC server and samples of implementing the MMF that can plug into this harness.
We secured multiple domains for Open Match and we are getting close to publishing a website for it. Currently the Github repository is github.com/GoogleCloudPlatform/open-match
. For go get, this is problematic because hyphenated last paths are not allowed, see #114 for details.
Given this restriction and the desire to keep this as consistent as possible I'd like to propose http://openmatch.dev/ as the primary URL for the website.
We can also have http://open-match.dev/ as well but we'll need to workaround the tail hyphenation problem that open-match does with golang.
As our APIs stabilize and we want to start exposing them to be consumed publically we'll need a directory that indicates "safe to depend on". Typically this is the pkg/ directory for golang applications as stated by golang-standards/project-layout/pkg. The official golang website does not have an opinion on this but using a well understood pattern I think is a reasonable fallback here.
At this point it's unclear when we'll reach that state but we should at least discuss and maybe put a doc.go file there to reserve the package for those changes. At the very latest it should be done populated once we start pushing v1.0.0 release candidates.
Some examples of stuff that should go into pkg/.
Sometimes function set.Difference() computes the difference incorrectly: https://play.golang.org/p/ZEQ_tFtfLy6
This causes MMLogic API to incorrectly apply ignore lists on GetPlayerPool() call.
As Open Match expands we'll need to provide guidance to customers on how to securely deploy it. More importantly we'll need best practices to make security informed decisions during our future designs to make sure that OM is secure by default.
I'll publish a draft for community review/edits once it's ready. As a preview it'll be simply just an index of all the best practices and recommendations for the dependencies of the project and how it relates to the project.
Typically in a git repository the master branch is where active development occurs for cutting a new release. This is outlined in the Github flow process. In addition, once a version is cut there's a few things that should happen,
Challenges for the old pattern:
go get -u github.com/GoogleCloudPlatform/open-match
and not really know what version you're working with.not branches. It has support to read from semver tags against master branch now.
How do we get there?
With this change we'll be modeling the development pattern that Kubernetes does.
Had three different users ask about proper configuration of an Kubernetes nginx ingress for the Frontend API. Would be great if someone can add a quick guide to the important annotations and such required to get this set up correctly for the HTTP2 gRPC Frontend API service. Could optionally cover more broadly the topic of making the Frontend API public, and should definitely cover what use cases require the (relatively complex) nginx ingress over a simple public service configured using the type: LoadBalancer
.
This change should do the following:
This is an issue tracking my investigation for use with the Godot Game Engine.
Hi. I used Google Cloud Platforn.
kubectl logs om-backendapi-568cb94d86-b5x9p
time="2019-01-07T03:52:14Z" level=fatal msg="Fatal error reading config file" app=openmatch caller=config/main.go component=config error="While parsing config: invalid character '.' looking for beginning of value"
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/om-backendapi-568cb94d86-b5x9p 0/1 CrashLoopBackOff 2 1m
pod/om-frontendapi-85b7457b89-f2fh2 0/1 CrashLoopBackOff 1 42s
pod/om-mmforc-687f45b6ff-55kcz 0/1 ContainerCreating 0 35s
pod/om-mmlogicapi-55c6d8766d-t5m4l 0/1 ContainerCreating 0 21s
pod/redis-master-b56bcf67d-sgddc 1/1 Running 0 3h
We should consolidate all our instructions of Open Match on a website and retire the markdown files.
https://docs.google.com/document/d/1PGXzIBgDoSS34QDHP9goI1JpI7i7vLZ2AFyj0WQ0WAI outlines the necessary articles that should be in the first release. Please add any sections you'd like to see.
In the part of the dev guide that covers applying the kubernetes configs for the prometheus operator, it has:
kubectl apply -f prometheus_operator.json
kubectl apply -f prometheus.json
kubectl apply -f prometheus_service.json
kubectl apply -f metrics_servicemonitor.json
A couple users have noticed errors creating these services that go away on applying them a second time. There's probably a resource that is out-of-order in one of the files, so it's not getting added to the cluster before another resource that depends on it. Could probably be fixed with a few minutes of trial-and-error. Low priority obviously, since everything works if the user just re-runs the offending kubectl apply
, which is probably the first thing everyone will try.
Basic work here would be something like:
redis-sentinel
to just redis
and update the OM codebase to match this change (so future users won't be confused by the naming)deployments/k8s/redis-deployment.json
/redis-service.json
and install/yaml/01-redis.yaml
that stands up a HA redis deployment that users can elect to use if they need it.It doesn't have to be the final one, but we should definitely have a pass before it ends up in the repo, as it doesn't conform to any of Jeremy's changes or have any of Caleb's input. So far, what we have right now is:
message MmfSpec {
string name = 1; // Arbitrary developer-chosen, human-readable string. Optional.
string host = 2; // Host or DNS name for service providing this MMF. Must be resolve-able by the backend API.
int32 port = 3; // Port number for service providing this MMF.
enum Type {
// Generic serving types.
GRPC = 0;
REST = 1;
// Knative types will be KNATIVEREST, KNATIVEGRPC, etc.
// Deprecated types
K8SJOB = 7;
}
Type type = 4; // Type of MMF
}
message CreateMatchRequest {
MatchObject matchobject = 1;
MmfSpec mmfspec = 2;
}
message ListMatchesRequest {
MatchObject matchobject = 1;
MmfSpec mmfspec = 2;
}
The MatchObject
message didn't change, so I didn't include it here.
The Backend API then generates the necessary database record keys and calls the function
gRPC API at the host
and port
it reads from the MmfSpec
message:
service Function {
// The assumption is that there will be one service for each MMF that is
// being served. Build your MMF in the appropriate serving harness, deploy it
// to the K8s cluster with a unique service name, then connect to that service
// and call 'Run()' to execute the fuction.
rpc Run(messages.Arguments) returns (messages.Result) {}
}
Its messages look like this, and definitely need to get updated to be in line with Jeremy's changes.
// Messages for gRPC-served matchmaking functions.
// The message for passing in the per-request identifiers
// to a matchmaking function; used so it knows which records to
// write/update in state storage. Internal to Open Match; this is populated by
// the Backend API and sent to your gRPC-served MMF.
message Request {
string profile_id = 1; // Developer-chosen profile name, a copy of the MatchObject name.
string request_id = 2; // Per-request generated unique ID, by convention XID.
string proposal_id = 3; // Intermediate result proposal ID. by convention, the string 'proposal.' prepended to the result_id.
string result_id = 4; // Final result ID. By convention, request_id.profile_id
int64 timestamp = 5; // Reserved for future use.
}
// The message for passing all the necessary arguments to a
// matchmaking function.
message Arguments{
Request request = 1;
MatchObject matchobject = 2;
}
So, let me know how we want it to look and I'll get to work on updating it for inclusion in the 0.4.0 RC.
As mentioned in the docs, right now the 'common' config used by various components is in a json file that gets added to the component containers at build time. This was always intended as an interim solution, we should discuss and evaluate moving config to a more 'kubernetes-native' format.
In the past we've discussed kubernetes ConfigMaps, and those might be a good fit but someone needs to evaluate them. This isn't an exhaustive list, just what springs to mind immediately.
In frontendclient, resultsChan
is only used to print:
case a := <-resultsChan:
pretty.PrettyPrint(a)
And the Assignment
is always empty:
player := players[0]
if connect && player.Assignment != "" {
err = udpClient(context.Background(), player.Assignment)
It should be:
case a := <-resultsChan:
pretty.PrettyPrint(a)
udpClient(context.Background(), a.Assignment)
For folks that need local development of Open Match, instructions and possibly a make file for Kubernetes in Docker (like the one Agones has) or even just minikube would be great. Just instructions on getting everything compiled and deployed into a local environment is all that's needed, I think.
I'd like to propose creating a vanity url for Open Match to abstract away the location of the repository and make it easier for users to access the code.
Something similar to agones.dev/agones. This is also tied to the website site hosting which I'll need to discuss with the Agones folks.
The changes consumers of Open Match will need to make are the following:
import "openmatch.dev/openmatch"
Example of getting the repository:
go get -u openmatch.dev/openmatch
cd src/openmatch.dev/openmatch/
To be clear, this does not change the physical location of the repository but only adds a layer of indirection. For contributing to Open Match you'll still need to be aware of the repository path.
In addition, I'd to also propose that we remove the hyphen from the package name because it causes issues in go.
The following happens when you create a doc.go file with openmatch as the package name.
./doc.go:18:13: syntax error: unexpected -, expecting semicolon or newline
It is possible to workaround this limitation but it means we’ll need to name the package openmatch within .go files like the following.
import openmatch "open-match.dev/open-match"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.