Comments (9)
thisisxvr
commented on May 9, 2024
2
Thanks @craigpastro. I've deployed main to our env, and it is now in use — I will report back if I run into issues.
from openfga.
jon-whit
commented on May 9, 2024
1
@thisisxvr sounds good. No worries. I think I can track it down with what you've already provided.
Thanks for reporting. I probably won't be able to get around to this today but I will tomorrow. I'll keep you updated here 👍 .
from openfga.
craigpastro
commented on May 9, 2024
1
Hi @thisisxvr ! Thanks for reporting this issue. I was able to easily reproduce this issue on the code prior to #113. There was a bug that wasn't releasing a Postgres connection back to the pool which caused pool starvation. It should be fixed in main, and we expect to release v0.1.5 very soon. If you can try main that would be great. Or please wait for v0.1.5. In any case, it would be great if you can report back when you get a chance.
Thanks again!
from openfga.
rhamzeh
commented on May 9, 2024
1
v0.1.6 is now out with the changes included in #113 that would help alleviate this issue.
@thisisxvr thank you for raising this issue. Let us know if this release resolves it for you or not!
from openfga.
thisisxvr
commented on May 9, 2024
1
I've had main running for nearly 48 hours now with no issues — which is the longest I've seen without this error popping up, so I'm happy to consider the issue resolved. Thanks very much @jon-whit and @craigpastro for the quick resolve 🙌.
from openfga.
jon-whit
commented on May 9, 2024
1
I've had
mainrunning for nearly 48 hours now with no issues — which is the longest I've seen without this error popping up, so I'm happy to consider the issue resolved. Thanks very much @jon-whit and @craigpastro for the quick resolve 🙌.
No problem! Thanks for reporting and for following through with confirmation of the fix. We appreciate it!
from openfga.
jon-whit
commented on May 9, 2024
@thisisxvr thanks for reporting this. I'll take a look! In the meantime, if you could upload any step by step instructions that reproduces the issue I would appreciate it 👍 That will help.
from openfga.
thisisxvr
commented on May 9, 2024
Hey @jon-whit. Unfortunately I am not aware of a reliable method to repro this issue. What I can tell you is that I've only seen it happen on a live env (on Azure) — never came across it while running locally. I'm fairly confident that it does not occur when it's just a single user using the service in the env (I have the OpenFGA service wrapped in feature flag, so I can control who is using it).
Here are a couple of examples from the logs:
{"log":"{\"level\":\"info\",\"ts\":1658701171.2792175,\"caller\":\"logger/logger.go:48\",\"msg\":\"HTTP server listening on ':8080'...\",\"build.version\":\"v0.1.4\",\"build.commit\":\"226299f267219b7e36c3449270ee00d4b1234693\"}\n","stream":"stderr","time":"2022-07-24T22:19:31.279311522Z"}
---4004 successful log entries over 19 hours---
{"log":"{\"level\":\"error\",\"ts\":1658769667.8109884,\"caller\":\"logger/logger.go:80\",\"msg\":\"grpc error\",\"build.version\":\"v0.1.4\",\"build.commit\":\"226299f267219b7e36c3449270ee00d4b1234693\",\"error\":\"rpc error: code = Canceled desc = context canceled\",\"request_url\":\"/stores/01G7QVP4XNWB9W64BEWACZ15FQ/write\",\"stacktrace\":\"github.com/openfga/openfga/pkg/logger.(*ZapLogger).ErrorWithContext\\n\\t/home/runner/work/openfga/openfga/pkg/logger/logger.go:80\\ngithub.com/openfga/openfga/server.New.func1\\n\\t/home/runner/work/openfga/openfga/server/server.go:136\\ngithub.com/grpc-ecosystem/grpc-gateway/v2/runtime.HTTPError\\n\\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway/[email protected]/runtime/errors.go:81\\ngo.buf.build/openfga/go/openfga/api/openfga/v1.RegisterOpenFGAServiceHandlerClient.func2\\n\\t/home/runner/go/pkg/mod/go.buf.build/openfga/go/openfga/[email protected]/openfga/v1/openfga_service.pb.gw.go:1429\\ngithub.com/grpc-ecosystem/grpc-gateway/v2/runtime.(*ServeMux).ServeHTTP\\n\\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway/[email protected]/runtime/mux.go:386\\ngithub.com/rs/cors.(*Cors).Handler.func1\\n\\t/home/runner/go/pkg/mod/github.com/rs/[email protected]/cors.go:231\\nnet/http.HandlerFunc.ServeHTTP\\n\\t/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084\\nnet/http.serverHandler.ServeHTTP\\n\\t/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916\\nnet/http.(*conn).serve\\n\\t/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966\"}\n","stream":"stderr","time":"2022-07-25T17:21:07.81184193Z"}
{"log":"{\"level\":\"info\",\"ts\":1658770655.936566,\"caller\":\"logger/logger.go:48\",\"msg\":\"HTTP server listening on ':8080'...\",\"build.version\":\"v0.1.4\",\"build.commit\":\"226299f267219b7e36c3449270ee00d4b1234693\"}\n","stream":"stderr","time":"2022-07-25T17:37:35.936721505Z"}
---125 successful log entries over 37 minutes---
{"log":"{\"level\":\"error\",\"ts\":1658772920.3901386,\"caller\":\"logger/logger.go:80\",\"msg\":\"grpc error\",\"build.version\":\"v0.1.4\",\"build.commit\":\"226299f267219b7e36c3449270ee00d4b1234693\",\"error\":\"rpc error: code = Canceled desc = context canceled\",\"request_url\":\"/stores/01G7QVP4XNWB9W64BEWACZ15FQ/check\",\"stacktrace\":\"github.com/openfga/openfga/pkg/logger.(*ZapLogger).ErrorWithContext\\n\\t/home/runner/work/openfga/openfga/pkg/logger/logger.go:80\\ngithub.com/openfga/openfga/server.New.func1\\n\\t/home/runner/work/openfga/openfga/server/server.go:136\\ngithub.com/grpc-ecosystem/grpc-gateway/v2/runtime.HTTPError\\n\\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway/[email protected]/runtime/errors.go:81\\ngo.buf.build/openfga/go/openfga/api/openfga/v1.RegisterOpenFGAServiceHandlerClient.func3\\n\\t/home/runner/go/pkg/mod/go.buf.build/openfga/go/openfga/[email protected]/openfga/v1/openfga_service.pb.gw.go:1450\\ngithub.com/grpc-ecosystem/grpc-gateway/v2/runtime.(*ServeMux).ServeHTTP\\n\\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway/[email protected]/runtime/mux.go:386\\ngithub.com/rs/cors.(*Cors).Handler.func1\\n\\t/home/runner/go/pkg/mod/github.com/rs/[email protected]/cors.go:231\\nnet/http.HandlerFunc.ServeHTTP\\n\\t/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084\\nnet/http.serverHandler.ServeHTTP\\n\\t/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916\\nnet/http.(*conn).serve\\n\\t/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966\"}\n","stream":"stderr","time":"2022-07-25T18:15:20.390695882Z"}
Let me know if there's anything I can do to get more info for you.
from openfga.
jon-whit
commented on May 9, 2024
@thisisxvr did the v0.1.6 release resolve your issue? If so, do you mind if we close?
from openfga.
Related Issues (20)
- Assertions with context HOT 1
- Export Check benchmarks HOT 1
- Unable to open swagger interface after deploying openfga container HOT 4
- Update default docker command to start server
- Set stricter RPC message size limits for the gRPC server
- Move `pkg/server/commands/reverseexpand` into `internal/graph` package HOT 1
- OpenFGA v1.3.8 panic HOT 10
- Upgrading to 1.3.8 caused: sql: Scan error on column index 5, name "condition_name": converting NULL to string is unsupported HOT 2
- Support for reverse queries for HOT 3
- If OTEL exporter is not available, server panics HOT 2
- Deleting tuples without known relation. HOT 3
- Unable to create Authorization Model using WriteAuthorizatoinModel (grpc) method HOT 4
- Extend request_duration_by_query_count_ms to list objects
- Check gives two responses at random if one of the branches of evaluation errors
- initialize mysql datastore: parse mysql connection dsn: invalid bool value: true HOT 1
- Production MySQL not listing any data HOT 1
- Make ReadAuthorizationModels faster for Postgres and MySQL HOT 4
- Support passing the query Condition Context in Expand
- OpenFGA reports that model is invalid - no entrypoints defined HOT 1
- Error when trying to save the directly pasted authorization models using openfga playground HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openfga.