ListObjects fails to retrieve all items about community HOT 8 CLOSED

It it planned to ever support returning all items a user has a relationship with? So that I can use ListObjects to get a full list of results, even if there are more items than defined in listObjectsMaxResults (maybe supported by pagination)?

@robinmanuelthiel it already is supported, but not well documented. If you provide a zero value for that config then it won't prematurely terminate. You can also use the StreamedListObjects API for streaming semantics, which may work better for your use case.

The danger with disabling (with zero values) the listObjectsMaxResults and listObjectsDeadline configurations, and the reason those exist in the first place, is that it's possible that operation could hang for a long time. Those are meant as safeguards for the server so that a client request can't overload the server with a ListObjects request that yields an insanely large number of objects.

Unfortunately, given the nature of the query resolution (it's a recursive breadth first search of a relationship graph), it's hard to paginate this dataset because it's not a fixed dataset and the graph evaluation is done concurrently (and thus is not ordered). The query is implemented as a dynamic graph traversal over relational data. This makes it particularly challenging to paginate, and hence the reason why we've chosen these alternative tradeoffs.

I recommend giving the StreamedListObjects API a try and see how that works for you. Disable the listObjectsDeadline by setting it to a zero value and then see if that works well for ya. If you need some assistance don't hesitate to reach out over Discord 👍

from community.

Thanks for raising this issue @danmunteanuevo!

Please note the ListObjects is considered very experimental and we note down some of the known issues with ListObjects here, so please take a look.

Just so that we enough data to know if this is a known issue or a new one can I ask you to share:

• What version of OpenFGA are you running?
• How are you running it (docker/docker-compose/binary/etc..)
• Whether you are using any of these config values/flags, and if so, what are their values
  • listObjectsMaxResults/ Env: OPENFGA_LISTOBJECTSMAXRESULTS
  • listObjectsDeadline/ Env: OPENFGA_LISTOBJECTSDEADLINE
  • experimentals/ Env: OPENFGA_EXPERIMENTALS
• How many relationship tuples do you have in your database?
• How many objects do you expect to be returned in the response

from community.

OpenFGA Latest Docker from 4 days ago: [v0.3.1]
.net SDK
I did not use any of those custom configs.
38 Tuples.
I expect to have returned 6 and receive 5.

from community.

I have enabled on the docker the --experimentals="list-objects-optimized". The query are now faster for the other users/access rights and they work.
But now exactly for the same 2 users -> only for the has_read access right (which was missing items before) I now get some times not always Internal error from the api:
Exception when calling OpenFgaApi.Check: One or more errors occurred. (Internal Server Error) Status Code: OpenFga.Sdk.Exceptions.FgaApiInternalError: Internal Server Error at OpenFga.Sdk.Client.BaseClient.SendRequestAsync[T](HttpRequestMessage request, IDictionary2 additionalHeaders, String apiName, CancellationToken cancellationToken)
at OpenFga.Sdk.Client.BaseClient.SendRequestAsync[T](RequestBuilder requestBuilder, IDictionary2 additionalHeaders, String apiName, CancellationToken cancellationToken) at OpenFga.Sdk.Client.ApiClient.<>c__DisplayClass4_01.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at OpenFga.Sdk.Client.ApiClient.Retry[TResult](Func1 retryable) at OpenFga.Sdk.Client.ApiClient.SendRequestAsync[T](RequestBuilder requestBuilder, String apiName, CancellationToken cancellationToken) at OpenFga.Sdk.Api.OpenFgaApi.ListObjects(ListObjectsRequest body, CancellationToken cancellationToken)

from community.

It it planned to ever support returning all items a user has a relationship with? So that I can use ListObjects to get a full list of results, even if there are more items than defined in listObjectsMaxResults (maybe supported by pagination)?

from community.

@jon-whit Thanks a lot for the detailed answer. Will give it a try and report back! Did I understand correctly, that streaming the results still requires a lot of calculation on the server and still gets throttled or limited by the listObjectsMaxResults and listObjectsDeadline variables? So streaming just gives me first results faster but internally uses the same logic, correct?

from community.

Did I understand correctly, that streaming the results still requires a lot of calculation on the server and still gets throttled or limited by the listObjectsMaxResults and listObjectsDeadline variables? So streaming just gives me first results faster but internally uses the same logic, correct?

@robinmanuelthiel That is correct, yes. That's a perfect way to describe it. It's a more responsive way to receive the result stream, whereas the alternative /list-objects or ListObjects API waits for the results (up to the max) before responding. So if, for example, you are building a UI that only needs the first 10 results, then the streaming variant may be more responsive for you.

from community.

@jon-whit Thanks for clarifying. Are there any plans to ever support pagination or something similar? My use-case is search, so I need to get all items of type Foo a user has access to and then filter my search results to only include these.

As I am building a multi-tenant system, I can't afford that one search affects the permissions-performance for all other tenants, I am looking for a solution that gives me a paginated list of all items a user can access without running into memory/time issues on the permissions server :/

from community.