Comments (9)
@bencoppock this is still an issue, internally we use our own fork of paginator that removes Ecto preloads for all one-to-many assocs when they are made on a paginated query and preloads them after the request in separate queries. But the best solution would be to construct a proper query that doesn't break with pagination.
It's important to notice that this library is probably still the best one our community has at the moment, so if you looking for one - use it. Other pagination libs are using offset paging that doesn't scale and suffer from similar problems.
from paginator.
@glennr it's been a few months since I looked into this, and I'm just now about to get back into it. From what I recall, Quarto seemed an improvement (in that I believe it handled NULLs better), but it also had a couple issues that might block us from using it. Namely, when I last checked:
- it didn't support paginating from the end of the result set backward
- it didn't support jumping to an arbitrary point and paginating forward/backward from there (e.g. if sorting users by last name, jumping to users with last names starting with "J")
from paginator.
Related Ecto issue; elixir-ecto/ecto#2926
from paginator.
@AndrewDryga is this still an issue? Or did the fix for the related Ecto issue fix the issue in Paginator as well?
from paginator.
Thanks, @AndrewDryga. Yeah, we definitely want keyset pagination as opposed to offset-based pagination, so that's what led me here. However, the fact that this library drops records with null in the cursor values is a non-starter for us unfortunately.
I see that there's a fork called Quarto that fixes the NULL
s issue. I'm currently investigating that to see if it'll also support paginating by dynamic values (i.e. values that aren't directly in the database but can be calculated in the database using other values)…
from paginator.
@AndrewDryga, are you able to share the changes you made to paginator
? We'd like to cover as much surface area as possible, so understanding your use cases and the implementation that was required would be really valuable.
from paginator.
@sgerrand we altered the paginator itself but for other purposes (having cursor on COALESCE(field1, field2)
). For pagination with joins we wrote our own wrapper that uses Ecto reflection that I've described above but it's aside of paginator library and would be hard to share :(
from paginator.
I'm currently investigating that to see if it'll also support paginating by dynamic values (i.e. values that aren't directly in the database but can be calculated in the database using other values)…
@bencoppock did you find a solution there?
from paginator.
@bencoppock it does work with both of your use cases but I think the second one is out of scope for any pagination library.
it didn't support paginating from the end of the result set backward
If I remember correctly you can add order_by: ...
to your paginated query and also define cursors with direction, so pagination from the end of the list is the same for paginator as paginating from its beginning. Basically, it doesn't care about the order.
it didn't support jumping to an arbitrary point and paginating forward/backward from there
And it should not do it for you, but you still can do it yourself. From your example, you can SELECT cursor_field FROM users WHERE last_name ILIKE 'J%' ORDER BY last_name LIMIT 1
and then by having cursor field on hand you can separately query a page before and after that cursor using pagination library. (Or you can UNION two queries that do that which is fancier, but writing such query in Ecto and maintaining it might not be worth the tiny speed optimization you will get out of it.)
from paginator.
Related Issues (20)
- help: validate cursors?
- Binary cursor field HOT 1
- What is the purpose of `cursor_fields` HOT 2
- Having wrong before and after key values whenever I'm using select in query HOT 5
- Reversed order of arguments in the reverse order of Ecto HOT 2
- Using total_count_limit raises an exception HOT 1
- Default fetch cursor value function fails when base schema has the same field has the binding's
- Query with a fragment seems to return the after and before keys swapped. HOT 2
- `cursor_fields` with virtual field HOT 3
- Comparison with composite to optimize performance?
- Feature suggestion: `.paginated_stream/2` that mimics `Repo.stream/2` HOT 1
- after in meta gets nil when limit is set to 1
- cursor fields can't find field inside select ordered by fragment
- Cursor decode does not work for custom sort field with `fragment` HOT 1
- The query is slow if `cursor_fields` contains the primary key HOT 1
- Missing tag 1.3.0
- null values get dropped from results HOT 5
- Suggestion: Change metadata HOT 1
- New release
- cursor_fields yield error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from paginator.