Code Monkey home page Code Monkey logo

order_query's Introduction

order_query Build Status Coverage Status

100% offset-free

This gem finds the next or previous record(s) relative to the current one efficiently using keyset pagination, e.g. for navigation or infinite scroll.

Installation

Add to Gemfile:

gem 'order_query', '~> 0.5.3'

Usage

Use order_query(scope_name, *order_option) to create scopes and class methods in your model and specify how you want results ordered. A basic example:

class Post < ActiveRecord::Base
  include OrderQuery
  order_query :order_home,
    [:pinned, [true, false]], # First sort by :pinned over t/f in :desc order
    [:published_at, :desc] # Next sort :published_at in :desc order
end

Each order option specified in order_query is an array in the following form:

  1. Symbol of the attribute name (required).
  2. An array of values to order by, such as %w(high medium low) or [true, false] (optional).
  3. Sort direction, :asc or :desc (optional). Default: :asc; :desc when values to order by are specified.
  4. A hash (optional):
option description
unique Unique attribute. Default: true for primary key, false otherwise.
sql Customize column SQL.
nulls If set to :first or :last, orders NULLs accordingly.

If no unique column is specified, [primary_key, :asc] is used. Unique column must be last.

Scopes for ORDER BY

Post.published.order_home         #=> #<ActiveRecord::Relation>
Post.published.order_home_reverse #=> #<ActiveRecord::Relation>

Before / after, previous / next, and position

First, get an OrderQuery::Point for the record:

p = Post.published.order_home_at(Post.find(31)) #=> #<OrderQuery::Point>

It exposes these finder methods:

p.before   #=> #<ActiveRecord::Relation>
p.after    #=> #<ActiveRecord::Relation>
p.previous #=> #<Post>
p.next     #=> #<Post>
p.position #=> 5

The before and after methods also accept a boolean argument that indicates whether the relation should exclude the given point or not. By default the given point is excluded, if you want to include it, use before(false) / after(false).

If you want to obtain only a chunk (i.e., a page), use before or after with ActiveRecord's limit method:

p.after.limit(20) #=> #<ActiveRecord::Relation>

Looping to the first / last record is enabled for next / previous by default. Pass false to disable:

p = Post.order_home_at(Post.order_home.first)
p.previous        #=> #<Post>
p.previous(false) #=> nil

Even with looping, nil will be returned if there is only one record.

You can also get an OrderQuery::Point from an instance and a scope:

posts = Post.published
post  = posts.find(42)
post.order_home(posts) #=> #<OrderQuery::Point>

Dynamic columns

Query with dynamic order columns using the seek(*order) class method:

space = Post.visible.seek([:id, :desc]) #=> #<OrderQuery::Space>

This returns an OrderQuery::Space that exposes these methods:

space.scope           #=> #<ActiveRecord::Relation>
space.scope_reverse   #=> #<ActiveRecord::Relation>
space.first           #=> scope.first
space.last            #=> scope_reverse.first
space.at(Post.first)  #=> #<OrderQuery::Point>

OrderQuery::Space is also available for defined order_queries:

Post.visible.order_home_space #=> #<OrderQuery::Space>

Alternatively, get an OrderQuery::Point using the seek(scope, *order) instance method:

Post.find(42).seek(Post.visible, [:id, :desc]) #=> #<OrderQuery::Point>
# scope defaults to Post.all
Post.find(42).seek([:id, :desc]) #=> #<OrderQuery::Point>

Advanced example

class Post < ActiveRecord::Base
  include OrderQuery
  order_query :order_home,
    # For an array of order values, default direction is :desc
    # High-priority issues will be ordered first in this example
    [:priority, %w(high medium low)],
    # A method and custom SQL can be used instead of an attribute
    [:valid_votes_count, :desc, sql: '(votes - suspicious_votes)'],
    # Default sort order for non-array columns is :asc, just like SQL
    [:updated_at, :desc],
    # pass unique: true for unique attributes to get more optimized queries
    # unique is true by default for primary_key
    [:id, :desc]
  def valid_votes_count
    votes - suspicious_votes
  end
end

How it works

Internally this gem builds a query that depends on the current record's values and looks like this:

-- Current post: pinned=true published_at='2014-03-21 15:01:35.064096' id=9
SELECT "posts".* FROM "posts"  WHERE
  ("posts"."pinned" = 'f' OR
   "posts"."pinned" = 't' AND (
      "posts"."published_at" < '2014-03-21 15:01:35.064096' OR
      "posts"."published_at" = '2014-03-21 15:01:35.064096' AND "posts"."id" < 9))
ORDER BY
  "posts"."pinned"='t' DESC, "posts"."pinned"='f' DESC,
  "posts"."published_at" DESC,
  "posts"."id" DESC
LIMIT 1

The actual query is a bit different because order_query wraps the top-level OR with a (redundant) non-strict column x0' AND (x0 OR ...) for performance reasons. This can be disabled with OrderQuery.wrap_top_level_or = false.

See the implementation in sql/where.rb.

See how this affects query planning in Markus Winand's slides on Pagination done the Right Way.

This project uses MIT license.

order_query's People

Contributors

dtrejo avatar glebm avatar jsonb-uy avatar niborg avatar rogercampos avatar sanzstez avatar stephannv avatar vkinelev avatar yujideveloper avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

order_query's Issues

Wrong order?

Hi! I am trying to use this gem in a CMS, to show the previous/next post related to the current one, and sorted by publishing date.

In my dev environment, if I run

blog.
    documents.
    published.
    posts.
    order(published_at: :asc).
    pluck(:id)

I get these IDs:

[195, 209, 210, 212, 213, 214, 215, 216, 217, 218, 255, 257, 258, 259, 260, 261, 262, 263]

If I ask the previous of 255 for example, I get 218 as expected. However if I ask for the next document, I get 263 instead of 257. But both of the queries generated seem to be wrong:

Previous

SELECT "documents".* FROM "documents" WHERE "documents"."blog_id" = $1 AND "documents"."published" = $2 AND "documents"."type" = $3 AND ("documents"."published_at" <= '2020-05-08 17:56:08.844567' AND ("documents"."published_at" < '2020-05-08 17:56:08.844567' OR "documents"."published_at" = '2020-05-08 17:56:08.844567' AND "documents"."id" < 255)) ORDER BY "documents"."published_at" DESC, "documents"."id" DESC LIMIT $4

Next

SELECT "documents".* FROM "documents" WHERE "documents"."blog_id" = $1 AND "documents"."published" = $2 AND "documents"."type" = $3 AND ("documents"."published_at" >= '2020-05-08 17:56:08.844567' AND ("documents"."published_at" > '2020-05-08 17:56:08.844567' OR "documents"."published_at" = '2020-05-08 17:56:08.844567' AND "documents"."id" > 255)) ORDER BY "documents"."published_at" DESC, "documents"."published_at" ASC, "documents"."id" ASC LIMIT $4

So for some reason order_query adds a where clause for the ID, which is incorrect. I only want to sort by publishing date (published_at).

The scope I give to order_query is as follow:

    scope = Current.blog.
    documents.
    published.
    posts

    @related ||= scope.order_related_at(@document)

And in the document model I have this:

order_query :order_related, [:published_at, :asc]

What am I missing?

Thanks a lot in advance!

Change base scope?

Thanks for the great gem, it's been really helpful.

We ran into a situation where we wanted to change the base scope, specifically to add a join (we were ordering on a joined table's field). It doesn't appear that there was a way to do that, so we defined our own scopes that wrapped the scopes generated by order_query. What would you think about a PR to enable overriding the base scope instead of hard-coding all?

Thanks!

Get point for reverse scope

Hi, first thank you for this awesome gem !

I need to get a point position for a record in an order_query scope in reverse order. Something like:

p = Post.published.order_home_reverse_at(post)

or

p = post.order_home_reverse(posts)

Is there a way to do this ? The only way I found is to define 2 order_query, one for each direction...
Thanks

new version release

Could you please release a new version into rubygems? I would want to reference to this gem from another gem, but I need the addition on #11

thanks!

Can this sort on nested attributes?

I've got some data that I'd like to get the next and previous results from based off a nested attribute. Something like ordering posts by last comment date on that post.

Calling prev and next when the ordering model attribute is enumerable creates an invalid SQL where clause

The model:

class Address < ApplicationRecord
  include OrderQuery
  belongs_to :network
  validates :inet, presence: true
  order_query :by_inet, [ :inet ]

  def next loop_around: false
    self.by_inet.next(loop_around)
  end

  def prev loop_around: false
    self.by_inet.previous(loop_around)
  end

end

The model field named inet is an IPAddress.
IPAddress includes Enumerable (a network IP address could be said to cover multiple IP addresses).

Calling @address.prev from a view fails with:

  Rendering addresses/show.html.erb
  Address Load (0.9ms)  SELECT "addresses".* FROM "addresses" WHERE ("addresses"."inet" <= inet '10.7.7.4/30',inet '10.7.7.5/30',inet '10.7.7.6/30',inet '10.7.7.7/30' AND ("addresses"."inet" < inet '10.7.7.4/30',inet '10.7.7.5/30',inet '10.7.7.6/30',inet '10.7.7.7/30' OR "addresses"."inet" = inet '10.7.7.4/30',inet '10.7.7.5/30',inet '10.7.7.6/30',inet '10.7.7.7/30' AND "addresses"."id" < 20)) ORDER BY "addresses"."inet" DESC, "addresses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  ↳ app/models/address.rb:47:in `prev'
  Rendered addresses/show.html.erb (Duration: 2.6ms | Allocations: 2915)
Completed 500 Internal Server Error in 4ms (ActiveRecord: 1.1ms | Allocations: 3562)
  
ActionView::Template::Error (PG::DatatypeMismatch: ERROR:  argument of AND must be type boolean, not type inet
LINE 1: ....4/30',inet '10.7.7.5/30',inet '10.7.7.6/30',inet '10.7.7.7/...
                                                             ^

The expected SQL where clause should not contain the 4 enumerated IP addresses covered by '10.7.7.7/30'. It should contain only a single inet '10.7.7.7/30', something like:

... WHERE ("addresses"."inet" <= inet '10.7.7.7/30' ... )

I suspect this would happen for any model attributes that include Enumerable, probably as a result of splatting the attribute when building the where clause ?

FYI: The inet '10.7.7.7/30' is created by a custom PostgreSQL adapter as required to handle PG inet data types. It works as expected.

FYI: Address.by_inet works as expected:

  Rendering addresses/index.html.erb within layouts/application
  Address Load (0.4ms)  SELECT "addresses".* FROM "addresses" ORDER BY "addresses"."inet" ASC, "addresses"."id" ASC
  ↳ app/views/addresses/_table.html.erb:29

BTW: Thanks for creating this gem, it solves an unexpectedly difficult problem very well.

is sorting by an associated column supported?

Given this AR model set:

class Sandwich < ActiveRecord::Base
  has_many toppings
  include OrderQuery
end

# has a name attribute
class Toppings < ActiveRecord::Base
end

Could you order by the associated column, Topping.name?

I have not had luck with this. It appears that the order clause prepends the base model's table name, unless you pass a sql option with the raw value:

# fails, tries to do "sandwich.toppings.name"
point = Sandwich.include(:toppings).joins(:toppings).seek(*[[:"toppings.name", :desc]])

# works, I suppose the first arg is ignored since the has is passed in as a 3rd arg
point = Sandwich.include(:toppings).joins(:toppings).seek(*[[:"toppings.name", :desc, {:sql=>'toppings.name'} ]])

However, calling point.after above throws an error:

undefined method `toppings.name' for #<Sandwich:00x00...>

.previous always showing first record when limiting scope

First of all, I am a total noob, so sorry if I have this set up incorrectly or if this is not the proper channel to communicate this type of issue. I still have so much to learn!

I have your order_query gem working beautifully for my Bottle class as such:

class Bottle < ActiveRecord::Base
  ....
  include OrderQuery
  order_query :order_home,
    [:name],
    [:id]
end

When I use it in my controller like @b = Bottle.order_home_at(@bottle), everything works as expected. I can use @b.next.id and @b.previous.id to do what I'm trying to accomplish.

However, when I try to limit the scope of the query to match my UI, with something like this @b = Bottle.order(:name, :id).includes(:liquor_type).where("liquor_types.name" => params[:type]).order_home_at(@bottle) I run into what may be a bug.

I seem to be able to use @b.next.id with expected behavior, but @b.previous.id always returns the id of the first record in the intended set.

Am I doing something wrong (probably), or have I discovered a bug? Thank you for your help and for making such a cool gem!

pass table and column to custom sql function

In trying to figure out some NULL sorting handling issues, I ran into #10, and in particular the suggestion to use COALESCE to turn a null value to a non-value.

In order to allow this kind of functionality for dynamic queries, I am suggesting that the table and column name obtained from the connection be passed to the sql function. In other words, in here, a change along the lines of:

          table = connection.quote_table_name(scope.table_name)
          column = connection.quote_column_name(column.name)

          if sql
            sql.respond_to?(:call) ? sql.call(table, column) : sql
          else
            "#{table}."\
            "#{column}"

This allows me to call COALESCE using the resolved table name, as I don't have access to the connection in the custom sql proc.

If you're open to this, I can submit a PR.

Thank you!

documentation on how to "paginate" results

I have put "paginate" in quotes because I am not referring to the paginate found in a offset/query based "page x of y" scheme, but as a "page" or chunk of rows in a infinite scrolling scheme.

The best I can tell from studying this library and the postings in the "use the index" site, the cursors are identified by the primary key of the row, and that I would still need to use limit in combination with those PKs to chunk the results.

With this in mind, I am not sure how to go about this properly with the Point/Space APIs in this library given a dynamic query.

  • On the first keyset paginated request for some records, the client would need to be given the following bits of info in the response:

    • The records in the current "page"
    • The cursor to get the previous "page"
    • The cursor to get the next "page"

    In order to get the first item above, I could do a seek per the docs:

      Post.visible.seek([:id, :desc]) 
    

    I could add a limit to change the behavior of the resulting after to infer the cursor for the next "page":

      Post.limit(10).visible.seek([:id, :desc]) 
    

    However, I actually would have to fetch limit +1 for thespace.last to point to the cursor for the next "page".

    I am already feeling like I am misunderstanding something. :)

  • On subsequent requests (e.g. now I have a cursor-- say id 42), I could use:

       Post.find(42).seek([:id, :desc])
    

    Now I have a Point with a before and after. The before I won't use since this is an infinite scroller and the data is already on the client (otherwise I'd have to do additional calculation to get the before "page"'s start position). The after once again would include all records, not just the next chunk, so calculating the cursor for the next "page" would involve similar limit +1 logic from above.

Now I definitely feel like I'm not using this library correctly!

If you could be so kind to provide some assistance, I would be willing to write some documentation on the wiki.

Support of Rails 7.1

Hello! Rails 7.1 released, so can you please update this dependencies for your gem?

From:

 s.add_dependency 'activerecord', '>= 5.0', '< 7.1'
 s.add_dependency 'activesupport', '>= 5.0', '< 7.1'

To:

 s.add_dependency 'activerecord', '>= 5.0', '<= 7.1'
  s.add_dependency 'activesupport', '>= 5.0', '<= 7.1'

Thanks!

Comparison predicates with row value constructors

SQL '92 standard, Predicates 209, defines row values comparison predicates as:

Let Rx and Ry be the two row value constructors of the comparison predicate and let RXi and RYi be the i-th row value constructor elements of Rx and Ry, respectively. "Rx comp op Ry" is true, false, or unknown as follows:

a) "x = Ry" is true if and only if RXi = RYi for all i.
b) "x <> Ry" is true if and only if RXi <> RYi for some i.
c) "x < Ry" is true if and only if RXi = RYi for all i < n and RXn < RYn for some n.
d) "x > Ry" is true if and only if RXi = RYi for all i < n and RXn > RYn for some n.

Example:

SELECT *
  FROM sales
 WHERE (sale_date, sale_id) < (?, ?)
 ORDER BY sale_date DESC, sale_id DESC

They are only on supported on certain databases (postgres yes, mysql no), and additional fiddling would probably be required to deal with array attributes (priority: high medium low) and differences in order direction between sort attributes.

I won't work on this in the near future, but this would be great to do on databases that support it.

Nulls ordering skips records if there is more than one NULL

Example from #20 (comment):

seek_args: [[:title, :desc, {:nulls=>:first}]]
Entry Load (1.5ms)  SELECT  `entries`.* FROM `entries` ORDER BY `entries`.`title` IS NULL DESC, `entries`.`title` DESC, `entries`.`id` ASC LIMIT 20
Entry Load (0.8ms)  SELECT  `entries`.* FROM `entries` WHERE `entries`.`id` = 1 LIMIT 1
Entry Load (1.0ms)  SELECT  `entries`.* FROM `entries` WHERE (`entries`.`title` IS NULL AND (`entries`.`title` IS NULL OR `entries`.`title` IS NULL AND `entries`.`id` > 1)) ORDER BY `entries`.`title` IS NULL ASC, `entries`.`title` ASC, `entries`.`id`     DESC LIMIT 1

seek_args: [[:title, :desc, {:nulls=>:last}]]
Entry Load (1.0ms)  SELECT  `entries`.* FROM `entries` ORDER BY `entries`.`title` DESC, `entries`.`id` ASC LIMIT 20
Entry Load (0.7ms)  SELECT  `entries`.* FROM `entries` WHERE `entries`.`id` = 1 LIMIT 1
Entry Load (0.8ms)  SELECT  `entries`.* FROM `entries` WHERE (`entries`.`title` IS NOT NULL AND (`entries`.`title` IS NOT NULL OR `entries`.`title` IS NULL AND `entries`.`id` > 1)) ORDER BY `entries`.`title` ASC, `entries`.`id` DESC LIMIT 1

/cc @twelve17

Feature Request:

Given an ActiveRecord::Relation, which is already ordered etc., provide next and previous methods.

some_ar_relation.next(post.id)
some_ar.relation.previous(post.id)

Ordering by NULLS FIRST / LAST

Hallo and thanks for this neat gem!

I'm trying to implement an order with some custom sql but I do not get results. I have a field posted_at that sometimes may be NULL. The order I want to achieve is posted_at DESC NULLS LAST

I tried using order_query in different ways but to no avail. Can you please tell me if it's possible and point me in the right direction?

Rails 6 support

Hi,
any chance to relax dependencies to support Rails 6?

s.add_dependency 'activerecord', '>= 5.0', '< 6.0'
s.add_dependency 'activesupport', '>= 5.0', '< 6.0'

Top-level disjunctive causes sub-optimal query plan

Issue from a comment by @fatalmind:

Besides the row values, there is also a performance problem with the logic as shown in the README:

x0 OR 
y0 AND (x1 OR
        y1 AND (x2 OR
                y2 AND ...))

The problem is that this kind of where-clause cannot use and index on these columns, even if it exists. No matter which database.

See here: http://use-the-index-luke.com/sql/partial-results/fetch-next-page#sb-equivalent-logic

To use an index, you must not have an OR on the top level comparison.

Let me know if you have any questions.

-Markus Winand

#seek or #at with previously specified order

If I do this:

relation = MyModel.order_foo

My relation is ordered according to the order_foo specified in the model. If I then want to get a Point I have to do this:

point = relation.order_foo_at(instance)

So I've now specified the ordering twice. If I change the ordering I have to change it in both places. If I make the ordering selection complicated based on some logic (parameterized, etc...) I have to have that logic in two places, or use relation.send, or use seek, etc...

Is there a way to avoid this repetitiveness? Since the relation has already been ordered using order_query, it feels like I should be able to do this:

point = relation.at(instance)

or:

point = instance.seek(relation)

or something similar?

Is there a way to get a point based on the previously used ordering without re-specifying it?

Not compatible with rails 5

Bundler could not find compatible versions for gem "activerecord":
In Gemfile:

    order_query was resolved to 0.3.2, which depends on
      activerecord (~> 4.0)

rails (= 5.0.0.beta1) was resolved to 5.0.0.beta1, which depends on
  activerecord (= 5.0.0.beta1)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.