Code Monkey home page Code Monkey logo

gobeam / mongo-go-pagination Goto Github PK

View Code? Open in Web Editor NEW
129.0 5.0 35.0 102 KB

Golang Mongodb Pagination for official mongodb/mongo-go-driver package which supports both normal queries and Aggregation pipelines with all information like Total records, Page, Per Page, Previous, Next, Total Page and query results.

Home Page: https://mongo-go-pagination.herokuapp.com/normal-pagination?page=1&limit=15

License: MIT License

Go 100.00%
go-mongo mongo-go-pagination mongo-go-driver mongo-pagination mongo pagination aggregation pipelines golang paginator

mongo-go-pagination's Introduction

Golang Mongo Pagination For Package mongo-go-driver

Workflow Build Go Report Card GoDoc Coverage Status

For all your simple query to aggregation pipeline this is simple and easy to use Pagination driver with information like Total, Page, PerPage, Prev, Next, TotalPage and your actual mongo result. View examples from here

๐Ÿ”ˆ ๐Ÿ”ˆ For normal queries new feature have been added to directly pass struct and decode data without manual unmarshalling later. Only normal queries support this feature for now. Sort chaining is also added as new feature

Example api response of Normal Query click here.
Example api response of Aggregate Query click here.
View code used in this example from here

Install

$ go get -u -v github.com/gobeam/mongo-go-pagination

or with dep

$ dep ensure -add github.com/gobeam/mongo-go-pagination

For Aggregation Pipelines Query

package main

import (
	"context"
	"fmt"
	. "github.com/gobeam/mongo-go-pagination"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/bson/primitive"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
	"time"
)

type Product struct {
	Id       primitive.ObjectID `json:"_id" bson:"_id"`
	Name     string             `json:"name" bson:"name"`
	Quantity float64            `json:"qty" bson:"qty"`
	Price    float64            `json:"price" bson:"price"`
}

func main() {
	// Establishing mongo db connection
	ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
	client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
	if err != nil {
		panic(err)
	}
	
	var limit int64 = 10
	var page int64 = 1
	collection := client.Database("myaggregate").Collection("stocks")

	//Example for Aggregation

	//match query
	match := bson.M{"$match": bson.M{"qty": bson.M{"$gt": 10}}}

	//group query
	projectQuery := bson.M{"$project": bson.M{"_id": 1, "qty": 1}}

    // set collation if required
    collation := options.Collation{
		Locale:    "en",
		CaseLevel: true,
	}

	// you can easily chain function and pass multiple query like here we are passing match
	// query and projection query as params in Aggregate function you cannot use filter with Aggregate
	// because you can pass filters directly through Aggregate param
	aggPaginatedData, err := New(collection).SetCollation(&collation).Context(ctx).Limit(limit).Page(page).Sort("price", -1).Aggregate(match, projectQuery)
	if err != nil {
		panic(err)
	}

	var aggProductList []Product
	for _, raw := range aggPaginatedData.Data {
		var product *Product
		if marshallErr := bson.Unmarshal(raw, &product); marshallErr == nil {
			aggProductList = append(aggProductList, *product)
		}

	}

	// print ProductList
	fmt.Printf("Aggregate Product List: %+v\n", aggProductList)

	// print pagination data
	fmt.Printf("Aggregate Pagination Data: %+v\n", aggPaginatedData.Data)
}

For Normal queries

func main() {
	// Establishing mongo db connection
	ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
	client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
	if err != nil {
		panic(err)
	}

	// Example for Normal Find query
    	filter := bson.M{}
    	var limit int64 = 10
    	var page int64 = 1
    	collection := client.Database("myaggregate").Collection("stocks")
    	projection := bson.D{
    		{"name", 1},
    		{"qty", 1},
    	}
    	// Querying paginated data
    	// Sort and select are optional
        // Multiple Sort chaining is also allowed
        // If you want to do some complex sort like sort by score(weight) for full text search fields you can do it easily
        // sortValue := bson.M{
        //		"$meta" : "textScore",
        //	}
        // aggPaginatedData, err := paginate.New(collection).Context(ctx).Limit(limit).Page(page).Sort("score", sortValue)...
        var products []Product
    	paginatedData, err := New(collection).Context(ctx).Limit(limit).Page(page).Sort("price", -1).Select(projection).Filter(filter).Decode(&products).Find()
    	if err != nil {
    		panic(err)
    	}
    
    	// paginated data or paginatedData.Data will be nil because data is already decoded on through Decode function
    	// pagination info can be accessed in  paginatedData.Pagination
    	// print ProductList
    	fmt.Printf("Normal Find Data: %+v\n", products)
    
    	// print pagination data
    	fmt.Printf("Normal find pagination info: %+v\n", paginatedData.Pagination)
}
    

Notice:

paginatedData.data //it will be nil incase of  normal queries because data is already decoded on through Decode function

Running the tests

$ go test

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

Acknowledgments

  1. https://github.com/mongodb/mongo-go-driver

MIT License

Copyright (c) 2021

mongo-go-pagination's People

Contributors

ghstahl avatar gobeam avatar littlecxm avatar yzherebukh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

mongo-go-pagination's Issues

getSkip decrements skip by one more, causing the pagination to pick up one less document

Describe the bug
A clear and concise description of what the bug is.
"getSkip" function decrements skip by one more, causing the pagination to pick up one less document.

To Reproduce
Steps to reproduce the behavior:
Create a collection with 2 documents;

  1. Call pagination function with page number 2 and limit 1.
  2. The function returns the first document instead of second document.

Expected behavior
The function should return second document in page 2.

Additional context
This happens in version v0.0.5 where the "getSkip" function decrements the "skip" value by one more after calculating the correct value.

// getSkip return calculated skip value for query

func getSkip(page, limit int64) int64 {
	page--
	skip := page * limit
	skip--

	if skip <= 0 {
		skip = 0
	}

	return skip
}

getSkip logic is broken

Describe the bug

The Problem

getSkip

func getSkip(page, limit int64) int64 {	
 page--	
 skip := page * limit	
 skip--
 if skip <= 0 {		
    skip = 0	
 }
return skip
}

if page == 2 and limit == 1
then the following should be returned.

skip == 1

Actual: skip == 0

The fix

func getSkip(page, limit int64) int64 {	
 page--	
 skip := page * limit	
  
 if skip <= 0 {		
    skip = 0	
 }
return skip
}

The current code will produce the same result given the following paging arguments.

page = 1
limit = 1
 and
page = 2
limit = 1

How to sort alphabetically?

I have a collection like this one:

[
   { address: "a" },
   { address: "b" },
   { address: "A" },
   { address: "B" },
   { address: "c" },
   { address: "D" },
]

My query looks like this:

paginatedData, err := mongopagination.New(collection).Context(ctx).
   Limit(limit).
   Page(page).
   Filter(paginationFilter).
   Decode(&addresses).
   Sort("address", 1).
   Find()

I'd expect the following order in the response: a, A, b, B, c, D
but in fact it is: A, B, D, a, b, c

How can I sort the elements in alphabetically order? Is there a way to set a "collation" or something else?

Total page is always 0

I got in this trouble total page is always 0
Aggregate Pagination Data: {Total:5 Page:1 PerPage:2 Prev:0 Next:0 TotalPage:0}

OffSet Usage

Hey is it possible to use or add the usage of an OffSet at all?

aggPaginatedData, err := New(collection).Limit(limit).OffSet(offset).Sort("price", -1).Aggregate(match, projectQuery)
...

How to dump the query?

We're running the performance test using mongo explain query.
Do you have any method that lets us just dump the query so we can just copy and paste the query instead of figuring out ourselves?

Thanks

Hardcoded Sort on Aggregate function

facet := bson.M{"$facet": bson.M{
		"data": []bson.M{
			{"$sort": bson.M{"createdAt": -1}},
			{"$skip": skip},
			{"$limit": paging.LimitCount},
		},
		"total": []bson.M{{"$count": "count"}},
	},
}

It makes it impossible to have a sort pipeline when doing aggregation

concurrent query CountDocuments

Is your feature request related to a problem? Please describe.
in func (paging *pagingQuery) Find(), Paging(...) could be executed concurrently with Find()

Describe the solution you'd like

func (paging *pagingQuery) Find() (paginatedData *PaginatedData, err error) {
         ...
	paginationInfoChan := make(chan *Paginator, 1)
	go Paging(paging, paginationInfoChan, false, 0)
        ...
}

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is there any way to sort by weight(score) for full text search fields?

Is your feature request related to a problem? Please describe.
As I know right now we have only .Sort() method which accept 2 arguments: field name and sort by ASC, DESC but it would be nice to have a method to sort by score(weight) for full text search fields. The query might be something like:
db.table.find({$text:{$search:"Test"}},{score:{$meta:"textScore"}}).sort({score:{$meta:"textScore"}})

Describe the solution you'd like
It would be nice to have a method like .sortByScore()

Skip calculation error

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Call function '...'
    mongopagination.New(models.RedPackCollection).Context(context.TODO()).Limit(int64(p.PageSize)).Page(int64(p.CurPage)).Filter()
  2. Pass value '...'
    p.PageSize=10
    p.CurPage=2
  3. See error

func getSkip(page, limit int64) int64 {
page--
skip := page * limit
skip-- / / produces an extra result

if skip <= 0 {
	skip = 0
}

return skip

}

Expected behavior
A clear and concise description of what you expected to happen.

I think we should delete him

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

opts agregation

you can make the *opt.AgregateOptions vailable, in the aggregation func

sample:

opts := &options.AggregateOptions{
AllowDiskUse: &diskUse,
Collation: &options.Collation{
Locale: "en",
NumericOrdering: true,
Strength: 2,
},
}

paginate.New(coll).Context(ctx).Limit(filtros.FindSet.Pagesize).Page(filtros.FindSet.Page).Aggregate(&opts, pipeline...)

unexported field

Hi,
i have a lot of implicit assignment of unexported field 'collection' when i try to use your code.

Page limit not applied for aggregation

Hi,

I have 3 entries in my db and I apply following aggregation query.

        matchStage := bson.M{"$match": bson.M{"user_id":  userId}}
	lookup := bson.M{"$lookup": bson.M{"from": "com", "localField": "com_id", "foreignField": "_id", "as": "com"}}
	unwindStage := bson.M{"$unwind": bson.M{"path": "$com", "preserveNullAndEmptyArrays": false}}
	
        cur, err := mongopagination.New(cr.GetCollection()).
		Page(1).
		Limit(1).
		Sort("role", -1).
		Aggregate(matchStage)
	if err != nil {
		panic(err)
	}

Expected
3 pages with 1 result per page.

Actual
3 pages with all 3 results per page.

{
   "data":[
      {
         "com":{
            "id":"5f68d3dc9b8155bb8807d1e6",
            "name":"ceE5u25yH2",
            "info":{
               "address":{
                  
               }
            },
            "is_open":true,
            "is_active":true
         },
         "role":2
      },
      {
         "com":{
            "id":"5f68d3dc9b8155bb8807d1e8",
            "name":"3QmOnaG3uy",
            "info":{
               "address":{
                  
               }
            },
            "is_open":true,
            "is_active":true
         },
         "role":2
      },
      {
         "com":{
            "id":"5f68d3dc9b8155bb8807d1ea",
            "name":"RDn8LlZGeX",
            "info":{
               "address":{
                  
               }
            },
            "is_open":true,
            "is_active":true
         },
         "role":2
      }
   ],
   "pagination":{
      "total":3,
      "page":1,
      "perPage":1,
      "prev":0,
      "next":2,
      "totalPage":3
   }
}

Cause:
File: pagingQuery.go
Method: Aggregate

facetData slice "$skip" (and "$limit" if sort is used) is overridden as slice 0 index value replaced.

        facetData := make([]bson.M, 1)
	facetData[0] = bson.M{"$skip": skip}
	facetData[0] = bson.M{"$limit": paging.LimitCount}
	if paging.SortField != "" {
		facetData[0] = bson.M{"$sort": bson.M{paging.SortField: paging.SortValue}}
	}

Use aggregate with optional match stage

Is your feature request related to a problem? Please describe.
How can i use aggregate with optional match stage. For example,
I have to use aggregate like this:

  var matchStage bson.M
  if accountId != "" {
      matchStage = bson.M{
            "$match": bson.M{ "_id": accountId, },
      }
  }
  aggPaginatedData, err := New(collection).Context(ctx).Limit(limit).Page(page).Sort("created_at",sort).Aggregate(matchStage)

Here, if there is value for accountId, then the aggregate works fine but if the accountId == "" then i got an error says A pipeline stage specification object must contain exactly one field
How can we deal with that scenario?
Thanks

Allow to disable pagination

Is your feature request related to a problem? Please describe.
How can i disable the pagination? When i tried to not chain the limit and page, i got an error says "page or limit cannot be less than 0"

Describe the solution you'd like
Like this NPM (JS) package here, there is an option "pagination" if it is set to false, it will return all docs without adding limit condition. (Default: True)

decode to struct we want instead of bson.raw

in line , you use a decoder and pass variable in type of bson.raw, in the example you use decode again to convert data to struct we want ...
in my opinion we can add func to app to get the struct and decode it in the first time ! what is your opinion about this ?

How to do multiple sort fields?

We can have a single sort by using .Sort("price", -1).
But how to add another sort field?

I tried .Sort("price", -1).Sort("_id", 1) didn't work.

Thanks

Feature to sort first before limit

Description

  • The default settings for Aggregate() are inconvenient. There are needs to sort and limit throughout total documents, but now limit first and then sorting within it.
  • Example code
collection := client.Database("test_database").Collection("test_sort")
// insert 20 documents in order
for i := 1; i <= 20; i++ {
 	loc, _ := time.LoadLocation("Asia/Seoul")
 	now := primitive.NewDateTimeFromTime(time.Now().In(loc))
 	time.Sleep(100 * time.Millisecond)
 	newDoc := map[string]interface{}{
 		"name":       i,
 		"created_at": now,
 	}
 	if _, err := collection.InsertOne(context.Background(), newDoc); err != nil {
 		panic(err)
 	}
}
var queries []interface{}
var docs []map[string]interface{}
pqr := pg.New(collection).Limit(int64(10)).Page(int64(1)).Sort("created_at", -1) // sort desc
paginatedData, err := pqr.Aggregate(queries...)
if err != nil {
 	panic(err)
}
for _, raw := range paginatedData.Data {
 	doc := make(map[string]interface{})
 	if marshallErr := bson.Unmarshal(raw, &doc); marshallErr == nil {
 		docs = append(docs, doc)
 	}
}
for _, doc := range docs {
 	fmt.Printf("Name : %v CreatedAt : %v\n ", doc["name"], doc["created_at"])
}
  • Expected result with 20 coming first, but 10.

Suggestion

  • How about adding interface to sort first?
type pagingQuery struct {
	Collection  *mongo.Collection
	...
	SortFirst   bool // added
}

type PagingQuery interface {
	// Find set the filter for query results.
	Find() (paginatedData *PaginatedData, err error)
        ...
	SetSortFirst(sortFirst bool) PagingQuery // added
}

// SetSortFirst is function to set sortFirst field
func (paging *pagingQuery) SetSortFirst(sortFirst bool) PagingQuery { // added
	paging.SortFirst = sortFirst
	return paging
}

func (paging *pagingQuery) Aggregate(filters ...interface{}) (paginatedData *PaginatedData, err error) {
	// checking if user added required params
	if err := paging.validateQuery(false); err != nil {
		return nil, err
	}
	if paging.FilterQuery != nil {
		return nil, errors.New(FilterInAggregateError)
	}

	var aggregationFilter []bson.M
	// combining user sent queries
	for _, filter := range filters {
		aggregationFilter = append(aggregationFilter, filter.(bson.M))
	}
	skip := getSkip(paging.PageCount, paging.LimitCount)
	var facetData []bson.M
	facetData = setFacetDataInOrder(paging.SortFirst, paging.SortFields, skip, paging.LimitCount) // replaced

	//if paging.SortField != "" {
	//	facetData = append(facetData, bson.M{"$sort": bson.M{paging.SortField: paging.SortValue}})
	//}
	// making facet aggregation pipeline for result and total document count
	facet := bson.M{"$facet": bson.M{
		"data":  facetData,
		"total": []bson.M{{"$count": "count"}},
	},
..........
}

// setFacetDataInOrder return facet data by pre-set sortFirst option
func setFacetDataInOrder(sortFirst bool, sortFields bson.D, skip, limitCount int64) []bson.M { // added
	var facetData []bson.M
	if sortFirst {
		if len(sortFields) > 0 {
			facetData = append(facetData, bson.M{"$sort": sortFields})
		}
		facetData = append(facetData, bson.M{"$skip": skip})
		facetData = append(facetData, bson.M{"$limit": limitCount})
	} else {
		facetData = append(facetData, bson.M{"$skip": skip})
		facetData = append(facetData, bson.M{"$limit": limitCount})
		if len(sortFields) > 0 {
			facetData = append(facetData, bson.M{"$sort": sortFields})
		}
	}
	return facetData
}

Example code after change

var queries []interface{}
var docs []map[string]interface{}
pqr := pg.New(collection).Limit(int64(10)).Page(int64(1)).Sort("created_at", -1).SetSortFirst(true) // look here!
paginatedData, err := pqr.Aggregate(queries...)
if err != nil {
 	panic(err)
}
for _, raw := range paginatedData.Data {
 	doc := make(map[string]interface{})
 	if marshallErr := bson.Unmarshal(raw, &doc); marshallErr == nil {
 		docs = append(docs, doc)
 	}
}
for _, doc := range docs {
 	fmt.Printf("Name : %v CreatedAt : %v\n ", doc["name"], doc["created_at"])
}
  • If SetSortFirst() is not called, the default value is false , so the existing code is not affected.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.