globalsign / mgo Goto Github PK
View Code? Open in Web Editor NEWThis project forked from domodwyer/mgo
The MongoDB driver for Go
License: Other
This project forked from domodwyer/mgo
The MongoDB driver for Go
License: Other
Hi,
Please is there a documentation for this on godoc?
Does this project build on top of go-mgo by Gustavo Niemier?
Is there a way to define aggreate options (and more precisely collation) by using .Pipe()
? Currently only AllowDisk
seems to be available.
When sess.SetPoolLimit(50)
not used many errors are occurring when mgo
is under stress like 10.000 concurrent connections.
When I limit the pool errors went away.
I've created a test-case source-code for this problem below, so you can test it in your own machine easily.
Do you have any suggestions about this behaviour?
package main
import (
"fmt"
"time"
mgo "github.com/globalsign/mgo"
"github.com/globalsign/mgo/bson"
)
// TODO: put some records into db first:
//
// use testapi
// db.competitions.insert([
// {game_id: 1, game_name: "foo"},
// {game_id: 2, game_name: "bar"},
// {game_id: 3, game_name: "jazz"}
// ])
// NOTE: you might want to increase this depending on your machine power
// mine is:
// MacBook (Retina, 12-inch, Early 2015)
// 1,2 GHz Intel Core M
// 8 GB 1600 MHz DDR3
const ops = 10000
type m bson.M
func main() {
sess, err := mgo.DialWithTimeout("localhost", time.Second)
if err != nil {
panic(err)
}
defer sess.Close()
// NOTE: without this setting there are many errors
// see the end of the file
// setting pool limit prevents most of the timeouts
// sess.SetPoolLimit(50)
// sess.SetSocketTimeout(60 * time.Second)
sess.SetMode(mgo.Monotonic, true)
time.Sleep(time.Second)
done := make(chan bool, ops)
for i := 0; i < ops; i++ {
go func() {
defer func() { done <- true }()
result, err := fetchSomething(sess)
if err != nil {
fmt.Printf("ERR: %s\n", err)
}
fmt.Printf("RESULT: %+v\n", result)
}()
}
for i := 0; i < ops; i++ {
<-done
}
}
func fetchSomething(sess *mgo.Session) ([]m, error) {
s := sess.Copy()
defer s.Close()
result := []m{}
group := m{
"$group": m{
"_id": m{
"id": "$game_id",
"name": "$game_name",
},
},
}
project := m{
"$project": m{
"_id": "$_id.id",
"name": "$_id.name",
},
}
sort := m{
"$sort": m{
"_id": 1,
},
}
err := col(s, "competitions").Pipe([]m{group, project, sort}).All(&result)
if err != nil {
return nil, err
}
return result, nil
}
// col is a helper for selecting a column
func col(sess *mgo.Session, name string) *mgo.Collection {
return sess.DB("testapi").C(name)
}
/*
ERRORS WITHOUT sess.SetPoolLimit(50)
$ go run socket_error.go
RESULT: [map[name:foo _id:1] map[_id:2 name:bar] map[_id:3 name:jazz]]
ERR: read tcp 127.0.0.1:52918->127.0.0.1:27017: read: connection reset by peer
ERR: write tcp 127.0.0.1:52084->127.0.0.1:27017: write: broken pipe
RESULT: []
RESULT: []
ERR: write tcp 127.0.0.1:53627->127.0.0.1:27017: write: broken pipe
ERR: write tcp 127.0.0.1:53627->127.0.0.1:27017: write: broken pipe
RESULT: []
ERR: write tcp 127.0.0.1:53627->127.0.0.1:27017: write: broken pipe
RESULT: []
ERR: write tcp 127.0.0.1:53627->127.0.0.1:27017: write: broken pipe
RESULT: []
ERR: write tcp 127.0.0.1:53627->127.0.0.1:27017: write: broken pipe
RESULT: []
ERR: write tcp 127.0.0.1:53627->127.0.0.1:27017: write: broken pipe
RESULT: []
RESULT: []
ERR: read tcp 127.0.0.1:53627->127.0.0.1:27017: read: connection reset by peer
RESULT: []
ERR: resetNonce: write tcp 127.0.0.1:53625->127.0.0.1:27017: write: broken pipe
RESULT: []
RESULT: [map[name:foo _id:1] map[_id:2 name:bar] map[_id:3 name:jazz]]
ERR: resetNonce: write tcp 127.0.0.1:54591->127.0.0.1:27017: write: broken pipe
RESULT: []
...
...
*/
Ported issue from go-mgo/mgo.
According to the documentation of mongodb 3.0, cursor.count() accept a boolean parameter named "applySkipLimit".
By default, calling cursor.count() will ignore the effects of skip() and limit().
If applySkipLimit is set to true: cursor.count(1), it will consider the effect of skip() and limit().
In mgo, Query.Count() doesn't accept a boolean parameter. It will also consider the effects of skip() and limit() by default, which is the opposite of the expected behavior.
Currently some features of MongoDB 3.4.x are implemented ( collations for example) but aren't documented ( Query.Collation(), views...), and some other aren't implemented at all (see #484 )
In addition, test aren't run against mongo 3.4.x in Travis CI, so it's not very clear whether this serie is supported or not.
So is there any plans to fully support Mongodb 3.4?
If nobody is working on this, I'm interested in doing it !
Usually, for unit tests in my projects, I add interface for the Collection and Query structures.
What if we will try to add that interfaces exactly in the driver?
I can implement it.
Additionally, I can implement mocks, because that structures will large.
It would be helpful for a testing not data access layer, controllers for example.
Hello !
I'm not sure if it's a normal behaviour, something to be aware of or a potential improvement.
Context:
I'm working with AIS data (positions of vessels delimited by time/area), so basically, on my database the document is about 20 fields (a mix of strings, integers and floats).
On my go project (doing stats for vessels) I do a request to get the a list of position/time for a vessel. What I really need here is just 3 fields so my type struct is as simple as Time, location and an integer which is a vessel identifiant.
Without a ".Select" of the 3 fields of interest :
A run grows up to 3gb in memory.
With a ".Select" of the 3 fields of interest (basically the select gets only the 3 fields I described on my struct):
.Select(bson.M{"location": 1, "SOG": 1, "RecvTime": 1})
A run grows up to 700mb (which is the volume of data I expected roughly)
Both runs get the data I need.
What I expect at the start:
My struct only have 3 fields (that I care about) but my database document can have 20 fields.
I thought it would have magically do the Select behind the scene so the database request is already optimised.
What I learned :
It's helpful to do a Select on the find to only get the data that really matters.
It's important to know it especially when you work with partial fields and not all the document.
Adding a select result in both lighter networking, lighter memory footprint, faster execution.
Maybe that's a rookie error, then, my bad tho.
Hi
I am a new to GO and its ecosystem. I was wondering if there is a way to run an embedded Mongo for integration test. I am aware of the globalsign/mgo/dbtest. I have configured an init_test.go to run the server prior to running my integration test. Unfortunately, the Mongo server that gets started by the dbserver.go runs on a random port and there is no way to communicate that port into the integration test.
Any directions would really be appreciated. Apologies if this kind of questions are not meant teo be published here, any forum or channel where we can post these kind of questions?
thanks in advance.
Ayache
Line 552 in 5be15cc
socket.Unlock()
must be before return, after socket.conn.Write()
I thinks the lock before socket.conn.write cause the problem: https://github.com/globalsign/mgo/blob/master/socket.go#L522
The unlock of this mutx is after the write opreation: https://github.com/globalsign/mgo/blob/master/socket.go#L561
Which caused the socket cannot read while writing.
NOTE: this issue is for an experimental feature, and does not affect the production driver.
The OP_MSG specification states:
Each OP_MSG MUST NOT exceed the maxMessageSizeBytes as configured by the MongoDB Handshake.
Currently the experimental support for OP_MSG (contributed by @feliixx - thanks!) does not gracefully handle a message exceeding maxMessageSizeBytes
- in the future this limit should be handled transparently by the driver.
In socket.go we return an error from sendMessage()
when the limit is exceeded - ideally the driver should split the requests into messages that do not exceed this limit internally.
If you have docs for the project I can add to the github main page and write some description.
I first time see this repo, so what about compatibility with mgov2? Did it save?
hi,anyone
how to use textsearch with mgo, in my case, i need to find two bson.M other one bson.M, insert a new filed score, and sort by score filed, i try it, the mgo and globalsign/mgo not support my case.
how to fix it ?
db.stores.find(
{ $text: { $search: "coffee shop cake" } },
{ score: { $meta: "textScore" } }
).sort( { score: { $meta: "textScore" } } )
my mongo version is 3.4
https://docs.mongodb.com/v3.4/core/text-search-operators/
Hi,
Every once in a while our mongo suite gets killed on TravicCI. We run go 1.10 and use docker for our test suites. Our Postgres and Neo4j test suites run just fine with this setup but with mgo and Mongo we're having these issues.
Stacktrace information can be found below. Any idea why this is happening?
+go test -v -race -coverprofile=coverage.out -covermode=atomic ./...
=== RUN TestMongoSuiteWithoutCredentials
2018/03/04 13:50:01 CREATING NEW POOL
2018/03/04 13:50:01 POOL CREATED <nil>
2018/03/04 13:50:01 RUNNING MONGO CONTAINER
2018/03/04 13:50:11 MONGO CONTAINER CREATED <nil>
2018/03/04 13:50:11 BEFORE testConnect
2018/03/04 13:50:11 START DialWithTimeout
2018/03/04 13:50:11 MONGO URL = mongodb://localhost:32768
SIGQUIT: quit
PC=0x474643 m=0 sigcode=0
goroutine 31 [syscall]:
runtime.notetsleepg(0x12be9e0, 0x37e09133e, 0x16)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/lock_futex.go:227 +0x42 fp=0xc420052760 sp=0xc420052730 pc=0x422022
runtime.timerproc(0x12be9c0)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/time.go:261 +0x2f9 fp=0xc4200527d8 sp=0xc420052760 pc=0x461889
runtime.goexit()
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200527e0 sp=0xc4200527d8 pc=0x472bd1
created by runtime.(*timersBucket).addtimerLocked
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/time.go:160 +0x107
goroutine 1 [chan receive]:
testing.(*T).Run(0xc42021c000, 0xd2d7a8, 0x20, 0xd41b80, 0xc4201e5c00)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc42021c000)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc42021c000, 0xc4201e5d48)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:777 +0x16e
testing.runTests(0xc4201378e0, 0x127b3e0, 0x1, 0x1, 0xc420160800)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc420160800, 0x0)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:978 +0x2ce
main.main()
_testmain.go:90 +0x325
goroutine 19 [syscall]:
os/signal.signal_recv(0x472bd1)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
/home/travis/.gimme/versions/go1.10.linux.amd64/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
/home/travis/.gimme/versions/go1.10.linux.amd64/src/os/signal/signal_unix.go:28 +0x4f
goroutine 20 [semacquire]:
sync.runtime_notifyListWait(0xc42023a6e8, 0xc400000000)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc42023a6d8)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/cond.go:56 +0x8e
github.com/globalsign/mgo.(*mongoCluster).AcquireSocket(0xc42023a6c0, 0x0, 0xc420240a01, 0x6fc23ac00, 0x6fc23ac00, 0x0, 0x0, 0x0, 0x1000, 0x1c5b320, ...)
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:644 +0xff
github.com/globalsign/mgo.(*Session).acquireSocket(0xc4202409c0, 0xb9e201, 0x0, 0x0, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:4853 +0x271
github.com/globalsign/mgo.(*Database).Run(0xc42017bc20, 0xc2ee40, 0xda60b0, 0x0, 0x0, 0x0, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:799 +0x5e
github.com/globalsign/mgo.(*Session).Run(0xc4202409c0, 0xc2ee40, 0xda60b0, 0x0, 0x0, 0xcf84e0, 0xc42023a6c0)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:2270 +0xba
github.com/globalsign/mgo.(*Session).Ping(0xc4202409c0, 0xc42023a6c0, 0x6fc23ac00)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:2299 +0x5d
github.com/globalsign/mgo.DialWithInfo(0xc4202c0000, 0x17, 0xc4202c0000, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:563 +0x566
github.com/globalsign/mgo.DialWithTimeout(0xc420026d20, 0x17, 0x6fc23ac00, 0x0, 0xc420167780, 0xc4200b0120)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:304 +0xc3
mongo_test.(*suite).testConnect(0xc42017bf48, 0xc42021c0f0)
/home/travis/build/qdentity/qdentity/go/src/mongo/mongo_test.go:36 +0xc8
mongo_test.TestMongoSuiteWithoutCredentials(0xc42021c0f0)
/home/travis/build/qdentity/qdentity/go/src/mongo/mongo_test.go:22 +0x187
testing.tRunner(0xc42021c0f0, 0xd41b80)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:824 +0x565
goroutine 30 [semacquire]:
sync.runtime_notifyListWait(0xc42023a6e8, 0xc400000001)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc42023a6d8)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/cond.go:56 +0x8e
github.com/globalsign/mgo.(*mongoCluster).AcquireSocket(0xc42023a6c0, 0x1, 0xc420240b01, 0x2540be400, 0x2540be400, 0x0, 0x0, 0x0, 0x1000, 0xc420082700, ...)
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:644 +0xff
github.com/globalsign/mgo.(*Session).acquireSocket(0xc420240b60, 0xc5f001, 0x0, 0x0, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:4853 +0x271
github.com/globalsign/mgo.(*Database).Run(0xc4200779b8, 0xc5f0c0, 0xc42000d200, 0xc10ec0, 0xc420232630, 0x0, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:799 +0x5e
github.com/globalsign/mgo.(*Session).Run(0xc420240b60, 0xc5f0c0, 0xc42000d200, 0xc10ec0, 0xc420232630, 0x0, 0x1)
/home/travis/gopath/src/github.com/globalsign/mgo/session.go:2270 +0xba
github.com/globalsign/mgo.(*mongoCluster).isMaster(0xc42023a6c0, 0xc4202c20f0, 0xc420232630, 0xc4202c20f0, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:182 +0x258
github.com/globalsign/mgo.(*mongoCluster).syncServer(0xc42023a6c0, 0xc4202c00e0, 0xd, 0xc42001ed20, 0xc4202c00e0, 0xc42023a6c0, 0xc440000000, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:231 +0x434
github.com/globalsign/mgo.(*mongoCluster).syncServersIteration.func1.1(0xc420292060, 0xc420026d2a, 0xd, 0xc420292070, 0xc420026d00, 0xc4202867b0, 0xc42023a6c0, 0xc4202867e0, 0xc420286810, 0x0, ...)
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:553 +0x1fb
created by github.com/globalsign/mgo.(*mongoCluster).syncServersIteration.func1
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:525 +0x175
goroutine 11 [semacquire]:
sync.runtime_Semacquire(0xc42029206c)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc420292060)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/waitgroup.go:129 +0xb3
github.com/globalsign/mgo.(*mongoCluster).syncServersIteration(0xc42023a6c0, 0x0)
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:582 +0x4c5
github.com/globalsign/mgo.(*mongoCluster).syncServersLoop(0xc42023a6c0)
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:390 +0x17c
created by github.com/globalsign/mgo.newCluster
/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:81 +0x2e3
goroutine 12 [sleep]:
time.Sleep(0x37e11d600)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/time.go:102 +0x146
github.com/globalsign/mgo.(*mongoServer).pinger(0xc4202c00e0, 0x479801)
/home/travis/gopath/src/github.com/globalsign/mgo/server.go:314 +0x7ad
created by github.com/globalsign/mgo.newServer
/home/travis/gopath/src/github.com/globalsign/mgo/server.go:89 +0x24b
goroutine 34 [IO wait]:
internal/poll.runtime_pollWait(0x7f50f3494f00, 0x72, 0x128aff0)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc420234e18, 0x72, 0xda9f00, 0x128aff0, 0xffffffffffffffff)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc420234e18, 0xc420028800, 0x24, 0x24)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc420234e00, 0xc420028840, 0x24, 0x24, 0x0, 0x0, 0x0)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc420234e00, 0xc420028840, 0x24, 0x24, 0x4ab9ed, 0xc420234e00, 0x0)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc42000e0c8, 0xc420028840, 0x24, 0x24, 0x0, 0xc4202c24b0, 0xc420062dc0)
/home/travis/.gimme/versions/go1.10.linux.amd64/src/net/net.go:176 +0x85
github.com/globalsign/mgo.fill(0xdb3660, 0xc42000e0c8, 0xc420028840, 0x24, 0x24, 0x0, 0x11)
/home/travis/gopath/src/github.com/globalsign/mgo/socket.go:567 +0x64
github.com/globalsign/mgo.(*mongoSocket).readLoop(0xc4202c24b0)
/home/travis/gopath/src/github.com/globalsign/mgo/socket.go:583 +0x15b
created by github.com/globalsign/mgo.newSocket
/home/travis/gopath/src/github.com/globalsign/mgo/socket.go:197 +0x341
rax 0xfffffffffffffffc
rbx 0x12bb3a0
rcx 0x474643
rdx 0x0
rdi 0x12be9e0
rsi 0x0
rbp 0xc4200526e8
rsp 0xc420052698
r8 0x0
r9 0x0
r10 0xc4200526d8
r11 0x202
r12 0xc420079c80
r13 0x12bb3a0
r14 0xc420001500
r15 0x1a354620
rip 0x474643
rflags 0x202
cs 0x33
fs 0x0
gs 0x0
*** Test killed with quit: ran too long (10m0s).
FAIL mongo 600.006s
The default behavior when JSON marshalling a bson.Raw
field doesn't print the data represented by the bson.Raw
field cleanly. Can a MarshalJSON
method be added to bson.Raw
which would allow the JSON marshaller to cleanly print a value in bson.Raw?
Current:
{
"Kind": 4,
"Data": "{something}"
}
Suggested:
{
"field": "value"
},
Example MarshalJSON:
// MarshalJSON serializes raw bson into JSON
func (raw Raw) MarshalJSON() ([]byte, error) {
var rawInterface interface{}
err := raw.Unmarshal(&rawInterface)
if err != nil {
return nil, errors.New("unable to unmarshal BSON data into interface")
}
return json.Marshal(rawInterface)
}
Hey guys, my co-workers and I are impressed by how well you're maintaining this project, and we are considering switching from the neglected go-mgo driver to your repo.
We have two quick questions if you don't mind:
We are wondering if you guys are keeping your repo up to date with new commits from 10gen/mgo, or do you work completely independently of them?
Do you know if there are any plans to classify this repo or the 10gen one as the "official" driver?
Thank you for taking them time to maintain this repo. We were relieved to find such an active and well maintained driver solution.
I notice there is a session, err := mgo.Dial(MONGODB_DIAL_URL)
function and also Clone
and Copy
.
If we are using MongoDB for a web app backend, I'm wondering what the best practice is out of these options.
Call Dial when we need to access database and close as soon as possible (usually with defer within same function).
Call Dial at start of request and close at end of request cycle after servicing request.
(Obviously For option 2, an improvement is to dial lazily but that's obvious to readers).
Assuming that we may need to do multiple database queries per request, how exactly do we use clone
and/or copy
?
One small bugbear I've had for a long time with MGO is that unlike Mapstructure you can't set the TagName to use on structures. The general use-case here is letting structures be decorated with json annotations, and then having both mapstructure, mgo and pretty much anything else all work off the same tags - so I don't have:
type SomeStruct struct {
SnakedField int `json:"snaked_field" mapstructure:"snaked_field" bson:"snaked_field"`
}
I've put an option in a fork of the code that enables this by adding a new global flag to the BSON parser, which can be set per-application. This allows for opting-in to using the JSON annotation if, and only if both the new option is enabled and there are no BSON annotations on the structure.
To 'truly' fix this at a BSON encoder level requires a major contract change, because it relies heavily on stateless global functions - so the notion of an encoder/decoder context would need to be added - and it'd break too much compatibility - unless I've missed something.
If folks are happy with it coming in, I can tweak it and submit the PR (or happy for someone to cherry-pick it) - it seems harmless enough and will aid mightily with struct-pollution in codebases. My main use case for MGO is as a provider that's one-of-many/pluggable - and not requiring every user to add 50 struct tags to a model to support all possible storage engines.
It should be nice to have context support. This is a long time requested feature on mgo.
I do not think this is a duplicate of #101 ,
we are running mongodb v3.4.10, and connections to the database seem to be completely unstable using the globalsign driver.
I created an ultra simple go app to showcase the issue:
func main() {
fmt.Println("Starting test")
session, err := mgo.Dial("mongodb://localhost:27017")
if err != nil {
panic(err)
}
defer session.Close()
fmt.Println("running")
wait()
}
func wait() os.Signal {
ch := make(chan os.Signal)
signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
return <-ch
}
when using the labix/mgo driver, there is no issue holding a connection, however when using globalsign/mgo, the connection lasts for about 10 seconds before closing and restarting. The mongod output looks like this
2018-02-12T13:48:03.014-0600 I NETWORK [thread1] connection accepted from 127.0.0.1:54685 #1 (1 connection now open)
2018-02-12T13:48:03.019-0600 I NETWORK [conn1] received client metadata from 127.0.0.1:54685 conn1: { driver: { name: "mgo", version: "globalsign" }, os: { architecture: "amd64", type: "darwin" } }
2018-02-12T13:48:34.530-0600 I - [conn1] end connection 127.0.0.1:54685 (1 connection now open)
2018-02-12T13:48:35.535-0600 I NETWORK [thread1] connection accepted from 127.0.0.1:54690 #2 (1 connection now open)
2018-02-12T13:48:35.536-0600 I NETWORK [conn2] received client metadata from 127.0.0.1:54690 conn2: { driver: { name: "mgo", version: "globalsign" }, os: { architecture: "amd64", type: "darwin" } }
2018-02-12T13:49:07.051-0600 I - [conn2] end connection 127.0.0.1:54690 (1 connection now open)
and will continue that loop ad infinitum. This is occurring on our production environment and creating a number of problems for us. Am I doing something wrong in the way I am initializing the connection?
we have new features like change streams
and lookup
in 3.6 https://docs.mongodb.com/manual/release-notes/3.6/
anybody plan to implement support?
Hi,
I have been tasked with setting up and testing go against mongodb (I am a dba). I need to prepare the basics for a developer to get productive. Are there docs on how to build this or use it in a simple CRUD app written in go?
Thanks!
In an attempt to prevent the ridiculous diffs on GitHub a new development
branch has been cut from the current master
head (896bbb8).
The old development branch is preserved as old-development
for the time being to allow anyone who had pinned to a commit to use it, and the existing changes from master have been cherry picked to the new development
branch so anyone using a branch name as a pin should also be covered.
This issue is here to point to if anything goes horribly wrong.
Currently we support collations, but there's no documentation!
https://godoc.org/github.com/globalsign/mgo#Query.Collation
Requires a quick PR adding documentation :)
Mongo 3.6 adds support for using DNS SRV
records to configure the initial set of ReplicaSet members instead of providing them in the connection string itself. See: https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format and https://github.com/mongodb/specifications/blob/master/source/initial-dns-seedlist-discovery/initial-dns-seedlist-discovery.rst
It would be great to see support for this in mgo!
https://github.com/globalsign/mgo/blob/master/session.go#L2925
This way if we want capped collections, we don't have the restriction of deployment of this driver to a 64bit machine
Is there any reason why we haven't published this through gopkg.in
? I'd make all API and documentations gathered in just one place while letting Gopkg manifests cleaner and less verbos.
MongoDB 3.4.X introduced support for read-only views - we should too!
Currently views aren't a priority, but they're considered a "nice to have".
Currently we run integration tests against the 2.6.X
, 3.0.X
and 3.2.X
releases of MongoDB.
We should also run against 3.4.X
- this is tricky as the test harness is difficult to work with.
If anyone wants to take a crack at it we're happy to host the binaries as we do for the other versions, see travis.yml - just leave a message below and we'll get the latest version uploaded.
Hi,
I am using version 3.4 of mongodb and its supporting db.collection.insertOne(), db.collection.inserMany()
As per requirement, I need to have the last insert id from the db. Mongodb is returning it but func (c *Collection) Insert(docs ...interface{}) error method is returning the error only, though I supposed to have the insertedId as well.
Hi, I am trying to implement authboss example using mgo
for example
storer > MemStorer.Create translates into
func (um UserManager) Create(id string, attr authboss.Attributes) error {
dbsession := um.Db.Copy()
defer dbsession.Close()
var user User
if err := attr.Bind(&user, true); err != nil {
return err
}
user.ID = bson.NewObjectId()
if err := dbsession.DB("tasksapp").C("users").Insert(&user); err != nil {
log.Fatal(err)
return err
}
fmt.Println("Create")
//spew.Dump(s.Users)
return nil
}
and when I run it, I get:
2018/03/17 22:49:36 [auth] Loading
2018/03/17 22:49:36 [confirm] Loading
2018/03/17 22:49:36 [lock] Loading
2018/03/17 22:49:36 [oauth2] Loading
2018/03/17 22:49:36 [recover] Loading
2018/03/17 22:49:36 [register] Loading
2018/03/17 22:49:36 [remember] Loading
2018/03/17 22:49:36 [auth] Route: /auth/login
2018/03/17 22:49:36 [auth] Route: /auth/logout
2018/03/17 22:49:36 [confirm] Route: /auth/confirm
2018/03/17 22:49:36 [oauth2] Route: /auth/oauth2/google
2018/03/17 22:49:36 [oauth2] Route: /auth/oauth2/callback/google
2018/03/17 22:49:36 [oauth2] Route: /auth/oauth2/logout
2018/03/17 22:49:36 [recover] Route: /auth/recover
2018/03/17 22:49:36 [recover] Route: /auth/recover/complete
2018/03/17 22:49:36 [register] Route: /auth/register
2018/03/17 22:49:36 listen tcp 127.0.0.1:3000: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
Hi,
I tried to answer that question myself but I can't figure out the code exactly. Do you happen to know the answer ?
Thanks !
Maybe it should be r2018.01.15
Create a CONTRIBUTING.md
file for the project.
Include at least test requirements, and that PRs should be targeted to development.
Now that go-mgo/mgo is unmaintained and pointing to this fork as a suggested alternative, perhaps it's time to start using semantic versioning (with a suitable starting version to avoid collisions with go-mgo/mgo). Semver should help ease the migration from go-mgo/mgo for those migrating over due to the cessation of maintenance.
So I use docker for mongo but I still build and run go in host machine, is there a way to do testing without actually installing mongod?
Currently, there's a rather unfortunate side effect of using this library in that you have to go pretty aggressively out of your way to use string-based IDs (especially in the case of microservices leveraging different DBs but using the same model library). One step toward improved support has been implemented in a PR against the originating fork: go-mgo#492
Yet more powerful might be the inclusion of a new tag that indicates that a given string field should be treated as an ObjectId for marshaling purposes.
Is there any resistance to the inclusion of this sort of feature?
Hi there.
I've been trying to make the geoNear
command work for me using this mgo fork, but somehow I am not happy with it.
I am using the db.Run(cmd, &r)
method for this purpose and the query is executing as expected. The problem is the result handling.
The result looks something like this:
map[stats:map[maxDistance:0 time:426 nscanned:3 objectsLoaded:0] ok:1 operationTime:6504358628374347777 $clusterTime:map[clusterTime:6504358628374347777 signature:map[hash:[77 13 197 18 149 243 243 102 160 148 122 23 81 135 43 40 227 229 141 95] keyId:6497522217260154881]] results:[]]
The problem is, that the result type provided in Run()
seems to apply to the full result above, rather than the results
section within. Is this expected? If so, any suggestions besides creating a struct for wrapping the results of a command?
Thanks in advance.
Marvin
Now that this repo is the new source of truth for mgo development, Will there be a dedicated site for non-godoc documentation? Something like http://labix.org/mgo but up to date with this repo?
Thanks for all the great work and continuing development on this library!
I'm running through this error after updating my codebase import from
"gopkg.in/mgo.v2/bson"
to "github.com/globalsign/mgo/bson"
.
The next func trigger as result an error/panic:
result argument must be a slice address
func Get(query, result interface{}) func(c *mgo.Collection) error {
return func(c *mgo.Collection) error {
return c.Find(query).One(result)
}
}
After a simple rollback, my code run well with the "gopkg.in/mgo.v2/bson"
import
I've updated the mgo library at the tag r2018.02.20
For debug purpose, here is the caller of the Get and the SearchBuilder implementation
The Get func is called by
//GetUserByLogin return database user
func GetUserByLogin(login string) (*User, error) {
var (
user *User
err error
)
if len(login) == 0 {
return nil, errors.New("Login or password empty")
}
err = database.SearchBuilder("User", database.Get(bson.M{"Login": bson.M{"$regex": bson.RegEx{Pattern: "^" + login + "$", Options: "i"}}}, &user))
if err != nil {
log.E(err.Error())
return nil, err
}
return user, err
}
func SearchBuilder(collection string, s func(*mgo.Collection) error) error {
App := app.GetInstance()
session := App.Session.Copy()
defer session.Close()
c := session.DB(App.Config.DatabaseName).C(collection)
return s(c)
}
Hi, we use in our application time.Duration with nanoseconds value and have noticed that using your driver it will round them to milliseconds. Is there any chance to store nanoseconds as well?
Thanks,
TeoV
If I run the tests (after successfully running make startdb), I consistently get cluster_test failures.
Is there anything I can do to make tests pass?
(running this against MongoDB 3.6.2)
----------------------------------------------------------------------
FAIL: cluster_test.go:527: S.TestModeMonotonicFallover
cluster_test.go:556:
c.Assert(err, Equals, io.EOF)
... obtained = nil
... expected *errors.errorString = &errors.errorString{s:"EOF"} ("EOF")
----------------------------------------------------------------------
FAIL: cluster_test.go:570: S.TestModeMonotonicWithSlaveFallover
Running serverStatus and isMaster with monotonic session
cluster_test.go:646:
// ... which is not the old one, since it's still dead.
c.Assert(ssresult.Host, Not(Equals), master)
... obtained string = "snazzy:40021"
... expected string = "snazzy:40021"
----------------------------------------------------------------------
FAIL: cluster_test.go:819: S.TestModePrimaryFallover
cluster_test.go:843:
c.Assert(err, ErrorMatches, "no reachable servers")
... value = nil
... regex string = "no reachable servers"
... Error value is nil
----------------------------------------------------------------------
FAIL: cluster_test.go:476: S.TestModePrimaryHiccup
cluster_test.go:510:
c.Assert(err, Equals, io.EOF)
... obtained = nil
... expected *errors.errorString = &errors.errorString{s:"EOF"} ("EOF")
----------------------------------------------------------------------
FAIL: cluster_test.go:776: S.TestModePrimaryPreferredFallover
cluster_test.go:797:
c.Assert(err, Equals, io.EOF)
... obtained = nil
... expected *errors.errorString = &errors.errorString{s:"EOF"} ("EOF")
----------------------------------------------------------------------
FAIL: cluster_test.go:717: S.TestModeSecondaryPreferredFallover
Waiting for cluster sync to finish...
cluster_test.go:759:
c.Assert(err, Equals, io.EOF)
... obtained = nil
... expected *errors.errorString = &errors.errorString{s:"EOF"} ("EOF")
----------------------------------------------------------------------
FAIL: cluster_test.go:435: S.TestModeStrongFallover
cluster_test.go:455:
c.Assert(err, Equals, io.EOF)
... obtained = nil
... expected *errors.errorString = &errors.errorString{s:"EOF"} ("EOF")
----------------------------------------------------------------------
FAIL: cluster_test.go:1819: S.TestNearestSecondary
cluster_test.go:1878:
c.Assert(hostPort(result.Host), Equals, hostPort(rs1b))
... obtained string = "40013"
... expected string = "40012"
----------------------------------------------------------------------
FAIL: cluster_test.go:1764: S.TestPrimaryShutdownOnAuthShard
cluster_test.go:1796:
c.Assert(err, Equals, io.EOF)
... obtained = nil
... expected *errors.errorString = &errors.errorString{s:"EOF"} ("EOF")
----------------------------------------------------------------------
FAIL: cluster_test.go:1491: S.TestRemovalOfClusterMember
cluster_test.go:1556:
c.Assert(err, NotNil)
... value = nil
----------------------------------------------------------------------
FAIL: cluster_test.go:1470: S.TestSecondaryModeWithMongosInsert
cluster_test.go:1488:
c.Assert(result.A, Equals, 1)
... obtained int = 0
... expected int = 1
----------------------------------------------------------------------
FAIL: cluster_test.go:986: S.TestSyncTimeout
cluster_test.go:1004:
c.Assert(err, ErrorMatches, "no reachable servers")
... value = nil
... regex string = "no reachable servers"
... Error value is nil
----------------------------------------------------------------------
OOPS: 231 passed, 19 skipped, 12 FAILED
--- FAIL: TestAll (517.10s)
The isMaster request fails in syncServersIteration()
, which is repeatedly executed, and disconnects from the existing mongodb server and repeats the connection again.
This causes user requests to mongodb to be periodically delayed.
It seems to occur when an isMaster request is made through a pooled mongodb connection.
2018/02/07 19:12:33 Cluster 0xc4202cc900 acquired (refs=3)
2018/02/07 19:12:33 New session 0xc4203ed380 on cluster 0xc4202cc900
2018/02/07 19:12:33 Session 0xc4203ed380: setting mode 1 with refresh=true (master=0x0, slave=0x0)
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: serializing op: &mgo.queryOp{query:bson.D{bson.DocElem{Name:"isMaster", Value:1}, bson.DocElem{Name:"client", Value:bson.M{"driver":bson.M{"name":"mgo", "version":"globalsign"}, "os":bson.M{"type":"linux", "architecture":"amd64"}}}}, collection:"admin.$cmd", serverTags:[]bson.D(nil), selector:interface {}(nil), replyFunc:(mgo.replyFunc)(0x869280), mode:1, skip:0, limit:-1, options:mgo.queryWrapper{Query:interface {}(nil), OrderBy:interface {}(nil), Hint:interface {}(nil), Explain:false, Snapshot:false, ReadPreference:bson.D(nil), MaxScan:0, MaxTimeMS:0, Comment:"", Collation:(*mgo.Collation)(nil)}, hasOptions:false, flags:0x4, readConcern:""}
2018/02/07 19:12:33 final query is &mgo.queryWrapper{Query:bson.D{bson.DocElem{Name:"isMaster", Value:1}, bson.DocElem{Name:"client", Value:bson.M{"driver":bson.M{"name":"mgo", "version":"globalsign"}, "os":bson.M{"architecture":"amd64", "type":"linux"}}}}, OrderBy:interface {}(nil), Hint:interface {}(nil), Explain:false, Snapshot:false, ReadPreference:bson.D{bson.DocElem{Name:"mode", Value:"secondaryPreferred"}}, MaxScan:0, MaxTimeMS:0, Comment:"", Collation:(*mgo.Collation)(nil)}
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: sending 1 op(s) (235 bytes)
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: updated write deadline to 5s ahead (2018-02-07 19:12:38.348446658 +0900 KST)
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: updated read deadline to 5s ahead (2018-02-07 19:12:38.348473501 +0900 KST)
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: got reply (187 bytes)
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: received document: bson.M{"ok":0, "errmsg":"The client metadata document may only be sent in the first isMaster", "code":186, "codeName":"ClientMetadataCannotBeMutated"}
2018/02/07 19:12:33 Run command unmarshaled: mgo.queryOp{query:bson.D{bson.DocElem{Name:"isMaster", Value:1}, bson.DocElem{Name:"client", Value:bson.M{"driver":bson.M{"name":"mgo", "version":"globalsign"}, "os":bson.M{"type":"linux", "architecture":"amd64"}}}}, collection:"admin.$cmd", serverTags:[]bson.D(nil), selector:interface {}(nil), replyFunc:(mgo.replyFunc)(0x869280), mode:1, skip:0, limit:-1, options:mgo.queryWrapper{Query:bson.D{bson.DocElem{Name:"isMaster", Value:1}, bson.DocElem{Name:"client", Value:bson.M{"driver":bson.M{"name":"mgo", "version":"globalsign"}, "os":bson.M{"type":"linux", "architecture":"amd64"}}}}, OrderBy:interface {}(nil), Hint:interface {}(nil), Explain:false, Snapshot:false, ReadPreference:bson.D{bson.DocElem{Name:"mode", Value:"secondaryPreferred"}}, MaxScan:0, MaxTimeMS:0, Comment:"", Collation:(*mgo.Collation)(nil)}, hasOptions:true, flags:0x4, readConcern:""}, result: bson.M{"ok":0, "errmsg":"The client metadata document may only be sent in the first isMaster", "code":186, "codeName":"ClientMetadataCannotBeMutated"}
2018/02/07 19:12:33 Closing session 0xc4203ed380
2018/02/07 19:12:33 unset master socket from session 0xc4203ed380
2018/02/07 19:12:33 Cluster 0xc4202cc900 released (refs=2)
2018/02/07 19:12:33 SYNC Command 'ismaster' to 10.101.216.180:27017 failed: The client metadata document may only be sent in the first isMaster
2018/02/07 19:12:33 Connections to 10.101.216.180:27017 closing (1 live sockets).
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: closing: Closed explicitly (abend=false)
2018/02/07 19:12:33 Socket 0xc420660000 to 10.101.216.180:27017: idle and close.
2018/02/07 19:12:33 Removed server 10.101.216.180:27017 from cluster.
related mongodb changes
ClientMetadataCannotBeMutated
error is returned.Affected mongo versions
The contributions guidelines states that
Code should pass
golint
,go vet
andgo fmt
Currently, 6 files aren't formatted with go fmt
, golint
returns 126 warnings and go vet
returns 486 warnings!
It would be nice to fix this in order to run gloint/go vet against new code direclty in travis
If you're interrested in this I'll send a PR!
I believe there's no way right now to watch a change stream with mgo. It would be great if this feature could be added.
I noticed a branch with some work started on this. I'm bringing it up-to-date with master and will start testing with that.
While running unit tests with go 1.9.4 on x86_64 for release 2018.02.20
+ go test -buildmode pie -compiler gc -ldflags ' -X github.com/globalsign/mgo/version.tag=r2018.02.20 -X github.com/globalsign/mgo/version=2018.02.20 -extldflags '\''-Wl,-z
,relro '\'''
----------------------------------------------------------------------
PANIC: dbserver_test.go:89: S.TestCheckSessions
... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x55E276888F1C)
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
/usr/lib/golang/src/runtime/panic.go:63
in panicmem
/usr/lib/golang/src/runtime/signal_unix.go:367
in sigpanic
/usr/lib/golang/src/os/exec_unix.go:56
in Process.signal
/usr/lib/golang/src/os/exec.go:121
in Process.Signal
dbserver.go:114
in DBServer.Stop
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
dbserver.go:72
in DBServer.start
dbserver.go:130
in DBServer.Session
dbserver_test.go:94
in S.TestCheckSessions
/usr/lib/golang/src/reflect/value.go:302
in Value.Call
/usr/lib/golang/src/runtime/asm_amd64.s:2337
in goexit
----------------------------------------------------------------------
PANIC: dbserver_test.go:99: S.TestCheckSessionsDisabled
... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x55E276888F1C)
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
/usr/lib/golang/src/runtime/panic.go:63
in panicmem
/usr/lib/golang/src/runtime/signal_unix.go:367
in sigpanic
/usr/lib/golang/src/os/exec_unix.go:56
in Process.signal
/usr/lib/golang/src/os/exec.go:121
in Process.Signal
dbserver.go:114
in DBServer.Stop
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
dbserver.go:72
in DBServer.start
dbserver.go:130
in DBServer.Session
dbserver_test.go:107
in S.TestCheckSessionsDisabled
/usr/lib/golang/src/reflect/value.go:302
in Value.Call
/usr/lib/golang/src/runtime/asm_amd64.s:2337
in goexit
----------------------------------------------------------------------
PANIC: dbserver_test.go:58: S.TestStop
... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x55E276888F1C)
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
/usr/lib/golang/src/runtime/panic.go:63
in panicmem
/usr/lib/golang/src/runtime/signal_unix.go:367
in sigpanic
/usr/lib/golang/src/os/exec_unix.go:56
in Process.signal
/usr/lib/golang/src/os/exec.go:121
in Process.Signal
dbserver.go:114
in DBServer.Stop
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
dbserver.go:72
in DBServer.start
dbserver.go:130
in DBServer.Session
dbserver_test.go:67
in S.TestStop
/usr/lib/golang/src/reflect/value.go:302
in Value.Call
/usr/lib/golang/src/runtime/asm_amd64.s:2337
in goexit
----------------------------------------------------------------------
PANIC: dbserver_test.go:35: S.TestWipeData
... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x55E276888F1C)
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
/usr/lib/golang/src/runtime/panic.go:63
in panicmem
/usr/lib/golang/src/runtime/signal_unix.go:367
in sigpanic
/usr/lib/golang/src/os/exec_unix.go:56
in Process.signal
/usr/lib/golang/src/os/exec.go:121
in Process.Signal
dbserver.go:114
in DBServer.Stop
/usr/lib/golang/src/runtime/panic.go:491
in gopanic
dbserver.go:72
in DBServer.start
dbserver.go:130
in DBServer.Session
dbserver_test.go:40
in S.TestWipeData
/usr/lib/golang/src/reflect/value.go:302
in Value.Call
/usr/lib/golang/src/runtime/asm_amd64.s:2337
in goexit
--- FAIL: TestAll (0.00s)
FAIL
OOPS: 0 passed, 4 PANICKED
exit status 1
FAIL github.com/globalsign/mgo/dbtest 0.011s
A unit test should always just work with go test and report sensible results
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.