elbywan / cryomongo Goto Github PK

So I may just be doing something dumb here, but I'm using https://kemalcr.com/ and initialize a single cryomongo connection that's then shared by every request (kemal uses fibers so there's only one executing at a time). What's worrisome is that over time Mongo Atlas reports that connections stack up until they hit the max allowed and then my app crashes and restarts.

I'm running at the DEBUG log level I'm not seeing any errors at all... could there be some sort of reconnect code in cryomongo that keeps old connections open but opens new ones perhaps?

Sorry there's not an easily way to immediately reproduce this issue... it stacks up over time. I'll investigate ways of making the problem show up faster in a test app...

Hey there! Great libraries - I discovered cryomongo and moongoon right before deploying a crystal app to production, and it's so much better I'm rewriting my app to use them instead of another ORM.

I'm actually adding a few things like validations, dirty field tracking, and more BSON::ObjectId id flexibility to moongoon (pull requests forthcoming), but during testing for that I ran into the current error (using the docker mongo 4.2.7 image):

mongo_1 | 2020-10-14T04:32:02.212+0000 I ACCESS [conn69] SASL SCRAM-SHA-256 authentication failed for tests on admin from client 172.18.0.1:38618 ; AuthenticationFailed: SCRAM authentication failed, storedKey mismatch
However, I've confirmed by using the shell that I'm using the correct password and the user exists in the admin database.

I've traced the problem to scram.cr, where Step 3 results in a failure. I've confirmed that @Iterations is set to 15000 (the default) and the @client_key contains data, so I'm not quite sure what the problem is... it seems to be generating a slightly malformed message.

When I enter:

username: tests
password: t

...it results in the following (for a single run.. as expected, the output does change each attempt):

auth_message;: n=tests,r=4XNLkk9/2lPlb8oXoiD13w==,r=4XNLkk9/2lPlb8oXoiD13w==4sfRkRvXuKt6v0ddInXSehaJ49sXoe1F,s=Z1msBKGAb6RJbh/73SgPH7Qq5y6ynXuHPPuwtQ==,i=15000,c=biws,r=4XNLkk9/2lPlb8oXoiD13w==4sfRkRvXuKt6v0ddInXSehaJ49sXoe1F

payload: Bytes[99, 61, 98, 105, 119, 115, 44, 114, 61, 52, 88, 78, 76, 107, 107, 57, 47, 50, 108, 80, 108, 98, 56, 111, 88, 111, 105, 68, 49, 51, 119, 61, 61, 52, 115, 102, 82, 107, 82, 118, 88, 117, 75, 116, 54, 118, 48, 100, 100, 73, 110, 88, 83, 101, 104, 97, 74, 52, 57, 115, 88, 111, 101, 49, 70, 44, 112, 61, 105, 51, 72, 82, 65, 53, 81, 72, 79, 71, 115, 115, 77, 83, 101, 105, 90, 87, 99, 109, 109, 122, 67, 112, 70, 107, 74, 52, 53, 85, 113, 65, 82, 122, 103, 86, 77, 112, 117, 101, 121, 115, 119, 61]

Is there any more debugging information I can provide that would be helpful?

Applications previously have worked fine with our standard configuration for several years, e.g.

Moongoon.connect("mongodb+srv://marcstein:[email protected]", database_name: "underwrite")

They are now generating an error

2021-11-27T17:58:47.523219Z INFO - moongoon: Connecting to MongoDB @ mongodb+srv://marcstein:[email protected]
Unhandled exception: Invalid uri: mongodb+srv://marcstein:[email protected], No SRV records found when querying url: underwrite.vbut1.mongodb.net (Mongo::Error)
from lib/cryomongo/src/cryomongo/uri/uri.cr:106:5 in 'parse'
from lib/cryomongo/src/cryomongo/client.cr:65:55 in 'initialize:connection_string:options:start_monitoring'
from lib/cryomongo/src/cryomongo/client.cr:60:5 in 'initialize'
from lib/cryomongo/src/cryomongo/client.cr:59:3 in 'new'
from lib/moongoon/src/database/database.cr:104:7 in 'connect:database_name'
from src/compnet.cr:37:1 in '__crystal_main'
from /opt/homebrew/Cellar/crystal/1.2.2/src/crystal/main.cr:110:5 in 'main_user_code'
from /opt/homebrew/Cellar/crystal/1.2.2/src/crystal/main.cr:96:7 in 'main'
from /opt/homebrew/Cellar/crystal/1.2.2/src/crystal/main.cr:119:3 in 'main'

Is this related to a version or driver mismatch?

Thanks!

Marc

The following lines probably don't do what they're supposed to do:

id : String just declares a local variable in the top level scope (actually in ID scope, but local variables inside types are shared with the top level).

Because of a compiler bug this causes programs using this shard to fail to build with --debug flag, see crystal-lang/crystal#9882

Not sure exactly how you'd want to implement this, but the new specific validation errors Mongo 5 gives are super helpful, so for my own use I've just added:

#Patch cryomongo /lib/cryomongo/src/cryomongo/error.cr : 164
if info = error["errInfo"]?.as?(BSON)
puts "Mongo errInfo: \n" + info.to_json
end

The errInfo is deeply nested an unpredictable, so to add it officially to the library should the Error perhaps have property info : BSON??

Hi, and thanks for your work ;)

I saw that you can instantiate the object with the database name directly in the MongoDB URL Mongo::Client.new("mongodb://address:port/database") but it seems that we need to retrieve the database first anyway with client["database_name"].

This is not really handy. Since the MongoDB URL can be passed from the command line then I need to parse the URL to know what is the name of the database I need to access.

Is there a way to retrieve the database that is set in the defined URL easily?

I'm getting a timeout error when I try to connect to an instance of an Azure CosmosDB. I'm not sure where to start trying to debug this but curious if you have any thoughts. My app is very simple. This is running in a simple Kemal app. The full app is below.

require "kemal"
require "cryomongo"

client = Mongo::Client.new ENV["MONGODB_URL"]

get "/featured_posts" do |env|
  database = client["blog"]
  postsCollection = database["posts"]
  projection = { title: 1, text: 1, createdAt: 1, updatedAt: 1 }
  posts = postsCollection.find(
    { title: { "$exists": true }},
    projection: projection,
    limit: 20,
    sort: { createdAt: 1 }
  ).to_a
  env.response.content_type = "application/json"
  posts.to_json
end

port = ENV["PORT"] || "80"
Kemal.run port.to_i

I see the following error when I try to load the page.

Timeout (00:00:30) reached while trying to select a suitable server with read preference primary.

Here is my shards.yml

name: Blog
version: 0.0.1

targets:
  app:
    main: src/app.cr
dependencies:
  kemal:
    github: kemalcr/kemal
  cryomongo:
    github: elbywan/cryomongo

crystal: 1.3.2

license: MIT

Cyromongo does not seem to respect the connect_timeout option. I set it to 10s but it still loops on retry after the time has elapsed:

2021-04-15T17:21:52.365633Z  ERROR - mongo: Monitoring handshake error: Error connecting to 'localhost:4001': Connection refused
2021-04-15T17:21:52.866395Z  ERROR - mongo: Monitoring handshake error: Error connecting to 'localhost:4001': Connection refused
2021-04-15T17:21:53.367184Z  ERROR - mongo: Monitoring handshake error: Error connecting to 'localhost:4001': Connection refused

...

2021-04-15T17:22:18.932129Z  ERROR - mongo: Monitoring handshake error: Error connecting to 'localhost:4001': Connection refused

Here's how I'm defining it:

Mongo::Options.new(
  connect_timeout: Time::Span.new(seconds: 10)
)

edit : forgot to drop the code!

I'm using mongoon (very cool tool, btw). The issue that I'm seeing is that each write to the Mongodb both opens files on the operating system (adding to the ulimit) and also keeps opening new connections to the Mongo server. Once I get to 1500 connections, the Mongo server maxes out and throws an error.

Is there a way to manually close these connections? I had thought that the pooling would handle this. Do I need to configure the pool to have it operate?

Thanks!