Getting the below error almost immediately when running. MongoDB 2.2

Options -w -p -n "my_namespace.*" mongodb://....

Total entries read: 0
Understood entries: 0
Unique recommendations: 0
Entries impacted by recommendations: 0
{
    "linesPassed": 0, 
    "linesRecommended": 0, 
    "results": [], 
    "uniqueRecommendations": 0, 
    "linesProcessed": 0
}
Traceback (most recent call last):
  File "/usr/local/share/python/dex", line 147, in <module>
    main(sys.argv[1:])
  File "/usr/local/share/python/dex", line 129, in main
    return md.watch_profile()
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dex/dex.py", line 194, in watch_profile
    for profile_entry in self._tail_profile(db, WATCH_INTERVAL_SECONDS):
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dex/dex.py", line 286, in _tail_profile
    for doc in cursor:
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/cursor.py", line 778, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/cursor.py", line 729, in _refresh
    self.__uuid_subtype))
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/cursor.py", line 686, in __send_message
    self.__uuid_subtype)
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/helpers.py", line 111, in _unpack_response
    as_class, tz_aware, uuid_subtype)
InvalidBSON: bad eoo

{
    'queryMask': '{"$query":{"$OR":[{"created_at":{"$gt":"<val>"}},{"launch_date":{"$gt":"<val>"}}]}}',
    'namespace': 'angellist.startups',
    'recommendation': {
        'index': '{"$OR": 1}',
        'namespace': 'angellist.startups',
        'shellCommand': 'db["startups"].ensureIndex({"$OR": 1}, {"background": true})'
    },
    'details': {
        'count': 1,
        'totalTimeMillis': 180,
        'avgTimeMillis': 180
    }
},

Not sure that's a desired behavior. Am I just using this incorrectly?

It would be useful if Dex were to add one (or a configurable number of) non-anonymized query samples in it's output, so that we can easily do an explain() and further optimisation ourselves on queries where Dex doesn't make recommendations.

Dex does not recognize "locks(micros)" as part of a query, and invalidates a query as unrecognizable.

Example:

kyle@db-stage-1:~$ cat test.log.orig
Wed Aug 29 00:02:49 [conn13221] command production-squash.$cmd command: { count: "leaderboard_triple_highscore_weekly20120826", query: { score: { $gt: 66610 }, ts: { $gte: new Date(1345964400000) } } } ntoreturn:1 reslen:48 0ms
Mon Sep 24 18:17:39 [conn1318] command staging-kiwi.$cmd command: { count: "analytics_playerdaysByRef", query: { value.dates: "20120826" } } ntoreturn:1 reslen:48 0ms
Mon Sep 24 18:17:39 [conn1318] command staging-kiwi.$cmd command: { count: "analytics_playerdays", query: { value.dates: "20120923" } } ntoreturn:1 reslen:48 0ms

kyle@db-stage-1:~$ dex -v -f test.log.orig mongodb://127.0.0.1:27017
Total entries read: 3
Understood entries: 2
Unique recommendations: 2

Entries impacted by recommendations: 2

---- When I remove " locks(micros)" it recognizes and understands the entry. ----

kyle@db-stage-1:$ cat test.log
Wed Aug 29 00:02:49 [conn13221] command production-squash.$cmd command: { count: "leaderboard_triple_highscore_weekly20120826", query: { score: { $gt: 66610 }, ts: { $gte: new Date(1345964400000) } } } ntoreturn:1 reslen:48 0ms
Mon Sep 24 18:17:39 [conn1318] command staging-kiwi.$cmd command: { count: "analytics_playerdaysByRef", query: { value.dates: "20120826" } } ntoreturn:1 reslen:48 0ms
Mon Sep 24 18:17:39 [conn1318] command staging-kiwi.$cmd command: { count: "analytics_playerdays", query: { value.dates: "20120923" } } ntoreturn:1 keyUpdates:0 r:125 reslen:48 0ms
kyle@db-stage-1:$

kyle@db-stage-1:~$ dex -v -f test.log mongodb://127.0.0.1:27017
Total entries read: 3
Understood entries: 3
Unique recommendations: 3
Entries impacted by recommendations: 3

kyle@db-stage-1:~$

Can I use dex to profile against a secondary in a replica set? I'm currently getting errors trying to do that:

Total entries read: 0
Understood entries: 0
Unique recommendations: 0
Entries impacted by recommendations: 0
{
    "linesPassed": 0, 
    "linesRecommended": 0, 
    "results": [], 
    "uniqueRecommendations": 0, 
    "linesProcessed": 0
}
Traceback (most recent call last):
  File "/usr/local/share/python/dex", line 147, in <module>
    main(sys.argv[1:])
  File "/usr/local/share/python/dex", line 129, in main
    return md.watch_profile()
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dex/dex.py", line 194, in watch_profile
    for profile_entry in self._tail_profile(db, WATCH_INTERVAL_SECONDS):
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dex/dex.py", line 279, in _tail_profile
    latest_doc = db['system.profile'].find_one()
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/collection.py", line 516, in find_one
    for result in self.find(spec_or_id, *args, **kwargs).limit(-1):
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/cursor.py", line 778, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/cursor.py", line 729, in _refresh
    self.__uuid_subtype))
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/cursor.py", line 686, in __send_message
    self.__uuid_subtype)
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/helpers.py", line 102, in _unpack_response
    raise AutoReconnect("master has changed")
AutoReconnect: master has changed

When I run dex during mongodump i get a recommendation to insert below index into my collections.

{
    "key" : {
        "query" : 1,
        "$snapshot" : 1
    },
    "collection" : "reportDetail"
},

Feedback from MongoDB support on the recommended index:

Dex is confused and wrong. That is not an index you should create. You may want to file an issue with them if you are active there.

I think dex is wonderful and I really like it to look through the logfiles to make optimizations. Sadly, It doesn't support ssl to connect to mongodb which would allow dex to learn from setup indexes. Is there a (undocumented) way to use ssl (with PEMKeyFile) or is this planned for dex?
thanks for your work!

I tried running dex on a Mongo shard log I've got and noticed that it missed a bunch of findAndModify lines. It turns out that the log has "findandmodify" instead of "findAndModify" for the command name. I hacked my copy of parsers.py locally to account for that one special case, but I'm not sure if something more comprehensive should be done. I can open a PR with my changes if it helps, but I'm not sure they're exactly what they should be (in particular, my installed version doesn't have SUPPORTED_COMMANDS in analyzer.py).

I have a collection, "features", with 3 fields, name, active, weight,

I query the collection by weight descending:

db.features.find({active:true},{name:1, weight:1}).sort({weight:-1})

Dex gives me suggestion:

db['features'].ensureIndex({'active': 1, 'weight': 1}, {'background': true})

While I thought I should create index on weight descend:

db['features'].ensureIndex({'active': 1, 'weight': -1}, {'background': true})

I have post a question on Stackoverflow
in more details, would you please check it out?

As this:

"errmsg" : "namespace name generated from index name "test.task.$_id_1__class_1_uuid_1_name_1_description_1_status_1_imageName_1_imageWidth_1_imageHeight_1_expiredTime_1_batchUuid_1_requesterId_1_businessId_1_originalImagePath_1_originalSheetImageName_1_qc_1_reviewed_1_qcResult_1_templateName_1_priority_1_createtime_1_createdBy_1" is too long (127 byte max)"

I'm getting permissions errors when I use dex now. Is specifying the auth database supported?

I get the following error when running dex against the profile collections. Profiling is enabled across the entire DB at the command line level.

db version v2.2.0, pdfile version 4.5
Thu Sep 20 12:04:51 git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207

kyle@db-stage-1:~$ pip freeze dex
Warning: cannot find svn location for distribute==0.6.16dev-r0
BeautifulSoup==3.2.1
Cheetah==2.4.4
Dex==0.4.1
GnuPGInterface==0.3.2
Landscape-Client==11.07.1.1
M2Crypto==0.20.1
PAM==0.4.2
PyYAML==3.10
Twisted-Core==11.0.0
Twisted-Web==11.0.0
apt-xapian-index==0.44
argparse==1.2.1
boto==2.0
chardet==2.0.1
cloud-init==0.6.1
command-not-found==0.2.44
configobj==4.7.2
dargparse==0.2.1

FIXME: could not find svn URL in dependency_links for this package:

distribute==0.6.16dev-r0
euca2ools==2.0.0
httplib2==0.7.1
iotop==0.4.3
keyring==0.6.2
launchpadlib==1.9.8
lazr.restfulclient==0.11.2
lazr.uri==1.0.2
oauth==1.0.1
paramiko==1.7.7.1
pexpect==2.3
pyOpenSSL==0.12
pycrypto==2.3
pycurl==7.19.0
pymongo==2.3
pyserial==2.5
python-apt==0.8.0ubuntu9
python-debian==0.1.20ubuntu2
simplejson==2.1.6
smart==1.4
stripogram==1.5
ufw==0.30.1-2ubuntu1
unattended-upgrades==0.1
wadllib==1.2.0
wsgiref==0.1.2
zope.interface==3.6.1
kyle@db-stage-1:~$

kyle@db-stage-1:$ dex -p mongodb://127.0.0.1:27019
{
"index": "{'fb_token': 1, 'expires': 1}",
"namespace": "staging-grabauth.tokens",
"shellCommand": "db['tokens'].ensureIndex({'fb_token': 1, 'expires': 1}, {'background': true})"
}
Traceback (most recent call last):
File "/usr/local/bin/dex", line 147, in
main(sys.argv[1:])
File "/usr/local/bin/dex", line 127, in main
return md.analyze_profile()
File "/usr/local/lib/python2.7/dist-packages/dex/dex.py", line 150, in analyze_profile
for profile_entry in profile_entries:
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 778, in next
if len(self.data) or self._refresh():
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 742, in _refresh
limit, self.__id))
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 686, in __send_message
self.__uuid_subtype)
File "/usr/local/lib/python2.7/dist-packages/pymongo/helpers.py", line 111, in _unpack_response
as_class, tz_aware, uuid_subtype)
File "/usr/local/lib/python2.7/dist-packages/bson/__init.py", line 515, in decode_all
tz_aware, uuid_subtype))
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 325, in _elements_to_dict
tz_aware, uuid_subtype)
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 315, in _element_to_dict
tz_aware, uuid_subtype)
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 147, in _get_string
return _get_c_string(data, position, length)
File "/usr/local/lib/python2.7/dist-packages/bson/init.py", line 114, in _get_c_string
value = data[position:end].decode("utf-8")
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x90 in position 7: invalid start byte
kyle@db-stage-1:$

In some cases the order of fields in an orderby clause is not being respected in the final recommendation.

To be safe if you use compound orderbys, the current workaround is to run dex in -v mode, and note the "s" field of the queriesCovered array. If the s document contains multiple fields, be sure to verify your index recommendation against the query itself.

There is always such an index (as per http://docs.mongodb.org/manual/core/indexes/#index-type-id ) except for local capped databases.

I'm not sure if this is a version thing or not, but I had to tweak the regex for the StandardQueryHandler to make it recognize any lines from the log file. It currently looks for 20 characters at the beginning of the line before the connection number. The lines in our log had more than 20 characters, so they never matched. Changing it to a simple .* worked.

Hello,

My name is Brad. I have used Dex quite a bit, but I found it frustrating that I would have to manually hand-tune the indexes after wards. Dex just doesn't go far enough.

The one particular feature that is really missing from Dex is the ability to automatically sort its recommended indexes by the cardinality of its fields.

After becoming very frustrated having to hand tune the results every time, I eventually created my own tool.

https://bitbucket.org/sensibill/mongo-dynamic-indexer/overview

I would really suggest you guys at Mongolab come check it out. I have developed an algorithm that can automatically sort indexes by their cardinality, and a whole lot more. I hope we can collaborate. Either way, I recommend field cardinality sorting as a feature for Dex.

Is this supposed to happen? I run db.collection.getIndexes() and it outputs indexes that Dex has suggested to me, again without the --nocheck flag. Is this the correct behavior? What do I do when this happens? It negates the potential benefit I might receive from adding the index, since the index is already there.

I'm having some trouble getting it installed. I originally tried pip install dex but it didn't end up in my path afterwards. And running it from build/dex/bin/dex gave a permission denied.

I then tried sudo pip install dex and got this with errors: https://gist.github.com/barmstrong/5a67ac1417fed7488b82

dex is now in my path but I'm guessing it didn't install correctly. When I try to run it I get errors:

$ dex -p mongodb://USERNAME:[email protected]:12345/my_db
Traceback (most recent call last):
  File "/usr/local/bin/dex", line 184, in <module>
    main(sys.argv[1:])
  File "/usr/local/bin/dex", line 160, in main
    return md.analyze_profile()
  File "/Library/Python/2.7/site-packages/dex/dex.py", line 135, in analyze_profile
    connection = pymongo.Connection(self._db_uri,document_class=OrderedDict)
  File "/Library/Python/2.7/site-packages/pymongo/connection.py", line 180, in __init__
    max_pool_size, document_class, tz_aware, _connect, **kwargs)
  File "/Library/Python/2.7/site-packages/pymongo/mongo_client.py", line 278, in __init__
    raise ConfigurationError("authentication failed")
ConfigurationError: authentication failed
$

Any ideas? This is on max os. I have the default install python 2.7.2 which is prob a bit old. A homebrew install would make things easier, but just one idea.

I tried install using python 3.4.1:

pip install dex
Downloading/unpacking dex
  Downloading Dex-0.6.tar.gz
  Running setup.py (path:...build/dex/setup.py) egg_info for package dex

Requirement already satisfied (use --upgrade to upgrade): pymongo in .../lib/python3.4/site-packages (from dex)
Downloading/unpacking pyyaml (from dex)
  Downloading PyYAML-3.11.tar.gz (248kB): 248kB downloaded
  Running setup.py (path:...build/pyyaml/setup.py) egg_info for package pyyaml

Downloading/unpacking dargparse>=0.2.2 (from dex)
  Downloading dargparse-0.2.5.tar.gz
  Running setup.py (path:..../build/dargparse/setup.py) egg_info for package dargparse
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "..../build/dargparse/setup.py", line 31
        except Exception, ex:
                        ^
    SyntaxError: invalid syntax
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "..../build/dargparse/setup.py", line 31

    except Exception, ex:

                    ^

SyntaxError: invalid syntax

----------------------------------------

[ ]'s

We are running dex on our production dataset but we are not getting the right indexes when a BinData field is used.

For example, for the following log line:

2016-03-14T15:33:49.177+0000 I WRITE    [conn826040] update collection query: { _id: BinData(0, 6165363837656337373438316233343862333033363464663961336231353262) } update: { $set: { a: 1 } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:0 keyUpdates:0 writeConflicts:0 numYields:1 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } 1440ms

We get the following suggestion:

        {
            'queryMask': '{"$query":{"6165363837656337373438316233343862333033363464663961336231353262)":"<val>","_id":"<val>"}}', 
            'namespace': 'db.collection', 
            'recommendation': {
                'index': '{"_id": 1, "6165363837656337373438316233343862333033363464663961336231353262)": 1}', 
                'namespace': 'db.collection', 
                'shellCommand': 'db["collection"].ensureIndex({"_id": 1, "6165363837656337373438316233343862333033363464663961336231353262)": 1}, {"background": true})'
            }, 
            'details': {
                'count': 1, 
                'totalTimeMillis': 1440, 
                'avgTimeMillis': 1440
            }
        }, 

Bot the commands:

1. dex -f /XXX/XXX/mongodb/var/log/mongodb/mongod.log mongodb://root:[email protected]:27017/databaseName

2. dex -f /mnt/pb1d1p1/mongodb/var/log/mongodb/mongod.log

yield:

Traceback (most recent call last):
File "/usr/local/bin/dex", line 233, in
main(sys.argv[1:])
File "/usr/local/bin/dex", line 218, in main
return md.analyze_logfile(options.logfile_path)
File "/usr/local/lib/python2.7/dist-packages/dex/dex.py", line 212, in analyze_logfile
self.analyze_logfile_object(file)
File "/usr/local/lib/python2.7/dist-packages/dex/dex.py", line 230, in analyze_logfile_object
for line in file_object:
SystemError: Negative size passed to PyString_FromStringAndSize

How do I fix this?

Hi,

This wonderful script have a lack; we need python, dex and pymongo.
As this is not possible on production environments, I have mongoexported the system.profile collection, but I cannot mongoimport it with the same name, as it's a system collection.

Thu Oct  3 11:29:20 [conn1] ERROR: attempt to insert in system namespace UserConcurrentControl.system.profile

I restored it as a collecion "profile", but dex is unable to look at other collection to find the same data. As I know enough python, I modified the dex.py module, and I succeeded.

It would be nice to have this collection name as an optional parameter, so anyone could benefit from it.

Thanks

Is this tool still ok for Mongo 3.0 and 3.2?

Is there a better tool to find slow queries?

thanks

I am having an issue running dex against a replica set. I am running it on the server hosting the primary and have tried using multiple variations of the the connection string for mongodb, using a user defined in the admin db, using a user defined in the local db i want to run dex against, using the proper replica set connection string requried by mongo, and just trying to connect as localhost, and all result in 'OperationFailure: Authentication failed.. Please see below example below :

 dex -n 'mydb.*' -p mongodb://myuser:[email protected]/mydb?authSource=admin

 dex -n 'mydb.*' -p mongodb://myuser:mypwd@localhost/mydb?authSource=admin

dex -n 'mydb.*' -p mongodb://myuser:mypwd@*.*.*.*:27017,*.*.*.*:27017,*.*.*.*:27017/mydb?authSource=admin&replicaSet=rs1

Hi,
While we tried to run dex in the below format using profiling, we are getting "TypeError: expected string or buffer" error. Pls guide us on the same.
command used:
dex -p 'mongodb://adminUser:adminPwd#[email protected]:27017/admin' -n "MyDB.*"
Error:
Traceback (most recent call last):
File "/usr/bin/dex", line 233, in
main(sys.argv[1:])
File "/usr/bin/dex", line 206, in main
return md.analyze_profile()
File "/usr/lib/python2.6/site-packages/dex/dex.py", line 162, in analyze_profile
profile_parser)
File "/usr/lib/python2.6/site-packages/dex/dex.py", line 88, in _process_query
line_time = get_line_time(input)
File "/usr/lib/python2.6/site-packages/dex/parsers.py", line 51, in get_line_time
match = ts_rx.match(line)
TypeError: expected string or buffer

Pls note that, it is working fine when targeted to a log file.

Parsing a Mongo 2.6.4 log file always seems to fail. For example:

2014-08-26T16:38:23.309+0200 [conn440] command akn.$cmd command: count { count: "content", query: { $and: [ { document.$id: "8d7b03e32414310c8b5684048d2ab114", isPublished: true, class: { $in: [ "Akn\Bundle\ContentBundle\Document\Rating" ] } }, { $or: [ { disabled: null }, { disabled: false } ] } ] } } planSummary: IXSCAN { class: 1, createdAt: -1 }, IXSCAN { document.$id: 1 } keyUpdates:0 numYields:2 locks(micros) r:1146670 reslen:48 671ms
2014-08-26T16:38:23.551+0200 [conn451] command akn.$cmd command: count { count: "content", query: { $and: [ { document.$id: "010c29056f623658a119522cb04e8edb", isPublished: true, class: { $in: [ "Akn\Bundle\ContentBundle\Document\Rating" ] } }, { $or: [ { disabled: null }, { disabled: false } ] } ] } } planSummary: IXSCAN { class: 1, createdAt: -1 }, IXSCAN { document.$id: 1 } keyUpdates:0 numYields:1 locks(micros) r:1013025 reslen:48 639ms
2014-08-26T16:38:45.153+0200 [conn444] query akn.content query: { $query: { $and: [ { document.$id: "da231061db8c3ca68be24ee79e70ceeb", isPublished: true, class: { $in: [ "Akn\Bundle\ContentBundle\Document\Rating" ] } }, { $or: [ { disabled: null }, { disabled: false } ] } ] }, $orderby: { createdAt: -1 } } planSummary: IXSCAN { document.$id: 1 }, IXSCAN { class: 1, createdAt: -1 } ntoreturn:20 ntoskip:0 nscanned:110857 nscannedObjects:0 keyUpdates:0 numYields:0 locks(micros) r:113738 nreturned:2 reslen:1941 113ms
2014-08-26T16:40:03.468+0200 [conn446] query akn.content query: { $query: { $and: [ { document.$id: "54863050aba53fa59097d58bcf97750e", isPublished: true, class: { $in: [ "Akn\Bundle\ContentBundle\Document\Rating" ] } }, { $or: [ { disabled: null }, { disabled: false } ] } ] }, $orderby: { createdAt: -1 } } planSummary: IXSCAN { document.$id: 1 }, IXSCAN { class: 1, createdAt: -1 } cursorid:130420393831 ntoreturn:20 ntoskip:0 nscanned:207813 nscannedObjects:0 keyUpdates:0 numYields:2 locks(micros) r:282352 nreturned:20 reslen:32862 266ms

Result:

{
    'runStats': {
        'linesRecommended': 0,
        'linesProcessed': 0,
        'linesPassed': 4
    },
    'results': []
}

I noticed from running Dex (and also this comment - #22 (comment)) that Dex doesn't currently recommend for $or queries. (Or quite a few other operators after looking at analyzer.py)

Is there a roadmap for when some of these operators will be included, or are there restrictions preventing them being added?

Thanks

When running Dex in profile mode and providing a non-admin URI (i.e., when the URI is not authing to /admin), Dex attempts to run .database_names(), which fails because "list databases" is an admin-only command, and the URI does not provide the necessary authentication.

dex -p mongodb://user:[email protected]:27307/dbname

Traceback (most recent call last):
File "/usr/local/bin/dex", line 129, in
main(sys.argv[1:])
File "/usr/local/bin/dex", line 114, in main
return md.analyze_profile()
File "/Library/Python/2.7/site-packages/dex/dex.py", line 137, in analyze_profile
databases = connection.database_names()
File "/Library/Python/2.7/site-packages/pymongo/connection.py", line 950, in database_names
self.admin.command("listDatabases")["databases"]]
File "/Library/Python/2.7/site-packages/pymongo/database.py", line 355, in command
msg, allowable_errors)
File "/Library/Python/2.7/site-packages/pymongo/helpers.py", line 129, in _check_command_response
raise OperationFailure(msg % response["errmsg"])

Hi,

I was trying to use Jackson's ObjectMapper in Java to read files with output from Dex and map the output into Java POJOs I made. But I realized this isn't JSON. It's reverse JSON, i.e. JSON with reversed double and single quotes. I did a find and replace, swapped the single quotes with an apostrophe (`), then swapped all double quotes with single quotes and finally swapped all apostrophes (former single quotes) with double quotes. And it worked. It has now been turned into valid JSON.

Please fix this, it shouldn't take you long now that it's been diagnosed and it's quite an annoying problem.

P.S. This used to be correct, why would you suddenly change it?

Visible error:

Traceback (most recent call last):
  File "/usr/local/bin/dex", line 233, in <module>
    main(sys.argv[1:])
  File "/usr/local/bin/dex", line 206, in main
    return md.analyze_profile()
  File "/usr/local/lib/python2.7/dist-packages/dex/dex.py", line 162, in analyze_profile
    profile_parser)
  File "/usr/local/lib/python2.7/dist-packages/dex/dex.py", line 101, in _process_query
    if parsed['supported']:
KeyError: 'supported'

How to reproduce:

• Replicaset, app read_preference = secondary_preferred
• Log in to a secondary, set slaveOk(), setProfilingLevel(2)
• Run loadtests
• setProfilingLevel(0)
• Run Dex from a remote machine with -p -n

Additional info:

• Dex installed today, using pip, from Github repo.
• MongoDB 2.4.8 running on all repset nodes.
• If I change the hostname to one of the other nodes, I get no error, but also no data as the system.profile collection doesn't exist there.
• I tried changing the PRIMARY_PREFERRED to NEAREST in dex.py and analyser.py, but no change in behaviour.

I'm going to export the collection and import it into the master, but any help here would be appreciated - I don't like having to turn on profiling globally.

Having the following collection:

> db.people.find()
{ "_id" : ObjectId("52779d40ad2f74e0deb3a7cf"), "name" : "Gerard", "age" : 31 }
{ "_id" : ObjectId("52779d83ad2f74e0deb3a7d0"), "name" : "Rose", "age" : 23 }
{ "_id" : ObjectId("52779d9ead2f74e0deb3a7d1"), "name" : "Yuki", "age" : 30 }
> 

I tried some basic queries, for example:

> db.people.find({age:{$gte:30}}).sort({name:1})
> db.people.find({age:{$gte:30}})

Passing the DEX tool to an empty system.profile collection, I can see that there's a suggestion when issuing the second query, but not when using the first one.

With sort operation:

> db.system.profile.drop()
true
> db.setProfilingLevel(2)
{ "was" : 0, "slowms" : 100, "ok" : 1 }
> db.people.find({age:{$gte:30}}).sort({name:1})
{ "_id" : ObjectId("52779d40ad2f74e0deb3a7cf"), "name" : "Gerard", "age" : 31 }
{ "_id" : ObjectId("52779d9ead2f74e0deb3a7d1"), "name" : "Yuki", "age" : 30 }
> db.setProfilingLevel(0)
{ "was" : 2, "slowms" : 100, "ok" : 1 }
> 

(env)gerard@antares:~/projects/dextest$ dex -p mongodb://localhost/test
{
    'runStats': {
        'linesRecommended': 0, 
        'linesProcessed': 1, 
        'linesPassed': 2
    }, 
    'results': []
}
(env)gerard@antares:~/projects/dextest$ 

Without sort operation:

> db.system.profile.drop()
true
> db.setProfilingLevel(2)
{ "was" : 0, "slowms" : 100, "ok" : 1 }
> db.people.find({age:{$gte:30}})
{ "_id" : ObjectId("52779d40ad2f74e0deb3a7cf"), "name" : "Gerard", "age" : 31 }
{ "_id" : ObjectId("52779d9ead2f74e0deb3a7d1"), "name" : "Yuki", "age" : 30 }
> db.setProfilingLevel(0)
{ "was" : 2, "slowms" : 100, "ok" : 1 }
> 

(env)gerard@antares:~/projects/dextest$ dex -p mongodb://localhost/test
{
    'runStats': {
        'linesRecommended': 1, 
        'linesProcessed': 1, 
        'linesPassed': 2
    }, 
    'results': [
        {
            'queryMask': '{"$query":{"age":{"$gte":"<val>"}}}', 
            'namespace': 'test.people', 
            'recommendation': {
                'index': '{"age": 1}', 
                'namespace': 'test.people', 
                'shellCommand': 'db["people"].ensureIndex({"age": 1}, {"background": true})'
            }, 
            'details': {
                'count': 1, 
                'totalTimeMillis': 0, 
                'avgTimeMillis': 0
            }
        }
    ]
}
(env)gerard@antares:~/projects/dextest$ 

This is the output when running pip install dex

Collecting dex
  Downloading https://files.pythonhosted.org/packages/cd/f4/46dd14cd7787b66f69b02cb5771247eaf6078913a5dccbff0f0ce0825bdb/Dex-0.6.tar.gz
Requirement already satisfied: pymongo in c:\users\jmorales\appdata\roaming\python\python35\site-packages (from dex) (3.3.0)
Requirement already satisfied: pyyaml in c:\users\jmorales\appdata\local\programs\python\python35\lib\site-packages (from dex) (3.11)
Collecting dargparse>=0.2.2 (from dex)
  Downloading https://files.pythonhosted.org/packages/ab/7a/8e6b3985396e164f09f3bb626b52ec2c9b1c88dd9f4dffffd5a2316538d2/dargparse-0.2.5.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\jmorales\AppData\Local\Temp\pip-install-_y0ezmr8\dargparse\setup.py", line 31
        except Exception, ex:
                        ^
    SyntaxError: invalid syntax

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\jmorales\AppData\Local\Temp\pip-install-_y0ezmr8\dargparse\

This is my setup:

Python version: 3.5
OS: Windows 10 Home
System Type: 64-bit Operating System, x64-based processor
Processor: AMD A10-8700P Radeon R6