wyattsam / mongo-rpc Goto Github PK

This tree contains a couple of new aggregation stages, $socketBSON and $httpGET

$httpGET takes a URI (optionally with paramters in the "query" field) and expects a single JSON document back.

$socketBSON creates a bidirectional stream of BSON documents. For each document in, you should get one document back.
If you don't specify a "localField", it currently expects a unidirectional input stream of BSON. Will add other options later.
$socketBSON only supports IPv4 - fix it if you care ;-)

Some of the tests use a sample mysql database found here: 
https://dev.mysql.com/doc/world-setup/en/world-setup-installation.html

Example xinetd conf files are in xinetd/ - put them in /etc/xinetd.d and do "service xinetd restart"
There's some sample test data in test/
Some python scripts of example connectors in the connectors/ directory

The trickiest thing with $socketBSON is unwanted IO buffering. Using "python -u" helps.

---------------------------------------------------------------------------

$httpGET examples:

db.test.aggregate({'$httpGET':{'uri:'https://www.quandl.com/api/v3/datasets/FRED/NGDPPOT.json', 'subfield':'dataset', 'as':"foo"}})

db.test.aggregate({$httpGET:{'uri':'https://api.enigma.io/v2/data/Ym7JIgsmje7zkU6kA6G8VgAyRB5INIpDXamVvxNwCJiFP7YTisfW3/us.gov.government-spending.awards.grants'}})

db.test.aggregate([{'$httpGET':{'uri':'http://api.nytimes.com/svc/search/v2/articlesearch.json','query':{'q':'obama','api-key':'c2fede7bd9aea57c898f538e5ec0a1ee:6:68700045'}}}), {$unwind: {path:"$response.docs"}}, {$limit:1}])


---------------------------------------------------------------------------

$socketBSON example - importing from mysql:

First, let's look at the data that get written to the pipe by mongod
$ cat ~/test/mysql_execute.json
{"host":"localhost", "user":"mbligh", "passwd":"", "db":"world", "sql":"SELECT CountryCode,Name,Population FROM City WHERE District=%s", "wherekeys":["district"]}
{"district":"Iowa"}
{"district":"Virginia"}

Now the data that will be in the reply.
$ json2bson < ~/test/mysql_execute.json | mysql_execute | bson2json
{"": [{"Name": "Des Moines", "CountryCode": "USA", "Population": 198682}, {"Name": "Cedar Rapids", "CountryCode": "USA", "Population": 120758}, {"Name": "Davenport", "CountryCode": "USA", "Population": 98256}]}
{"": [{"Name": "Virginia Beach", "CountryCode": "USA", "Population": 425257}, {"Name": "Norfolk", "CountryCode": "USA", "Population": 234403}, {"Name": "Chesapeake", "CountryCode": "USA", "Population": 199184}, {"Name": "Richmond", "CountryCode": "USA", "Population": 197790}, {"Name": "Newport News", "CountryCode": "USA", "Population": 180150}, {"Name": "Arlington", "CountryCode": "USA", "Population": 174838}, {"Name": "Hampton", "CountryCode": "USA", "Population": 146437}, {"Name": "Alexandria", "CountryCode": "USA", "Population": 128283}, {"Name": "Portsmouth", "CountryCode": "USA", "Population": 100565}, {"Name": "Roanoke", "CountryCode": "USA", "Population": 93357}]}


Now let's make that happen with aggregate:
> db.test.aggregate({'$socketBSON':{host:'localhost',port:668, localField:'', as:'cities', initial:{"host":"localhost", "user":"mbligh", "passwd":"", "db":"world", "sql":"SELECT CountryCode,Name,Population FROM City WHERE District=%s", "wherekeys":["district"]}}}).pretty()
{
        "_id" : ObjectId("5723cbb9f08e20a32022a52f"),
        "district" : "Iowa",
        "cities" : {
                "" : [
                        { "Name" : "Des Moines", "CountryCode" : "USA", "Population" : NumberLong(198682) },
                        { "Name" : "Cedar Rapids", "CountryCode" : "USA", "Population" : NumberLong(120758) },
                        { "Name" : "Davenport","CountryCode" : "USA", "Population" : NumberLong(98256) }
                ]
        }
}
{
        "_id" : ObjectId("5723cbb9f08e20a32022a530"),
        "district" : "Virginia",
        "cities" : {
                "" : [
                        { "Name" : "Virginia Beach",  "CountryCode" : "USA","Population" : NumberLong(425257) },
                        { "Name" : "Norfolk", "CountryCode" : "USA",  "Population" : NumberLong(234403) },
                        { "Name" : "Chesapeake", "CountryCode" : "USA", "Population" : NumberLong(199184) },
          ....
                ]
}

---------------------------------------------------------------------------

$socketBSON example - looking at electrical rate data by zip code:

 $ wget -q -O - http://en.openei.org/doe-opendata/dataset/3e440383-a146-49b5-978a-e699334d2e1f/resource/3f00482e-8ea0-4b48-8243-a212b6322e74/download/iouzipcodes2011.csv | head -3
zip,eiaid,utility_name,state,service_type,ownership,comm_rate,ind_rate,res_rate
35218,195,Alabama Power Co,AL,Bundled,Investor Owned,0.105761195393,0.0602924366735,0.114943267065
35219,195,Alabama Power Co,AL,Bundled,Investor Owned,0.105761195393,0.0602924366735,0.114943267065


$ echo '{"uri":"http://en.openei.org/doe-opendata/dataset/3e440383-a146-49b5-978a-e699334d2e1f/resource/3f00482e-8ea0-4b48-8243-a212b6322e74/download/iouzipcodes2011.csv"}' | json2bson | csv2bson | bson2json | head
{"ind_rate": "0.0602924366735", "zip": "35218", "res_rate": "0.114943267065", "comm_rate": "0.105761195393", "utility_name": "Alabama Power Co", "eiaid": "195", "state": "AL", "ownership": "Investor Owned", "service_type": "Bundled"}
{"ind_rate": "0.0602924366735", "zip": "35219", "res_rate": "0.114943267065", "comm_rate": "0.105761195393", "utility_name": "Alabama Power Co", "eiaid": "195", "state": "AL", "ownership": "Investor Owned", "service_type": "Bundled"}
...


> db.test.aggregate({'$socketBSON':{host:'localhost',port:669, initial:{"uri":"http://en.openei.org/doe-opendata/dataset/3e440383-a146-49b5-978a-e699334d2e1f/resource/3f00482e-8ea0-4b48-8243-a212b6322e74/download/iouzipcodes2011.csv"}, output:false}}).pretty()
{
        "ind_rate" : "0.0602924366735",
        "zip" : "35218",
        "res_rate" : "0.114943267065",
        "comm_rate" : "0.105761195393",
        "utility_name" : "Alabama Power Co",
        "eiaid" : "195",
        "state" : "AL",
        "ownership" : "Investor Owned",
        "service_type" : "Bundled"
}
{
        "ind_rate" : "0.0602924366735",
        "zip" : "35219",
        "res_rate" : "0.114943267065",
        "comm_rate" : "0.105761195393",
        "utility_name" : "Alabama Power Co",
        "eiaid" : "195",
        "state" : "AL",
        "ownership" : "Investor Owned",
        "service_type" : "Bundled"
}
...