sangheestyle / bigdiff-code Goto Github PK

http://stackoverflow.com/questions/2900674/how-do-i-convert-a-property-in-mongodb-from-text-to-date-type

        "created_at" : ISODate("2013-06-01T17:45:45Z"),
        "updated_at" : "2014-04-13T17:35:33Z",

https://github.com/thlorenz/readdirp

first depth only

var fs = require('fs');
fs.readdirSync('dir')

However, we need to know one more depth.

> var readdirp = require('readdirp');
> readdirp({root:'repos', depth:1}, function(fileInfo) {}, function (err, res) {console.log});
...
     { name: 'SMSSync',
       path: 'ushahidi/SMSSync',
       fullPath: '/home/sangheestyle_gmail_com/repos/ushahidi/SMSSync',
       parentDir: 'ushahidi',
       fullParentDir: '/home/sangheestyle_gmail_com/repos/ushahidi',
       stat: [Object] } ],

Check list:

• search repos
• filter and clone
• get issues

One of my friend in South Korea suggested me that we might what to consider different ways to implement our repo search. We can see his implementation at the following:

• https://github.com/jooyunghan/bigdiff-code/commit/b035d5912b341e4c39f8f3289f08ef0926ef6903
• https://gist.github.com/jooyunghan/7497419caaca7e579357

He used expand, observable, and concatMap instead of AyncIO. Just check it out and we can choose better way. :)

Especially this,

ianhanniballake/ContractionTimer@3989504
ianhanniballake/ContractionTimer#97

We might want to split library and application one day. For example, we can make bigdiff-code more general purpose and publish it into npm. If so, we can use this npm when we develop service for our server.

pros:

• We can use npm and use it for more public way. (Practice and good for showing what you do)
• Clear to maintain. It means we can make git repo of library for public and another for service implementation for private.

cons:

• Have two git repo.

parsing error when the result have no item

UPDATED: 2010-02-08 7 items
/home/sangheestyle_gmail_com/dev/bigdiff-code/lib/github.js:161
            pageNum = headers.link.split(',')[1].match(regex)[1];
                                                ^
TypeError: Cannot read property 'match' of undefined
    at /home/sangheestyle_gmail_com/dev/bigdiff-code/lib/github.js:161:49
    at /home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/octonode/lib/octonode/search.js:31:18
    at Client.errorHandle (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/octonode/lib/octonode/client.
js:183:14)
    at Request._callback (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/octonode/lib/octonode/client.j
s:197:24)
    at Request.self.callback (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/octonode/node_modules/requ
est/request.js:373:22)
    at Request.emit (events.js:110:17)
    at Request.<anonymous> (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/octonode/node_modules/reques
t/request.js:1318:14)
    at Request.emit (events.js:129:20)
    at IncomingMessage.<anonymous> (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/octonode/node_module
s/request/request.js:1266:12)
    at IncomingMessage.emit (events.js:129:20)

I need to put this into git.log.

Problem on server

TypeError: undefined is not a function
   at Object.git.execCommand (/home/sangheestyle_gmail_com/dev/bigdiff-code/lib/git.js:133:16)
   at Object.git.log (/home/sangheestyle_gmail_com/dev/bigdiff-code/lib/git.js:61:21)
   at /home/sangheestyle_gmail_com/dev/bigdiff-code/index.js:42:7
   at Layer.handle [as handle_request] (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/express/lib/router/layer.js:82:5)
   at next (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/express/lib/router/route.js:110:13)
   at Route.dispatch (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/express/lib/router/route.js:91:3)
   at Layer.handle [as handle_request] (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/express/lib/router/layer.js:82:5)
   at /home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/express/lib/router/index.js:267:22
   at Function.proto.process_params (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/express/lib/router/index.js:321:12)
   at next (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/express/lib/router/index.js:261:10)

UPDATED: 2010-02-07 3 items
UPDATED: 2010-02-08 7 items
Progress: 40/1886 ...
/home/sanghee/Dev/bigdiff-code/lib/github.js:215
numberOfPages = headers.link.split(',')[1].match(regex)[1];
^
TypeError: Cannot read property 'match' of undefined
at /home/sanghee/Dev/bigdiff-code/lib/github.js:215:51

We will use cassandra for storing git(not git repository, but git data), github, and social artifacts.
So, we might need to discuss about why we need it for this and how.

https://github.com/datastax/nodejs-driver

This is how to run node app in server.

http://stackoverflow.com/a/4018223/568901

We need to release bigdiff-code 0.1.0.

Release Schedule

The releases:

• 0.1.0: Feburary 22, 2015
• 0.1.1: Feburary 24, 2015

Features for 0.1.0

• #1
• #2
• #3

Misc

This version is for our presentation.

Need to transfer Issues to MongoDB instead of cassandra

How can we make the following part more gracefully? For example, how can we avoid using many ? mark?

 var insertRepo = 'INSERT INTO bigdiff.repo_info (id, name, full_name, owner_login, '
    + 'owner_id, owner_url, private, html_url, description, url, clone_url, size, '
    + 'stargazers_count, watchers_count, language, has_issues, forks_count, '
    + 'open_issues_count, score) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '
    + '?, ?, ?, ?, ?, ?);';

de2a466

I think mongodb might be better to this project in terms of rich query features. Here is my plan.

a. I make a feature branch: mongodb
b. replace cassandra part to by mongodb

We need to release bigdiff-code 0.1.1.

Release Schedule

The releases:

• 0.1.0: Feburary 22, 2015
• 0.1.1: Feburary 24, 2015

Features for 0.1.1

• #4
• #7

Misc

This version is for our presentation.

Not Found in CyanogenMod/android_packages_apps_Calendarfatal: bad default revision 'HEAD'child_process.js:1382 throw err; ^Error: Command failed: git log --pretty=format:'{ "id": "%H", "author_name": "%an", "author_email": "%ae", "date": "%ad", "subject": "%s", "body": "%b"},' -G".setTag([^,|^(],[^,])" -- *.javafatal: bad default revision 'HEAD' at checkExecSyncError (child_process.js:1339:13) at execSync (child_process.js:1379:13) at Object.git.execCommand (/home/sangheestyle_gmail_com/dev/bigdiff-code/lib/git.js:148:16) at Object.git.log (/home/sangheestyle_gmail_com/dev/bigdiff-code/lib/git.js:67:21) at ReaddirpReadable. (/home/sangheestyle_gmail_com/dev/bigdiff-code/app.js:41:13) at ReaddirpReadable.emit (events.js:107:17) at ReaddirpReadable. (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/readdirp/node_modules/readable-stream/lib/stream_readable.js:786:14) at ReaddirpReadable.emit (events.js:104:17) at emitReadable (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/readdirp/node_modules/readable-stream/lib/_stream_readable.js:448:10) at emitReadable (/home/sangheestyle_gmail_com/dev/bigdiff-code/node_modules/readdirp/node_modules/readable-stream/lib/_stream_readable.js:444:5)sangheestyle_gmail_com@instance-1:~/dev/bigdiff-code$

Cloned: bgcngm/mtk-tools
Username for 'https://github.com': Cloned: lasar/titanium-lib
Cloned: perry/sass-font-face
Cloned: vowei/iQuery
Cloned: teamblueridge/a

It's not really error but need to figure out what's the Username for ... part

I think we can show what happen in our server on web page. (NOT NOW) For example,

a. clone status (e.g. how many git repo stored or list repo name)
b. update status
c. trend graph

Whatever.

I think we can make some different script for update and run it with bash script first.

Is it good? Mimic their service.

https://github.com/sangheestyle/bigdiff-code/search?utf8=%E2%9C%93&q=android

Our code generates ton of log with console.log() so we might want to make some good prefix for someday. This is because the way will allow us to grep some pattern easily for logging or error analysis.

83 Issues Stored in DB.
98 Issues Stored in DB.
Cloned: 47deg/android-swipelistview
18 Issues Stored in DB.
Cloned: Comcast/FreeFlow
Cloned: emilsjolander/StickyListHeaders
251 Issues Stored in DB.
Cloned: redsolution/xabber-android

Let's think about what's the good log naming sense.

Need to get reviews from google play for apps. Here is a basic code snippet. I worked.

var request = require('request');
var cheerio = require('cheerio');

// Set the headers
var headers = { 'User-Agent': 'Wild/0.0.1'
              , 'Content-Type': 'application/x-www-form-urlencoded'
              }

// Configure the request
var options = { url: 'https://play.google.com/store/getreviews'
              , method: 'POST'
              , headers: headers
              , form: { 'reviewType': 0
                      , 'pageNum': 1
                      , 'id': 'com.github.mobile'
                      , 'reviewSortOrder': 2
                      }
              }

// Start the request
request(options, function (error, response, body) {
    if (!error && response.statusCode == 200) {
        var htmlString = JSON.parse(body.slice(7, -1))[2];
        var reviews = [];
        $ = cheerio.load(htmlString);
        $('.single-review').each(function(i, elem) {
          reviews[i] = $(this).text();
        });
        console.log(reviews[0]);
    }
})

From server

{ [Error: API rate limit exceeded for sangheestyle.] message: 'API rate limit exceeded for sangheestyle.', statusCode: 403, headers: { server: 'GitHub.com', date: 'Wed, 25 Feb 2015 07:09:58 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '124', connection: 'close', status: '403 Forbidden', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': '1424850215', 'x-github-media-type': 'github.v3; format=json', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'content-security-policy': 'default-src 'none'', 'access-control-allow-credentials': 'true', 'access-control-expose-headers': 'ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval', 'access-control-allow-origin': '*', 'x-github-request-id': '689A4A5D:11B4:4AD3456:54ED7545', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-content-type-options': 'nosniff' } }

When you run github.js, getRepoIssues works approximately 50% of the time, the other times it will hang for some unknown reason. It is crucial to repair this. Is it a problem that I implemented a recursive method in order to gather all issues with pagination?

Local git repositories and information of repositories on github should be updated periodically i.e., daily.

From Ken,

Search harvested repos for instances of stack traces that appear in issues on the repo or stack traces in commit logs (even small snippets of stack traces)

This will involve coming up with a set of heuristics for how to search for stack traces: i.e. search for the words "Exception" and "throws" in the indexed documents and then see what comes back to identify other terms that can be used to find stack traces.

If there has a way to use github REST APIs without actual password, it might be very helpful to us.

74b83e5