demian85 / gnip Goto Github PK

It seems that when adding rules in bulk, errors are not carried on the body.error property, but rather in an array in body.detail.

I think adding a clause for this possibility to append more information to the errStr is appropriate, something like this:

if (body && body.detail) {
    errStr = body.detail.filter(r => r.message).reduce((msg, r) =>
        msg + '\n' + r.message
    , errStr);
}

This would result in an error string like

Error: Unable to add rules. Request Failed with status code: 422
Rule 'something' has some error

https://github.com/demian85/gnip/blob/master/lib/rules.js#L49

From the GNIP docs:

I'm testing PowerTrack 2.0 and noticed in each tweet the matching_rules:id is being rounded. All the rules end in 00. Here is an example of the one of the tweets. Any thoughts?

"gnip" : { "matching_rules" : [{ "tag" : "1956", "id" : 781250306873012200 } ]

When stream ends (on purpose or not on purpose) I get

Error: unexpected end of file
File "zlib.js", line 154, in Zlib.zlibOnError [as onerror]
on the error event
This doesn't happen every time, but quire frequent
This doesn't seem to be a "real" error but some kind of stream noise. is there a way to silence/solve this error?

Hi @demian85

There hasn't been much activity on this project for a while even though some PRs and issue reports are very useful. I think it would be sad if this project got scattered throughout many independent forks.

I'm willing to maintain this project if you allow me collaborator access? Or someone else if there's any interest?

Let me know what you think!

Thanks

Gnip API endpoint conventions seem to have changed. Might want to update the examples?

e.g. PowerTrack is now: https://gnip-stream.twitter.com/stream/powertrack/accounts/xxx/publishers/twitter/prod.json

and rules is now:
https://gnip-api.twitter.com/rules/powertrack/accounts/xxx/publishers/twitter/prod.json

When I have a stream open and I remove all the rules I start getting tons of parse errors in the console:
Error: Error parsing JSON: Cannot set property 'tweet' of undefined.
followed by the tweet object json and then:

    at Parser.receive (/Users/[directory_redacted]/node_modules/gnip/lib/JSONParser.js:30:28)
    at Gunzip.<anonymous> (/Users/[directory_redacted]/node_modules/gnip/lib/index.js:109:17)
    at Gunzip.emit (events.js:223:5)
    at addChunk (_stream_readable.js:309:12)
    at readableAddChunk (_stream_readable.js:290:11)
    at Gunzip.Readable.push (_stream_readable.js:224:10)
    at Gunzip.Transform.push (_stream_transform.js:150:32)
    at Zlib.processCallback (zlib.js:525:10)```

When trying to make multiple connections to a single GNIP stream, GNIP is sending an error in the stream alongside tweets

{"error":{"message":"This stream is currently at the maximum allowed connection limit","sent":"2018-02-16T04:59:50+00:00","transactionId":"00ad405500190622"}}
{"id":"tag:search.twitter.com,2005:964363458044076032","objectType":"activity","verb":"post","postedTime":"2018-02-16T04:59:00.000Z","generator":{"displayName":"Twitter for Android","link":"http:\/\/twitter.com\/download\/android"},"provider":{"objectType":"service","displayName":"Twitter","link":"http:\/\/www.twitter.com"},"link":"http:\/\/twitter.com\/MfMats\/statuses\/964363458044076032","body":"I vote #LandRoverFanClub for the #WesBankCOTY2018 win, who\u2019s your vote? @LandRoverZA @AutoTraderSA @W

however tweets have a \r\n as the separator, while errors only have \n. I noticed that your JSONParser defaults to \r\n as its seperator, and this is causing the events given in the example above to throw the Error parsing JSON error.

Has this been updated to use PowerTrack 2.0?

Just tested it with replay v2.0 and doesn't work. In index.js on line 89 is has this path : streamUrl.path + '?' + qs,. If qs is null then it appends the ? to then end of the url which gnip then rejects as invalid.

Try testing with replay with only url, user, and password options.

With one open PR remaining it's time to work towards a 0.3.0 release and have it pushed to the npm repository.

Todo:

• Wait for #7 to be ready and merged to master
• Create 0.3.0 tag
• Publish tag to npm

i asked gnip if they plan on adding id_str, like all the other twitter apis, no response yet.

When I get a 406 http error from replay stream, I only get the http error object
While the replay docs Specify that I'm expected to get

Will contain a JSON message indicating the issue -- e.g. "This connection requires compression. To enable compression, send an 'Accept-Encoding: gzip' header in your request and be ready to uncompress the stream as it is read on the client end." or "Invalid date for query parameter 'toDate'. Can't ask for tweets from within the past 30 minutes."

I'll be happy to help with a PR if you can point me in the right direction

I'm getting this when I try to write over 2K rules:

Error: socket hang up
at createHangUpError (http.js:1476:15)
at CleartextStream.socketCloseListener (http.js:1526:23)
at CleartextStream.emit (events.js:117:20)
at tls.js:693:10
at process._tickCallback (node.js:419:13)

There are a bunch of good pull requests that would be helpful in the pipeline.