bwbwbwbw / ingress-exporter Goto Github PK

Can't extract munge data from Intel:

2015-07-01T14:14:59.765Z - info: [MungeDetector] Trying to use internal munge data.
2015-07-01T14:15:00.048Z - warn: [MungeDetector] Failed.
2015-07-01T14:15:00.048Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
{ [Error: Line 2: Unexpected token <]
  index: 1,
  lineNumber: 2,
  column: 1,
  description: 'Unexpected token <' }
2015-07-01T14:15:00.248Z - warn: [MungeDetector] Failed.
2015-07-01T14:15:00.248Z - error: [MungeDetector] Could not detect munge data. Tasks are terminated.

This can be solved if I edit munge data manually in database.
But anyway ingress-exporter can't find any portals at now. And I don't know where problem is.

Does anyone have the same issue and maybe someone know to solve this?

Hi,

i tried to update my database today, but i'm getting no Portals/Links/Fields.

2015-04-13T20:24:00.862Z - �[32minfo�[39m: [MungeDetector] Trying to use internal munge data.
2015-04-13T20:24:02.563Z - �[32minfo�[39m: [MungeDetector] Detect successfully.
2015-04-13T20:24:02.564Z - �[32minfo�[39m: [AccountInfo] Fetching current account information...
2015-04-13T20:24:04.154Z - �[32minfo�[39m: [AccountInfo] XXXXXX (RESISTANCE)
2015-04-13T20:24:04.154Z - �[33mwarn�[39m: [AccountInfo] �[7m�[33mPlease immediately press Ctrl+C if you are using an incorrect account.�[39m�[27m
2015-04-13T20:24:04.178Z - �[32minfo�[39m: [Portals] Prepared 2025 tiles
2015-04-13T20:24:04.181Z - �[32minfo�[39m: [Portals] Begin requesting...
2015-04-13T20:24:05.920Z - �[32minfo�[39m: [Portals] 0% [1/507] 1185ms 0 portals, 0 links, 0 fields
2015-04-13T20:24:06.394Z - �[32minfo�[39m: [Portals] 0% [2/507] 1659ms 0 portals, 0 links, 0 fields
2015-04-13T20:24:06.698Z - �[32minfo�[39m: [Portals] 1% [3/507] 777ms 0 portals, 0 links, 0 fields

Any ideas?

Not an issue just a heads up just in case you decided to try export this I thought I would point out it is inaccurate, it APPEARS (without digging through activity logs I can't confirm this yet) for Uncaptured Portals it shows the date/time it went uncaptured but for captured portals it just shows the date/time the data was collected.

2014-05-24T14:27:46.108Z - error: [MungeDetector] Failed to read mungedata from database: failed to connect to [localhost:27017]
2014-05-24T14:27:46.110Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
2014-05-24T14:27:46.709Z - error: [Auth] Authentication failed. Please update the cookie.

Getting this when I try to run:
node build/app.js --portals --new

Have an error while trying to use/detect munge data on different accounts/servers starting from 18 october.

2014-10-19T15:55:44.251Z - info: [MungeDetector] Trying to use internal munge data.
2014-10-19T15:55:45.063Z - warn: [MungeDetector] Failed.
2014-10-19T15:55:45.064Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
2014-10-19T15:55:46.396Z - warn: [MungeDetector] Failed.
2014-10-19T15:55:46.397Z - error: [MungeDetector] Could not detect munge data. Tasks are terminated.

Someone has the same trouble?

error: [Details] unknown server response
error: [Broadcasts] unknown server response

help, pls

Hi,

today i got these errors while runnig the script.

2015-01-13T19:24:23.094Z - warn: [Entity] Unknown entity type, id=xxxxxxxxxxxxxxxxxxxx, type=undefined

Entity.coffee contains the following code.

        if data.type is 'portal'
            createPortalEntity id, timestamp, data, callback
        else if data.type is 'region'
            createFieldEntity id, timestamp, data, callback
        else if data.type is 'edge'
            createLinkEntity id, timestamp, data, callback
        else
            logger.warn "[Entity] Unknown entity type, id=#{id}, type=#{data.type}"
            callback && callback()

It seems it is the same prob iitc got three days ago:

10th January 2015

IITC 0.20.0 released. This is a critical update to fix loading of portals/links/fields after a protocol change by Niantic.

http://iitc.jonatkins.com/

Does anybody know how to fix this?

Edit:

I played a little bit with the js-Files and that is what i found:

Niantic changed the format of the "data".

request-portals.js

About line 282 i created some log entries:

      if (t.tile.gameEntities != null) {
        return async.each(t.tile.gameEntities, function(entity, callback) {
          return Entity.add(entity[0], entity[1], entity[2], function(type) {
          logger.info("####: ID=" + entity[0]);
          logger.info("####: TIMESTAMP=" + entity[1]);
          logger.info("####: DATA=" + entity[2]);

Portal.

2015-01-13T20:22:55.743Z - info: ####: ID=58701dcc7c64422ca0b5ccbd2fc8fffd.16
2015-01-13T20:22:55.743Z - info: ####: TIMESTAMP=1417021375226
2015-01-13T20:22:55.743Z - info: ####: DATA=p,N,52566862,7593227,1,0,0,http://lh5.ggpht.com/bgyvgnnNqjflA6fgH_7TfJ_dxeDeoDS12zaL9L3azhFK9Qt67kOyAqQZklxNpOk7BHGsZeYdqBSUm-fWmP4tZw,Luna Park,

You can see, the "DATA" is now an array:

TYPE,FACTION,LAT,LNG,LEVEL,XM,RESONATOR-COUNT,PICTURE-URL,TITLE

Edge:

2015-01-13T20:26:05.741Z - info: ####: ID=647b7ba8c8224e88b835f5b26b78ebfd.9
2015-01-13T20:26:05.741Z - info: ####: TIMESTAMP=1421093566634
2015-01-13T20:26:05.741Z - info: ####: DATA=e,E,1025d2fa138143dcb61e8dcd87b46c2a.16,52716320,6767534,7e407a37512348be8f747429289810c5.16,52833075,6366633

Field:

2015-01-13T20:28:25.307Z - info: ####: ID=5e33289ce78a4d6c9b4c7de4f62ea931.b
2015-01-13T20:28:25.307Z - info: ####: TIMESTAMP=1420818476250
2015-01-13T20:28:25.307Z - info: ####: DATA=r,R,618813f6d82c4c6d810c729ddf6a6dcd.16,52527632,7196913,bc00081f51594dd8b8c4f432e43bb111.16,52443113,7091642,7a963deaf168482f884af9390f596658.16,52691670,7270650

entity.js:

I changed lines like data.type == 'portal' to data[0] === 'p'

add: function(id, timestamp, data, callback) {
  var main;
  main = function() {
    Entity.entityCount++;
    if (data[0] === 'p') {
      return createPortalEntity(id, timestamp, data, callback);
    } 
    else if (data[0] === 'r') {
      return createFieldEntity(id, timestamp, data, callback);
    } else if (data[0] === 'e') {
      return createLinkEntity(id, timestamp, data, callback);
    } else {
      logger.warn("[Entity] Unknown entity type, id=" + id + ", type=" + data[0]);
      return callback && callback();
   }
  };

In createPortalEntity i changed the calculation of lat lng and the if statement.

createPortalEntity = function(id, timestamp, data, callback) {
data[2] = data[2] / 1e6;
data[3] = data[3] / 1e6;
return createEntity('Portals', id, timestamp, data, function() {
if (data[1] !== 'N' && argv.detail !== 'false') {
return Entity.requestPortalDetail(id, function() {
return callback && callback('portal');
});
} else {
return callback && callback('portal');
}
});
};

Afterwards, the scripts calls the createEntity method:

createEntity = function(collection, id, timestamp, data, callback) {
data.time = timestamp;
return Database.db.collection(collection).update({
_id: id
}, {
$set: data
}, {
upsert: true
}, function(err) {
if (err) {
logger.error('[Entity] Failed to insert entity (id=%s) into database: %s', id, err.message);
}
return callback();
});
};

This causes this kind of error:

2015-01-13T20:28:25.300Z - error: [Entity] Failed to insert entity (id=8d20d05d599d4207af7cd41cae7f2391.9) into database: Invalid modifier specified: $set

I think "data" was in json and Database.db.collection could handle this, but it isn't anymore :(

I'm stuck and i don't know what to to next :(

Perhaps somebody can use my "work" ;) to fix this issue!

Hi,

sometimes when I do grunt I get this compilation error:

Running "copy:project" (copy) task
Copied 2 files

Running "coffee:project" (coffee) task

Running "cson:project" (cson) task
Parsing config.cson failed.
>> Syntax error on line 49, column 0: unexpected '\u0000' (\u0000)
>> 46 :     Options:
>> 47 :
>> 48 :         safe:       true
>> 49 :
>> ^^ :~^
Warning: Task "cson:project" failed. Use --force to continue.

This is my config.cson:

Auth:

    # You could fill this field via copying from:
    #     Chrome Dev Tools -> Network -> (select a non-static request entity) ->
    #     Request Headers -> Cookie
    # Please fetch the cookie from non-SSL ingress/intel site.
    # Otherwise it won't work currently.
    CookieRaw: '.....my cookie....;'


ZoomLevel: 14

Region: [[12.30743408203125,45.88714049635727],[12.85675048828125,46.021045392202666],[12.582435607910156,45.94834486134022],[12.719764709472656,45.98008480803993]]

Request:

    UserAgent:     'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36'
    MaxParallel:   10
    MinIntervalMS: 100  # Too fast requesting would get IP banned

Tile:

    MaxFailRetry:  3    # server responses a failure
    MaxErrorRetry: 3    # network errors, etc

TileBucket:

    Max: 4  # tiles may get TIMEOUT if larger than 4
    Min: 4

Chat:

    SplitTimespanMS:        3*60*60*1000   # split tasks by 3hrs
    TraceTimespanMS:    10*24*60*60*1000   # new task: trace 10 days
    FetchItemCount:     300

PlayerLookup:

    Max: 15

Database:

    ConnectString: 'mongodb://localhost:27017/ingress'

    MaxParallel:    10

    Options:

        safe:       true

I haven't skill in cson. May you help me?

v.

Anyone else having issues mine is failing non-stop today, I even made sure to freshly pull it from git and reinstall it all with the https cookies?

It looks like they changed intel over the weekend and now I'm just getting the MungeDetector Failed error when trying to extract any data.

A saw that markup.PORTAL1.guid is missing in last month logs.
How to fix this problem?

hi, i got these errors while runnig the script:

2015-02-02T11:37:50.646Z - info: [MungeDetector] Trying to use internal munge data.
2015-02-02T11:37:51.179Z - warn: [MungeDetector] Failed.
2015-02-02T11:37:51.180Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
[Error: Failed to extract version]
2015-02-02T11:37:51.687Z - warn: [MungeDetector] Failed.
2015-02-02T11:37:51.688Z - error: [MungeDetector] Could not detect munge data. Tasks are terminated.

Any ideas?

Niantic introduced Google's botguard protection to the Intel map. This prevents ingress-exporter to work after a few requests.

IITCs current approach is to eval the code that's being sent with every request.

Is This Broken???

2015-06-14T11:01:28.635Z - info: [Portals] 0%   [88/41842]      1708ms  15 portals, 0 links, 0 fields

/home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/base.js:245
        throw message;      
              ^
TypeError: Cannot read property 'length' of undefined
    at Object.async.each (/home/vavoluk/ingress-exporter/node_modules/async/lib/async.js:120:17)
    at Object.GLOBAL.Agent.resolveFromPortalDetail (/home/vavoluk/ingress-exporter/build/lib/agent.js:38:20)
    at /home/vavoluk/ingress-exporter/build/lib/entity.js:100:26
    at /home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/collection/core.js:743:14
    at /home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/db.js:1132:7
    at /home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/db.js:1846:9
    at Server.Base._callHandler (/home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/base.js:445:41)
    at /home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/server.js:468:18
    at MongoReply.parseBody (/home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
    at null.<anonymous> (/home/vavoluk/ingress-exporter/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/server.js:426:20)

How to fix this?

Hi,
No data retrieved when requesting portals infos.
Maybe an API change ?

http://pastebin.com/mGi1cx23

Getting the following, already used mrcerf's is_tablet fix.
2014-12-29T13:34:00.246Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
[ReferenceError: Invalid left-hand side in assignment]
2014-12-29T13:34:01.484Z - warn: [MungeDetector] Failed.
2014-12-29T13:34:01.484Z - error: [MungeDetector] Could not detect munge data. Tasks are terminated.

Any fixes for this one as well?

Ingress intel is now forcing a secure connection.

I checked 3 browsers on 3 servers and nothing.... no one can show me ID.

Hi,

first: tanks for this amazing project!

second: When I run:

node build/app.js --portals --new
2014-05-02T05:51:56.243Z - info: [MungeDetector] Trying to use internal munge data.
2014-05-02T05:52:01.393Z - info: [MungeDetector] Detect successfully.
2014-05-02T05:52:01.396Z - info: [AccountInfo] Fetching current account information...
2014-05-02T05:52:02.506Z - info: [AccountInfo] myAccount (RESISTANCE)
2014-05-02T05:52:02.507Z - warn: [AccountInfo] Please immediately press Ctrl+C if you are using an incorrect account.

Hang in this point. No data. If I look into mongodb collection I have only one entry in MungeData.

May you help me?

Thanks in advance.
v.

Today IITC updated, please update the files, get "unknown server response", account is not banned, error persist on new account.

Starting from today have trouble with MungeDetector:

2014-12-19T20:41:30.682Z - info: [MungeDetector] Trying to use internal munge data.
2014-12-19T20:41:31.056Z - warn: [MungeDetector] Failed.
2014-12-19T20:41:31.058Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
[ReferenceError: IS_TABLET is not defined]
2014-12-19T20:41:31.940Z - warn: [MungeDetector] Failed.
2014-12-19T20:41:31.942Z - error: [MungeDetector] Could not detect munge data. Tasks are terminated.

Anyone have ideas how to fix this?