langleu / ma_crawler Goto Github PK

spend a day testing the implementation of the client using various docker files.

could be used to check against duplication.
for error logging of crappy crawled files.

e.g. MongoDB

• check how grakn deals with duplicates when it comes to the same key
  • if one tries to commit duplicates those will be automatically ignored and all uncommitted data will be removed as on every insert we immediately commit only the duplicate should be lost
• maybe just write errors down to disk or sentry?

let graqlInsertQuery = `match $deployment isa deployment, has rid "${deploymentId}"; `;
graqlInsertQuery += `$service isa service, has name "${serviceName}"; `;

generate unique hash for services based on name + deployment sha

convert a grakn graph to gephi.

has nothing directly to do with the crawler, but for now I'll keep it here.

the grakn workbase is just simply too slow to handle multiple hundred thousand entries.

https://gephi.org/gexf/format/schema.html (1.2)

GraphDB (figure out which one)
possible mongodb for metadata?
dive deeper into neo4j and grakn. Take notes and decision making to the other repository.

or comparable that works better with koa

e.g. how far nodes are with crawling.
how the current status is of nodes (healthy, reachable etc...)

based on the API

based on the web app via webcrawlers

for MA_Crawler and MA_Thesis

possible first step intitle:docker-compose.yml site:github.com