digitalpebble Goto Github PK

ansible-storm

Ansible playbook for deploying a Storm cluster

behemoth

Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.

behemoth-commoncrawl

Support for old (pre 2013) CommonCrawl dataset in Behemoth

behemoth-elasticsearch

ElasticSearch module for Behemoth

behemoth-textclassification

Module for classifying Behemoth documents with a model from our Text Classification API

benchmark

StormCrawler topology to evaluate the performance of different backends and configurations

crawler-commons

A set of reusable Java components that implement functionality common to any web crawler

crawler4j-frontier-battle

crawlurlfrontier

Crawl config used to test URL Frontier on a large scale and produce WARCs for CommonCrawl.

digitalpebble.github.io

Resources for the DigitalPebble website

docs

Documentation for Docker Official Images in docker-library

ngrams-api

Java API for querying a N-Grams corpus. Uses Lucene for searching and indexing from the Google Web-1T format

nutch

Apache Nutch is an extensible and scalable web crawler

nutchfight

Resources for comparison between 1.8 and 2.x of Apache Nutch

sc-warc

WARC resources for StormCrawler

storm

Mirror of Apache Storm

storm-crawler

A scalable, mature and versatile web crawler based on Apache Storm

stormcrawler-docker

Resources for running StormCrawler with Docker services

stormcrawlerfight

Crawl configurations for benchmarking / testing StormCrawler

tescobank

Setup for crawling tescobank with SC

textclassification

A Text Classification API in Java originally developed by DigitalPebble Ltd. The API is independent from the ML implementations used and can be used as a front end to various ML algorithms. libSVM and liblinear are currently embedded.

textclassification-examples

Use cases for DigitalPebble's TextClassification API

textclassificationplugin

GATE Processing Resource wrapping DigitalPebble's TextClassification API

tika

The Apache Tika toolkit detects and extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF).

tika-cc

resources for generating a corpus of docs from CC for Tika

tika-detector-stormcrawler

Wraps the charset detection logic from StormCrawler as a Tika module

urlfrontier-client

URLFrontier client written in Rust (mostly as a way of learning Rust)