lemurproject Goto Github PK

cast-bert-reranker

clueweb12pp

Extension to the Clueweb12 dataset

clueweb12pp-core

Unifies all code written for processing clueweb12++

clueweb22

crawl-browser

crawling-tools

A collection of tools for running the crawler, including scripts and scrapers to collect seeds.

forum-seed

Code and other stuff for discussion-forum seeds.

heritrix-jobs

Configuration files and scripts needed to run the Heritrix jobs

heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

kba-tools

Tools for processing the Trec KBA dataset

language-id

language identification module in racket

lucindri

Indri search implementation on top of Lucene search engine

mturk

nabble-tools

Tools for processing our nabble crawl

nopol

Simple command line tool to export the ClueWeb dataset as HTML files.

oauth-racket

Oauth Implementation in Racket (PLT Scheme)

python-blekko

bindings for the Blekko search engine API

reddit_crawler

Download the reddit dataset for ClueWeb12++

scrapers

Collection of Scrapers I am writing

site-sampling

Code to sample forums and check stats on pages and their information content

warc

Python library for reading and writing warc files

warc-tools

Command line utilities for working with WARC files

ygroups-tools

Processing the Yahoo groups crawl (part of clueweb12++)