A collection of scripts to crawl English legal corpora ๐ from open public domains.
- The current version supports the following domains:
Corpus | Domain | Corpus alias |
---|---|---|
๐ช๐บ EU legislation | https://eur-lex.europa.eu/ | eu |
๐ฌ๐ง UK legislation | https://legislation.gov.uk/ | uk |
๐จ๐ฆ Canadian legislation | http://laws.justice.gc.ca/eng/ | ca |
๐ฏ๐ต Japanese legislation | http://www.japaneselawtranslation.go.jp/law/ | jp |
๐ซ๐ฎ Finish legislation | https://www.finlex.fi/en | fi |
๐บ๐ธ US case law* | https://case.law/bulk/download/ | us |
* In order to use the script for US case law, you need to first apply for a researcher account at https://case.law.
- For US public filings, e.g., contracts, please use the library OpenEDGAR (https://github.com/LexPredict/openedgar) by LexPredict.
- Documents are saved in raw text format, amend the code if you wish to better handle metadata, document structure, etc.
- If you aim to use the code, please carefully read the individual license agreements with respect to re-use, re-publication, terms of use, etc. ๐
- The text cleansing from the original PDF/HTML files is minimal. Consider amending the scripts and/or writing your own post-processing data cleansing process that better fit for each corpus. ๐ง
- These scripts aim to give researchers a kick start for scraping legal corpora from public domains. They should not considered a stand-alone qualified solution. ๐ง
- json-lines
- tqdm
- beautifulsoup4
The following linux packages are used to process PDF documents:
- pdftocairo
- pdftotext
- mutool
- gs
pip install -r requirements.txt
sudo apt-get install libcairo2-dev
sudo apt-get install libpango1.0-dev
sudo apt-get install -y xpdf
sudo apt-get install mupdf mupdf-tools
python download_legal_corpora.py --corpus ca
python download_legal_corpora.py --corpus eu
python download_legal_corpora.py --corpus all