apmoore1 / bl-19-books Goto Github PK

Currently when batching files we copy the data, it would be better if the batching script just creates a new directory with M files each containing N relative files names to the book folder directory, whereby each relative file name is unique. This resolves the issue of copying data and the only data that is ever needed is the original book folder data (all of the British Library OCR book corpus.). This will require changing the Spacy processing and language ID script to take a file of file names rather than a directory of files to process.

The TSV format that the spacy tagging produces does not use a CSV/TSV library, this is required as currently it does not escape certain characters that a CSV library would like: " which should be when escaped """.