A set of simple Python scripts for pre-processing large files, things like splitting and format conversion. The names phraug comes from a great book, Made to Stick, by Chip and Dan Heath.

See http://fastml.com/processing-large-files-line-by-line/ for the basic idea.

There's always at least one input file and usually one or more output files. An input file always stays unchanged.

csv2libsvm.py <input file> <output file> [<label index = 0>] [<skip headers = 0>]

Convert CSV to LIBSVM format. If there are no labels in the input file, specify label index = -1. If there are headers in the input file, specify skip headers = 1.

csv2vw.py <input file> <output file> [<label index = 0>] [<skip headers = 0>]

Convert CSV to VW format. Arguments as above.

libsvm2csv.py <input file> <output file> <input file dimensionality>

Convert LIBSVM to CSV. You need to specify dimensionality, that is a number of columns (not counting a label).

libsvm2vw.py <input file> <output file>

Convert LIBSVM to VW.

tsv2csv.py <input file> <output file>

Convert tab-separated file to comma-separated file.

chunk.py <input file> <number of output files> [<random seed>]

Split a file randomly line by line into a number of smaller files. Might be useful for preparing cross-validation. Output files will have the base nume suffixed with a chunk number, for example data.csv will be chunked into data_0.csv, data_1.csv etc.

count.py <input file>

Count lines in a file. On Unix you can do it with wc -l

sample.py <input file> <output file> [<P = 0.5>]

Sample lines from an input file with probability P. Similiar to split.py, but there's only one output file. Useful for sampling large datasets.

split.py <input file> <output file 1> <output file 2> [<P = 0.9>] [<random seed>]

Split a file into two randomly. Default P (probability of writing to the first file) is 0.9. You can specify any string as a seed for random number generator.

subset.py <input file> <output file> [<offset = 0>] [<lines = 100>]

Save a subset of lines from an input file to an output file. Start at offset (default 0), save lines (default 100).