rd_tool.py is a script for collecting rate-distortion curves across a series of either local or Amazon AWS nodes.

This script is run by arewecompressedyet.com, but can also be run locally to produce the same data.

You will need Python 3.4 or later, as well as boto3.

On Ubuntu, install with:

sudo apt install python3-tornado python3-boto3

Individual build machines do not need Python, but do need bash. Each machine should be configured with a user and work_root (such as that user's home directory). This work directory must be populated with a folder called daalatool and dump_ciede2000, which needs to contain a checkout of Daala and dump_ciede git with the tools built:

sudo apt install autoconf libogg-dev libjpeg-dev libpng-dev check python3-numpy python3-scipy

git clone https://git.xiph.org/daala.git daalatool
cd daalatool
./autogen.sh
./configure --disable-player
make tools -j4

For dump_ciede2000, exit the daalatool directory

cd ../

Install rust if you haven't already. You only need rust to compile the binary and don't need it on the individual machines.

curl -sf -L https://static.rust-lang.org/rustup.sh | sh

Build dump_ciede2000.

git clone https://github.com/KyleSiefring/dump_ciede2000
cd dump_ciede2000
cargo build --release

rd_tool will automatically create one slot directory per core, and upload codec binaries into that directory.

You will need a ~/.aws configuration for boto with your AWS information and credentials.

Specify the autoscaling groukp to use with -awsgroup.

You can specify all of the machines you want to use in a JSON file:

[
  {
    "host": "localhost",
    "user": "thomas",
    "cores": 4,
    "port": 22,
    "work_root": "/home/thomas/tmp",
    "media_path": "/home/thomas/sets"
  },
  {
    ...
  }
]

Specify this configuration on the command line with -machineconf.

Specify the path to the git checkout of thetested codec with the -bindir argument. rd_tool.py expects to find prebuilt binaries of the codec - it will not build the codec automatically.

Output files are written into the current directory. The format of the .out files is one quantizer per line, of the format

...

See RDRun.reduce() in work.py for an up-to-date list of metrics.