drmy / exip-eval Goto Github PK

This project is a public release of the evaluation of the dedicated EXI grammar
generator implemented as part of the EXIP library. The grammar generator accepts
EXI encoded XML Schema definitions as an input
A set of 5 XML schema documents (available in the Schemas folder) are used for decoding 15 instances
(XML examples that conform to the schemas and available in XML-Instances folder; 3 instances per
each schema document) by 3 different EXI processors - EXIficient v0.9.1 Java,
OpenEXI v1.0238.0 Java, and EXIP v0.5.3 C.
The EXI encoded XML instance documents are located in EXI-Output.
The XML schema documents are: netconf.xsd, SenML.xsd, sep.xsd, OPC-UA-Types.xsd,
and XMLSchema.xsd. All of them are accessed from the local hard-drive,
including the imported XML schema files, so there were no dependencies on the
network performance.

The Tests folder contains the 3 implementations and 2 bash scripts for compiling
the sources of the 3 EXI processors (compileAll.sh) and running the evaluation (run-eval.sh).

The run-eval.sh script creates a CSV file called evaluation.csv which contains the
times to generate the EXI grammars dynamically for each processor and for each
XML Schema and the times to process the instances. All times are in nanoseconds.
Each row is one run of the evaluation for one instance.
The number of runs can be set in run-eval.sh by changing the value of NUMBER_OF_RUNS variable.

The compileAll.sh sets -O0 flag to gcc compiler (i.e. no compile time optimizations).
For compiling exip with optimizations (i.e. -O3 flag) use compileAllOptimize.sh

By default run-eval.sh executes OpenEXI and EXIficient tests in a fresh JVM and thus
the JVM runs the tests with limited run-time optimizations (such as JIT) as the JVM
cannot gain enough statistics for performing code optimizations.

When passing optimize parameter to run-eval.sh script (i.e. ./run-eval.sh optimize)
Each decoding test in OpenEXI and EXIficient is executed 5 times before the
timed decoding is performed thus the JVM has enough statistic to perform
run-time optimizations on the code.

For comparing in a optimization scenario use compileAllOptimize.sh to compile
and ./run-eval.sh optimize to run the tests as opposed to compileAll.sh and run-eval.sh.