WeaveQ: A program and module for pivoting and joining across collections of data, with special support for pivoting and joining across JSON files, CSV files and Elasticsearch resultsets.
Prerequisites
In order to install WeaveQ you will need:
- Python 2.7 or higher (including Python 3)
- pip, a Python package installer (the pip package is usually called
python-pip
in Linux package repositories)
For development of WeaveQ, you will need:
- tox >= 2.7.0 (available from PyPi)
- six >= 1.5.2 (available from PyPi)
- pyparsing >= 2.2.0 (available from PyPi)
- elasticsearch >= 5.1.0 (available from PyPi)
- elasticsearch_dsl >= 5.1.0 (available from PyPi)
- Doxygen (optional, available from your OS package repository)
- Sphinx (optional, available from PyPi)
Installing
To install the latest stable WeaveQ, use pip (you may need to sudo
):
$ pip install weaveq
To install from source, run the setup.py
file located in the WeaveQ
repository root (you may need to sudo
):
$ python setup.py install
Documentation
Comprehensive WeaveQ documentation is located at: <https://readthedocs.org/projects/weaveq/>
To build the documentation in the WeaveQ repository locally, from the repository root:
$ doxygen $ cd docs $ make html
In-code HTML documentation will be written to docs/code/html
and
user/developer HTML documentation will be written to docs/build/html
.
There are 3 types of WeaveQ tests:
- Unit: Exercises individual classes.
- System: Exercises unit integration, integration with external modules and integration with Elasticsearch.
- Performance: Measures average performance of pre-defined operations to help gauge performance impact of code changes.
For the system tests to work correctly, you need to have an Elasticsearch
instance accessible from your development system (a vanilla, single local
node should be sufficient). You will have to configure the system tests to
point to the Elasticsearch node - do this by entering the host and port in
the tests/system/elastic_test_config.json
file.
To run the unit and system tests, from the repository root run:
$ tox
tox
is used to manage virtual environments for testing that WeaveQ installs
and runs with multiple Python versions. To successfully test both Python 2.7
and Python 3 compatibility, you need to have these interpreters installed on
your system.
To run the performance tests, from the repository root run:
$ python perf_tests.py
Note that WeaveQ compatibility with the following has not been tested in this release (though this doesn't necessarily mean it won't work):
- Python versions < 2.7.6 or > 3.4
- Operating systems other than Linux
- Elasticsearch versions < 5.1.2
In order to use Elasticsearch data sources in WeaveQ queries, you must supply a configuration file on the command line. This file specifies the addresses/host names of Elasticsearch nodes, authentication certificates and other settings.
If deploying to a controlled environment with an Elasticsearch instance that
you want users to access by default, consider providing a pre-written
configuration file (for example, in /etc/weaveq
) so that users can get up
and running quickly. You may also want to alias the weaveq
command to
specify the default configuration without requiring users to specify the
--config
option.
The configuration file format is documented at <https://readthedocs.org/projects/weaveq/>
WeaveQ versioning follows the SemVer specification: <http://semver.org/>
You can see the tagged WeaveQ versions at: <https://github.com/jamesmistry/weaveq/tags>
This project is licensed under the MIT License - see the LICENSE file.