the-sequence-ontology / soba Goto Github PK

The Sequence Ontology Bioinformatics Analysis command line tool (SOBAcl) will generate a variety of tables, graphs and reports from the data in GFF3 files and format the output in a variety of ways. Selections can be specified which will cause a subset of the data within files to be used to generate the output. The sections below give descriptions of the major features of SOBAcl. Detailed descriptions of each of the command line options will be printed by running SOBAcl --help. And finally, the shell script t/sobacl_test.sh has a lot of working examples of most of the major functionality. The command line examples given in this file can also be executed in the t directory.

This tools is also available for web based analysis here.

SOBAcl allows you to flexibly generate data tables by specifying the data that is to be presented in each row of the table, how the summarized data should be grouped in each column of the table, and what type of summary statistic should be reported in each cell of the table. All output for tables is printed to STDOUT.

Examples of tables include:

• Create a table of the count of every feature type grouped by file for multiple files.
• Create a table of the mean length of all features in a file.
• Create a table of the CDS footprint of all mRNAs by chromosome.

To create the above examples you would specify the rows, columns, data and data_type. For example the command line to generate a table of the count of every feature type grouped by file for multiple files would look like this - try it:

SOBAcl --columns file --rows type --data type --data_type count   \
  data/dmel-all-r5.30_0001000.gff data/dmel-all-r5.30_0010000.gff 

Tables can be output in 4 different formats: text, tab, html and html_page. The text format (used in the example above) produces an ASCII table that is aligned for viewing with fixed length fonts such as on a terminal screen. The tab format produces tab delimited files appropriate for import into a spreadsheet application such as Excel or for post processing with other command line tools like perl or awk. The html format will produce just the table in html format (for inclusion into a larger html page) and the html_page format will create an entire html page appropriate for viewing in a browser.

SOBAcl will produce a variety of graphs: lines, bars, hbars (horizontal bar chart), points, linespoints, area, and pie. For each of these graphs the --x_axis [rows] parameter specifies the data that becomes the x-axis values. The --series [columns] parameter defines the values that become the data series. The --y_axis [data] parameter determines the data used for the y-axis and the --data_type parameter specifies the summary statistic used to report the y-axis value. The name of the of the output file is specified on the command line with the --output parameter. The --collapse parameter determines if series will presented each on a different chart (this is always the case for pie charts) or whether series data will be collapsed onto a single chart.

The GD graphics library is used to create the graphs and parameters can be passed to the perl module GD::Graph as key=value pairs to the --gd parameter.

Examples of graphs include:

• Create a graph of the mean length of every feature by chromosome. Each feature would apprear on the x-axis, the mean length would be the y-axis and a different bar would apprear for each chromosome for every feature on the x-axis.
• Create a pie chart based on the footprint of each type of feature in the file.
• Create a line graph of the total length of CDSs on each chromosome.

To create a graph you would specify the x-axis, y-axis, series and data_type. For example the command line to generate a graph of the mean length of every feature by chromosome would be written:

SOBAcl --series seqid --x_axis type --y_axis length --data_type mean \
       --layout bars --gd width=600 --output seqid_type_hbars        \
       --format png data/dmel-all-r5.30_0001000.gff

If a --layout type of graphviz is specified, a graphical view of the directed acycilical graph (DAG) of the onology terms found in the file and the relationships between them will be generated. These graphs are generated by the GraphViz library and you can pass parameters to the perl module Graphviz as key=value pairs via the --gv --gv_node and --gv_edge parameters. This allows you to control many details (the color, shape, etc) of the graph as well as it's nodes and edges.

Some types of data are too complex to be easily summarized by charts or tables. SOBAcl provides built in reports to describe the data in GFF3 files in complex ways. Currently only one report is provided, but more will be developed in the future. The report currently supported is a count of all of the attributes in a file grouped by feature source and type. You can generate this report with the following command line:

SOBAcl --report attributes --format html_page \
     data/dmel-all-r5.30_0001000.gff

All text output from SOBAcl is generated by Perl's Template Toolkit templating system. You can pass a custom template on the command line to generate fully customized output. Template Toolkit offers a simple but very powerful templating syntax. SOBAcl processes a GFF3 file, prepares the data in various ways and then passes variables containing those data to the template engine. Those variables and the data they contain are then available within to the custom templates that you write. The contents of these variables include all of the parameter values that you specified on the command line to SOBAcl, as well as the data from the file summarized as you specified on the command line. Within your template you can create loops to iterate through the data, filter, transform or format the data in various ways and finally include the data in your output. A SOBAcl command line and a very simple template is given below. See the excellent documentation provided on the Template Toolkit website and have a look at the templates provided in the templates directory of this distribution to learn more.

SOBAcl --columns file --rows type --data length --data_type mean \
       --layout table --format tab                               \
       --template soba_custom_template_text.tt                   \
       data/dmel-all-r5.30_0001000.gff                           \
       data/dmel-all-r5.30_0010000.gff

My Template
[% title %]
[%- FOR row = data_set.keys.sort -%]
[%- FOR column = data_set.$row.keys.sort -%]
[%- data_set.$row.$column.stats.count -%]
[%- "\t" -%]
[%- data_set.$row.$column.stats.$data_type %]
[%- loop.last ? "\n" : "\t" -%]
[%- END -%]
[%- END -%]

Command line parameters passed to SOBAcl are passed along to the templates. Within the templates the values of those parameters are available as template variables via the name of the parameter. Template toolkit uses '[%' and '%]' to delimit templating commands. In the tempalte example above the text "My Template" would be printed in the output unchanged while '[% title %]' would print the title passed to SOBAcl via the --title parameter. The template commands '[% FOR %]' and '[% END %]' define loops which will iterate over the data summarized by SOBAcl. You will need to have a pretty good understanding of Template Toolkit to take full advantage of custom templates.

The data processed by SOBAcl is passed to the template via the data_set variable. For tables this data structure is a hash. The hash structure is shown below (first as it would appear in Perl and then as it would be used in a template:

$data_set{$row}{$column}{stats}->mean
data_set.$row.$column.stats.mean

The data prepared by SOBAcl for the attributes report looks like this:

$data_set{att_count}{$source}{$type}{$attribute_key}
data_set.att_count.$source.$type.$key

Notice how the template uses the '.' for both hash references and method calls. The '$' sigil is used by the template system to disambiguate an interpolated variable from a hash key or method call.

See the templates in the template directory for more examples.

GFF3 files may optionally be loaded into a database before by SOBAcl and SOBAcl can then use the database rather than a GFF3 flat file. This is most useful when complex SQL based data selection is needed as described below.

Sometime you only want to report or summarize some of the data in a GFF3 file. You can do this with SOBAcl on the fly by using the --select parameter. The format of values passed to the --select parameter is borrowed from Perl's SQL::Abstract module. Breifly, you pass a Perl hash to SOBAcl. Hash keys specify GFF3 columns and hash values specify the value to constrain that column's data to. SQL::Abstract allows very complex queries that expose most SQL feature via this interface. If you load your data into a SOBA database first (see above) you can exploit the full power of SQL::Abstract to build complex SQL queries from the command line. If you are processing a GFF3 text file rather than a database only one level of key value pairs is supported in the select parameter. The examples below show simple command lines that use a select statement:

SOBAcl --columns seqid --rows type --data length --data_type mean \
--layout table --format text --select 'type => ["ne", "contig"]'  \
data/refseq_short.gff3

SOBAcl --columns seqid --rows type --data length --data_type mean \
--layout table --format text                                      \
--select 'start => [">=", "1000"], end => ["<=", "1000000"]'      \
data/refseq_short.gff3

Note that both select values above use custom comparison operators. The following comparison operators are available:

== eq != ne < lt <= le > gt >= ge =~ !~

The '=' and '!' only available when processing text files because SQL does not have robust regular expression support. The 'eq' operator is the default if none is given in the statement in which case the array reference collapses to a simple value as seen in this select statements:

--select 'type => "contig"'