Code Monkey home page Code Monkey logo

q2galaxy's Introduction

q2galaxy

An interface for generating Galaxy tool descriptions automatically from QIIME 2 actions.

Table of Contents

The Very Quick Start

This is not necessarily the best way to run q2galaxy, but it is the quickest way to take a peek at this interface.

Before starting, please be aware that your history will not persist between sessions, so you must download any results you wish to keep track of. To avoid this, use the Docker instructions instead.

First, activate your QIIME 2 environment, it will have q2galaxy already installed. Then, install Planemo:

pip install planemo

Then create a directory to store data in for q2galaxy

export Q2GALAXY_DATA=$HOME/q2galaxy_data
mkdir -p $Q2GALAXY_DATA

Next, template out the tools in your environment:

q2galaxy template all $Q2GALAXY_DATA

Finally run planemo (this will take a while as it must build Galaxy from source, using Docker will avoid this):

planemo serve --install_galaxy \
  --galaxy_branch qiime2 \
  --galaxy_source https://github.com/ebolyen/galaxy.git \
  --no_conda_auto_install \
  --no_conda_auto_init \
  --no_cleanup \
  --file_path $Q2GALAXY_DATA \
  $Q2GALAXY_DATA

Once that has finished, navigate to http://localhost:9090

!~ IMPORTANT ~!

When using planemo, there is not a convenient way to persist the database which means you will lose information in your History. The above command will store the datasets in $Q2GALAXY_DATA, however these are not conveniently named (they will end with .dat) and you may need to use qiime tools peek to figure out what they used to be.

To avoid losing History state, consider using Docker instead.

Known Limitations

This interface is currently in alpha, as such there are a few known problems and likely many other currently unknown issues. If you come across something you do not see listed, please create an issue!

Viewing QZV files

  • To use the view at qiime2view links, your Galaxy instance must be running over HTTPS with appropriate CORS headers set. Configuring those settings can be straight-forward if you are already familiar with these terms in a production environment but it is beyond the scope of these instructions.

Docker

  • Tools are not pre-installed, so the first job will take an inordinate amount of time as the environment is constructed.

Tool Environments

  • All tools share the same environment to reduce the burden of installing one for each plugin. This is likely to change in the future.

TypeMap

  • Constraints for inputs are not yet implemented
  • TypeMaps are treated as naive Unions

Semantic Properties

  • The generated tools will not accept artifacts which have been annotated with semantic properties

Metadata

  • Using artifacts as metadata is not yet supported
  • When providing metadata columns, it is possible to provide the ID column which will result in an error
  • Metadata columns are not yet typed

Usage

There are three subcommands to q2galaxy:

  • run
  • version
  • template

run and version are internal details and are what the Galaxy tool XML files will call (this means that q2galaxy needs to be installed as part of the tool definition, but this is handled automatically for you).

What you will be most interested in will be the template subcommand, which provides four additional subcommands:

  • template
    • all
    • builtins
    • plugin
    • tests

Each of these will take a directory to place the generated tools into. So if you wanted to create all of the tools available in your QIIME 2 environment (including builtins) you might run:

q2galaxy template all <some directory>

If you wanted to template only a single plugin, for example qiime feature-table, you can run:

q2galaxy template plugin feature_table <some directory>

(note that the plugin is provided as the ID form with underscores rather than dashes)

Once this is done, you can use the generated tool suites in a modified Galaxy installation. See below for additional details.

Galaxy Quickstart

As this is an alpha interface, some of the changes to add Galaxy datatypes are not yet finalized, which means that for the moment, you will need to use this fork and branch of Galaxy:

In the future this will not be necessary.

To skip most of the setup involved in a Galaxy deployment, there are two simple ways to test drive this interface, docker and planemo.

Docker

Docker is the more robust option at the moment for running q2galaxy. To learn more about how to use Docker, see this documentation:

Once you have Docker installed, you will want to pull the image from https://quay.io/repository/qiime2/q2galaxy. On the command line this might look like:

docker pull quay.io/qiime2/q2galaxy

Once you have the image downloaded, the next step is to run the container. Our container is based on:

and the instructions in that README will generally apply (just remember to change the container name). To persist any data, make sure you mount the /export directory as described in those instructions.

Usually you will start the server on http://localhost:8080, but this can be changed via the -p parameter.

Example:

docker run --platform linux/amd64 -d -p 8080:80 -p 8021:21 -p 8022:22 -v $HOME/q2galaxy_data/:/export/ quay.io/qiime2/q2galaxy

Building the image yourself

This can be skipped if you are not interested in customizing the image. If you are interested, see the readme here.

Planemo

This is more useful for those building plugin tool definitions who want to take a quick look at the results. Persisting history state does not appear to be possible.

In your QIIME 2 environment, run:

pip install planemo

Then you will want to template the tools you are interested in (see Usage above). Then you can run this command (in your QIIME 2 environment)

planemo serve --install_galaxy \
  --galaxy_branch qiime2 \
  --galaxy_source https://github.com/ebolyen/galaxy.git \
  --no_conda_auto_install \
  --no_conda_auto_init \
  --database_type sqlite \
  --no_cleanup \
  --file_path <directory to store data> \
  <directory of tools here>

This command will skip installation of any tools as they are assumed to already be available in your environment. To disable that, you can omit the --no_conda flags from the above command.

Note: this command can take quite some time as it will build the Galaxy UI from source. This will involve creating a lot of node modules via webpack which may appear intimidating.

Once that is finished, the server will be running on: http://localhost:9090

DIY Everything

For the very bravest, there are really only a few unusual parts to setting up the Galaxy instance. The first is to use the QIIME 2 specific fork and branch described above, and the next is to ensure that you have the correct configuration for conda and a tool config file so that your generated tools are accessible. For hints on this, see our Dockerfile, and tool_conf.xml and adapt as necessary.

q2galaxy's People

Contributors

bebatut avatar chriskeefe avatar colinvwood avatar ebolyen avatar hagenjp avatar lizgehret avatar oddant1 avatar q2d2 avatar thermokarst avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

q2galaxy's Issues

ToolShed deposit

Hi,
I've seen several people asking to install Qiime2 on different Galaxy usegalaxy.* servers.
What I see on this repo is interesting but it seems like it's not possible to install the tools the "standard way" from the ToolShed. Is this planned at some point on your roadmap?

Ensure that

Addition Description
A clear and concise description of what the addition is.

Current Behavior
Please provide a brief description of the current behavior, if applicable.

Proposed Behavior
Please provide a brief description of the proposed behavior.

Questions

  1. An enumerated list of questions related to the proposal.
  2. If not applicable, please delete this section.

Comments

  1. An enumerated list of comments related to the proposal that don't fit anywhere else.
  2. If not applicable, please delete this section.

References

  1. An enumerated list of links to relevant references, including forum posts, stack overflow, etc.
  2. If not applicable, please delete this section.

Some small things to improve

This is a small list of things that can make the wrappers a bit better. All of those are not important for the first version I think.

    <param name="seqs" type="data" format="qza" label="seqs: MultiplexedPairedEndBarcodeInSequence" help="[required]  The paired-end sequences to be demultiplexed.">

The [required] string can go away. If you do not use optional="true" everything is required and Galaxy will make sure the user knows about it.

    <section name="__q2galaxy__GUI__section__extra_opts__" title="Click here for additional options">

The __ might be confusing down the line. E.g. this will appear in Galaxy workflows etc and might be confusing for users that deal with it.

        <repeat name="front_f" help="[optional]  Sequence of an adapter ligated to the 5' end. The adapter and any preceding bases are trimmed. Partial matches at the 5' end are allowed. If a `^` character is prepended, the adapter is only found if it is at the beginning of the read. Search in forward read." title="front_f: List[Str]">

The title does not look nice. [optional] should be removed

            <param name="column" type="text" label="Column Name">

Should this not be a column parameter?

        <param name="minimum_length" type="integer" min="1" value="1" label="minimum_length: Int % Range(1, None)" help="[default: 1]  Discard reads shorter than specified value. Note, the cutadapt default of 0 has been overridden, because that value produces empty sequence records."/>

The [default: 1] can be removed, it is already encoded in the UI.

Config files and dynamic input params don't seem to cooperate

Given this xml:

   <command>q2galaxy run diversity pcoa '$inputs'</command>
   <configfiles>
      <inputs name="inputs"/>
   </configfiles>
   <inputs>
      <param name="example" type="integer" value="0" />
      <param format="qza" help="The distance matrix on which PCoA should be computed." name="distance_matrix" type="data">
         <options options_filter_attribute="metadata.semantic_type">
            <filter type="add_value" value="DistanceMatrix"/>
         </options>
      </param>
   </inputs>

The $inputs file is not including a key/value pair for distance_matrix. This could be related to my patch galaxyproject/galaxy#6443, or perhaps this has never quite worked right with options_filter_attribute/filter on data-params.

In any case, we have two options:

  1. fix the config file in galaxy (probably the right way to do this)
  2. actually make a command line interface for galaxy with key/value pairs

I'm definitely leaning towards option 1 and I'll see what I can dig up inside galaxy for why this might be happening.

cc @bgruening

Consider using docker-compose for stand-alone deployments

Addition Description
Docker-compose is one way to orchestrate docker services. It allows users or devs to define infrastructure-as-code and could help users deploy this plugin more easily.

Current Behavior
Docker services are built and launched one at a time

Proposed Behavior
Provide infrastructure to deploy q2galaxy along with the rest of galaxy, perhaps like this:
https://github.com/bgruening/docker-galaxy-stable

Questions
Project scope: Given that Qiime2 is focused on end users, deployment and orchestration could be out of scope for this plugin
Where's the best place to put this code? Within this repo, a separate repo, or as part of an existing docker infrastructure project?
What orchestration engine to use? Why not use Kubernetes instead of docker-compose?

User stories:
  • Qiime2 forums x-ref; user successfully deploys docker and q2galaxy on AWS, but has some trouble setting up ports

Running on Apple Silicon M1

Bug Description
Using the recommended command line input to start an image
docker run -d -p 8080:80 -p 8021:21 -p 8022:22 -v $HOME/q2galaxy_data/:/export/ quay.io/qiime2/q2galaxy
results in a warning
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
furthermore it seems to prevent exposure of port 8080 on the localhost.

Steps to reproduce the behavior

  1. Follow the docker steps in the README for Docker setup up to the last terminal command.
  2. While running macOS on a M1 chip go to terminal.
  3. Run docker run -d -p 8080:80 -p 8021:21 -p 8022:22 -v $HOME/q2galaxy_data/:/export/ quay.io/qiime2/q2galaxy
  4. See error WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Expected behavior
I expected the docker image to be created successfully.

Computation Environment

  • OS: [macOS Sonoma 14.1.1]
  • QIIME 2 Release [2024.2.1]

Fix

  1. Run this command to start the container instead (added --platform linux/amd64)
    docker run --platform linux/amd64 -d -p 8080:80 -p 8021:21 -p 8022:22 -v $HOME/q2galaxy_data/:/export/ quay.io/qiime2/q2galaxy

References

  1. Where I found the fix

qiime2_core__tools__import stacktrace from usegalaxy.eu

Hi,

we got this error report, let me know if you need more information.

Thanks for this great tool suite!

Tool ID: toolshed.g2.bx.psu.edu/repos/q2d2/qiime2_core__tools__import/qiime2_core__tools__import/2022.11.1+dist.h2bda5906.2

Error:

Traceback (most recent call last):
  File "/opt/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 291, in prepare_job
    job_wrapper.prepare()
  File "/opt/galaxy/server/lib/galaxy/jobs/__init__.py", line 1254, in prepare
    ) = tool_evaluator.build()
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 566, in build
    global_tool_logs(self._build_config_files, config_file, "Building Config Files")
  File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 96, in global_tool_logs
    raise e
  File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 92, in global_tool_logs
    return func()
           ^^^^^^
  File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 633, in _build_config_files
    self.__write_workdir_file(config_filename, config_text, param_dict, is_template=is_template)
  File "/opt/galaxy/server/lib/galaxy/tools/evaluation.py", line 756, in __write_workdir_file
    value = fill_template(content, context=context, python_template_version=self.tool.python_template_version)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/server/lib/galaxy/util/template.py", line 143, in fill_template
    raise first_exception or e
  File "/opt/galaxy/server/lib/galaxy/util/template.py", line 87, in fill_template
    return unicodify(t, log_exception=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/galaxy/server/lib/galaxy/util/__init__.py", line 1153, in unicodify
    value = str(value)
            ^^^^^^^^^^
  File "/opt/galaxy/venv/lib/python3.11/site-packages/Cheetah/Template.py", line 1053, in __unicode__
    return getattr(self, mainMethName)()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cheetah_DynamicallyCompiledCheetahTemplate_1699011302_5547879_70105.py", line 119, in respond
  File "cheetah_DynamicallyCompiledCheetahTemplate_1699011302_5547879_70105.py", line 102, in stringify
  File "cheetah_DynamicallyCompiledCheetahTemplate_1699011302_5547879_70105.py", line 113, in stringify
AttributeError: 'TreeDict' object has no attribute 'is_collection'

Add visualization plugin for .qzv files

Addition Description
There needs to be a way to view .qzv files inside Galaxy. It isn't yet clear how to dynamically serve files as each .qzv file is essentially a mini website with static assets and an index.html file.

Questions

  1. Should a visualization plugin be used for this? Or should it be a display application to q2view? The latter won't work with non-TLS endpoints.

References

  1. https://galaxyproject.org/develop/visualizations/

cc @blankenberg, any ideas here?

Add QIIME 2 metadata handling

Improvement Description
This will require a new QIIME 2 metadata dataset format in Galaxy. We should be able to map the column types to Galaxy metadata, but I'm not sure how much work moving that logic from the framework will be.

Extra text in tool version

Seems that the tool version command outputs some extra text:

Matplotlib created a temporary config/cache directory at /tmp/matplotlib-kaxxsc2g because the default path (/home/qiime2/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
diversity version 2022.11.1

I guess MPLCONFIGDIR should be set for the version command (and maybe even for command).

Alternatively one could use some redirection / grep / tail to get only the actual version.

QZA import/export inside Galaxy

Addition Description
We need a built-in tool for importing data into a QZA file and exporting as well. Ideally we let Galaxy's auto-detect figure out the export situation. Perhaps in the future we could use EDAM to do some of the translation...

Ensure that hidden actions are rendered in q2galaxy

Improvement Description
Typically we use hidden actions for the inner part of a parallel pipeline loop, e.g.

def some_parallel_pipeline():
  ...
  collection = partition_data()
  for key, val in collection.items():
    result[key] =  _hidden_method(val)
  viz = collate_data(result)
  ...

where partition_data, _hidden_method, and collate_data are all qiime2 actions.

With Galaxy, it is better to use the built-in for-each on a collection which will play nicely with whatever scheduler is present and will accomplish the same goal as the pipeline. So the user should be using the underlying actions directly instead of the pipeline.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.