This is the GitHub repository for the program Clumppling (CLUster Matching and Permutation Program that uses integer Linear programmING), a framework for aligning clustering results of population structure analysis.
Current version v 0.1.0 (beta version)
The current version has been tested on Windows 10 with Python 3.8 to 3.10, Ubuntu 20.04 LTS with Python 3.10, MacOS 13 with Python 3.11, and RHEL (Red Hat Enterprise Linux) 8.6 with Python 3.11.
Detailed instructions can be found in the pdf Manual.
Questions and feedback are welcome. Contact Xiran Liu at [email protected].
There are two ways to run Clumppling.
- You can run it remotely on the server, which does not require downloading or installing the program locally. The remote version provides the core functionalities of the program. Check out the Remote Version section.
- You can download and install the Python package onto your local machine and run the program locally. The local version provides an extended list of functionalities (see the pdf Manual for details). Check out the Local Version section.
The remote version is available through an online Colaboratory notebook, which is a Jupyter notebook that runs in the cloud served by Google. If you are interested, more details about Colab notebooks can be found at https://colab.google/.
There is no need to download and install the program locally.
To run Clumppling remotely, click on THIS LINK which will bring you to the notebook. Next, follow the instructions in the notebook.
One by one, Click the run (little round-shaped buttons with a triangle in the middle) buttons next to each block on the left.
Upload input files (e.g., the example files provided here) as a zip folder, specify the input data format, and change input parameters (if needed) following the instructions.
You will be able to download a zipped file containing the alignment results at the end of the notebook.
The local version requires downloading and installing the program to your local machine.
- Linux and macOS users can use the built-in Terminal.
- For Windows users, you will need to obtain a terminal. For example:
- After you follow Step 3 to install Conda, you can use the built-in Anaconda Prompt available from the Anaconda Navigator. Note that the installation of Python and Conda on Windows only requires running the installers and there is no need for running commands in the command window.
- You may also use the built-in (for Windows 10) Windows PowerShell.
- Or, you can use Git Bash after you install Git by downloading and running the executable Git installer from https://git-scm.com/download/win.
You can download the Python installer from https://www.python.org/downloads/.
- For Windows users, go to https://www.python.org/downloads/windows/ to download the installer corresponding to your operating system, e.g., Windows installer (64-bit). Run the executable installer and check the box 'Add Python to environment variables' during the installation.
- For macOS users, go to https://www.python.org/downloads/macos/ to download the macOS 64-bit universal2 installer and double-click on the python--macosx.pkg file to start the Python installer.
- For Linux users, if Python is not pre-installed, you can install it via command lines (
sudo yum install -y python3
for CentOS and Red Hat Linux andsudo apt-get install python3
for all other Linux systems).
You can verify the installation by running
python --version
in the command line interpreter, which should give you the version of the installed Python.
Go to https://www.anaconda.com/download to download the conda installer and run the installer. Conda is a popular package management system and environment management system.
A virtual environment is a Python environment such that the Python interpreter, libraries and scripts installed into it are isolated from those installed in other virtual environments, and (by default) any libraries installed in a “system” Python, i.e., one which is installed as part of your operating system"
Using a virtual environment helps to keep the dependencies required by different projects separate and to avoid conflicts between projects.
Create a virtual environment named clumppling-env
by typing the following command in the command-line interpreter
conda create -n clumppling-env python
Activate the virtual environment by
conda activate clumppling-env
(1) Install the package
If you have Git installed, run the following command:
pip install git+https://github.com/PopGenClustering/Clumppling
If you don't have Git, run the following command:
pip install https://github.com/PopGenClustering/Clumppling/archive/master.zip
(2) Download the example files from the input directory in the GitHub repository
For each example dataset, unzip the files into a folder with the same name as the zip file, and put it inside a folder called "input" under a path of your choice.
For example, you may put it inside a path C:/Users/YOUR_USER_NAME/Clumppling
. Then the path for all the example files should be C:/Users/YOUR_USER_NAME/Clumppling/input
. The path for the Cape Verde example data should be C:/Users/YOUR_USER_NAME/Clumppling/input/capeverde
.
More will be discussed in the section How to Run (with an example).
Run the following command:
python -m clumppling -h
If the installation was successful, you should see the usage of the program in the command window. The usage tells you the required and optional arguments to the program. It should look like:
usage: __main__.py [-h] -i INPUT_PATH -o OUTPUT_PATH -f INPUT_FORMAT [-v VIS] [--cd_param CD_PARAM]
[--use_rep USE_REP] [--merge_cls MERGE_CLS] [--cd_default CD_DEFAULT] [--plot_modes PLOT_MODES]
[--plot_modes_withinK PLOT_MODES_WITHINK] [--plot_major_modes PLOT_MAJOR_MODES]
[--plot_all_modes PLOT_ALL_MODES] [--custom_cmap CUSTOM_CMAP]
required arguments:
-i INPUT_PATH, --input_path INPUT_PATH
path to load input files
-o OUTPUT_PATH, --output_path OUTPUT_PATH
path to save output files
-f INPUT_FORMAT, --input_format INPUT_FORMAT
input data format
optional arguments:
-v VIS, --vis VIS whether to generate visualization: 0 for no, 1 for yes (default)
--cd_param CD_PARAM the parameter for community detection method (default 1.0)
--use_rep USE_REP whether to use representative replicate as mode consensus: 0 for no (default), 1 for yes
--merge_cls MERGE_CLS
whether to merge all pairs of clusters to align K+1 and K: 0 for no (default), 1 for yes
--cd_default CD_DEFAULT
whether to use default community detection method (Louvain): 0 for no, 1 for yes (default)
--plot_modes PLOT_MODES
whether to display aligned modes in structure plots over a multipartite graph: 0 for no, 1 for
yes (default)
--plot_modes_withinK PLOT_MODES_WITHINK
whether to display modes for each K in structure plots: 0 for no (default), 1 for yes
--plot_major_modes PLOT_MAJOR_MODES
whether to display all major modes in a series of structure plots: 0 for no (default), 1 for
yes
--plot_all_modes PLOT_ALL_MODES
whether to display all aligned modes in a series of structure plots: 0 for no (default), 1 for
yes
--custom_cmap CUSTOM_CMAP
customized colormap as a comma-separated string of hex codes for colors: if empty (default),
using the default colormap, otherwise use the user-specified colormap
The main module takes in three required arguments and several optional ones. The required arguments are
-i
(--input_path
) path to load input files-o
(--output_path
) path to save output files-f
(--input_format
) input data format. This choice must be one of "structure", "admixture", "fastStructure", and "generalQ".
The optional arguments are cd_param
, use_rep
, merge_cls
, cd_default
, plot_modes
, plot_modes_withinK
, plot_major_modes
, plot_all_modes
, and custom_cmap
. Detailed explanation of these arguments can be found in the Manual.
As a quick start, let's use the Cape Verde data as an example. The data files are available in the zip file input/capeverde.zip.
-
Ensure that the data files have been successfully downloaded and put under the right directory. Following Step 4(2) from above, you should have already downloaded the zip file "capeverde.zip" and extracted the data files inside the zip file to a folder named "capeverde" under a directory called "input" that you created.
For example, the data files are extracted toC:/Users/YOUR_USER_NAME/Clumppling/input/capeverde
. -
Ensure that the current path is the parent directory of the "input" folder. In your command-line interpreter, make sure that you navigate to the directory where the folder "input" is located. For the Cape Verde example, you should be in the directory
C:/Users/YOUR_USER_NAME/Clumppling
.If you do not know your current path, you can run
pwd
to see it in the command window.The Cape Verde data files should be located in
input/capeverde
under your current path. I.e., When you runls input/capeverde
you should see a list of Cape Verde data files (CAPEVERDE_Rep1.2.indivq, CAPEVERDE_Rep1.3.indivq, etc.) -
Run the program under the default setting
python -m clumppling -i input/capeverde -o output/capeverde -f admixture
The outputs will be saved in
output/capeverde
under your current directory and a zipped file of the same name will also be generated. For example, if you are in the directoryC:/Users/YOUR_USER_NAME/Clumppling
, then the input will be saved inC:/Users/YOUR_USER_NAME/Clumppling/output/capeverde
and zipped inC:/Users/YOUR_USER_NAME/Clumppling/output/capeverde.zip
.Notes:
-
If your data files are in a directory different from the given example, make sure to specify the corresponding input path after
-i
. -
You can also change the parameters from those in the default setting. Here is another example for the chicken example (data available in the zip file input/chicken.zip), where we specify the parameters
cd_param
andcustom_cmap
:python -m clumppling -i input/chicken -o output/chicken_color -f structure --cd_param 1.05 --custom_cmap #D65859,#00AAC1,#01C0F6,#FDF0C4,#F1B38C,#AAD6BD,#6BB582,#B5DDF7,#AE8557,#FCEC73,#A4A569,#4264AC,#A1CDB2,#DE9D5D,#D9439A,#ABB2BA,#8775B3,#B3865C,#DADDE6,#E7BDD1,#FF9999
Here we are setting the community detection parameter
cd_param
to be 1.05. This parameter controls the resolution, i.e., the size of the detected communities, of the Louvain algorithm used for community detection. The default value is 1.0, and a larger value will result in smaller communities and more of them, thus detecting more modes. Please see Section 4.3 of the paper. If the default parameter value is giving undesirable mode detection results, for instance, assigning each replicate to its own mode or assigning obviously different replicates to a single mode, you may vary this parameter to get a desired size and number of the modes. We suggest increasing or decreasing the parameter incrementally by 0.05 to search for a good choice of value for your dataset.
We are also providing a custom colormap (custom_cmap
) for visualization of the alignment results. The clusters will be colored based on colors specified by the list of comma-separated HEX codes. -
More example commands to call the program can be found under the scripts directory in the GitHub repository.
-
Detailed instructions to run the program can be found in the Manual.
Liu, Kopelman, & Rosenberg (2023). Clumppling: cluster matching and permutation program with integer linear programming. [in prep]
The Cape Verde data used as the example comes from:
Verdu, P., Jewett, E. M., Pemberton, T. J., Rosenberg, N. A., & Baptista, M. (2017). Parallel trajectories of genetic and linguistic admixture in a genetically admixed creole population. Current Biology, 27(16), 2529-2535. https://doi.org/10.1016/j.cub.2017.07.002.
The chicken data used as the example comes from:
Rosenberg, N. A., Burke, T., Elo, K., Feldman, M. W., Freidlin, P. J., Groenen, M. A., ... & Weigend, S. (2001). Empirical evaluation of genetic clustering methods using multilocus genotypes from 20 chicken breeds. Genetics, 159(2), 699-713. https://doi.org/10.1093/genetics/159.2.699.
We thank Egor Lappo for helping with the packaging of the program. We thank Egor Lappo, Daniel Cotter, Maike Morrison, Chloe Shiff, and Juan Esteban Rodriguez Rodriguez for helping with the testing of the program.