jyyulab / mica Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 3.0 509.05 MB

Mutual Information-based Non-linear Clustering Analysis

License: Apache License 2.0

Python 49.53% Shell 0.65% Common Workflow Language 1.52% Jupyter Notebook 17.43% R 30.78% Dockerfile 0.10%

clustering-analysis dimensionality-reduction graph-embedding mutual-information single-cell-rna-seq

mica's People

Contributors

Stargazers

Watchers

Forkers

chengzhouwu

mica's Issues

unable to lock file

Traceback (most recent call last):
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 87, in
main()
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 42, in main
args.umap_min_dist, args.tsne_perplexity, args.plot_dim, args.n_thread, args.dim_km)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 69, in clustering
common_name=out_name, dims=dim_km, num_processes=n_processes)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 47, in km_multiprocess
hdf = pd.HDFStore(mi_file)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 505, in init
self.open(mode=mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 627, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace

File "H5F.c", line 511, in H5Fopen
unable to open file
File "H5Fint.c", line 1519, in H5F_open
unable to lock the file
File "H5FD.c", line 1650, in H5FD_lock
driver lock request failed
File "H5FDsec2.c", line 941, in H5FD_sec2_lock
unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'

End of HDF5 error back trace

Unable to open/create file '/research/rgs01/scratch_lsf/lsftmpdir/nodecn257/93666268.tmpdir/tmp44ejg7gf/stg738b285c-7c95-47f0-b268-f59ca310f275/PBMC12k_reduced.h5'
Traceback (most recent call last):
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 87, in
main()
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 42, in main
args.umap_min_dist, args.tsne_perplexity, args.plot_dim, args.n_thread, args.dim_km)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 69, in clustering
common_name=out_name, dims=dim_km, num_processes=n_processes)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 47, in km_multiprocess
hdf = pd.HDFStore(mi_file)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 505, in init
self.open(mode=mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 627, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace

End of HDF5 error back trace

Unable to open/create file '/research/rgs01/scratch_lsf/lsftmpdir/nodecn257/93666268.tmpdir/tmpat8phv8x/stg0d07df0c-2800-4570-b2b5-578da606b2ff/PBMC12k_reduced.h5'
Traceback (most recent call last):
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 87, in
main()
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 42, in main
args.umap_min_dist, args.tsne_perplexity, args.plot_dim, args.n_thread, args.dim_km)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 69, in clustering
common_name=out_name, dims=dim_km, num_processes=n_processes)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 47, in km_multiprocess
hdf = pd.HDFStore(mi_file)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 505, in init
self.open(mode=mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 627, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace

End of HDF5 error back trace

Unable to open/create file '/research/rgs01/scratch_lsf/lsftmpdir/nodecn257/93666268.tmpdir/tmpygbj2irf/stgd5725581-afe0-4cf9-8759-1e05501f7ef5/PBMC12k_reduced.h5'
Traceback (most recent call last):
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 87, in
main()
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 42, in main
args.umap_min_dist, args.tsne_perplexity, args.plot_dim, args.n_thread, args.dim_km)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 69, in clustering
common_name=out_name, dims=dim_km, num_processes=n_processes)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 47, in km_multiprocess
hdf = pd.HDFStore(mi_file)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 505, in init
self.open(mode=mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 627, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace

End of HDF5 error back trace

Unable to open/create file '/research/rgs01/scratch_lsf/lsftmpdir/nodecn257/93666268.tmpdir/tmpx7y22iru/stgbf77246b-014b-44c4-87d1-d50c26f59f3d/PBMC12k_reduced.h5'
Traceback (most recent call last):
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 87, in
main()
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 42, in main
args.umap_min_dist, args.tsne_perplexity, args.plot_dim, args.n_thread, args.dim_km)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 69, in clustering
common_name=out_name, dims=dim_km, num_processes=n_processes)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 47, in km_multiprocess
hdf = pd.HDFStore(mi_file)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 505, in init
self.open(mode=mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 627, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace

End of HDF5 error back trace

Unable to open/create file '/research/rgs01/scratch_lsf/lsftmpdir/nodecn257/93666268.tmpdir/tmpoepfbbbb/stgdafed1ba-cf6d-4110-96bb-86e62d6f90be/PBMC12k_reduced.h5'
Traceback (most recent call last):
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 87, in
main()
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 42, in main
args.umap_min_dist, args.tsne_perplexity, args.plot_dim, args.n_thread, args.dim_km)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 69, in clustering
common_name=out_name, dims=dim_km, num_processes=n_processes)
File "/research/rgs01/home/clusterHome/lding/.conda/envs/py361_test/lib/python3.6/site-packages/MICA-0.1.0-py3.6.egg/MICA/bin/clustering.py", line 47, in km_multiprocess
hdf = pd.HDFStore(mi_file)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 505, in init
self.open(mode=mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/pandas/io/pytables.py", line 627, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/lding/.conda/envs/py361_test/lib/python3.6/site-packages/tables-3.5.2-py3.6-linux-x86_64.egg/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace

End of HDF5 error back trace

Unable to open/create file '/research/rgs01/scratch_lsf/lsftmpdir/nodecn257/93666268.tmpdir/tmp33igsuo1/stg721ff009-e8c7-4c7c-8084-13e3d868b7fa/PBMC12k_reduced.h5'
?[1;30mINFO?[0m [job clustering_5] Max memory used: 15MiB
?[1;30mINFO?[0m [job clustering_6] Max memory used: 18MiB
?[1;30mINFO?[0m [job clustering] Max memory used: 17MiB
?[1;30mINFO?[0m [job clustering_3] Max memory used: 19MiB
?[1;30mINFO?[0m [job clustering_7] Max memory used: 18MiB
?[1;30mERROR?[0m ?[31m[job clustering_5] Job error:
Error collecting output for parameter 'out_fig':

Use cwlexe config file in the installed location by default

Remove "--project-name" argument?

The best practice is probably to let users specify a meaningful output dir name using argument "--output-dir" and put all the final output files in it.

In terms of temporary files, it'd be good to add temp dir support as an argument as both cwltool and cwlexec support it.

MICA PermissionError: [Errno 13] Permission denied: 'prep.py'

Hi i met following error when i run MICA:

(omicverse6) root@jiang:/home/jzhou/MICA/MICA/bin# mica mds -i /mnt/c/@@@@@@@@TCF1_T_scATAC/new/CD8TMICA.h5ad -o /mnt/c/@@@@@@@@TCF1_T_scATAC/new/ -pn TRAMP -nc 16
INFO:root:Start MDS mode...
INFO:root:/mnt/c/@@@@@@@@TCF1_T_scATAC/new/mica.yml
INFO:root:infile:
class: File
path: /mnt/c/@@@@@@@@TCF1_T_scATAC/new/CD8TMICA.h5ad
project_name: TRAMP
k: [16]
num_neighbor: None
visualization: UMAP
dim_reduction: MDS
iterations_km: 10
dims_km: [19]
dims: 19
dims_plot: 19
perplexity: 30
min_dist: 0.6
slice_size: 1000
thread_number: 10
dist_metrics: mi

INFO:root:cwltool --leave-tmpdir --outdir /mnt/c/@@@@@@@@TCF1_T_scATAC/new/ /root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/cwl/mica.cwl /mnt/c/@@@@@@@@TCF1_T_scATAC/new/mica.yml
INFO /root/miniconda3/envs/omicverse6/bin/cwltool 3.0.20210124104916
INFO Resolved '/root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/cwl/mica.cwl' to 'file:///root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/cwl/mica.cwl'
INFO [workflow ] start
INFO [workflow ] starting step prep
INFO [step prep] start
INFO [job prep] /tmp/yaoonq_5$ prep.py
-i
/tmp/29l_o7i3/stg1f004596-266a-4d71-b37b-9a6a03ba4e1f/CD8TMICA.h5ad
-o
TRAMP
-s
1000
ERROR Exception while running job
Traceback (most recent call last):
File "/root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/cwltool/job.py", line 388, in _execute
default_stderr=runtimeContext.default_stderr,
File "/root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/cwltool/job.py", line 948, in _job_popen
universal_newlines=True,
File "/root/miniconda3/envs/omicverse6/lib/python3.7/subprocess.py", line 800, in init
restore_signals, start_new_session)
File "/root/miniconda3/envs/omicverse6/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: 'prep.py'
WARNING [job prep] completed permanentFail
ERROR [step prep] Output is missing expected field file:///root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/cwl/mica.cwl#prep/mat_pair_array
WARNING [step prep] completed permanentFail
INFO [workflow ] completed permanentFail
{
"clustering_plots": null,
"clustering_res": null,
"initial_plots": null,
"mi.h5": null,
"mi.reduced": null
}
WARNING Final process status is permanentFail
Traceback (most recent call last):
File "/root/miniconda3/envs/omicverse6/bin/mica", line 33, in
sys.exit(load_entry_point('MICA==0.2.2', 'console_scripts', 'mica')())
File "/root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/mica.py", line 96, in main
mica_mds.mica_mds(args)
File "/root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/mica_mds.py", line 122, in mica_mds
run_shell_command_call(cmd)
File "/root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/mica_mds.py", line 162, in run_shell_command_call
subprocess.check_call(cmd_to_exec)
File "/root/miniconda3/envs/omicverse6/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cwltool', '--leave-tmpdir', '--outdir', '/mnt/c/@@@@@@@@TCF1_T_scATAC/new/', '/root/miniconda3/envs/omicverse6/lib/python3.7/site-packages/MICA/cwl/mica.cwl', '/mnt/c/@@@@@@@@TCF1_T_scATAC/new/mica.yml']' returned non-zero exit status 1.
(omicverse6) root@jiang:/home/jzhou/MICA/MICA/bin#

Can you help me?

best
Jiang

Error suspected in step03 in running kmeans.py locally

nodecn112 rprod MICA$ ./tests/test_PBMC_mini_no_install_local.sh
[INFO] --> [MICA] Finished.

OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 1033162 max

OSError: ./test_data/outputs/test_no_install_local/MICA_test_no_install_local_MDS_3/MICA_out/.tmp does not exist
Traceback (most recent call last):
File "/home/lding/Git/MICA/MICA/kmeans.py", line 7, in
utils.kmeans(sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4]), sys.argv[5], sys.argv[6])
File "/research/rgs01/home/clusterHome/lding/Git/MICA/MICA/utils.py", line 277, in kmeans
hdf = pd.HDFStore(in_mat_file)
File "/hpcf/apps/python/install/3.6.1/lib/python3.6/site-packages/pandas/io/pytables.py", line 467, in init
self.open(mode=mode, **kwargs)
File "/hpcf/apps/python/install/3.6.1/lib/python3.6/site-packages/pandas/io/pytables.py", line 580, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/hpcf/apps/python/install/3.6.1/lib/python3.6/site-packages/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/hpcf/apps/python/install/3.6.1/lib/python3.6/site-packages/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 371, in tables.hdf5extension.File._g_new
File "/hpcf/apps/python/install/3.6.1/lib/python3.6/site-packages/tables/utils.py", line 185, in check_file_access
check_file_access(filename, 'w')
File "/hpcf/apps/python/install/3.6.1/lib/python3.6/site-packages/tables/utils.py", line 175, in check_file_access
raise IOError("%s does not exist" % (parentname,))

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.