jlevy44 / joshua-levy-synteny-analysis Goto Github PK

x - done
~ - WIP/testing

Maybe:

• -Sliders for clusterGraph
• -Getting pipeline to shut down
• - Changing iteration calculation on clusterGraph for spring layout

Going to fix:

• -Easy feature coloring for clusterGraph
• -adding low memory option
• -Add generating centromere bed

Fixes after v1:

• - Classifier
• -Allow user to specify any versus all in calculating differential Kmers or which scaffold belongs to which subgenome... Modifying scaffasta and kmerCompare
• - Adding more feature extraction options to extractTelo/Centromeres
• - Physics simulation of spectral clustering in multidimensions
• - Adding more dimensionality reduction techniques and fix factor analysis memory issues (find a library that will output spicy sparse matrices after dimensionality reduction)
• - Add SGE and slurm support
• - Fix bug that causes config file to get erased sometimes
• - Add documentation and comment code

Not sure if there are others... Will check notes.
Will look into memory issues caused by kpca transformation...
Could be related to total number of scaffolds

c6/98d480] Submitted process > BlastOff (1)
WARN: No more task to compute -- Execution may be stalled (see the log file for details)
WARN: No more task to compute -- Execution may be stalled (see the log file for details)
[25/5ac550] Submitted process > blast2bed (1)
ERROR ~ Error executing process > 'blast2bed (1)'

Caused by:
Process blast2bed (1) terminated with an error exit status (1)

Command executed:

#!/bin/bash
cd /global/projectb/scratch/sgordon/polysCRACKER/run_dirs/wheat.1_5Mbp.k23
python subgenomeClusteringInterface.py blast2bed ./blast_files/Chinese_Spring_v10_pseudomolecules_161010_shreded_split.kcount.BLASTtsv.txt 1

Command exit status:
1

Command output:
(empty)

Command error:
Traceback (most recent call last):
File "subgenomeClusteringInterface.py", line 1737, in
main()
File "subgenomeClusteringInterface.py", line 1735, in main
optionsfunct
File "subgenomeClusteringInterface.py", line 145, in blast2bed
b = pybedtools.BedTool('blasted.bed').sort().merge(c=4,o='collapse',)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/pybedtools/bedtool.py", line 806, in decorated
result = method(self, *args, **kwargs)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/pybedtools/bedtool.py", line 337, in wrapped
decode_output=decode_output,
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/pybedtools/helpers.py", line 356, in call_bedtools
raise BEDToolsError(subprocess.list2cmdline(cmds), stderr)
pybedtools.helpers.BEDToolsError:
Command was:

    bedtools sort -i blasted.bed

Error message was:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

[7e/4f92f5] Submitted process > genClusterMatrix_kmerPrevalence (1)
FAN_r11_split.kcount.fa
[93/956254] Submitted process > transform_main (2)
[df/56161c] Submitted process > transform_main (1)
ERROR ~ Error executing process > 'transform_main (1)'

Caused by:
Process transform_main (1) terminated with an error exit status (1)

Command executed:

#!/bin/bash
cd /global/projectb/scratch/sgordon/polysCRACKER/run_dirs/OctoStrawberry.250kb.k23
python subgenomeClusteringInterface.py transform_main 1 ./recluster_files/ factor 2 cosine

Command exit status:
1

Command output:
(empty)

Command exit status:
1

Command output:
(empty)

Command error:
Traceback (most recent call last):
File "subgenomeClusteringInterface.py", line 1599, in
main()
File "subgenomeClusteringInterface.py", line 1597, in main
optionsfunct
File "subgenomeClusteringInterface.py", line 365, in transform_main
transform_plot((main,reclusterFolder,model,n_subgenomes,metric))
File "subgenomeClusteringInterface.py", line 445, in transform_plot
data = KernelPCA(n_components=499).fit_transform(data)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/sklearn/decomposition/kernel_pca.py", line 260, in fit_transform
self.fit(X, **params)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/sklearn/decomposition/kernel_pca.py", line 236, in fit
K = self._get_kernel(X)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/sklearn/decomposition/kernel_pca.py", line 164, in _get_kernel
**params)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/sklearn/metrics/pairwise.py", line 1399, in pairwise_kernels
return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/sklearn/metrics/pairwise.py", line 1083, in _parallel_pairwise
return func(X, Y, **kwds)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/sklearn/metrics/pairwise.py", line 735, in linear_kernel
return safe_sparse_dot(X, Y.T, dense_output=True)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/sklearn/utils/extmath.py", line 186, in safe_sparse_dot
ret = ret.toarray()
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/scipy/sparse/compressed.py", line 964, in toarray
return self.tocoo(copy=False).toarray(order=order, out=out)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/scipy/sparse/coo.py", line 252, in toarray
B = self._process_toarray_args(order, out)
File "/global/homes/s/sgordon/.conda/envs/polysCRACKER090617/lib/python2.7/site-packages/scipy/sparse/base.py", line 1039, in _process_toarray_args
return np.zeros(self.shape, dtype=self.dtype, order=order)
MemoryError
.command.run.1: line 99: 23466 Terminated nxf_trace "$pid" .command.trace

Work dir:
/global/projectb/scratch/sgordon/polysCRACKER/run_dirs/OctoStrawberry.250kb.k23/work/df/56161cf1e10d9bb0f217bf78cc03e7

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out

-- Check '.nextflow.log' file for details
WARN: Killing pending tasks (1)

http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.KernelPCA.html http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html#sklearn.neighbors.DistanceMetric https://docs.scipy.org/doc/scipy/reference/spatial.distance.html