diffDomain is working for "dvsd one", however I am unable to get "dvsd multiple" to work with 2 hic files and a bed file.

Using the code below, I get "1 not found in the file." repeated for the total number of TADs in the bed file, then the program hangs indefinitely.
If the chr column is populated with "chr1", I get the message below.
If the chr column is populated with "1", I get the message below.
If I explicitly add --chrn "1" to the command, I get the message below.
If I explicitly add --chrn "chr1" to the command, the program immediately finishes and the output file only has the # commented rows.

python /home/########/diffDomainENV/diffDomain/diffdomain-py3/diffdomains.py dvsd multiple Control.hic Treatment.hic Control_TADs_chr1.txt --ofile Ctrl.vs.Treatment.at.Ctrl_chr1.txt --hicnorm KR --ncore 1 --reso 10000 --min_nbin 5
diffdomains.py
1 not found in the file.
1 not found in the file.
1 not found in the file.
1 not found in the file.
etc
etc
etc.

Hi author:
The diffDomain is a useful tool. But there seems to be something strange about my results, all the TADs have no p-values, is it possible that the input file should provide the original matrix not the iced matrix.
thanks.

Hello again!

I have an issue while I was testing dvsd one.

I use as follow :

python ../diffDomain/diffdomain-py3/diffdomains.py dvsd one chr2 10000000 15000000 --reso 40000 data1.mcool data2.mcool

And I obtain the error :

Traceback (most recent call last):
  File "../diffDomain/diffdomain-py3/diffdomains.py", line 45, in <module>
    result = comp2domins_by_twtest(chrn=opts['<chr>'], start=int(opts['<start>']), end=int(opts['<end>']), reso=int(opts['--reso']), hicnorm=opts['--hicnorm'], fhic0=opts['<hic0>'], fhic1=opts['<hic1>'], min_nbin=int(opts['--min_nbin']), f=opts['--f'])
  File "../diffdomain-py3/utils.py", line 398, in comp2domins_by_twtest
    Diffmatnorm = normDiffbyMeanSD(D=Diffmat)
  File "../diffDomain/diffdomain-py3/utils.py", line 291, in normDiffbyMeanSD
    b[k] = np.max(val1)
  File "<__array_function__ internals>", line 180, in amax
  File "/env/products/python/3.8.11/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 2791, in amax
    return _wrapreduction(a, np.maximum, 'max', axis, None, out,
  File "/env/products/python/3.8.11/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity

It's seem that this issue isn't visible when you use a hic file.

Thank you in advance for your answers.

Hi! Thanks for this tool!
I have a question about the format of input file.
Only file.hic are accepted ? Or can we use an other format (like cool or mcool) with a trick ?
If not are you planning to add more format possibilities or to stay with only the hic format ?

Thank you in advance for your answers.

Hi, I have replicates for the two conditions to test. How I can I use them? I don't see option to use replicates.

I attached a screenshot of the "dvsd multiple" output file, and I can see the first 4 columns are TAD information I provided as input, and the 6th column is the p-value as mentioned in the wiki. Could you explain what do the 5th column and 7th column mean?

Also, I provided 2127 TAD entries in the input file, why the output table only gives information of 1885 of them?

Thank you!

Hi,

I tried your example 2. The commands and output results(adjusted_TADs2.txt_types.txt) are attached.

# ========= Example2: Testing TADs on all chromosomes ============= 

# <bed> data/GSE63525_GM12878_primary+replicate_Arrowhead_domainlist.txt
# <ofile: output file> temp/temp.txt
python3 ${diffdomain_py3}diffdomains.py dvsd multiple ${hic0} ${hic1} ${diffdomain_home}data/GSE63525_GM12878_primary+replicate_Arrowhead_domainlist.txt --ofile ${res_path}temp/temp.txt

# <input> temp/temp.txt
# <output> adjusted_TADs2.txt 
python ${diffdomain_py3}diffdomains.py adjustment fdr_bh ${res_path}temp/temp.txt ${res_path}adjusted_TADs2.txt 
# --filter true adj_pvalue<=0.05
# The final output is saved to <diff_res/reorganized_TADs_GM12878_K562.tsv>
python ${diffdomain_py3}diffdomains.py adjustment fdr_bh ${res_path}temp/temp.txt ${res_path}reorganized_TADs_GM12878_K562.tsv --filter true

cd ${res_path}
# -d diffdoamin's outcome
# -t the other tadlist
# output diffDomain/diff_res/{}_types.txt
python ${diffdomain_py3}classification.py -d ${res_path}adjusted_TADs2.txt  -t ${diffdomain_home}data/GSE63525_K562_Arrowhead_domainlist.txt
python ${diffdomain_py3}classification.py -d ${res_path}reorganized_TADs_GM12878_K562.tsv  -t ${diffdomain_home}data/GSE63525_K562_Arrowhead_domainlist.txt

In the result, it is observed that, for some regions, there are only condition 1 entries whose type is loss; while for other regions, the type of consition 2 are always nan.
I am not sure what happend. May I ask whether I made any mistakes when using the diffDomain tools? Could you please help with this problem?

Thank you.

Best regards,
Stacey LIU

adjusted_TADs2.txt
adjusted_TADs2.txt_types.txt

Hello,

Is there any plan to upgrade to Python 3.X, as some dependencies failed to install due to the lack of support for Python 2.7?
e.g. h5py==2.9,

What’s new in h5py 2.9

Support for old Python

Support for Python 3.3 has been dropped.
Support for Python 2.6 has been dropped.

Hello,
Thanks for this new tool. I am attempting to run diffDomain using the command python diffdomain-py3/diffdomains.py dvsd multiple input/h9_merged_30_25kb_25000.cool input/smpc_merged_30_25kb_25000.cool input/h9_merged_30_25kb_normKR.bed --reso 25000 --ofile output/ --oprefix hPSC_vs_FetalSMPC --oprefixFig hPSC_vs_FetalSMPC --hicnorm KR
but keep getting this error:
`multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/data/anaconda3/envs/diffdomain/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "diffdomain-py3/diffdomains.py", line 59, in comp2domins_by_twtest_parallel
fhic0=opts[''], fhic1=opts[''],min_nbin=int(opts['--min_nbin']),f=opts['--f'])
File "/data/diffDomain/diffdomain-py3/utils.py", line 379, in comp2domins_by_twtest
Diffmatnorm = normDiffbyMeanSD(D=Diffmat)
File "/data/diffDomain/diffdomain-py3/utils.py", line 260, in normDiffbyMeanSD
b[k] = np.max(val1)
File "<array_function internals>", line 6, in amax
File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2755, in amax
keepdims=keepdims, initial=initial, where=where)
File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "diffdomain-py3/diffdomains.py", line 76, in
result.append(i.get())
File "/data/anaconda3/envs/diffdomain/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
ValueError: zero-size array to reduction operation maximum which has no identity`

I installed using conda. I have tried both .mcool and cool files as well as .hic files. neither work. When using .hic files I receive the error:
Traceback (most recent call last): File "diffdomain-py3/diffdomains.py", line 54, in <module> tadb = loadtads(opts['<bed>'], sep=opts['--sep'], chrnum=opts['--chrn'], min_nbin=int(opts['--min_nbin']), reso=int(opts['--reso'])) File "/data/diffDomain/diffdomain-py3/utils.py", line 44, in loadtads tadb.iloc[:,1:3] = tadb.iloc[:,1:3].astype(int) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/generic.py", line 5815, in astype new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 418, in astype return self.apply("astype", dtype=dtype, copy=copy, errors=errors) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 327, in apply applied = getattr(b, f)(**kwargs) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 591, in astype new_values = astype_array_safe(values, dtype, copy=copy, errors=errors) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/dtypes/cast.py", line 1309, in astype_array_safe new_values = astype_array(values, dtype, copy=copy) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/dtypes/cast.py", line 1257, in astype_array values = astype_nansafe(values, dtype, copy=copy) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/dtypes/cast.py", line 1095, in astype_nansafe result = astype_nansafe(flat, dtype, copy=copy, skipna=skipna) File "/data/anaconda3/envs/diffdomain/lib/python3.7/site-packages/pandas/core/dtypes/cast.py", line 1174, in astype_nansafe return lib.astype_intsafe(arr, dtype) File "pandas/_libs/lib.pyx", line 679, in pandas._libs.lib.astype_intsafe TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Not quite sure what I am doing wrong, but any help would be great!
Thanks!

Hello,
I am trying to run the tool with the following command:

python diffdomains.py dvsd multiple /data/T1_10000.cool /data/T2_10000.cool /data/TADS_DM.bed --reso 10000

But after appearing to run fine for a while it throws the following error:

...
chr9:75210000:75430000
chr9:106940000:107100000
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "diffdomains.py", line 66, in
comp2domins_by_twtest_parallel(0)
File "diffdomains.py", line 59, in comp2domins_by_twtest_parallel
fhic0=opts[''], fhic1=opts[''],min_nbin=int(opts['--min_nbin']),f=opts['--f'])
File "/diffdomain-py3/utils.py", line 338, in comp2domins_by_twtest
mat0 = contact_matrix_from_hic(chrn, start, end, reso, fhic0, hicnorm)
File "/diffDomain/diffdomain-py3/utils.py", line 181, in contact_matrix_from_hic
c = cooler.Cooler(f'{hic_norm}')
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 85, in init
self._refresh()
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 89, in _refresh
with open_hdf5(self.store, **self.open_kws) as h5:
File "/home/micromamba/envs/diffDomain3/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/util.py", line 525, in open_hdf5
fh = h5py.File(fp, mode, *args, **kwargs)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 427, in init
swmr=swmr)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/micromamba/envs/diffDomain3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "diffdomains.py", line 59, in comp2domins_by_twtest_parallel
fhic0=opts[''], fhic1=opts[''],min_nbin=int(opts['--min_nbin']),f=opts['--f'])
File "/diffDomain/diffdomain-py3/utils.py", line 338, in comp2domins_by_twtest
mat0 = contact_matrix_from_hic(chrn, start, end, reso, fhic0, hicnorm)
File "/diffDomain/diffdomain-py3/utils.py", line 181, in contact_matrix_from_hic
c = cooler.Cooler(f'{hic_norm}')
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 85, in init
self._refresh()
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/api.py", line 89, in _refresh
with open_hdf5(self.store, **self.open_kws) as h5:
File "/home/micromamba/envs/diffDomain3/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/cooler/util.py", line 525, in open_hdf5
fh = h5py.File(fp, mode, *args, **kwargs)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 427, in init
swmr=swmr)
File "/home/micromamba/envs/diffDomain3/lib/python3.6/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (file signature not found)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "diffdomains.py", line 76, in
result.append(i.get())
File "/home/micromamba/envs/diffDomain3/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
OSError: Unable to open file (file signature not found)

Any help will be highly appreciated!
Thanks :)

Hi,
I saw your updated DiffDomain for python3. https://pypi.org/project/DiffDomain/0.0.8/
I successfully install it on python 3.9.12 as well as all of the dependencies.
However, when I run the command below, the program crashes immediately with the trace-back below.
I look forward to using this program as the paper is very promising!

`python /home/#########/software/diffDomain_v0.0.8/lib/python3.9/site-packages/diffdomain/diffdomains.py dvsd multiple
Control.hic Treatment.hic Control.bed --hicnorm KR --reso 10000 --oprefix=Control.vs.Treatment

Traceback (most recent call last):
File "/home/#########/software/diffDomain_v0.0.8/lib/python3.9/site-packages/diffdomain/diffdomains.py", line 31, in
from .utils import comp2domins_by_twtest, loadtads, visualization
ImportError: attempted relative import with no known parent package`

Hi,

Thanks for this wonderful tool to see the TAD changes among 2 condition!
How to identity the gained loops in condition 2?

Thank you,
Pinpin Sui

Hi,

We're facing a few failed attempts when trying to install diffdomain on our HPC cluster (from the container itself)...
For approach one, a conda environment, we tried our personal account on a standalone server with internet access, because it appears to access a mirror site is not in our firewall list. the error message with the command line
conda env create --name diffdomain -f environment_linux.yml

... CondaEnvException: Pip failed
Pip subprocess error:
ERROR: Ignored the following versions that require a different python version: 0.12.0 Requires-Python >=3.8; 0.12.0rc1 Requires-Python >=3.8; 0.12.1 Requires-Python >=3.8
ERROR: Could not find a version that satisfies the requirement diffdomain-py3==0.2.0 (from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4,
0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.1, 0.2.2)
ERROR: No matching distribution found for diffdomain-py3==0.2.0

For a container approach, I am able to pull a apptainer image on a login node on seadragon2 using the command:
apptainer pull docker://guming5/diffdomain-centos7:v1
But the instruction of using the container, " "In this image, there is a contact conda environment named diffdomain (/home/work/.conda/envs/diffdomain) meeting all requests, in which you can use the diffDomain Python3 version directly."
I failed to change to user 'work', it asks a password.

I was wondering if you could provide any suggestions to resolve the above errors.

Thank you!

Archit

Hello again!
I have questions about the use ofTAD_list generated before the use of DiffDomain.
I don't really understand your documentation on that. You wrote on the main method :

python diffdomains.py dvsd multiple <hic0> <hic1> <tadlist_of_hic0.bed> [options] 

Questions :
1- can we put a <tadlist_of_hic1.bed> with it ? or it's just for hic0 ?
2- can you give an exemple of usable <tadlist_of_hic0.bed> ?
3- if we can't used <tadlist_of_hic1.bed>, how can we compare 2 samples hic with a TAD_list for each ?

Thank you in advance for your answers.

Hi. I campare condition1 to condition2. How to know the differential TAD (strength type) is more strength in condition1 or condition2 ?