kosinskilab / pytme Goto Github PK

Hi Mark and Valentin,

I'm trying to run a match_template with a binary segmentation mask for the target (with the same shape/pixel size as the original tomogram), but I'm having the following error:

Am I missing some point during the process or do you have any other tips on how I can run match_template with these binary masks?

[hrosa@srv-mattei-01 template_matching_pytme]$ match_template.py -m L062_ts_016_15.44Apx.mrc --target_mask L062_ts_016_15.44Apx_Seg_mito-ribo-no-dilation_flipped.mrc --bandpass_band 0,0.25 --gaussian_sigma 1 --tilt_range 60,60 --tilt_step 3 --wedge_axes 0,2 --tilt_step 3 -i MitoRibo_eEF2_job261_filtsharp.mrc -a 15 --no_edge_padding --no_fourier_padding -o L062_ts_016_mask.pickle -s MCC

•                                 pyTME v0.1.6                                 *

Resampling template to [15.44 15.44 15.44]. Consider providing a template with the same sampling rate as the target.

Target
  - Initial Shape:       (400, 512, 512)
  - Sampling Rate:       (15.44, 15.44, 15.44)
  - Final Shape:         (400, 512, 512)

Template
  - Initial Shape:       (30, 30, 30)
  - Sampling Rate:       (15.44, 15.44, 15.44)
  - Final Shape:         (52, 52, 52)

Template Mask
  - Inital Shape:        (30, 30, 30)
  - Sampling Rate:       (15.44, 15.44, 15.44)
  - Final Shape:         (52, 52, 52)

Template Matching Options
  - CPU Cores:             4
  - Run on GPU:            False [N=0]
  - Use Mixed Precision:   False
  - Assigned Memory [MB]:  191143.0 [out of 224874.0]
  - Temporary Directory:   /g/icem/hrosa/SP-RIBO/template_matching_pytme
  - Extend Fourier Grid:   False
  - Extend Target Edges:   False
  - Interpolation Order:   3
  - Score:                 MCC
  - Setup Function:        <function 'tme.matching_exhaustive.mcc_setup'>
  - Scoring Function:      <function 'tme.matching_exhaustive.mcc_scoring'>
  - Angular Sampling:      15.0 [1992 rotations]
  - Scramble Template:     False
  - Target Splits:         0:1, 1:1, 2:1 [N=1]

Score Analysis Options
  - Analyzer:          <class 'tme.analyzer.MaxScoreOverRotations'>
  - score_threshold:   0.0
  - number_of_peaks:   1000
  - convolution_mode:  valid
  - use_memmap:        False

Distributing 1 split on 1 job each using 4 cores.
Running Template Matching. This might take a while ...
Traceback (most recent call last):
  File "/g/easybuild/x86_64/Rocky/8/rome/software/Python-bundle-PyPI/2023.06-GCCcore-12.3.0/lib/python3.11/site-packages/joblib/parallel.py", line 862, in dispatch_one_batch
    tasks = self._ready_batches.get(block=False)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/queue.py", line 168, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/g/easybuild/x86_64/Rocky/8/rome/software/pyTME/0.1.6-foss-2023a-CUDA-12.1.1/bin/match_template.py", line 773, in
    main()
  File "/g/easybuild/x86_64/Rocky/8/rome/software/pyTME/0.1.6-foss-2023a-CUDA-12.1.1/bin/match_template.py", line 737, in main
    candidates = scan_subsets(
                 ^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/pyTME/0.1.6-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/tme/matching_exhaustive.py", line 1474, in scan_subsets
    results = Parallel(n_jobs=outer_jobs)(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/Python-bundle-PyPI/2023.06-GCCcore-12.3.0/lib/python3.11/site-packages/joblib/parallel.py", line 1085, in call
    if self.dispatch_one_batch(iterator):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/Python-bundle-PyPI/2023.06-GCCcore-12.3.0/lib/python3.11/site-packages/joblib/parallel.py", line 873, in dispatch_one_batch
    islice = list(itertools.islice(iterator, big_batch_size))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/pyTME/0.1.6-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/tme/matching_exhaustive.py", line 1478, in
    matching_data=matching_data.subset_by_slice(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/pyTME/0.1.6-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/tme/matching_data.py", line 223, in subset_by_slice
    ret.target_mask = self.subset_array(
                      ^^^^^^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/pyTME/0.1.6-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/tme/matching_data.py", line 133, in subset_array
    arr = Density.from_file(arr.data.filename, subset=arr_slice).data
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/easybuild/x86_64/Rocky/8/rome/software/pyTME/0.1.6-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/tme/density.py", line 200, in from_file
    if filename.endswith(".em") or filename.endswith(".em.gz"):
       ^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'endswith'

Dear developers,

Hi my name is Dimitrios Bellos and I am member of the AI & I theme in the Rosalind Franklin Institute, UK. We are interested in using your software for particle picking. We have start using it and reading through the documentation (https://kosinskilab.github.io/pyTME/quickstart/match_template.html), however we would some help interpreting the output of the match_template.py command-line tool (output.pickle) . Can you please help us or gives an example of how to take it apart e.g. which part the coordinates of the candidates, scores for each candidate, etc. ?

Thank you in advance for your time reading and responding to us.

Kind regards,
Dimitrios Bellos

Dear Marc and Valentin,

I'm installing pyTME (on the EMBL cluster), and followed your installation guide. I have issues with running template matching. I get a 'permission error' when running the template matching. (Error at the end of the issue).

I tried to install it by following your installation instruction precisely but to make sure I did everything correctly I reinstalled the environment as below:

First I create the conda environment:

module load Miniconda3
conda create --name pytme2 -c conda-forge python=3.11 pyfftw napari magicgui pyqt

Afterwards I install pytme:

source activate pytme2
module load GCC
~/.conda/envs/pytme2/bin/python -m pip install git+https://github.com/KosinskiLab/pyTME.git

I then install napari:

~/.conda/envs/pytme2/bin/python -m pip install napari magicgui pyqt5
~/.conda/envs/pytme2/bin/python -m pip install git+https://github.com/maurerv/napari-density-io.git

And finally installed CuPy:
~/.conda/envs/pytme2/bin/python -m pip install cupy-cuda12x

I run template matching:

module load CUDA/12.2.0
match_template.py -m 075.gaussian.i.mrc -i ref.mrc -n 1 -a 60 --use_gpu

ERROR/LOG

•                             pyTME v0.1.3                                 *
  

Target

• Inital Shape: (450, 1440, 1022)
• Sampling Rate: (6.52, 6.52, 6.52)
• Final Shape: (450, 1440, 1022)

Template

• Inital Shape: (67, 67, 67)
• Sampling Rate: (6.52, 6.52, 6.52)
• Final Shape: (71, 71, 71)

Template Mask

• Inital Shape: (67, 67, 67)
• Sampling Rate: (6.52, 6.52, 6.52)
• Final Shape: (71, 71, 71)

Template Matching Options

• CPU Cores: 1
• Run on GPU: True [N=1]
• Use Mixed Precision: False
• Assigned Memory [MB]: 12080.0 [out of 14212.0]
• Temporary Directory: /g/scb/mahamid/rasmus/pytme/pyTME/input_data
• Extend Fourier Grid: True
• Extend Target Edges: True
• Interpolation Order: 3
• Score: CC
• Setup Function: <function 'tme.matching_exhaustive.cc_setup'>
• Scoring Function: <function 'tme.matching_exhaustive.corr_scoring'>
• Angular Sampling: 60.0 [24 rotations]
• Scramble Template: False
• Target Splits: 0:2, 1:2, 2:2 [N=8]

Score Analysis Options

• Analyzer: <class 'tme.analyzer.MaxScoreOverRotations'>
• score_threshold: 0.0
• number_of_peaks: 1000
• convolution_mode: valid
• use_memmap: False

Distributing 8 splits on 1 job each using 1 core.
Running Template Matching. This might take a while ...
Process SharedMemoryManager-1:
Traceback (most recent call last):
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/managers.py", line 592, in _run_server
server = cls._Server(registry, address, authkey, serializer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/managers.py", line 1280, in init
Server.init(self, *args, **kwargs)
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/managers.py", line 156, in init
self.listener = Listener(address=address, backlog=16)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/connection.py", line 464, in init
self._listener = SocketListener(address, family, backlog)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/connection.py", line 607, in init
self._socket.bind(address)
PermissionError: [Errno 1] Operation not permitted

File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/site-packages/tme/matching_exhaustive.py", line 1192, in inner_function
with SharedMemoryManager() as smh:
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/managers.py", line 645, in enter
self.start()
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/managers.py", line 567, in start
self._address = reader.recv()
^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/multiprocessing/connection.py", line 399, in _recv
raise EOFError
Traceback (most recent call last):
File "/home/kjeldsen/.conda/envs/pytme2/bin/match_template.py", line 733, in
main()
File "/home/kjeldsen/.conda/envs/pytme2/bin/match_template.py", line 699, in main
candidates = scan_subsets(
^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/site-packages/tme/matching_exhaustive.py", line 1473, in scan_subsets
results = Parallel(n_jobs=outer_jobs)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/site-packages/joblib/parallel.py", line 1863, in call
return output if self.return_generator else list(output)
^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/site-packages/joblib/parallel.py", line 1792, in _get_sequential_output
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/site-packages/tme/matching_exhaustive.py", line 47, in _run_inner
return scan(**kwargs)
^^^^^^^^^^^^^^
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/site-packages/tme/matching_exhaustive.py", line 1199, in inner_function
handle_traceback(last_type, last_value, last_traceback)
File "/home/kjeldsen/.conda/envs/pytme2/lib/python3.11/site-packages/tme/matching_utils.py", line 48, in handle_traceback
raise Exception(last_value)
Exception

Unrelated:
You should put a link to the preprint on the github frontpage :-)

Hi,
I was wondering if it would be better to select extraction points as a function of hi-low CC threshold rather than picking n points. One of the outputs of the matching program could be the mean CC, score and hi, low values and the user can then decide where to set the thresholds during the extraction step.
Best,
Pranav