mli-lab / deepdewedge Goto Github PK

Hello,

I've been enjoying trying out DeepDeWedge to denoise/correct some of our tomograms. However, on our data I noticed that the losses (both training and validation) were very high compared to the tutorial data, and that after refining the tomograms it produces a weird, grid-like artifact. I thought it may be due to something wrong with input dimensions, etc and reassembling of the refined subtomograms into the full tomogram, however when I extracted subtomograms and reassembled them from an unrefined tomogram (using ddw functions), everything looked fine. This made me curious if the high loss of the model is just resulting in poor refinement of subtomogram edges?

I've attached screenshots below - the first is a screenshot of the training loss curves from fitting to the entire tomogram, the second is the training loss curves from fitting to a sub-region of the tomogram that contains mostly the cell of interest and excludes empty regions, and the third is a screenshot of the refined tomogram. As can be seen, removing the empty areas from the tomogram improves the trends of the training loss curves (I could have ran it for much longer), however the losses are still very high and it didn't seem to improve the refinement very much.

Training curve from the full tomogram

Training curve for the tomogram region with most empty ice exluded

Snapshot of the refined tomogram

I also noticed that the tutorial used bin 6 data, whereas this data is bin 4 (pixel size ~1.06 nm). I'm rerunning it now with bin 6 data (pixel size ~1.6 nm) and the loss looks better, but it is still hovering above 900 so I'm not sure how the results will look. Let me know if you have any advice about this issue!

Best,
Matt

Hi,

I just wanted to share that following the instructions provided for installation did not work for me. Here is how I got to install DeepDeWedge:

First, the required Python version was not specified anywhere. After some trial and error (regarding the pytorch dependency and such) I found that python=3.10 works on my system (CentOS 7, NVIDIA driver 545.23.08, CUDA 12.3)

git clone https://github.com/MLI-lab/DeepDeWedge
cd DeepDeWedge
mamba create -n DeepDeWedge_env python=3.10

Then I had to edit the requirements.txt to be as follows:

torch==1.12.1
mrcfile
matplotlib
pandas
pytorch-lightning==1.8.0.post1
scikit-image

and then

pip install -r requirements.txt`

finally worked.

Hope this helps other people out there!

Thank you,
Ricardo

Hi,
it is a bit of a pain to erase all folders in subtomo-dir every time the run crashes and I want to re-run the fixed command. I am sure this is an easy fix.

When trying to resume ddw fit-model from a previous checkpoint, it seems to ignore it and start fitting from scratch again. I am adding the resume-from-checkpoint line to my config.yaml file:

fit_model:
    unet_params_dict:
      chans: 32
      num_downsample_layers: 3
      drop_prob: 0.1
    adam_params_dict: 
      lr: 0.0004
    num_epochs: 600
    batch_size: 10
    update_subtomo_missing_wedges_every_n_epochs: 10
    check_val_every_n_epochs: 10
    save_n_models_with_lowest_val_loss: 5
    save_n_models_with_lowest_fitting_loss: 5
    save_model_every_n_epochs: 50
    logger: "csv"
    resume-from-checkpoint: "ddw_imod/logs/version_0/checkpoints/epoch/epoch=399.ckpt"

I know it's starting from scratch because the fitting loss at this epoch was 0.14867860078811646 in the previous run (which crashed), but upon starting this new run it says the fitting loss is 0.1634301096200943 in the first epoch. Does that make sense?

Thank you!

Hi Simon,

I am consistently observing the validation loss almost always being lower than the fitting loss, which is quite puzzling. Isn't the validation loss expected to be higher than the fitting loss, on average?
My guess is some issue with the normalization of the validation subtomograms, not sure. The curve can probably still be used to identify overfitting, regardless of the scale, I think.

Here are 4 examples from different cases (2 sets of tomograms reconstructed in 2 different ways each):

Thank you!

I installed the software yesterday (after implemented overwrite in prepare_data comment) and i got the following error in the tutorial run (provided config with updated : model_checkpoint_file: "./tutorial_project/logs/version_3/checkpoints/epoch/epoch=999.ckpt")

ddw refine-tomogram --config ./config.yaml

ddw_refine.log

          Running the second command returns this error. I am not clear on what is happening there.

Command: ddw fit-model --unet-params-dict chans 32 drop_prob 0.0 num_downsample_layers --adam-params-dict lr 0.0004 --num-epochs 100 --batch-size 5 --subtomo-size 64 --mw-angle 60 --gpu 0 --num-workers 3

Error:
Traceback (most recent call last) ────────────────────────────────╮
│ /cemcofsw/em/deepDeWedge/240507/bin/ddw:8 in │
│ │
│ 5 from ddw.app import main │
│ 6 if name == 'main': │
│ 7 │ sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ ╭───────────────────────────────────── locals ─────────────────────────────────────╮ │
│ │ main = <function main at 0x7f4db165f0a0> │ │
│ │ re = <module 're' from '/cemcofsw/em/deepDeWedge/240507/lib/python3.10/re.py'> │ │
│ │ sys = <module 'sys' (built-in)> │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/ddw/app.py:14 in main │
│ │
│ 11 │
│ 12 │
│ 13 def main(): │
│ ❱ 14 │ app() │
│ 15 │
│ 16 │
│ 17 if name == "main": │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/typer/main.py:326 in call │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/typer/main.py:309 in call │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/click/core.py:1157 in call │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/typer/core.py:723 in main │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/typer/core.py:193 in _main │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/click/core.py:1686 in invoke │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/click/core.py:943 in make_context │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/click/core.py:1408 in parse_args │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/click/core.py:2400 in │
│ handle_parse_result │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/click/core.py:2362 in process_value │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/site-packages/typer/main.py:1001 in wrapper │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/ast.py:110 in literal_eval │
│ │
│ 107 │ │ │ │ else: │
│ 108 │ │ │ │ │ return left - right │
│ 109 │ │ return _convert_signed_num(node) │
│ ❱ 110 │ return _convert(node_or_string) │
│ 111 │
│ 112 │
│ 113 def dump(node, annotate_fields=True, include_attributes=False, *, indent=None): │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _convert = <function literal_eval.._convert at 0x7f4c4099a680> │ │
│ │ _convert_num = <function literal_eval.._convert_num at 0x7f4c4099a560> │ │
│ │ _convert_signed_num = <function literal_eval.._convert_signed_num at │ │
│ │ 0x7f4c4099a5f0> │ │
│ │ _raise_malformed_node = <function literal_eval.._raise_malformed_node at │ │
│ │ 0x7f4c4099a440> │ │
│ │ node_or_string = <ast.Name object at 0x7f4c409b4850> │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/ast.py:109 in _convert │
│ │
│ 106 │ │ │ │ │ return left + right │
│ 107 │ │ │ │ else: │
│ 108 │ │ │ │ │ return left - right │
│ ❱ 109 │ │ return _convert_signed_num(node) │
│ 110 │ return _convert(node_or_string) │
│ 111 │
│ 112 │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _convert = <function literal_eval.._convert at 0x7f4c4099a680> │ │
│ │ _convert_num = <function literal_eval.._convert_num at 0x7f4c4099a560> │ │
│ │ _convert_signed_num = <function literal_eval.._convert_signed_num at │ │
│ │ 0x7f4c4099a5f0> │ │
│ │ _raise_malformed_node = <function literal_eval.._raise_malformed_node at │ │
│ │ 0x7f4c4099a440> │ │
│ │ node = <ast.Name object at 0x7f4c409b4850> │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/ast.py:83 in _convert_signed_num │
│ │
│ 80 │ │ │ │ return + operand │
│ 81 │ │ │ else: │
│ 82 │ │ │ │ return - operand │
│ ❱ 83 │ │ return _convert_num(node) │
│ 84 │ def _convert(node): │
│ 85 │ │ if isinstance(node, Constant): │
│ 86 │ │ │ return node.value │
│ │
│ ╭──────────────────────────────────── locals ────────────────────────────────────╮ │
│ │ _convert_num = <function literal_eval.._convert_num at 0x7f4c4099a560> │ │
│ │ node = <ast.Name object at 0x7f4c409b4850> │ │
│ ╰────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/ast.py:74 in _convert_num │
│ │
│ 71 │ │ raise ValueError(msg + f': {node!r}') │
│ 72 │ def _convert_num(node): │
│ 73 │ │ if not isinstance(node, Constant) or type(node.value) not in (int, float, comple │
│ ❱ 74 │ │ │ _raise_malformed_node(node) │
│ 75 │ │ return node.value │
│ 76 │ def _convert_signed_num(node): │
│ 77 │ │ if isinstance(node, UnaryOp) and isinstance(node.op, (UAdd, USub)): │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _raise_malformed_node = <function literal_eval.._raise_malformed_node at │ │
│ │ 0x7f4c4099a440> │ │
│ │ node = <ast.Name object at 0x7f4c409b4850> │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /cemcofsw/em/deepDeWedge/240507/lib/python3.10/ast.py:71 in _raise_malformed_node │
│ │
│ 68 │ │ msg = "malformed node or string" │
│ 69 │ │ if lno := getattr(node, 'lineno', None): │
│ 70 │ │ │ msg += f' on line {lno}' │
│ ❱ 71 │ │ raise ValueError(msg + f': {node!r}') │
│ 72 │ def _convert_num(node): │
│ 73 │ │ if not isinstance(node, Constant) or type(node.value) not in (int, float, comple │
│ 74 │ │ │ _raise_malformed_node(node) │
│ │
│ ╭────────────────── locals ───────────────────╮ │
│ │ lno = 1 │ │
│ │ msg = 'malformed node or string on line 1' │ │
│ │ node = <ast.Name object at 0x7f4c409b4850> │ │
│ ╰─────────────────────────────────────────────╯

Originally posted by @Mishaczech in #3 (comment)