caiyuanhao1998 / sax-nerf Goto Github PK

Is it possible to use this framework on 2D RGB X-ray images instead of CT scans?

Here are my data specifications and I want to know if it is sensible and possible to train it using your designs:

1. Collection of 2D rgb xray images
2. Lacks camera parameters
3. Dynamic in nature

Thanks for sharing your wonderful project !!
I was really looking forward to reading the scripts.

I would like to ask about the hardware requirements for training a model.
In your paper, you mentioned that all experiments were performed on RTX8000 (VRAM: 48GB).
How much VRAM does it take, for example, python train_mlg.py --config config/Lineformer/chest_50.yaml ?
With V100 (VRAM: 16GB), I got an OOM error.

I would appreciate it if you could let me know.
Thank you in advance.

Hello! The img.mat file was missing when running generatedata_backpack.py. Could you upload this data, please? FileNotFoundError: [Errno 2] No such file or directory: './dataGenerator/raw_data/backpack/img.mat' In addition, if I want to use a nii.gz format CT file to generate a training set, what should I do? I found in the generatedata.py code that the input CT needs a dictionary, which contains a lot of information.

Thanks for sharing your awesome project!
I try to run python3 eval_traditional.py --algorithm fdk --category chest --config config/FDK/chest_50.yaml .
But retrun some error
Traceback (most recent call last): File "/workspace/SAX-NeRF/eval_traditional.py", line 9, in <module> from src.evaluator import Evaluator File "/workspace/SAX-NeRF/src/evaluator.py", line 5, in <module> from .dataset import TIGREDataset_Traditional as Dataset ImportError: cannot import name 'TIGREDataset_Traditional' from 'src.dataset' (/workspace/SAX-NeRF/src/dataset/__init__.py)
It seems like there is no function called TIGREDataset_Traditional。

So Nice to share the code.
I tried to follow the author's instructions to install CUDA = v11.3, and then followed a series of environment installations. But I encountered an error:

“
from .backend import _backend
File "/ai4hh/ai4h/mawen/Project/SAX-NeRF-master/src/encoder/hashencoder/backend.py", line 18, in
_backend = load(name='_hash_encoder',
File "/home/wenma/anaconda3/envs/sax_nerf/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1144, in load return _jit_compile(
File "/home/wenma/anaconda3/envs/sax_nerf/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1357, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/wenma/anaconda3/envs/sax_nerf/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1469, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/wenma/anaconda3/envs/sax_nerf/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1756, in _run_ninja_build
raise RuntimeError(message) from e

RuntimeError: Error building extension '_hash_encoder': [1/3] /usr/bin/nvcc
nvcc fatal : Unsupported gpu architecture 'compute_86'
ninja: build stopped: subcommand failed.
”

And
ninja --version = 1.11.1.git.kitware.jobserver-1
gcc & g++ version = 9.4.0
nvcc: 11.3

Who can save this child...

I appreciate the excellence of your research.
Upon reading the paper, I was wondering when I could download the X3D dataset used in the paper.
Or could you provide information on where the dataset can be downloaded?

Thank you for good research.

Hello.
I am encountering an error while testing with test.py. It seems to be a compilation-related message. How should I resolve it?

Dear author,
If my raw data are in nii format, how could I convert them to the suitable form for SAX-NeRF, or what parameters should I stay the same?

Hello!
I appreciate your research.
Your research has led to some interesting collaborations.

The data we use is too big, so we are trying to reshape it to 256256256 size like the sample you provided.
However, if we set it to 204820482160, an error occurs.
What does the z-axis mean here? What should go in place of 2160?

In addition, are there any parameters to keep in mind when using a custom dataset?
Below is our Dicom data.

Scan
Scan mode: CT
2048 Number of Detector Elements
2160 CT: Number of Projections
19 DR: Number of Lines
1 Linebinning Settings
1 Pixelbinning Settings
Measurement extension: none
1 Multi-Scan: Number of Slices
184.20 Multi Slice: Distance between two Slices [mm]
2.14 Magnification (Object Center)
0.09366268 2D-Pixel Size [mm]
0.09366286 3D-XY-Pixel Size [mm]
0.09514938 3D-Z-Pixel Size [mm]
1 CT: Number of Slices

Detector
Detector Type: Y.XRD1621
409.60 Detectorlength

Reconstruction
2048 Number of actual Detector Pixel
2048 Image Dimension
2016 Number of Z-slices
2160 Number of Projections
0.00 Rotational Offset in degree
Dose Rate Correction: on
0.00000000 Center Offset [Pixel]
0.00000000 Center Offset [mm]
1.00 Convolution Coefficient
409.60 ROI [mm]

Axis
257.12 X-Axis: Measurement Position [mm]
750.00 Y-Axis: Measurement Position [mm]
1225.60 YD-Axis: Measurement Position [mm]
281.00 ZS-Axis: Measurement Position [mm]
1225.60 Focus-Detector distance [mm]
651.64 Object-Detector distance [mm]
573.97 Focus-Object distance [mm]
500.00 Integration Time [ms] for CT and DR

X-Ray
X-Ray-Tube: MF X-Ray-Tube
Tube head: D-Target FXE 225.99 (48)
185.00 Voltage [kV]
0.49 Current [mA]
Focus: small
TXI: on
SecondTubeHead: off

Filter
Al 0.50 [mm]
Cu 0.00 [mm]
Sn 0.00 [mm]
Fe 0.00 [mm]

Remarks

Hi Authors,

Thank you for deploying the code. It's really helpful to have all architectures in one place. I have a question about how your models would perform on dynamic data. Is it possible to incorporate a temporal aspect into the existing architecture, considering that my data modality changes with time?

Nice Work!

I have run your code on the Chest-50 data. I find that your proposed LineFormer requires much more training time than other methods. Specifically, your LineFormer requires about 6 hours for 1500 epochs while Naf only requires about 15 minutes.

Could you please provide more details about why LineFormer requires a much longer training process? Is it because LineFormer has more parameters or is it related to MLG sampling strategy?

Hello. Thank you for your great job!
I have a question for your code.

I try to convert a 2D x-ray image (only one view) to 3D images.
I just want to know if your code has this function or not.
If it has, then,
do the training data also only require only one view 2D x-ray image?

A great work on CT reconstruction.
I wanna ask a question. The strategies you developed in your paper: lineformer and MLG, are they suitable for natural scenes, i.e. RGB NeRF? It seems that they are not specific for X-rays.

Hi, thanks for sharing this awesome project!

I have already generated .mat file.
However, when I try to run 'python dataGenerator/generateData_backpack.py', I got the following errors as shown below. I followed the steps for downloading and installing TIGRE exactly without any errors during those steps. Is there any effective way to solve this problem? Thanks a lot!

Great work! Are there any plans to release the code?

Hello,

Apparently, SAX-NeRF code requires cuda 11.3, which seems suprising as it looks to be built on naf hash encoder that I could run on cuda 12.2. In colab, the default version is 12.2, and it is not convenient to modify that. Do you know if there is any solution to run your code in colab?

Here is what I obtain when running the code:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
Traceback (most recent call last):
File "/content/drive/MyDrive/SAX-NeRF/train_mlg.py", line 23, in
from src.trainer_mlg import Trainer
File "/content/drive/MyDrive/SAX-NeRF/src/trainer_mlg.py", line 17, in
from .encoder import get_encoder
File "/content/drive/MyDrive/SAX-NeRF/src/encoder/init.py", line 1, in
from .hashencoder import HashEncoder
File "/content/drive/MyDrive/SAX-NeRF/src/encoder/hashencoder/init.py", line 1, in
from .hashgrid import HashEncoder
File "/content/drive/MyDrive/SAX-NeRF/src/encoder/hashencoder/hashgrid.py", line 8, in
from .backend import _backend
File "/content/drive/MyDrive/SAX-NeRF/src/encoder/hashencoder/backend.py", line 6, in
_backend = load(name='_hash_encoder',
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1309, in load
return _jit_compile(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1719, in _jit_compile
_write_ninja_file_and_build_library(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1819, in _write_ninja_file_and_build_library
_write_ninja_file_to_build_library(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 2206, in _write_ninja_file_to_build_library
cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1987, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range

Thank you for your time.
Quentin

python : 3.9
cuda : 11.3
pytorch : 1.10
torch.cuda.is_available() : true

I an trying to run testing about part of sax-nerf which does not work, could you have a look please? thanks

Traceback (most recent call last):
  File "C:\3dgs\SAX-NeRF\test.py", line 29, in <module>
    from src.encoder import get_encoder
  File "C:\3dgs\SAX-NeRF\src\encoder\__init__.py", line 1, in <module>
    from .hashencoder import HashEncoder
  File "C:\3dgs\SAX-NeRF\src\encoder\hashencoder\__init__.py", line 1, in <module>
    from .hashgrid import HashEncoder
  File "C:\3dgs\SAX-NeRF\src\encoder\hashencoder\hashgrid.py", line 8, in <module>
    from .backend import _backend
  File "C:\3dgs\SAX-NeRF\src\encoder\hashencoder\backend.py", line 6, in <module>
    _backend = load(name='_hash_encoder',
  File "C:\Anaconda\envs\test\lib\site-packages\torch\utils\cpp_extension.py", line 1144, in load
    return _jit_compile(
  File "C:\Anaconda\envs\test\lib\site-packages\torch\utils\cpp_extension.py", line 1357, in _jit_compile
    _write_ninja_file_and_build_library(
  File "C:\Anaconda\envs\test\lib\site-packages\torch\utils\cpp_extension.py", line 1456, in _write_ninja_file_and_build_library
    _write_ninja_file_to_build_library(
  File "C:\Anaconda\envs\test\lib\site-packages\torch\utils\cpp_extension.py", line 1857, in _write_ninja_file_to_build_library
    cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
  File "C:\Anaconda\envs\test\lib\site-packages\torch\utils\cpp_extension.py", line 1626, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range```

when using imageio to save image ,P.S. in this sentence specificlly iio.imwrite(osp.join(proj_pred_origin_dir, f"proj_pred_{str(i)}.png"), (cast_to_image(projs_pred[i])*255).astype(np.uint8))
there comes the problem of Can't write images with one color channel. How can this be solved?like using cv instead?

In all the config files of the experiments, you set the line size as 2 for the Lineformer network, which means only the self-attention between 2 points is considered. I also try line size 4 and 8, but actually 2 is best. However, it is counter-intuitive and strange that self-attention between 2 points is good enough for reconstruction. Generally, I think more points considered will achieve better results. Can you explain why you choose 2 as the line size?