drprojects / superpoint_transformer Goto Github PK

Official PyTorch implementation of Superpoint Transformer introduced in [ICCV'23] "Efficient 3D Semantic Segmentation with Superpoint Transformer" and SuperCluster introduced in [3DV'24 Oral] "Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering"

License: MIT License

Shell 0.37% Jupyter Notebook 3.29% Python 96.34%

3d deep-learning efficient fast hierarchical lightweight partition point-cloud pytorch semantic-segmentation

superpoint_transformer's Introduction

Superpoint Transformer

Official implementation for

Efficient 3D Semantic Segmentation with Superpoint Transformer (ICCV 2023)

Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering (3DV 2024 Oral)

If you ❤️ or use this project, don't forget to give it a ⭐, it means a lot to us !

@article{robert2023spt,
  title={Efficient 3D Semantic Segmentation with Superpoint Transformer},
  author={Robert, Damien and Raguet, Hugo and Landrieu, Loic},
  journal={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2023}
}

@article{robert2024scalable,
  title={Scalable 3D Panoptic Segmentation as Superpoint Graph Clustering},
  author={Robert, Damien and Raguet, Hugo and Landrieu, Loic},
  journal={Proceedings of the IEEE International Conference on 3D Vision},
  year={2024}
}

📌 Description

Superpoint Transformer

Superpoint Transformer (SPT) is a superpoint-based transformer 🤖 architecture that efficiently ⚡ performs semantic segmentation on large-scale 3D scenes. This method includes a fast algorithm that partitions 🧩 point clouds into a hierarchical superpoint structure, as well as a self-attention mechanism to exploit the relationships between superpoints at multiple scales.

✨ SPT in numbers ✨
📊 SOTA on S3DIS 6-Fold (76.0 mIoU)
📊 SOTA on KITTI-360 Val (63.5 mIoU)
📊 Near SOTA on DALES (79.6 mIoU)
🦋 212k parameters (PointNeXt ÷ 200, Stratified Transformer ÷ 40)
⚡ S3DIS training in 3h on 1 GPU (PointNeXt ÷ 7, Stratified Transformer ÷ 70)
⚡ Preprocessing x7 faster than SPG

SuperCluster

SuperCluster is a superpoint-based architecture for panoptic segmentation of (very) large 3D scenes 🐘 based on SPT. We formulate the panoptic segmentation task as a scalable superpoint graph clustering task. To this end, our model is trained to predict the input parameters of a graph optimization problem whose solution is a panoptic segmentation 💡. This formulation allows supervising our model with per-node and per-edge objectives only, circumventing the need for computing an actual panoptic segmentation and associated matching issues at train time. At inference time, our fast parallelized algorithm solves the small graph optimization problem, yielding object instances 👥. Due to its lightweight backbone and scalable formulation, SuperCluster can process scenes of unprecedented scale at once, on a single GPU 🚀, with fewer than 1M parameters 🦋.

✨ SuperCluster in numbers ✨
📊 SOTA on S3DIS 6-Fold (55.9 PQ)
📊 SOTA on S3DIS Area 5 (50.1 PQ)
📊 SOTA on ScanNet Val (58.7 PQ)
📊 FIRST on KITTI-360 Val (48.3 PQ)
📊 FIRST on DALES (61.2 PQ)
🦋 212k parameters (PointGroup ÷ 37)
⚡ S3DIS training in 4h on 1 GPU
⚡ 7.8km² tile of 18M points in 10.1s on 1 GPU

📰 Updates

28.02.2024 Major code release for panoptic segmentation, implementing Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering. This new version also implements long-awaited features such as lightning's predict() behavior, voxel-resolution and full-resolution prediction. Some changes in the dependencies and repository structure are not backward-compatible. If you were already using anterior code versions, this means we recommend re-installing your conda environment and re-running the preprocessing or your datasets❗
15.10.2023 Our paper Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering was accepted for an oral presentation at 3DV 2024 🥳
06.10.2023 Come see our poster for Efficient 3D Semantic Segmentation with Superpoint Transformer at ICCV 2023
14.07.2023 Our paper Efficient 3D Semantic Segmentation with Superpoint Transformer was accepted at ICCV 2023 🥳
15.06.2023 Official release 🌱

💻 Environment requirements

This project was tested with:

Linux OS
64G RAM
NVIDIA GTX 1080 Ti 11G, NVIDIA V100 32G, NVIDIA A40 48G
CUDA 11.8 and 12.1
conda 23.3.1

🏗 Installation

Simply run install.sh to install all dependencies in a new conda environment named spt.

# Creates a conda env named 'spt' env and installs dependencies
./install.sh

Note: See the Datasets page for setting up your dataset path and file structure.

🔩 Project structure

└── superpoint_transformer
    │
    ├── configs                   # Hydra configs
    │   ├── callbacks                 # Callbacks configs
    │   ├── data                      # Data configs
    │   ├── debug                     # Debugging configs
    │   ├── experiment                # Experiment configs
    │   ├── extras                    # Extra utilities configs
    │   ├── hparams_search            # Hyperparameter search configs
    │   ├── hydra                     # Hydra configs
    │   ├── local                     # Local configs
    │   ├── logger                    # Logger configs
    │   ├── model                     # Model configs
    │   ├── paths                     # Project paths configs
    │   ├── trainer                   # Trainer configs
    │   │
    │   ├── eval.yaml                 # Main config for evaluation
    │   └── train.yaml                # Main config for training
    │
    ├── data                      # Project data (see docs/datasets.md)
    │
    ├── docs                      # Documentation
    │
    ├── logs                      # Logs generated by hydra and lightning loggers
    │
    ├── media                     # Media illustrating the project
    │
    ├── notebooks                 # Jupyter notebooks
    │
    ├── scripts                   # Shell scripts
    │
    ├── src                       # Source code
    │   ├── data                      # Data structure for hierarchical partitions
    │   ├── datamodules               # Lightning DataModules
    │   ├── datasets                  # Datasets
    │   ├── dependencies              # Compiled dependencies
    │   ├── loader                    # DataLoader
    │   ├── loss                      # Loss
    │   ├── metrics                   # Metrics
    │   ├── models                    # Model architecture
    │   ├── nn                        # Model building blocks
    │   ├── optim                     # Optimization 
    │   ├── transforms                # Functions for transforms, pre-transforms, etc
    │   ├── utils                     # Utilities
    │   ├── visualization             # Interactive visualization tool
    │   │
    │   ├── eval.py                   # Run evaluation
    │   └── train.py                  # Run training
    │
    ├── tests                     # Tests of any kind
    │
    ├── .env.example              # Example of file for storing private environment variables
    ├── .gitignore                # List of files ignored by git
    ├── .pre-commit-config.yaml   # Configuration of pre-commit hooks for code formatting
    ├── install.sh                # Installation script
    ├── LICENSE                   # Project license
    └── README.md

Note: See the Datasets page for further details on `data/``.

Note: See the Logs page for further details on `logs/``.

🚀 Usage

Datasets

See the Datasets page to set up your datasets.

Evaluation

Use the following command structure for evaluating our models from a checkpoint file checkpoint.ckpt, where <task> should be semantic for using SPT and panoptic for using SuperCluster:

# Evaluate for <task> segmentation on <dataset>
python src/eval.py experiment=<task>/<dataset> ckpt_path=/path/to/your/checkpoint.ckpt

Some examples:

# Evaluate SPT on S3DIS Fold 5
python src/eval.py experiment=semantic/s3dis datamodule.fold=5 ckpt_path=/path/to/your/checkpoint.ckpt

# Evaluate SPT on KITTI-360 Val
python src/eval.py experiment=semantic/kitti360  ckpt_path=/path/to/your/checkpoint.ckpt 

# Evaluate SPT on DALES
python src/eval.py experiment=semantic/dales ckpt_path=/path/to/your/checkpoint.ckpt

# Evaluate SuperCluster on S3DIS Fold 5
python src/eval.py experiment=panoptic/s3dis datamodule.fold=5 ckpt_path=/path/to/your/checkpoint.ckpt

# Evaluate SuperCluster on S3DIS Fold 5 with {wall, floor, ceiling} as 'stuff'
python src/eval.py experiment=panoptic/s3dis_with_stuff datamodule.fold=5 ckpt_path=/path/to/your/checkpoint.ckpt

# Evaluate SuperCluster on ScanNet Val
python src/eval.py experiment=panoptic/scannet ckpt_path=/path/to/your/checkpoint.ckpt

# Evaluate SuperCluster on KITTI-360 Val
python src/eval.py experiment=panoptic/kitti360  ckpt_path=/path/to/your/checkpoint.ckpt 

# Evaluate SuperCluster on DALES
python src/eval.py experiment=panoptic/dales ckpt_path=/path/to/your/checkpoint.ckpt

Note:

The pretrained weights of the SPT and SPT-nano models for S3DIS 6-Fold, KITTI-360 Val, and DALES are available at:

The pretrained weights of the SuperCluster models for S3DIS 6-Fold, S3DIS 6-Fold with stuff, ScanNet Val,, KITTI-360 Val, and DALES are available at:

Training

Use the following command structure for train our models on a 32G-GPU, where <task> should be semantic for using SPT and panoptic for using SuperCluster:

# Train for <task> segmentation on <dataset>
python src/train.py experiment=<task>/<dataset>

Some examples:

# Train SPT on S3DIS Fold 5
python src/train.py experiment=semantic/s3dis datamodule.fold=5

# Train SPT on KITTI-360 Val
python src/train.py experiment=semantic/kitti360 

# Train SPT on DALES
python src/train.py experiment=semantic/dales

# Train SuperCluster on S3DIS Fold 5
python src/train.py experiment=panoptic/s3dis datamodule.fold=5

# Train SuperCluster on S3DIS Fold 5 with {wall, floor, ceiling} as 'stuff'
python src/train.py experiment=panoptic/s3dis_with_stuff datamodule.fold=5

# Train SuperCluster on ScanNet Val
python src/train.py experiment=panoptic/scannet

# Train SuperCluster on KITTI-360 Val
python src/train.py experiment=panoptic/kitti360 

# Train SuperCluster on DALES
python src/train.py experiment=panoptic/dales

Use the following to train on a 11G-GPU 💾 (training time and performance may vary):

# Train SPT on S3DIS Fold 5
python src/train.py experiment=semantic/s3dis_11g datamodule.fold=5

# Train SPT on KITTI-360 Val
python src/train.py experiment=semantic/kitti360_11g 

# Train SPT on DALES
python src/train.py experiment=semantic/dales_11g

# Train SuperCluster on S3DIS Fold 5
python src/train.py experiment=panoptic/s3dis_11g datamodule.fold=5

# Train SuperCluster on S3DIS Fold 5 with {wall, floor, ceiling} as 'stuff'
python src/train.py experiment=panoptic/s3dis_with_stuff_11g datamodule.fold=5

# Train SuperCluster on ScanNet Val
python src/train.py experiment=panoptic/scannet_11g

# Train SuperCluster on KITTI-360 Val
python src/train.py experiment=panoptic/kitti360_11g 

# Train SuperCluster on DALES
python src/train.py experiment=panoptic/dales_11g

Note: Encountering CUDA Out-Of-Memory errors 💀💾 ? See our dedicated troubleshooting section.

Note: Other ready-to-use configs are provided in configs/experiment/. You can easily design your own experiments by composing configs:
# Train Nano-3 for 50 epochs on DALES
python src/train.py datamodule=dales model=nano-3 trainer.max_epochs=50
See Lightning-Hydra for more information on how the config system works and all the awesome perks of the Lightning+Hydra combo.

Note: By default, your logs will automatically be uploaded to Weights and Biases, from where you can track and compare your experiments. Other loggers are available in configs/logger/. See Lightning-Hydra for more information on the logging options.

PyTorch Lightning `predict()`

Both SPT and SuperCluster inherit from LightningModule and implement predict_step(), which permits using PyTorch Lightning's Trainer.predict() mechanism.

from src.models.semantic import SemanticSegmentationModule
from src.datamodules.s3dis import S3DISDataModule
from pytorch_lightning import Trainer

# Predict behavior for semantic segmentation on S3DIS
dataloader = S3DISDataModule(...)
model = SemanticSegmentationModule(...)
trainer = Trainer(...)
batch, output = trainer.predict(model=model, dataloaders=dataloader)

This, however, still requires you to instantiate a Trainer, a DataLoader, and a model with relevant parameters.

For a little more simplicity, all our datasets inherit from LightningDataModule and implement predict_dataloader() by pointing to their corresponding test set by default. This permits directly passing a datamodule to PyTorch Lightning's Trainer.predict() without explicitly instantiating a DataLoader.

from src.models.semantic import SemanticSegmentationModule
from src.datamodules.s3dis import S3DISDataModule
from pytorch_lightning import Trainer

# Predict behavior for semantic segmentation on S3DIS
datamodule = S3DISDataModule(...)
model = SemanticSegmentationModule(...)
trainer = Trainer(...)
batch, output = trainer.predict(model=model, datamodule=datamodule)

For more details on how to instantiate these, as well as the output format of our model, we strongly encourage you to play with our demo notebook and have a look at the src/eval.py script.

Full-resolution predictions

By design, our models only need to produce predictions for the superpoints of the $P_1$ partition level during training. All our losses and metrics are formulated as superpoint-wise objectives. This conveniently saves compute and memory at training and evaluation time.

At inference time, however, we often need the predictions on the voxels of the $P_0$ partition level or on the full-resolution input point cloud. To this end, we provide helper functions to recover voxel-wise and full-resolution predictions.

See our demo notebook for more details on these.

Parameterizing SuperCluster graph clustering

One specificity of SuperCluster is that the model is not trained to explicitly do panoptic segmentation, but to predict the input parameters of a superpoint graph clustering problem whose solution is a panoptic segmentation.

For this reason, the hyperparameters for this graph optimization problem are selected after training, with a grid search on the training or validation set. We find that fairly similar hyperparameters yield the best performance on all our datasets (see our paper's appendix). Yet, you may want to explore these hyperparameters for your own dataset. To this end, see our demo notebook for parameterizing the panoptic segmentation.

Notebooks & visualization

We provide notebooks to help you get started with manipulating our core data structures, configs loading, dataset and model instantiation, inference on each dataset, and visualization.

In particular, we created an interactive visualization tool ✨ which can be used to produce shareable HTMLs. Demos of how to use this tool are provided in the notebooks. Additionally, examples of such HTML files are provided in media/visualizations.7z

📚 Documentation

README - General introduction to the project
Data - Introduction to NAG and Data, the core data structures of this project
Datasets - Introduction to Datasets and the project's data/ structure
Logging - Introduction to logging and the project's logs/ structure

Note: We endeavoured to comment our code as much as possible to make this project usable. Still, if you find some parts are unclear or some more documentation would be needed, feel free to let us know by creating an issue !

👩‍🔧 Troubleshooting

Here are some common issues and tips for tackling them.

SPT or SuperCluster on an 11G-GPU

Our default configurations are designed for a 32G-GPU. Yet, SPT and SuperCluster can run on an 11G-GPU 💾, with minor time and performance variations.

We provide configs in configs/experiment/semantic for training SPT on an 11G-GPU 💾:

# Train SPT on S3DIS Fold 5
python src/train.py experiment=semantic/s3dis_11g datamodule.fold=5

# Train SPT on KITTI-360 Val
python src/train.py experiment=semantic/kitti360_11g 

# Train SPT on DALES
python src/train.py experiment=semantic/dales_11g

Similarly, we provide configs in configs/experiment/panoptic for training SuperCluster on an 11G-GPU 💾:

# Train SuperCluster on S3DIS Fold 5
python src/train.py experiment=panoptic/s3dis_11g datamodule.fold=5

# Train SuperCluster on S3DIS Fold 5 with {wall, floor, ceiling} as 'stuff'
python src/train.py experiment=panoptic/s3dis_with_stuff_11g datamodule.fold=5

# Train SuperCluster on ScanNet Val
python src/train.py experiment=panoptic/scannet_11g

# Train SuperCluster on KITTI-360 Val
python src/train.py experiment=panoptic/kitti360_11g 

# Train SuperCluster on DALES
python src/train.py experiment=panoptic/dales_11g

CUDA Out-Of-Memory Errors

Having some CUDA OOM errors 💀💾 ? Here are some parameters you can play with to mitigate GPU memory use, based on when the error occurs.

Parameters affecting CUDA memory.

Legend: 🟡 Preprocessing | 🔴 Training | 🟣 Inference (including validation and testing during training)

Parameter	Description	When
`datamodule.xy_tiling`	Splits dataset tiles into xy_tiling^2 smaller tiles, based on a regular XY grid. Ideal square-shaped tiles à la DALES. Note this will affect the number of training steps.	🟡🟣
`datamodule.pc_tiling`	Splits dataset tiles into 2^pc_tiling smaller tiles, based on a their principal component. Ideal for varying tile shapes à la S3DIS and KITTI-360. Note this will affect the number of training steps.	🟡🟣
`datamodule.max_num_nodes`	Limits the number of $P_1$ partition nodes/superpoints in the training batches.	🔴
`datamodule.max_num_edges`	Limits the number of $P_1$ partition edges in the training batches.	🔴
`datamodule.voxel`	Increasing voxel size will reduce preprocessing, training and inference times but will reduce performance.	🟡🔴🟣
`datamodule.pcp_regularization`	Regularization for partition levels. The larger, the fewer the superpoints.	🟡🔴🟣
`datamodule.pcp_spatial_weight`	Importance of the 3D position in the partition. The smaller, the fewer the superpoints.	🟡🔴🟣
`datamodule.pcp_cutoff`	Minimum superpoint size. The larger, the fewer the superpoints.	🟡🔴🟣
`datamodule.graph_k_max`	Maximum number of adjacent nodes in the superpoint graphs. The smaller, the fewer the superedges.	🟡🔴🟣
`datamodule.graph_gap`	Maximum distance between adjacent superpoints int the superpoint graphs. The smaller, the fewer the superedges.	🟡🔴🟣
`datamodule.graph_chunk`	Reduce to avoid OOM when `RadiusHorizontalGraph` preprocesses the superpoint graph.	🟡
`datamodule.dataloader.batch_size`	Controls the number of loaded tiles. Each train batch is composed of `batch_size``datamodule.sample_graph_k` spherical samplings. Inference is performed on entire validation and test tiles*, without spherical sampling.	🔴🟣
`datamodule.sample_segment_ratio`	Randomly drops a fraction of the superpoints at each partition level.	🔴
`datamodule.sample_graph_k`	Controls the number of spherical samples in the train batches.	🔴
`datamodule.sample_graph_r`	Controls the radius of spherical samples in the train batches. Set to `sample_graph_r<=0` to use the entire tile without spherical sampling.	🔴
`datamodule.sample_point_min`	Controls the minimum number of $P_0$ points sampled per superpoint in the train batches.	🔴
`datamodule.sample_point_max`	Controls the maximum number of $P_0$ points sampled per superpoint in the train batches.	🔴
`callbacks.gradient_accumulator.scheduling`	Gradient accumulation. Can be used to train with smaller batches, with more training steps.	🔴

💳 Credits

This project was built using Lightning-Hydra template.
The main data structures of this work rely on PyToch Geometric
Some point cloud operations were inspired from the Torch-Points3D framework, although not merged with the official project at this point.
For the KITTI-360 dataset, some code from the official KITTI-360 was used.
Some superpoint-graph-related operations were inspired from Superpoint Graph
The hierarchical superpoint partition and graph clustering are computed using Parallel Cut-Pursuit

Citing our work

If your work uses all or part of the present code, please include the following a citation:

@article{robert2023spt,
  title={Efficient 3D Semantic Segmentation with Superpoint Transformer},
  author={Robert, Damien and Raguet, Hugo and Landrieu, Loic},
  journal={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2023}
}

@article{robert2024scalable,
  title={Scalable 3D Panoptic Segmentation as Superpoint Graph Clustering},
  author={Robert, Damien and Raguet, Hugo and Landrieu, Loic},
  journal={Proceedings of the IEEE International Conference on 3D Vision},
  year={2024}
}

You can find our SPT paper 📄 and SuperCluster paper 📄 on arxiv.

Also, if you ❤️ or use this project, don't forget to give it a ⭐, it means a lot to us !

superpoint_transformer's People

Contributors

Stargazers

Watchers

Forkers

d-roberts jk4011 umrlastig whuhxb ziwenjie aamsoftware swoonge peoplelu ceibasheep liuxinren456852 anhquancao perception4d hyunkoome jackyccl kg571852741 defe41251135 tiantian205 lyp-deeplearning rjanvier jinghere11 lihu577 hiyyg sguldbaek yanlingz charlesgaydon whu-lyh cell1604 owen718 josephorng simonbuusjensen mbendjilali doublepoints ustbzgn s-na oogetyboogety seekerzero lixiao-bai vlehtola cchampeyroux qingxia1994 changhaonan benjamingejian scw0819 eltociear sorokinvld depth-estimation 1a7r0ch3 pc2005 cainiaoshidai enginbozkurt zhangzw12319 kms-team qb-1314 igasparini contactrichrobotics chengwei920412

superpoint_transformer's Issues

how to extracting the features?

i am not getting feature extraction code.

[Environment Configuration] SPT and SPG Environment Compatibility

Hi, thanks for your excellent work in point clustering. I am interested in this field and have been working on reproducing your previous work, the SuperPoint Transformer, alongside your previous excellent project, the SuperPoint Graph.

From my understanding, SPT includes point partitioning, which is similar to that found in SPG. However, I encountered some difficulties while attempting to install the required environment for SPG, while the installation process for SPT was much smoother and successful. Therefore, I am curious to know if the SPT environment also supports the SPG project, allowing me to conduct SPG under the SPT environment.

Custom Dataset Index Error

Hello, I have a custom dataset of .las files that I've converted to .ply with x, y, z, intensity, and sem_class attributes. The dataset is similar to DALES3D but has lower average points per meter and covers a much larger area. The ground points are non-planar, so I've followed the suggestion in #32 to remove 'elevation' from partition_hf and point_hf in my config (I also removed the GroundElevation pre_transform).

When running src/train.py on my dataset/config I get an Index Error (see stack trace below) inside the RadiusHorizontalGraph transform. This error is directly related to the edge_index variable in src/utils/neighbors.py cluster_radius_nn(), which when printed out returns a tensor with size (2, 0). I believe this empty tensor suggests that there exist clusters without nearby neighbors.

I've attempted to solve this issue by increasing the config's graph_gap value but did not succeed in getting past the IndexError. I also tried scaling the XYZ points by a constant factor (before running src/train.py) to increase the points per meter. The scaling was met with partial success allowing me to train on a small subset of my data (+datamodule.mini=True), but some files still produced an IndexError.

Any suggestions on how I should proceed would be greatly appreciated. Perhaps I need to modify additional values inside my config.

Thanks for the amazing repo!

  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/datasets/base.py", line 493, in _process
    self.process()
  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/datasets/base.py", line 528, in process
    self._process_single_cloud(p)
  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/datasets/base.py", line 559, in _process_single_cloud
    nag = self.pre_transform(data)
  File "/home/acardaras/anaconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/transforms/compose.py", line 24, in __call__
    data = transform(data)
  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/transforms/transforms.py", line 23, in __call__
    return self._process(x)
  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/transforms/graph.py", line 649, in _process
    nag = _horizontal_graph_by_radius(
  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/transforms/graph.py", line 753, in _horizontal_graph_by_radius
    nag = _horizontal_graph_by_radius_for_single_level(
  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/transforms/graph.py", line 807, in _horizontal_graph_by_radius_for_single_level
    edge_index, distances = cluster_radius_nn(
  File "/home/acardaras/PycharmProjects/Superpoint/superpoint_transformer/src/utils/neighbors.py", line 382, in cluster_radius_nn
    d_nn = (x_points[anchors[0]] - x_points[anchors[1]]).norm(dim=1)

Output prediction resolution & PyTorch Lightning `predict` behavior

Hi Damien,

I've modified the code a bit to ingest a different dataset. and I have a few questions

The number of points that are inputs don't match the number of outputs when doing the pre-processing. I've narrowed this down to the grouping that is done in the pre-processing. Is there a way to infer on the entire dataset and not just the mean of the points per voxel?
Is it because it's just modified from the notebook?
How do I use the predict function in pytorch lightning, I've seen that the eval.py is just stock standard - is there an implementation for prediction for an entire dataset rather than a subset?

Thanks!

How to get the Superpoint Partitions of every room in S3DIS

Hello @drprojects,

Thank you for your wonderful work, SPT! Your Superpoint Partition algorithm holds great promise for various 3D point cloud tasks, such as instance segmentation. Currently, I'm interested in obtaining the superpoint partitions (1st-level partitions) for each room within the S3DIS dataset.

I attempted to use your notebook, but I noticed that it merges the entire area into a single pointcloud. Could you show me how to separate it into distinct rooms? Furthermore, I examined the point cloud sizes, and it appears that the first level of the nag object for S3DIS area_5 contains just 9,308,118 points. However, when I load the point clouds for all the rooms in Area_5 and concatenate them, the total number of points amounts to 78,649,818 points. I'm curious about how I can recover the superpoints for each room's point cloud.

Thank you for your assistance, and I eagerly await your response.

How to find `state_dict` in a `.ckpt` file ?

sorry i am having trouble accessing the trained model weights of the
PointSegmentationModule and especially the encoder and the decoder,
i would like to inspect them and edit them how can i do it? thank you

cfg = init_config(overrides=[
    "experiment=kitti360",
    "ckpt_path=mycheckpoint.ckpt"
])
datamodule = hydra.utils.instantiate(cfg.datamodule)
datamodule.prepare_data()
datamodule.setup()
model = hydra.utils.instantiate(cfg.model)
model = model.load_from_checkpoint(cfg.ckpt_path, net=model.net, criterion=model.criterion)
model = model.eval().cuda()
model.state_dict() # error here

Project root directory not found. Indicators: ('.project-root', 'setup.cfg', 'setup.py', '.git', 'pyproject.toml')

Hello, I am running pyrootutils.setup_ root(".", pythonpath=True).The error reported during root code is as follows:Project root directory not found. Indicators: ('.project-root', 'setup.cfg', 'setup.py', '.git', 'pyproject.toml')
How to solve it?

Issue while running inference on custom data

Hi
We are using Superpoint transform pretrained weights to perform inference on our custom data.
We are using Dales pretrained weights to perform the inference on our custom data. Our custom data has other classes . We are mapping the classes in custom data to Dales classes before loading the data.

We are performing this mapping as follows :

DCILDATA_TO_DALESMAP={ 1 : 8.0,
                       2 : 0,
                       3 : 1.0,
                       4 : 1.0 ,
                       5 : 1.0 , 
                       6 : 7.0 , 
                       7:  8.0 ,
                       8 : 8.0 ,
                       9:  8.0 ,
                      10:  8.0 ,
                      11:  8.0 ,
                      12:  8.0 , 
                      13:  8.0 ,
                      14:  8.0 , 
                      15 : 8.0 , 
                      16 : 1.0 ,
                      17:  1.0 ,
                      18:  6.0 , 
                      19:  2.0,
                      20:  4.0 ,
                      21 : 4.0 ,
                      22:  8.0 ,
                      30 : 8.0 ,
                      31: 8.0 ,
                      32 : 8.0 ,
                      33 : 8.0 ,
                      38: 8.0 ,
                      99 : 8.0
                     }

def read_dales_tile(
        filepath, xyz=True, intensity=True, semantic=True, instance=False,
        remap=False):
    data = Data()
    key = 'testing'
    
    print("~~~~~~~~~~~spt:src:datasets:dales~~~~~~~~~~~~")
    with open(filepath, "rb") as f:
        tile = PlyData.read(f)

        if xyz:
#             print('tile---------------------------------',tile[key][axis])
            data.pos = torch.stack([
                torch.FloatTensor(tile[key][axis])
                for axis in ["x", "y", "z"]], dim=-1)

        if intensity:
            # Heuristic to bring the intensity distribution in [0, 1]
            data.intensity = torch.FloatTensor(
                tile[key]['scalar_Intensity']).clip(min=0, max=60000) / 60000

        **if semantic:
            myclass = tile[key]['scalar_Classification']
            
            newclass = [DCILDATA_TO_DALESMAP[cls] for cls in myclass]** 
            y = torch.LongTensor(np.array(newclass))
            data.y = torch.from_numpy(ID2TRAINID)[y] if remap else y

        if instance:
            data.instance = torch.LongTensor(tile[key]['ins_class'])
    print("data.pos",data.pos)
    print("data.y",data.y)
    return data

The following error is seen when the code tries to perform transformations in line 147 of superpoint_transformer/src/transforms/sampling.py

RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

Please let us know how can we resolve this issue .

Thanks and Regards
Soumya Manjunath

Inference with pretrain model

Good morning, thank you for sharing this repository and for your magnificent work. I would like to know, if it is possible to use one of your model pretrained, for example in Kitti3D, to do inference about a new point cloud. In my case i would like to do a 3D semantic segmentation on an acquired pointcloud. Thank you very much!

custom dataset problem

sorry i have errors as below by running train.py using a modified the kitti-360 dataset,
i did it by simply modifying the manifest of where the train,val,test files point to, and renaming kitti-360 to another instance in the configuration files
how do i solve this? thank you

Traceback (most recent call last):
  File "/superpoint_transformer/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))ying
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 980, in _run
    results = self._run_stage()
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1023, in _run_stage
    self.fit_loop.run()
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 198, in run
    self.on_run_start()
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 316, in on_run_start
    call._call_callback_hooks(trainer, "on_train_start")
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 195, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/root/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/callbacks/lr_monitor.py", line 107, in on_train_start
    raise MisconfigurationException(
lightning_fabric.utilities.exceptions.MisconfigurationException: Cannot use `LearningRateMonitor` callback with `Trainer` that has no logger.

Test with custom dataset, and running error with existing dataset

Hi @drprojects , great project! quite new to this type of models, some questions hope you can help with:

I have some point cloud in ( x, y, z, intensity) format, can I use one of the existing model s3dis do the training and apply to my custom dataset? which script should I use or modify from?
my dataset is quite special (like cup shapes), do I necessarily need some custom training? , If I need to prepare training dataset, what is the format I need to prepare?

I have a 12Gb RAM GPU, currently git clone the project and download the training dataset, in structure:

-./github/superpoint_transformer/data/s3dis/
|--Stanford3dDataset_v1.2.zip  
|--processed  
|--raw
|----Area#
|------....

some errors when try directly use existing dataset to start:
- run command: python src/train.py experiment=s3dis_11g datamodule.fold=5
with error:

[2023-09-26 17:33:44,718][src.utils.utils][INFO] - Closing loggers...
[2023-09-26 17:33:44,719][src.utils.utils][INFO] - Closing wandb!
wandb: Waiting for W&B process to finish... (success).
wandb: 🚀 View run SPT-64 at: https://wandb.ai/czeng/spt_s3dis/runs/u7amf1ra
wandb: ️⚡ View job at https://wandb.ai/czeng/spt_s3dis/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjEwMTM3NTE5Ng==/version_details/v0
wandb: Synced 6 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./logs/train/runs/2023-09-26_17-24-34/wandb/run-20230926_172436-u7amf1ra/logs
Error executing job with overrides: ['experiment=s3dis_11g', 'datamodule.fold=5']
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/root/miniconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/root/miniconda3/envs/spt/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "./github/superpoint_transformer/src/utils/multiprocessing.py", line 41, in apply_args_and_kwargs
    return fn(*args, **kwargs)
  File "./github/superpoint_transformer/src/datasets/s3dis.py", line 164, in read_s3dis_room
    xyz_data = torch.from_numpy(np.concatenate(xyz_list, 0)) if xyz else None
  File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate
"""
The above exception was the direct cause of the following exception:
...

after download the checkpoints files from page, stored under:
\superpoint_transformer\checkpoints\xxx.ckpt , then

run command: python src/eval.py experiment=s3dis datamodule.fold=5 ckpt_path=/path/to/your/checkpoint.ckpt
it fails with error message:

...
├── extras
│   └── ignore_warnings: false
│       enforce_tags: true
│       print_config: true
│
├── task_name
│   └── eval
├── tags
│   └── ['dev']
├── compile
│   └── False
├── ckpt_path
│   └── ./checkpoints/spt-2_s3dis_fold5.ckpt
└── float32_matmul_precision
    └── highest
[2023-09-26 22:30:56,523][__main__][INFO] - Instantiating datamodule <src.datamodules.s3dis.S3DISDataModule>
[2023-09-26 22:30:56,690][__main__][INFO] - Instantiating model <src.models.segmentation.PointSegmentationModule>
[2023-09-26 22:30:56,864][__main__][INFO] - Instantiating loggers...
[2023-09-26 22:30:56,865][__main__][INFO] - Instantiating trainer <pytorch_lightning.Trainer>
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
[2023-09-26 22:30:57,001][__main__][INFO] - Starting testing!
Processing...
  0%|                                                                                                                                                                                                                                | 0/10 [00:00<?, ?it/s]
[process exited with code 1 (0x00000001)]
You can now close this terminal with Ctrl+D, or press Enter to restart.

does the evaluation also requires 32GB ram GPU?

thanks,
Chui

Implementing PyTorch Lightning `predict` behavior

Hello there,

Having successfully trained a custom dataset and evaluated it, I would like to uncomment the trainer.predict() line located in the eval.py script for visual inspection. Unfortunately, I've been battling to implement the predict_dataloader, predict_dataset and other new variables needed for this.

To be more specific, my only issue is that after a successful instantiation of the cfg.datamodule, the variable datamodule.predict_dataloader().dataset is set to None while it should should be set to MyCustomDataset(len(predict_dataset)).

Up to this point, I tried to mimick the implementation of test_dataloader, test_dataset etc. in various scripts:

I added predict_transform and on_device_predict_transform to the configs/datamodule/MyCustomDataset.yaml script.
I initialized self.predict_dataset, self.predict_transform and self.on_device_predict_transform to None, added lines to the prepare_data() and setup() functions, implemented predict_dataloader() in src/datamodules/base.py.
I added a predict_dir to the process() function in src/datasets/base.py.

Could you please help me understand what is missing or located in the code where are loaded the datasets ?

Issue with loading checkpoints

Hi Damien,

Thanks for the great work and the code!

I'm currently performing some experiments with a docker-ized SPT based on nvidia/cuda:11.8.0-devel-ubuntu22.04 , on Kitty360.

I have no problem running the training script (both standard and 11g configs run smoothly), however, the evaluation script automatically fails when reading the saved checkpoints with the following error:

Traceback (most recent call last):
  File "/app/superpoint_transformer/src/models/segmentation.py", line 545, in load_state_dict
    super().load_state_dict(state_dict, strict=strict)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PointSegmentationModule:
	Unexpected key(s) in state_dict: "criterion.criteria.0.weight", "criterion.criteria.1.weight".

The same error happens with epoch-XXX.ckpt, latest.ckpt or the pretrained weights downloaded from your git,

A quick google search points to the loading modules that may try reading a distinct model from what has been stored, but since I didn't modify the config files except for adding the path to the data, this seems weid.

Any hints regarding this error?

Best regards,
Stephane

I have downloaded the data, how to obtain the preprocessed data

Hi Damien:

I have prepared the s3dis dataset and do not know how to obtain the preprocessed data

└── data
└── s3dis # Structure for S3DIS
├── Stanford3dDataset_v1.2.zip # (optional) Downloaded zipped dataset with non-aligned rooms
├── raw # Raw dataset files
│ └── Area_{{1, 2, 3, 4, 5, 6}} # S3DIS's area/room/room.txt structure
│ └── Area_{{1, 2, 3, 4, 5, 6}}alignmentAngle.txt # Room alignment angles required for entire floor reconstruction
│ └── {{room_name}}
│ └── {{room_name}}.txt
****└── processed # Preprocessed data
└── {{train, val, test}} # Dataset splits
└── {{preprocessing_hash}} # Preprocessing folder
└── Area{{1, 2, 3, 4, 5, 6}}.h5 # Preprocessed Area file****

s3dis dataset corrupt

when i downloaded the Stanford3dDataset_v1.2.zip
this file in line 323473 has a symbol that makes it corrupted
Area_5/office_19/Annotations/ceiling_1.txt

when i train using
python src/train.py experiment=s3dis datamodule.xy_tiling=2 datamodule.fold=6
it runs fine

python src/eval.py experiment=s3dis datamodule.xy_tiling=2 datamodule.fold=6 ckpt_path=mycheckpoint
it cant run
spt_error.txt

could this data corruption cause the error?

Adjusting the partition hyperparameters `regularization`, `cutoff`, `spatial_weight`

Hello,

I'm using the code to process my own data, and I visualized the results of partition, I noticed that each superpoint in the first level is very small. I want to adjust the hyperparameters to obtain larger superpoints in the first stage. How can I adjust the parameters regularization, cutoff, spatial_weight to achieve this?

Also, I would like to understand the functions of these three parameters in the partition process. Could you briefly explain, or guide me to where I can find relevant documentation regarding their functions?

Thanks a lot! 😄

Error During Data Pre-processing on Custom MLS Dataset

Hello there @drprojects, @rjanvier, @loicland, @CharlesGaydon ! Its very nice to see a very well documented, state-of-art architecture which is user-friendly when it comes to setting up and running. Thanks for your work on the Superpoint Transformer.

We (@pyarelalchauhan, @xbais) are trying to train the architecture on a custom dataset collected in India. We have prepared the dataset as Binary PLY files similar to those in the DALES Object dataset (please see the header of one of our files attached below):

We have generated the relevant Configuration files and other Python files for our dataset taking inspiration from similar files available for DALES and S3DIS datasets provided in your repository. Some of the changes we have made according to our dataset are in these directories:

/configs/datamodule : added our custom YAML file
configs/experiment : added relevant YAML files for our dataset
/data/ : added custom_data/raw/train and custom_data/raw/test
/src/datamodules : added relevant Python file for our dataset.
/src/datasets/ : added relevant custom-data.py and custom-data_config.py files

We have read the posts #32 (related to RANSAC), #36 (in which you talk about the parameters voxel, knn, knn_r, pcp_regularization, pcp_spatial_weight, pcp_cutoff). But we are still facing issues. It will be greate if you can help us out here!!

👉 Regarding Errors and Warnings

We are getting the following errors and warnings which we are unable to resolve at the moment :

Warning in Sckit-Learn Regression :
NAG Related Issue : Cannot compute radius-based horizontal graph :
ValueError min_samples may not be larger than number of samples: n_samples = 2 :

(Following your advice on #32 , we have already removed "elevation" from partition_hf and point_hf, but still could not get the training to start.
Torch.cat() : expected a non-empty list of Tensors

👉 Regarding Understanding the Configuration

Could you also explain the significance of the value pcp_regularization, pcp_spatial_weight and pcp_cutoff parameters in the /configs/datamodule/custom_data.yaml file.

We are currently using the following configuration values :

We have tried tweaking these, but cannot get beyond the processing stage for our dataset. Tweaking these params gives one or more of the above mentioned errors and warnings at different stages of processing. Kindly help.

PS : We have already ⭐ ed your repo 😉

The number of items in the Loss function is different from the number of stages matching

Hi,When I run train.py, found 4 stages, but 2 criteria in the loss.

Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
    return _target_(*args, **kwargs)
  File "/home/zhaojing/code/superpoint_transformer-master/src/models/segmentation.py", line 53, in __init__
    assert len(self.net.out_dim) == len(self.criterion), \
AssertionError: The number of items in the multi-stage loss must match the number of stages in the net. Found 4 stages, but 2 criteria in the loss.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/zhaojing/code/superpoint_transformer-master/src/train.py", line 139, in main
    metric_dict, _ = train(cfg)
  File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 48, in wrap
    raise ex
  File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "/home/zhaojing/code/superpoint_transformer-master/src/train.py", line 82, in train
    model: LightningModule = hydra.utils.instantiate(cfg.model)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    return instantiate_node(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
    return _call_target(_target_, partial, args, kwargs, full_key)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'src.models.segmentation.PointSegmentationModule':
AssertionError('The number of items in the multi-stage loss must match the number of stages in the net. Found 4 stages, but 2 criteria in the loss.')
full_key: model
python-BaseException

Where can I get pretrained model?

I want to try this project, but I‘m not able to find a pre-trained model. Where can I get pretrained model?

Looking forward your help, thank you！

Evaluation on KITTI-360 Test set

I have a problem viewing the performance evaluation numbers when i run

python src/eval.py experiment=kitti360  ckpt_path='downloaded checkpoint from your website'

below is the result i got

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test/loss         │            0.0            │
│         test/macc         │            0.0            │
│         test/miou         │            0.0            │
│          test/oa          │            0.0            │
└───────────────────────────┴───────────────────────────┘

but when i trained a new model from scratch with the kitti360

python src/train.py experiment=kitti360

i can get view the numbers during training

val/miou_best: 63.757 val/oa_best: 92.886 val/macc_best: 79.989

I get this warning during evaluation
"You are using cuda device ('nvidia geforce rtx 4090') that has Tensor Cores. To properly utilize them you should set 'torch.set_float32_matmul_precision('medium' | 'high) which will trade-off precision for performance. For more details read https://pytorch.org/docs/stable/generated.torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision

i tried using high and medium precision but theres no change in the evaluation result.

i used this project in a docker container with gpu passthrough according to your setup. cuda11.8
please help, thank you for the project

run train.py

Hi,When I ran train.py, I encountered the following error

[2023-07-01 22:16:13,128][src.utils.utils][INFO] - Instantiating logger <pytorch_lightning.loggers.wandb.WandbLogger>
Error executing job with overrides: []
[2023-07-01 22:16:13,822][src.utils.utils][ERROR] - 
Traceback (most recent call last):
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
    return _target_(*args, **kwargs)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/loggers/wandb.py", line 358, in __init__
    _ = self.experiment
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/lightning_fabric/loggers/logger.py", line 114, in experiment
    return fn(self)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/loggers/wandb.py", line 406, in experiment
    self._experiment = wandb.init(**self._wandb_init)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1171, in init
    raise e
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1148, in init
    wi.setup(kwargs)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 289, in setup
    wandb_login._login(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 298, in _login
    wlogin.prompt_api_key()
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 228, in prompt_api_key
    raise UsageError("api_key not configured (no-tty). call " + directive)
wandb.errors.UsageError: api_key not configured (no-tty). call wandb.login(key=[your_api_key])

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "/home/zhaojing/code/superpoint_transformer-master/src/train.py", line 88, in train
    logger: List[Logger] = utils.instantiate_loggers(cfg.get("logger"))
  File "/home/zhaojing/code/superpoint_transformer-master/src/utils/utils.py", line 132, in instantiate_loggers
    logger.append(hydra.utils.instantiate(lg_conf))
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
    return instantiate_node(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
    return _call_target(_target_, partial, args, kwargs, full_key)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'pytorch_lightning.loggers.wandb.WandbLogger':
UsageError('api_key not configured (no-tty). call wandb.login(key=[your_api_key])')
full_key: logger.wandb
[2023-07-01 22:16:13,824][src.utils.utils][INFO] - Closing loggers...
Error in call to target 'pytorch_lightning.loggers.wandb.WandbLogger':
UsageError('api_key not configured (no-tty). call wandb.login(key=[your_api_key])')
full_key: logger.wandb

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Inference on own data

I want to run the inference on our data using dales p retrained weights. my data classes is similar to dales data classes. how can i convert my data into proper format. or how can i run the inference on my data.

Is CUDA 12.0 supported now?

Prediction without sampling

Hello,
Thanks for the great work. I trained a model on my own data. The results are good and I could run the inference on the test dataset. However, I noticed that we can't run the inference on data without sub-sampling, thus reducing the number of points. Is there any possibility to run inference without sampling and keep the original data points?

In my .yaml file I noticed there is a GridSampling operation:

# Preprocessing
pre_transform:
    - transform: DataTo
      params:
        device: 'cuda'
    - transform: GridSampling3D
      params:
        size: ${datamodule.voxel}
        hist_key: 'y'
        hist_size: ${eval:'${datamodule.num_classes} + 1'}

If I delete this transformation, I get this error:

Traceback (most recent call last):
  File "src/train.py", line 139, in main
    metric_dict, _ = train(cfg)
  File "/home/anass/superpoint_transformer/src/utils/utils.py", line 48, in wrap
    raise ex
  File "/home/anass/superpoint_transformer/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 932, in _run
    self._data_connector.prepare_data()
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
    call._call_lightning_datamodule_hook(trainer, "prepare_data")
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 166, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/home/anass/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data
    self.dataset_class(
  File "/home/anass/superpoint_transformer/src/datasets/base.py", line 193, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
    super().__init__(root, transform, pre_transform, pre_filter, log)
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in __init__
    self._process()
  File "/home/anass/superpoint_transformer/src/datasets/base.py", line 493, in _process
    self.process()
  File "/home/anass/superpoint_transformer/src/datasets/base.py", line 528, in process
    self._process_single_cloud(p)
  File "/home/anass/superpoint_transformer/src/datasets/base.py", line 559, in _process_single_cloud
    nag = self.pre_transform(data)
  File "/home/anass/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/transforms/compose.py", line 24, in __call__
    data = transform(data)
  File "/home/anass/superpoint_transformer/src/transforms/transforms.py", line 23, in __call__
    return self._process(x)
  File "/home/anass/superpoint_transformer/src/transforms/partition.py", line 195, in _process
    assert d1.y.dim() == 2, \
AssertionError: Expected Data.y to hold `(num_nodes, num_classes)` histograms, not single labels

It appears that the Data.y tensor doesn't have the expected shape.

RANSAC Error on Custom Dataset

Hello,
I've followed the documentation guidelines to train SPT on my custom dataset, but I fail to go through the training step as a ValueError raised by ransac emerges on one of the point clouds.

Error executing job with overrides: ['experiment=HAG', 'ckpt_path=../HAG.ckpt']
Traceback (most recent call last):
  File "src/train.py", line 139, in main
    metric_dict, _ = train(cfg)
  File "/data/Moussa/superpoint_transformer/src/utils/utils.py", line 48, in wrap
    raise ex
  File "/data/Moussa/superpoint_transformer/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 932, in _run
    self._data_connector.prepare_data()
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
    call._call_lightning_datamodule_hook(trainer, "prepare_data")
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 166, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/data/Moussa/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data
    self.dataset_class(
  File "/data/Moussa/superpoint_transformer/src/datasets/base.py", line 200, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
    super().__init__(root, transform, pre_transform, pre_filter, log)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in __init__
    self._process()
  File "/data/Moussa/superpoint_transformer/src/datasets/base.py", line 511, in _process
    self.process()
  File "/data/Moussa/superpoint_transformer/src/datasets/base.py", line 546, in process
    self._process_single_cloud(p)
  File "/data/Moussa/superpoint_transformer/src/datasets/base.py", line 577, in _process_single_cloud
    nag = self.pre_transform(data)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/transforms/compose.py", line 24, in __call__
    data = transform(data)
  File "/data/Moussa/superpoint_transformer/src/transforms/transforms.py", line 23, in __call__
    return self._process(x)
  File "/data/Moussa/superpoint_transformer/src/transforms/point.py", line 223, in _process
    ransac = RANSACRegressor(random_state=0, residual_threshold=1e-3).fit(
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/sklearn/linear_model/_ransac.py", line 358, in fit
    raise ValueError(
ValueError: `min_samples` may not be larger than number of samples: n_samples = 2.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Could you help me on this issue ?

Can't resolve OOM Error caused by FRNN

Hello,
I've been trying to run a training on dales using the command line

python src/train.py experiment=dales_11g ckpt_path=../dales_11g.ckpt

on a 2080Ti GPU with 11 Gb of RAM, but I always end up crashing because of a torch.cuda.OutOfMemoryError. I tried to tweak some of the parameters proposed in the README, but nothing does it.
From my perspective, it looks like the training script always crash when calling frnn_grid_points. Here is an exemple traceback :

Error executing job with overrides: ['experiment=dales_11g', 'ckpt_path=../dales_11g.ckpt', 'datamodule.xy_tiling=6', 'datamodule.graph_k_max=2', 'datamodule.dataloader.batch_size=8']
Traceback (most recent call last):
  File "src/train.py", line 139, in main
    metric_dict, _ = train(cfg)
  File "/data/Moussa/spt/superpoint_transformer/src/utils/utils.py", line 48, in wrap
    raise ex
  File "/data/Moussa/spt/superpoint_transformer/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 932, in _run
    self._data_connector.prepare_data()
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
    call._call_lightning_datamodule_hook(trainer, "prepare_data")
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 166, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/data/Moussa/spt/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data
    self.dataset_class(
  File "/data/Moussa/spt/superpoint_transformer/src/datasets/base.py", line 193, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
    super().__init__(root, transform, pre_transform, pre_filter, log)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in __init__
    self._process()
  File "/data/Moussa/spt/superpoint_transformer/src/datasets/base.py", line 493, in _process
    self.process()
  File "/data/Moussa/spt/superpoint_transformer/src/datasets/base.py", line 528, in process
    self._process_single_cloud(p)
  File "/data/Moussa/spt/superpoint_transformer/src/datasets/base.py", line 559, in _process_single_cloud
    nag = self.pre_transform(data)
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/transforms/compose.py", line 24, in __call__
    data = transform(data)
  File "/data/Moussa/spt/superpoint_transformer/src/transforms/transforms.py", line 23, in __call__
    return self._process(x)
  File "/data/Moussa/spt/superpoint_transformer/src/transforms/neighbors.py", line 46, in _process
    neighbors, distances = knn_1(
  File "/data/Moussa/spt/superpoint_transformer/src/utils/neighbors.py", line 36, in knn_1
    distances, neighbors, _, _ = frnn.frnn_grid_points(
  File "/data/Moussa/spt/superpoint_transformer/src/dependencies/FRNN/frnn/frnn.py", line 331, in frnn_grid_points
    idxs, dists, sorted_points2, pc2_grid_off, sorted_points2_idxs, grid_params_cuda = _frnn_grid_points.apply(
  File "/home/moussa/miniconda3/envs/spt/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/data/Moussa/spt/superpoint_transformer/src/dependencies/FRNN/frnn/frnn.py", line 174, in forward
    idxs, dists = _C.find_nbrs_cuda(sorted_points1, sorted_points2,
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 10.75 GiB total capacity; 45.41 MiB already allocated; 44.50 MiB free; 62.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Got error "Indices must be dense" using my own dataset

Hi, thank you for sharing the code of your promising paper! I want to try the code with my own dataset, but I got an error during preprocess. When does this AssertionError happen? I believe I made a custom loader to load my dataset referring to s3dis's loader, but it or my own dataset seem to be causing the error. It seems properly to load dataset for a while during the preprocess, but stops in the middle of it. The following is the traceback or the error. Could you tell me what the cause would be for this error?

Traceback (most recent call last):
  File "src/train.py", line 139, in main
    metric_dict, _ = train(cfg)
  File "/home/ubuntu/superpoint_transformer_custom/src/utils/utils.py", line 48, in wrap
    raise ex
  File "/home/ubuntu/superpoint_transformer_custom/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 529, in fit
    call._call_and_handle_interrupt(
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 568, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 925, in _run
    self._data_connector.prepare_data()
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
    call._call_lightning_datamodule_hook(trainer, "prepare_data")
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 164, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/home/ubuntu/superpoint_transformer_custom/src/datamodules/base.py", line 144, in prepare_data
    self.dataset_class(
  File "/home/ubuntu/superpoint_transformer_custom/src/datasets/custom.py", line 240, in __init__
    super().__init__(*args, val_mixed_in_train=True, **kwargs)
  File "/home/ubuntu/superpoint_transformer_custom/src/datasets/base.py", line 193, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
    super().__init__(root, transform, pre_transform, pre_filter, log)
  File "/home/ubuntu/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in __init__
    self._process()
  File "/home/ubuntu/superpoint_transformer_custom/src/datasets/base.py", line 493, in _process
    self.process()
  File "/home/ubuntu/superpoint_transformer_custom/src/datasets/base.py", line 528, in process
    self._process_single_cloud(p)
  File "/home/ubuntu/superpoint_transformer_custom/src/datasets/base.py", line 576, in _process_single_cloud
    nag.save(
  File "/home/ubuntu/superpoint_transformer_custom/src/data/nag.py", line 236, in save
    data.save(
  File "/home/ubuntu/superpoint_transformer_custom/src/data/data.py", line 572, in save
    save_dense_to_csr(val, sg, fp_dtype=fp_dtype)
  File "/home/ubuntu/superpoint_transformer_custom/src/utils/io.py", line 111, in save_dense_to_csr
    pointers, columns, values = dense_to_csr(x)
  File "/home/ubuntu/superpoint_transformer_custom/src/utils/sparse.py", line 53, in dense_to_csr
    pointers = indices_to_pointers(index[0])[0]
  File "/home/ubuntu/superpoint_transformer_custom/src/utils/sparse.py", line 18, in indices_to_pointers
    assert is_dense(indices), "Indices must be dense"
AssertionError: Indices must be dense

How to train or validate over s3dis and s3disroom?

Hello,
Suppose we have two datamodule, for instance s3dis and s3disroom.
I would like to train a model on s3dis and validate over s3dis and s3disroom to check how the model behave on a subsample of s3dis,

Do you know if their is an easy way to do so?

What I already tried:

First, according to pytorch lightning I could iterate over multiple data_loaders. However, when I do trainer.test(model=model,dataloader=datamodule.test_dataloader()) I got an error: AttributeError: 'NAG' object has no attribute 'x'. I suspect that some operation on NAG isn't coded in the dataloader :(
I tried to create a new datamodule

fromtyping import Any, Dict
from lightning import LightningDataModule
from src.loader.dataloader import DataLoader
from src.data import NAGBatch
from lightning.pytorch.utilities.combined_loader import CombinedLoader

from lightning.pytorch.utilities.types import TRAIN_DATALOADERS
class ConcatenateDatamodule1(LightningDataModule):
    KEY_MAIN=0
    def __init__(self,l_datamodule,**kwargs):
        super().__init__()
        self.save_hyperparameters(logger=False)
        self.kwargs = kwargs

        assert len(l_datamodule)>0,"non"
        self.l_datamodule:BaseDataset=l_datamodule
        self.main_datamodule=self.l_datamodule[self.KEY_MAIN]
        self.train_dataset=[]
        self.val_dataset=[]
        self.test_dataset=[]
        self.pred_dataset=[]

        self.pre_transform = []
        self.train_transform = []
        self.val_transform = []
        self.test_transform = []
        self.on_device_train_transform = []
        self.on_device_val_transform = []
        self.on_device_test_transform = []
        self.on_device_pred_transform = []
        self.pred_transform=[]
        self.set_transforms()

    def set_transforms(self):
        for datamodule in self.l_datamodule:
            KEY=["pre_transform","train_transform","val_transform","test_transform","on_device_train_transform","on_device_val_transform","on_device_test_transform","on_device_pred_transform"]
            for key in KEY: 
                self.__getattribute__(key).append(datamodule.__getattribute__(key))


    @property
    def dataset_class(self):
        """Return the LightningDataModule's Dataset class.
        """
        return self.main_datamodule.dataset_class()
    @property
    def train_stage(self):
        return self.main_datamodule.train_stage()
    @property
    def val_stage(self):
        return self.main_datamodule.val_stage(


        )
    def prepare_data(self):
        for datamodule in self.l_datamodule:
            datamodule.prepare_data() #?
    def setup(self,stage:str):
        KEY=["train_dataset","val_dataset","test_dataset"]
        for datamodule in self.l_datamodule:
            datamodule.setup(stage=stage)
            for key in KEY:
                self.__getattribute__(key).append(datamodule.__getattribute__(key))



    def train_dataloader(self) -> TRAIN_DATALOADERS:
        l=[]
        for datamodule in self.l_datamodule:
            l.append(datamodule.train_dataloader())
        return l
    def val_dataloader(self) -> TRAIN_DATALOADERS:
        l=[]
        for datamodule in self.l_datamodule:
            l.append(datamodule.val_dataloader())
        return l
    def test_dataloader(self) -> TRAIN_DATALOADERS:
        l=[]
        for datamodule in self.l_datamodule:
            l.append(datamodule.test_dataloader())
        return l
    def predict_dataloader(self):
        # any iterable or collection of iterables
        l=[]
        for datamodule in self.l_datamodule:
            l.append(datamodule.predict_dataloader())
        return self.main_datamodule.predict_dataloader()

    def teardown(self, stage: str) -> None:
        pass 
    def state_dict(self):
        return {}
    def load_state_dict(self, state_dict: Dict[str, Any]) -> None:
        pass 

        
    @torch.no_grad()
    def on_after_batch_transfer(self, nag_list, dataloader_idx):
        """Intended to call on-device operations. Typically,
        NAGBatch.from_nag_list and some Transforms like SampleSubNodes
        and SampleSegments are faster on GPU, and we may prefer
        executing those on GPU rather than in CPU-based DataLoader.

        Use self.on_device_<stage>_transform, to benefit from this hook.
        """
        # Since NAGBatch.from_nag_list takes a bit of time, we asked
        # src.loader.DataLoader to simply pass a list of NAG objects,
        # waiting for to be batched on device.
        nag = NAGBatch.from_nag_list(nag_list)
        del nag_list

        # Here we run on_device_transform, which contains NAG transforms
        # that we could not / did not want to run using CPU-based
        # DataLoaders
        if self.trainer.training:
            on_device_transform = self.on_device_train_transform[dataloader_idx]
        elif self.trainer.validating:
            on_device_transform = self.on_device_val_transform[dataloader_idx]
        elif self.trainer.testing:
            on_device_transform = self.on_device_test_transform[dataloader_idx]
        elif self.trainer.predicting:
            #TODO: predict_dataloader
            on_device_transform=self.on_device_pred_transform[dataloader_idx]
        elif self.trainer.evaluating:
            on_device_transform = self.on_device_test_transform[dataloader_idx]
        elif self.trainer.sanity_checking:
            on_device_transform = self.on_device_train_transform[dataloader_idx]
        else:
            print(
                'Unsure which stage we are in, defaulting to '
                'self.on_device_train_transform')
            on_device_transform = self.on_device_train_transform[dataloader_idx]

        # Skip on_device_transform if None
        if on_device_transform is None:
            return nag

        # Apply on_device_transform only once when in training mode and
        # if no test-time augmentation is required
        if self.trainer.training \
                or self.hparams.tta_runs is None \
                or self.hparams.tta_runs == 1 or \
                (self.trainer.validating and not self.hparams.tta_val):
            return on_device_transform(nag)

        # We return the input NAG as well as the augmentation transform
        # and the number of runs. Those will be used by
        # `LightningModule.step` to accumulate multiple augmented runs
        return nag, on_device_transform, self.hparams.tta_runs

    def __repr__(self):
        return f'{self.main_datamodule.__class__.__name__ } '

but when I run trainer=trainer.predict(model=model,datamodule=ConcatenateDatamodule(datamodule,datamodule))
I got this shady error : TypeError: An invalid dataloader was passed to Trainer.predict(dataloaders=...). Found S3DISDatamodule

I eventually tried to hard code it in a callbacks and I got some strange errors about logging :(

I know this isn't exactly a superpoint_transformer question but it is highly related,
Sincerely,
Pierrick Bournez

RuntimeError: DataLoader worker (pid 15163) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit

Hello,
Thanks you for sharing your amazing repo,
I have a question:
When you trained your model and especially with the 11GPU, how much share memory did you have?
After struggling on preprocessing I have now a lot of issue with share memory error :(
My error is this one
RuntimeError: DataLoader worker (pid 15163) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit

Preprocessing s3dis and multiprocessing freezes after the first area

First, thanks for all the effort in making your project usable for the community 💪.

Issue
Preprocessing of the s3dis dataset hangs/freezes directly after the first batch (Area_1). The process does not crash, and no error messages are printed. Debugging shows that a lot of subprocesses are spawned; all rooms are processed, but the workers do not join anymore - all subprocesses are still alive, also with the fact that the read_s3dis_room function returns data. However, this line is never reached.

Environment
I use the Docker container nvidia/cuda:11.7.1-cudnn8-devel-ubuntu20.04 and executed the install.sh.

Package                   Version
------------------------- ----------------
absl-py                   1.4.0
aiohttp                   3.8.5
aiosignal                 1.3.1
ansi2html                 1.8.0
antlr4-python3-runtime    4.9.3
anyio                     3.5.0
appdirs                   1.4.4
argon2-cffi               21.3.0
argon2-cffi-bindings      21.2.0
arrow                     1.2.3
asttokens                 2.0.5
async-lru                 2.0.4
async-timeout             4.0.3
attrs                     23.1.0
Babel                     2.12.1
backcall                  0.2.0
beautifulsoup4            4.12.2
bleach                    4.1.0
cachetools                5.3.1
certifi                   2023.7.22
cffi                      1.15.1
charset-normalizer        3.2.0
click                     8.1.6
cloudpickle               2.2.1
cmake                     3.27.2
colorhash                 1.2.1
colorlog                  6.7.0
comm                      0.1.4
contourpy                 1.1.0
cycler                    0.11.0
dash                      2.11.1
dash-core-components      2.0.0
dash-html-components      2.0.0
dash-table                5.0.0
debugpy                   1.6.7
decorator                 5.1.1
defusedxml                0.7.1
docker-pycreds            0.4.0
entrypoints               0.4
executing                 0.8.3
fastjsonschema            2.16.2
filelock                  3.12.2
Flask                     2.2.5
fonttools                 4.42.0
fqdn                      1.5.1
frnn                      0.0.0
frozenlist                1.4.0
fsspec                    2023.6.0
gdown                     4.7.1
gitdb                     4.0.10
GitPython                 3.1.32
google-auth               2.22.0
google-auth-oauthlib      1.0.0
grpcio                    1.57.0
h5py                      3.9.0
hydra-colorlog            1.2.0
hydra-core                1.3.2
hydra-submitit-launcher   1.2.0
idna                      3.4
importlib-metadata        6.8.0
importlib-resources       5.2.0
ipykernel                 6.25.0
ipython                   8.12.2
ipython-genutils          0.2.0
ipywidgets                8.1.0
isoduration               20.11.0
itsdangerous              2.1.2
jedi                      0.18.1
Jinja2                    3.1.2
joblib                    1.3.2
json5                     0.9.14
jsonpointer               2.4
jsonschema                4.19.0
jsonschema-specifications 2023.7.1
jupyter_client            7.4.9
jupyter_core              5.3.0
jupyter-dash              0.4.2
jupyter-events            0.7.0
jupyter-lsp               2.2.0
jupyter_server            2.7.0
jupyter_server_terminals  0.4.4
jupyterlab                4.0.5
jupyterlab-pygments       0.1.2
jupyterlab_server         2.24.0
jupyterlab-widgets        3.0.8
kiwisolver                1.4.4
lightning-utilities       0.9.0
lit                       16.0.6
llvmlite                  0.40.1
lxml                      4.9.2
Markdown                  3.4.4
markdown-it-py            3.0.0
MarkupSafe                2.1.1
matplotlib                3.7.2
matplotlib-inline         0.1.6
mdurl                     0.1.2
mistune                   0.8.4
mpmath                    1.3.0
multidict                 6.0.4
nb-conda-kernels          2.3.1
nbclassic                 0.5.5
nbclient                  0.5.13
nbconvert                 6.5.4
nbformat                  5.7.0
nest-asyncio              1.5.6
networkx                  3.1
notebook                  6.5.4
notebook_shim             0.2.2
numba                     0.57.1
numpy                     1.24.4
nvidia-cublas-cu11        11.10.3.66
nvidia-cuda-cupti-cu11    11.7.101
nvidia-cuda-nvrtc-cu11    11.7.99
nvidia-cuda-runtime-cu11  11.7.99
nvidia-cudnn-cu11         8.5.0.96
nvidia-cufft-cu11         10.9.0.58
nvidia-curand-cu11        10.2.10.91
nvidia-cusolver-cu11      11.4.0.1
nvidia-cusparse-cu11      11.7.4.91
nvidia-nccl-cu11          2.14.3
nvidia-nvtx-cu11          11.7.91
oauthlib                  3.2.2
omegaconf                 2.3.0
overrides                 7.4.0
packaging                 23.0
pandas                    2.0.3
pandocfilters             1.5.0
parso                     0.8.3
pathtools                 0.1.2
pexpect                   4.8.0
pickleshare               0.7.5
Pillow                    10.0.0
pip                       23.2.1
pkgutil_resolve_name      1.3.10
platformdirs              2.5.2
plotly                    5.9.0
plyfile                   1.0.1
prefix-sum                0.0.0
prometheus-client         0.14.1
prompt-toolkit            3.0.36
protobuf                  4.24.0
psutil                    5.9.0
ptyprocess                0.7.0
pure-eval                 0.2.2
pyasn1                    0.5.0
pyasn1-modules            0.3.0
pycocotools               2.0.7
pycparser                 2.21
pyg-lib                   0.2.0+pt20cu118
Pygments                  2.15.1
pyparsing                 3.0.9
pyrootutils               1.0.4
pyrsistent                0.18.0
PySocks                   1.7.1
python-dateutil           2.8.2
python-dotenv             1.0.0
python-json-logger        2.0.7
pytorch-lightning         2.0.6
pytz                      2023.3
PyYAML                    6.0.1
pyzmq                     25.1.1
referencing               0.30.2
requests                  2.31.0
requests-oauthlib         1.3.1
retrying                  1.3.4
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rich                      13.5.2
rpds-py                   0.9.2
rsa                       4.9
scikit-learn              1.3.0
scipy                     1.10.1
seaborn                   0.12.2
Send2Trash                1.8.0
sentry-sdk                1.29.2
setproctitle              1.3.2
setuptools                68.0.0
six                       1.16.0
smmap                     5.0.0
sniffio                   1.2.0
soupsieve                 2.4
stack-data                0.2.0
submitit                  1.4.5
sympy                     1.12
tenacity                  8.2.3
tensorboard               2.14.0
tensorboard-data-server   0.7.1
terminado                 0.17.1
threadpoolctl             3.2.0
tinycss2                  1.2.1
tomli                     2.0.1
torch                     2.0.1
torch-cluster             1.6.1+pt20cu118
torch-geometric           2.3.1
torch-scatter             2.1.1+pt20cu118
torch-sparse              0.6.17+pt20cu118
torch-spline-conv         1.2.2+pt20cu118
torch-tb-profiler         0.4.1
torchmetrics              1.0.3
torchvision               0.15.2
tornado                   6.3.2
tqdm                      4.66.1
traitlets                 5.7.1
triton                    2.0.0
typing_extensions         4.7.1
tzdata                    2023.3
uri-template              1.3.0
urllib3                   1.26.16
wandb                     0.15.8
wcwidth                   0.2.5
webcolors                 1.13
webencodings              0.5.1
websocket-client          0.58.0
Werkzeug                  2.2.3
wheel                     0.38.4
widgetsnbextension        4.0.8
yarl                      1.9.2
zipp                      3.11.0

S3DIS running [Errno 111] Connection refused')': /api/4504800232407040/envelope/

Hello, Professor! I'm running python src/train.py experiment=s3dis_11g datamodule.fold=5 and have the following problem:

[2023-07-10 19:24:00,196][src.utils.utils][INFO] - Closing loggers...
[2023-07-10 19:24:00,196][src.utils.utils][INFO] - Closing wandb!
wandb: Waiting for W&B process to finish... (success).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/vge/spt/superpoint_transformer-master/logs/train/runs/2023-07-10_19-23-47/wandb/offline-run-20230710_192353-kkv84qta
wandb: Find logs at: ./logs/train/runs/2023-07-10_19-23-47/wandb/offline-run-20230710_192353-kkv84qta/logs
[2023-07-10 19:24:01,331][urllib3.connectionpool][WARNING] - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f884af8d9d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/4504800232407040/envelope/
[2023-07-10 19:24:01,333][urllib3.connectionpool][WARNING] - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f884af8dc40>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/4504800232407040/envelope/

Hope to get your help.

Problem with processing of the pointcloud

Good morning,
I have added another dataset, composed only of one pointcloud, which contains 1000k points. After the preprocessing step the number of points is reduced to 79000, which is less than 10% of the original number of points. Have you got any idea about how to solve this problem or which function I need to modify?
Sorry to bother you, thank you for your work

How to train my own dataset

Hello Professor！
My point cloud only has three properties of xyz, and the fourth column is the label value, how to train my own dataset using SPT network

About Superpoint-Graphs Computation

Hi, thanks for your amazing job!
I am interested in the implementation of Superpoint-Graphs Computation, could you tell me where the implementation please?

cuda out of memory when test

Hi Damien,
When I tested, my cuda of memory ran out.However,I can train s3dis =11g.I want to know how much cuda memory is needed for test.

Beside, I have trained s3dis-11g for 500 epochs,wanb shows me state of crashed.

NAG.select() reveals `torch-geometric>=2.4` backward compatibility error

Hi Damien, thank's for sharing SPT.

Environnement : NVIDIA RTX A6000 and Torch 2.0.0 // CUDA 11.7

I try to use NAG.select() and I get :

Code :
n = NAG.load('Area_6_conferenceRoom_1.h5') ; n.select(2, 10)

Do you have any ideas ?

Data preprocessing within Docker environment

Thanks for all the effort in making your project usable for the community.
When I was reproducing your code, I downloaded S3DIS to the specified location.
But after extracting data/s3dis/stanford3dDataset_v1.2.zip, then it keeps showing processing and the progress bar keeps showing 0%.No errors or crashes

how to use the network for semantic segmentation?

Hello, this page looks very powerful.
I simply ask what are the required steps to test my own point clouds for semantic segmentation?
thank you for your help.

Implementing SensatUrban

Hi Damian,

Thanks for SPT!
We are three university students trying to use SPT to classify point clouds.

We have been looking into incorporating the SensatUrban dataset into SPT.

It seems as if the data read correctly, as shown in the picture above.

We tried looking into issue #15 to resolve it, but it did not work.

Would you happen to have any suggestions on how to resolve this issue? We have been through the guide for setting up our own dataset several times and have found a lot of inspiration in how dales are setup.

Looking forward to hearing from you.
Best regards Jeppe

[enviroment configuration] Compilation of third-party libraries (eg `FRNN`)

Hi Damien:
When I run install.sh until the installation of third-party libraries such as FRNN, errors related to g++ and gcc are output. I can make sure that the system has gcc-7 & g++-7, but it doesn't seem to work.

running install
/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
running bdist_egg
running egg_info
writing prefix_sum.egg-info/PKG-INFO
writing dependency_links to prefix_sum.egg-info/dependency_links.txt
writing top-level names to prefix_sum.egg-info/top_level.txt
reading manifest file 'prefix_sum.egg-info/SOURCES.txt'
writing manifest file 'prefix_sum.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    setup(
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/command/install.py", line 80, in run
    self.do_egg_install()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/command/install.py", line 129, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 164, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 150, in call_command
    self.run_command(cmdname)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/command/install_lib.py", line 111, in build
    self.run_command('build_ext')
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
    super().run_command(command)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
    self.build_extensions()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 485, in build_extensions
    compiler_name, compiler_version = self._check_abi()
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 869, in _check_abi
    _, version = get_compiler_abi_compatibility_and_version(compiler)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 337, in get_compiler_abi_compatibility_and_version
    if not check_compiler_ok_for_platform(compiler):
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 291, in check_compiler_ok_for_platform
    which = subprocess.check_output(['which', compiler], stderr=subprocess.STDOUT)
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/shi/anaconda3/envs/spt/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['which', 'g++']' returned non-zero exit status 1.

Focal Loss

Hi @drprojects , I saw there was an implementation src/loss/focal.py of focal loss and I would like to apply it as loss function during traning on my dataset (imbalanced). Could you please advise how to do so? Thanks

Environment configuration issues

Hello, thank you for your great work. I'm currently replicating the effect in your paper, but I'm having the following problems configuring the environment:

⭐ Installing Parallel Cut-Pursuit

fatal: destination path 'src/dependencies/parallel_cut_pursuit' already exists and is not an empty directory.
fatal: destination path 'src/dependencies/grid_graph' already exists and is not an empty directory.
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cpython/cp_d0_dist_cpy.cpp
gcc: error: ../src/cp_kmpp_d0_dist.cpp: No such file or directory
error: command '/usr/bin/gcc' failed with exit code 1


🚀 Successfully installed SPT

There are no cp_kmpp_d0_dist.cpp in the path spt/superpoint_transformer-master/scripts/src/dependencies/parallel_cut_pursuit/python/cpython, only cp_d0_dist.cpp

fatal error: eigen3/Eigen/Dense: No such file or directory

Thank you for your hard working.
When i run install.sh, I have some issue.
Especially in installing point geometric features, there are the problem :

fatal error: eigen3/Eigen/Dense: No such file or directory

I think it is cpp config error, but sorry i dont have any idea about this solution..
Can you help me?
Thank you already !

Custom Dataset - AssertionError: Cloud ids must be unique across all the 'val' and 'test' stages, unless `test_mixed_in_val=True`

Hi Damien,

I tried to train the model in a custom dataset but encounter below error. (I have followed the docs to set up env, dataset & configs)

Traceback (most recent call last):
  File "/home/steven/Desktop/git/superpoint_transformer/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 989, in _run
    results = self._run_stage()
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1033, in _run_stage
    self._run_sanity_check()
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1062, in _run_sanity_check
    val_loop.run()
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 134, in run
    self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 365, in _evaluation_step
    batch = call._call_strategy_hook(trainer, "batch_to_device", batch, dataloader_idx=dataloader_idx)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 269, in batch_to_device
    return model._apply_batch_transfer_handler(batch, device=device, dataloader_idx=dataloader_idx)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/core/module.py", line 334, in _apply_batch_transfer_handler
    batch = self._call_batch_hook("on_after_batch_transfer", batch, dataloader_idx)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/core/module.py", line 322, in _call_batch_hook
    return trainer_method(trainer, hook_name, *args)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 179, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/steven/Desktop/git/superpoint_transformer/src/datamodules/base.py", line 336, in on_after_batch_transfer
    return on_device_transform(nag)
  File "/home/steven/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/transforms/compose.py", line 24, in __call__
    data = transform(data)
  File "/home/steven/Desktop/git/superpoint_transformer/src/transforms/transforms.py", line 23, in __call__
    return self._process(x)
  File "/home/steven/Desktop/git/superpoint_transformer/src/transforms/data.py", line 347, in _process
    raise ValueError(
ValueError: Input NAG does not have `is_val` attribute at level `0`

My data directory is like this:

/data/DATASET/
        └── raw/
            └── {train, test}/
                └── {file_name}.laz

and I want to train on train files and test on test files. In this case there is no validation dataset, is it condisered test_mixed_in_val ?
Could you please help on this? Thanks!

[Checkpoint Loading] Issues on loading s3dis ckpt weights when running 'demo_s3dis.ipynb'

Hi, thanks again for your great project.
I have encountered a problem while attempting load pre-trained checkpoint file spt-2_s3dis_fold5.ckpt when I run demo_s3dis.ipynb.
The detailed error report is as follows:

AttributeError                            Traceback (most recent call last)
Cell In[11], line 10
      7 model = hydra.utils.instantiate(cfg.model)
      9 # Load pretrained weights from a checkpoint file
---> 10 model = model.load_from_checkpoint(cfg.ckpt_path, net=model.net, criterion=None)
     11 model.criterion = hydra.utils.instantiate(cfg.model).criterion
     12 model = model.eval().cuda()

File /home/pai/envs/spt/lib/python3.8/site-packages/pytorch_lightning/core/module.py:1520, in LightningModule.load_from_checkpoint(cls, checkpoint_path, map_location, hparams_file, strict, **kwargs)
   1440 @classmethod
   1441 def load_from_checkpoint(
   1442     cls,
   (...)
   1447     **kwargs: Any,
   1448 ) -> Self:
   1449     r"""
   1450     Primary way of loading a model from a checkpoint. When Lightning saves a checkpoint
   1451     it stores the arguments passed to ``__init__``  in the checkpoint under ``"hyper_parameters"``.
   (...)
   1518         y_hat = pretrained_model(x)
   1519     """
-> 1520     loaded = _load_from_checkpoint(
   1521         cls,
   1522         checkpoint_path,
   1523         map_location,
   1524         hparams_file,
   1525         strict,
   1526         **kwargs,
   1527     )
   1528     return cast(Self, loaded)

File /home/pai/envs/spt/lib/python3.8/site-packages/pytorch_lightning/core/saving.py:90, in _load_from_checkpoint(cls, checkpoint_path, map_location, hparams_file, strict, **kwargs)
     88     return _load_state(cls, checkpoint, **kwargs)
     89 if issubclass(cls, pl.LightningModule):
---> 90     model = _load_state(cls, checkpoint, strict=strict, **kwargs)
     91     state_dict = checkpoint["state_dict"]
     92     if not state_dict:

File /home/pai/envs/spt/lib/python3.8/site-packages/pytorch_lightning/core/saving.py:156, in _load_state(cls, checkpoint, strict, **cls_kwargs_new)
    154 # load the state_dict on the model automatically
    155 assert strict is not None
--> 156 keys = obj.load_state_dict(checkpoint["state_dict"], strict=strict)
    158 if not strict:
    159     if keys.missing_keys:

File /mnt/user/superpoint_transformer/src/models/segmentation.py:551, in PointSegmentationModule.load_state_dict(self, state_dict, strict)
    549 # Special treatment for MultiLoss
    550 if self.multi_stage_loss:
--> 551     class_weight_bckp = self.criterion.weight
    552     self.criterion.weight = None
    554     # Recover the class weights from any 'criterion.weight' or
    555     # 'criterion.*.weight' key and remove those keys from the
    556     # state_dict

File /home/pai/envs/spt/lib/python3.8/site-packages/torch/nn/modules/module.py:1614, in Module.__getattr__(self, name)
   1612     if name in modules:
   1613         return modules[name]
-> 1614 raise AttributeError("'{}' object has no attribute '{}'".format(
   1615     type(self).__name__, name))

AttributeError: 'MultiLoss' object has no attribute 'weight'

Empty tensor when running inference on custom dataset

Hi!

We are trying to use the pre-trained dales model to segment our dataset. Our dataset is similar to dales and is setup in SPT in the same way as dales. When we use the dales model to segment the dales dataset we do not have any issues.

When running inference we get the following error:

It seems like the 'data.pos' tensor is loaded correctly at first. However, when we run the script the tensor 'coords' is empty. As 'coords' is a result of 'data.pos'/'grid_size' we tried to change 'grid_size' as 'data.pos' seem to be read correctly.
Furthermore we tried adding our data to the dales dataset in SPT, as we thought we might have made an error when setting up our own dataset. This did not solve the issue.
It seems as the issue is related to the dataset, but we are having issues locating the error. Do you have any suggestions?

Once again thank you for the great project! 😃

Best regards
Jeppe

Problem running train.py

Hello Professor ！
Thank you for your excellent work! When I was running train.py, the following error occurred

Traceback (most recent call last):
  File "/home/zhaojing/下载/pycharm-community-2022.2.4/plugins/python-ce/helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/zhaojing/下载/pycharm-community-2022.2.4/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/zhaojing/code/superpoint_transformer-master/src/train.py", line 3, in <module>
    root = str(pyrootutils.setup_root(
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pyrootutils/pyrootutils.py", line 151, in setup_root
    path = find_root(search_from, indicator)
  File "/home/zhaojing/anaconda3/envs/spt/lib/python3.8/site-packages/pyrootutils/pyrootutils.py", line 73, in find_root
    raise FileNotFoundError(f"Project root directory not found. Indicators: {indicator}")
FileNotFoundError: Project root directory not found. Indicators: ['.git', 'pyproject.toml']

OutOfMemoryError when run eval.py

Hi @drprojects
Thank you for your research, it is very useful.
I completed training the model on s3dis data and got the checkpoint. But when I run the command line for evaluate the model, the log shows as bellow.
I am using RTX 4090 Ti, my GPU memory is free at that time (~24 GB).
torch.cuda.torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.62 GiB (GPU 0; 23.62 GiB total capacity; 15.77 GiB already allocated; 7.08 GiB free; 15.78 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CON: CUDA out of memory. Tried to allocate 7.62 GiB (GPU 0; 23.62 GiB total capacity; 15.77 GiB already allocated; 7.08 GiB free; 15.78 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CON

nag.select().save() RuntimeError

Hi,

I use Nag.select() and I try to save it. I get this :

RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

Code :

hierarchy = nag.select(3, 0)
hierarchy.save( path_to_save_file + '.h5')

Value of hierarchy :

NAG(num_levels=4, num_points=[2690, 131, 49, 1], device=cuda:0)

Terminal output :

Any ideas ?

training not working with Dales dataset

Hi,

I've installed spt in an Ubuntu 22.04 VM on WSL2. Installation appears to have been finished successfully. Unfortunately I get errors when trying to run training with Dales. I've manually downloaded the PLY version of Dales and placed the train and test folders in the ~/superpoint_transformer/data/dales/raw folder, but the presence of the data appears to be ignored as it looks like the download is still attempted. I then get this error message:

[2023-06-17 16:14:05,526][src.datasets.base][INFO] - WARNING: You are about to download DALES data.
[2023-06-17 16:14:05,526][src.datasets.base][INFO] - Files will be organized in the following structure:
[2023-06-17 16:14:05,526][src.datasets.base][INFO] -
    /home/toby/superpoint_transformer/data/dales/
        └── raw/
            └── {train, test}/
                └── {tile_name}.ply

[2023-06-17 16:14:05,527][src.datasets.base][INFO] -
[2023-06-17 16:14:05,527][src.datasets.dales][INFO] - Please, register yourself by filling up the form at https://docs.google.com/forms/d/e/1FAIpQLSefhHMMvN0Uwjnj_vWQgYSvtFOtaoGFWsTIcRuBTnP09NHR7A/viewform?fbzx=5530674395784263977
[2023-06-17 16:14:05,527][src.datasets.dales][INFO] - ***
[2023-06-17 16:14:05,529][src.utils.utils][ERROR] -
Traceback (most recent call last):
  File "/home/toby/superpoint_transformer/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
    call._call_and_handle_interrupt(
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
    self._data_connector.prepare_data()
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
    call._call_lightning_datamodule_hook(trainer, "prepare_data")
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/home/toby/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data
    self.dataset_class(
  File "/home/toby/superpoint_transformer/src/datasets/base.py", line 193, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
    super().__init__(root, transform, pre_transform, pre_filter, log)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 94, in __init__
    self._download()
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 199, in _download
    self.download()
  File "/home/toby/superpoint_transformer/src/datasets/base.py", line 447, in download
    self.download_dataset()
  File "/home/toby/superpoint_transformer/src/datasets/dales.py", line 124, in download_dataset
    self.download_zip()
  File "/home/toby/superpoint_transformer/src/datasets/dales.py", line 143, in download_zip
    gdown.download(
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/gdown/download.py", line 161, in download
    res = sess.get(url, stream=True, verify=verify)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 575, in request
    prep = self.prepare_request(req)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 486, in prepare_request
    p.prepare(
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/models.py", line 368, in prepare
    self.prepare_url(url, params)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/models.py", line 439, in prepare_url
    raise MissingSchema(
requests.exceptions.MissingSchema: Invalid URL '1IW_3xZFGf_A05Qqmv3b1eOjtYj461KFX': No scheme supplied. Perhaps you meant https://1IW_3xZFGf_A05Qqmv3b1eOjtYj461KFX?
[2023-06-17 16:14:05,542][src.utils.utils][INFO] - Closing loggers...
[2023-06-17 16:14:05,542][src.utils.utils][INFO] - Closing wandb!
wandb: Waiting for W&B process to finish... (success).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/toby/superpoint_transformer/logs/train/runs/2023-06-17_16-13-53/wandb/offline-run-20230617_161400-3edj9ae0
wandb: Find logs at: ./logs/train/runs/2023-06-17_16-13-53/wandb/offline-run-20230617_161400-3edj9ae0/logs
Error executing job with overrides: ['experiment=dales']
Traceback (most recent call last):
  File "src/train.py", line 139, in main
    metric_dict, _ = train(cfg)
  File "/home/toby/superpoint_transformer/src/utils/utils.py", line 48, in wrap
    raise ex
  File "/home/toby/superpoint_transformer/src/utils/utils.py", line 45, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "src/train.py", line 114, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
    call._call_and_handle_interrupt(
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in _run
    self._data_connector.prepare_data()
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 94, in prepare_data
    call._call_lightning_datamodule_hook(trainer, "prepare_data")
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 160, in _call_lightning_datamodule_hook
    return fn(*args, **kwargs)
  File "/home/toby/superpoint_transformer/src/datamodules/base.py", line 144, in prepare_data
    self.dataset_class(
  File "/home/toby/superpoint_transformer/src/datasets/base.py", line 193, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 57, in __init__
    super().__init__(root, transform, pre_transform, pre_filter, log)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 94, in __init__
    self._download()
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 199, in _download
    self.download()
  File "/home/toby/superpoint_transformer/src/datasets/base.py", line 447, in download
    self.download_dataset()
  File "/home/toby/superpoint_transformer/src/datasets/dales.py", line 124, in download_dataset
    self.download_zip()
  File "/home/toby/superpoint_transformer/src/datasets/dales.py", line 143, in download_zip
    gdown.download(
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/gdown/download.py", line 161, in download
    res = sess.get(url, stream=True, verify=verify)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 575, in request
    prep = self.prepare_request(req)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/sessions.py", line 486, in prepare_request
    p.prepare(
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/models.py", line 368, in prepare
    self.prepare_url(url, params)
  File "/home/toby/miniconda3/envs/spt/lib/python3.8/site-packages/requests/models.py", line 439, in prepare_url
    raise MissingSchema(
requests.exceptions.MissingSchema: Invalid URL '1IW_3xZFGf_A05Qqmv3b1eOjtYj461KFX': No scheme supplied. Perhaps you meant https://1IW_3xZFGf_A05Qqmv3b1eOjtYj461KFX?

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Additionally, I think it would be very nice and useful if there was a dataset class that could operate on LAS/LAZ files.

drprojects / superpoint_transformer Goto Github PK

superpoint_transformer's Introduction

Superpoint Transformer

📌 Description

Superpoint Transformer

SuperCluster

📰 Updates

💻 Environment requirements

🏗 Installation

🔩 Project structure

🚀 Usage

Datasets

Evaluation

Training

PyTorch Lightning predict()

Full-resolution predictions

Parameterizing SuperCluster graph clustering

Notebooks & visualization

📚 Documentation

👩‍🔧 Troubleshooting

SPT or SuperCluster on an 11G-GPU

CUDA Out-Of-Memory Errors

💳 Credits

Citing our work

superpoint_transformer's People

Contributors

Stargazers

Watchers

Forkers

superpoint_transformer's Issues

👉 Regarding Errors and Warnings

👉 Regarding Understanding the Configuration

Recommend Projects

Recommend Topics

Recommend Org

PyTorch Lightning `predict()`