Nerfstudio to MLflow: Challenges in Instantiating Nerfacto Models about nerfstudio HOT 2 CLOSED

These are both dataset-dependent.

How they're computed in the typical Nerfstudio pipeline might be helpful:

If you want to instantiate the model without the full pipeline, perhaps you could save these values at the start of training?

For example, you could add a few lines below where the dataparser_transforms.json is saved:

from nerfstudio.

Hi @brentyi ,

Thank you for your prompt and insightful response! Your suggestion about saving the necessary arguments at the beginning of training seems like a viable approach to me. However, I have a few concerns regarding its implementation:

1. I'm currently utilizing the Docker image of Nerfstudio, and while cloning/forking and modifying the Nerfstudio repository is feasible, it's not the most convenient option for me.
2. Considering the cost of software development, both in terms of initial feature development and ongoing maintenance, keeping a private copy of this repository with modified file nerfstudio/engine/trainer.py might be challenging. This is especially true if Nerfstudio undergoes significant updates later, and it could also pose difficulties in sharing my work with others.

Given these considerations, I’m leaning towards alternative strategies that might better suit a collaborative engineering environment. Here are my thoughts:

1. Integration with MLflow's Experiment Tracking: Instead of utilizing MLflow’s model registry, I'm considering integrating Nerfstudio with MLflow’s experiment tracking feature. This would involve wrapping the entire Nerfstudio pipeline in an MLflow experiment run and logging the complete training dataset and outputs directory as artifacts of MLflow experiment run. Organizing and tagging runs, rather than models in the model registry, could be a more effective strategy.
2. Potential Pull Request: Another option could be for either of us to create a Pull Request to implement your suggested change, saving the required arguments in the outputs directory at the start of training. However, I'm concerned about whether the feature of allowing users to instantiate a trained model outside of a pipeline would be broadly useful or if it might lead to unintended complexities in the repository.
3. PyTorch Lightning Integration: A more labor-intensive solution could involve rewriting the PyTorch models in the repository to use PyTorch Lightning trainers (here is a link to the PyTorch Lightning docs in case you are unfamiliar with this tool). This would allow users to leverage MLflow’s auto-logging feature for PyTorch Lightning, but it represents a significant overhaul of the current system.

For now, I plan to proceed with the first option and integrate Nerfstudio with MLflow’s experiment tracking. I'll be closing this issue, but your feedback has been invaluable. Please feel free to share any further thoughts you have on this matter. You're also welcome to reopen this issue and tag me if you’d like to discuss the possibility of a Pull Request in more detail.

Thanks again for your valuable input! 🙏 👏

from nerfstudio.