vilab-ucsd / photoscene Goto Github PK

[CVPR2022] Official implementation of PhotoScene, a framework to transfer photorealistic material and lighting for indoor scene from photo.

License: MIT License

Shell 1.82% Python 98.18%

photoscene's Introduction

PhotoScene

The official implementation of PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes.
Yu-Ying Yeh, Zhengqin Li, Yannick Hold-Geoffroy, Rui Zhu, Zexiang Xu, Miloš Hašan, Kalyan Sunkavalli, Manmohan Chandraker
IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022

[Paper] [Project]

Prerequisite

This repo is tested on RTX2080Ti GPU. There are reports showing that it might not work on newer GPUs.
Clone this repository including submodules

git clone --recursive https://github.com/ViLab-UCSD/PhotoScene.git
cd PhotoScene

Install OptiX 5.1.1 under third_party/OptiX. We assume OptiX install directory is third_party/OptiX/NVIDIA-OptiX-SDK-5.1.1-linux64. Note that the version should be 5.1.1 upon which our renderer is built.
Download Total3D, MaskFormer, InvRenderNet pretrained model by running the following script.

bash scripts/download_model.sh

Set up Substance Designer

Install Substance Designer with Linux (.rpm) version 2021.1.2

Get free license here if you're a student or teacher. Download Substance Designer version 11.1.2 (437M) (LICENSES->Substance Designer All builds->Linux (.rpm) version 2021.1.2) and run the following to install. The default installation path will be /opt/Allegorithmic/Substance_Designer.

sudo apt-get update
sudo apt-get install alien
sudo alien -i Substance_Designer-11.1.2-4593-linux-x64-standard.rpm

Note: It is a requirement for DiffMat

Requirement

Install Docker

Our framework consists of multiple dependencies. We strongly recommend using our provided docker image to run this repository. The provided script assumes running docker in rootless mode with a single GPU. Might need to modify /etc/nvidia-container-runtime/config.toml for the entry under [nvidia-container-cli] to no-cgroups = true to disable the use of cgroups by the NVIDIA container runtime. After installation, run the following to pull our image and launch the environment.

docker pull yyeh/photoscene:v1
bash scripts/run_docker_interactive.sh

Build OptixRenderer

Please make sure OptiX 5.1.1 is installed under third_party/OptiX before running the following script. Note that from this step all the commands must be run inside the docker environment.

bash scripts/build_renderer.sh

Run PhotoScene

Now we can run the full PhotoScene framework on a scene specified by $yamlFile.

bash run_photoscene.sh $yamlFile

For example, $yamlFile is set default as configs/total3d/Total_246.yaml. Alternatively, you can run each single step sequentially:

# Initialization and Alignment
python3 photoscene/preprocess.py --config $yamlFile

# Graph Selection
python3 photoscene/selectGraphFromCls.py --config $yamlFile

# First Round Material Optimization
python3 photoscene/optimizeMaterial.py --config $yamlFile --mode first

# Lighting Optimization
python3 photoscene/optimizeLight.py --config $yamlFile

# Second Round Material Optimization
python3 photoscene/optimizeMaterial.py --config $yamlFile --mode second

# Render Final PhotoScene Result
python3 photoscene/renderPhotoScene.py --config $yamlFile

Additional Input Data

You can first use provided Total3D examples to run the entire framework. If you want to try more scenes, please download Total3D preprocessed data by running the following script or following Total3D to download preprocessed SUN RGBD data (12G) to third_party/Total3D/data/sunrgbd/sunrgbd_train_test_data and copy the scenes (<scene_id>.pkl) to data/total3d/inputs.

bash scripts/download_total3d_data.sh

Citation

Please cite our paper if you find that our method is helpful!

@InProceedings{Yeh_2022_CVPR,
    author    = {Yeh, Yu-Ying and Li, Zhengqin and Hold-Geoffroy, Yannick and Zhu, Rui and Xu, Zexiang and Ha\v{s}an, Milo\v{s} and Sunkavalli, Kalyan and Chandraker, Manmohan},
    title     = {PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {18562-18571}
}

photoscene's People

Contributors

Stargazers

Watchers

Forkers

jackzhousz ericustc youngxiao13 peterzs slayyden jasonsongpeng

photoscene's Issues

Docker GPU problem

Hello

I am having problems running GPU from docker as follows:

$ bash scripts/run_docker_interactive.sh

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown.

I followed many online workarounds, but nothing worked so far. Last attempt was following this link https://discuss.linuxcontainers.org/t/docker-with-gpu-support-in-unprivileged-lxd-container/5783/2 :

`after the docker starts, map GPU buses /proc/driver/nvidia/gpus/xxxxxx from the host to the LXD container. The xxxxx pci buses should be those of the GPUs passed through from LXD to docker. The needed pci buses can be identified with nvidia-smi.

I installed lxc, but I do not know how to map GPU to busto LXD container, and not sure if your project uses LXD container or is useful at all.

Any help is very appreciated,

Thank you,

Manal

Renderer issue: pure virtual method called

Thanks for your excellent work @yuyingyeh. I have some trouble with the renderer.
My GPU is RTX3090 with cuda 11.1 and I have successfully compiled the renderer. (I change sm_30 to sm_64 in the cmake file to avoid compile error.)
The cmd I use: python3 photoscene/preprocess.py --config configs/total3d/Total_246.yaml
The core dumped issue I met:

I have locate the code which leads to this issue: (in photoscene/utils/render.py renderPass Function)

So I print the cmd:
third_party/OptiX_Renderer/OptixRenderer_MatPart/src/bin/optixRenderer -f /mnt/data2/dangziqiang2/ProceduralMaterial/PhotoScene/outputs/total3d/scene246/preprocess/xmlInit/main.xml -c /mnt/data2/dangziqiang2/ProceduralMaterial/PhotoScene/outputs/total3d/scene246/preprocess/cameras/cam_all.txt -o /mnt/data2/dangziqiang2/ProceduralMaterial/PhotoScene/outputs/total3d/scene246/preprocess/renderInit/partId/im.hdr -m 7 --modelIdFile /mnt/data2/dangziqiang2/ProceduralMaterial/PhotoScene/outputs/total3d/scene246/preprocess/meshes/models.txt --seed 0 > /mnt/data2/dangziqiang2/ProceduralMaterial/PhotoScene/logs/renderPartId.txt

Please help me, I don't know where I went wrong. Thanks very much for your reply!

Inquiry Regarding Missing "layout.mat" File in the Codebase

Hello, I've been encountering an issue while running the code, specifically a missing "layout.mat" file. Despite searching through the file system, I couldn't locate it. In the code, it is referenced as an input, and its location is within the "outputs" folder. Does this mean that this is the output of a submodule?

I observed that during the project clone, I faced network issues, and as a result, the "third_party" submodule appears to be empty. Could this be the reason behind the submodule-related problems and the absence of the "layout. mat" file as its output?

I appreciate any insights or guidance you can provide to help resolve this issue.

Inquiry About Increasing Material Resolution Beyond 256

Nice work on the project! I'm really impressed with what you've achieved here. I have a question regarding the resolution of optimized materials. Currently, the code seems to only output materials at a 256 resolution. However, from what I observed in the teaser images, the resolution appears to be quite high. Could you guide me on how to increase the resolution of these materials beyond 256? How can the resolution of optimized materials be increased? Any tips or code modifications that would allow achieving higher-resolution outputs would be greatly appreciated. Looking forward to your response, and thanks again for your fantastic work!

How to download Substance Designer All builds

Thanks for your excellent work. I have some trouble downloading Substance Designer All builds. There is no this app in the website you offered, and I asked the helper of adobe, they still give me nothing about the downloading link.
Looking to your reply, best wishes.

And I have searched all websites of adobe, still nothing.