Project page | Paper | Video
This repo contains the official implementation for the paper "BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation".
Pytorch3d is required for postprocess. Come to offical page for installation instructions.
We modified diffusers to adapt triplane structure. Users should build based on the ./src
# clone this repo
python setup.py install
Pretrained weights of MLP, VAE, and Diffuser(Condition/Unconditioned). Download the model here [10GB]
and extract to ./checkpoints.
To do uncondtional inference, run
python unconditioned_prediction.py --batch 4 --save output/uncond
To do single block conditional inference, run
python conditioned_prediction.py --layout samplelayouts/exp1_32-56-24/0_0.npy --save output/cond
We provide some sample layouts for demo. If you would like to draw your own layout maps, you can use drawtkinter.py to create your own layout. Here is a tutorial:
The unit of the graduations are in meters, furnitures need to be in reasonable scale. Walls should surround the floor and furniture should be placed on the floor.
The output layouts directory named after expname_xscale-zscale-stride, xscale-zscale-stride are given in decimeter.
Usage:
python drawtkinter.py --h 56 --w 56 --expname draw
Full pipeline(conditional prediction + extrapolation + resample) is contained in fullpipeline.py. Before running, specifying the layout directory.
python fullpipeline.py --layout samplelayouts/exp2_32-56-24 --resample 15 --save output
If you find our code or paper helps, please consider citing:
@article{blockfusion,
title={BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation},
author={Wu, Zhennan and Li, Yang and Yan, Han and Shang, Taizhang and Sun, Weixuan and Wang, Senbo and Cui, Ruikai and Liu, Weizhe and Sato, Hiroyuki and Li, Hongdong and Ji, Pan},
journal={ACM Transactions on Graphics},
volume={43},
number={4},
year={2024},
doi={10.1145/3658188}
}
blockfusion's People
Contributors
Stargazers
Watchers
blockfusion's Issues
Any practical installation guide?
Would you mind offer a enviroment version list you are using?
I've tried py39, pytorch 2.3.1 + cu118, build pytorch3d from source code, and build the custom diffusers according to your setup.py script.
And during executing the command: python unconditioned_prediction.py --batch 4 --save output/uncond ,got a lot of errors about missing modules like open3d, einops, xformers, etc... after I pip install them each by each. Still got this error:
undefined symbol: _ZN3c1015SmallVectorBaseIjE8grow_podEPvmm
It could be painful to guess the correct runtime enviroment...
FYI, I thinks its a matter of xformers version
how to visualize the model output ?
I run fullpipeline.py on Linux and obtained many files. How can I visualize the output results
about the grad of sdf model
hi,
thanks for your great work.
Is it convenient for you to share the code for calculating the SDF gradient? I have implemented the Eq.6 for calculating gradients in your article, but the results are always incorrect
Triplane fitting MLP pretrained weight
Hello,
Thanks for the impressive work!
I’m trying to test the triplane fitting code located in the fit_triplane directory. However, it seems that the provided pretrained weights do not include the mlp.tar file. Could you provide an alternative link for the pretrained MLP weights(mlp.tar)?
Releasing training code
Thanks for the great work!
Do you have any plans to release the training codes?
link for pretrained weights
Thanks for the awesome work! The link for pretrained weights seem to be wrong, is there an alternative link?
issues with linear attention mode when running the main
Get errors below when running the main function:
constructing SpatialTransformer_ of depth 1 w/ 512 channels and 16 heads
Attention mode 'linear' is not available. Falling back to native attention. This is not a problem in Pytorch >= 2.0. FYI, you are running with PyTorch version 2.4.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.