zqbai-jeremy / dfnrmvs Goto Github PK

View Code? Open in Web Editor NEW

167.0 11.0 26.0 31.77 MB

Source code for CVPR 2020 paper "Deep Facial Non-Rigid Multi-View Stereo"

License: GNU General Public License v3.0

Python 100.00%

dfnrmvs's Introduction

Deep Facial Non-Rigid Multi-View Stereo

Source code for CVPR 2020 paper "Deep Facial Non-Rigid Multi-View Stereo" [paper] [supp] [video].

Installation

(1) Create an Anaconda environment with python 3.6.

conda create -n DFNRMVS python=3.6
source activate DFNRMVS

(2) Clone the repository and install dependencies.

git clone https://github.com/zqbai-jeremy/DFNRMVS.git
cd DFNRMVS
conda install --yes --file requirements_conda.txt
pip install -r requirements_pip.txt

(3) Setup 3DMM

Clone this repository, which is forked and modified from YadiraF/face3d, to "<DFNRMVS directory>/external/".

mkdir external
cd external
git clone https://github.com/zqbai-jeremy/face3d.git
cd face3d

Setup face3d as in YadiraF/face3d.
Download "Exp_Pca.bin" from Guo et al. (in "CoarseData" link of their repository) and copy to "<DFNRMVS directory>/external/face3d/examples/Data/BFM/Out/".
Download "std_exp.txt" from Deng et al. and copy to "<DFNRMVS directory>/external/face3d/examples/Data/BFM/Out/".

(4) Install face-alignment.

conda install -c 1adrianb face_alignment

(5) Download pre-trained model (2views_model.pth or 3views_finetune_model.pth; May be used for research purpose only) to "<DFNRMVS directory>/net_weights/". Need to create the folder.

Run Demo

Modify directory paths in demo.py and run

cd <DFNRMVS_directory>
python demo.py

All images in the input directory will be used for reconstruction. Per-view results will be saved to the output directory.
Some examples are in "<DFNRMVS directory>/examples/". The corresponding outputs are in "<DFNRMVS directory>/out_dir/".
The model usually gives good results for 2 views input with +-30 degree yaw angles.

Training

Training requires 256x256 images with ground truth 3D scans. Loss functions and training parameters are provided in "<DFNRMVS directory>/train/losses.py"
Need to setup torch-batch-svd to use the losses.

Citation

@inproceedings{bai2020deep,
  title={Deep Facial Non-Rigid Multi-View Stereo},
  author={Bai, Ziqian and Cui, Zhaopeng and Rahim, Jamal Ahmed and Liu, Xiaoming and Tan, Ping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5850--5860},
  year={2020}
}

dfnrmvs's People

Contributors

Stargazers

Watchers

dfnrmvs's Issues

rundemo.py报错 FileNotFoundError

您好，我按readme.md 操作到run demo.py

为啥会有这个错误

FileNotFoundError: [Errno 2] No such file or directory: './external/face3d/examples/Data/BFM\Out/BFM.mat'

Code questions

Thanks for your great work. But I have one question about this line:

The ep_norm is 0 ? why

Buffer has wrong number of dimensions (expected 2, got 3)

File "y:\python\human\face\DFNRMVS\data\BFM\utils.py", line 347, in get_pixel_vert_idx_and_weights
depth_buffer, triangle_buffer, barycentric_weight = mesh.render.rasterize_triangles(vertices, triangles, h, w)
File "y:\python\human\face\DFNRMVS\external\face3d\face3d\mesh\render.py", line 50, in rasterize_triangles
h, w)
File "mesh_core_cython.pyx", line 49, in mesh_core_cython.rasterize_triangles_core
ValueError: Buffer has wrong number of dimensions (expected 2, got 3)

请问你训练用到的数据集是什么

The camera parameters

How did you get camera parameters information from the dataset？How do I apply for Stirling/ESRC 3D face database？

TypeError when transform "face_region_mask" from numpy to torch

Hi, thank you very much for your work and the code, it is very interesting.

I tried to run your demo. However, there were sth wrong while transforming "face_region_mask" from numpy to torch.

Traceback (most recent call last):
File "demo.py", line 143, in
model = init_model(checkpoint_path)
File "demo.py", line 47, in init_model
model = FNRMVSNet(opt_step_size=1e-2)
File "/home/yyan/3d_reco/DFNRMVS/networks/sub_nets.py", line 1083, in init
self.opt_layer = NRMVSOptimization(opt_step_size=opt_step_size, MM_base_dir=MM_base_dir)
File "/home/yyan/3d_reco/DFNRMVS/networks/sub_nets.py", line 558, in init
self.bfm_torch = bfm_utils.MorphabelModel_torch(self.bfm)
File "/home/yyan/3d_reco/DFNRMVS/data/BFM/utils.py", line 589, in init
self.face_region_mask = torch.from_numpy(bfm.face_region_mask).unsqueeze(0) # (1, nver, 1)
TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.

There is probably an easy hotfix. However, I am new to your code so I am not quite sure what is an appropriate type for your self.face_region_mask.
Could u please have a look? Thank you very much.

想看到Landmark数据和Multi-Level重建

您好，我在用自己的照片想复现您这篇面部重建，请问我要修改什么语句可以看到检测面部Landmark的数据，和分别看到三个不同精度Level的脸部模型呢？谢谢！

Licence

Hi. Interesting work! What is the license for your code ?

深度图寻求帮助

您好，我想使用您论文的方法生成深度图来辅助人脸活检的训练？
请问真脸和拍摄屏幕得到的假脸，重构出来的人脸深度图，会有明显差异么？
我的目的是真脸可以重构出深度图，假脸重构出来深度图得分都特别低。

Building wheel for tinyobjloader (setup.py) ... error

系统是Ubuntu16.04 ，64位。根据指示一步一步安装，
requirements_pip 安装到 tinyobjloader 时报错。请问如何解决？

ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r5_8JE/tinyobjloader/setup.py'"'"'; file='"'"'/tmp/pip-install-r5_8JE/tinyobjloader/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-v2e1H6
cwd: /tmp/pip-install-r5_8JE/tinyobjloader/
Complete output (11 lines):
running bdist_wheel
running build
running build_ext
building 'tinyobjloader' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I../ -I../pybind11/include -I/usr/include/python2.7 -c bindings.cc -o build/temp.linux-x86_64-2.7/bindings.o -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
bindings.cc:1:31: fatal error: pybind11/pybind11.h: 没有那个文件或目录
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

ERROR: Failed building wheel for tinyobjloader

Running setup.py clean for tinyobjloader
Failed to build tinyobjloader
DEPRECATION: Could not build wheels for tinyobjloader which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at pypa/pip#8368.
Installing collected packages: tinyobjloader
Running setup.py install for tinyobjloader ... error

ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r5_8JE/tinyobjloader/setup.py'"'"'; file='"'"'/tmp/pip-install-r5_8JE/tinyobjloader/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-nMzlKD/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python2.7/tinyobjloader
cwd: /tmp/pip-install-r5_8JE/tinyobjloader/
Complete output (11 lines):
running install
running build
running build_ext
building 'tinyobjloader' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I../ -I../pybind11/include -I/usr/include/python2.7 -c bindings.cc -o build/temp.linux-x86_64-2.7/bindings.o -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
bindings.cc:1:31: fatal error: pybind11/pybind11.h: 没有那个文件或目录
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r5_8JE/tinyobjloader/setup.py'"'"'; file='"'"'/tmp/pip-install-r5_8JE/tinyobjloader/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-nMzlKD/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python2.7/tinyobjloader Check the logs for full command output.

Best,
Yumi

KeyError: 'model'

/usr/local/lib/python3.7/site-packages/skimage/io/manage_plugins.py:23: UserWarning: Your installed pillow version is < 8.1.2. Several security issues (CVE-2021-27921, CVE-2021-25290, CVE-2021-25291, CVE-2021-25293, and more) have been fixed in pillow 8.1.2 or higher. We recommend to upgrade this library.
from .collection import imread_collection_wrapper
Initializing model ...
Traceback (most recent call last):
File "demo.py", line 143, in
model = init_model(checkpoint_path)
File "demo.py", line 47, in init_model
model = FNRMVSNet(opt_step_size=1e-2)
File "/content/DFNRMVS/networks/sub_nets.py", line 1083, in init
self.opt_layer = NRMVSOptimization(opt_step_size=opt_step_size, MM_base_dir=MM_base_dir)
File "/content/DFNRMVS/networks/sub_nets.py", line 553, in init
self.bfm = bfm_utils.load_3DMM(MM_base_dir)
File "/content/DFNRMVS/data/BFM/utils.py", line 69, in load_3DMM
bfm = MorphabelModel(os.path.join(MM_base_dir, 'Out/BFM.mat'))
File "/content/DFNRMVS/external/face3d/face3d/morphable_model/morphabel_model.py", line 30, in init
self.model = load.load_BFM(model_path)
File "/content/DFNRMVS/external/face3d/face3d/morphable_model/load.py", line 32, in load_BFM
model = C['model']
KeyError: 'model'

Instead of a "BFM.mat", I used a "01_MorphableModel.mat" and renamed it to a "BFM.mat". It's right?