tranluan / nonlinear_face_3dmm Goto Github PK
View Code? Open in Web Editor NEWSource code for "Nonlinear 3D Face Morphable Model"
Home Page: http://cvlab.cse.msu.edu/project-nonlinear-3dmm.html
License: Apache License 2.0
Source code for "Nonlinear 3D Face Morphable Model"
Home Page: http://cvlab.cse.msu.edu/project-nonlinear-3dmm.html
License: Apache License 2.0
Greetings, I wish you a good day. I have managed to train the network after many attempts but now I faced the challenge of testing it and apparently DCGAN does not have the test function, could you please provide this function?
Thanks to the great work of @tranluan ! I am reading the paper named "Towards High-fidelity Nonlinear 3D Face Morphable Model", and interested in the Section 4.5 Facial Editing
.
More specifically, I want to add stickers to human faces, and the illustration in Sec 4.5 is impressive. However, the detail of this experiment in this paper is lost. So can you give me some details of this experiment (code better) or some related literature? I am a complete beginner. Thanks a lot! :)
Hi,
It seems that perceptual loss has not been calculated in the code. Right? Would you please share that missing part of the code.
Thanks.
Thank you for sharing your code. But I have a n error: cuda_op_kernel_v2_sz224.cu.o: No such file or directory. Would you help me? Thank you
what's meaning?the loss is reduce 10 times or iterations is 10 times?
Hi, I really like this work. I just wonder whether you have the plan to reimplement it using Pytorch? Or release the pytorch pre-trained model?
the function warp_texture()
in rendering_ops.py could be better written to process the whole batch instead of using the for loop.
Thanks for your great work,I have a new 3D Face Morphable Model ,and want to map it to UV map. Cause you only give the 2 dat file ,I have no ideal how did you get them,and how did you choose a1,a2 b1,b2 ,the constant scale and translation scalars?
Could you please share the generation code?
@tranluan when I try to switch the L loss, but I encounter the error.
'DCGAN' object has no attribute 'test'
I just took my first step in scientific research focused on 3D face reconstruction. .Specific module details about the environment configuration were not found in the article and code.So could you please provide the details of the environment configuration of the article if possible(including python, tensorflow, cuda, cudnn ubantu version, etc.)? I would be very appreciated if receiving the reply.Thank you very much !
Hi, this project is a great work with fine details.
But here is a little problem, when I tried the shade generation function, the loaded predefined barycoord is zero everywhere actually.
Nonlinear_Face_3DMM/_3dmm_utils.py
Lines 122 to 130 in 1c0c7d3
Which means the shade generation function below is actually makes non-sense. The generated shade is the same every in the texture.
Nonlinear_Face_3DMM/model_non_linear_3DMM.py
Line 158 in 1c0c7d3
So, it is designed like this or the file '3DMM_tri_2d_barycoord.dat' is broken?
Hi, can you tell me the L1 loss of texture i can get after the pretrain step? Thks.
I found that some values of the UV map you provided were less than 0 or more than 224, so that there would be mistakes in sampling shape from shape uv?
I try to train the net, but I can't get good result. I use uv map is from prnet https://github.com/YadiraF/PRNet。 Looking forward release train code.
Hi,
I'm trying to run the main_non_linear_3DMM.py and it gives me this error. "InternalError: Failed to create session."
Any advice would be highly appreciated.
I just took my first step in scientific research focused on 3D face reconstruction. Specific module details about the environment configuration were not found in the article and code.So could you please send me the details of the environment configuration of the article if possible(including python, tensorflow, cuda, cudnn ubantu version, etc.)? I would be very appreciated if receiving the reply.Thank you very much !
Hello, I wish you a good day. The problem I have happens just when I get to the "Train" function, in the update part of G.
To be exact on line 487 in model_non_linear_3DMM.py, what happens is the following:
-The program stops on that line for a few minutes
-During that time, my entire laptop suffers a lot of delay, even the mouse takes time to respond.
-Finally the program ends and "terminated (killed)" appears.
I think it is because G is too big for my laptop, since I only gave 30 GB to the partition where Ubuntu resides (Here where I run the program). I would like to know if the solution is simply to give more memory to my Ubuntu or my laptop is unable to run the program, I have a Geforce MX150 (Similar to a GeForces 1030), it is the only graphics card I have. I appreciate any help, I am new to this.
Hey!
I have an issue about the reconstruction loss in
Nonlinear_Face_3DMM/model_non_linear_3DMM.py
Line 198 in 1c0c7d3
Nonlinear_Face_3DMM/model_non_linear_3DMM.py
Line 164 in 1c0c7d3
I got the above error When i was compiling. Can anyone help me?
Hi, it's a fantastic job, but after the pre-training model converges, the training model with reconstruction loss and perceptual loss can not converge. Is there anything else I need to pay attention to?
Hi,
@tranluan the new commits posted last week. I tried running the code and I'm getting some variables that aren't defined. Do you know what edits we need to make to fix it?
Input: python main_non_linear_3DMM.py
Output
{'batch_size': 64,
'beta1': 0.5,
'c_dim': 3,
'checkpoint_dir': 'checkpoint',
'dataset': 'celebA',
'df_dim': 32,
'dfc_dim': 512,
'epoch': 1000,
'gf_dim': 32,
'gfc_dim': 512,
'gpu': '0',
'h': False,
'help': False,
'helpfull': False,
'helpshort': False,
'image_size': 108,
'is_batchwise_white_shading': False,
'is_const_albedo': False,
'is_const_local_albedo': False,
'is_crop': False,
'is_gt_m': False,
'is_partbase_albedo': False,
'is_pretrain': False,
'is_reduce': False,
'is_smoothness': False,
'is_train': True,
'is_using_frecon': False,
'is_using_graddiff': False,
'is_using_landmark': False,
'is_using_linear': False,
'is_using_recon': False,
'is_using_symetry': False,
'is_with_y': True,
'learning_rate': 0.0002,
'output_size': 224,
'sample_size': 64,
'samples_dir': 'samples',
'train_size': 5000000,
'visualize': False,
'z_dim': 50}
2019-07-22 11:46:36.925753: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-22 11:46:37.039450: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 11:46:37.040339: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a2dbb0 executing computations on platform CUDA. Devices:
2019-07-22 11:46:37.040355: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
2019-07-22 11:46:37.062342: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2208000000 Hz
2019-07-22 11:46:37.062777: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a97540 executing computations on platform Host. Devices:
2019-07-22 11:46:37.062796: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-07-22 11:46:37.062964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 5.77GiB freeMemory: 4.55GiB
2019-07-22 11:46:37.062980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-22 11:46:37.065177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-22 11:46:37.065190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-07-22 11:46:37.065199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-07-22 11:46:37.065300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4723 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading ./3DMM_definition/3DMM_shape_basis.dat ...
DONE
Loading ./3DMM_definition/3DMM_exp_basis.dat ...
DONE
Making make_parallel for 1 gpu(s)
WARNING:tensorflow:From /home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[64, 102]
[64, 102]
[64, 256]
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:128: conv2d_transpose (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d_transpose instead.
[64, 256]
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/rendering_ops.py:544: calling l2_normalize (from tensorflow.python.ops.nn_impl) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
sh.get_shape()
(64, 43008, 1, 9)
L1.get_shape()
(64, 43008, 9, 1)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading 3DMM keypoints ...
Loading 3DMM keypoints ...
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:222: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
Loading AFW ...
DONE. Finish loading AFW with 5207 images
Loading AFW_Flip ...
DONE. Finish loading AFW_Flip with 5207 images
Loading HELEN ...
DONE. Finish loading HELEN with 37676 images
Loading HELEN_Flip ...
DONE. Finish loading HELEN_Flip with 37676 images
Loading IBUG ...
DONE. Finish loading IBUG with 1786 images
Loading IBUG_Flip ...
DONE. Finish loading IBUG_Flip with 1786 images
Loading LFPW ...
DONE. Finish loading LFPW with 16556 images
Loading LFPW_Flip ...
DONE. Finish loading LFPW_Flip with 16556 images
Traceback (most recent call last):
File "main_non_linear_3DMM.py", line 87, in
tf.app.run()
File "/home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "main_non_linear_3DMM.py", line 70, in main
dcgan.train(FLAGS)
File "/home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/model_non_linear_3DMM.py", line 405, in train
print("Valid images %d/%d" % ( len(valid_300W_idx), self.images_300W.shape[0] ))
NameError: global name 'valid_300W_idx' is not defined
I just took my first step in scientific research focused on 3D face reconstruction. Specific module details about the environment configuration were not found in the article and code.So could you please send me the details of the environment configuration of the article if possible(including python, tensorflow, cuda, cudnn ubantu version, etc.)? I would be very appreciated if receiving the reply.Thank you very much !
Hello,
I tried to train the model. However I got a few issues. @tranluan can you help me to resolve this? Thanks in advance.
RT
training command:
python3 tridmm/main_non_linear_3DMM.py --batch_size 16 --sample_size 128 --learning_rate 0.001 --ouput_size 224 --gf_dim 32 --df_dim 32 --dfc_dim 320 --gfc_dim 320 --z_dim 20 --c_dim 3 --is_using_landmark True --shape_loss l2 --tex_loss l1 --is_using_recon True --is_using_frecon True --is_partbase_albedo True --is_using_symetry True --is_albedo_supervision True --is_batchwise_white_shading True --is_const_albedo True --is_const_local_albedo True --is_smoothness True --gpu 0 --checkpoint_dir ckpt/ckpt_3dmm/ --samples_dir samples/tridmm/ --is_train True
Some people mentioned the code can only run in such an environment:
ubuntu 16.04
tensorflow-gpu 1.3.0
cuda 8.0,cudnn 6.0
python3.5
Can the author give the TRUE configuration environment?
In line 138 of TF_newop/cuda_op_kernel_v2_sz224.cu.cc we have to set block_count and thread_per_block to 1 ZbufferTriKernel<<<1, 1>>>(s2d, tri,...)
since it need access (read/write) into the zbuffer
.
Implementing some sort of critical section for the block from line 102 - 106 could fix it.
This can lead to a significant speed up of this operation.
I just took my first step in scientific research focused on 3D face reconstruction. Specific module details about the environment configuration were not found in the article and code.So could you please send me the details of the environment configuration of the article if possible(including python, tensorflow, cuda, cudnn ubantu version, etc.)? I would be very appreciated if receiving the reply.Thank you very much !
I can't run rendering_example.py successfully, I get the following error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'ZbufferTriV2Sz224' used by node ZbufferTriV2Sz224 (defined at :82) with these attrs: []
Registered devices: [CPU, XLA_CPU, XLA_GPU]
Registered kernels:
device='GPU'
I see that others came across the same issue, but a clear solution does not exist.
My system configuration is:
Is there a system configuration that is suggested to run this code?
Thanks
tri_map_2d, mask_i = ZBuffer_Rendering_CUDA_op_v2_sz224(vertex2d_i, tri_const, visible_tri_i)
the mask_i is 224*224 , shape is (-1,);
masks.append(mask_i)
masks = tf.stack(masks)
return images, masks
where the mask shape is wrong ,that
G_images_300W_mask = tf.multiply(input_masks_300W, tf.expand_dims(G_images_300W_mask, -1))
G_images_300W = tf.multiply(G_images_300W, G_images_300W_mask) + tf.multiply(input_images_300W, 1 - G_images_300W_mask)
I'm not familiar with CUDA, if someone have rewritten cuda_op_kernel_v2_sz224 using python/tensorflow/, thanks a lot
get textures and masks from original 30w_LP data,
Hi @tranluan, your paper looks very interesting, and I want to try the released source code. But I found that the implementation of the decoder and encoder function was left empty, e.g. GCGAN.generator_encoder and GCGAN.generator_decoder_shape. Could you help resolve this?
While reading your great paper, i am bit confused with the input/output of the FC layer which is in-between Encoder and Decoder.
Please make me correct if i get it wrong.
Thanks in advance for your help to make my understanding clear.
Hi, firstly - really interesting article!
I'm receiving the same issue as @yyssmm . I'm currently trying to run this on a windows 10 system with bash installed on it. Would you have a clue on were to start looking for a potential error?
Thanks in advance.
Originally posted by @tklinterg in #4 (comment)
Hello,
I tried to train the model. However I got a few issues. During pre-training, after 200 epochs, the training converges and doesn't improve. The rendered image doesn't look good, especially the pose and texture. Then I switched to unsupervised training. However after 100 epochs, the image rendered is super blurred.
I'm not sure if the configuration in the code is the default setting you use. Thus, could you share your configuration for training, e.g.
pre-training:
training
Thanks a lot
Hi,tranluan,
Thank you for your job.
I ran rendering_example.py, then I got a rendered_img.png and a texture.png. But where can I find the model of test data?And how can I produce test data of my own images?
I just took my first step in scientific research focused on 3D face reconstruction. Specific module details about the environment configuration were not found in the article and code.So could you please send me the details of the environment configuration of the article if possible(including python, tensorflow, cuda, cudnn ubantu version, etc.)? I would be very appreciated if receiving the reply.Thank you very much !
Hi, i meet some problems when compiling the rendering layer.
InternalError (see above for traceback): Blas xGEMMBatched launch failed : a.shape=[16,4,3], b.shape=[16,53215,4], m=3, n=53215, k=4, batch_size=16 [[Node: MatMul = BatchMatMul[T=DT_FLOAT, adj_x=true, adj_y=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](concat, concat_1)]] [[Node: Reshape_43/_147 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1301_Reshape_43", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
I use tensorflow1.8, cuda9.0, g++4.8.
Thanks in advance.
Excuse me, first of all congratulate him on his work which is really interesting. The problem I have is because I want to run the code but when analyzing the ".sh" that you provide, you used TF 1.8. Is there a way to compile it with TF 2.0 ?, some modification or maybe do the compilation "manually"
Hi Luan, I played around with the code you released. And I found when I changed the output_size from 224, the default one, to 512, and run the program rendering_example.py, it constantly throws this error:
InvalidArgumentError: Incompatible shapes: [262144] vs. [50176]
The error seems from line 157 of rendering_ops.file. Can you please help have a look? Thanks!
anyone knows??
Hello Trabluan,
Thank you for your great work.
When I run your code(I use python 3.6), I change the code in opts.py:
out_split.append(fn(**{k : v[i] for k, v in in_splits.items()}))
is replaced by:
dict_tem = {}
for k,v in in_splits.items():
if(type(v)==list):
dict_tem[k] = v[i]
else:#'zip'
if i == 0:
temp_zip[k] = list(v)
dict_tem[k] = temp_zip[k][i]
else:
dict_tem[k] = temp_zip[k][i]
out_split.append(fn(**dict_tem))
Because I got the error when I used the original code: TypeError: 'zip' object is not subscriptable
I can run the 'main' file with only one GPU, but when I set 'flags.DEFINE_string("gpu", "0,1", "GPU to use [0]")' , the computer will be shut down when the program was run.
The initial settings are below:
flags.DEFINE_integer("epoch", 25, "Epoch to train [25]")
flags.DEFINE_float("learning_rate", 0.001, "Learning rate of for adam [0.0002]")
flags.DEFINE_float("beta1", 0.5, "Momentum term of adam [0.5]")
flags.DEFINE_integer("train_size", 5000000, "The size of train images [np.inf]")
flags.DEFINE_integer("batch_size", 16, "The size of batch images [64]")
flags.DEFINE_integer("sample_size", 64, "The size of batch samples images [64]")
flags.DEFINE_integer("image_size", 108, "The size of image to use (will be center cropped) [108]")
flags.DEFINE_integer("output_size", 224, "The size of the output images to produce [64]")
flags.DEFINE_integer("c_dim", 3, "Dimension of image color. [3]")
flags.DEFINE_boolean("is_with_y", True, "True for with lable")
flags.DEFINE_string("dataset", "celebA", "The name of dataset [celebA, mnist, lsun]")
flags.DEFINE_string("checkpoint_dir", "checkpoint", "Directory name to save the checkpoints [checkpoint]")
flags.DEFINE_string("samples_dir", "samples", "Directory name to save the image samples [samples]")
flags.DEFINE_boolean("is_train", True, "True for training, False for testing [False]")
flags.DEFINE_boolean("is_reduce", False, "True for 6k verteices, False for 50k vertices")
flags.DEFINE_boolean("is_crop", False, "True for training, False for testing [False]")
flags.DEFINE_boolean("visualize", False, "True for visualizing, False for nothing [False]")
flags.DEFINE_integer("gf_dim", 32, "")
flags.DEFINE_integer("gfc_dim", 320, "")
flags.DEFINE_integer("df_dim", 32, "")
flags.DEFINE_integer("dfc_dim", 320, "")
flags.DEFINE_integer("z_dim", 20, "")
flags.DEFINE_string("gpu", "0,1", "GPU to use [0]")
flags.DEFINE_boolean("is_pretrain", False, "Is in pretrain stage [False]")
flags.DEFINE_boolean("is_using_landmark", False, "Using landmark loss [False]")
flags.DEFINE_boolean("is_using_symetry", True, "Using symetry loss [False]")
flags.DEFINE_boolean("is_using_recon", False, "Using rescontruction loss [False]")
flags.DEFINE_boolean("is_using_frecon", False, "Using feature rescontruction loss [False]")
flags.DEFINE_boolean("is_using_graddiff", False, "Using gradient difference [False]")
flags.DEFINE_boolean("is_gt_m", False, "Using gt m [False]")
flags.DEFINE_boolean("is_partbase_albedo", False, "Using part based albedo decoder [False]")
flags.DEFINE_boolean("is_using_linear", False, "Using linear model supervision [False]")
flags.DEFINE_boolean("is_batchwise_white_shading", True, "Using batchwise white shading constraint [False]")
flags.DEFINE_boolean("is_const_albedo", True, "Using batchwise const albedo constraint [False]")
flags.DEFINE_boolean("is_const_local_albedo", True, "Using batchwise const albedo constraint [False]")
flags.DEFINE_boolean("is_smoothness", True, "Using pairwise loss [False]")
I think it was not cpu or memery's problem. Could you please help to run this code with multiple GPUs.
Python3.5 & Tensorflow1.12
Thanks!!
@tranluan
Is G_loss_frecon missing? I can't find the relevant code in this model_nonlinear_3DMM.py
can you share the code to me?
Thank you very much!
hi, could you tell me where are the m(matrix data) and shape datas come from ?
acording the paper says "With the face profiling technique, Zhu et al. [33] expand the
300W dataset [68] into 122, 450 images with fitted 3DMM shapes and projection parameters m."
but i can't find it, could you give me the accuracy network link?
Hi,
@tranluan I want to test out the model on some images and see the results (3D mesh, rendered image....)
How can I try ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.