eugenelet / meta-rppg Goto Github PK

Implementation of "Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner"

License: MIT License

Python 100.00%

meta-rppg's Introduction

Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner

This repository is the official implementation of Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner that has been accepted to ECCV 2020.

Heatmap Visualization

Left to right:

Cropped input image
End-to-end trained model (baseline)
Meta-rPPG (transducive inference)
Top to down: rPPG signal, Power Spectral Density (PSD), Predicted and ground truth heart rate

Requirements

To install requirements:

pip install -r requirements.txt

All experiments can be run on a single NVIDIA GTX1080Ti GPU.

The code was tested with python3.6 the following software versions:

Software	version
cuDNN	7.6.5
Pytorch	1.5.0
CUDA	10.2

Training

Training Data Preparation

Download training data (example.pth) from Google Drive. Due to privacy issue (face images), provided data contains only a subset of the entire training data, i.e. contains faces of the authors of this paper.

Move example.pth to data/ directory:

mv example.pth data/

Begin Training

To begin training, run:

python3 train.py

Validation Data

Validation data can be requested from:

MAHNOB-HCI

UBFC-rPPG

Contributing

If you find this work useful, consider citing our work using the following bibTex:

@inproceedings{lee2020meta,
  title={Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner},
  author={Lee, Eugene and Chen, Evan and Lee, Chen-Yi},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2020}
}

meta-rppg's People

Contributors

Stargazers

Watchers

meta-rppg's Issues

Help! testing model

Can you please provide a way to test the model that has been trained using example.pth? much appreciated!

Inference pipeline

Can you provide code and weights for inference purpose only?

作者可以把example.pth的连接重新放一下吗，已经失效了

或者各位大佬谁还有example.pth文件可以传我一份吗

train on VIPL dataset

Hi @eugenelet @EvanChen022kan,
Thanks for your work, I know that BVP is the signal that results when highpass filtering the PPG signal, so can I use BVP in VIPL dataset as ground truth label in model? How do you normalize ppg label in example.pth file?

Error

thanks for codes sharing !

I installed requirements and run python train.py

but I have bellow error

RuntimeError: result type Float can't be cast to the desired output type Byte

Please advise me!

UBFC Video - In which part of the code we need to pass it ?

Hi All,

I'm working on this meta-rPPG. I got access for UBFC dataset for validation.

But I'm not sure how to pass the bunch of subject.

In the current work, author directly passed .pth format.

Please help me with it

Example.pth link not working

Custom Dataset Template file not found

Hi there,

You mentioned that there is a custom dataset template file but I did not find in here. Can you please share it?

Thank you so much!

How to predict heart rate?

Could you please tell me whether to obtain the average heart rate it is enough to pass the output of the ordinal regression through the Hanh window, then Butterworth filter and Welch density decomposition, like the get_bpm function suggests?

Meta-rPPG/data/data_utils.py

Line 127 in bab00d9

def get_bpm(Sig, rate= 30.0):

If possible, I would be grateful if you could give an intuitive understanding of what the Butterworth bandpass does, since you use it both in pre-processing and post-processing.

"Algorithm 1 Training of Meta-Learner"

Where in the code occurs the division into frames V and W mentioned in the article in "Algorithm 1 Training of Meta-Learner".

What is "L steps" in the code?

Was "fewshot_test" supposed to be adaptation steps mentioned in the article?

Pretrained model

Hi,
Thank you for releasing the code for Meta-rPPG, very interesting paper indeed.
However, It would be really helpful if you could provided access to the custom training data-set or if privacy is a concern you could maybe upload the pretrained model.

Thank you.

PreProcessing Issue

It is calling baseline process function, after checking baseline process it seems you are doing mean subtraction, mask multiplication and butter processing, but when I checked the output It is actually only doing mean subtraction and not mask multiplication and butter etc. Is it intentional or is it a bug in code ? Thanks

self.baseline_procress(inputs, masks.clone())
ppg = self.quantify(ppg)

learning phase

Thanks for releasing the code for the meta-rPPG paper! Very interesting paper!

I am wondering where is the code for the learning phase described in the algrotihm 1 (Training of Meta-Learner)? Is it 'def fewshot_learning' whic is currently unifished? If so, does the trainng pipeline work without it?

Thanks!

This model is very difficult to train. What training skills are there？

Thanks a lot for releasing the code of Meta-rPPG paper!
We use the Deap dataset train Meta-rPPG model, but the model is difficult to converge. Do you have any training skills?
Thaks a lot!

What should be the input for the method set_input_for_test?

Based on the example.pth, I thought it should be a tensor with size (1, 60, 3, 64, 64) representing the input returned by the test SlideWindowDataLoader, but when I try to use this method with model.test(), it gives me the following error: "Expected hidden[0] size (4, 1, 60), got (4, 3, 60)". So I would like to know what should be the shape. Also, if possible, could tell me the diference between model.fewshot_test() and model.test()?

Duplicate few-shot updates of the encoder

Thanks so much for releasing the code for the great paper!

At the moment the encoder seems to get updated twice during validation, judging from the following two loops:

Meta-rPPG/model/main_model.py

Line 367 in bab00d9

for i in range(self.opt.fewshots):

Meta-rPPG/model/main_model.py

Line 376 in bab00d9

for i in range(self.opt.fewshots):

Is it a bug? Am I missing anything?

about AttentionNet

Hi, could you share the AttentionNet with me? I want to learn how your net design~

RuntimeError: result type Float can't be cast to the desired output type Byte

(py370) C:\Users\bunny\Desktop\Meta-rPPG>py train.py
----------- Networks initialized -------------
Total number of parameters : 0.380 M
---------------------end----------------------
dataset [rPPGDataset-train] was created
dataset [rPPGDataset-test] was created
Data Size: 650 ||||| Batch Size: 3 ||||| initial lr: 0.001000
Traceback (most recent call last):
  File "C:\Users\bunny\Desktop\Meta-rPPG\train.py", line 38, in <module>
    data = dataset[task_list, 0]
  File "C:\Users\bunny\Desktop\Meta-rPPG\data\dataload.py", line 68, in __getitem__
    dat = self.dataset[idx, items[1]]
  File "C:\Users\bunny\Desktop\Meta-rPPG\data\pre_dataload.py", line 97, in __getitem__
    self.baseline_procress(inputs, masks.clone())
  File "C:\Users\bunny\Desktop\Meta-rPPG\data\pre_dataload.py", line 120, in baseline_procress
    mask /= 255
RuntimeError: result type Float can't be cast to the desired output type Byte

Excuse me, Could you upload "test" code?

Real time on webcam?

Is the model capable of running real time on a webcam?

Where do the benchmarks for other methods on UBFC come from?

In the paper there is a comparison of Meta-rPPG with other methods, but not all of the referenced authors list MAE, RMSE and r on UBFC. Could you please tell me whether you have trained these models on your own data as well?

How to produce example.pth?

Thank you for provide the source code, I want to training the model with some other data, the training and testing dataset, exmple.pth, how to produce this file?

Why does the test set used in the example contain a video fragment of a subject that appears in the training?

Does this have to do with the partition of the dataset of query set and support set or was it an error that occurred when generating the exemple.pth?

If it is not an error then must the testset used in fewshot method must contain video fragments of the subjects contained in the training?

Thanks

Pretrained

Can you upload a pre-trained model?

Thanks a lot for releasing the code of Meta-rPPG paper!
I understand that due to privacy issue, it is hard to get your training dataset. But can you provide a pre-trained model? This will help me a lot!