hello,
https://folk.ntnu.no/haakohu/RetinaFace_ResNet50.pth is not available

****

Hi, I apply your creative code on my own data, but find some results of the face swap are scary. I think going back to targeted training might help. Could you teach me how to train with my own data？ thank you so much!

Can't be used, I can't anonymize the pictures in the folder using "python3 anonymize.py -s input_path -t output_path" after using the command "python3 setup.py install"

Hello, thank you for this great work.

I am trying to train retinanet512 model on my own dataset, but there aren't examples of usage of train.py in version 2.
How do I train on my own dataset in version 2 ?

I am very interested in your work and have studied it carefully.But there are some questions when I run the program in docker.I wonder if there is a problem with the docker I created.
Traceback (most recent call last):
File "/workspace/DeepPrivacy/deep_privacy/inference/infer.py", line 44, in load_model_from_checkpoint
ckpt = get_checkpoint(cfg.output_dir, validation_checkpoint_step)
File "/workspace/DeepPrivacy/deep_privacy/engine/checkpointer.py", line 48, in get_checkpoint
return load_checkpoint(path)
File "/workspace/DeepPrivacy/deep_privacy/engine/checkpointer.py", line 29, in load_checkpoint
raise FileNotFoundError(f"Did not find path: {ckpt_path}")
FileNotFoundError: Did not find path: outputs/workspace/deep_privacy_cache/fdf_512/checkpoints

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "anonymize.py", line 5, in
cli.main()
File "/workspace/DeepPrivacy/deep_privacy/cli.py", line 113, in main
return_cfg=True)
File "/workspace/DeepPrivacy/deep_privacy/build.py", line 66, in build_anonymizer
generator = load_model_from_checkpoint(cfg)
File "/workspace/DeepPrivacy/deep_privacy/inference/infer.py", line 47, in load_model_from_checkpoint
ckpt = load_checkpoint_from_url(cfg.model_url)
File "/workspace/DeepPrivacy/deep_privacy/engine/checkpointer.py", line 18, in load_checkpoint_from_url
model_url, map_location=_get_map_location())
File "/opt/conda/lib/python3.6/site-packages/torch/hub.py", line 494, in load_state_dict_from_url
raise RuntimeError('Only one file(not dir) is allowed in the zipfile')
RuntimeError: Only one file(not dir) is allowed in the zipfile

Hello, thanks for the great work !
I'am currently new to this, but im trying to use your code to anonymize face on cctv footage, do you think i should use it as it is or i should re-train your model on a cctv dataset ( using your pre-trained weights obviously ) ?
I have two other questions :

• what does those parameters mean : use_static_z , max_face_size ?
• if i want to keep the generated face of a person in the next frame (i.e. face swapping the generated face to the original face in the next frame) because that person's face didnt move much, can i do that with stitch_face method ? (if you can explain what stitch_face and replace_face methods exactly do i will be very grateful).

Thank you so much !

Is it possible to anonymize only a part of a face using bounding boxes? Using a custom im_bboxes with a high truncation level gives this traceback because z_shape is 0 in the first dimension

Traceback (most recent call last):
  File "DeepPrivacy/anonymize.py", line 5, in <module>
    cli.main()
  File "DeepPrivacy/deep_privacy/cli.py", line 142, in main
    anonymizer.anonymize_image_paths(image_paths, image_target_paths, [np.array([[250,400,500,550]])])
  File "DeepPrivacy/deep_privacy/inference/anonymizer.py", line 61, in anonymize_image_paths
    anonymized_images, image_annotations = self.detect_and_anonymize_images(
  File "DeepPrivacy/deep_privacy/inference/anonymizer.py", line 43, in detect_and_anonymize_images
    anonymized_images = self.anonymize_images(
  File "/usr/lib/python3/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "DeepPrivacy/deep_privacy/inference/deep_privacy_anonymizer.py", line 74, in anonymize_images
    z = infer.truncated_z(
  File "DeepPrivacy/deep_privacy/inference/infer.py", line 96, in truncated_z
    while z.abs().max() >= truncation_level:
RuntimeError: operation does not have an identity.

Without a truncation level the model outputs an identical image without changes

When running the webcam module, it crashes after some time with a division by zero error.

python webcam.py None 2020-11-23 16:53:20,565 [INFO ] Loaded model:fdf_512 2020-11-23 16:53:23,362 [INFO ] Generator initialized with 47.39M parameters {'type': 'RCNNDetector', 'keypoint_threshold': 0.2, 'densepose_threshold': 0.3, 'simple_expand': False, 'align_faces': False, 'resize_background': True, 'face_detector_cfg': {'name': 'RetinaNetResNet50', 'confidence_threshold': 0.1, 'nms_iou_threshold': 0.3, 'max_resolution': 1080, 'fp16_inference': True, 'clip_boxes': True}, 'rcnn_batch_size': 8} Traceback (most recent call last): File "webcam.py", line 48, in <module> print("FPS:", frames / (time.time() - t), end="\r") ZeroDivisionError: float division by zero

The same happens with a different model:
python webcam.py -m fdf128_retinanet128 None 2020-11-23 16:52:23,881 [INFO ] Loaded model:retinanet128 2020-11-23 16:52:25,888 [INFO ] Generator initialized with 3.17M parameters {'type': 'BaseDetector', 'keypoint_threshold': 0.2, 'densepose_threshold': 0.3, 'simple_expand': True, 'align_faces': False, 'resize_background': True, 'face_detector_cfg': {'name': 'RetinaNetResNet50', 'confidence_threshold': 0.1, 'nms_iou_threshold': 0.3, 'max_resolution': 1080, 'fp16_inference': True, 'clip_boxes': True}} Traceback (most recent call last): File "webcam.py", line 48, in <module> print("FPS:", frames / (time.time() - t), end="\r") ZeroDivisionError: float division by zero

This should be
anonymized_frames = self.anonymize_images(frames, im_bboxes=im_bboxes)
as im_boxes is a positional argument.
Otherwises this method throws an error when it is called with_keypoints = False which is the default

Hey, thank you for your research and publishing your source code.

Are you aware of brighter.ai, a Berlin based startup - they are doing more or less what your paper does, but as a product. Maybe you should get in contact and/or mention them in the paper?

Best,
Christoph

When running the program and doing a conversion (after downloading the model and setting up all dependencies), I keep getting a Killed error about what seems to be half way through the process:

(base) stephenlizcano@deepfacelab-vm:~/DeepPrivacy$ python -m deep_privacy.inference.anonymize_video models/isvc_large/config.yml --source_path /home/stephenlizcano/deepfakeResize.mp4 --target_path /home/stephenlizcano/video_anonymized.mp4
CONFIG USED:
================================================================================
logging.num_ims_per_log                            500
logging.num_ims_per_save_image                     100000
logging.num_ims_per_checkpoint                     200000
max_imsize                                         128
use_full_validation                                False
load_fraction_of_dataset                           False
dataset                                            yfcc100m128
train_config.transition_iters                      1200000
train_config.learning_rate                         0.00175
train_config.amp_opt_level                         O1
train_config.batch_size_schedule                   {4: 256, 8: 256, 16: 256, 32: 128, 64: 96, 128: 64}
models.start_channel_size                          512
models.pose_size                                   14
models.image_channels                              3
models.generator.running_average_decay             0.999
models.discriminator.structure                     normal
config_path                                        models/isvc_large/config.yml
checkpoint_dir                                     models/isvc_large/checkpoints
generated_data_dir                                 models/isvc_large/generated_data
summaries_dir                                      models/isvc_large/summaries
source_path                                        /home/stephenlizcano/deepfakeResize.mp4
target_path                                        /home/stephenlizcano/video_anonymized.mp4
anonymize_source                                   False
max_face_size                                      1.0
without_source                                     False
================================================================================
 [*] Loading checkpoint from models/isvc_large/checkpoints/step_40000000.ckpt succeed!
extending G 512
extending G 512
extending G 512
extending G 256
extending G 128
Anonymizer initialized. Keypoint threshold: 0.1Face threshold: 0.6
================================================================================
Anonymizing video.
Duration: 90.38. Total frames: 2259, FPS: 25.0
Anonymizing from: 0(0.0), to: 2259(90.36)
Reading frames: 100%|███████████████████████| 2259/2259 [00:30<00:00, 72.91it/s]
Batch detecting faces:   0%|                           | 0/2259 [00:00<?, ?it/s]Finished loading DSFD model!
Batch detecting faces: 100%|████████████████| 2259/2259 [10:02<00:00,  3.79it/s]
Killed

I am trying to train on a new dataset using train.py but it is showing me this error continuously. I am not sure or aware what this .debug/before.torch file is and is trying to do

Does this work on ears since they are increasingly being used for identification?

Thank you very much for the impressive work

It is not clear to me:

1) How to fine-tune the existing models on another dataset? 
2) How to train a deep privacy model from scratch? 

In both cases, what are the exact steps, scripts, formats, etc

Regards

Hello,
I was wondering how the resolution of the output images is chosen? Is this a parameter of the model, which could be trained to output higher resolution images?

Alternatively do you think it is reasonable to cut-out the face from a higher resolution image and input this into your program, then re-integrate into the original resolution image? If so, how much context surrounding the face would be good to include in these subsets?

My goal is to benchmark how this tool works on high resolution images.

Thanks,
James

Hello, Thank you for this awesome work.

So, I'm trying to compute the SSIM metric between real and generated images inside the loss.py program to see it's evolution while training the model.

I tried to do this by transforming real_data tensors to RGB image but the result was not satisfying. So I was wondering if there's a way to get the images (in a pixelwise structure), since SSIM is using pixels not tensors.

Finally, kindly share any ideas or tips to complete this task.

Thanks for the detailed write up of your amazing work. I have a few questions, in particular relating to "Image Inpainting with Learnable Feature Imputation".

1. In GCPR.md mask_infer.py is referenced but I can't seem to find it in this github repo.
2. I'm trying to reproduce the results in the paper "Image Inpainting with Learnable Feature Imputation", would it be possible to train a new model from scratch, providing I have the training and mask dataset.

Thank you for the great work!

Regarding the paper, it is mentioned that:

• "By design, our generator never observes the original face, ensuring removal of any privacy-sensitive information"
• "Our model is based on a conditional generative adversarial network, generating images considering the original pose and image background"

My question is how the model preserves the gender, age and other related features while generating the faces?

Regards

First of all, I should say thanks a lot for a detailed write-up of your amazing work. It is really useful for me
I have some questions. I want to train this model from scratch with the FDF dataset. I download it from the Github that you mentioned:

1. There is no mask dataset in it, should I make it? how?
2. It did not separate data to training and validation, how should I do that?
   As you have told I download the dataset and run the code "python train.py configs/fdf/512.py" but İ got the error, "AssertionError: Did not find landmarks at: data/fdf/train/landmarks.npy". I don't know how should İ solve this problem :(

Whenever you get some time, I appreciate your eyes on this.

In the README it states that the model fdf128_rcnn512 uses DSFD as face detector. However, the respective config file at: https://folk.ntnu.no/haakohu/configs/fdf_512.json has RetinaNetResNet50 as face detector. Why exactly is that?

Could you help me?
python3 anonymize.py -s/home/daqua/Pictures/h.png -t/home/daqua/Pictures/h2.png

None
Traceback (most recent call last):
File "anonymize.py", line 5, in
cli.main()
File "/home/daqua/Videos/DeepPrivacy/deep_privacy/cli.py", line 111, in main
anonymizer, cfg = build_anonymizer(
File "/home/daqua/Videos/DeepPrivacy/deep_privacy/build.py", line 62, in build_anonymizer
cfg = get_config(config_urls[model_name])
File "/home/daqua/Videos/DeepPrivacy/deep_privacy/build.py", line 36, in get_config
torch.hub.download_url_to_file(config_url, cfg_path)
File "/home/daqua/.local/lib/python3.8/site-packages/torch/hub.py", line 394, in download_url_to_file
u = urlopen(req)
File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 308: Permanent Redirect

As I understand, this algorithm works well on small faces in the image. But for large faces (larger than 128 x 128), the output is not suitable.
Is there a solution?
Is this a problem because the output of the generator network is 128 x 128?

Hi,
I am quite new to GANs and experimenting a bit with loss functions. While trying to train my own model I struggled a bit unraveling the code of your losses. Could you please explain what the loss function with id = 1 stands for?

And is it correct to say that loss function with id 0 represents the wasserstein distance, loss function 2 the true positive rate of the discriminator of real images and loss function 3 the adversarial loss for the generator (fakes scores of discriminator)?

Best regards and thanks in advance!

python3.6 -m deep_privacy.inference.anonymize_folder model/default/config.yml --source_path lina2.jpg --target_path lina2_anonymized.jpg

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/DeepPrivacy/deep_privacy/inference/anonymize_folder.py", line 1, in <module>
    from deep_privacy.inference import deep_privacy_anonymizer, infer
  File "/opt/DeepPrivacy/deep_privacy/inference/deep_privacy_anonymizer.py", line 7, in <module>
    from .anonymizer import Anonymizer
  File "/opt/DeepPrivacy/deep_privacy/inference/anonymizer.py", line 7, in <module>
    from deep_privacy.inference import infer
  File "/opt/DeepPrivacy/deep_privacy/inference/infer.py", line 5, in <module>
    from deep_privacy import config_parser, utils
  File "/opt/DeepPrivacy/deep_privacy/utils.py", line 4, in <module>
    from apex.amp._amp_state import _amp_state
  File "/usr/local/lib/python3.6/site-packages/apex/__init__.py", line 18, in <module>
    from apex.interfaces import (ApexImplementation,
  File "/usr/local/lib/python3.6/site-packages/apex/interfaces.py", line 10, in <module>
    class ApexImplementation(object):
  File "/usr/local/lib/python3.6/site-packages/apex/interfaces.py", line 14, in ApexImplementation
    implements(IApex)
  File "/usr/local/lib/python3.6/site-packages/zope/interface/declarations.py", line 483, in implements
    raise TypeError(_ADVICE_ERROR % 'implementer')
**TypeError: Class advice impossible in Python3.  Use the @implementer class decorator instead.**

why？

Not sure why this issue is popping up now...I've run this code in the past to anonymize thousands of images...

CUDA version: 10.1

Linux learner001 4.15.0-118-generic #119-Ubuntu SMP Tue Sep 8 12:30:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

>>> torch.__version__
'1.7.0.dev20200923+cu101'
>>> torchvision.__version__
'0.8.0.dev20200923+cu101'

(deep-privacy) (base) lrm@learner001:~/repos/DeepPrivacy$ python -m deep_privacy.inference.anonymize_folder models/large/config.yml --source_path /home/lrm/data/inline_2020-09-17_2020-09-18/ --target_path /home/lrm/data/anon_inline_2020-09-17_2020-09-18/

CONFIG USED:
================================================================================
logging.num_ims_per_log                            500
logging.num_ims_per_save_image                     100000
logging.num_ims_per_checkpoint                     200000
max_imsize                                         128
use_full_validation                                False
load_fraction_of_dataset                           False
dataset                                            yfcc100m128
train_config.transition_iters                      1200000
train_config.learning_rate                         0.00175
train_config.amp_opt_level                         O1
train_config.batch_size_schedule                   {4: 256, 8: 256, 16: 256, 32: 128, 64: 96, 128: 64}
models.start_channel_size                          512
models.pose_size                                   14
models.image_channels                              3
models.generator.running_average_decay             0.999
models.discriminator.structure                     normal
config_path                                        models/large/config.yml
checkpoint_dir                                     models/large/checkpoints
generated_data_dir                                 models/large/generated_data
summaries_dir                                      models/large/summaries
source_path                                        /home/lrm/data/inline_2020-09-17_2020-09-18/
target_path                                        /home/lrm/data/anon_inline_2020-09-17_2020-09-18/
================================================================================
 [*] Loading checkpoint from models/large/checkpoints/step_40000000.ckpt succeed!
extending G 512
extending G 512
extending G 512
extending G 256
extending G 128
Anonymizer initialized. Keypoint threshold: 0.3Face threshold: 0.1
Batch detecting faces:   0%|                                                                | 0/25359 [00:00<?, ?it/s]Finished loading DSFD model!
Batch detecting faces:  60%|███████████████████████████████▏                    | 15202/25359 [14:15<09:31, 17.78it/s]
Traceback (most recent call last):
  File "/home/lrm/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/lrm/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/inference/anonymize_folder.py", line 10, in <module>
    anonymizer.anonymize_folder(source_path, save_path)
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/inference/anonymizer.py", line 35, in anonymize_folder
    im_bboxes=im_bboxes)
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/inference/anonymizer.py", line 42, in anonymize_image_paths
    face_threshold=self.face_threshold
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/detection/detection_api.py", line 43, in batch_detect_faces_with_keypoints
    im_bboxes = batch_detect_faces(images, face_threshold)
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/detection/detection_api.py", line 25, in batch_detect_faces
    det = face_detector.detect_face(im[:, :, ::-1], face_threshold)
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/detection/dsfd/detect.py", line 46, in detect_face
    y = self.net(x, confidence_threshold, self.nms_iou_threshold)
  File "/home/lrm/.pyenv/versions/deep-privacy/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/detection/dsfd/face_ssd.py", line 193, in forward
    nms_threshold
  File "/home/lrm/repos/DeepPrivacy/deep_privacy/detection/dsfd/utils.py", line 61, in forward
    indices = (conf_scores >= confidence_threshold).nonzero().squeeze()
RuntimeError: CUDA error: misaligned address

Thank you for this work

It is mentioned in the paper, temporal consistency can be one of the proposed improvements.

• Did you implement any kind of temporal consistency method in this code? (Based on the results shown as the animated gif, I do not think so)
• Do you have any recommendations, guidelines etc?

Regards

Hi, it seems that the config.yml file for the large "best" model is missing. The other model folder seems to have this. Would it be possible to update the google drive with the large config file?

Thanks for your code! I am trying to train a model from scratch using my own facial keypoint detection network (rather than using the standard pre-trained mask RCNN provided in this repository). My keypoint network provides 63 facial points as compared to the 7 points used by your code. Is it possible to train the network with 63 keypoints? Where should I make the changes to your code? Thanks in advance!

I tried to install that on Windows, all packages were successful downloaded.

Now i'm trying to run command

python -m deep_privacy.inference.anonymize_folder model/default/config.yml --source_path testim.jpg --target_path testim_anonymized.jpg

I changed after failure model to models

python -m deep_privacy.inference.anonymize_folder models/default/config.yml --source_path testim.jpg --target_path testim_anonymized.jpg

after running this command i get

`C:\Users\Alexander\Work\NN\DeepPrivacy>python -m deep_privacy.inference.anonymize_folder models/default/config.yml --source_path testim.jpg --target_path testim_anonymized.jpg
CONFIG USED:
logging.num_ims_per_log 500
logging.num_ims_per_save_image 100000
logging.num_ims_per_checkpoint 200000
max_imsize 128
use_full_validation False
load_fraction_of_dataset False
dataset yfcc100m128
train_config.transition_iters 1200000
train_config.learning_rate 0.00175
train_config.amp_opt_level O1
train_config.batch_size_schedule {4: 256, 8: 256, 16: 256, 32: 128, 64: 96, 128: 64}
train_config.criterion.type wgan
train_config.criterion.gradient_penalty_weight 10
train_config.criterion.epsilon_penalty_weight 0.001
train_config.criterion.different_images False
models.start_channel_size 256
models.pose_size 14
models.image_channels 3
models.generator.running_average_decay 0.999
models.discriminator.structure normal
config_path models/default/config.yml
checkpoint_dir models/default\checkpoints
generated_data_dir models/default\generated_data
summaries_dir models/default\summaries
source_path testim.jpg
target_path testim_anonymized.jpg

Traceback (most recent call last):
File "C:\Users\Alexander\Miniconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\Alexander\Miniconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Alexander\Work\NN\DeepPrivacy\deep_privacy\inference\anonymize_folder.py", line 5, in
generator, imsize, source_path, image_paths, save_path = infer.read_args()
File "C:\Users\Alexander\Work\NN\DeepPrivacy\deep_privacy\inference\infer.py", line 176, in read_args
ckpt = utils.load_checkpoint(config.checkpoint_dir)
File "C:\Users\Alexander\Work\NN\DeepPrivacy\deep_privacy\utils.py", line 43, in load_checkpoint
with open(os.path.join(ckpt_dir_or_file, 'latest_checkpoint')) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'models/default\checkpoints\latest_checkpoint'`

How can i fix it?

Hello,
I was wondering how the resolution of the output images is chosen? Is this a parameter of the model, which could be trained to output higher resolution images?

Alternatively do you think it is reasonable to cut-out the face from a higher resolution image and input this into your program, then re-integrate into the original resolution image? If so, how much context surrounding the face would be good to include in these subsets?

My goal is to benchmark how this tool works on high resolution images.

Thanks,
James

Hi,
it seems like
Reading of the images is successfully completed, dataset is loaded, validation of the model seems to be completed too, but then the following error happens:

Traceback (most recent call last):
File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec)
File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals)
File "/workspace/deep_privacy/train.py", line 484, in <module> trainer.train()
File "/workspace/deep_privacy/train.py", line 464, in train self.maybe_validate_model()
File "/workspace/deep_privacy/train.py", line 412, in maybe_validate_model self.validate_model()
File "/workspace/deep_privacy/train.py", line 326, in validate_model log_to_validation=True)
File "/workspace/deep_privacy/logger.py", line 58, in save_images torchvision.utils.save_image(images, filepath, nrow=10)
File "/opt/conda/lib/python3.6/site-packages/torchvision/utils.py", line 101, in save_image normalize=normalize, range=range, scale_each=scale_each)
File "/opt/conda/lib/python3.6/site-packages/torchvision/utils.py", line 31, in make_grid raise TypeError('tensor or list of tensors expected, got {}'.format(type(tensor)))
TypeError: tensor or list of tensors expected, got <class 'numpy.ndarray'> 

I wrote a print statement to logger.py file to see which "images" are being forwarded to torchvision.utils.save_image function and I saw that the docker crashes every time fakes*.jpg images are forwarded into the save_image function. Their are of <class 'numpy.ndarray'> class, whereas all the others (at which the docker doesn't crash) are of torch.Tensor type. However, I wrote an if statement if the images are numpy array class, and I change their type when that happens into torch.Tensor, but then this error happens:

RuntimeError: The expanded size of the tensor (3) must match the existing size (8) at non-singleton dimension 0. Target sizes: [3, 8, 3]. Tensor sizes: [8, 8, 3]

What can I do that it will go through as it should?

Dear

Is there a way (API) to get the bounding boxes and key points detected? (For example, in terms of a list of x,y,w,h for bounding boxes and x,y for each key point)

Best Regards

Hello,

I just tried to build the docker image, but sadly it fails:

 => ERROR [ 8/14] RUN pip install -e /detectron2_repo                                                                                                                                                                                                   3.6s
------
 > [ 8/14] RUN pip install -e /detectron2_repo:
#12 1.137 Obtaining file:///detectron2_repo
#12 3.374     ERROR: Command errored out with exit status 1:
#12 3.374      command: /opt/conda/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/detectron2_repo/setup.py'"'"'; __file__='"'"'/detectron2_repo/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
#12 3.374          cwd: /detectron2_repo/
#12 3.374     Complete output (6 lines):
#12 3.374     Traceback (most recent call last):
#12 3.374       File "<string>", line 1, in <module>
#12 3.374       File "/detectron2_repo/setup.py", line 15, in <module>
#12 3.374         assert torch_ver >= [1, 6], "Requires PyTorch >= 1.6"
#12 3.374     AssertionError: Requires PyTorch >= 1.6
#12 3.374     No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
#12 3.374     ----------------------------------------
#12 3.375 ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
------
executor failed running [/bin/sh -c pip install -e /detectron2_repo]: exit code: 1

It says No CUDA runtime is found but that's not mentioned in the requirements.
Do I need to install it manually or what's the case here?

Hello @cclauss @hardmaru @hukkelas @StianHanssen @SajjadAemmi
Great work on the paper and even the results look amazing!

Can you please guide me with step by step process on how we need to prepare a dataset for a custom image dataset with a custom mask (if possible)?

How the masks will look like in the sense that whether it will be a colour image on which white patches (the part that will be inpainted from the model) is drawn or it will be a completely black image on which white patches (the part that will be inpainted from the model) is drawn? (in short how to prepare mask and then how to supply it for model training)

I referred to your steps but couldn't completely follow them.
Thank you in advance!

Dears

Regarding keypoint_threshold and face_threshold, could you please explain in some detial how they work and affect the anonymization.

anonymizer = deep_privacy_anonymizer.DeepPrivacyAnonymizer(generator,
                                                           batch_size=32,
                                                           use_static_z=True,
                                                           keypoint_threshold=.1,
                                                           face_threshold=.6)

For example, I noticed that making face_threshold too small, encourage the network to predict anything as face. and making it too large, leads to missing faces. How this will work with the keypoint_threshold?

Additionally, what is replace_tight_bbox used for?

Finally, kindly share any tips to enable detection and anonymization of small faces.

Thank you and best regards

seems like the demo with face configuration does not allow the upload of image, but tries to access the webcam in both modalities

i have Nvidia GPU 8 GB total capacity. i run anonymize video on a mp4 video with resolution 240x160 and lenght 00:00:01.
and get this error:

RuntimeError: CUDA out of memory. Tried to allocate 850.00 MiB (GPU 0; 8.00 GiB total capacity; 4.64 GiB already allocated; 298.65 MiB free; 5.96 GiB reserved in total by PyTorch)

what is wrong?
thank you

When you start anonymize.py you get an ImportError

Did someone else already received this error and knows a solution?

Thanks for the great repository!

I just wonder, could the anonymization network generate different results on the same input image?
There is no manual seed selection in code, that I could find in your repository.

The only thing that works for me:

anonymizer = build_anonymizer(model_name="fdf128_rcnn512", truncation_level=100)
torch.manual_seed(int(time.time()))
np.random.seed(int(time.time()))
res = anonymizer.detect_and_anonymize_images([in_image])

Hey quite new to this , can you help in a doubt . when you say install pythorch and python , where exactly should I type those codelines .

Hello,

I am trying to build the docker container using proposed docker file, but I am getting this error:
-- cut here --
Found link https://files.pythonhosted.org/packages/cb/28/91f26bd088ce8e22169032100d4260614fc3da435025ff389ef1d396a433/pip-20.2.4-py2.py3-none-any.whl#sha256=51f1c7514530bd5c145d8f13ed936ad6b8bfcb8cf74e10403d0890bc986f0033 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.,!=3.4.), version: 20.2.4
Found link https://files.pythonhosted.org/packages/0b/f5/be8e741434a4bf4ce5dbc235aa28ed0666178ea8986ddc10d035023744e6/pip-20.2.4.tar.gz#sha256=85c99a857ea0fb0aedf23833d9be5c40cf253fe24443f0829c7b472e23c364a1 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.,!=3.1.,!=3.2.,!=3.3.,!=3.4.), version: 20.2.4
The command '/bin/sh -c pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./' returned a non-zero code: 1
-- cut here --

This is related to this line in the Dockerfile:
RUN pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Any advise?
FM

I tried use it with docker, but I am stuck on the command with nvidia-docker: nvidia-docker run --rm -it -v $PWD:/workspace -e CUDA_VISIBLE_DEVICES=0 deep_privacy python -m deep_privacy.train models/large/config.yml

After installation of nvidia-docker, I could command like this docker run --gpus all nvidia/cuda:10.0-base nvidia-smi but not like nvidia-docker run .

I am not familiar with nvidia-docker, I'd really appreciate that if you could tell me how to run your program on docker.

nvidia-docker: command not found · Issue #1028 · NVIDIA/nvidia-docker

How to prepare your own dataset for training

when i input a video, but got output like this. All is blank

Hello, thank you very much for your work!
I have a problem with loading the checkpoint: when I use checkpoint places_E.ckpt with config configs/gcpr/places/places_E_conv.py, I end up with RuntimeError which cannot match the weights of my model with pretrained one:

RuntimeError: Error(s) in loading state_dict for MSGGenerator:
	Missing key(s) in state_dict: "decoder.skip_connection8.conv.layers.0.weight", "decoder.skip_connection8.conv.layers.0.bias", "decoder.skip_connection16.conv.layers.0.weight", "decoder.skip_connection16.conv.layers.0.bias", "decoder.skip_connection32.conv.layers.0.weight", "decoder.skip_connection32.conv.layers.0.bias", "decoder.skip_connection64.conv.layers.0.weight", "decoder.skip_connection64.conv.layers.0.bias", "decoder.skip_connection128.conv.layers.0.weight", "decoder.skip_connection128.conv.layers.0.bias", "decoder.skip_connection256.conv.layers.0.weight", "decoder.skip_connection256.conv.layers.0.bias". 
	Unexpected key(s) in state_dict: "decoder.basic_block4.layers.0.expected_value_updater.input_updater.weight", "decoder.basic_block4.layers.0.expected_value_updater.input_updater.bias", "decoder.basic_block4.layers.0.mask_updater.weight", "decoder.basic_block4.layers.3.expected_value_updater.input_updater.weight", "decoder.basic_block4.layers.3.expected_value_updater.input_updater.bias", "decoder.basic_block4.layers.3.mask_updater.weight", "decoder.skip_connection8.beta0", "decoder.skip_connection8.beta1", "decoder.basic_block8.layers.0.expected_value_updater.input_updater.weight", "decoder.basic_block8.layers.0.expected_value_updater.input_updater.bias", "decoder.basic_block8.layers.0.mask_updater.weight", "decoder.basic_block8.layers.3.expected_value_updater.input_updater.weight", "decoder.basic_block8.layers.3.expected_value_updater.input_updater.bias", "decoder.basic_block8.layers.3.mask_updater.weight", "decoder.skip_connection16.beta0", "decoder.skip_connection16.beta1", "decoder.basic_block16.layers.0.expected_value_updater.input_updater.weight", "decoder.basic_block16.layers.0.expected_value_updater.input_updater.bias", "decoder.basic_block16.layers.0.mask_updater.weight", "decoder.basic_block16.layers.3.expected_value_updater.input_updater.weight", "decoder.basic_block16.layers.3.expected_value_updater.input_updater.bias", "decoder.basic_block16.layers.3.mask_updater.weight", "decoder.skip_connection32.beta0", "decoder.skip_connection32.beta1", "decoder.basic_block32.layers.0.expected_value_updater.input_updater.weight", "decoder.basic_block32.layers.0.expected_value_updater.input_updater.bias", "decoder.basic_block32.layers.0.mask_updater.weight", "decoder.basic_block32.layers.3.expected_value_updater.input_updater.weight", "decoder.basic_block32.layers.3.expected_value_updater.input_updater.bias", "decoder.basic_block32.layers.3.mask_updater.weight", "decoder.skip_connection64.beta0", "decoder.skip_connection64.beta1", "decoder.basic_block64.layers.0.expected_value_updater.input_updater.weight", "decoder.basic_block64.layers.0.expected_value_updater.input_updater.bias", "decoder.basic_block64.layers.0.mask_updater.weight", "decoder.basic_block64.layers.3.expected_value_updater.input_updater.weight", "decoder.basic_block64.layers.3.expected_value_updater.input_updater.bias", "decoder.basic_block64.layers.3.mask_updater.weight", "decoder.skip_connection128.beta0", "decoder.skip_connection128.beta1", "decoder.basic_block128.layers.0.expected_value_updater.input_updater.weight", "decoder.basic_block128.layers.0.expected_value_updater.input_updater.bias", "decoder.basic_block128.layers.0.mask_updater.weight", "decoder.basic_block128.layers.3.expected_value_updater.input_updater.weight", "decoder.basic_block128.layers.3.expected_value_updater.input_updater.bias", "decoder.basic_block128.layers.3.mask_updater.weight", "decoder.skip_connection256.beta0", "decoder.skip_connection256.beta1", "decoder.basic_block256.layers.0.expected_value_updater.input_updater.weight", "decoder.basic_block256.layers.0.expected_value_updater.input_updater.bias", "decoder.basic_block256.layers.0.mask_updater.weight", "decoder.basic_block256.layers.3.expected_value_updater.input_updater.weight", "decoder.basic_block256.layers.3.expected_value_updater.input_updater.bias", "decoder.basic_block256.layers.3.mask_updater.weight", "encoder.basic_block256.layers.0.expected_value_updater.input_updater.weight", "encoder.basic_block256.layers.0.expected_value_updater.input_updater.bias", "encoder.basic_block256.layers.0.mask_updater.weight", "encoder.basic_block256.layers.3.expected_value_updater.input_updater.weight", "encoder.basic_block256.layers.3.expected_value_updater.input_updater.bias", "encoder.basic_block256.layers.3.mask_updater.weight", "encoder.basic_block128.layers.0.expected_value_updater.input_updater.weight", "encoder.basic_block128.layers.0.expected_value_updater.input_updater.bias", "encoder.basic_block128.layers.0.mask_updater.weight", "encoder.basic_block128.layers.3.expected_value_updater.input_updater.weight", "encoder.basic_block128.layers.3.expected_value_updater.input_updater.bias", "encoder.basic_block128.layers.3.mask_updater.weight", "encoder.basic_block64.layers.0.expected_value_updater.input_updater.weight", "encoder.basic_block64.layers.0.expected_value_updater.input_updater.bias", "encoder.basic_block64.layers.0.mask_updater.weight", "encoder.basic_block64.layers.3.expected_value_updater.input_updater.weight", "encoder.basic_block64.layers.3.expected_value_updater.input_updater.bias", "encoder.basic_block64.layers.3.mask_updater.weight", "encoder.basic_block32.layers.0.expected_value_updater.input_updater.weight", "encoder.basic_block32.layers.0.expected_value_updater.input_updater.bias", "encoder.basic_block32.layers.0.mask_updater.weight", "encoder.basic_block32.layers.3.expected_value_updater.input_updater.weight", "encoder.basic_block32.layers.3.expected_value_updater.input_updater.bias", "encoder.basic_block32.layers.3.mask_updater.weight", "encoder.basic_block16.layers.0.expected_value_updater.input_updater.weight", "encoder.basic_block16.layers.0.expected_value_updater.input_updater.bias", "encoder.basic_block16.layers.0.mask_updater.weight", "encoder.basic_block16.layers.3.expected_value_updater.input_updater.weight", "encoder.basic_block16.layers.3.expected_value_updater.input_updater.bias", "encoder.basic_block16.layers.3.mask_updater.weight", "encoder.basic_block8.layers.0.expected_value_updater.input_updater.weight", "encoder.basic_block8.layers.0.expected_value_updater.input_updater.bias", "encoder.basic_block8.layers.0.mask_updater.weight", "encoder.basic_block8.layers.3.expected_value_updater.input_updater.weight", "encoder.basic_block8.layers.3.expected_value_updater.input_updater.bias", "encoder.basic_block8.layers.3.mask_updater.weight", "encoder.basic_block4.layers.0.expected_value_updater.input_updater.weight", "encoder.basic_block4.layers.0.expected_value_updater.input_updater.bias", "encoder.basic_block4.layers.0.mask_updater.weight", "encoder.basic_block4.layers.3.expected_value_updater.input_updater.weight", "encoder.basic_block4.layers.3.expected_value_updater.input_updater.bias", "encoder.basic_block4.layers.3.mask_updater.weight", "from_rgb.0.expected_value_updater.input_updater.weight", "from_rgb.0.expected_value_updater.input_updater.bias". 
	size mismatch for from_rgb.0.weight: copying a param with shape torch.Size([16, 3, 1, 1]) from checkpoint, the shape in current model is torch.Size([16, 5, 1, 1]).

Process finished with exit code 1

Will be grateful for any help!