Code Monkey home page Code Monkey logo

gfvc's Introduction

Awesome 🎉Generative Face Video Coding

GFVC_framework

Awesome GitHub stars

Implementation Codes

We optimize the implemention codes of three representative GFVC works, i.e., FOMM, CFTE and FV2V, and further provide the unified codes regarding the encoder and decoder processes. Current extensions to the original code include DAC and HDAC

  • Download the CFTE-checkpoint.pth.tar, FOMM-checkpoint.pth.tar, and FV2V-checkpoint.pth.tar to the path ./GFVC/CFTE/checkpoint/, ./GFVC/FOMM/checkpoint/, and ./GFVC/FV2V/checkpoint/ respectively. The checkpoints for FOMM, CFTE and FV2V are available at this link while DAC and HDAC are available here.

  • The overall testing dataset is available at this link.

  • Download HEVC and VVC and place both codecs in source/anchors/ directory.

  • The specific details can be seen in the subfolder source.

Sample Demos

Demo: Similar Bitrate && Similar Quality

similar.mp4

Demo: Animating Virtual Faces

Animating.virtual.faces.mp4

Demo: Interacting with Facial Expression and Head Posture

cotrol.mp4

Rate-distortion Performance

RDperformance

Technical Summary


  • (DCC 2024) [GFVC_Review] Generative Face Video Coding Techniques and Standardization Efforts: A Review Chen Bolin, Chen Jie, Wang Shiqi, Ye Yan paper
  • (DCC 2024) [GFVC_Translator] Enabling Translatability of Generative Face Video Coding: A Unified Face Feature Transcoding Framework Yin Shanzhi, Chen Bolin, Wang Shiqi, Ye Yan

  • (ICIP 2023) [RDAC] Predictive Coding for Animation-Based Video Compression Goluck Konuko, Stéphane Lathuilière, Giuseppe Valenzise paper
  • (arXiv 2023) [IFVC] Interactive Face Video Coding: A Generative Compression Framework Chen Bolin, Wang Zhao, Li Binzhe, Wang Shurun, Wang Shiqi, Ye Yan paper
  • (TCSVT 2023) [CTTR] Compact Temporal Trajectory Representation for Talking Face Video Compression Chen Bolin, Wang Zhao, Li Binzhe, Wang Shiqi, Ye Yan paper

  • (ICME 2022) [Bi-Net] Generative Compression for Face Video: A Hybrid Scheme Anni Tang, Yan Huang, Jun Ling, Zhiyu Zhang, Yiwei Zhang, Rong Xie, Li Song paper
  • (BMVC 2022) [CVC_STR] Compressing Video Calls using Synthetic Talking Heads Madhav Agarwal, Anchit Gupta, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C V Jawahar paper
  • (ICIP 2022) [HDAC] A Hybrid Deep Animation Codec for Low-bitrate Video Conferencing Goluck Konuko, Stéphane Lathuilière, Giuseppe Valenzise paper
  • (ICIP 2022) [DMRGP] Dynamic Multi-Reference Generative Prediction for Face Video Compression Wang Zhao, Chen Bolin, Ye Yan, Wang Shiqi paper
  • (CVPRW 2022) [MAX-RS] Neural Face Video Compression Using Multiple Views Anna Volokitin, Stefan Brugger, Ali Benlalah, Sebastian Martin, Brian Amberg, Michael Tschannen paper
  • (DCC 2022) [C3DFD] Towards Ultra Low Bit-Rate Digital Human Character Communication via Compact 3D Face Descriptors Li Binzhe, Chen Bolin, Wang Zhao, Wang Shiqi, Ye Yan paper
  • (DCC 2022) [CFTE] Beyond Keypoint Coding: Temporal Evolution Inference with Compact Feature Representation for Talking Face Video Compression Chen Bolin, Wang Zhao, Li Binzhe, Lin Rongqun, Wang Shiqi, Ye Yan paper
  • (DCC 2022) [SNRVC] Semantic Neural Rendering-based Video Coding: Towards Ultra-Low Bitrate Video Conferencing Hu Yujie, Xu Youmin, Chang Jianhui, Zhang Jian paper

  • (CVPRW 2021) [Mob M-SPADE] Low Bandwidth Video-Chat Compression Using Deep Generative Models Maxime Oquab, Pierre Stock, Daniel Haziza, Tao Xu, Peizhao Zhang, Onur Celebi, Yana Hasson, Patrick Labatut, Bobo Bose-Kolanu, Thibault Peyronel, Camille Couprie paper
  • (CVPR 2021) [Face_vid2vid] One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing Wang Ting-Chun, Mallya Arun, Liu Ming-Yu paper
  • (ICMEW 2021) [VSBNet] A Generative Compression Framework For Low Bandwidth Video Conference Feng Dahu, Huang Yan, Zhang Yiwei, Ling Jun, Tang Anni, Song Li paper
  • (ICASSP 2021) [DAC] Ultra-Low Bitrate Video Conferencing Using Deep Image Animation Goluck Konuko, Giuseppe Valenzise, Stéphane Lathuilière paper

  • (NeurIPS 2021) [FOMM] First Order Motion Model for Image Animation Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe paper

Standardization Efforts


  • (JVET January 2024) [JVET-AG0042] AHG16: Proposed Common Software Tools and Testing Conditions for Generative Face Video Compression Bolin Chen, Jie Chen, Ru-Ling Liao, Yan Ye, Shiqi Wang Proposal
  • (JVET January 2024) [JVET-AG0048] AHG16: Interoperability Study on Parameter Translator of Generative Face Video Coding Shanzhi Yin, Bolin Chen, Jie Chen, Ru-Ling Liao, Yan Ye, Shiqi Wang Proposal
  • (JVET January 2024) [JVET-AG0087] AHG9: On the generative face video SEI message M. M. Hannuksela, F. Cricri, H. Zhang Proposal
  • (JVET January 2024) [JVET-AG0088] AHG9: Usage of the neural-network post-filter characteristics SEI message to define the generator NN of the generative face video SEI message M. M. Hannuksela, F. Cricri, H. Zhang Proposal
  • (JVET January 2024) [JVET-AG0139] AHG16: Depthwise separable convolution for generative face video compression Renjie Zou, Ru-Ling Liao, Bolin Chen, Jie Chen, Yan Ye Proposal
  • (JVET January 2024) [JVET-AG0187] AHG16: Study text for common test conditions and evaluation procedures for generative face video coding (draft 1) Sean McCarthy, Peng Yin, Bolin Chen, Yan Ye, Shiqi Wang Proposal
  • (JVET January 2024) [JVET-AG0203] AHG9/AHG16: Common text for proposed generative face video SEI message Jie Chen, Bolin Chen, Yan Ye, Shanzhi Yin, Shiqi Wang, Sean McCarthy, Peng Yin, Guan-Ming Su, Anustup Kuma Choudhury, Walt Husak, Gary J. Sullivan Proposal

  • (MPEG October 2023) [m64987] On VVC-assisted Ultra-low Rate Generative Face Video Coding Yan Ye, Sean McCarthy, Han Boon Teo, Zhuoyi Lv, Shiqi Wang, Kai Zhang, Marta Karczewicz, Iole Moccagatta Proposal
  • (JVET October 2023) [JVET-AF0234] AHG9: Common text for proposed generative face video SEI message Bolin Chen, Jie Chen, Yan Ye, Shiqi Wang, Sean McCarthy, Peng Yin, Guan-Ming Su, Anustup Kuma Choudhury, Walt Husak, Gary J. Sullivan Proposal
  • (JVET October 2023) [JVET-AF0146] AHG9: On Face Motion Information for Generative Face Video Han Boon Teo, Jing Yuan Thong, Karlekar Jayashree, Chong Soon Lim, Kiyofumi Abe Proposal
  • (JVET October 2023) [JVET-AF0048] A Study on Decoder Interoperability of Generative Face Video Compression Bolin Chen, Shanzhi Yin, Jie Chen, Yan Ye, Shiqi Wang Proposal
  • (JVET July 2023) [JVET-AE0280] AHG9: Common text for proposed generative face video SEI message Bolin Chen, Jie Chen, Yan Ye, Shiqi Wang, Sean McCarthy, Peng Yin, Guan-Ming Su, Anustup Kuma Choudhury, Walt Husak Proposal
  • (JVET July 2023) [JVET-AE0088] AHG9: A study on Generative Face Video SEI Message Han Boon Teo, Jing Yuan Thong, Karlekar Jayashree, Chong Soon Lim, Kiyofumi Abe Proposal
  • (JVET July 2023) [JVET-AE0083] AHG9: Common SEI Message of Generative Face Video Bolin Chen, Jie Chen, Yan Ye, Shiqi Wang Proposal
  • (JVET July 2023) [JVET-AE0080] AHG9: Generative Face Video SEI message Sean McCarthy, Peng Yin, Guan-Ming Su, Anustup Kuma Choudhury, Walt Husak Proposal
  • (JVET April 2023) [JVET-AD0051] AHG9: Common SEI Message of Generative Face Video Bolin Chen, Jie Chen, Yan Ye, Shiqi Wang Proposal
  • (JVET January 2023) [JVET-AC0088] AHG9: Generative Face Video SEI Message Bolin Chen, Jie Chen, Shurun Wang, Yan Ye, Shiqi Wang Proposal

Additional Notes

Acknowledgement

We really appreciate all authors for making their codes available to public.

  • The GFVC software package includes FOMM, CFTE and FV2V.
  • The testing dataset is sourced and preprocessed from VoxCeleb and CFVQA.
  • The quality assessment metrics include DISTS and LPIPS.

gfvc's People

Contributors

berlin0610 avatar goluck-konuko avatar

Stargazers

Wan Rui avatar  avatar Shunpu Tang avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.