Continuing editing (Not finished yet)

The goal of project is focus on Audio-driven Gesture Generation with output is 3D keypoints gesture.
Input: Audio, Text, Gesture, ..etc. -> Output: Gesture Motion

Gesture Generation is the process of generating gestures from speech or text. The goal of Gesture Generation is to generate gestures that are natural, realistic, and appropriate for the given context. The generated gestures can be used to animate virtual characters, robots, or embodied conversational agents.

ACM CCS: • Human-centered computing → Human computer interaction (HCI).

Paper by Folder : 📁/survey || 📁/approach || 📁/papers || 📁/dataset || 📁/books

• 1. Survey
• 2. Papers
  • Relation of speech and gesture
  • GENEA Challenge
  • 2023
  • 2022
  • 2021
  • 2020
  • 2019
  • 2018
  • Before 2017
  • Others
• 3. Selected Approach
  • 3.1 Selected rule base approach
  • 3.2 Selected data-driven approach
    • 3.2.a Statistical approach
    • 3.2.b Deep learning approach
• 4. Pipeline
• 5. Learning objective
• 6. Metric Evaluation
• 7. Datasets
• 8. Toolkit
• 9. Playlist & Talks
• 10. Code
• 11. Books

Main resource

• paperswithcode.com/Gesture Generation
• GENEA Workshop
• github/topic/gesture-generation
• twitter/WorkshopGENEA
• Scholar/gesture generation

• 【EUROGRAPHICS 2023】A Comprehensive Review of Data-Driven Co-Speech Gesture Generation; [paper]

• 2014 - Gesture and speech in interaction: An overview ; [paper]

• 【HAI 2021】Speech-based Gesture Generation for Robots and Embodied Agents: A Scoping Review [paper]

• 【IEEE 2022】 A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents [paper] ;

• The GENEA Challenge 2020: A large, crowdsourced evaluation of gesture generation systems on common data [paper] ; [homepage] ; [youtube] ; [Svito-zar/genea_numerical_evaluations]
• The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation [paper] ; [homepage] ; [youtube] ; [web]
• The GENEA Challenge 2023: A large-scale evaluation of gesture [paper]
• GENEA Workshop 2021: The 2nd Workshop on Generation and Evaluation of Non-verbal Behaviour for Embodied Agents [paper] ; [homepage]
• Evaluating gesture-generation in a large-scale open challenge - The GENEA Challenge 2022 [paper]

• The Relation of Speech and Gestures: Temporal Synchrony Follows Semantic Synchrony [paper]
• Complexity Matters E05: Complexity Matching and Synchronization between Gestures and Speech [paper]
• Easier Said Than Done? Task Difficulty's Influence on Temporal Alignment, Semantic Similarity, and Complexity Matching Between Gestures and Speech [paper]
• Advances in Visual Semiotics - The Semiotic Web [book]
• Gestural beats: The rhythm hypothesis [paper]

• 【CVPR 2023】 Co-speech Gesture Synthesis by Reinforcement Learning with Contrastive Pre-trained Rewards [paper] ; [RLracer/RACER]
• 【PAKDD 2023】 RLMixer: A Reinforcement Learning Approach For Integrated Ranking With Contrastive User Preference Modeling [paper]
• 【IJCAI 2023】 DiffuseStyleGesture - Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models [paper] ; [YoungSeng/DiffuseStyleGesture] ; [youtube]
• 【CVPR 2023】 Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation [paper] ; [Advocate99/DiffGesture]
• 【CVPR 2023】 Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglement [paper] ; [XingqunQi-lab/Diverse-3D-Hand-Gesture-Prediction]
• 【CVPR 2023】 QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation [paper] ; [YoungSeng/QPGesture] ; [video]
• 【CVPR 2023】 GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents [paper] ; [homepage] ; [video]
• 【CVPR 2023】 Continual Learning for Personalized Co-Speech Gesture Generation [paper]; [homepage]
• 【CVPR 2023】 Guided Motion Diffusion for Controllable Human Motion Synthesis [paper] ; [homepage]
• 【CVPR 2023】 Sequential Texts Driven Cohesive Motions Synthesis with Natural Transitions [paper]
• 【CVPR 2023】 EMMN: Emotional Motion Memory Network for Audio-driven Emotional Talking Face Generation [paper]
• 【CVPR 2023】 FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation [paper]
• 【CVPR 2023】 Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models [paper]
• 【CVPR 2023】 Speech4Mesh: Speech-Assisted Monocular 3D Facial Reconstruction for Speech-Driven 3D Facial Animation [paper]
• 【CVPR 2023】 Semi-supervised Speech-driven 3D Facial Animation via Cross-modal Encoding [paper]
• 【ACM MM 2023】UnifiedGesture - A Unified Gesture Synthesis Model for Multiple Skeletons [paper] ; [YoungSeng/UnifiedGesture]
• 【ICMI 2023】 AQ-GT: a Temporally Aligned and Quantized GRU-Transformer for Co-Speech Gesture Synthesis [paper] ; [hvoss-techfak/AQGT]
• DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model [paper]
• BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer [paper]
• EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation [paper] ;
• Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis [paper] ; [homepage] ; [video]
• EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model [paper]
• MRecGen: Multimodal Appropriate Reaction Generator [paper] ; [SSYSteve/MRecGen]
• Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model [paper]
• The KCL-SAIR team’s entry to the GENEA Challenge 2023 Exploring Role-based Gesture Generation in Dyadic Interactions: Listener vs. Speaker [paper]
• The KU-ISPL entry to the GENEA Challenge 2023-A Diffusion Model for Co-speech Gesture generation [paper]
• Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation [paper]
• Co-Speech Gesture Generation via Audio and Text Feature Engineering [paper]
• Gesture Motion Graphs for Few-Shot Speech-Driven Gesture Reenactment [paper]
• Gesture Generation with Diffusion Models Aided by Speech Activity Information [paper]
• FEIN-Z: Autoregressive Behavior Cloning for Speech-Driven Gesture Generation [paper]
• Discrete Diffusion for Co-Speech Gesture Synthesis [paper]
• DiffuGesture: Generating Human Gesture From Two-person Dialogue With Diffusion Models [paper]
• The FineMotion entry to the GENEA Challenge 2023: DeepPhase for conversational gestures generation [paper]
• Am I listening - Evaluating theQuality of Generated Data-driven Listening Motion [paper]
• Unified speech and gesture synthesis using flow matching [paper] ; [homepage] ;

• 【SIGGRAPH 2022】 A Motion Matching-based Framework for Controllable Gesture Synthesis from Speech [paper] ; [homepage]
• 【ICMI 2022】The DeepMotion entry to the GENEA Challenge 2022 [paper]
• 【ICMI 2022】The ReprGesture entry to the GENEA Challenge 2022 [paper] ; [YoungSeng/ReprGesture] ; [youtube]
• 【CVPR 2022】 HA2G - Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation [paper] ; [alvinliu0/HA2G]
• 【CVPR 2022】 SEEG - SEEG: Semantic Energized Co-Speech Gesture Generation [paper] ; [akira-l/seeg]
• 【CVPR 2022】 DiffGAN - Low-Resource Adaptation for Personalized Co-Speech Gesture Generation [paper]
• 【CVPR 2022】 Audio-Driven Neural Gesture Reenactment With Video Motion Graphs [paper]
• 【ICMI 2022】 ZeroEGGS Exemplar-based stylized gesture generation from speech: An entry to the GENEA Challenge 2022 [paper]
• 【AAMAS 2022】 Multimodal analysis of the predictability of hand-gesture properties [paper]
• 【ICMI 2022】 GestureMaster GestureMaster: Graph-based Speech-driven Gesture Generation [paper]
• 【ICRA 2022】Context-Aware Body Gesture Generation for Social Robots [paper]
• 【IROS 2022】Gesture2Vec: Clustering Gestures using Representation Learning Methods for Co-speech Gesture Generation [paper] [pjyazdian/Gesture2Vec] ; [youtube] ; [youtube]
• Evaluating Data-Driven Co-Speech Gestures of Embodied Conversational Agents through Real-Time Interaction [paper]
• ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech [paper] ; [ubisoft/ubisoft-laforge-ZeroEGGS] ; [youtube]
• Voice2Face: Audio-Driven Facial and Tongue Rig Animations [paper] ; [youtube] ; [web]
• Deep Gesture Generation for Social Robots Using Type-Specific Libraries [paper] ; [youtube] ; [web]
• Automatic text‐to‐gesture rule generation for embodied conversational agents [paper] [youtube]
• Evaluating Data-Driven Co-Speech Gestures of Embodied Conversational Agents through Real-Time Interaction [paper] ; [web]
• Towards Context-Aware Human-like Pointing Gestures with RL Motion Imitation [paper]
• Text/Speech-Driven Full-Body Animation [paper]
• Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding [paper]

• 【ICCV 2021】 Speech Drives Templates: Co-Speech Gesture Synthesis With Learned Templates [paper] ; shenhanqian/speechdrivestemplates ; [youtube] ; poster
• 【ICCV 2021】 Audio2Gestures Audio2Gestures: Generating Diverse Gestures From Speech Audio With Conditional Variational Autoencoders [paper]
• 【ICMI 2021】 Probabilistic Human-like Gesture Synthesis from Speech using GRU-based WGAN [paper]
• 【ICMI 2021】 Crossmodal Clustered Contrastive Learning: Grounding of Spoken Language to Gesture [paper] ; [dondongwon/CC_NCE_GENEA]
• 【IVA 2021】 Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech [paper] ; [homepage]
• 【IVA 2021】 Learning Speech-driven 3D Conversational Gestures from Video [paper]
• 【CASA 2021】 ExpressGesture ExpressGesture: Expressive gesture generation from speech through database matching [paper] ; [youtube]
• 【AAMAS 2021】 CMCF CCFM: An Architecture for Realtime Gesture Generation by Clustering Gestures by Communicative Function and Motion [paper]
• 【IEEEVR 2021】 Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents [paper] ; [UttaranB127/Text2Gestures] ; homepage
• Evaluating Data-Driven Co-Speech Gestures of Embodied Conversational Agents [paper]
• Multimodal analysis of the predictability of hand-gesture properties [paper]
• Deep Gesture Generation for Social Robots Using Type-Specific Libraries [paper]
• A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents [paper] ; [youtube] ; [homepage] ; [nagyrajmund/gesturebot]
• Moving Fast and Slow: Analysis of Representations and Post-Processing in Speech-Driven Automatic Gesture Generation [paper]
• ExpressGesture: Expressive gesture generation from speech through database matching [paper]
• Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speech [paper]

• 【SIGGRAPH Asia 2020】 Trimodal Speech gesture generation from the trimodal context of text, audio, and speaker identity [paper] ; [ai4r/Gesture-Generation-from-Trimodal-Context]
• 【ICMI 2020】 Gesticulator Gesticulator: A framework for semantically-aware speech-driven gesture generation [paper] ; [Svito-zar/gesticulator]
• 【ECCV 2020】 Mix-StAGE Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach [paper]
• 【EUROGRAPHICS 2020】 StyleGestures Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows [paper] ; [simonalexanderson/StyleGestures] ; [youtube]
• 【EUROGRAPHICS 2020】 StyleGestures Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows [paper] ; [simonalexanderson/StyleGestures] ; [youtube]
• 【EMNLP 2020】 AiSLE No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures [paper] ; [pchahuja/aisle]
• The GENEA Challenge 2020: A large, crowdsourced evaluation of gesture generation systems on common data [paper] ; [homepage] ; [youtube] ; [youtube] ; [Svito-zar/genea_numerical_evaluations]
• Gesticulator: A framework for semantically-aware speech-driven gesture generation [paper] ; [youtube] ; [Svito-zar/gesticulator] ; [homepage] ; [dataset]
• Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings [paper] ; [youtube] ; [homepage]
• Can we trust online crowdworkers? Comparing online and offline participants in a preference test of virtual agents [paper]
• Affective synthesis and animation of arm gestures from speech prosody [paper]
• FineMotion - Audio and Text-Driven approach for Conversational Gestures Generation [paper] ; [FineMotion/GENEA_2020]
• Modeling the Conditional Distribution of Co-Speech Upper Body Gesture Jointly Using Conditional-GAN and Unrolled-GAN [paper] ; [wubowen416/co-speech-gesture-generation-using-CGAN]
• 【IVA 2020】 Generating coherent spontaneous speech and gesture from text [paper] ; [video] ; [homepage]

• 【SIGGRAPH MIG 2019】 Multi-objective adversarial gesture generation [paper]
• 【ICMI 2019】 DRAM To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations [paper]
• 【CVPR 2019】 Speech2Gesture Learning Individual Styles of Conversational Gesture [paper]
• Analyzing Input and Output Representations for Speech-Driven Gesture Generation [paper] ; [GestureGeneration/Speech_driven_gesture_generation_with_autoencoder] ; [youtube] ; [youtube] ; [homepage]
• On the Importance of Representations for Speech-Driven Gesture Generation [paper]

• A Neural Network Approach to Missing Marker Reconstruction in Human Motion Capture [paper] ; [youtube] ; [youtube] ; [Svito-zar/NN-for-Missing-Marker-Reconstruction]
• Data Driven Non-Verbal Behavior Generation for Humanoid Robots [paper]
• A Neural Network Approach to Missing Marker Reconstruction in Human Motion Capture [paper] ; [Svito-zar/NN-for-Missing-Marker-Reconstruction] ; [youtube]
• Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network [paper]
• A Speech-Driven Hand Gesture Generation Method and Evaluation in Android Robots [paper] ; [youtube]
• Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots [paper]

• DCNF Predicting Co-verbal Gestures - A Deep and Temporal Modeling Approach [paper]
• 2017 - CDBN Speech-driven animation with meaningful behaviors [paper]

Others

• 【SIGGRAPH 2022】 GANimator for generate data GANimator: Neural Motion Synthesis from a Single Sequence [paper] ; PeizhuoLi/ganimator ; [youtube]
• 【CVPR 2021】 Body2Hands: Learning To Infer 3D Hands From Conversational Gesture Body Dynamics [paper]
• Rig Inversion by Training a Differentiable Rig Function [paper] ; [youtube]

• [1994] Rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents [paper]

• Speech to sequence gesture

  • 【SIGGRAPH 2001】 BEAT: the Behavior Expression Animation Toolkit [paper]
  • 【HRI 2012】 Robot Behavior Toolkit: Generating Effective Social Behaviors for Robots [paper]
  • Gesture Generation by Imitation: From Human Behavior to Computer Character Animation [books]

• Probabilistic model of speech to gesture

  • 【IVA 2006】 Towards a Common Framework for Multimodal Generation: The Behavior Markup Language [paper]
  • 【SIGGRAPH 2010】 Gesture Controllers [paper]

• Probabilistic model of personal style

  • 【ACM Transactions on Graphics 2008】Gesture modeling and animation based on a probabilistic re-creation of speaker style [paper]

• Neural classification model of personal style

  • 【IVA 2015】Predicting Co-verbal Gestures: A Deep and Temporal Modeling Approach [paper]

This section is -- not accurate --> continue edditing

• MLP (Multilayer perceptron)

  • 【ICMI 2020】 Gesticulator Gesticulator: A framework for semantically-aware speech-driven gesture generation [paper] ; [Svito-zar/gesticulator]

• RNN (Recurrent Neural Networks)

  • 【MM 2021】 Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning [paper] ; [UttaranB127/speech2affective_gestures] ; [homepage]
  • 【IVA 2018】Evaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network [paper]
  • 【CVPR 2022】 HA2G - Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation [paper] ; alvinliu0/HA2G
  • 【SIGGRAPH Asia 2020】 Trimodal Speech gesture generation from the trimodal context of text, audio, and speaker identity [paper] ; [ai4r/Gesture-Generation-from-Trimodal-Context]
  • 【ICMI 2022】TransGesture: Autoregressive Gesture Generation with RNN-Transducer [paper]

• CNN (Convolutional Networks)

  • 【IVA 2021】 Learning Speech-driven 3D Conversational Gestures from Video [paper]

• Transformers

  • 【IEEEVR 2021】 Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents [paper] ; [UttaranB127/Text2Gestures] ; homepage

• Generative models -- not accurate -- continue edditing

  • Normalising Flows

    • 【EUROGRAPHICS 2020】 StyleGestures Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows [paper] ; [simonalexanderson/StyleGestures] ; [youtube]

  • WGAN

    • 【ICMI 2021】 Probabilistic Human-like Gesture Synthesis from Speech using GRU-based WGAN [paper]

  • VAEs

    • 【ICCV 2021】 Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders ; [paper] ; [JingLi513/Audio2Gestures] ; [homepage]
    • Freeform Body Motion Generation from Speech [paper] ; [TheTempAccount/Co-Speech-Motion-Generation] ; [youtube]
    • 【CVMP 2021】 Flow-VAE Speech-Driven Conversational Agents using Conditional Flow-VAEs [paper]

  • Learnable noise codes

    • 【ICCV 2021】 Speech Drives Templates: Co-Speech Gesture Synthesis With Learned Templates ; [paper] ; [ShenhanQian/SpeechDrivesTemplates] ;

  • CaMN BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis [paper] ; [PantoMatrix/BEAT]

• Diffusion

  • 【SIGGRAPH 2023】 Listen, denoise, action! Audio-driven motion synthesis with diffusion models [paper] ; (Code repository (coming soon)) ; [youtube] ; [homepage] ; [video]
  • 【IJCAI 2023】 DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models [paper] ; youngseng/diffusestylegesture ; [youtube]
  • 【CVPR 2023】 Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation [paper] ; [advocate99/diffgesture]

• Periodic autoencoders (DeepPhase)

  • Rhythmic Gesticulator - Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings [paper] ; [Aubrey-ao/HumanBehaviorAnimation] ; [youtube] ; [youtube]
  • 【CVPR 2023】QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation [paper] ; [YoungSeng/QPGesture] ; [video]

• Text to Gesture

  • 【CVPR 2022】 Generating Diverse and Natural 3D Human Motions from Text [paper] [homepage] ; [poster] ; [EricGuo5513/text-to-motion]

• Others (Uncategory)

  • 【SIGGRAPH 2022】 A Motion Matching-based Framework for Controllable Gesture Synthesis from Speech [paper] ; [homepage]

  • 【CVPR 2022】 SEEG - SEEG: Semantic Energized Co-Speech Gesture Generation [paper] ; [akira-l/seeg]

  • 【CVPR 2022】 DiffGAN - Low-Resource Adaptation for Personalized Co-Speech Gesture Generation [paper]

  • 【ICMI 2022】 ZeroEGGS Exemplar-based stylized gesture generation from speech: An entry to the GENEA Challenge 2022 [paper]

  • 【CVPR 2022】 Audio-Driven Neural Gesture Reenactment With Video Motion Graphs [paper]

  • 【AAMAS 2022】 Multimodal analysis of the predictability of hand-gesture properties [paper]

  • 【ICMI 2022】 GestureMaster GestureMaster: Graph-based Speech-driven Gesture Generation [paper]

  • 【ICCV 2021】 Audio2Gestures Audio2Gestures: Generating Diverse Gestures From Speech Audio With Conditional Variational Autoencoders [paper]

  • 【IVA 2021】 Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech [paper] ; [homepage]

  • 【ECCV 2020】 Mix-StAGE Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach [paper]

  • 【CVPR 2019】 Speech2Gesture Learning Individual Styles of Conversational Gesture [paper]

• Fréchet Inception Distance (FID) -
• Fréchet Gesture Distance (FGD) -
• Fréchet Template Distance (FTD) -

• Human-likeness : looks like the motion of a real human

• Appropriateness (specificity) : appropriate for the given speech, controlling for the human-likeness of the motion

  • 🧑‍🦲 : Upper-body tier || 🧍 : Full-body tier

  • 🧍‍♂️ : motion || 📃 : text || 🔊 : audio || ⚙️ : custom by teams

• Canonical correlation analysis

• Modalities type:

  • 🔊 : audio || 📃 : text || 🤯 : emotion || 🚶 : gesture motion || ℹ️ : gesture properties || 🎞️ : gesture segment

• Type

  • 👥 : Dialog (Conversation between two people 🤼) || 👤 : Monolog (Self conversation 🧍)

• Challenge dataset: GENEA Challenge 2022 Dataset Files
• 3D coordinates of submitted motion: GENEA Challenge 2022 3D coordinates of submitted motion
• Submitted BVH files: GENEA Challenge 2022 submitted BVH files
• User-study video stimuli: GENEA Challenge 2022 user-study video stimuli

• Algorithms

  • SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents [paper] ; [homepage] ; [youtube]

• Recognition:

  • OpenPose - CMU-Perceptual-Computing-Lab/openpose
  • MMPose - open-mmlab/mmpose
  • AlphaPose - MVIG-SJTU/AlphaPose

• Audio pre-processing:

• Mesh processing:

  • Utility to trim BVH files: github.com/ghenter/trim_bvh

• Visualization:

  • github.com/TeoNikolov/genea_visualizer
  • [ICMI 2021] HEMVIP: Human Evaluation of Multiple Videos in Parallel ; [jonepatr/hemvip/] ; [data and analysis code]

• The GENEA Challenge 2022 Playlist

• The GENEA Challenge 2022 Playlist

• GENEA Workshop 2022 Playlist

• GENEA 2021: Crossmodal clustered contrastive learning: Grounding of spoken language to gesture

• GENEA Workshop 2020 Playlist

• Objective evaluation code: github.com/genea-workshop/genea_numerical_evaluations
• Text-based baseline: github.com/youngwoo-yoon/Co-Speech_Gesture_Generation
• Audio-based baseline: github.com/genea-workshop/Speech_driven_gesture_generation_with_autoencoder
• Interface for subjective evaluations: jonepatr/hemvip
• Code for creating attention-check videos: youngwoo-yoon/create_attention_check
• Utility to trim BVH files: github.com/ghenter/trim_bvh
• Modified PyMO for the challenge dataset: youngwoo-yoon/PyMO

• 2000 - Paired Speech and Gesture Generation in Embodied Conversational Agents
• 2005 - Gesture Generation by Imitation: From Human Behavior to Computer Character Animation
• 2009 - Gesture in Embodied Communication and Human Computer Interaction
• 2013 - Nonverbal Communication in Human Interaction
• 2013 - Nonverbal Communication: Science and Applications

• TEDTalk (Extract skeleton from video Dataset)

  • AQ-GT: a Temporally Aligned and Quantized GRU-Transformer for Co-Speech Gesture Synthesis [paper] ; [hvoss-techfak/AQGT]
  • Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings [paper] ; [aubrey-ao/humanbehavioranimation] ; [youtube]
  • Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation [paper] alvinliu0/HA2G ; [youtube] ; [homepage]
  • Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning [paper] [UttaranB127/speech2affective_gestures] ; [homepage] ; [youtube]
  • Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity [paper] ; [ai4r/Gesture-Generation-from-Trimodal-Context]

• BEAT (Motion Capture Dataset)

  • BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis [paper] ; [PantoMatrix/BEAT]
  • Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity [paper] ; [ai4r/Gesture-Generation-from-Trimodal-Context]
  • Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders [paper]
  • Learning Individual Styles of Conversational Gesture [paper]
  • Robots Learning to Say `No': Prohibition and Rejective Mechanisms in Acquisition of Linguistic Negation [paper]

Your contributions are always welcome! Please take a look at the contribution guidelines first.

This project is licensed under the MIT License - see the LICENSE.md file for details.

OpenHuman.ai - Open Store for Realistic Digital Human