Existing applications:

• video conference reactions (https://support.apple.com/guide/iphone/use-video-conferencing-features-iphaa0b5671d/ios): 8 gestures recognized
• Avatar animation
• online meeting camera focus for signers ("presenter" focus with signing instead of speech; forget which service has this)
• physical exercise: push-up, squat counter (https://developers.google.com/ml-kit/vision/pose-detection/classifying-poses)
• golf swing analysis

Potential applications:

• gesture-controlled smart home devices (with camera) / computer user interface in general (think "minority report")
• in-car driver monitor and gesture control
• sign language recognition (and communicating with non-signers)

MediaPipe issues to overcome:

• FPS varies from device to device (train-inference skew)
• incorrectly recognized keypoints between correctly recognized frames (smoothing? would it work well with fast movements?)
• holistic model is in the process of being updated

• Build client device-side data pipeline for processing both live webcam stream and recorded video
• Feature engineering: hand-crafted or latent representation (embedding) learning
• Models: sequence classification / sequence tagging / sequence-to-sequence generation
• Stream data decoding

• connection between client-server
  • websocket
• client
  • video, webcam modes
  • set client id
  • set server settings
  • video/webcam resolution
• server
  • asynchronous, multi-threaded
  • data format
    • client id
    • video source, dimension
    • keypoints, timestamp
    • mediapipe model version, config
    • (annotations)
  • database format