auduno / clmtrackr Goto Github PK

Hi mate,

I get some trouble using the face detection on images. Here is the process of my script:

• Launch webcam and init clmtrackr
• On button click, drawImage() of webcam in "canvas.image"
• Start clmtrackr on "canvas.image" to detect face
• Draw in "canvas.mapping"

My problem is that the first time I click on the button clmtrackr detects a face in the upper left corner of the image. If I click on the button again, everything works fine.

Do you have an idea why the first detection is not working as expected ?

Here is my script:

'use strict';

import $ from 'jquery';
import BaseView from 'common/base-view';


export default class Home extends BaseView {
  constructor() {
    super();

    this.el = '.container';

    this.events = {
      'click .js-snapshot' : '_drawImage',
      'click .js-mapping' : '_drawLoop'
    };

    // Define variables
    this._hasUserMedia = this._checkUserMedia();

    this.webcam      = document.getElementById('webcam');
    this.image       = document.getElementById('image');
    this.imageCtx    = this.image.getContext('2d');
    this.mapping     = document.getElementById('mapping');
    this.mappingCtx  = this.mapping.getContext('2d');

    // Init User Media && Canvas
    if (this._hasUserMedia) {
      this._initWebcam();
    }
  }


  /**
   * Initialize Webcam && tracker
   * @private
   */
  _initWebcam() {
    navigator.getUserMedia = this._getUserMedia();

    navigator.getUserMedia({video: true}, (stream) => {
      const object = window.URL.createObjectURL(stream);

      this.webcam.width  = window.innerWidth;
      this.webcam.height = window.innerHeight;

      this.image.width  = window.innerWidth;
      this.image.height = window.innerHeight;

      this.mapping.width  = window.innerWidth;
      this.mapping.height = window.innerHeight;

      // Init clmtrackr
      this.ctrack = new clm.tracker({stopOnConvergence: true});
      this.ctrack.init(pModel);

      // Play stream
      this.webcam.src = object;
    }, (e) => {
      console.log(e);
    });
  }


  /**
   * Draw image from video
   */
  _drawImage() {
    this.imageCtx.clearRect(0, 0, this.image.width, this.image.height);
    this.imageCtx.drawImage(this.webcam, 0, 0);

    this._drawLoop();
  }


   /**
   * Draw mapping from image
   */
  _drawLoop() {
      this.mappingCtx.clearRect(0, 0, this.mapping.width, this.mapping.height);
      this.ctrack.start(this.image);
      this.ctrack.draw(this.mapping);
  }


  /**
   * Check user media
   */
  _checkUserMedia() {
    return !!(navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia);
  }


  /**
   * Get user media
   */
  _getUserMedia() {
    return navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia;
  }
}

Thanks a lot !

Hello, I'm fascinated by your works, but I face a problem.
I download your full package of clmtrackr and try the example inside, the example is like this one:http://auduno.github.io/clmtrackr/example.html
The problem is the green mask doesn't move at all, always stands at the upper left and no coordinates showed below...
No matter how I tired I still couldn't make it. I don't know what I missed...

I was wondering how you can use an <iframe> for the video source?

Any help would be appreciated!

Hi,

First of all, thanks for the time you've spent coding this library, it's very useful.
Although, is there any way to get the name and center position of a specific object ?

I need to know the center position of the mouth for example in order to compare the values with default ones and activate some events if the values are close enough.

The problem is that we currently can't say if the x and y positions came from the mouth, the jawline, the left eye, etc... And we can't know the center position of a specific object.

Is a function like getCenterPosition('mouth') possible ?
This function could return the same array as the getCurrentPosition() function but with the x and y values of the mouth's center position or any other object.

Best regards,
Anthony

Hi,

When I use clmtracker on a video element with a fix size, drawing on a canvas, it works perfectly.
But when trying to change the video element and the canvas size for a responsive html page clmtracker draws on a small part of my canvas (looks like the original dimension of the canvas) and doesn't clear outside of it.
Here an attach image for a better understanding of what I'm describing.

Any plans or appointment to make it run on node/io?

Is it possible to use clmtrackr for smile detection?

I guess this is a feature request to enable the face deformer to work with a png with transparency as a mask. I was looking through the code and couldn't quite figure why it is not working as it is parsing the texture on as RGBA as far as I can see.

Update: read up on the differences between webgl and opengl here http://www.html5rocks.com/en/tutorials/webgl/webgl_fundamentals/#toc-webglalpha and proposed a fix. I tested that it works with both transparent png's and with all of the existing images in the face mask example.

I downloaded the clmtrackr source and looked over the code in the examples provided, but face recogntion in videos does not work (although the face recogntion for the webcam DOES work).

Also the example on the homepage also does not work for recognizing faces.

(Also note that the example pages only ran on firefox for me and did not work for chrome.)

Any help would be much appreciated!! (ive been searching for days!)

Hi, sorry for the question, i just don't understand if the library only works on browsers that support WebGL, can you just clarify this to me?

Thank you, this is an amazing project!

Hi,

thanks for your work. I'm wondering if we could detect when eyes are closed or not. Is it possible ?

Fantastic library. Detection fails when target is wearing glasses though. Surely not a very difficult fix, but as it stands, detection fails on about 50% of all users!

Hello

I guess that the camera was dark or white.
The error occurred.

a patchresponse was monotone, causing normalization to fail. Leaving it unchanged.

And later.

Uncaught TypeError: Failed to execute 'getImageData' on 'CanvasRenderingContext2D': float parameter 1 is non-finite.

sometimes the error happened.

when the error happened the camera can't detect later.

Could you tell me how to fix it?
Or is it bug of library?

Hi there,

I'm using the following model: model_pca_20_svm.js, and it consistently plots the points a few pixels above the target.

I'm not sure if this is a bug or bad code on my part. Any insight is appreciated. Thanks!

Here's a link to my code:
https://github.com/piinthecloud/p5js-demo/blob/master/live-demo.js

Hello . I'm developing some exercises to learn more about cmltrackr and have 2 questions :
• How to getting the mouth closed during track
• How do I extend the track for the entire face, like a mask.

Thank you
Diogo

Hi, is it possible to use this library to do eye tracking as well in order to know where the user is looking on the screen?

Thanks!

Thank you for your very inspiring work! I had a look at the Eigen values and I am wondering if I can get some similar data out in real time. E.g no 1 seems to be the side to side orientation of the face and no 3 and 4 the opening of the mouth. Is there a way to get the current values of these parameters from the tracked image?
I looked at the ctrack.getParameters() but it didn't seem like what I was looking for as there are 20 EigenValues but 24 values returned by getParameters.

Hi, sorry to bother you, but I was wondering if clmtrackr has a way to cut out a face from a photo; like isolate the face from a photo and return the canvas image data.
Thanks!

Could please share the model training code?

I am looking for a way to return the RGB value of a pixel inside of this library, are there any solutions?

Is there have any tutorial can let me know how to make 3d model matrix for Face Deformer?

Hey there,

Firstly, great library. And the clmtools are incredibly useful as well.

Unfortunately the impressive technical details underpinning clmtrackr and mean shifts are a bit of a black boxed system for me so I'm having difficulty understanding a custom console trace you be included re: patch response being monotone are you able to send me a very brief sentence explaining what causes this?

If it helps my code basically grabs standard html img elements, draws them to a canvas and uses faceDeformer for substitution onto a canvas

Hi. I have my camera rotated by 90deg and because of this clmtrackr can't capture my face. It just goes crazy. I've checked all of your examples. Is there any way to fix this? Thanks anyway.

When writing an application to work with Facebook photos, I discovered that images smaller than 101 pixels in either width or height cause an error. This error also happens with many images of larger sizes in Mozilla Firefox. I trekked around in the source for a while and found the source of the issue. In the 7400s and 7800s, some lines made calls to drawImage and getImageData. Occasionally, these calls are made in such a way that a zero is passed as a width or height of a rectangle, and that throws the error. This can be fixed with either a try-catch block or some Math.max(1, pyr[i_4-next_4].width) or similar. After this fix, I had it working fine in all browsers.

i am well aware that ie9 is not one of the newest browsers but many people running windows 7 are still using it. however, i understand that this is probably not a priority for anyone so i realize that this may get just closed

anyways, when trying to init the clmtrackr model (for example, a model from the examples), the IE9 in about a minute or two, while other browsers do it in a couple of miliseconds

i have done some thorough benchmarking and it seems that the problems lies when doing the 2d fast fourier transforms.

this is the bottleneck

the very root of the slow performance lies when the FFT.real_fft2d is being performed and fft1d called within it. it lasts for a couple of miliseconds but adds up massively, since it's done in a loop. google chrome, for example does it within 1/2 miliseconds

perhaps it would work with another FFT implementation faster on IE9?

note: in order to make the library work on IE9 you need to include a TypedArray from somewhere, like from here

As I would need it for a new project: is it possible to detect ears, the top part of the face above the eyes (i.e. head and / or hair), or generally speaking: the face outline?

I used getCurrentPosition() to get the audrey.jpg positions array. The array is too much different from the array in the fack_mask.html. Does the model's position array needed to be trained.
Does the position array determine how the face crop from the image? I saw draw() works well and the yellow polygonal line fits the face faultlessly.

But when I use the array as mask:

can it work out without the support of webgl? I want to perform it without browser.thanks a lot

Hey,

this is aweeeesome! (I would put there even more "e"s ;) )

One question, we created some html with which we are able to shot some photos with ios and android phones on mobile webpages.

Is it possible, after having shot the photo to use your libary to detect emotion/mood of the picture?
I saw the demo for video, so probably it also works for taken pictures?

I ask because on a iOS or android phones you are not getting access to the camera directly on the mobile website only in the native camera-app. So the detection would only work out after returning back to the safari with the safed picture, right?

Is the libary based on js/webgl?
Is it possible after the detection to use some js event to show some happy/sad/angry smiley?

Greets sight :)

Hi Auduno,
First of all i want to say thank you that you build such a incredible JS.
Your JS is actually give position of Eyes, Lips, Nose but not Hair And how i can replace the hair with exact mapping point by other custom Hairs?
Please help me if possible.

I started playing with your library, and I'm fascinated by the results my only problem is that I'm not able to use your image example. There is a js error when you click on start, so to see it working you have to manually select a face. Also if I try to manually select the whole picture the tracker fail to converge

Hi, great project!
I'm trying to use this library to detect faces that are only of a certain minimum size. Is there a way to return the face size (width/height)?

And is it possible to get other features as well, like the face features available with face OSC (mouth size, eyebrow height, etc)

Thanks!

It thinks that the top of my beard is my upper lip.
Is there anything i can do other than saving my beard?

For some applications it would be very useful to allow for manual corrections of the face feature positions and to get the matching face model parameters from them. This would allow more precise face classification, which is what I am working on.
The library can already do that from the patches positions, but I'm having troubles hacking the overridden positions there.

My approach is to override vecposvector in the line 505 so that it represents the manually set position and then let the parameters converge. Does it sound like it might work or am I completely out of track?

I would also like to disable the shape constraint since the input data is human curated but I got lost there.

Thanks!

hello, your framework is awesome thanks for sharing with everyone, and i'm sorry for my terrible english but i will try to do my best:

I tried to understand in your code what is the coordinate representation based in the ctrack.getCurrentPosition() method. I did some test for figure out but didn't work. I suppose that you are using cartesian coordinates.

with a console.log in my chrome browser i can check the values, for example the point 19 (eyebrow, in your face-image representation on your webpage), I expected with the origin 0,0 (x,y coordinate) they were in upper left on the html video tag, but the values suggest a no. The y point give me a expected value but the problem is the x value.

In that way my question is where is the origin point (0,0) ? I tested if was on upper left, upper right, and center but my values not give me a dependable value.

thanks and regards...

Is there a way to generate mosse filter for any other object or it's features like you used for nose, left_eye, right_eye etc?
I'm sure there must be some code to generate the filters but I'm unable to locate in clmtrackr and the clmtools both.

Hi again, once again great job with this library.
I am getting really good tracking of eyes and nose but I am having trouble with the opening of the mouth. If I open the mouth wide as if singing or screaming the mouth is not tracked as open.
First of all have you encountered this problem/weakness? Do you have any tips for improving this?

It keeps on showing "wait..Loading..video". Please help !

            function getPositions() {
                ctrack.start(document.getElementById('image'));
                var positions = ctrack.getCurrentPositions();
                document.getElementById('cpo').innerHTML = positions;
            }

The third line returns "Uncaught TypeError: undefined is not a function"

I am having a consistent image load problem with the face substitution example. It seems like it is an order of operations issues. My guess is that sometimes the mask image files get loaded before the .load callback is applied to them, and therefore the callback never gets triggered, and the mask canvases never get created. Sometimes they load, sometimes they don't, but so far I have not been able to find a solution.

Hi @auduno. This is more a question than an issue.

Do you have to use a Canvas with an Image rendered into it, or can this just be run on an run-of-the-mill Image tag?

Hi @auduno , hope you are well.
Does clmtrackr "tracking in video" works with all colour of faces, I mean, black, white, brown, etc? I tested it with brown faces and it had trouble finding the face. Also, if I have glasses it messes up a little bit. Do you know how can I improve this?
Thanks again :)

Hi Auduno, I just want to find faces and its elements( eyes, nose, ...) in an image using node.js Can I use clmtrackr with node.js? Or do you recommend me any other library for server side image processing?
Thanks again!

Just a note that Google Chrome's FaceTime camera access on MacBook Air (Mid 2013) has been intermittently broken for a few months (noticed working on Chrome WebLab's Sketchbot).

The problem exists as of OS X 10.8.5 (and persists in 10.9). That includes Canary (as of 34.0.1782.2).

Nothing to do your end but suggest FireFox.

For more information:

• Apple KB: MacBook Air (Mid 2013): FaceTime HD Camera may not work with some applications
• Google Products Forums > Hangouts discussion with workaround

Hi, @auduno. This is more a question, not an issue.

I need to know some things:

• the getParameters() function which values returns?
• if I have 2 photoes and tracking the facial model, can I scale the 2 face (for example size of them) to a fixed value? If yes, there is a parameter that help me to know face size? For example if face size in photo 1 is 10 and in photo 2 is 20, can I scale them to a fixed value like 50?

1. While the image demo works, I couldn't upload an image to use.

2. The video example simply didn't work at all. It fails here,

   lastError = gl.getProgramInfoLog (program);

with error "Variable undefined in strict mode"

hi @auduno using clmtrackr.js is there a way to get the 'Z' position of the face like in headtrackr.js ? I tried to use both libraries at the same time (in order to get the Z position) , but apparently clmtrackr.js gets confused when using the same video tag for both. Also the browser asks the user to "Allow" the access to the webcam 2 times.
Thanks !

Hi,

Is it possible to detect and get information about two faces at the same time?

thanks