Code Monkey home page Code Monkey logo

Comments (33)

bruce987 avatar bruce987 commented on August 19, 2024 1

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

@bruce987

  • What is Vector2?
    • If this is Unity, perhaps I can not support
  • Minimal code to reproduce issue if you can provide

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

@takuya-takeuchi

Vector2 is part of Microsoft's System.Numerics. That class has Vector, Vector2, Vector3, and Vector4... and it is not part of Unity, though Unity does have its own Vector classes.

Here is the method that fails (RetrieveHandData). At this point I'm trying to get Hand movement to determine which direction it's being waved. The first argument is which hand and the second is the datum which is derived from the "StdSharedPtr" which you will see in the examples of OpenPose. It comes from the WUserOutput.cs.

    protected override void WorkConsumer(StdSharedPtr<Datum>[] datumsPtr)
    {
        try
        {
            // User's displaying/saving/other processing here
            // datum.cvOutputData: rendered frame with pose or heatmaps
            // datum.poseKeypoints: Array<float> with the estimated pose
            if (datumsPtr != null && datumsPtr.Length != 0)
            {
                var datum = datumsPtr[0].Get();

                var rightHandVector = RetrieveHandData(BodyPart.Right, datum);

                var leftHandVector = RetrieveHandData(BodyPart.Left, datum);

                ProcessData(rightHandVector, leftHandVector);

                DisplayResults(datum);
            }
        }
        catch (Exception e)
        {
            Stop();
            LogHelper.Error($"{MethodBase.GetCurrentMethod()?.Name}: {e.Message}");
        }
    }

    private static Vector2 RetrieveHandData(BodyPart whichSide, Datum datum)
    {
        var handVector = Vector2.Zero;

        try
        {
            switch (whichSide)
            {
                case BodyPart.Right:
                {
                    var handKeyPoints = datum.HandKeyPoints[1];

                    handVector = new Vector2(handKeyPoints[new[] {0, 10, 0}], // 10 = middleJoint2
                                             handKeyPoints[new[] {0, 10, 1}]);
                    break;
                }
                case BodyPart.Left:
                {
                    var handKeyPoints = datum.HandKeyPoints[0];

                    handVector = new Vector2(handKeyPoints[new[] {0, 10, 0}], // 10 = middleJoint2
                                             handKeyPoints[new[] {0, 10, 1}]);
                    break;
                }
                default:
                    handVector = default;
                    break;
            }
        }
        catch (Exception e)
        {
            LogHelper.Info($"{MethodBase.GetCurrentMethod()?.Name}: {whichSide} {e.Message}");
        }

        return handVector;
    }

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

@bruce987
One question.
Where does numer of middleJoint2 (10) come from?
I can not find this number from openpose example code.
openpose example does not show how to get hand key points,

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

@takuya-takeuchi

The number for middlejoint2 (10) comes from the OpenPose site itself. Scroll down toward the bottom to see the definition.

https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/output.md

I had to dig around to find it. They explicitly define the Body25 parts, but not for the face or hand. The show the numeric number for face and hand, but do not actually give them names like Body25. So I assign them names. The previous post shows how I retrieved the hand data. (RetrieveHandData).

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

@bruce987
Wow!!!! Great!! Thanks a lot for your works!!

OK.
Let me check program.

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

@bruce987
I checked 07_HandFromImage.

image

It works fine.
Did you set true value to Flags.Hand?

This 07_HandFromImage uses this snipet.

// Required flags to enable heatmaps
Flags.Body = 0;
Flags.Hand = true;
Flags.HandDetector = 2;

But if this is changed to this, demo crashes due to above watch window value.

// Required flags to enable heatmaps
Flags.Body = 1;
Flags.Hand = false;
Flags.HandDetector = 0;

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

Name of parts for hand.
https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/41d8c087459fae844e477dda50a6f732e70f2cb8/src/openpose/pose/poseParameters.cpp#L182

But where is 0?

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

Ok, let me check my code and look at example 7. I'm doing it a little different. This is my code that initializes the Wrapper.

    private static void ConfigureWrapper(Wrapper<Datum> opWrapper)
    {
        try
        {
            // Configuring OpenPose

            // logging_level
            OpenPose.CheckBool(0 <= Flags.LoggingLevel && Flags.LoggingLevel <= 255, "Wrong logging_level value.");
            ConfigureLog.PriorityThreshold = (Priority)Flags.LoggingLevel;
            Profiler.SetDefaultX((ulong)Flags.ProfileSpeed);
            
            // // For debugging
            // // Print all logging messages
            // ConfigureLog.PriorityThreshold = Priority.None;
            // // Print out speed values faster
            // Profiler.setDefaultX(100);

            // Applying user defined configuration - GFlags to program variables
            // producerType
            Flags.Camera = 1;
            var (producerType, producerString) = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);

            // cameraSize
            var cameraSize = OpenPose.FlagsToPoint(Flags.CameraResolution, "-1x-1");
            
            // outputSize
            var outputSize = OpenPose.FlagsToPoint(Flags.OutputResolution, "-1x-1");
            
            // netInputSize
            var netInputSize = OpenPose.FlagsToPoint(Flags.NetResolution, "-1x-1");
            
            // faceNetInputSize
            var faceNetInputSize = OpenPose.FlagsToPoint(Flags.FaceNetResolution, "368x368 (multiples of 16)");
            
            // handNetInputSize
            var handNetInputSize = OpenPose.FlagsToPoint(Flags.HandNetResolution, "368x368 (multiples of 16)");
            
            // poseMode
            var poseMode = OpenPose.FlagsToPoseMode(Flags.Body);
            
            // poseModel
            var poseModel = OpenPose.FlagsToPoseModel(Flags.ModelPose);
            
            // JSON saving
            if (!string.IsNullOrEmpty(Flags.WriteKeyPoint))
            {
                LogHelper.Info("Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json` instead.", Priority.Max);
            }

            // key point Scale
            var keypointScale = OpenPose.FlagsToScaleMode(Flags.KeyPointScale);
            
            // heatmaps to add
            var heatMapTypes = OpenPose.FlagsToHeatMaps(Flags.HeatmapsAddParts, Flags.HeatmapsAddBackground, Flags.HeatmapsAddPAFs);
            var heatMapScale = OpenPose.FlagsToHeatMapScaleMode(Flags.HeatmapsScale);
            
            // >1 camera view?
            var multipleView = Flags.Enable3D || Flags.Views3D > 1 || Flags.FlirCamera;
            
            // Face and hand detectors
            var faceDetector = OpenPose.FlagsToDetector(Flags.FaceDetector);
            var handDetector = OpenPose.FlagsToDetector(Flags.HandDetector);
            
            // disable Google Logging
            const bool enableGoogleLogging = false;

            // Initializing the user custom classes
            // GUI (Display)
            var wUserOutput = new StdSharedPtr<UserWorkerConsumer<Datum>>(new WUserOutput());

            // Add custom processing
            opWrapper.SetWorker(WorkerType.Output, wUserOutput);

            // Pose configuration (use WrapperStructPose{} for default and recommended configuration)
            Flags.NumGpu = 1;
            //Flags.NumGpuStart = 1; // throws exception
            Flags.NumberPeopleMax = 1;
            Flags.FpsMax = 60;
            var pose = new WrapperStructPose(poseMode,
                                             netInputSize,
                                             outputSize,
                                             keypointScale,
                                             Flags.NumGpu,
                                             Flags.NumGpuStart,
                                             Flags.ScaleNumber,
                                             (float)Flags.ScaleGap,
                                             OpenPose.FlagsToRenderMode(Flags.RenderPose, multipleView),
                                             poseModel,
                                             !Flags.DisableBlending,
                                             (float)Flags.AlphaPose,
                                             (float)Flags.AlphaHeatmap,
                                             Flags.PartToShow,
                                             Flags.ModelFolder,
                                             heatMapTypes,
                                             heatMapScale,
                                             Flags.PartCandidates,
                                             (float)Flags.RenderThreshold,
                                             Flags.NumberPeopleMax,
                                             Flags.MaximizePositives,
                                             Flags.FpsMax,
                                             Flags.PrototxtPath,
                                             Flags.CaffeModelPath,
                                             (float)Flags.UpsamplingRatio,
                                             enableGoogleLogging);

            // Face configuration (use op::WrapperStructFace{} to disable it)
            //Flags.Face = true;
            var face = new WrapperStructFace(Flags.Face,
                                             faceDetector,
                                             faceNetInputSize,
                                             OpenPose.FlagsToRenderMode(Flags.FaceRender, multipleView, Flags.RenderPose),
                                             (float)Flags.FaceAlphaPose,
                                             (float)Flags.FaceAlphaHeatmap,
                                             (float)Flags.FaceRenderThreshold);

            // Hand configuration (use op::WrapperStructHand{} to disable it)
            Flags.Hand = true;
            var hand = new WrapperStructHand(Flags.Hand,
                                             handDetector,
                                             handNetInputSize,
                                             Flags.HandScaleNumber,
                                             (float) Flags.HandScaleRange,
                                             OpenPose.FlagsToRenderMode(Flags.HandRender, multipleView, Flags.RenderPose),
                                             (float) Flags.HandAlphaPose,
                                             (float) Flags.HandAlphaHeatmap,
                                             (float) Flags.HandRenderThreshold);

            // Extra functionality configuration (use op::WrapperStructExtra{} to disable it)
            //Flags.Enable3D = true; // throws exception
            var extra = new WrapperStructExtra(Flags.Enable3D,
                                               Flags.MinViews3D,
                                               Flags.Identification,
                                               Flags.Tracking,
                                               Flags.IkThreads);

            // Producer (use default to disable any input)
            Flags.ProcessRealTime = true;
            var input = new WrapperStructInput(producerType,
                                               producerString,
                                               Flags.FrameFirst,
                                               Flags.FrameStep,
                                               Flags.FrameLast,
                                               Flags.ProcessRealTime,
                                               Flags.FrameFlip,
                                               Flags.FrameRotate,
                                               Flags.FramesRepeat,
                                               cameraSize,
                                               Flags.CameraParameterPath,
                                               Flags.FrameUndistort,
                                               Flags.Views3D);

            // Output (comment or use default argument to disable any output)
            var output = new WrapperStructOutput(Flags.CliVerbose,
                                                 Flags.WriteKeyPoint,
                                                 OpenPose.StringToDataFormat(Flags.WriteKeyPointFormat),
                                                 Flags.WriteJson,
                                                 Flags.WriteCocoJson,
                                                 Flags.WriteCocoJsonVariants,
                                                 Flags.WriteCocoJsonVariant,
                                                 Flags.WriteImages,
                                                 Flags.WriteImagesFormat,
                                                 Flags.WriteVideo,
                                                 Flags.WriteVideoFps,
                                                 Flags.WriteVideoWithAudio,
                                                 Flags.WriteHeatmaps,
                                                 Flags.WriteHeatmapsFormat,
                                                 Flags.WriteVideoAdam,
                                                 Flags.WriteBvh,
                                                 Flags.UdpHost,
                                                 Flags.UdpPort);

            // No GUI. Equivalent to: opWrapper.configure(op::WrapperStructGui{});
            opWrapper.Configure(pose);
            opWrapper.Configure(face);
            opWrapper.Configure(hand);
            opWrapper.Configure(extra);
            opWrapper.Configure(input);
            opWrapper.Configure(output);

            // No GUI. Equivalent to: opWrapper.configure(op::WrapperStructGui{});
            // Set to single-thread (for sequential processing and/or debugging and/or reducing latency)
            Flags.DisableMultiThread = false;
            if (Flags.DisableMultiThread)
            {
                opWrapper.DisableMultiThreading();
            }
        }
        catch (Exception e)
        {
            LogHelper.Error($"{nameof(ConfigureWrapper)} {e.Message}");
        }
    }

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

Keep in mind I'm capturing a video feed from a webcam and processing in real-time.

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

OK. But i can not find Flags.HandDetector. It could be 0 as default value.
It may be important.
Because OpenPose.FlagsToDetector(Flags.HandDetector); returns different value.

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

memo

XXXX = 0,
ThumbCarpometacarpal,
ThumbMetacarpophalangeal,
ThumbInterphalangeal,
ThumbFingerTip,
IndexMetacarpophalangeal,
IndexProximalInterphalangeal,
IndexDistalInterphalangeal,
IndexFingerTip,
MiddleMetacarpophalangeal,
MiddleProximalInterphalangeal,
MiddleDistalInterphalangeal,
MiddleFingerTip,
RingMetacarpophalangeal,
RingProximalInterphalangeal,
RingDistalInterphalangeal,
RingFingerTip,
PinkyMetacarpophalangeal,
PinkyProximalInterphalangeal,
PinkyDistalInterphalangeal,
PinkyFingerTip

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

I will change the names next week. The value 10 still works for the middle finger joint.

I tried this...
Flags.HandDetector = 2;
var handDetector = OpenPose.FlagsToDetector(Flags.HandDetector);

This causes an early exception. So I put it back to just the FlagsToDetector without setting it to 2, then it works.

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

If you hook up your webcam and run the app. While just looking at the camera, raise your hand so the hand is detected. Then move your hand toward the camera... the exception will occur.

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

I modified 11_AsynchronousCustomOutput.

        private static void ConfigureWrapper(Wrapper<Datum> opWrapper)
        {
            try
            {
+                Flags.Hand = true;
+                Flags.HandDetector = 0;

                // logging_level

image

It works fine while 1 minute running.
I tried

  • raise right hand
  • raise left hand
  • hidd right hand
  • hide left hand
  • raise both hands

But there is no problem

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

Would like my project that does all this?

What does it mean?
Do you expect that I try the above all code, right? If so, no.

Did you try to move your hand to the webcam? I use a Logitech C922 on a tripod. I found the error when I moved my hand to adjust the camera.

Yes, my webcam Microsoft LifeCam Studio Q2F-00021 can not detect hand first time.
So I tried to move hand to camera and leave from camera to detect hand by openpose.

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

oh,... I forgot to insert new [] {0, 10, 0}

please wait!!

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

@bruce987
OK, I was able to reproduce your issue.
Uhh.. it is weird.

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

App crash this scene.

image

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

@bruce987
I think this issue does not relate to distance between camera and hand.
Perhaps, it may occur when openpose does not detect any parts, face, hand or body.

After this, app crashes.

image

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

Yes the app will run without issues for a very long time, but if you move your hand to the camera... EXCEPTION! I believe it may be happening in the C++ code.

I use a fixed size queue to store 20 data points. The queue is a ConcurrentQueue so its thread safe...

public class FixedSizedConcurrentQueue<TValue> : IProducerConsumerCollection<TValue>, IReadOnlyCollection<TValue>
{
    private readonly ConcurrentQueue<TValue> queue;

    private readonly object syncObject = new object();

    public int Size { get; }

    public bool IsEmpty => queue.IsEmpty;

    public int Count => queue.Count;

    bool ICollection.IsSynchronized => ((ICollection) queue).IsSynchronized;

    object ICollection.SyncRoot => ((ICollection) queue).SyncRoot;

    public FixedSizedConcurrentQueue(int size)
    {
        queue = new ConcurrentQueue<TValue>();
        Size = size;
    }

    public FixedSizedConcurrentQueue(int size, IEnumerable<TValue> collection)
    {
        queue = new ConcurrentQueue<TValue>(collection);
        Size = size;
    }

    // Not supported until .NET Standard 2.1
    public void Clear() => queue.Clear();

    public void CopyTo(TValue[] array, int index) => queue.CopyTo(array, index);

    void ICollection.CopyTo(Array array, int index) => ((ICollection) queue).CopyTo(array, index);

    public IEnumerator<TValue> GetEnumerator() => queue.GetEnumerator();

    IEnumerator IEnumerable.GetEnumerator() => ((IEnumerable<TValue>) this).GetEnumerator();

    public TValue[] ToArray() => queue.ToArray();

    public bool TryAdd(TValue item)
    {
        Enqueue(item);
        return true;
    }

    bool IProducerConsumerCollection<TValue>.TryTake(out TValue item) => TryDequeue(out item);

    public void Enqueue(TValue obj)
    {
        queue.Enqueue(obj);
        lock (syncObject)
        {
            while (queue.Count > Size)
            {
                queue.TryDequeue(out _);
            }
        }
    }

    public bool TryDequeue(out TValue result) => queue.TryDequeue(out result);

    public bool TryPeek(out TValue result) => queue.TryPeek(out result);
}

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

You can check array property.

Crash

image

Normal

image

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024
                var left = datum.HandKeyPoints[0];
                if (left.Volume != 0)
                {
                    //left.NumberDimensions
                    OpenPose.Log($"Left hand keypoints: {left[new[] { 0, 10, 0 }]}", Priority.High);
                }

It works fine!!

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

from openposedotnet.

takuya-takeuchi avatar takuya-takeuchi commented on August 19, 2024

OK.
Will you fix this issue on your company next week?
Please close this issue.

Thanks a lot for giving happy debugging time :)

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

@takuya-takeuchi

I'm not seeing the "Close Issue" at the bottom of the issue.

from openposedotnet.

bruce987 avatar bruce987 commented on August 19, 2024

@takuya-takeuchi

I think you own this thread. I can't close.

from openposedotnet.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.