mprib / caliscope Goto Github PK
View Code? Open in Web Editor NEWMulticamera Calibration + Pose Estimation --> Open Source Motion Capture
Home Page: https://mprib.github.io/caliscope/
License: BSD 2-Clause "Simplified" License
Multicamera Calibration + Pose Estimation --> Open Source Motion Capture
Home Page: https://mprib.github.io/caliscope/
License: BSD 2-Clause "Simplified" License
Rather than frame by frame, take the comprehensive point data used for the bundle adjustment and triangulate it using various methods. Point agreement/charuco corner relative position can then be assessed.
# TODO: if file exists then change the name
This may be better described as a VideoStream object.
This is needed to play back video via synchronizer. Should be (hopefully) a quick fix...just in the middle of other things now.
Stereocalibration currently just opens an OpenCV window of paired views that
drop off when a sufficient number of points have been captured. This needs
to be integrated into the standard GUI, and perhaps additional feedback
provided. This feedback might include a calculation of the redundency in the
stereo connections. Perhaps creation of an acceptable anchor camera and a list
of all the represented camera pairs.
This should allow hitting a target frame rate in a clean and explicit manner while also reducing the number of queues floating around, simplifying the code base.
# TODO: Create FPS display on config dialog
#TODO Split out the image update to its own method that returns a modified frame
Align the method of grabbing frames for processing used in mono and stereo
calibration.
if tab is open , the stereocalibrate button should be unclickable
A decent array initialization can be achieved using stereopairs that only have 20 or so grids between them. This does not, however, appear adequete for a good bundle adjustment calibration (from the perspective of accurate registering of distances).
I believe that a good path for the process is this:
From there it's a whole new set of issues. But just to thnk through them..
results of calibration will be presented (histograms and scatter plots)
perhaps play back the post bundle adjusted data in the 3d viewer? (this would be an undertaking all its own...)
allow acceptance of the calibration
move on to set the origin (or alternatively use the last frame of the calibration to set the origin?)
I'm going to start creating project items as this is becoming a bit big...
following tutorial here:
The implementation of this in the current Jupyter notebook is a travesty. I think it's right, but it is slow...
This may become the basis for testing bundle adjusting on the basis of matching charuco corner distances if it ends up running quickly enough...
What needs to be done?
Stereocalibration parameters are not named in a standard way. Example, it could
be stereo_0_2_RMSE on one run and stereo_2_0_RMSE on the next one. This creates
duplicated records with out of synch data. Stick to a convention of always
putting the smaller number first.
#TODO: figure out how to get this to stop automatically
# TODO: Create FPS display on config dialog
I suspect this may be at the heart of the rather slow processing of frames in the entire workflow.
Pandas is a crutch here...just buckle down and work it out in numpy. Should be a small scale refactor that
could yield significant processing speed improvements.
# TODO: avg_delta_time was zero when testing on the laptop...is this necessary?
# TODO: in the future, this could perhaps get pushed to the corner tracker
fps = 0 # TODO: calculate based off of rate of getting from q
I may mock this up in a jupyter notebook reading in saved out paired_point data to make sure that I get the details of the input transformations correct. From there I can test out the performance of the triangulation code as well as whether or not it resolves the mirroring that I'm seeing in my current projected points.
Original triangulation code was based on the excellent tutorial here:
https://temugeb.github.io/opencv/python/2021/02/02/stereo-camera-calibration-and-triangulation.html
but that does not take distortion into account.
Undistorting all pixels in each frame prior to point identification is one option, though this has considerable processing overhead. Another option is to only undistort points, but it appears that there are problems with the native opencv
function to for this:
https://answers.opencv.org/question/230847/undistort-and-undistortpoints-are-inconsistent/
but that this can be resolved with a function described here:
https://yangyushi.github.io/code/2020/03/04/opencv-undistort.html
perhaps default to some simple naming convention for folders...
recording_1, recording_2, etc...
You may want to be able to go back and double check the distortion params in a previously configured setup. So if distortion is loaded, Undistort should be clickable
# TODO: track down why this is necessary
# TODO: Create FPS display on config dialog
Refactor within PointData and bundle_adjust_test.
provide point data with least squares output and a threshold value to subset the values (from _full
variables)
Additional refactor to make the PointData object as the primary input of an optimize
method. A side effect of this is that it will set the least_sq_result variable within the PointData. I'm not super happy about that...but it's starting to feel like the simplest and most straightforward way to manage things while looping through this code over and over again.
create a single folder to hold session configs used for testing... a related issue to the "use tests" issue
Need to also ignore the hidden files (i.e. ".*"). In general, clean up what gets saved.
# TODO: don't pass a csv_path...you can't really reuse this function if that is the case
Only 8 dependencies in the entire project at the moment and this is one of them. It's only used in one line of code and I believe cv2 has similar functionality.
Some references that may be useful and that you will likely want to come back to multiple times:
https://www.youtube.com/watch?v=sobyKHwgB0Y
and
https://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html
Should include
A good idea for a figure (from the Anipose paper):
Histograms of length variation and RMSE of reproj may also be way to display this. Show both before and after Bundle Adjustment. Also show how the basic parameters changed from before to after Bundle Adjustment.
# TODO: Create FPS display on config dialog
This is confusing language given that it has no relationship with "bundle adjustment." In general, that "bundle" of data may more accurately be described as a "packet" that is passed into the thread queue. This packet could be a dataclass like has been done with most recent additions to the code base where data is passed from one thread to another.
Likely place to start is stereocalibrator.store_stereo_data()
Make sure that data from all pairs is being stored at each eligible capture time for later refinement of calibration.
After an initial round of bundle adjustment, remove from the 2_d points the image points that have the worst reprojection error, then bundle adjust again. This is an implementation of a basic idea within anipose.
After that, I would be interested in re-running bundle adjustment on the complete dataset, but the parameters to optimize would be the camera intrinsics and distortion. Going iteratively back and forth in this three step process seems like it might help square away the outliers due to initially mis-estimated camera intrinsics:
repeat the above and see if results improve....
this is going deeper into the hierarchy than I would like. Consider refactoring.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.