mprib / caliscope Goto Github PK

Rather than frame by frame, take the comprehensive point data used for the bundle adjustment and triangulate it using various methods. Point agreement/charuco corner relative position can then be assessed.

        # TODO: if file exists then change the name

https://github.com/mprib/calicam/blob/1de11f8958abaee977100434a831929aec3d0951/src/recording/video_recorder.py#L91

This may be better described as a VideoStream object.

        # TODO: Start Here Mac

https://github.com/mprib/calicam/blob/cb9de910fffb5ef01f87e0e4c1cafcd6add4f8b4/src/cameras/camera_array.py#L128

This is needed to play back video via synchronizer. Should be (hopefully) a quick fix...just in the middle of other things now.

Stereocalibration currently just opens an OpenCV window of paired views that
drop off when a sufficient number of points have been captured. This needs
to be integrated into the standard GUI, and perhaps additional feedback
provided. This feedback might include a calculation of the redundency in the
stereo connections. Perhaps creation of an acceptable anchor camera and a list
of all the represented camera pairs.

This should allow hitting a target frame rate in a clean and explicit manner while also reducing the number of queues floating around, simplifying the code base.

        # TODO: Create FPS display on config dialog

https://github.com/mprib/calicam/blob/8baf95643c10743387b10195571c56b9b7a8904c/src/gui/camera_config_dialogue.py#L249

        #TODO Split out the image update to its own method that returns a modified frame

https://github.com/mprib/calicam/blob/3e4b9a71c10fb9b3f5474b0affb97566f786159c/src/calibration/monocalibrator.py#L57

Align the method of grabbing frames for processing used in mono and stereo
calibration.

if tab is open , the stereocalibrate button should be unclickable

A decent array initialization can be achieved using stereopairs that only have 20 or so grids between them. This does not, however, appear adequete for a good bundle adjustment calibration (from the perspective of accurate registering of distances).

I believe that a good path for the process is this:

• collect at least 100 grids per stereopair (or whatever the ceiling will be)
• display a pair-grid that has counts or colors to indicate how many each pair has
• when the ceiling is reached, move that pairs display to the bottom
• allow auto-termination if all ceilings reached, alternatively
• user can attempt to proceed with calibration data as is
• upon terminating,
  • determine camera intrinsics based on a subset of the collected images
  • stereocalibraition for all pairs using the first 20 (or whatever) grids to initialize the camera array
• bundle adjustment will proceed with the full set of data
  • all frames during recording session will be saved for processing

From there it's a whole new set of issues. But just to thnk through them..

• results of calibration will be presented (histograms and scatter plots)

• perhaps play back the post bundle adjusted data in the 3d viewer? (this would be an undertaking all its own...)

• allow acceptance of the calibration

• move on to set the origin (or alternatively use the last frame of the calibration to set the origin?)

I'm going to start creating project items as this is becoming a bit big...

following tutorial here:

https://realpython.com/pypi-publish-python-package/

The implementation of this in the current Jupyter notebook is a travesty. I think it's right, but it is slow...

This may become the basis for testing bundle adjusting on the basis of matching charuco corner distances if it ends up running quickly enough...

What needs to be done?

• unpack least_squares_result.x to get camera parameters
• reshape parameters to be one camera per row
• update each camera with the revised intrinics

Stereocalibration parameters are not named in a standard way. Example, it could
be stereo_0_2_RMSE on one run and stereo_2_0_RMSE on the next one. This creates
duplicated records with out of synch data. Stick to a convention of always
putting the smaller number first.

                    #TODO: figure out how to get this to stop automatically

https://github.com/mprib/calicam/blob/2c16a8a9191e0ddb978db7d624d673b0682d934d/src/triangulate/array_triangulator.py#L97

        # TODO: Create FPS display on config dialog

https://github.com/mprib/calicam/blob/8baf95643c10743387b10195571c56b9b7a8904c/src/gui/camera_config_dialogue.py#L249

        I suspect this may be at the heart of the rather slow processing of frames in the entire workflow.
        Pandas is a crutch here...just buckle down and work it out in numpy. Should be a small scale refactor that
        could yield significant processing speed improvements.

https://github.com/mprib/calicam/blob/ed5244a8a1a985e290c900f372bed05869069b0b/src/triangulate/paired_point_stream.py#L58

        # TODO: avg_delta_time was zero when testing on the laptop...is this necessary?

https://github.com/mprib/calicam/blob/e050e9064b8c54be98f178128f2301e4fc2839e0/src/cameras/video_stream.py#L50

                    # TODO: in the future, this could perhaps get pushed to the corner tracker

https://github.com/mprib/calicam/blob/10bf2123f37b9ef3dea5db7c5808fb7b8ff36ff3/src/triangulate/paired_point_stream.py#L150

                fps = 0 # TODO: calculate based off of rate of getting from q

https://github.com/mprib/calicam/blob/3e4b9a71c10fb9b3f5474b0affb97566f786159c/src/gui/frame_emitter.py#L52

I may mock this up in a jupyter notebook reading in saved out paired_point data to make sure that I get the details of the input transformations correct. From there I can test out the performance of the triangulation code as well as whether or not it resolves the mirroring that I'm seeing in my current projected points.

Original triangulation code was based on the excellent tutorial here:

https://temugeb.github.io/opencv/python/2021/02/02/stereo-camera-calibration-and-triangulation.html

but that does not take distortion into account.

Undistorting all pixels in each frame prior to point identification is one option, though this has considerable processing overhead. Another option is to only undistort points, but it appears that there are problems with the native opencv function to for this:

https://answers.opencv.org/question/230847/undistort-and-undistortpoints-are-inconsistent/

but that this can be resolved with a function described here:

https://yangyushi.github.io/code/2020/03/04/opencv-undistort.html

perhaps default to some simple naming convention for folders...
recording_1, recording_2, etc...

You may want to be able to go back and double check the distortion params in a previously configured setup. So if distortion is loaded, Undistort should be clickable

        # TODO: track down why this is necessary 

https://github.com/mprib/calicam/blob/417f90d3113a97010e27411221551b1a1e432bc1/src/calibration/stereocalibrator.py#L240

        # TODO: Create FPS display on config dialog

https://github.com/mprib/calicam/blob/8baf95643c10743387b10195571c56b9b7a8904c/src/gui/camera_config_dialogue.py#L249

Refactor within PointData and bundle_adjust_test.

provide point data with least squares output and a threshold value to subset the values (from _full variables)

Additional refactor to make the PointData object as the primary input of an optimize method. A side effect of this is that it will set the least_sq_result variable within the PointData. I'm not super happy about that...but it's starting to feel like the simplest and most straightforward way to manage things while looping through this code over and over again.

create a single folder to hold session configs used for testing... a related issue to the "use tests" issue

Need to also ignore the hidden files (i.e. ".*"). In general, clean up what gets saved.

# TODO: don't pass a csv_path...you can't really reuse this function if that is the case

https://github.com/mprib/calicam/blob/7e8d5138b553fa2caf9070e0237391afba86e7e6/src/calibration/bundle_adjustment/bundle_adjust_functions.py#L98

Only 8 dependencies in the entire project at the moment and this is one of them. It's only used in one line of code and I believe cv2 has similar functionality.

Some references that may be useful and that you will likely want to come back to multiple times:

https://www.youtube.com/watch?v=sobyKHwgB0Y

and

https://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html

Should include

• the reprojection error
• the agreement between triangulated point distances and actual charcoal distances...
• the "spread" of a simple 3d point triangulation

A good idea for a figure (from the Anipose paper):

Histograms of length variation and RMSE of reproj may also be way to display this. Show both before and after Bundle Adjustment. Also show how the basic parameters changed from before to after Bundle Adjustment.

        # TODO: Create FPS display on config dialog

https://github.com/mprib/calicam/blob/8baf95643c10743387b10195571c56b9b7a8904c/src/gui/camera_config_dialogue.py#L249

This is confusing language given that it has no relationship with "bundle adjustment." In general, that "bundle" of data may more accurately be described as a "packet" that is passed into the thread queue. This packet could be a dataclass like has been done with most recent additions to the code base where data is passed from one thread to another.

Likely place to start is stereocalibrator.store_stereo_data()

Make sure that data from all pairs is being stored at each eligible capture time for later refinement of calibration.

After an initial round of bundle adjustment, remove from the 2_d points the image points that have the worst reprojection error, then bundle adjust again. This is an implementation of a basic idea within anipose.

After that, I would be interested in re-running bundle adjustment on the complete dataset, but the parameters to optimize would be the camera intrinsics and distortion. Going iteratively back and forth in this three step process seems like it might help square away the outliers due to initially mis-estimated camera intrinsics:

1. Optimize 6dof based on full data set
2. Optimize 6dof after removing outliers with high reprojection error
3. Optimize Intrinsics (camera matrix and distortion) on complete dataset

repeat the above and see if results improve....

this is going deeper into the hierarchy than I would like. Consider refactoring.

https://github.com/mprib/calicam/blob/09d13d19800000da6101fc8306c42c9ab4cffde5/src/calibration/monocalibrator.py#L52