Code Monkey home page Code Monkey logo

bioslam's Introduction

Bioslam: IMU-Based Human Skeletal Pose Estimation

What is Bioslam?

Bioslam is a C++/MATLAB toolbox for estimation of human skeletal pose from IMU data, using robust factor-graph based global optimization techniques.

Citing Bioslam

If using this software repository, please cite bioslam through the following Zenodo DOI: DOI

Core publications

If using bioslam, also please cite the following publication:

  • T. McGrath and L. Stirling, “Body-Worn IMU Human Skeletal Pose Estimation Using a Factor Graph-Based Optimization Framework,” Sensors, vol. 20, no. 23, p. 6887, Dec. 2020. [link]

  • T. McGrath and L. Stirling, “Body-Worn IMU-Based Human Hip and Knee Kinematics Estimation during Treadmill Walking,” Sensors, vol. 22, no. 7, p. 2544, Mar. 2022. [link]

Related publications

The novel hinge joint kinematic model is based on:

  • T. McGrath, R. Fineman, and L. Stirling, “An Auto-Calibrating Knee Flexion-Extension Axis Estimator Using Principal Component Analysis with Inertial Sensors,” Sensors, vol. 18, no. 6, p. 1882, Jun. 2018. [link]

The optimization backend (GTSAM) and IMU dynamics/preintegration is based on:

  • L. Carlone, Z. Kira, C. Beall, V. Indelman, and F. Dellaert, “Eliminating conditionally independent sets in factor graphs: A unifying perspective based on smart factors,” in Proceedings - IEEE International Conference on Robotics and Automation, 2014. [link]

  • C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “IMU preintegration on manifold for efficient visual-inertial maximum-a-posteriori estimation,” in Robotics: Science and Systems, 2015. [link]

Installation

Supported systems

Bioslam is tested on the following systems:

  • Ubuntu 20.04 with GCC 7, GCC 9, and Clang 9

using Boost 1.67, GTSAM 4.0.3, and Eigen 3.3.9.

On other systems, a bioslam-installed Ubuntu image can be spun up in a Docker container using a provided Dockerfile.

Required Dependencies

bioslam < v1.1

  • CMake (>= 3.17)
  • boost (>= 1.65 && <1.74)
    • Ubuntu: sudo apt-get install libboost-all-dev
  • Eigen 3.3.9+
  • hdf5
    • Ubuntu: sudo apt-get install libhdf5-serial-dev
  • gtsam 4.0.3 (note: bioslam MATLAB wrapper may have a known compatibility issue.)
  • imuDataUtils
  • HighFive >= 2.3 && < 2.8
  • Optional: Intel's Thread Building Blocks (TBB)
    • ubuntu: sudo apt install libtbb-dev

bioslam >= v1.1 && <1.2

similar to < v1.1, but:

bioslam >= 1.2

similar to v1.1, but:

Optional Recommended Dependencies

for faster performance

  • Intel MKL can offer situationally-dependent performane improvements
  • NOTE: MKL is not compatible with Eigen 3.3.4 because of a bug in Eigen. Use a newer version of Eigen.

Note: if any of these three optional dependencies are installed, you must change a corresponding GTSAM flag to use them:

  • GTSAM_WITH_EIGEN_MKL=ON Tells Eigen to use Intel MKL if available
  • GTSAM_WITH_TBB=ON Tells GTSAM to use Intel TBB if available
  • set MKL_FFTW_INCLUDE_DIR, MKL_INCLUDE_DIR, MKL_ROOT_DIR if you wanna use MKL
  • set TBB_INCLUDE_DIRS, TBB_tbb_LIBRARY_RELEASE, TBB_tbbmalloc_LIBRARY_RELEASE if you wanna use TBB

Installation Instructions

Installing library via CMake

Starting from the root of the repo:

mkdir build
cd build
cmake ..
make install # may require root privileges
make test # optional, runs unit tests

Testing

make test # remember to `make install` first!

Want more verbose output (including std::cout) while testing?

# from build/
CTEST_OUTPUT_ON_FAILURE=1 make test

Development

A convenient devcontainer configuration is provided for use with VSCode's Dev Containers extension.

Bioslam MATLAB interface

In the matlab/ directory there is an implementation of bioslam for MATLAB. This feature is experimental and prone to instability.

Building bioslam for MATLAB

Set the appropriate CMake option in bioslam's CMakelists.txt to build the MATLAB wrapper. Then in the build/ folder (or wherever you build bioslam) there will be a directory called wrap/. The contents of this directory will include a directory wrap/bioslam/+bioslam. The +bioslam/ directory must be added to your MATLAB path. Additionally, so does the file wrap/bioslam_mex/bioslam_wrapper.mexa64. These will be installed if you make install.

Required MATLAB Toolboxes

  • Bioinformatics Toolbox
  • Deep Learning Toolbox
  • Robotics System Toolbox
  • Mapping Toolbox

Notes

  • Depending on your system and MATLAB configuration, the version of libstdc++ may be different between MATLAB and your system. This can cause errors. In these cases, it might be easiest to just remove the version of libstdc++ that shipped with MATLAB, forcing MATLAB to find it on your system instead. Note: this requires it be properly added to your system environment paths (below). Related question/answer on MATLAB central
    • You can add the standard location for MATLAB with export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib/x86_64-linux-gnu/ in .bashrc
  • You will need to set your system environment variables for LD_RUN_PATH and LD_LIBRARY_PATH for MATLAB to find the built libraries for GTSAM and bioslam. Add the following lines to your .bashrc:
    • export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
    • export LD_RUN_PATH=$LD_RUN_PATH:/usr/local/lib/
  • In general, if -march=native is enabled for any libraries, subsequent libraries should pass the same build flag. Make sure that the build option BIOSLAM_BUILD_WITH_MARCH_NATIVE matches the GTSAM build option GTSAM_BUILD_WITH_MARCH_NATIVE.

Additional Information

bioslam is open sourced under the MIT license. See LICENSE for details.

bioslam's People

Contributors

tmcg0 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

bioslam's Issues

Save imuPoseEstimator results to .h5 file

Similar to lowerBodyPoseEstimator, it is desired to save results to an .h5 file instead of individual .csv files. This would make the data much more organized and compact.

As an example, see lowerBodyPoseEstimator::saveResultsToH5File()

Check and disable building of MATLAB wrapper in CMake

By default, CMake option BIOSLAM_BUILD_MATLAB_WRAPPER is set to ON. In the CMake built script, we should check to make sure the user has MATLAB installed properly on their system and set BIOSLAM_BUILD_MATLAB_WRAPPER=OFF if not.

Updating CMake for modern Boost compatibility

Previously I was using Boost 1.56 on Ubuntu 18. But in 20, the default Boost version is like 1.71. In the newer version, they changed the way CMake works with Boost. Update bioslam in order to adjust for this new approach.

This issue was fixed in GTSAM here, do exactly what this was: borglab/gtsam#145

Setting up CI

For every platform that bioslam supports, a continuous integration (CI) pipeline should be setup via GitHub actions to automate unit testing. This issue can track conversation and observations related to CI setup.

To start, I'd like to test bioslam on clean setups of Ubuntu 16, 18, and 20.

Missing OpalIMUData Function in matlab examples

Hi,
I am trying your examples in matlab. I am not sure but it seems like your from

_% construct a data file
imus=OpalIMUData(fullfile(testDataDir,'20170412-162455-Y2_TUG_5.h5'));

The function OpalIMUData is missing. Thanks in advance

Numerical derivatives inaccurate for mathutils::atan2()

See test of the jacobians numerically in testHipIntExtRotAngFactor.cpp.

The derivatives pass most of the time. But sometimes they fail, with error about 1e-3. I can't tell if that's too high in that our derivatives are incorrect. Sometimes the numerical derivatives also blow up and return unlikely values.

This issue is likely either due to either

  1. incorrect analytical jacobians or
  2. choosing ill-conditioned values for your input variables to the function for numerical derivative approximation, leading to numerical derivatives which are ill-conditioned as well. this would be a problem in the unit test.

How can I obtain a visualization of the human skeleton?

I would like to inquire which file I can run to obtain the GIF animation described in the README. Additionally, does the code provided include the part for visualizing human skeletal movement from IMU calculations? Thank you.
Your code is excellent.I would like to further develop your code to display the entire human skeleton. Thank you.

GTSAM compatibility

Currently we require GTSAM 4.0.3 because that's when numericalDerivative6 was added. In order to support all the way back to GTSAM 4.0, I think it's best to just implement our own numericalDerivative.h file with numericalDerivative1-8. If ever in the future GTSAM gets to supporting 8-factor numerical derivatives, we can deprecate that new file.

problem with return type of gtsam::noiseModels

See testNoiseModels.cpp

For some reason, all of my noise models calls (i.e., gtsam::noiseModel::Diagonal::Sigmas) are all returning shared pointers to gtsam isotropic noise models or constrained noise models (i.e., the wrong type--we want diagonal noise models). It seems to be working okay in lowerbodypoesestimator though. Figure this problem out.

Transition away from Boost in favor of modern C++

Much like the philosophy of GTSAM, bioslam would be made better by upgrading to at least C++ 17 (likely 20 for future proofing).

This issue tracks boost changes that need to be made, with reference counts in the bioslam codebase as of Dec 3, 2023.

958 references to boost:: in the codebase:

  • 11 to boost::filesystem* -> directly switchable to std::filesystem (C++ 17)
  • 39 to boost::shared_ptr -> std::shared_ptr since C++11
  • 7 to boost::dynamic_pointer_cast -> std::dynamic_pointer_cast for const lvalue reference since C++11
  • 19 to boost::static_pointer_cast -> std::static_pointer_cast for const lvalue reference since C++11
  • 30 to boost::is_base_of -> std::is_base_of since C++11
  • 314 to boost::optional -> std::optional since C++17
  • 538 to boost::none -> std::nullopt since C++17

Note: direct swap replacements will not work because GTSAM often returns boost types, not std types. And these types are not implicitly converted at compile time. Before bioslam can move away from Boost, GTSAM must move away from Boost.

Fixing broken MATLAB wrapper

Recently, bioslam was upgraded to be compatible with exactly GTSAM 4.0.3 (as opposed to my previous hacked-up version of GTSAM). This upgrade broke the MATLAB wrapper. Issues affecting fixing the MATLAB wrapper should be documented here.

  • Update MATLAB code to handle preintegrated (combined) IMU measurements and parameters.

  • this shouldn't be too hard. In GTSAM 4.0.3, it looks like the wrapper code has been updated to expose these interfaces, which is why I previously was hacking up a version of GTSAM to do exactly that. Go through and find MATLAB examples in the GTSAM repo to update your code.

  • write documentation to show users how to edit GTSAM according to my PR which gave a little more functionality that we need in the bioslam MATLAB implementation.

Known issue: timing is incorrect in C++

In C++, I use a scheme of:
double tic=clock()
double timeElapsedSec=(clock()-tic)/CLOCKS_PER_SEC
to time things in seconds.

However, the timing seems to be accurate in debug mode, but it seems to be inaccurate in release mode. Possible that the -O3 optimization flag does something funky with moving these variables around.

Consider switching to a different timing method in the standard library (if one exists?) or switch to using the boost timers (think it's in the chrono library).

Use GTSAM random objects

In testutils.h we reinvent the wheel with methods that return (possibly-biased) random quantities, like doubles, but also GTSAM objects like Pose3, Rot3, etc. GTSAM probably has methods which already do this and we should use those.

Make Bioslam available on Windows

This is a placeholder issue to remind us to allow bioslam to run on windows.

What needs to be done? As we find out more, we can document here.

Building docker image on Windows 11 failed at "/bin/sh -c make install -j4"

Platform Windows 11Pro
Docker version 20.10.17, build 100c701
Execution command: cd docker ; docker build .
Error log:

#37 5.598 /usr/local/include/highfive/bits/H5Converter_misc.hpp:83:19: error: static assertion failed: Booleans are not supported yet.
#37 5.598    83 |     static_assert(!std::is_same<type, bool>::value, "Booleans are not supported yet.");
#37 5.598       |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#37 8.713 /usr/src/bioslam/src/mathutils.cpp: In function 'Eigen::VectorXd mathutils::angleUnwrapOnKnownDomain(const VectorXd&, const double&, const double&, const double&)':
#37 8.713 /usr/src/bioslam/src/mathutils.cpp:410:49: warning: unused variable 'domainCenter' [-Wunused-variable]
#37 8.713   410 |         double domainWidth=domainMax-domainMin, domainCenter=(domainMax+domainMin)/2;
#37 8.713       |                                                 ^~~~~~~~~~~~
#37 8.792 /usr/src/bioslam/src/mathutils.cpp: In function 'Eigen::VectorXd mathutils::adjustRegressionSlopeToTargetSlope(const VectorXd&, const VectorXd&, const double&)':
#37 8.792 /usr/src/bioslam/src/mathutils.cpp:452:40: warning: unused variable 'tol' [-Wunused-variable]
#37 8.792   452 |         double newSlope, newIntercept, tol=1.0e-10;
#37 8.792       |                                        ^~~
#37 9.074 In file included from /usr/local/include/highfive/bits/H5Attribute_misc.hpp:22,
#37 9.074                  from /usr/local/include/highfive/bits/H5Annotate_traits_misc.hpp:18,
#37 9.074                  from /usr/local/include/highfive/H5File.hpp:94,
#37 9.074                  from /usr/local/include/highfive/H5Easy.hpp:58,
#37 9.074                  from /usr/src/bioslam/src/lowerBodyPoseEstimator.cpp:25:
#37 9.074 /usr/local/include/highfive/bits/H5ReadWrite_misc.hpp: At global scope:
#37 9.074 /usr/local/include/highfive/bits/H5ReadWrite_misc.hpp:106:1: error: 'HighFive::details::BufferInfo<T>::BufferInfo(const HighFive::DataType&, F) [with F = HighFive::SliceTraits<Derivate>::write(const T&) [with T = bool; Derivate = HighFive::DataSet]::<lambda()>; T = bool]', declared using local type 'HighFive::SliceTraits<Derivate>::write(const T&) [with T = bool; Derivate = HighFive::DataSet]::<lambda()>', is used but never defined [-fpermissive]
#37 9.074   106 | BufferInfo<T>::BufferInfo(const DataType& dtype, F getName)
#37 9.074       | ^~~~~~~~~~~~~
#37 9.490 make[2]: *** [CMakeFiles/bioslam.dir/build.make:63: CMakeFiles/bioslam.dir/src/lowerBodyPoseEstimator.cpp.o] Error 1
#37 9.490 make[2]: *** Waiting for unfinished jobs....
#37 10.32 /usr/src/bioslam/src/gtsamutils.cpp: In function 'uint gtsamutils::nearestIdxToVal(std::vector<double>, double)':
#37 10.32 /usr/src/bioslam/src/gtsamutils.cpp:189:16: warning: 'nearestIdx' may be used uninitialized in this function [-Wmaybe-uninitialized]
#37 10.32   189 |         return nearestIdx;
#37 10.32       |                ^~~~~~~~~~
#37 15.54 make[1]: *** [CMakeFiles/Makefile2:173: CMakeFiles/bioslam.dir/all] Error 2
#37 15.54 make: *** [Makefile:141: all] Error 2
#37 ERROR: executor failed running [/bin/sh -c make install -j4]: exit code: 2
------
 > [33/33] RUN make install -j4:
------
executor failed running [/bin/sh -c make install -j4]: exit code: 2

Creating custom h5 wrapper

The h5 file format needs of bioslam are really slim: basic reading and writing of common types to and from files. As a result, requiring HighFive as a dependency is a little overkill. I would like to remove this dependency by creating a lightweight .h5 wrapper for bioslam purposes.

Recommendation: create a namespace called h5utils which contains the few simple functions we need. h5utils should include the raw h5df cpp utilities (i.e., those included when apt install libhdf5-serial-dev is run, which is already a stated dependency of bioslam via HighFive). This would allow us to cut out the middle man, so to say.

Tracking issue: GTSAM 4.2 and boost 1.74

upgrade dependency to GTSAM 4.2 and Boost 1.74

This was the prior note on upgrading to GTSAM 4.1.1, expecting at least this to need to be done:
Notes on upgrading for GTSAM 4.1.1
Major thing that changed which will impact bioslam should be the gtsam::Point3 interface. Point3 is just typedef'd to a vector3 now. So there's no more Point3::vector() or dot or cross. These are just namespaced functions to gtsam.

  • remove all gtsam::Point3::vector() calls. Just remove the vector() part, it's already typedef'd to a Vector3.
  • Point3::cross() -> gtsam::cross()
  • Point3::dot() -> gtsam::dot()
  • Point3::norm() -> gtsam::norm3()

Updating magnetometer factor to GTSAM's new built-in `MagPoseFactor`

Background:

When I first started developing bioslam, gtsam only had a magnetometer factor which hooked into a Rot3 object, not a Pose3 object (which are the objects we use to store the pose of IMUs in bioslam). So I developed a custom factor (MagPose3Factor) to add a magnetometer factor to a Pose3 object. Recently, a pull request was merged on gtsam that added a factor which does exactly what I wanted in the first place. It adds the factor MagPoseFactor which is probably way more robust and better-tested than the one I had made. We should switch bioslam to use this.

What to do:

Swap out bioslam's custom MagPose3Factor with the new gtsam built-in MagPoseFactor.

Caveats:

  • This new GTSAM factor just got merged and won't be available in an official GTSAM release til GTSAM 4.1. So don't do this switch until GTSAM 4.1 is out and we force bioslam to require GTSAM >= 4.1.
  • Right now it's in gtsam_unstable, and I'm not sure why. Consider if we want bioslam to depend on gtsam_unstable components.
    • As of GTSAM PR borglab/gtsam#788, this factor is now in stable! Will be available whenever GTSAM 4.1 is tagged.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.