dattalab / keypoint-moseq Goto Github PK

Hi,

When I install in conda for linux/GPU the environment doesn't show up. Then, when I install with pip for linux/GPU I get the following error in jupyter notebook when I try to run "import keypoint_moseq as kpms":

2023-05-08 15:05:56.736333: W external/org_tensorflow/tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-05-08 15:05:56.773764: W external/org_tensorflow/tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-05-08 15:05:56.776625: W external/org_tensorflow/tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory

Do you have any suggestions?

Thanks,

Evan

Hi, Just started using keypoint moseq, and am getting an error when trying to load the DLC data:

TypeError                                 Traceback (most recent call last)
Cell In[31], line 6
      3 coordinates, confidences = kpms.load_deeplabcut_results(dlc_results_directory)
      5 # format data for modeling
----> 6 data, labels = kpms.format_data(coordinates, confidences=confidences, **config())

Cell In[28], line 5, in <lambda>()
      1 import keypoint_moseq as kpms
      3 project_dir = 'demo_project'
----> 5 config = lambda: kpms.load_config(project_dir)

File ~\Anaconda3\envs\keypoint_moseq\lib\site-packages\keypoint_moseq\io.py:217, in load_config(project_dir, check_if_valid, build_indexes)
    214     config = yaml.safe_load(stream)
    216 if check_if_valid: 
--> 217     check_config_validity(config)
    219 if build_indexes:
    220     config['anterior_idxs'] = jnp.array(
    221         [config['use_bodyparts'].index(bp) for bp in config['anterior_bodyparts']])

File ~\Anaconda3\envs\keypoint_moseq\lib\site-packages\keypoint_moseq\io.py:162, in check_config_validity(config)
    157     if not bodypart in config['bodyparts']:
    158         error_messages.append(           
    159             f'ACTION REQUIRED: `use_bodyparts` contains {bodypart} '
    160             'which is not one of the options in `bodyparts`.')
--> 162 for bodypart in sum(config['skeleton'],[]):
    163     if not bodypart in config['bodyparts']:
    164         error_messages.append(
    165             f'ACTION REQUIRED: `skeleton` contains {bodypart} '
    166             'which is not one of the options in `bodyparts`.')

TypeError: 'NoneType' object is not iterable

My skeleton is set to null, and my body parts are the same in the bodypart list as they are in use_bodyparts. I think the lamda function is having issues pulling the config? Thanks!

I have been working through the tutorial and I keep running into the same issue when running this line:

kpms.generate_trajectory_plots(coordinates=coordinates, name=name, project_dir=project_dir, **config())

This is the error I get.:

### ValueError                                Traceback (most recent call last)
[<ipython-input-40-07fec078becb>](https://localhost:8080/#) in <cell line: 1>()
----> 1 `kpms.generate_trajectory_plots(coordinates=coordinates,` name=name, project_dir=project_dir, **config())

4 frames
[/usr/local/lib/python3.9/dist-packages/keypoint_moseq/viz.py](https://localhost:8080/#) in generate_trajectory_plots(coordinates, results, output_dir, name, project_dir, results_path, pre, post, min_frequency, min_duration, use_reindexed, use_estimated_coords, skeleton, bodyparts, use_bodyparts, num_samples, keypoint_colormap, plot_options, sampling_options, padding, save_individually, save_gifs, save_mp4s, fps, projection_planes, **kwargs)
   1062 
   1063     sampling_options['n_neighbors'] = num_samples
-> 1064     sampled_instances = sample_instances(
   1065         syllable_instances, num_samples, coordinates=coordinates,
   1066         centroids=centroids, headings=headings, **sampling_options)

[/usr/local/lib/python3.9/dist-packages/keypoint_moseq/util.py](https://localhost:8080/#) in sample_instances(syllable_instances, num_samples, mode, pca_samples, pca_dim, n_neighbors, coordinates, pre, post, centroids, headings, filter_size)
    719             centroids=centroids, headings=headings, filter_size=filter_size
    720             ) for syllable,instances in syllable_instances.items()}
--> 721         X = np.vstack(list(trajectories.values()))
    722 
    723         if X.shape[0]>pca_samples:

/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py in vstack(*args, **kwargs)

[/usr/local/lib/python3.9/dist-packages/numpy/core/shape_base.py](https://localhost:8080/#) in vstack(tup)
    280     if not isinstance(arrs, list):
    281         arrs = [arrs]
--> 282     return _nx.concatenate(arrs, 0)
    283 
    284 

/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py in concatenate(*args, **kwargs)

ValueError: need at least one array to concatenate

I am not certain what it means.

Hi caleb, we are running into an issue with the grid movies being really cropped and not showing any of the keypoints or the full frame

but crowd movies look fine

We are running kp.moseq 0.1.0 on faces of headfixed animals (the same dataset i have been posting about this is Shawn from the stowers lab btw just switching to my own github account :) )

Hi,

I installed keypoint-moseq on Linux using pip, but after importing it with "import keypoint_moseq" I get the following error: AttributeError: module 'jax' has no attribute 'typing'". I am using jax 0.3.22 (GPU version).

"
...
File ~/.conda/envs/keypoint_moseq/lib/python3.9/site-packages/jax_moseq/utils/distributions.py:3
1 import jax, jax.numpy as jnp, jax.random as jr
2 import tensorflow_probability.substrates.jax.distributions as tfd
----> 3 from dynamax.hidden_markov_model.inference import hmm_posterior_sample
4 from jax_moseq.utils import nan_check
5 na = jnp.newaxis

File ~/.conda/envs/keypoint_moseq/lib/python3.9/site-packages/dynamax/hidden_markov_model/init.py:1
----> 1 from dynamax.hidden_markov_model.models.abstractions import HMM, HMMEmissions, HMMInitialState, HMMTransitions, HMMParameterSet, HMMPropertySet
2 from dynamax.hidden_markov_model.models.arhmm import LinearAutoregressiveHMM
3 from dynamax.hidden_markov_model.models.bernoulli_hmm import BernoulliHMM

File ~/.conda/envs/keypoint_moseq/lib/python3.9/site-packages/dynamax/hidden_markov_model/models/abstractions.py:2
1 from abc import abstractmethod, ABC
----> 2 from dynamax.ssm import SSM
3 from dynamax.types import Scalar
4 from dynamax.parameters import to_unconstrained, from_unconstrained

File ~/.conda/envs/keypoint_moseq/lib/python3.9/site-packages/dynamax/ssm.py:9
7 from jax import jit, lax, vmap
8 from jax.tree_util import tree_map
----> 9 from jaxtyping import Float, Array, PyTree
10 import optax
11 from tensorflow_probability.substrates.jax import distributions as tfd

File ~/.conda/envs/keypoint_moseq/lib/python3.9/site-packages/jaxtyping/init.py:33
30 del jax
32 # First import some things as normal
---> 33 from .array_types import (
34 AbstractArray as AbstractArray,
35 AbstractDtype as AbstractDtype,
36 get_array_name_format as get_array_name_format,
37 set_array_name_format as set_array_name_format,
38 )
39 from .decorator import jaxtyped as jaxtyped
40 from .import_hook import install_import_hook as install_import_hook

File ~/.conda/envs/keypoint_moseq/lib/python3.9/site-packages/jaxtyping/array_types.py:667
665 PRNGKeyArray = Key[jax.Array, "2"]
666 Scalar = Shaped[jax.Array, ""]
--> 667 ScalarLike = Shaped[jax.typing.ArrayLike, ""]

AttributeError: module 'jax' has no attribute 'typing'"

Thank you,
Bram

while installing keypoint-moseq from the environment file provided (environment.win64_gpu.yml), we ran into this error

The conflict is caused by:
The user requested jax-moseq==0.0.0
keypoint-moseq 0.0.1 depends on jax-moseq==0.0.1

changing the version to "jax-moseq==0.0.1" under pip solves it.

not sure if the other environment files for other OS should also be updated

I was wondering if it is possible to change the frame rate of the acquired data or change the optimal syllable duration. I am acquiring at 60fps but it looks like the duration was optimized for 30fps.

Hello, everyone! I saw change-scores appeared in the article, but I can't find the function to get the change-scores. Could you tell me how to get the change-scores? Thanks for your attention!

Hi,

With the Windows GPU version installed, all of the suggested jupyter lab code runs fine until the PCA cell, and then the kernel dies and there is no further error message.

When I run "python -c "import jax; print(jax.devices())"", I get:

[StreamExecutorGpuDevice(id=0, process_index=0)]

Does this mean the GPU is not being used?

Thanks,

Evan

I am running the code on a jupyter notebook, and the Kernel keeps dying at the step of full model fitting. My PC and GPU should be powerful enough for that (256 GB RAM and the Nvidia Geforce RTX 3080), so I am not sure what the issue is.

Hi all,
While running the Keypoint MoSeq Analysis Visualization Notebook, in the Load Progress step, we enter this info into the cell:

import keypoint_moseq as kpms
model_dirname = '2023_06_05-12_29_0' # model directory name for the model to analyze
project_dir = 'model training notebook 6-5-2023' # project directory
progress_filename = 'progress.yaml' # progress file name
progress_paths = kpms.track_progress(model_dirname, project_dir, 'progress.yaml')

However, when we run this cell, we get the following error:

AttributeError Traceback (most recent call last)
Cell In[6], line 7
4 project_dir = 'model training notebook 6-5-2023' # project directory
5 progress_filename = 'progress.yaml' # progress file name
----> 7 progress_paths = kpms.track_progress(model_dirname, project_dir, 'progress.yaml')

AttributeError: module 'keypoint_moseq' has no attribute 'track_progress'

Any idea what might be causing it? Did I enter the project directory/model directory name in the incorrect format? Thanks!

I get the following output when running interactive_group_setting:

Is that intended? Or are there supposed to be rows for the various recordings? My index.yaml file seems to be fully populated:

files:
- filename: 21_11_8_one_mouse.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: ae4f8e94-53e3-49a0-8cd6-8ae29ad63952
- filename: 21_12_10_def6a_1_1.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: 217f6b8c-cfb9-4e3a-90fc-d2d7d6c1e8ec
- filename: 21_12_10_def6a_3.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: 558adfe5-3351-4244-971c-5ad5345b2481
- filename: 21_12_10_def6b_3.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: bd58f1ed-e3c3-4bfc-8f3c-3b75e7f523bf
- filename: 21_12_2_def6a_1.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: 1d588560-104a-41c5-9c60-ce332380c8a2
- filename: 21_12_2_def6b_2.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: cadc2edb-5ccd-490d-9f89-7783470c611e
- filename: 22_04_26_cage4_0.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: 3853f96f-a62a-4af3-8a62-199de2a0688b
- filename: 22_04_26_cage4_0_2.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: e7608a8e-d6e0-43ab-a855-d4070ecfaa06
- filename: 22_04_26_cage4_1_1.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: e402beff-acb0-4e2d-9cd8-9835a80c5ee5
- filename: 22_27_04_cage4_mouse2_0.top.irDLC_resnet50_moseq_exampleAug21shuffle1_500000
  group: default
  uuid: 053f1e4e-15c4-4dc3-804d-972c22fa571a

and here's my progress.yaml file

base_dir: demo_project/
config_file: demo_project/config.yml
crowd_movie_dir: demo_project/2023_05_23-15_19_03/crowd_movies
grid_movie_dir: demo_project/2023_05_23-15_19_03/grid_movies
index_file: demo_project/index.yaml
model_checkpoint: demo_project/2023_05_23-15_19_03/checkpoint.p
model_dir: demo_project/2023_05_23-15_19_03/
model_name: 2023_05_23-15_19_03/
model_results: demo_project/2023_05_23-15_19_03/results.h5
plot_dir: demo_project/2023_05_23-15_19_03/plots
progress_filepath: demo_project/progress.yaml
syll_info_path: demo_project/syll_info.yaml
trajectory_plot_dir: demo_project/2023_05_23-15_19_03/trajectory_plots

Hi all,

When using the most up-to-date version of the tutorial notebook for model training (https://github.com/dattalab/keypoint-moseq/blob/main/docs/source/tutorial.ipynb), we've run into an error during the Load Data step. Any idea what be causing this issue? We run the following cell and receive the resulting error:

#load data (e.g. from DeepLabCut)
keypoint_data_path = 'Identity tracking test\Identity tracked keypoint videos' # can be a file, a directory, or a list of files
coordinates, confidences, bodyparts = kpms.load_keypoints(keypoint_data_path, 'sleap')
#format data for modeling
data, labels = kpms.format_data(coordinates, confidences=confidences, **config())

AttributeError Traceback (most recent call last)
Cell In[21], line 3
1 # load data (e.g. from DeepLabCut)
2 keypoint_data_path = 'Identity tracking test\Identity tracked keypoint videos' # can be a file, a directory, or a list of files
----> 3 coordinates, confidences, bodyparts = kpms.io.load_sleap_results(keypoint_data_path)
5 # format data for modeling
6 data, labels = kpms.format_data(coordinates, confidences=confidences, **config())

AttributeError: module 'io' has no attribute 'load_sleap_results'

I am receiving the following error when attempting to generate grid movies. I saw that this was an issue for crowd movies and was fixed in the most recent release (crowd movies do not produce this error for me).

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[76], line 1
----> 1 kpms.generate_grid_movies(name=name, project_dir=project_dir, coordinates=coordinates, **config())
      2 #kpms.generate_crowd_movies(name=name, project_dir=project_dir, coordinates=coordinates, **config())

File ~\anaconda3\envs\keypoint_moseq\lib\site-packages\keypoint_moseq\viz.py:856, in generate_grid_movies(results, output_dir, name, project_dir, results_path, video_dir, video_paths, rows, cols, filter_size, pre, post, min_frequency, min_duration, dot_radius, dot_color, quality, window_size, use_reindexed, coordinates, bodyparts, use_bodyparts, sampling_options, video_extension, max_video_size, **kwargs)
    852 centroids,headings = filter_centroids_headings(
    853     centroids, headings, filter_size=filter_size)
    855 if window_size is None:
--> 856     window_size = get_grid_movie_window_size(
    857         sampled_instances, centroids, headings, 
    858         coordinates, pre, post)
    860 # in practice we may need a smaller window...
    861 scaled_window_size = max_video_size/max(rows,cols)

File ~\anaconda3\envs\keypoint_moseq\lib\site-packages\keypoint_moseq\viz.py:666, in get_grid_movie_window_size(sampled_instances, centroids, headings, coordinates, pre, post, pctl, fudge_factor, blocksize)
    664 ax_distances = np.max(np.abs(all_trajectories), axis=1)
    665 window_size = np.percentile(ax_distances, pctl) * fudge_factor * 2
--> 666 window_size = int(np.ceil(window_size / blocksize) * blocksize)
    667 return window_size

ValueError: cannot convert float NaN to integer

I got the following error following your pip installation procedure:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
autograd 1.4 requires future>=0.15.2, which is not installed.

pip install future>=0.15.2 allowed me to import keypoint_moseq without the kernel image issue reported in readthedocs.
I think you might need to update the dependency list to include this package?

I got this error when loading the data. The path is correct since the config file went to the right place. I have also updated my version of keypoint moseq to v 0.1.2

We have videos of headfixed mice that are being flipped upside down during the error calibration and PCA extraction (probably because of rotational alignment) so we are wondering if that can be made optional or if the direction of alignment can be changed

Dear all,

I was trying to use keypoint-moseq to analyse 3d coordinate data generated from SLEAP-Anipose. I first used SLEAP to track the keypoints (2d) from multiple camera angles. I then generated a calibration.toml using checkerboard points annotated using DLT and generated coefficients for 3d reconstruction. I then modified this file to make it suitable for SLEAP-Anipose ( following this https://github.com/backyardbiomech/DLCconverterDLT/blob/master/DLTcameraPosition.py) and reconstructed xyz coordinates of the keypoints using the calibration function in SLEAP-anipose (https://github.com/talmolab/sleap-anipose). The resulting file was a .h5 file with tracks of shape (3 x K x 1 x N ) where 3 refers to the 3 coordinates, K is the number of keypoints 1 refers to the single individual tracked and N is the total no. of frames.

However, I now see this error when I try to plot PC's using keypoint-moseq.

Could you please direct me to what I can do to fix this and what is going wrong here?

Thank you very much!

When I run the generate trajectory plots command, I get this keyword error but I don't see anywhere I can change the keyword from fps to 'duration' in my code when I command+f 'fps'

What does this error mean? I have successfully ran other files without getting this error? I have also tried restarting and running the command multiple times

**For reference, our PC has a GPU and the following specs:

• Windows 11 Pro OS build 22000.1936
• 2th Gen Intel(R) Core(TM) i7-12700K, 3.60 GHz
• 64.0 GB installed RAM

After trying to install Keypoint-MoSeq using the conda instructions, we encountered an error in which there were multiple conflicts, preventing installation. Using the Anaconda shell, we entered the following commands and received the following errors:**

(base) C:\Users\Abdus-saboor>git clone https://github.com/dattalab/keypoint-moseq
Cloning into 'keypoint-moseq'...
remote: Enumerating objects: 1488, done.
remote: Counting objects: 100% (57/57), done.
remote: Compressing objects: 100% (34/34), done.
Receiving objects: 100% (1488/1488), 10.01 MiB | 46.18 MiB/s, done.31

Resolving deltas: 100% (1037/1037), done.

(base) C:\Users\Abdus-saboor>cd keypoint-moseq

(base) C:\Users\Abdus-saboor\keypoint-moseq>conda env create -f conda_envs\environment.win64_gpu.yml
Collecting package metadata (repodata.json): done
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
Examining conflict for python cudnn cudatoolkit pip: 25%|███████▊ | 2/8 [00:00<00:00, 18.44it/s]\failed
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
Examining conflict for cudnn cudatoolkit: 38%|███████████████▊ | 3/8 [00:00<00:00, 7.69it/s]|failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
pip -> python[version='2.7.|3.5.|3.6.|>=2.7,<2.8.0a0|>=3.10,<3.11.0a0|>=3.11,<3.12.0a0|>=3.8,<3.9.0a0|>=3.9,<3.10.0a0|>=3.7,<3.8.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0|>=3.7|>=3.6|>=3|3.4.']
pip -> setuptools -> python[version='!=3.0,!=3.1,!=3.2,!=3.3,!=3.4|>=3.8|2.7.*|>=3.6|2.7|>=3.6|>=2.7|>=3.7,<4.0|>=3.6,<4.0']

Package ucrt conflicts for:
python=3.9 -> openssl[version='>=1.1.1t,<1.1.2a'] -> ucrt[version='>=10.0.20348.0']
cudnn=8.2 -> cudatoolkit[version='>=11.0,<11.4'] -> ucrt[version='>=10.0.20348.0']
cudatoolkit=11.1 -> ucrt[version='>=10.0.20348.0']
pip -> python[version='>=3.11,<3.12.0a0'] -> ucrt[version='>=10.0.20348.0']

Package cudatoolkit conflicts for:
cudatoolkit=11.1
cudnn=8.2 -> cudatoolkit[version='10.2.|>=11.0,<11.4|11.']

Package vs2015_runtime conflicts for:
python=3.9 -> openssl[version='>=1.1.1t,<1.1.2a'] -> vs2015_runtime[version='>=14.15.26706|>=14.27.29016|>=14.29.30139|>=14.32.31332|>=14.34.31931|>=14.28.29325|>=14.29.30037']
python=3.9 -> vs2015_runtime[version='>=14.16.27012|>=14.16.27012,<15.0a0|>=14.16.27033']

Package vc conflicts for:
python=3.9 -> openssl[version='>=1.1.1t,<1.1.2a'] -> vc[version='9.*|>=14.2,<15|>=14,<15.0a0']
python=3.9 -> vc[version='>=14.1,<15|>=14.1,<15.0a0']
Note that strict channel priority may have removed packages required for satisfiability.

Hi!

I'm having some difficulty installing and importing on Windows 10. I tried both the CPU and GPU versions, and multiple installation types (conda and pip). I'm consistently getting this error
"AttributeError: module 'jax' has no attribute 'typing'"

and think there's some incompatibility between the jax and jaxlib libraries. Do you have any recommendations for how to fix this?

I've also tried on the Google Colab, but the same error pops up. Attached a picture below from the Google Colab.

Thank you for developing/maintaining kpms, it looks like a great tool.

Hi all,

We're at the syllable labeling tool step in the Keypoint MoSeq Analysis Visualization Notebook, and have encountered an interesting error. Thanks for all the help so far and it would be amazing if we could get some insight into this! We can successfully load the SLEAP .H5 files but then encounter the error "cannot convert float NaN to integer". We're trying to generate grid movies, but also get the same issue when we try crowd movies. We use this cell, and then get the following error:

#video_dir = 'dlc_project/videos'
#keypoint_data_type = 'deeplabcut'
movie_type='grid' # either 'grid' or 'crowd'
kpms.label_syllables(project_dir, model_dirname, video_dir, keypoint_data_type, movie_type=movie_type)

BokehJS 3.1.1 successfully loaded.
Loading from sleap: 100%|██████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 17.57it/s]
No grid movies found in the directory. Generating grid movies
Writing grid movies to model training notebook 6-5-2023\2023_06_05-12_29_03\grid_movies

ValueError Traceback (most recent call last)
Cell In[13], line 5
1 # video_dir = 'dlc_project/videos'
2 # keypoint_data_type = 'deeplabcut'
4 movie_type='grid' # either 'grid' or 'crowd'
----> 5 kpms.label_syllables(project_dir, model_dirname, video_dir, keypoint_data_type, movie_type=movie_type)

File ~\miniconda3\envs\keypoint_moseq\lib\site-packages\keypoint_moseq\wrappers.py:74, in label_syllables(project_dir, model_dirname, video_dir, keypoint_data_type, movie_type)
72 if len(grid_movies)==0:
73 print('No grid movies found in the directory. Generating grid movies')
---> 74 generate_grid_movies(name=model_dirname, project_dir=project_dir, coordinates=coordinates, **config_data)
75 # record the movie paths
76 grid_movies=glob(os.path.join(project_dir, model_dirname, 'grid_movies', '*.mp4'))

File ~\miniconda3\envs\keypoint_moseq\lib\site-packages\keypoint_moseq\viz.py:856, in generate_grid_movies(results, output_dir, name, project_dir, results_path, video_dir, video_paths, rows, cols, filter_size, pre, post, min_frequency, min_duration, dot_radius, dot_color, quality, window_size, use_reindexed, coordinates, bodyparts, use_bodyparts, sampling_options, video_extension, max_video_size, **kwargs)
852 centroids,headings = filter_centroids_headings(
853 centroids, headings, filter_size=filter_size)
855 if window_size is None:
--> 856 window_size = get_grid_movie_window_size(
857 sampled_instances, centroids, headings,
858 coordinates, pre, post)
860 # in practice we may need a smaller window...
861 scaled_window_size = max_video_size/max(rows,cols)

File ~\miniconda3\envs\keypoint_moseq\lib\site-packages\keypoint_moseq\viz.py:666, in get_grid_movie_window_size(sampled_instances, centroids, headings, coordinates, pre, post, pctl, fudge_factor, blocksize)
664 ax_distances = np.max(np.abs(all_trajectories), axis=1)
665 window_size = np.percentile(ax_distances, pctl) * fudge_factor * 2
--> 666 window_size = int(np.ceil(window_size / blocksize) * blocksize)
667 return window_size

ValueError: cannot convert float NaN to integer

Right now to pick a value of kappa, users have to manually perform fitting over and over again with different kappas until they achieve the target syllable durations. We should automate this by making a kappa_scan function. There are two ways to approach this:

1. We could use a simulated annealing-like approach that algorithmically generates kappa proposals and hones in on the target value (as a human would do)

2. We could use a simpler parameter-scan approach that systematically tests log-spaced kappa values between some user-specified min and max. The function could then plot the results and print the kappa value that was closest to the target, along with the median duration associated with this kappa.

I lean toward (2) since it sounds simpler to implement and understand. If we go that route, here are some additional considerations:

• Should the min, max and grid-spacing be set in the config or specified at runtime? I had originally included the relevant variables in the config (they are currently commented out).
• The number of iterations for each kappa could be fixed ahead of time, or could be determined dynamically based on when the median duration stabilizes. Maybe we could use an API similar to scikit-learn where there's a tol and a max_iter.
• The scanning function should produce a figure with two subplots:
  - one showing the median duration (y-axis) vs. fitting iteration (x-axis) for each tested kappa value (perhaps coloring the lines and including a colorbar). This will show whether the durations converged within the specified number of iterations.
  - a second plot showing median duration (y-axis) vs. kappa (x-axis), along the target duration (hline) and selected kappa value (vline). This will show whether the min, max, and spacing were sufficient to hit the target.
• Should we also save the results to disk? I'm ambivalent. Maybe it's a enough to just make the plot and save that to disk.
• To make the function idiot-proof, we could include the following checks and warnings:
  - if the target duration falls outside the range of durations achieved through the scan, we could issue a warning that prompts the user to try again and perhaps suggests a new min/max.
  - check if the median durations stabilize for each kappa value. If not, issue a warning telling the user to try again with more iterations

Once this function is written, we will need to update the tutorial notebook and colab notebook accordingly.

Thanks for the great work! I am hoping to replicate some of the results in the paper with regards to the benchmarking datasets. It is mentioned in the applying keypoint-MoSeq section of the paper that:

The one exception was $\kappa$ which we adjusted separately for each dataset to ensure a median state duration of 400ms.

I am wondering if $\kappa$ is different while training on the benchmarking datasets showed in Fig. 6.
Thank you very much!

Currently generate_grid_movies only works for 2D data because it assumes that keypoint coordinates are points in pixel space. We should address this by creating a new function generate_grid_movies_3D that takes camera calibrations as additional input.

One complication is that for each recording session, there will be multiple video files, and potentially separate calibration files. Furthermore, the format in which these are named may vary from method to method. Therefore, I propose that we make people create a table that explicitly links these together... so each row of the table would contain:

1. the name of the recording session (corresponding to a key in the results.h5 output by keypoint-moseq)
2. the path to a video file
3. the path to a calibration file
4. [optional] the path to a file that contains a list of frames indexes. this would be necessary if some video frames were thrown out during synchronization.

When generating grid movies, the caller should provide this table (or a link to the table), a string specifying which method they used (e.g. "sleap-anipose"), and a flag stating whether the camera they have chosen is a top-down view. Also, if the method is "anipose" or "sleap-anipose", the caller should provide the camera name, since this is needed to read the calibration file.

To write generate_grid_movies_3D, we need to make the following changes from generate_grid_movies

1. Get session/video correspondences from the table, rather than by string matching
2. If the user provided frame indexes, make sure that the total number of such indexes matches the length of the keypoint-MoSeq output for each session. If they didn't, then we should make a warning asking them to supply frame indexes
3. If the user specified that the camera is a top-down view, then we should perform egocentric alignment of the videos as usual. Otherwise, we should do centering but not rotation.
4. Extract calibration parameters. The format of the parameters will differ by method so we might need a few loading functions.
5. Use the calibration params to project 3D coordinates (and centroids) into the 2D image plane. I wrote a util for this in another library: https://github.com/calebweinreb/multicam-calibration/blob/127fba29a0067323a41df1d66235fe0829669fa7/multicam_calibration/geometry.py#L147

In terms of documenting this, I think it would be nice to make a separate tutorial notebook that describes data loading and visualization for people who have 3D data.

Hello, everyone! I have encountered problems while attempting to install keypoint_moseq. I have tried two different methods, pip and conda, but unfortunately, I have experienced errors with both approaches.

I attempted to use pip for installation, and the attached Image 1 shows the error message that I encountered. Subsequently, I tried using conda to install, but I also faced an issue in this case, which is illustrated in the attached Image 2.

Do you have any methods to solve it? Thanks for your attention!

It would be nice to compute the log probability of the model during each fitting iteration of fitting and plot it so that people have a sense of whether the model has converged or not. Here are a few notes on implementation:

• The log likelihood can be computed using jax_moseq.models.keypoint_slds.model_likelihood
• model_likelihood returns a dictionary that maps each variable name to the conditional probability of that variable. We could store all of these values separately, but it might be simpler to just store their sum (the total joint log prob)
• In terms of where/when to call model_likelihood, one option would be every resample iteration (and perhaps store the result in the model dictionary). Another option would be just when updating the history during model fitting (i.e. here)
• We can then plot the history of logprobs in the plot_progress function
• When plotting, there's going to be a huge increase in the first few iterations and then a slow increase after that. If the y-axis includes the full range, it may be hard to see then the slow increase becomes a plateau. Therefore, I propose that we superimpose two copies of the plot, one with the full-yaxis and one with a zoomed in y-axis (and perhaps use color coded axis ticks on the left and right sides of the subplot). Another option would be to just show the zoomed-in version.

Moving through the tutorial to load data, I keep getting this error with various h5 files I am trying. In the past I was able to run kp moseq on a data set and then all i did was create more h5 files to test and these new ones I cant get past this line with.

Despite specifying 50 iterations of model fitting, model only runs one iteration and then moves on to the next line of code. This happens with different files I have tried.

Able to generate grid movies but not crowd movies.

Is it possible when we save history to add P(Y | model) to the history?

I am running off my Mac laptop and I installed pip as such: pip install "jax[cpu]" and now I am getting this error just when running the first line of the keypoint-MoSeq walkthrough

After launching the most up-to date version of the Keypoint MoSeq Analysis Visualization Notebook, we start with the "Set up notebook for interactive widgets" step. We launched the jupyter notebbok via the Anaconda navigator using the environment called keypoint_moseq. The cell we run has these lines:

#Install and Enable widget extensions configurator
!jupyter nbextension install --py jupyter_nbextensions_configurator --sys-prefix
!jupyter nbextension enable --py --sys-prefix widgetsnbextension
!jupyter nbextension enable --py --sys-prefix --py qgrid

#Install bokeh extensions
!jupyter nbextension install --sys-prefix --symlink --py jupyter_bokeh
!jupyter nbextension enable jupyter_bokeh --py --sys-prefix

However, when we run this cell, we get the following error message:

Traceback (most recent call last):
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\Scripts\jupyter-nbextension.EXE_main.py", line 7, in
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 277, in launch_instance
return super().launch_instance(argv=argv, **kwargs)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\config\application.py", line 1043, in launch_instance
app.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 972, in start
super().start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 266, in start
self.subapp.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 702, in start
self.install_extensions()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 675, in install_extensions
full_dests = install(self.extra_args[0],
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 203, in install_nbextension_python
m, nbexts = _get_nbextension_metadata(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 1107, in _get_nbextension_metadata
m = import_item(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\utils\importstring.py", line 38, in import_item
return import(parts[0])
ModuleNotFoundError: No module named 'jupyter_nbextensions_configurator'
Traceback (most recent call last):
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\Scripts\jupyter-nbextension.EXE_main.py", line 7, in
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 277, in launch_instance
return super().launch_instance(argv=argv, **kwargs)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\config\application.py", line 1043, in launch_instance
app.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 972, in start
super().start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 266, in start
self.subapp.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 882, in start
self.toggle_nbextension_python(self.extra_args[0])
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 855, in toggle_nbextension_python
return toggle(module,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 470, in enable_nbextension_python
return _set_nbextension_state_python(True, module, user, sys_prefix,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 368, in _set_nbextension_state_python
m, nbexts = _get_nbextension_metadata(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 1107, in _get_nbextension_metadata
m = import_item(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\utils\importstring.py", line 38, in import_item
return import(parts[0])
ModuleNotFoundError: No module named 'widgetsnbextension'
Traceback (most recent call last):
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\Scripts\jupyter-nbextension.EXE_main.py", line 7, in
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 277, in launch_instance
return super().launch_instance(argv=argv, **kwargs)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\config\application.py", line 1043, in launch_instance
app.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 972, in start
super().start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 266, in start
self.subapp.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 882, in start
self.toggle_nbextension_python(self.extra_args[0])
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 855, in toggle_nbextension_python
return toggle(module,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 470, in enable_nbextension_python
return _set_nbextension_state_python(True, module, user, sys_prefix,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 368, in _set_nbextension_state_python
m, nbexts = _get_nbextension_metadata(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 1107, in _get_nbextension_metadata
m = import_item(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\utils\importstring.py", line 38, in import_item
return import(parts[0])
ModuleNotFoundError: No module named 'qgrid'
Traceback (most recent call last):
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\Scripts\jupyter-nbextension.EXE_main.py", line 7, in
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 277, in launch_instance
return super().launch_instance(argv=argv, **kwargs)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\config\application.py", line 1043, in launch_instance
app.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 972, in start
super().start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 266, in start
self.subapp.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 702, in start
self.install_extensions()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 675, in install_extensions
full_dests = install(self.extra_args[0],
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 203, in install_nbextension_python
m, nbexts = _get_nbextension_metadata(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 1107, in _get_nbextension_metadata
m = import_item(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\utils\importstring.py", line 38, in import_item
return import(parts[0])
ModuleNotFoundError: No module named 'jupyter_bokeh'
Traceback (most recent call last):
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\Scripts\jupyter-nbextension.EXE_main.py", line 7, in
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 277, in launch_instance
return super().launch_instance(argv=argv, **kwargs)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\config\application.py", line 1043, in launch_instance
app.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 972, in start
super().start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\jupyter_core\application.py", line 266, in start
self.subapp.start()
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 882, in start
self.toggle_nbextension_python(self.extra_args[0])
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 855, in toggle_nbextension_python
return toggle(module,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 470, in enable_nbextension_python
return _set_nbextension_state_python(True, module, user, sys_prefix,
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 368, in _set_nbextension_state_python
m, nbexts = _get_nbextension_metadata(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\notebook\nbextensions.py", line 1107, in _get_nbextension_metadata
m = import_item(module)
File "C:\Users\Abdus-saboor\miniconda3\envs\keypoint_moseq\lib\site-packages\traitlets\utils\importstring.py", line 38, in import_item
return import(parts[0])
ModuleNotFoundError: No module named 'jupyter_bokeh'

What might be causing this issue? Thanks!

When running the following cell of the new analysis notebook:

# Install and Enable widget extensions configurator
!jupyter nbextension install --py jupyter_nbextensions_configurator --sys-prefix
!jupyter nbextension enable --py --sys-prefix widgetsnbextension
!jupyter nbextension enable --py --sys-prefix  --py qgrid

# Install bokeh extensions
!jupyter nbextension install --sys-prefix --symlink --py jupyter_bokeh
!jupyter nbextension enable jupyter_bokeh --py --sys-prefix

I get the following error message

Traceback (most recent call last):
  File "/Users/calebweinreb/miniconda3/bin/jupyter-nbextension", line 10, in <module>
    sys.exit(main())
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 269, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 980, in start
    super().start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 258, in start
    self.subapp.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 708, in start
    self.install_extensions()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 679, in install_extensions
    full_dests = install(self.extra_args[0],
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 203, in install_nbextension_python
    m, nbexts = _get_nbextension_metadata(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 1114, in _get_nbextension_metadata
    m = import_item(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/utils/importstring.py", line 38, in import_item
    return __import__(parts[0])
ModuleNotFoundError: No module named 'jupyter_nbextensions_configurator'
Enabling notebook extension jupyter-js-widgets/extension...
      - Validating: OK
Traceback (most recent call last):
  File "/Users/calebweinreb/miniconda3/bin/jupyter-nbextension", line 10, in <module>
    sys.exit(main())
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 269, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 980, in start
    super().start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 258, in start
    self.subapp.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 888, in start
    self.toggle_nbextension_python(self.extra_args[0])
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 861, in toggle_nbextension_python
    return toggle(module,
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 474, in enable_nbextension_python
    return _set_nbextension_state_python(True, module, user, sys_prefix,
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 372, in _set_nbextension_state_python
    m, nbexts = _get_nbextension_metadata(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 1114, in _get_nbextension_metadata
    m = import_item(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/utils/importstring.py", line 38, in import_item
    return __import__(parts[0])
ModuleNotFoundError: No module named 'qgrid'
Traceback (most recent call last):
  File "/Users/calebweinreb/miniconda3/bin/jupyter-nbextension", line 10, in <module>
    sys.exit(main())
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 269, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 980, in start
    super().start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 258, in start
    self.subapp.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 708, in start
    self.install_extensions()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 679, in install_extensions
    full_dests = install(self.extra_args[0],
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 203, in install_nbextension_python
    m, nbexts = _get_nbextension_metadata(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 1114, in _get_nbextension_metadata
    m = import_item(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/utils/importstring.py", line 38, in import_item
    return __import__(parts[0])
ModuleNotFoundError: No module named 'jupyter_bokeh'
Traceback (most recent call last):
  File "/Users/calebweinreb/miniconda3/bin/jupyter-nbextension", line 10, in <module>
    sys.exit(main())
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 269, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 980, in start
    super().start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/jupyter_core/application.py", line 258, in start
    self.subapp.start()
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 888, in start
    self.toggle_nbextension_python(self.extra_args[0])
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 861, in toggle_nbextension_python
    return toggle(module,
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 474, in enable_nbextension_python
    return _set_nbextension_state_python(True, module, user, sys_prefix,
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 372, in _set_nbextension_state_python
    m, nbexts = _get_nbextension_metadata(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/notebook/nbextensions.py", line 1114, in _get_nbextension_metadata
    m = import_item(module)
  File "/Users/calebweinreb/miniconda3/lib/python3.9/site-packages/traitlets/utils/importstring.py", line 38, in import_item
    return __import__(parts[0])
ModuleNotFoundError: No module named 'jupyter_bokeh'

The module appears to be installed though, e.g. pip list | grep bokeh returns

bokeh                             2.4.3
jupyter-bokeh                     2.0.3

and also I can import jupyter_bokeh without error. I'm not sure whether this affects things downstream though. Bokeh seems to load sucessfully:

I have more data than can fit in GPU memory to train my model with. If I reload a checkpoint, can I resume fitting with a different Y and mask?

Hi, I want to know how to choose the state number. Is it automatically calculated by model ? Is there a suitable method to get a suitable state number ?

When only one group is present, the following error occurs. Probably it would be better to generate a warning explaining why multiple groups are needed for this step.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[33], line 18
     15 # significance threshold
     16 thresh=0.05
---> 18 kpms.plot_syll_stats_with_sem(stats_df, progress_paths, plot_sig = plot_sig, thresh=thresh, stat=stat, ordering=ordering, 
     19                          groups=groups, ctrl_group=ctrl_group, exp_group=exp_group, join=join, figsize=(10, 5))                          

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/analysis.py:936, in plot_syll_stats_with_sem(stats_df, progress_paths, plot_sig, thresh, stat, ordering, groups, ctrl_group, exp_group, colors, join, figsize)
    933 sig_sylls = None
    934 if plot_sig:
    935     # run kruskal wallis and dunn's test
--> 936     _, _, sig_pairs = run_kruskal(stats_df, statistic=stat, thresh=thresh)
    937     # plot significant syllables for control and experimental group
    938     if ctrl_group is not None and exp_group is not None:
    939         # check if the group pair is in the sig pairs dict

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/analysis.py:716, in run_kruskal(stats_df, statistic, n_perm, seed, thresh, mc_method)
    713 N_m, N_s = syllable_data.shape
    715 # Run KW and return H-stats
--> 716 h_all, real_ranks, X_ties = run_manual_KW_test(
    717     df_usage=df_only_stats,
    718     merged_usages_all=syllable_data,
    719     num_groups=num_groups,
    720     n_per_group=n_per_group,
    721     cum_group_idx=cum_group_idx,
    722     n_perm=n_perm,
    723     seed=seed,
    724 )
    726 # find the real k_real
    727 df_k_real = pd.DataFrame(
    728     [
    729         stats.kruskal(
   (...)
    733     ]
    734 )

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/analysis.py:545, in run_manual_KW_test(df_usage, merged_usages_all, num_groups, n_per_group, cum_group_idx, n_perm, seed)
    543 p_i = np.random.randint(n_perm)
    544 s_i = np.random.randint(N_s)
--> 545 kr = stats.kruskal(
    546     *np.array_split(
    547         merged_usages_all[perm[p_i, :], s_i], np.cumsum(n_per_group[:-1])
    548     )
    549 )
    550 assert (kr.statistic == h_all[p_i, s_i]) & (
    551     kr.pvalue == p_vals[p_i, s_i]
    552 ), "manual KW is incorrect"
    554 return h_all, real_ranks, X_ties

File ~/miniconda3/envs/keypoint_moseq/lib/python3.9/site-packages/scipy/stats/_axis_nan_policy.py:502, in _axis_nan_policy_factory.<locals>.axis_nan_policy_decorator.<locals>.axis_nan_policy_wrapper(***failed resolving arguments***)
    500 if sentinel:
    501     samples = _remove_sentinel(samples, paired, sentinel)
--> 502 res = hypotest_fun_out(*samples, **kwds)
    503 res = result_to_tuple(res)
    504 res = _add_reduced_axes(res, reduced_axes, keepdims)

File ~/miniconda3/envs/keypoint_moseq/lib/python3.9/site-packages/scipy/stats/_stats_py.py:8584, in kruskal(nan_policy, *samples)
   8582 num_groups = len(samples)
   8583 if num_groups < 2:
-> 8584     raise ValueError("Need at least two groups in stats.kruskal()")
   8586 for sample in samples:
   8587     if sample.size == 0:

ValueError: Need at least two groups in stats.kruskal()

Hi all, we are very excited to try the new keypoint moseq, but just wanted to ask some questions and maybe offer some suggestions.

1. I could not find any information on training requirements/recommendations and had to dig through slack to find that the recommended training requirements for keypoint moseq was about 20 hours of video. Does the recommended video duration change if the keypoints are higher in likelihood/confidence (say if we have brightly lit headfixed animals?)
2. Do you all have a guesstimate of what would be a good overall confidence/likehood range that keypoint-moseq could work with and when we might want to say invest more time in retraining pose estimation models or drop a bodypoint to be included while modelling?
3. Do you all have any data/idea about how deviations from that total video duration requirement might affect modelling success? (For instance, if all we can collect is 5 hours of video, would that still be sufficient if not ideal?)
4. A problem with the previous version depth-moseq is that size of the animals (say males vs females or older mice vs younger mice) was a problem, is that still an issue with keypoint-moseq (in case we wanted to combine data from both males and females)?
5. Should we blank and interpolate low confidence points for the ones we need included?
6. The noise calibration is a bit confusing to use, what does it mean by : "Continue until the regression line stabilizes.". does that mean till the slope of the line stops shifting? becomes horizontal? all the points lie close to the line?
7. Other than having prior assumptions about syllable durations, how else would you recommend optimising on kappa (i know in the previous version depth moseq, there was a way to compare syllable durations to changepoint durations to optimise kappa, but that seems to have been removed, is that method no longer valid?)
8. is the recommended number of iterations for training either the initial AR-HMM or full model the same as in the tutorial or different? (i see that the bioxirv paper uses 500 iterations for the full model
9. maybe a FAQ on the website with some of these questions and answers might be helpful, sorry if i missed something somewhere!

Currently when 3D data is passed to generate_trajectory_plots, it projects it into 2D in the x/y and x/z planes. I propose that we add an addition viz function called generate_trajectory_plots_3D that instead renders the trajectory plots in an interactive viewer with plotly.

Something like this already implemented for PCs in plot_pcs_3D, which includes a slider to switch between different PCs. We could similarly use a slider to toggle between different syllables for the trajectory plots. Also, once this is implemented, it would be nice to add it to the 3D-specific tutorial notebook mentioned in #64.

For developing this feature, it may be useful to use these 3D tracking data and modeling results.

Hi,

With the Windows GPU version installed, I keep getting the following error:

jax requires jaxlib to be installed

....when I run

import keypoint_moseq as kpms
project_dir = 'demo_project'
config = lambda: kpms.load_config(project_dir)

I uninstalled jaxlib in a previous attempt at the environment - how do I reinstall it, and why didn't the uninstall only affect the previous environment?

Thanks,

Evan

So not sure if im doing anything wrong but i have run into a strange bug. I have done the step of registering the keypoint moseq kernel. If I keypoint moseq in a jupyter notebook from its own environment though with that kernel selected as the backend it throws me this error

XlaRuntimeError: UNKNOWN: no kernel image is available for execution on the device
in external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_asm_compiler.cc(64): 'status'

However if i launch it from another environment it goes through with no issues

If you have two videos, 1.avi and 10.avi, the find_matching_videos will think both 1.avi and 10.avi will match 10.csv,

For example, my videos are in the format {camera}.{datetime}.{framenumber}.avi (Top.22181547.486027.avi), which causes the same error.

This is because of
os.path.basename(key).startswith(video)

Error:

AssertionError: More than one video matches Top.22181547.4860270DLC_dlcrnetms5_23_03_2
1_DLC_enhancedMar21shuffle1_200000.h5
(['/n/groups/datta/tim_sainburg/projects/23-02-24-chronic-
pipeline/models/dlc/23_03_21_DLC_enhanced-
tim-2023-03-21/videos/Top.22181547.486027.avi',
'/n/groups/datta/tim_sainburg/projects/23-02-24-chronic-
pipeline/models/dlc/23_03_21_DLC_enhanced-
tim-2023-03-21/videos/Top.22181547.4860270.avi'])

one fix would be to select the longest match.

e.g.

if len(matches) > 0:
  longest_match = np.argmax([len(i) for i in matches])
  matches = [matches[longest_match]]

or, if you somehow knew the DLC model name (here DLC_dlcrnetms5_23_03_2) you could look for a specific match rather than startswith

Currently the syllable labeler code seems to assume that (a) grid and crowd movies both exist; (b) the number of each is the same. I think ideally it should tolerate when both assumptions are violated. In other words, I think it should gracefully handle when there are no crowd movies, no grid movies, or none of either. If the user requested to use grid/crowd movies and none are present, it would be nice to issue a warning and explain how to generate them. Also the number of grid/crowd movies may differ since they use slightly different criteria to decide which syllables to include.

Here are the various errors I got when the assumptions were broken:

Yes grid movies but no/fewer crowd movies

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[20], line 2
      1 movie_type = 'grid' # either 'grid' or 'crowd'
----> 2 progress_paths = kpms.label_syllables(progress_paths, movie_type = movie_type)

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/wrappers.py:106, in label_syllables(progress_paths, movie_type)
    102     with open(progress_paths['progress_filepath'], 'w') as f:
    103         yaml.safe_dump(progress_paths, f, default_flow_style=False)
--> 106 labeler = SyllableLabeler(progress_paths['base_dir'], progress_paths['model_name'],
    107                           progress_paths['index_file'], movie_type, progress_paths['syll_info_path'])
    108 output = widgets.interactive_output(labeler.interactive_syllable_labeler, {'syllables': labeler.syll_select})
    109 display(labeler.clear_button, labeler.syll_select, output)

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/widgets.py:531, in SyllableLabeler.__init__(self, base_dir, model_name, index_file, movie_type, syll_info_path)
    529     self.syll_info = read_yaml(syll_info_path)
    530 else:
--> 531     self.syll_info = self._initialize_syll_info_dict(self.model_name)
    532     # Write to file
    533     with open(self.syll_info_path, 'w') as f:

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/widgets.py:570, in SyllableLabeler._initialize_syll_info_dict(self, movie_dir)
    566 grid_movie_files = sorted(glob(os.path.join(self.base_dir, self.model_name, 'grid_movies',
    567                           '*.mp4')), key=lambda x: int(os.path.basename(x).split('.')[0][8:]))
    568 crowd_movie_files = sorted(glob(os.path.join(self.base_dir, self.model_name,
    569                            'crowd_movies', '*.mp4')), key=lambda x: int(os.path.basename(x).split('.')[0][8:]))
--> 570 return {i: {'label': '', 'desc': '', 'movie_path': [grid_movie_files[i], crowd_movie_files[i]], 'group_info': {}} for i in range(len(grid_movie_files))}

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/widgets.py:570, in <dictcomp>(.0)
    566 grid_movie_files = sorted(glob(os.path.join(self.base_dir, self.model_name, 'grid_movies',
    567                           '*.mp4')), key=lambda x: int(os.path.basename(x).split('.')[0][8:]))
    568 crowd_movie_files = sorted(glob(os.path.join(self.base_dir, self.model_name,
    569                            'crowd_movies', '*.mp4')), key=lambda x: int(os.path.basename(x).split('.')[0][8:]))
--> 570 return {i: {'label': '', 'desc': '', 'movie_path': [grid_movie_files[i], crowd_movie_files[i]], 'group_info': {}} for i in range(len(grid_movie_files))}

IndexError: list index out of range

No grid movies

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
Cell In[32], line 2
      1 movie_type = 'grid' # either 'grid' or 'crowd'
----> 2 progress_paths = kpms.label_syllables(progress_paths, movie_type = movie_type)

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/wrappers.py:106, in label_syllables(progress_paths, movie_type)
    102     with open(progress_paths['progress_filepath'], 'w') as f:
    103         yaml.safe_dump(progress_paths, f, default_flow_style=False)
--> 106 labeler = SyllableLabeler(progress_paths['base_dir'], progress_paths['model_name'],
    107                           progress_paths['index_file'], movie_type, progress_paths['syll_info_path'])
    108 output = widgets.interactive_output(labeler.interactive_syllable_labeler, {'syllables': labeler.syll_select})
    109 display(labeler.clear_button, labeler.syll_select, output)

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/widgets.py:525, in SyllableLabeler.__init__(self, base_dir, model_name, index_file, movie_type, syll_info_path)
    523 self.movie_type = movie_type
    524 self.syll_info_path = syll_info_path
--> 525 self.video_dims = video_dims
    527 # check if syllable info file exists
    528 if os.path.exists(syll_info_path):

UnboundLocalError: local variable 'video_dims' referenced before assignment

When I ran

layout='circular' # could be circular or spring
kpms.plot_transition_graph_difference(groups, trans_mats,usages, layout=layout)

I got the following output

It'd be helpful to have some more links to the docs on the GitHub repo.

Right now the docs sit at: https://dattalab.github.io/moseq2-website/readthedocs/keypoint-moseq

Not the easiest URL to remember, though I'm assuming you're planning on eventually hosting it on ReadTheDocs?

In any case, this URL could be added to the website field in the GitHub repository settings so it shows up on the top-right of the repo. A nice prominent link in the README wouldn't be bad either!

I get the following error when running the bigram widget without grid movies in my model directory.

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[43], line 2
      1 normalize='bigram'
----> 2 trans_mats, usages, groups = kpms.generate_transition_matrices(progress_paths, normalize=normalize)
      3 kpms.visualize_transition_bigram(groups, trans_mats, normalize=normalize)

File ~/Dropbox (HMS)/caleb weinreb/PROJECTS/MOUSE_SURVEILANCE/22_3_1_keypoint_MOSEQ/official_pipelines/keypoint-moseq/keypoint_moseq/analysis.py:1248, in generate_transition_matrices(progress_paths, normalize, syll_key)
   1245     print('No syllable movies or index file found. Please generate syllable movies and the index file first, and make sure the paths are recorded in progress.yaml.')
   1246 else:
   1247     # get the number of max syllable to include
-> 1248     max_syllables = len(os.listdir(progress_paths.get('grid_movie_dir')))
   1249     if max_syllables == 0:
   1250         print('No syllable movies found. Please generate syllable moives and make sure the path is recorded in progress.yaml.')

FileNotFoundError: [Errno 2] No such file or directory: 'demo_project/2023_03_16-15_50_11/grid_movies'

When dattalab/jax-moseq#23 goes through, we should bump the pinned jax-moseq version in setup.cfg and also add a parallel_message_passing flag. If the flag is None, serial vs. parallel should be determined automatically depending whether the caller has a access to a GPU.