Hi All ,
It appears that we cannot clone the repo
https://github.com/ARM-software/ML-zoo.git

$ git clone https://github.com/ARM-software/ML-zoo.git
Cloning into 'ML-zoo'...
remote: Enumerating objects: 1320, done.
remote: Counting objects: 100% (66/66), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 1320 (delta 27), reused 22 (delta 22), pack-reused 1254
Receiving objects: 100% (1320/1320), 413.77 KiB | 721.00 KiB/s, done.
Resolving deltas: 100% (480/480), done.
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_fp32/testing_output/TFLite_Detection_PostProcess:1/0.npy'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

The path that git checks is models/object_detection/ssd_mobilenet_v1/tflite_fp32/testing_output/TFLite_Detection_PostProcess:1/

But the actual path seems to be,
/models/object_detection/ssd_mobilenet_v1/tflite_fp32/testing_input/normalized_input_image_tensor/

#git checkout master
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_fp32/testing_output/TFLite_Detection_PostProcess:1/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_fp32/testing_output/TFLite_Detection_PostProcess:2/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_fp32/testing_output/TFLite_Detection_PostProcess:3/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_int8/testing_output/TFLite_Detection_PostProcess:01/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_int8/testing_output/TFLite_Detection_PostProcess:02/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_int8/testing_output/TFLite_Detection_PostProcess:03/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_int8/testing_output/TFLite_Detection_PostProcess:04/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_uint8/testing_output/TFLite_Detection_PostProcess:1/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_uint8/testing_output/TFLite_Detection_PostProcess:2/0.npy'
error: invalid path 'models/object_detection/ssd_mobilenet_v1/tflite_uint8/testing_output/TFLite_Detection_PostProcess:3/0.npy'

Thanks,

In your paper titled "MICRONETS: NEURAL NETWORK ARCHITECTURES FOR DEPLOYING TINYML APPLICATIONS ON COMMODITY MICROCONTROLLERS" you also mention an anomaly detection model. However I am not able to spot it. Is it included in this repo?

@gregory-bronstain
Have you deployed transformer to TFLite micro kernel? I found that the model has different inference result with TFLite Mirco kernel than TFLite kernel with Python interface.
I also posted an issue to Tflite micro repo: tensorflow/tflite-micro#960 (comment)

Perhaps I'm missing something obvious, but the repository actually doesn't contain model files.
https://github.com/ARM-software/ML-zoo/blob/master/models/object_detection/ssd_mobilenet_v1/tflite_fp32/ssd_mobilenet_v1.tflite

This is just plan text file with the following content
version https://git-lfs.github.com/spec/v1
oid sha256:d15f8495fce308b774fd25cf4584e6f682f2d9d1b7cb6aa81cea2d784874c51b
size 27286108

The size 26 MB, but even when I press on Raw it only shows the same information as above.

• this link leads to .pb and saved model files and does not contain .tflite model
  http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz

I'm testing ARM NN and trying to understand the performance difference between FP32 and INT8 models. Having official models for the comparison would help greatly.

Hi,
I tried to test the QAT tflite model, but error reported.
ValueError: Cannot set tensor: Got value of type FLOAT32 but expected type INT8 for input 0, name: serving_default_input_1

Based on my understanding, input tensor should be 0.~1., but the model need INT8 input.
Would you give some introduction about test code for QAT model?
Thanks a lot.

I run your github code with following command:
python test.py --clean_wav_folder=clean_testset_wav/ --noisy_wav_folder=noisy_testset_wav/ --tflite_path=rnnoise_INT8.tflite

The document said Avg pesq is 2.9 but I on get 2.664

I download test dataset with provided script file.

Hi @idan-arm @phorsman-arm ,
I try to use the inference code and the tiny_wav2letter_int8.tflte to inference the same audio file, but the result is different with yours.
Your result which LER is 0.03123

My result which LER is 0.390625

My error rate is significant greater than yours.
Do you have any idea?

Thanks in advance,

OS:
Ubuntu 20.04.4 LTS

package version:
Python 3.8.10
librosa 0.8.1
numpy 1.21.0
tensorflow 2.4.1
jiwer 2.3.0

Instead of just the frozen tflite files, is it possible to upload h5 or hdf5 or pb or onnx or pt files?

Hey there,

It would be really helpful if these models had the appropriate label files to go with.

Thanks,

Hi there,

I have been working to test the model CNN Large INT8 model of Keyword Spotting in a Tensorflow Lite for Micro (TFLM) program. I got some issues when loading the input file.

When I tried to use numpy to load 0.npy, I got some errors as follows.

>>> import numpy as np
>>> np.load('0.npy')

Traceback (most recent call last):
  File "/Applications/MBEDCLI.app/Contents/Resources/miniconda/envs/ml-zoo/lib/python3.6/site-packages/numpy/lib/npyio.py", line 426, in load
    return pickle.load(fid, **pickle_kwargs)
_pickle.UnpicklingError: invalid load key, 'v'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/MBEDCLI.app/Contents/Resources/miniconda/envs/ml-zoo/lib/python3.6/site-packages/numpy/lib/npyio.py", line 429, in load
    "Failed to interpret file %s as a pickle" % repr(file))
OSError: Failed to interpret file '0.npy' as a pickle

>>> np.load('0.npy', allow_pickle=True)

Traceback (most recent call last):
  File "/Applications/MBEDCLI.app/Contents/Resources/miniconda/envs/ml-zoo/lib/python3.6/site-packages/numpy/lib/npyio.py", line 426, in load
    return pickle.load(fid, **pickle_kwargs)
_pickle.UnpicklingError: invalid load key, 'v'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/MBEDCLI.app/Contents/Resources/miniconda/envs/ml-zoo/lib/python3.6/site-packages/numpy/lib/npyio.py", line 429, in load
    "Failed to interpret file %s as a pickle" % repr(file))
OSError: Failed to interpret file '0.npy' as a pickle

I'm using Python 3.6.13 and numpy 1.12.1.

I did some research and checked 0.npy's version using command xxd:

$ xxd 0.npy
00000000: 7665 7273 696f 6e20 6874 7470 733a 2f2f  version https://
00000010: 6769 742d 6c66 732e 6769 7468 7562 2e63  git-lfs.github.c
00000020: 6f6d 2f73 7065 632f 7631 0a6f 6964 2073  om/spec/v1.oid s
00000030: 6861 3235 363a 3661 6239 3364 3438 3263  ha256:6ab93d482c
00000040: 3035 6365 3533 6664 3436 3766 3431 3964  05ce53fd467f419d
00000050: 3637 3635 3333 3531 6330 6236 6136 6434  67653351c0b6a6d4
00000060: 3436 3634 3834 6139 3034 3138 3363 3936  466484a904183c96
00000070: 3764 3736 3135 0a73 697a 6520 3230 3838  7d7615.size 2088
00000080: 0a

It seems to be v1 npy file. I'm not sure if this was the problem.

Would you mind give me some advice about how to read the input file? Also by any chance could you tell me how to load this npy file into the TFLM project which is using C/C++?

Thank you so much!

What's the performance differences between the different key word spotting models? I see accuracy differences, but no performance metrics. Would be helpful to pick a model before running it.

Hello and thanks for your work. I need to run model for Keyword Spotting from this zoo, but there is no any validation script with preprocessing for my .wav files. How should i prepare my data to apply models?

Hello,

I am trying to evaluate the quantized wav2letter model provided at:

project link

It also includes an input sample and its corresponding output:

• input: testing_input/input_2_int8/0.npy
• output: testing_output/Identity_int8/0.npy

Also, it is suggested to decode the class probabilities (output) using algorithms like beam search and explicitly it is refereed to
ctc_beam_search_decoder from tensorflow.

Accordingly, as the model is already quantized, the output tensor is dequantized using the the scales and zero_points.

• scales: 0.00390625
• zero_points: -128

Then, the dequntized output is generated as below:

deq_output = (output.astype(np.float32)-zero_points) * scales

And finally, it is passed to the decoder in order to create the sequence of letters:

# batch size: 1
# deq_output.shape after reshaping: (148, 1, 29) ==> (sequence length, batch size, # classes/characters)
# sequence_length : 148

(decoded, log_probabilities) = tf.nn.ctc_beam_search_decoder(inputs=deq_output,
                          sequence_length=[sequence_length],
                          beam_width=25,
                          top_paths=5)

tf versions: tested on both 2.4 and 2.5

However here is a sample decoded array:

(as a reminder, the alphabet is equal to "abcdefghijklmnopqrstuvwxyz' @")

 print("".join([alphabet[ch_ind] for ch_ind in np.array(decoded[0].values)])) :


 oe aie bm'abaqabaiebajakyiepaubacacua ara 

P.S. the provided model is correct as the model prediction for the provided input matches to the provided output.

====================================================================================
And here are my questions:

1. is the decoding process correct from your side?
2. what's your idea about decoded output?
3. is the given input a fake generated tensor?
4. could you please share your validated decoding process?
5. could you please share your feature extraction process and its corresponding preprocessing steps (i.e., normalization, etc.), in case that a raw audio wave is being used for inference.

Thanks in advance,