analogdevicesinc / ai8x-training Goto Github PK

What Should I change in order to process only 6 labels and not all 35 ?

Hello,

I would like to create my own model in PyTorch and port it to the MAX78000FTHR.
However, I don't exactly understand how to deal with the YAML architecture files (especially processors).

Is there any strategy on how to deal with the processors when the model contains parallel branches (3 residual connections)?
Is it possible to create a script that automatically generates the YAML file according to the PyTorch implementation of the model? This would really make the embedding process easier.

Thanks for the help!

Hello i was iwondering if training the model using recorded samples from the max microphone will increase the accuracy ?
I mean training the model with same microphone it will be tested on in real life ?
If yes can you provide us with necessary steps to do so ?

From the readme, normalization with ai8x.normalization should set the training/test data to range [-128, 128]. But, in some of the scripts eg datasets/imagenet.py, it says to map the data to this range [-128/128, +127/128]. Which one should we use?

Also, Is there a function to unnormalize this process so I can view the input image? I am using this: (128 + (img.detach().cpu().numpy()) + 1 ).astype(np.uint8) where my img is :
img = ai8x.normalize(torch.transform.ToTensor(PIL.Image.open(img_path)))

I got an index error after I ran the following line of code:
python gen_vggface2_embeddings.py -r <path_to_vggface2_dataset_folder> -d <path_to_store_generated_dataset> --type train

IndexError: The shape of the mask [58] at index 0 does not match the shape of the indexed tensor [59, 4] at index 0.

Hi,

I am having this issue while running all the KWS training examples:

ResourceWarning: unclosed file <_io.BufferedReader name=48>
Exception ignored in: <_io.FileIO name=46 mode='rb' closefd=True>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/audioread/__init__.py", line 86, in available_backends
    if ffdec.available()

I have synced up the repository and updated it to the latest version of the distiller. Also, this error remains while using the --cpu option.
Any help on this issue would be great.
Thanks!

Hello running the .sh file to train the kws20 model got me to this error :
pylogger = PythonLogger(msglogger, log_1d=True)
init() got an extra argument 'log_1d'

Hello,

I posted this as a comment on #135 but it's closed so I didn't know if it would be seen.

I'm training an object-detection model and I'm experiencing a related issue. When I add --save-sample, my mAP goes to 0.0000. Attached (ex.1 , ex.2) are some of my tests. Additionally, I tried the changed train file in PR #136 and I am still receiving a lower mAP than I have in previous continuations (ex.3). I also have a control example with the unedited SVHN_74_simplified with the changed train.py (ex.4).

When I use the samples from the different train.py's in the synthesis quantize and generation, they receive the same output which I have in ex.4. They are both using checkpoints from the same previous svhn training session with the original training script.

Are there any differences with object detection in this way?

Thanks for the help!

sample_examples.zip

Hello,

I create custemize model:

Then, I trained the model but I can't do it Synthesis part. After tranning these files were created:

I used this command line:
python quantize.py proj/best.pth.tar proj/proj_q8.pth.tar --device MAX78000
proj_q8.pth.tar folder is created.

I wrote the yaml file for my custom model:

After that, I runned this command line:
python ai8xize.py --verbose --test-dir demos --prefix doruk-kws20 --checkpoint-file proj/best.pth.tar --config-file networks/kws20-doruk.yaml --device MAX78000 --compact-data --mexpress --softmax
But result:

Can you help me? What am I doing wrong?

Is it possible to use a different activation function than relu/abs? I would like to use Mish/Swish/Leaky Relu. Is it possible to use these for the Max78002

Hello,

when using the --save-sample parameter in the training script, the console output seems to be incorrect.

I use the following parameters in my script within the scripts folder.

#!/bin/sh python train.py --epochs 200 --save-sample 15 --confusion --optimizer Adam --lr 0.001 --deterministic --compress schedule_kws20.yaml --model ai85kwslednet --dataset KWS_LED --confusion --device MAX78000 "$@"

This results in the following output during training

...
Epoch: [0][ 1000/ 1016]    Overall Loss 0.957286    Objective Loss 0.957286                                        LR 0.001000    Time 0.848247    
Epoch: [0][ 1010/ 1016]    Overall Loss 0.954101    Objective Loss 0.954101                                        LR 0.001000    Time 0.848627    
Epoch: [0][ 1016/ 1016]    Overall Loss 0.952487    Objective Loss 0.952487    Top1 76.271186    LR 0.001000    Time 0.848052    
--- validate (epoch=0)-----------
28888 samples (256 per mini-batch)
==> Saving sample at index 15 to sample_kws_led.npy
==> Best [Top1: 0.000   Sparsity:0.00   Params: 1280 on epoch: 0]
Saving checkpoint to: logs/2022.03.18-164941/checkpoint.pth.tar


Training epoch: 259997 samples (256 per mini-batch)
Epoch: [1][   10/ 1016]    Overall Loss 0.661458    Objective Loss 0.661458                                        LR 0.001000    Time 1.023104    
Epoch: [1][   20/ 1016]    Overall Loss 0.648343    Objective Loss 0.648343                                        LR 0.001000    Time 0.911043
...

Compared to the output of the same training script, when the --save-sample argument is omitted:

...
Epoch: [0][ 1010/ 1016]    Overall Loss 0.954101    Objective Loss 0.954101                                        LR 0.001000    Time 0.836035    
Epoch: [0][ 1016/ 1016]    Overall Loss 0.952487    Objective Loss 0.952487    Top1 76.271186    LR 0.001000    Time 0.835643    
--- validate (epoch=0)-----------
28888 samples (256 per mini-batch)
Epoch: [0][   10/  113]    Loss 0.664605    Top1 70.507812    
Epoch: [0][   20/  113]    Loss 0.684539    Top1 71.464844    
Epoch: [0][   30/  113]    Loss 0.696856    Top1 71.315104    
Epoch: [0][   40/  113]    Loss 0.694776    Top1 71.152344    
Epoch: [0][   50/  113]    Loss 0.690818    Top1 70.812500    
Epoch: [0][   60/  113]    Loss 0.698156    Top1 70.957031    
Epoch: [0][   70/  113]    Loss 0.693342    Top1 70.987723    
Epoch: [0][   80/  113]    Loss 0.685269    Top1 71.127930    
Epoch: [0][   90/  113]    Loss 0.684183    Top1 71.219618    
Epoch: [0][  100/  113]    Loss 0.685887    Top1 71.183594    
Epoch: [0][  110/  113]    Loss 0.681640    Top1 71.214489    
Epoch: [0][  113/  113]    Loss 0.677707    Top1 71.209499    
==> Top1: 71.209    Loss: 0.678

==> Confusion:
[[  900    50    43    17    44]
 [   84   591   336    10    54]
 [   28   193   756     5    78]
 [   34     5     2   873   123]
 [ 1339   719   710  4443 17451]]

==> Best [Top1: 71.209   Sparsity:0.00   Params: 1280 on epoch: 0]
Saving checkpoint to: logs/2022.03.18-171610/checkpoint.pth.tar


Training epoch: 259997 samples (256 per mini-batch)
Epoch: [1][   10/ 1016]    Overall Loss 0.661458    Objective Loss 0.661458                                        LR 0.001000    Time 1.138955    
Epoch: [1][   20/ 1016]    Overall Loss 0.648343    Objective Loss 0.648343                                        LR 0.001000    Time 0.996905    
Epoch: [1][   30/ 1016]    Overall Loss 0.657448    Objective Loss 0.657448                                        LR 0.001000    Time 0.947590    
Epoch: [1][   40/ 1016]    Overall Loss 0.665805    Objective Loss 0.665805                                        LR 0.001000    Time 0.921748    
...

Is this the expected behaviour?

BR,
Jonas

I want to test a model with my dataset. (Not train only test) . I trained a model and record it. But i could not handle test. How to do that?
How do i test an trained model.

Hi everyone,

I am working on keyword detection. So, I trained model like your repository. I created the test audio files. After Model trained, I need to see accuracy. Thats why I should make Confision Matrix. But I trained model, I saved like that;
torch.save(model.state_dict(), PATH)

I couldn't load like that;
model.load_state_dict(torch.load(PATH))

Would help me please? After solving this problem. I want to know How I load test dataset into the custom model?

Hello again,
I was following these instructions : https://github.com/MaximIntegratedAI/MaximAI_Documentation/blob/master/Guides/Making%20Your%20Own%20Audio%20and%20Image%20Classification%20Application%20Using%20Keyword%20Spotting%20and%20Cats-vs-Dogs.md

I wanted to add a new label to the dataset and completed all the mentioned steps, When I build the code i faced this problem.
" .bin_storage' will not fit in region FLASH' "
What did I wrong ?

Running train_svhn_tinierssd.sh with --enable-tensorboard flag raises TypeError. Running the script as it is doesn't run into this issue. Here is a screenshot of the terminal

I installed the requirements with cuda-enabled

Hello,

I am working on KWS. I need to ask something about training.

First of all, How do we choose this value? Does it have any meaning?
And in KWS, input shape is (514,64) why it is not (128,128)?

Second, when I use this script => scripts/train_kws20_v3.sh .Training is happenning like below.

But when I use this arguments:

"args"   : [
                    "--epochs", "50", 
                    "--optimizer", "Adam", 
                    "--lr", "0.005", 
                    "--deterministic", 
                    "--compress", "policies/schedule_kws20.yaml", 
                    "--model", "ai85kws20netv3",                     
                    "--dataset", "KWS_20", 
                    "--confusion", 
                    "--device",  "MAX78000",
                    "--enable-tensorboard",    
                ]

There is no training

Only difference is --wd from arguments, script is using --wd. But why? Can you explain please?

Upon following the instructions given in the README file in ai8xtraining/datasets/face_id (https://github.com/MaximIntegratedAI/ai8x-training/tree/master/datasets/face_id), I got an Unbound local error: local variable 'box' referenced before assignment after I ran the following line of code:
python gen_vggface2_embeddings.py -r <path_to_vggface2_dataset_folder> -d <path_to_store_generated_dataset> --type train

It seems like there's an issue in one of the python files inside the facenet_pytorch folder.
Thanks.

Hello Whenever I run Scripts/train_kws20_v3.sh i get this error "ResourceWarning: unclosed file <_io.BufferedReader name=12>" right after downloading and extracting the labels.
Any Ideas ? thanks in advance.

Hi,

I am currently evaluating the faceid model. I understand it uses VGGFace2 as training and validation datasets and YouTubeFace as a test dataset.

I wonder what task the model is doing? Face detection or face classification? Why does the evaluation script use MSE as a measurement for accuracy?

Thanks.

In the fpndetector.py, there are 2 functions for the classification and regression net that use wide_scores and wide_locations. what are these variables and why do we need them here, and if we change the input to the model (using a different image modality eg not RGB, or making the image bigger, does this change the need for the wide_scores or wide_locations?

In the FPN definition we have:

        self.classication_net = ClassificationModel(num_classes=self.num_classes, num_anchors = self.num_anchors, feature_size=self.feature_size, wide=wide_scores, **kwargs)
        self.regression_net = RegressionModel(feature_size=self.feature_size, num_anchors=self.num_anchors, wide=wide_locations, **kwargs)

Also these 2 functions return the conv wide feature

    def are_locations_wide(self):
        """
        Returns whether model uses wide outputs for the regression part (box locations)
        """
        return self.regression_net.conv5.wide

    def are_scores_wide(self):
        """
        Returns whether model uses wide outputs for the classification part (box predictions)
        """
        return self.classication_net.conv5.wide

Hello,
Reading this in
Data Loader Design for MAX78000 Model Training : Example: def AISegment352_get_datasets(data, load_train=True, load_test=True).
I was not able to understand what the argument data is.
Especially that I saw in many examples this line of code : ( data_dir , args ) = data
Am I obliged to do so if I wanna develop my own dataloader.

Hello! I would like to ask how to accomplish multi-label (not multi-class) classification using the train.py script. Indeed, I have an image processing task with 14 output classes that are not mutually exclusive, so where multiple labels can be true at the same time (i.e. person smiling, wearing a hat, having long hair, etc.).

The only flag I saw to change the loss function was "--regression", but nothing else - basically my question is how to remove the softmax on the outputs, and only apply a sigmoid on each one of the 14 output values.

Thank you!

Can you give me the mAP and final loss numbers obtained from the PASCAL VOC's fpn detector model's mAP and loss? Also, is there a pretrained model (pretrained on imagenet) for the resnet, or for any of the object detection models? I would like to do transfer learning with an imagenet pretrained model rather than training everything from scratch.

Hello,
I was using pyttsx3 to create synthetized words in order to create a new label in my dataset and getting to the augmentation phase i faced this issue any idea on how to solve it ?
Please note that i change the rate to 16000 using : engine.setProperty("rate", 16000).
THANKS IS ADVANCE ❤️

I want to use ai85net20 model and i changed dimension on train file and model file (dimension = 128,128). They are not match with this model. I set dimension 0 and 1 index and i set num_channels dimension[2]. Then i got this error:
" Expected 4-dimensional input for 4-dimensional weight [15, 1, 3, 3], but got 3-dimensional input of size [1, 128, 128] instead". Why dimension is not match ?

I am trying to do a test train of the feature pyramid network model and I am receiving assert errors. I am receiving one for the associated train and evaluation scripts. Screenshots of the outputs below. The extra line below "Training epoch: ...." is print(train_loader) since I initially assumed the issue was there.

For evaluate_catsdogs, I receive the same assert error, however I do not have any errors when training.

I initially did this in my preexisting Anaconda train environment (training) using 3.8.11 after updating with the new requirements.txt. When I received this error, I made a fresh install of the repo in a new directory and used a new Anaconda environment using python 3.8.11. I installed the requirements.txt and made sure to run the install for distiller as well. I am working on native Windows 10 without CUDA enabled. I am also using the newest update of the toolchain MaximSDK (updated as of today, 7/7/23) and I am working with the rest of the msdk from the Develop repo. I see that other people have posted about using FPN so I am unsure where this could be happening since I have tried a clean install.

Thank you!

how do I start multigpu training? I have tried to use --gpus 0,1,2 or set the CUDA_VISIBLE_DEVICES=0,1,2. However, the model appears to only be running on the first specified gpu (in this case 0). Am I using this correctly?

I saw the description of the ai8x-training which mentioned the neural network layers of max78000 up to 32, but the max78000 datasheet said it can up to 64 layers, so which is the right one? Thank you.

Best regards,
Jason

Hi, I am currently trying to run the training and evaluation scripts of faceid. They only work when I set batch size 1, and other sizes will give me this tensor size mismatch error.

e.g. with batch size 100

Traceback (most recent call last):
File "train.py", line 1794, in
main()
File "train.py", line 438, in main
return evaluate_model(model, criterion, test_loader, pylogger, activations_collectors,
File "train.py", line 1495, in evaluate_model
top1, _, _, _, mAP = test(test_loader, model, criterion, loggers, activations_collectors,
File "train.py", line 958, in test
top1, top5, vloss, APs, mAP = _validate(test_loader, model, criterion, loggers, args)
File "train.py", line 1163, in _validate
classerr.add(output.data.permute(0, 2, 3, 1).flatten(start_dim=0,
File "/opt/conda/lib/python3.8/site-packages/torchnet/meter/msemeter.py", line 21, in add
self.sesum += torch.sum((output - target) ** 2)
RuntimeError: The size of tensor a (512) must match the size of tensor b (51200) at non-singleton dimension 1

Could you please give me some help on how to fix this? Thanks.

In general, HWC is faster since each memory read can deliver data to up to four processors in parallel. On the other hand, four processors must share one data memory instance, which reduces the maximum allowable dimensions of the input layer.

Are the data memory instances all pointing to a single data memory (which is single-port) in the accelerator? (Is it also same for the weight/bias mem?)
I thought each group (of 4 processors) had separate memory.
That being said, for each clock, only one of the groups is getting data from memory?

Hello,

I am working on tensorflow kws20. I need to ask something about training. According to the README.md I have completed the installation. But in the training part, it downloads the dataset and then does not train it.

Why could it be? Am I missing something about installation?

I trained FashionMNIST with --use-bias and a bias was added to every layer.
I trained Cifar-10 with --use bias and only the fully connected layer had a bias added.

(gdb) monitor reset halt
SWD DPIDR 0x2ba01477
target halted due to debug-request, current mode: Thread
xPSR: 0x81000000 pc: 0x00002120 msp: 0x20003ff0
(gdb) c
Continuing.

Program stopped.
0x00002120 in ?? ()
(gdb) Quit

In the final deployment phase, it will stop automatically when running the C file

I am looking into building a custom dataloader,
I am following this documentation : Data Loader Design for MAX78000 Model Training

coming to this part : 'Expected Data Range
For training, input data is expected to be in the range . When evaluating quantized weights, or when running on hardware, input data is instead expected to be in the native MAX7800X range of [-128, +127].

As described in the following sections, the data loader function takes the data path and some arguments as input arguments. The arguments field includes two required fields, act_mode_8bit and truncate_testset. When set to True, the first argument refers to the case normalization should be done correctly for the native MAX7800X range, i.e., to range [-128, +127]. When set to False, the normalization should be in the range of for training. '

1/ I was not able to understand the role of truncate_testset.

2/ Dataloaders will take part in training phase and eval phase how can i change that argument for each phase.

3/ After building my own dataloader what is the next step in order to make my own model trained and deployed on MAX78000

Thanks in advance, A LOT !

how do I modify the network yaml in ai8x-synthesis/networks folder for the ai8x_fpndetector.py model trained on pascal voc? I am using the same model but the number of channels input was updated from 3 to 1.

I am able to train the model with ai8x-training, and evaluate the quantized version. However, when running gen_demos_...sh I am getting an error that Layer52 (class_preds_from_32_40) : 12 output channels using 1 pass (12 processors per pass), but the processor output map 0xffffffff[etc] has 64 bits instead of the expected number of 12)

Can you point me to which part of the yaml for the network should be updated with a change in the class number? Also, is there a program that generates this yaml file?

Hey I launched the train_kws20_v3.sh file and it took a while "gathering dataset from existing raw data" as it mentioned, reaching 20/35 it crashed and showed a "killed" message. Now i want to resume the process from that logs folder what can I do ?

Hi,
I ran train.py in windows 10 naitive enviroment and got an error: ModuleNotFoundError: No module named 'resource'.

The cause was the added import resource.

I commented out the import resource related lines (L140-L144) and it works as before.
https://github.com/MaximIntegratedAI/ai8x-training/blob/6ad19921c68bbd1a1e192b329666e79b2e71abce/train.py#L63

Hi there!

I'm trying to understand how weights are multiplied with the input from previous layer in the CNN accelerator.
So far I understand that, as you can see in the image below taken from the mnist examples, in the kernel map is explained how many kernels each layer takes into account when doing the convolution. In the example below I'm talking about the last layer.

From the kernel map of the last layer you can understand that the input size is 12 (which is correct) and the output channels would be 18 when in fact are just 10.Is that an error?
I read the things just mentioned above form here:

In addition I took the weights in the weights.c file and translate from hex numbers to mod2 numbers with the formula given in the README. Then I tried to convolve them with the flatten layer 192x1x1 but the results don't add up.

These are the first kernel of the last layer's weights turn into mod2:

weights_last=np.array([-28,12,-7,-107,44,94,46,-69,-1,-53,-68,-16,9,-3,9,11,-56,-56,-69,-11,-43,-8,0,9,59,-26,-18,-50,12,7,31,19,-7
,30,-3,20,28,-5,-31,25,-30,-58,-43,8,-10,16,8,17,-4,31,28,-40,72,22,22,12,81,12,-62,43,-31,-49,-89,42,-33,61,77,21,-22,19,-26,-15,
-16,-18,-7,-3,65,-3,-55,-25,40,71,63,62,-118,55,56,79,-119,-116,-72,-116,74,95,50,-121,84,97,99,51,111,-54,42,124,-60,-73,-5,125,8
8,74,59,33,40,51,33,39,-1,-49,-102,-5,36,-4,-19,-6,-94,-128,-31,-58,70,53,22,-10,-15,-121,-71,77,-78,-88,-128,-108,-76,-88,-128,-
128,90,29,51,60,57,84,49,22,25,0,0,25,43,20,74,38,0,0,81,83,-23,14,81,-1,36,11,73,37,-47,52,109,37,-17,-11,-40,-13,108,49,-46,-23,1
26,127,49,55,-5,80,31,80])

I tried to multiply them with the flatten layer, which is:

flatten = [127, 127, 127 , 68 ,127, 127, 127 ,127 , 62,  24,  11,   0 ,  0 ,  0 ,  0 ,  0,  11 ,  8,  25,  58,  94, 105,  52 , 98 , 99 ,127,  
38 , 43,  60, 127 , 36 ,  0, 127 ,127 ,127 ,110 ,113, 107,  86,  59,   0  , 0  , 0 , 34 ,  0  , 0  , 0, 29  , 0 , 15  , 3  ,36  , 0  , 0  , 0 , 61  , 0  , 0 
,  0 ,  0  , 0  , 0  , 0  , 0  , 0  , 0  , 0 , 42 , 17   ,0  , 0  , 0 , 59 , 91 ,  0 ,  0 , 49 , 45  , 0  , 0 , 52 , 15 , 33 , 57  , 0  , 0 , 22  , 0  , 0  , 9  , 0  
, 0 , 46  , 2, 0 , 24, 127, 127, 109  , 0, 127, 127, 127  , 0 , 89  ,64 , 46 , 52 , 56 , 73, 127, 115, 127, 127 , 93 , 66, 127, 127, 105 , 10, 127 , 
81  , 0  , 0 , 56  , 0 , 12 , 51  , 0 ,  0,  32  , 0  , 0  , 0  , 0  , 0 , 44, 127, 109, 127 , 48, 127, 127, 127 ,  0 , 34 , 47 , 35  , 0  , 0  , 0 , 53 , 52  
, 0  , 0  , 0  , 0  , 0   ,0  , 0 , 82, 100 ,126, 101, 104, 127, 127 , 20, 127, 127, 127 , 59 , 86, 127, 127 , 29  ,61 ,127, 127, 127, 111, 127, 127, 
127, 127, 127 , 74  , 0 ,113 , 86  , 2  , 0]

But the result are very different from the one in the log.txt file, which is:

-77992   

and to me the output was:

99297

When instead I multiply and sum the flatten layer (mentioned above) with the output channel 0 of the last layer it result something similar:

output_channel_0=np.array([-16,44,94,46,-69,-28,12,-7,-107,-69,-11,9,-3,9,11,-1,37,36,11,73,37,-23,14,81,-1,-46,-23,-17,-
11,-40,-13,-47,-22,-17,64,82,-11,-10,65,113,1,44,115,126,41,74,126,104,17,2,-11,-28,-14,-72,-74,-45,-53,-116,-124,-34,-8,-
69,-75,-25,70,40,-11,-46,41,19,-120,-115,35,30,-118,-122,23,13,-80,-73,40,89,104,40,62,106,109,-15,52,119,75,-28,52,73,
37,0,12,13,33,8,-15,-9,-10,-54,-60,-96,-74,-71,-15,-56,-47,-26,4,82,112,94,-89,-22,124,125,-126,-126,73,101,-114,-83,69,
69,8,-37,-34,17,45,-17,-49,5,5,-6,-29,17,-16,-77,-84,-23,-29,-42,-35,13,-42,-54,-56,-12,-14,-32,23,78,-34,1,94,115,0,6,5,
-9,-48,-57,-36,-38,-58,-57,-23,-45,-33,-42,2,10,-23,9,60,4,-28,1,32,-34,-45,-50,-28,-42,-47,-63,-13,-14])

result:

-80808
# if I add bias:
-80808 +22 = -80786

Which make no sense to me because the output channel 0 should be the output of the last layer in the first channel, right?

My final goal is to update the weights and add neurons to the last layer with an Online learning technique, so I need to understand how the weights are being used.
In the README is explained that the multiplication with a flatten layer occur with the first weights that match the length of the input flatten array.
Example: the layer is 192x1x1 long and is multiply with the first 192 mod2 weights taken from the first 192/4=48 hex values.
Is it correct?
Can you explain to me how the last layer convolve with the weights and how the results are compute?

Thank you!

While creating a custom model I tried to use the "groups" argument for a ConvTranpose2d layer but it had no effect. I checked the ai8x.py code and apparently the argument is missing as shown in the following:

I just want to confirm if the groups argument is not present for a reason or if I just add it manually I shouldn't have problems when it comes to synthetizing the model.

Hi

I think there might be a problem in the MaxPool1D class. Especially in the keyword arguments naming going up/down the class hierarchy.

In the Conv1D class if Pooling is not None there is an assertion at the end to check that stride == 1. Since the default value for stride is 3 an explicit setting is needed. Setting stride when using the MaxPool1D seems impossible as the stride keyword in the MaxPool1D gets used as the pooling_stride(line 1262).

I could not find where the CNN weights and biases are initialized. Could you direct me there? and are the weights and biases in the fpndetector.py all initialized to 0 at the start or if there is a different initialization used?

The date at the top of the README.MD and README.PDF in training and synthesis repos is January 14, 2020. Am hoping the next update is dated 2021!

Tensorflow branch is missing FaceID model.

Hi,

I can see there is an UNet demo that is trained and tested using the CamVid dataset. I think it is doing semantic segmentation which the accuracy is typically measured by IoU.

However, your script shows a top1 accuracy. I wonder what it is?

Thanks.

When I use launch.json, such a like that;

Early epochs accuracy are high but after that accuracy decreases. For instance 199 epoch:

200 epoch:

Why does the confusion matrix come out like this? Could you give an idea?

Is it possible to have an mscoco sample training setup? I'm working from the svhn setup and I'm not quite clear what transforms have to be in place for the images and the boxes.

Hello! I’m not sure if this question should be asked in ai8x-training or ai8x-syntheis. Since I am using mostly unedited scripts in synthesis, I believe it has something to do with my training process. The training was done in a fresh install of ai8x-training.

When I save a sample of my object detection models and use it in quantization in ai8x-synthesis, I receive the below output when using the default . Full output in generation_output_layer_11.txt .

ERROR: Layer 11 (conv12_2): 8-bit weight quantization supports an output_shift range of [-15, +15]. The specified value of output_shift is -9223372036854775808 which exceeds the system limits. I’ve experienced this in the past with unsimplified training parameters. Layer 11 is always included in this problem, but it may include other layers as well. The value is always negative but changes.

I see that output_shift is defined in the training software. I have not changed the network file used in the

By manually setting the output_shift to 0 in layer 11 in the network (svhn-tinierssd_v4), I was able to create the full demo using the attached gen_svhn_human_detection.sh . Below is the full output is also listed in the attached generation_output.txt as well as the full log in generation_log.txt .

Arranging weights... ---------------------------------------- 100% Storing weights... ---------------------------------------- 100% WARNING: Layer 5: All bias values are zero. Ignoring the input. WARNING: Layer 6: All bias values are zero. Ignoring the input. WARNING: Layer 7 (backbone_conv8): All bias values are zero. Ignoring the input. WARNING: Layer 10 (conv12_1): All bias values are zero. Ignoring the input. Creating network... ------ WARNING:--------------------------------- Layer 3: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... -------- ------------------------------- 20%WARNING: Layer 4: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... ---------- ----------------------------- 25%WARNING: Layer 5: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... ------------ --------------------------- WARNING: 30% Layer 6: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... -------------- ------------------------- 35%WARNING: Layer 7 (backbone_conv8): All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... ---------------- ----------------------- 40%WARNING: Layer 8 (backbone_conv9): All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. WARNING: Layer 9 (backbone_conv10): All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... --------------------WARNING: Layer 10 (conv12_1): All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... ------------------------ --------------- 60%WARNING: Layer 13: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. WARNING: Layer 14: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. WARNING:Creating network... Layer 17: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. ---------------------------------- ----- 85%WARNING: Layer 18: All output values for the given sample input are zero. The generated known-answer test for this network may not be meaningful. See the log file for details. Creating network... ---------------------------------------- 100%

It looks like this is related to the sample generated from --evaluate and --save-sample in the ai8x-training. In the ai8x-synthesis repo, I can see an example of creating an object detection demo without these missing layers. My previous demos using my personal database as well as the unchanged SVHN database have produced the same missing layers. I’ve had the same results when training simplified and not simplified versions of the SVHN dataloader with the same data, dataloader, policies, and parameters.

Do these empty layers have something to do with saving the sample, or something relating to my data or edited files being used to read the data?

Thank you in advance!
2022.12.23-165123.zip

Hello,
I'm trying to train KWS20 Demo on my System.

My the scripts start to train the Model, every time the system kills the process.
Here is the Screenshot with the message out of memory:

Here are some additional information about my system:
RAM 16Gb
Ubuntu 22.04
Geforce 4060
Cuda 11.8 with latest graphic card Driver 535

Could you please help
Thanks and best regards

Where is this .sh file located?