byeonghu-na / matrn Goto Github PK

Thanks for your great work.

Can you tell me how long the model needs to be trained under the configuration of 4 NVIDIA GeForce RTX 3090GPUs to converge to the results in the paper? If it is convenient, could you provide your training logs?

I'm in the process of reproducing it now, but I found that the loss became jittery after a period of training, I don't know if I configured it wrong or if it's inherently so, slowly converging to the result in a long period of jitters. So I hope the author will provide a training log, if possible（thanks a lot).

Thank you very much!

Nice paper! Do you plan to add a demo?

Hello there!

• Great work. I'd like to ask how to train the align model with more characters. The current implementation can only recognize 36 characters (09, az). I want to recognize 90 characters (09, az, A~Z, and some symbols).
• I tried to modify some code and now I can train on 90 characters. However, I am facing a problem that I can not load the pre-trained language model and vision model, as they are trained on 36 characters. Is there any way to modify the code so that I can load the pre-trained weights?

Hello author, the following error occurred in the model when I used Resnet as the model backbone instead of ResTransformer. No such error occurred when I ran ABINet using ResNet as the backbone of the model.

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:
 [torch.FloatTensor [10, 256, 512]], which is output 0 of ViewBackward, is at version 28; expected version 0 instead. 
Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I found that backbone must have transformer in your model, if there is no transformer behind CNN, there will be mistakes, but I can't find more specific reasons.
I only changed the backbone in the train_matrn.yaml configuration file.

Thanks for your reply!

Could you please explain how to train vision and language models for Korean and Japanese languages?

Hi!
Thanks for your great work. When I debug your code, I find Line81 and Line 82 in file main.py seems like they're written backwards. I have no idea whether this is correct.
Thanks :D

First, thank you very much for your work, it is very impressive. However, when I evaluate with the pre-trained model provided by the link, I get results that are lower than the performance of the report. What is the reason for this? Thank you very much for your answer.
And my results on 6 datasets of IIIT5k_3000, SVT, SVTP, IC13_857, IC15_1811, CUTE80 are as follows:

[2022-03-03 23:28:19,374 main.py:276 INFO train-matrn] validation time: 62.44528245925903 / batch size: 384
[2022-03-03 23:28:19,374 main.py:281 INFO train-matrn] eval loss = 1.435, ccr = 0.957, cwr = 0.904, ted = 1542.000, ned = 297, ted/w = 0.213.

you results of the same six datasets average cwr is 93.450.

Thank you very much again!

Thank you for sharing the code！

The 34-th line in modules/model_matrn_iter.py: the self.semantic_visual has no attribute about pe.
So I get the error "torch.nn.modules.module.ModuleAttributeError: 'BaseSemanticVisual_backbone_feature' object has no attribute 'pe'"

So as the 39-th and 44-th lines.

Is there something wrong here？

Excuse me?
Why does the process still end when the training reaches epoch9 after the configuration file is modified

I noticed that texts(index encoding) is passed to the forward function, but not used anywhere. Just curious that, are you going to "ADD" text embedding to the final output? My guess is that, you probably have tried it out, but got limited performance improvements. I've been thinking about the usage of text embedding for a while, but it's too hard to convince myself to add text embedding to the training pipeline. As in the inference time, no text information will be given. Please correct me if my guess is wrong. Thanks.

In the requirements.txt

editdistance should be added.

How does it perform in a handwritten data set