Inference result is different from test (test.py) results. about donut HOT 13 CLOSED

Hi,
Check if "dataset_name_or_paths" parameter value is same or not during train & test.

from donut.

@vishal-nayak1 Thank you for the response.

I have checked the dataset_name_or_path, it is the same for training and inference.

from donut.

@BakingBrains you can refer to this article for inferencing on the image- https://towardsdatascience.com/ocr-free-document-understanding-with-donut-1acfbdf099be

from donut.

@vishal-nayak1 I referred the same blog. I don't know why the results are different.

from donut.

Hey @BakingBrains did you solve this problem, i m stuck with this.

from donut.

@Codedrainer Yeah, I basically started everything from scratch again. Created a new env, prepared a small dataset and started the training as mentioned. All worked fine.

from donut.

Hey @BakingBrains did you solve this problem, i m stuck with this.

@Codedrainer Hi, Have you solved this problem? I also meet this problem.

from donut.

@wawaa Yeahh, Actually i am using wrong prompt values here.

output = model.inference(image=image, prompt="<s_id_number>")

i have replaced <s_id_number> to <s_dataset> and it's working.

from donut.

@wawaa Yeahh, Actually i am using wrong prompt values here.

output = model.inference(image=image, prompt="<s_id_number>")

i have replaced <s_id_number> to <s_dataset> and it's working.

OK, Thanks!

from donut.

Hello @BakingBrains, I need some assistance with dataset information. I currently have a dataset consisting of 800 images for training, 100 for testing, and 100 for validation. I've successfully trained the model and received the following accuracy scores:

Total number of samples: 100
Tree Edit Distance (TED) based accuracy score: 0.9476799242424243
F1 accuracy score: 0.5213032581453634
After achieving a TED accuracy of 94%, I decided to delve deeper into the dataset for a more thorough analysis. The goal is to parse a document containing a 15-digit alphanumeric invoice number. Upon reviewing the testing results, I noticed a consistent issue: everything was predicted correctly except for the third character from the end of the invoice number. In this position, the model consistently predicted a '2' as a '1'. Out of 100 images tested, 94 contained this inaccuracy.

This is a significant issue because an error in predicting any character of the invoice number renders the entire invoice number incorrect, undermining the utility of using AI for this prediction task.

Could you provide guidance on how to improve the model’s accuracy in predicting this specific character position in the invoice number?

from donut.

@Codedrainer I faced a similar issue with two of the keys from a document,
What I did is, Pretrained on the dataset with image size 2560 X 1920 for 20 epochs. And using the checkpoint I did the finetuning with lower image size.. The issue was not completely solved but the accuracy was above 96%.

from donut.

Hello @BakingBrains, thank you for your time. Initially, I had a comprehensive document from which I aimed to extract only the invoice number. Direct training of the Donut model on the entire document didn't yield satisfactory accuracy. To improve this, I applied YOLOv8 object detection to isolate the section containing the invoice number. After cropping this specific section, I retrained the model, which led to a distinct situation as mentioned earlier.

from donut.

@BakingBrains I'm currently pondering why the Donut model isn’t delivering promising results despite being provided with a clear image featuring a 15-character alphanumeric text. Presently, the images, akin to the one I'm utilizing, have dimensions of 428 x 24 pixels.

from donut.