Comments (4)
Hi,
I think that the good results is a subjective topic (maybe, depends upon your usage of application, i.e how you wish to use this architecture). From my perspective, you can just to tune the hyperparameters and observe the results (I have integrated W&B for visualization of progress), and come to a conclusion
Currently, if you are talking about the 4th Notebook (i.e the Kaggle Notebook), no we are not using the pre-trained weights (I have seperately described the pre-training on MLM), but rather directly training it from scratch
If you want to go with pre-training, here is a simple approach. Pre-train the docformer with MLM (from 3rd notebook in the example), and then use the 4th notebook (and load the specific weights from the checkpoint you saved)
Hope this helps.
Regards,
Akarsh
from docformer.
@BakingBrains Hi,
Actually, the way I trained DocFormer on RVL-CDIP has some assumptions (you can check out the Kaggle Notebook, where I mentioned the assumptions, and these assumptions were based on the fact that, the GPU memory was limited and the time constraints)
You can check out this line of code, where I load the weights. Ckpt. If that doesn't work, the checkpoint are saved in the version 5 of the kaggle notebook (shared in the examples/docformer_pl/ of cloned repo of docformer)
And for the NER task, in some time I would be uploading the script as well, so stay tuned. And, as soon as I upload it, I would let you know the same
Hope this helps
Regards,
Akarsh
from docformer.
@uakarsh Thank you.
What do you think the number of epochs required to get good output for document classification.
Also, in the paper author has mentioned:
DocFormer is pre-trained for 5 epochs, then we remove all
three task heads. We add one linear projection head and
fine-tune all components of the model for all downstream
tasks.
I see here adding Linear layer.
self.resnet = ResNetFeatureExtractor(hidden_dim = config['max_position_embeddings'])
self.embeddings = DocFormerEmbeddings(config)
self.lang_emb = LanguageFeatureExtractor()
self.config = config
self.dropout = nn.Dropout(config['hidden_dropout_prob'])
self.linear_layer = nn.Linear(in_features = config['hidden_size'], out_features = len(id2label))
In section 3.2. Are we doing the same thing for document classification as well?
Had this doubt.
Thanks and Regards
from docformer.
@uakarsh Thank you😄
from docformer.
Related Issues (20)
- Again Device issue HOT 3
- Weird output HOT 7
- Reference code of pre-training tasks HOT 4
- [Errno 2] No such file or directory: 'rvl_cdip_dataset.csv'
- Predictions are wrong. HOT 1
- can you provide visual question answering task code HOT 1
- Unable to convert model to onnx HOT 3
- DocFormer for Token Classification. HOT 3
- Shape mismatch during sanity check HOT 1
- finetune the Docformer HOT 8
- Pre-trained models HOT 5
- Error in Example: Please provide the bounding box and words or pass the argument "use_ocr" = True HOT 1
- DocFormer for key-value pairs extraction HOT 1
- Using pre-trained models HOT 17
- Inference for token classification. HOT 1
- We can not find DocFormer_For_IR in modeling.py and modeling_l.py HOT 1
- Permission Denied Pretraining Weights HOT 2
- NotImplementedError: Support for `validation_epoch_end` has been removed in v2.0.0. `DocFormer` implements this method HOT 2
- pre-training code tutorial HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docformer.