Comments (7)
Sorry for the delay, but can do let me know, from which layer did you extract the output?
Regards,
from docformer.
Hi @uakarsh
Thanks for your response.
I tried below code
config = {
"coordinate_size": 96,
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"image_feature_pool_shape": [7, 7, 256],
"intermediate_ff_size_factor": 4,
"max_2d_position_embeddings": 1000,
"max_position_embeddings": 512,
"max_relative_positions": 8,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"shape_size": 96,
"vocab_size": 30522,
"layer_norm_eps": 1e-12,
}
fp = "img.jpeg"
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
encoding = dataset.create_features(fp, tokenizer, add_batch_dim=True)
feature_extractor = modeling.ExtractFeatures(config)
docformer = modeling.DocFormerEncoder(config)
v_bar, t_bar, v_bar_s, t_bar_s = feature_extractor(encoding)
output = docformer(v_bar, t_bar, v_bar_s, t_bar_s) # shape (1, 512, 768)
then I visualized the output.
from docformer.
HI,
Actually, we know that the output is (512, 768), now, this output results from the attention of three different entities:
- Image feature of (512, 768)
- Language Feature of (512, 768)
- Spatial Dimension of (512, 768)
Now, when we perform any downstream task, we have an encoded version of these three modalities, so the diagram (which you have plotted) would be helpful for the model to know, which encoding to attend to when performing the downstream task.
The same can be seen in Pg No. 15, Figure 11. B of DocFormer Paper. Hope it helps
from docformer.
Thanks for your info. How can I do entity level classification like in FUNSD dataset?
from docformer.
from docformer.
I have almost finished the training script for RVL-CDIP (Document Classification), and have started working on FUNSD for token classification.
You can visit my cloned repo (https://github.com/uakarsh/docformer/tree/master/examples/docformer_pl), and in the examples/docformer_pl, you can get the
- Data visualizing
- Dataset making
- MLM with Pytorch Lightning
- Document Classification with DocFormer (would be uploaded soon)
And next would be NER with FUNSD.
Would update you soon!!
from docformer.
@uakarsh Hello,
Any update on NER with FUNSD using docformer?
from docformer.
Related Issues (20)
- checkpoint of docformer HOT 3
- About rvl_cdip_dataset.csv HOT 6
- Again Device issue HOT 3
- Reference code of pre-training tasks HOT 4
- [Errno 2] No such file or directory: 'rvl_cdip_dataset.csv'
- NER task HOT 4
- Predictions are wrong. HOT 1
- can you provide visual question answering task code HOT 1
- Unable to convert model to onnx HOT 3
- DocFormer for Token Classification. HOT 3
- Shape mismatch during sanity check HOT 1
- finetune the Docformer HOT 8
- Pre-trained models HOT 5
- Error in Example: Please provide the bounding box and words or pass the argument "use_ocr" = True HOT 1
- DocFormer for key-value pairs extraction HOT 1
- Using pre-trained models HOT 17
- Inference for token classification. HOT 1
- We can not find DocFormer_For_IR in modeling.py and modeling_l.py HOT 1
- Permission Denied Pretraining Weights HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docformer.