Comments (10)
Rakshith2597
commented on September 2, 2024
1
@dkurt , Sure, below is the model definition.
Additionally, I have provided the model definition, model weights, onnx file, error and debug log from mo in GDrive if you need it here.
Thank you for helping. I really appreciate it.
class SUMNet(nn.Module):
def __init__(self,in_ch,out_ch):
super().__init__()
self.encoder = models.vgg11_bn(pretrained = True).features
self.preconv = nn.Conv2d(in_ch, 3, 1)
self.conv1 = self.encoder[0]
self.bn1 = self.encoder[1]
self.pool1 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv2 = self.encoder[4]
self.bn2 = self.encoder[5]
self.pool2 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv3a = self.encoder[8]
self.bn3 = self.encoder[9]
self.conv3b = self.encoder[11]
self.bn4 = self.encoder[12]
self.pool3 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv4a = self.encoder[15]
self.bn5 = self.encoder[16]
self.conv4b = self.encoder[18]
self.bn6 = self.encoder[19]
self.pool4 = nn.MaxPool2d(2, 2, return_indices = True)
self.conv5a = self.encoder[22]
self.bn7 = self.encoder[23]
self.conv5b = self.encoder[25]
self.bn8 = self.encoder[26]
self.pool5 = nn.MaxPool2d(2, 2, return_indices = True)
self.unpool5 = nn.MaxUnpool2d(2, 2)
self.donv5b = nn.Conv2d(1024, 512, 3, padding = 1)
self.donv5a = nn.Conv2d(512, 512, 3, padding = 1)
self.unpool4 = nn.MaxUnpool2d(2, 2)
self.donv4b = nn.Conv2d(1024, 512, 3, padding = 1)
self.donv4a = nn.Conv2d(512, 256, 3, padding = 1)
self.unpool3 = nn.MaxUnpool2d(2, 2)
self.donv3b = nn.Conv2d(512, 256, 3, padding = 1)
self.donv3a = nn.Conv2d(256,128, 3, padding = 1)
self.unpool2 = nn.MaxUnpool2d(2, 2)
self.donv2 = nn.Conv2d(256, 64, 3, padding = 1)
self.unpool1 = nn.MaxUnpool2d(2, 2)
self.donv1 = nn.Conv2d(128, 32, 3, padding = 1)
self.output = nn.Conv2d(32, out_ch, 1)
def forward(self, x):
preconv = F.relu(self.preconv(x), inplace = True)
conv1 = F.relu(self.bn1(self.conv1(preconv)), inplace = True)
pool1, idxs1 = self.pool1(conv1)
conv2 = F.relu(self.bn2(self.conv2(pool1)), inplace = True)
pool2, idxs2 = self.pool2(conv2)
conv3a = F.relu(self.bn3(self.conv3a(pool2)), inplace = True)
conv3b = F.relu(self.bn4(self.conv3b(conv3a)), inplace = True)
pool3, idxs3 = self.pool3(conv3b)
conv4a = F.relu(self.bn5(self.conv4a(pool3)), inplace = True)
conv4b = F.relu(self.bn6(self.conv4b(conv4a)), inplace = True)
pool4, idxs4 = self.pool4(conv4b)
conv5a = F.relu(self.bn7(self.conv5a(pool4)), inplace = True)
conv5b = F.relu(self.bn8(self.conv5b(conv5a)), inplace = True)
pool5, idxs5 = self.pool5(conv5b)
unpool5 = torch.cat([self.unpool5(pool5, idxs5), conv5b], 1)
donv5b = F.relu(self.donv5b(unpool5), inplace = True)
donv5a = F.relu(self.donv5a(donv5b), inplace = True)
unpool4 = torch.cat([self.unpool4(donv5a, idxs4), conv4b], 1)
donv4b = F.relu(self.donv4b(unpool4), inplace = True)
donv4a = F.relu(self.donv4a(donv4b), inplace = True)
unpool3 = torch.cat([self.unpool3(donv4a, idxs3), conv3b], 1)
donv3b = F.relu(self.donv3b(unpool3), inplace = True)
donv3a = F.relu(self.donv3a(donv3b))
unpool2 = torch.cat([self.unpool2(donv3a, idxs2), conv2], 1)
donv2 = F.relu(self.donv2(unpool2), inplace = True)
unpool1 = torch.cat([self.unpool1(donv2, idxs1), conv1], 1)
donv1 = F.relu(self.donv1(unpool1), inplace = True)
output = self.output(donv1)
return output
from openvino_pytorch_layers.
Rakshith2597
commented on September 2, 2024
1
@dkurt Thanks a lot for your help. I am able to convert the model to IR.
from openvino_pytorch_layers.
dkurt
commented on September 2, 2024
@likholat, @alexeyhorkin, do you still have an alternative solution with MO Python extension which allows replace MaxUpooling to known ops? Cannot find in openvinotoolkit/openvino_notebooks#487
Update: Never mind
@Rakshith2597, may I ask you to try this approach which let you don't use custom extensions at all: openvinotoolkit/openvino#11400. You need create an alias for unpooling layer, export PyTorch model with it and then convert to OpenVINO IR using Model Optimized extension from the PR.
from openvino_pytorch_layers.
Rakshith2597
commented on September 2, 2024
@dkurt , Here are the steps I did, please let me know if I am doing it right.
- Created max_unpool_2d_decomposition at tools/mo/openvino/tools/mo/front/onnx/max_unpool2d_decomposition.py based on link.
- Exported the pytorch model to ONNX as mentioned in the PR.
- Tried converting the ONNX model to OpenVINO IR.
Got the following error:
OpenVINO runtime found in: /home/deeptensor/rakshith_codes/training_extensions/misc/pytorch_toolkit/lung_nodule_detection/venv/lib/python3.9/site-packages/openvino
OpenVINO runtime version: 2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version: 2022.1.0-7019-cdb9bec7210-releases/2022/1
[ ERROR ] Cannot infer shapes or values for node "ATen_53".
[ ERROR ] There is no registered "infer" function for node "ATen_53" with op = "ATen". Please implement this function in the extensions.
For more information please refer to Model Optimizer FAQ, question #37. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=37#question-37)
[ ERROR ]
[ ERROR ] It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ] Or because the node inputs have incorrect values/shapes.
[ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ] Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'openvino.tools.mo.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "ATen_53" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)
I think i might have missed something here
You need create an alias for unpooling layer, export PyTorch model with it
Could you please elaborate this? Sorry, I'm new to this.
from openvino_pytorch_layers.
dkurt
commented on September 2, 2024
Seems fine, but can you try operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK?
from openvino_pytorch_layers.
Rakshith2597
commented on September 2, 2024
I'm converting my PyTorch model to ONNX using,
torch.onnx.export(model, dummy_input, res_path,
input_names=['input'], output_names=['output'],
operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK,
verbose=False)
This raises the error mentioned in the previous comment.
from openvino_pytorch_layers.
dkurt
commented on September 2, 2024
@Rakshith2597, got it, thanks! Is there a way to share model definition so I can reproduce and propose a solution?
from openvino_pytorch_layers.
dkurt
commented on September 2, 2024
@Rakshith2597, seems like the following steps helped me convert a model:
- Restore normal unpooling usage:
self.unpool1 = nn.MaxUnpool2d(2, stride=2)
...
unpool1 = torch.cat([self.unpool1(donv2, idxs1), conv1], 1) # not .apply- Workaround some internal bug by adding intermediate op between
pool5andunpool5(have not looked how to fix it). Required only forupool5.
unpool5 = torch.cat([self.unpool5(pool5 + 1e-10, idxs5), conv5b], 1)- Convert to ONNX with
operator_export_type=torch.onnx.OperatorExportTypes.ONNX_FALLTHROUGH - Modify max_unpool2d_decomposition.py:
--- a/mo_extensions/front/onnx/max_unpool2d_decomposition.py
+++ b/mo_extensions/front/onnx/max_unpool2d_decomposition.py
@@ -20,7 +20,7 @@ class MaxUnpoolFrontReplacer(FrontReplacementSubgraph):
nodes=[
("max_pool0", dict(op="MaxPool")),
("max_pool1", dict(op="MaxPool")),
- ("slice", dict(op="AttributedSlice")),
+ ("slice", dict(op="Slice")),
("sub", dict(op="Sub")),
("unpool", dict(op="max_unpool2d")),
],from openvino_pytorch_layers.
Rakshith2597
commented on September 2, 2024
@dkurt Hi, on trying to do inference with the ONNX model generated, I'm getting the following error.
[ONNXRuntimeError] : 1 : FAIL : Load model from downloads/model_weights/stage1/lung_seg.onnx failed:Fatal error: max_unpool2d is not a registered function/op
Code used to do inference
net = onnxruntime.InferenceSession('model.onnx')
ort_inputs = {net.get_inputs()[0].name: to_numpy(inputs)}
net_out = net.run(None, ort_inputs)
Code used to create ONNX model
torch.onnx.export(model, dummy_input, res_path,
input_names=['input'], output_names=['output'],
operator_export_type=torch.onnx.OperatorExportTypes.ONNX_FALLTHROUGH,
verbose=False)
Am I missing something?
from openvino_pytorch_layers.
dkurt
commented on September 2, 2024
Not sure that ONNX Runtime can handle this layer. I've tested OpenVINO 2022.2 and inference was fine.
from openvino_pytorch_layers.
Related Issues (7)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openvino_pytorch_layers.