Model instantiation without loading from disk about vision HOT 2 OPEN

Thank you for the request @AndreasKaratzas

However, every time I query my framework for some workload, I have to first use Python and compile a jit instance to later load in my framework

It wouldn't completely address your situation, but is it feasible for you to create the scripted model (in Python) once and for all and just load that one file every time?

I'm sorry that this is not the answer you're looking for, but I'm afraid the C++ models won't be coming back. For some context, we used to support direct C++ model access in the past, but the decision was made to deprecate that API in favour of torchscript. Whether this was a user-friendly decision in the long-term is up for debate, but the main reason at the time was that maintaining both the Python and C++ backend was just too much maintenance work. Some relevant past discussion: #4375 (comment)

If that helps and if it's an option for you, those C++ models were removed in https://github.com/pytorch/vision/pull/6632/files so you'll find the original C++ implementations there (some of these models may have been updated in Python since then).

from vision.

Hello @NicolasHug,

Thank you for your prompt response.

It wouldn't completely address your situation, but is it feasible for you to create the scripted model (in Python) once and for all and just load that one file every time?

This would be a nice solution generally. However, to my problem, there are several parameters that need to be taken into account. My domain primarily evolves around embedded systems, where the workload changes rapidly and frequently. In particular, regarding the case of edge data centers, there are various clients with different needs who post different queries on the edge systems. Therefore, these constrained devices must be capable of quickly serving each client. I opted for torch/TensorRT to boost the performance of my framework. However, there is a huge computational burden and, worse of all, a power burden if I have to read the memory too frequently. Furthermore, due to the variety of available torch models (torchvision alone has more than 70 models), the solution of storing several hundreds of GBs on the embedded devices is not scalable and cannot streamline efficiency. I strongly think that my use case is not an extreme scenario and that torch/TensorRT primarily targets such cases where performance and efficiency are primary challenges.

I also reviewed the pull request referenced. As pointed out, the models there are old and point to a rather old release of torchvision.

Overall, I think that in the case of torch/TensorRT, there is a huge benefit to reinstating and maintaining the CPP models. This is due to the target group that torch/TensorRT attracts, which mostly chases performance, efficiency, and scalability in constrained devices.

from vision.