HuggingFace model multilingual-e5-small fails to open on Windows due to max path limitations about djl HOT 4 CLOSED

For a workaround, you can modify the metadata.json and remove the properties. You may also need to change file references from relative to absolute. Loading this would avoid the long path problems.

In terms of the larger usage of properties inside model paths, the main reason we do so is to differentiate different artifacts within the same metadata. Properties make it work with a fairly clean directory tree for most cases. This also only applies for the local cache of downloaded models and datasets. The code for it is in Artifact.getResourceUri() if you are interested.

For a complete solution, we would need to replace it with a new system. The obvious one is to use the artifact name instead of the properties. This requires that all artifacts have names and that they are unique within a metadata.json file. I am not sure off the top of my head that this is true, so we would have to verify. Assuming that is fine, it should be a fairly easy change. @frankfliu what do you think?

from djl.

the properties are generated by model importing tool, we need to update the tool to combine languages into a single property.

We can manually change the metadata.json for now.

from djl.

I did a scan of existing model zoo, all models (except mxnet yolo) artifact name + version are unique in metadata.json.

We actually don't need to use properties as file path to avoid file path clash.

from djl.

@frankfliu - many thanks for fixing this. I can confirm on Windows this now works as expected using 0.27.0-SNAPSHOT.

from djl.