Comments (3)
Ah yes, you are completely right. I had this problem before. Initially, I just added a check for which models have processors:
transformers.js/scripts/convert.py
Lines 193 to 195 in 6078e1c
but I removed it since it was extremely hacky, and would be difficult to maintain.
The other option would be to just copy all .json files:
transformers.js/scripts/convert.py
Line 203 in 492c117
Unless there is a better way? I assume this is a problem with optimum and the transformers' save_pretrained
function, which seems to ignore the tokenizer.json
from transformers.js.
I did some digging, and looks like transformers has a handy function for this! I've updated the conversion script to handle this. Let me know if you know of other files which aren't copied by save_pretrained
from transformers.js.
Closing as this was fixed in 1f14ca0
from transformers.js.
Related Issues (20)
- Options for the "translation" pipeline when using Xenova/t5-small
- Have considered using wasm technology to implement this library? HOT 1
- Example not working on Chrome/Arc v.124(M1 Mac) HOT 5
- Can you use all transformers models with transformers.js?
- Request to support suno-ai/bark model HOT 2
- Something went wrong during model construction (most likely a missing operation). Using `wasm` as a fallback. HOT 4
- v3: Issue loading T5-Small with webgpu backend HOT 2
- WebAssembly Compilation Error Due to CSP Restrictions HOT 5
- [v3] webgpu-chat demo fails with `no available backend found` HOT 1
- How to point to a specific model path in order to use compressed models? (brotli) HOT 1
- Method to send `HF_TOKEN` from the client side
- Pipeline execution time with 'image-classification' pipeline HOT 1
- Mistral Tokenizer V2
- Allow overriding model file name to support custom implementations of various compressions (brotli, gzip, etc)
- Progress callback for Moondream? HOT 1
- NanoLlava is repeating itself a lot HOT 1
- Shouldn't this work? #v3 HOT 3
- Using AutoModel, AutoTokenizer with distilbert models HOT 1
- Support stopping criteria for sequence generation HOT 1
- WebGPU and WASM Backends Unavailable within Service Worker (V3 issue) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.js.