Comments (3)
can be fixed by running pip install rust
before installation of the package
from marqo.
Thanks Danyil! You are correct, the current default behavior is that huggingface will use the rust based tokenizers. We will potentially switch to the python tokenizer as the small batch performance should not be any different and can be switched using a flag as described here https://github.com/huggingface/transformers/blob/v4.21.1/src/transformers/models/auto/tokenization_auto.py#L472. Although, the other consideration is that others may want to use the fast tokenizers (or not be aware) so maybe the install is a better approach
from marqo.
This has been taken care of after the move to Docker.
from marqo.
Related Issues (20)
- [ENHANCEMENT] More descriptive `get_loaded_models` endpoint
- Help installing container on TrueNAS Scale (containerd) HOT 6
- [BUG] Add documents validation failure when use_existing_tensors=True
- [BUG] Docker container does not run on M1 Mac HOT 4
- [ENHANCEMENT] Disk persistent storage configuration HOT 1
- Usage on Google Cloud Platform
- [ENHANCEMENT] Marqo on Azure HOT 1
- ONNX Support - CPU HOT 1
- [BUG] Cannot pull docker image on latest 2.1.0 HOT 1
- Marqo Integration into cacheGPT HOT 1
- Error Status: 400 after using add_documents() to add the images from docker to index
- [ENHANCEMENT] Wildcard support HOT 1
- Marqo encountered an unexpected internal error, status_code: 500[BUG] HOT 1
- Support OpenAPI spec generation [ENHANCEMENT]
- [ENHANCEMENT] Allow Pydantic > 2 HOT 2
- Unable to delete index [BUG] HOT 4
- [ENHANCEMENT] Nicer error message if not enough memory HOT 1
- [ENHANCEMENT] Highlighting Exact Matches HOT 2
- [BUG] This is a test HOT 1
- [BUG] HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from marqo.