Comments (3)
@StrangeBytesDev You can pass to the generateCompletion
function an array of tokens instead of a string - this way you can tokenize the input however you want
from node-llama-cpp.
@StrangeBytesDev This issue was already fixed in version 3 beta.
Using the version 3 beta, to tokenize an input with special tokens you should enable the specialTokens
parameter:
import {fileURLToPath} from "url";
import path from "path";
import {getLlama} from "node-llama-cpp";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const llama = await getLlama();
const model = await llama.loadModel({
modelPath: path.join(__dirname, "models", "functionary-small-v2.2.q4_0.gguf")
});
const text = "<|from|>user\n<|content|>Hello";
console.log("With special tokens:", model.tokenize(text, true));
console.log("Without special tokens:", model.tokenize(text));
from node-llama-cpp.
Oh awesome, I totally missed that. I like that its available optionally. I don't think I've seen any other library or API that has it as an option, and I can see some use cases where it would useful to have both.
I'm having a bit of a hard time getting my head around how the tokenization in the generateCompletion function is handled. I'm under the impression that there isn't a way to enable the specialTokens param from a completion currently. Is that the case?
from node-llama-cpp.
Related Issues (20)
- illegal hardware instruction on M2 HOT 4
- Problem when running some models with cuda HOT 7
- Receiving error when compiling for cuda HOT 2
- Error: Conversation roles must alternate user/assistant/user/assistant/.. HOT 12
- Prefix of function call problem HOT 3
- Error Creating Jinja Template Chat Wrapper in node-llama-cpp Latest Beta Version HOT 9
- When I use node-llama-cpp to run inference, cloudrun fails with a 503 error HOT 2
- Error Creating Jinja Template Chat Wrapper in node-llama-cpp Latest Beta Version HOT 1
- Common JS Support HOT 1
- GGML_ASSERT(check == -n_length) failed HOT 2
- Unable to create a context HOT 9
- Model outputs � in Korean/Chinese HOT 10
- feat: React Native support HOT 3
- Make Chat History optional HOT 3
- Token Info Provider HOT 2
- Cuda Support HOT 2
- feat: support Functionary new chat format HOT 7
- Installation with CUDA-support fails early with "npm ERR! canceled" HOT 1
- Free VRAM programmatically instead with GC HOT 2
- Error: vk::Queue::submit: ErrorDeviceLost HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-llama-cpp.