Comments (12)
myoldusername
commented on August 18, 2024
1
It is working as expected , THANK YOU SO MUCH .
You made my day !
Can i send you a donation ?
from fasttext.js.
loretoparisi
commented on August 18, 2024
@myoldusername I have just updated the library with several improvements for the child process run. I have also added a server example that will help your needs. Please check the README.
from fasttext.js.
myoldusername
commented on August 18, 2024
Upstanding.... I will test it today since i am out of town, today i will give you a feedback.
You are awesome......
from fasttext.js.
myoldusername
commented on August 18, 2024
Some times it crach when i pass the string text if the string is unicode, like Chinese
I advice you to add normalization method to remove all non characters, e.g all special characters and smiles characters...
from fasttext.js.
loretoparisi
commented on August 18, 2024
@myoldusername yes this is a good point there are minor functions in utils like Util.removeDiacritics here https://github.com/loretoparisi/fasttext.js/blob/master/lib/util.js#L238
and the dataset is normalized in FastText.normalize https://github.com/loretoparisi/fasttext.js/blob/master/lib/index.js#L438
but of course for symbolics languages it's different, since it must be handled with Unicode i.e. unicode conversion and normalization before prediction.
Be aware that this normalization must be done on the training set too i.e. you have to apply the same normalization to training/test set and to the sample for the inference.
In my backend I do unicode normalization in Java, but here I would prefer a node solution. Will look into!
from fasttext.js.
myoldusername
commented on August 18, 2024
Well i am working with language classification training set which provide by fastText with respect to them.
I use to pass some languages paragraphs to the localhost url it works, but some time it suddenly crashed even with normalized strings.. I am not sure i will make farther test to see if my copy paste string has some hidden characters.. Since unicode has some nasty stuff lol.
Regarding node solution, i think it will be an awesome idea to apply.
With respect.
Yours
from fasttext.js.
loretoparisi
commented on August 18, 2024
Yes this could be a very tricky task when dealing with languages that needs Unicode. By the way I'm using the same model too, so I have added the compressed version of the model in the example, and some env var so that you can go:
cd examples/
export MODEL=./data/lid.176.ftz
export PORT=9001
node server
and then
that will be correctly detected as KO:
{
"response_time": 0.001,
"predict": [{
"label": "KO",
"score": "1"
},
{
"label": "TR",
"score": "1.95313E-08"
}
]
}NOTE
My input text was 랄랄라%20차차차%20랄랄라\n랄랄라%20차차차%20장윤정%20트위스트%20춤을%20춥시다, but when you put in a url it will be automatically encoded with the encodeUriComponent method.
from fasttext.js.
myoldusername
commented on August 18, 2024
Well i like to bring to your attention that sometime when i pass a regular string, for unknown reasons the node server file freeze and i have to kill it and restart it again..
from fasttext.js.
loretoparisi
commented on August 18, 2024
@myoldusername put here that text and the url as cut&paste from the browser
from fasttext.js.
myoldusername
commented on August 18, 2024
http://localhost:3030/?text=bader
from fasttext.js.
loretoparisi
commented on August 18, 2024
uhm I guess you have some issues in your env:
$ export PORT=3030
$ export MODEL=./data/lid.176.ftz
$ node server.js
model loaded
server is listening on 3030
you therefore call http://localhost:3030/?text=bader and you get:
{
response_time: 0.002,
predict: [{
label: "EN",
score: "0.125931"
},
{
label: "CA",
score: "0.0847617"
}
]
}This should work without any issues:
$ time curl -s "http://localhost:3030/?text=bader"
{
"response_time": 0,
"predict": [
{
"label": "EN",
"score": "0.125931"
},
{
"label": "CA",
"score": "0.0847617"
}
]
}
real 0m0.027s
user 0m0.005s
sys 0m0.006s
and now we do some benchmarking as well calling 1, 10 and 100 times iteratively:
$ ab -n 1 "http://localhost:3030/?text=bader"
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient).....done
Server Software:
Server Hostname: localhost
Server Port: 3030
Document Path: /?text=bader
Document Length: 164 bytes
Concurrency Level: 1
Time taken for tests: 0.001 seconds
Complete requests: 1
Failed requests: 0
Total transferred: 271 bytes
HTML transferred: 164 bytes
Requests per second: 712.76 [#/sec] (mean)
Time per request: 1.403 [ms] (mean)
Time per request: 1.403 [ms] (mean, across all concurrent requests)
Transfer rate: 188.63 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1 1 0.0 1 1
Waiting: 1 1 0.0 1 1
Total: 1 1 0.0 1 1
[loretoparisi@:mbploreto task]$ ab -n 10 "http://localhost:3030/?text=bader"
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient).....done
Server Software:
Server Hostname: localhost
Server Port: 3030
Document Path: /?text=bader
Document Length: 164 bytes
Concurrency Level: 1
Time taken for tests: 0.011 seconds
Complete requests: 10
Failed requests: 4
(Connect: 0, Receive: 0, Length: 4, Exceptions: 0)
Total transferred: 2726 bytes
HTML transferred: 1656 bytes
Requests per second: 941.00 [#/sec] (mean)
Time per request: 1.063 [ms] (mean)
Time per request: 1.063 [ms] (mean, across all concurrent requests)
Transfer rate: 250.50 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 1 0.5 1 2
Waiting: 0 1 0.3 1 1
Total: 1 1 0.5 1 2
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 2
95% 2
98% 2
99% 2
100% 2 (longest request)
[loretoparisi@:mbploreto task]$ ab -n 100 "http://localhost:3030/?text=bader"
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient).....done
Server Software:
Server Hostname: localhost
Server Port: 3030
Document Path: /?text=bader
Document Length: 168 bytes
Concurrency Level: 1
Time taken for tests: 0.095 seconds
Complete requests: 100
Failed requests: 73
(Connect: 0, Receive: 0, Length: 73, Exceptions: 0)
Total transferred: 27208 bytes
HTML transferred: 16508 bytes
Requests per second: 1054.37 [#/sec] (mean)
Time per request: 0.948 [ms] (mean)
Time per request: 0.948 [ms] (mean, across all concurrent requests)
Transfer rate: 280.15 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 1 1.2 0 9
Waiting: 0 1 1.2 0 9
Total: 0 1 1.2 1 9
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 2
95% 3
98% 7
99% 9
100% 9 (longest request)
from fasttext.js.
loretoparisi
commented on August 18, 2024
I have added here some benchmarkes therefore I'm closing this issue. Feel free to re-open it if you have any problem.
from fasttext.js.
Related Issues (20)
- Time for scale up, fastText.Js with Redis and Clustering mode HOT 7
- better error message when .bin is not found on load() HOT 5
- Move repo to a GitHub *org* HOT 9
- async/await version HOT 3
- bug in FastText.prototype.nn() HOT 1
- calculate distance feature? HOT 4
- DeprecationWarning: Buffer() is deprecated due to security and usability issues when i move my script to another server
- Executable not found in path on Windows HOT 7
- Question : usage of pretrainedVectors? HOT 1
- Trying to get in touch regarding a security issue HOT 1
- [Question] Are the binaries provided always up to date with the latest version of fasttext ? HOT 1
- Using in the browser with WASM HOT 2
- I cannot run this project on apple M1 silicon HOT 4
- Different sentence vector compared to Python HOT 1
- npm module: loadSentence is not a function HOT 2
- Incremental training
- Fail to load WASM error HOT 7
- I like Your code in https://github.com/huggingface/tokenizers/issues/1076 HOT 4
- This package cannot be used in node.js worker thread HOT 1
- Is it library outdated?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fasttext.js.