rse / fasttext-lid Goto Github PK

View Code? Open in Web Editor NEW

10.0 3.0 0.0 18 KB

Language Identification with Facebook FastText for Node.js

Home Page: http://npmjs.com/fasttext-lid

JavaScript 100.00%

fasttext lid language identification prediction model

fasttext-lid's People

Contributors

Stargazers

Watchers

fasttext-lid's Issues

The URL of the pretrained model is no longer valid

FAIR updated the URL, which broke this lib.

For those searching:

♕ resource: FastText LID-176 model
★ location: https://s3-us-west-1.amazonaws.com/fasttext-vectors/supervised_models/lid.176.bin
⚙ download: 278 bytes received with 279B/s... npm-install-fetch: ERROR: download failed: HTTP response: 403

The fix is in #1. For a temporary workaround you can do npm install git+https://[email protected]/modelfront/fasttext-lid.git instead.

Pre-processing

fastText did some pre-processing before training, so it's safest to do the same, especially lowercasing:

> (async () => { console.log(await lid.predict("Ótimo")) })()
> [ { lang: 'hu', prob: 0.3540060520172119 } ]
> (async () => { console.log(await lid.predict("ótimo")) })()
> [ { lang: 'pt', prob: 0.9964836835861206 } ]
> (async () => { console.log(await lid.predict("Uma boa experiencia")) })()
> [ { lang: 'en', prob: 0.5653783679008484 } ]
> (async () => { console.log(await lid.predict("uma boa experiencia")) })()
> [ { lang: 'pt', prob: 0.9949126839637756 } ]

Recommend Projects