Comments (3)
hatttruong
commented on July 17, 2024
7
@pengqiuyuan have you resolved this issue yet? I have the same problem with you and I find out that when training model, FastText added </s>word whenever it read "\n" from training file. Read readWord function for more detail.
It means that </s> exists in vocabulary and has an impact on the predict probability. When you call "./fasttext test weibo.bin weibo2.txt", the text string in "weibo2.txt" contains "\n" and function getLine replaces it with </s>
However, the value in parameter text does not contain "\n", which results in the different result. In my case, I just added "\n" at the end of text string.
Hope it helps :)
from jfasttext.
pengqiuyuan
commented on July 17, 2024
@vinhkhuc thank you.
for (int i = 0; i < 100; i++) {
log.debug("测试接口："+ fastTextService.findByText(text));
}
public String findByText(String text){
JFastText jft = new JFastText();
jft.loadModel(weibo_zong);
String probLabel = jft.predict(text);
jft.unloadModel();
return probLabel;
}
15:48:27.266 [main] DEBUG com.stq.FastTextTest - 测试接口：__label__行业文章
15:48:27.433 [main] DEBUG com.stq.FastTextTest - 测试接口：__label__行业文章
15:48:27.599 [main] DEBUG com.stq.FastTextTest - 测试接口：__label__行业文章
15:48:27.770 [main] DEBUG com.stq.FastTextTest - 测试接口：__label__行业文章
java(21732,0x700004cd6000) malloc: *** error for object 0x400007fdf0d59ccc: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
/bin/sh: line 1: 21732 Abort trap: 6 /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/bin/java -Xmx1024M -jar /Users/admin/pqy/github/es-server/target/surefire/surefirebooter6458186226506641090.jar /Users/admin/pqy/github/es-server/target/surefire/surefire5222514888140147610tmp /Users/admin/pqy/github/es-server/target/surefire/surefire_08480922186965078148tmp
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.605 s
[INFO] Finished at: 2017-06-23T15:48:28+08:00
[INFO] Final Memory: 50M/561M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project stq: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
[ERROR] Command was/bin/sh -c cd /Users/admin/pqy/github/es-server && /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/bin/java -Xmx1024M -jar /Users/admin/pqy/github/es-server/target/surefire/surefirebooter6458186226506641090.jar /Users/admin/pqy/github/es-server/target/surefire/surefire5222514888140147610tmp /Users/admin/pqy/github/es-server/target/surefire/surefire_08480922186965078148tmp
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
from jfasttext.
johndowding
commented on July 17, 2024
Thanks very much harttruong, that fixed my problem too.
from jfasttext.
Related Issues (20)
- Trying to load .ftz model and getting "wrong file format" HOT 6
- Cannot rebuilt jfasttext wrapper with Fasttext version 0.1.0 HOT 2
- This project has a bug that bothers me for many days
- How to build for multiple platforms?
- Dimension of pretrained vectors does not match -dim option HOT 1
- ERROR
- Model file has wrong file format!
- print-word-vectors command does not work HOT 1
- error C2664: cannot convert parameter from "std::istringstream" to "int32_t"
- JVM crashed when load model on Ubuntu HOT 2
- Different results from command line tool HOT 3
- Support for test-label command
- Is the Fasttext dump supported in this wrapper ?
- the prediction is not same as predicted using official c++ HOT 2
- fastText ver 0.9.1 released HOT 1
- Perform unit test on JFastText
- Model file's format is not compatible with this JFastText version!
- Unable to build using mvn
- do not support get sentence vector HOT 1
- Error when building on Mac OS 10.15
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jfasttext.