duolingo / halflife-regression Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
I have noticed the evaluation results for the half-life regression model are slightly different each time I run the code. So I was wondering how the evaluation results presented in the paper are determined.
Are the results given in table 2 of the paper based on one run or are they based on multiple runs (using the best result or the average or something else)?
Thanks!
Thanks for making the great app even better and for uploading the code!
I am working on extending the model to allow each lexeme to have more features and have some minor doubts about the SGD implementation in the code:
The learning rate for the hlr
case seems to differ from the method given in the appendix of the paper by a factor of 1 / (1 + inst.p)
. Is there any particular reason for that?
Usually, SGD iterations are repeated until some convergence criteria is met. In the code, however, it seems that only one pass is made on the complete data-set. Was that because empirically you observe that the results converge by then?
Thanks!
Hi. First of all, thanks for making those codes public. While I was running the codes and data on my laptop, I found the experiment results are a little bit strange and counterintuitive. Though the recall error rate is quite similar to the results in paper, some other feature values really confused me:
else: # fv.append((intern('right'), right)) # fv.append((intern('wrong'), wrong)) fv.append((intern('right'), math.sqrt(1+right))) fv.append((intern('wrong'), math.sqrt(1+wrong)))
I have tried to reproduce your figure of 0.175
MAE for the fixed p-bar of 0.859. On neither the test set, nor the training set can I get this value. Could you detail for me how you have calculated it?
For context I am seeing if I can reproduce some of your results, or try various different modules on the same set of data. Firstly, I would like to reproduce your results as a baseline, but this one has eluded me.
Thanks.
I am trying to run the experiment file, but I get the following error with the dataset that you have provided:
iterator should return strings, not bytes (did you open the file in text mode?) at line 217.
I am using python 3.7.6 and executing experiment.py with the following arguments:
python3 experiment.py data/settles.acl16.learning_traces.13m.csv.gz
If I understood correctly, I see that pp in the result is the predicted probability of forgetting a word. So if pp is low, I should recommend a word associated with the pp as a user might have forgotten the word already. But from the result, I dont see associated word with pp. Then how can I recommend words to users?
And again, is my saying correct? (words with low pp values should be recommended)
Finally, I checked roughly the result and compared p (ground truth) and pp (prediction) and seemed to find that p and pp are quite not matching when they should. For example, when p is 0.0001 pp is 0.9999 and when p is 0.9999, pp is 0.0001. I saw this case quite often when I was skimming. But MAE shows it is very similar as your paper which means training is not broken. Is it normal that p and app are not so matching often? Should I still recommend repeating lexeme according to the pp values?
@sueyhan previously opened a ticket for this, but then closed it without getting a response.
These are the model weights I get for training without lexical features (python experiment.py -l settles.acl16.learning_traces.13m.csv.gz
):
wrong -0.2245
right -0.0125
bias 7.5365
I do not see how it can be correct that the right
feature has a negative weight. This will cause the half life to get shorter as a user gets more correct answers, and therefore the model will predict a lower and lower probability of the user getting correct answers.
How can this be correct?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.