gsbdbi / bm_model Goto Github PK

@susanathey

Following in the applied direction for this paper. I investigated the predictive performance of the model used in the Smart Homework application (two-parameter model).

This application updated the model parameter on a weekly basis. This allows us to define a different test set than the previous sections. We're now using data from production in the week after each training as the test set, which better mirrors how other real-world applications would work.

This is the updated version of the paper, the new section is from pages 23 - 29. I'm primarily looking for some high-level feedback on whether this is a useful addition to keep in the paper before trying to refine it further.

Shi_bayesian_student_model_20211208.pdf

Next steps:

1. Smart Homework used the two-parameter model, but I replicate the training scheme with other models to compare predictive performance using the same test set.
2. Explore "elasticity" between different knowledge points using the best performing factorization model

Also tagging @shanjukta-nath on this. She has been graciously meeting with me to discuss ideas and results for the past months.

Workplan (pending ideation session with Susan on 06.21)

Ideas for further development of paper

1. I propose a customized ELBO to make my models run even faster -- make the paper about computational gains with superior predictive performance
2. Do more interpreting of the latents like we discussed in the defense -- make the paper about balancing prediction and interpretation.
3. More fancy modeling, e.g. modeling teachers' effects as transformation matrix on the student latents -- make the paper to be about innovative modeling
4. Wait on experimental results from smart-homework and publish together as one paper -- make this paper about ML + real-world experiment (maybe something similar to https://academic.oup.com/qje/article-abstract/133/1/237/4095198)

This issue tracks the readying of paper 3 for Henry's dissertation submission:

• revamp literature review with a survey of recent models and the competition
• compare compuational performance against emIRT and VIBO
• compare overall predictive performance against emIRT and VIBO
• compare by knowledge label predictive performance against emIRT and VIBO