Comments (7)
I believe that will make the page too crowded, and the calculation required to perform that would impact the loading time of the page. This is usually the landing page for instructor view of an assessment, so the page should be lean and fast. If you feel this is a valid approach, I think a different page should be used for it.
from prairielearn.
Would the question statistics table or the detailed question statistics table in the question_statistics page be viable?
from prairielearn.
The original intention for this column was to show the "difficulty" of each question, to help instructors balance difficulties within pools, arrange the assessment so that questions start easy and then work up to hard, etc. I still this that this is something we need and that this is the right page for it.
However, I agree that we don't want to be calculating this on the fly. Ideally we'd store this in a question stats table and update this as needed in the background (we should be doing the same for the instance question stats).
Exactly what this column should show is also up for debate. Ideally I think we want to show (at least) two values per question:
- A pure measure of one-shot difficulty (e.g., predicted probability correct on first attempt on an exam)
- The predicted points that will be awarded for this question, taking into account the points array. This would also factor in the expected success rate on second (and further) attempts.
The overall goal would be to show a predicted mean score (or maybe distribution of scores) for an exam, to help guide instructors when they are assembling questions into an assessment.
Of course there are lots of unanswered questions above about exactly how to do this. Other factors such as shared questions make this even more complicated, but also more useful. I'm eager to try and work some of these things out.
But as a first step, I think just getting some overall difficulty measure like (1) above would be great.
from prairielearn.
A simple formula for difficulty estimation in multiple choice questions is P=R/T, where:
P = difficulty
R = correct responses
T = total responses
Further iterations taking into account weighted questions could be seen as P = ∑Res/T*100, where:
P = difficulty
∑Res = sum of all grades in a question
T*100 = total responses * 100 percent (assuming 100% as the weight)
from prairielearn.
After analyzing the detailed question statistics provided by PrairieLearn we discussed a potential enhancement that could aid instructors in quickly understanding the difficulty level of questions. This feature would automatically categorize questions into "easy," "medium," or "hard" based on their median scores and the percentage of perfect scores achieved by students.
Here’s a brief explanation of the proposed system:
-
Easy Questions:
- Median score is above a high threshold (e.g., 75% of the maximum possible score).
- Percentage of perfect scores is above a certain level (e.g., more than 60% of students achieve a perfect score).
-
Medium Questions:
- Median score is moderate (e.g., between 40% and 75% of the maximum score).
- Percentage of perfect scores is moderate (e.g., between 30% and 60% of students achieve a perfect score).
-
Hard Questions:
- Median score is below a lower threshold (e.g., less than 40% of the maximum score).
- Percentage of perfect scores is low (e.g., less than 30% of students achieve a perfect score).
This feature could be implemented as an additional analytic tool within the instructor dashboard, providing at-a-glance insights that can inform instructional adjustments and future question design. It would also be valuable for identifying outliers and ensuring that assessments are well-calibrated to measure a range of abilities effectively.
Our first idea would be to implement this feature as an extra column in the question stats page, just as we did with the median.
We would like to know what you think of this iteration and any suggestions would be more than welcome.
from prairielearn.
A few corrections to this previous suggestion:
Categorize the thresholds in order to not have a problem with multiple dimensions, here is a first draft of what they could be:
- Easy Questions:
Median score is above a high threshold, OR the percentage of perfect scores is very high. - Medium Questions:
Median score is moderate, AND the percentage of perfect scores is moderate.
This can be considered the "default" category if neither "easy" nor "hard" criteria are met. - Hard Questions:
Median score is below a low threshold, OR the percentage of perfect scores is very low.
from prairielearn.
After discussion with the core dev group we feel that this issue is too complex for implementation at this point. Instead we recommend focusing on #9699 as originally discussed.
from prairielearn.
Related Issues (20)
- pl-order-blocks uses incorrect format for _files object
- Customise pl-order-blocks answer message
- Questions page: clear filter not working/delayed
- Add semantic headings to all pages
- Add client-side input validation for issue reporting form
- Some elements don't render expected thing for missing input HOT 2
- Ensure LTI 1.3 config/JWKS URLs are always accessible
- Feature request (post-Bootstrap 5): Sticky exam timer HOT 4
- Improve navigational landmarks
- Inconsistent headings HOT 2
- Dropdown display in multiple choice: add placeholder option
- Ensure buttons have accessible labels
- Ensure form inputs have labels HOT 1
- Improve popover accessibility
- Ensure tables have labels
- Simultaneous Workspaces Constantly Refresh HOT 5
- Upgrade to Cheerio 1.0.0 HOT 1
- Manual grading conflict modal: review accessibility
- Transparency Bug on Questions Page
- Fix points/percentage edit popover styling
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prairielearn.