douglasrizzo / catsim Goto Github PK

In 1 PL model I have all the items with high theta (>2). We start the assessment with theta 0. If the first item response is wrong new estimated theta is going above 2 and every subsequent false response is also increasing the theta.
initializer = FixedPointInitializer(0)
selector = MaxInfoSelector()
estimator = NumericalSearchEstimator()

if set(response_vector) == 1:
            return float('inf')
elif set(response_vector) == 0:
            return float('-inf')

These statements will always return false.

When using one of the following item selectors: AStratSelector, AStratBBlockSelector, MaxInfoStratSelector, MaxInfoBBlockSelector; the method stratifies the item bank but does not select the item that is maximally informative for a given theta value inside that stratum.

This issue has already been fixed in both the master and dev branches, as well as in version 0.17.2 submitted recently to Pypi.

I'm just waiting for confirmation from the interested third-party that sent me the issue via e-mail to close it.

The snippet below was used to reproduce the issue, using an item bank of 5000 items. When the response vector is changed, the same item was selected both for a low-proficiency and high-proficiency examinee.

import numpy as np
from catsim.estimation import *
from catsim.initialization import *
from catsim.irt import *
from catsim.selection import *
from catsim.simulation import *
from catsim.stopping import *

items = np.genfromtxt("test.txt")

administered_items = [1, 2346, 17, 4444, 3490]
responses = [True] * len(administered_items)
# responses = [False] * len(administered_items)
print(f"Response vector is: {responses}")

initializer = FixedPointInitializer(0)
selector = MaxInfoStratSelector(test_size=2000)
estimator = NumericalSearchEstimator()
stopper = MinErrorStopper(.2)

new_theta = estimator.estimate(items=items,
                               response_vector=responses,
                               administered_items=administered_items,
                               est_theta=initializer.initialize())
print(f'Estimated Theta is: {new_theta}')

_, start_pointer, end_pointer = selector._get_stratum(items, 5)
organized_items = selector.postsort_items(items, est_theta=new_theta, using_simulator_props=False)
print(f"Items in current stratum are: {organized_items[start_pointer:end_pointer]}")

stratum_items = items[organized_items[start_pointer:end_pointer]]
print(f"Item information in current stratum for current theta is: {irt.inf_hpc(new_theta, stratum_items)}")

item_index = selector.select(items=items,
                             administered_items=administered_items,
                             est_theta=new_theta)

print(f"Selected item index is: {item_index}")

Plot a histogram of the number of times items have been used, ordered by their difficulty.

Hey Douglas,

Just a minor issue, the HTML documentation link you referenced in the the README seems to be pointing to a 404. Perhaps this should be the URL? http://douglasrizzo.github.io/catsim/

Hi,
Can the code be applied to real time data in the place of simulated ones.

I think in the html document, there should be a negative sign above the e, right?

It is that we can understand the base methods and its uses but seems unaware about its usage.

A python notebook where you can view how one one use it on small exaple of student id and question response dataset would become greatly useful and help tracking further errors if encountered.

Hi Douglas,

I noticed that you extend the range for bounded estimation search methods by an arbitrary error margin.
Why is that necessary?
In some of my test cases (1PL item bank, all scores correct except 1 item) I get estimates over 4.0, which is the upper limit of my item bank. Is there a way to limit the estimates to the [-4,4] range?

Thank you!

Kind regards,
Yaiza

Hi! This looks like a great library. Thanks for your work on it!

One thing that might be nice is to link to some projects which can estimate IRT parameters. since this seems to be a common request. I've found the following which seem to be maintained:

https://github.com/nd-ball/py-irt
https://github.com/eribean/girth_mcmc
https://github.com/eribean/girth

I may have missed something, but it looks like some selectors are selecting minimally informative items.

https://github.com/douglasrizzo/catsim/blob/master/catsim/selection.py#L840-L842

Since np.argsort() sorts in ascending order, the Randomesque selector is choosing from the least X informative items.

Should be a quick fix! Either by multiplying info by -1 or using reversed. Happy to open a PR!

Hey, I think that oftentimes the HillClimbingEstimator only does 1 round of estimation (rather than 10, iteratively more precise rounds). The issue is...

• after round 1, round 2 checks values to the left and right of the best theta, starting from the left (value checked before best theta, that best theta has > likelihood than).
• likelihood of the leftmost estimate is always < best theta (by definition), so it ends round 2 right away.
• crazy things start happening, because it sets the lowerbound as the upperbound, and interval sizes can become negative, etc..

I need to patch within the next couple days on DataCamp/catsim, so can submit a PR to fix!

Solution

Code here:

https://github.com/douglasrizzo/catsim/blob/master/catsim/estimation.py#L138-L139

seems like the solution is that max_ll needs to be reset each round?

The links such as this, which refer to a github.io page, don't exist:

https://douglasrizzo.github.io/catsim/item_matrix.html

When trying to run your Catsim in Colab, step #4, initializing theta for the examinee, gives me the following error code:

NameError: name 'initializer' is not defined.

I'm trying to read your code here, but I'm new to Python. Could you help?

Hello Mr. Douglas,

I have a question about the theta estimation returned from the estimator in the example Colab notebook.

The example code goes like below:

# response data
responses = [True, True, False, False]
administered_items = [1435, 3221, 17, 881]

# create objects
initializer = FixedPointInitializer(0)
selector = MaxInfoSelector()
estimator = NumericalSearchEstimator()
stopper = MinErrorStopper(.2)

# initialize the theta
est_theta = initializer.initialize()

# new theta estimation
new_theta = estimator.estimate(items=items, administered_items=administered_items, response_vector=responses, est_theta=est_theta)

# stop?
_stop = stopper.stop(administered_items=items[administered_items], theta=est_theta)
item_index = selector.select(items=items, administered_items=administered_items, est_theta=est_theta)

# simulate a response
true_theta = 0.8
a, b, c, d = items[item_index]
prob = icc(true_theta, a, b, c, d)
correct = prob > random.uniform(0, 1)

administered_items.append(item_index)
responses.append(correct)

Then, I should go back to the new theta estimation step.

My question is whether I should replace est_theta with new_theta before re-calling the estimator method. But then, it would replace the est_theta with the result based on only a few responses, which is probably not good. But when conducting adaptive testing for a new user (say, est_theta is initialized to 0), this approach may make sense (or not).

An alternative (as in the example) is to keep using the est_theta as initialized and accumulating responses until the stop condition is met. In this case, the final estimate is the new_theta, which is not used in the process. In this case, the stop condition and item selection are based on the initial theta. Therefore, the information is based on the initial theta, too.

So, Is either approach appropriate based on the usage? Or I should always follow the way the example indicates. I'd like to know your opinion.

Hi,

I am trying to do some work on item exposure control and would like to use the a-stratification method as a basis for comparison. When I put in the test length argument for this selector, it seems to select 30 items and always administer them. Am I putting the incorrect value here?

Thanks!

Hey Douglas 👋,

I have a question about the package. Shouldn't the logic of the package dictate that if the examinee answers one question correctly, the next question must have a higher "b" parameter, and vice versa? I'm wondering if it's possible that the question selection method isn't always following this logic.

I experimented with the package, and at times, I chose the correct answer, but some questions seemed to have lower "b" parameters than expected. I'd appreciate your input on this.

Thanks for your help! 🙏

I would like to know if there are plans to support MIRT in catsim. If not, does anyone know whether it is feasible or major changes are required?

Hi,

I 've just started studing about IRT (using IRT by Frank Baker and SeockHo Kim), and the catsim library (thx it looks like great work)

While this might be in the documentation (havent found it yet), does catsim support estimation for the item difficult parameter theta?

My problem at hand is: I have a bunch of questions/candidates and interactions without any further knowledge (questions on same topic) and need to properly estimate the question difficulty as well as the candidates proficiency.

Any examples/documentation referrals wld be deeply appreciated. Thx again.

I believe there is a little mistake in the code documentation that is used to generate the docs.

I believe the second theta is not a hat theta:

https://github.com/douglasrizzo/catsim/blob/master/catsim/cat.py#L42

http://douglasrizzo.com.br/catsim/cat.html

where θ^i is examinee i estimated proficiency and θ^i is examinee i actual proficiency.
---------------------------------------- This one ^

Currently there are type hints in the source code, but it looks like no type checking is being performed. Possible set up steps are:

• Ensure a type checker is installed as a dev requirement
• Add type checking as a pre-commit hook
• Add type checking in GitHub CI

Once the package type checks it can be marked as having types so that downstream packages can use it by adding a py.typed marker file: https://www.python.org/dev/peps/pep-0561/#packaging-type-information

Hey,

Not sure if this project is still supported but I'm trying to run the "Autonomous Usage" code that was shown in the docs and I'm getting the below error. Can you say if something has changed or needs to be updated?

File "/Users/rram/PycharmProjects/CAT_Testing/mac_venv/lib/python3.7/site-packages/catsim/selection.py", line 83, in <listcomp>

valid_indexes_low_r = [index for index in valid_indexes if items[index, 4] < self._r_max]

IndexError: index 4 is out of bounds for axis 1 with size 4```

pip installed it, when i try:

from catsim.estimation import *

i get

File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/catsim/estimation.py", line 18
    def __init__(self, precision: int = 6, dodd: bool = False, verbose: bool = False):
                                ^
SyntaxError: invalid syntax

any advice?

I am a bit confused by the current behaviour of MaxInfoSelector. I would expect it to select the item with the maximum fisher information given the current ability estimate, however it seems to instead select based upon where individual item response functions reach their maximum, ignoring the differences in scale on the y-axis between them . See e.g.:

(Figure taked from A Visual Guide to IRT https://www.metheval.uni-jena.de/irt/VisualIRT.pdf )

Imagine there is a small distance on the x-axis between the peaks of the blue and the black curve. If I have an ability estimate at the peak of the black curve I would expect the next item chosen by MaxInfoSelector to be the blue one since it still has higher information even a bit away from its peak.

Basically, the behaviour I want is equivalent to RandomesqueSelector(bin_size=1) (which is what I've now moved onto --- temporarily at least) but the behaviour I actually get for the 2PL model is the same as UrrySelector.

What is the specification for MaxInfoSelector? The citation given is Lion 1977, but in that article there are no equations. If the current behaviour is the intended behaviour, maybe the documentation could make it clear since it's a bit surprising based on the current name/doc to me.

It looks like when items is passed, it is supposed to use those. However, it tries to use self.simulator.items (which might not exist, and shouldn't be used in this case). Here's a link to a diff of how I was able to fix.

datacamp/catsim@d18a8d7