Some weird first forgetting curves about srs-benchmark HOT 11 OPEN

@dae, is it possible to include the preset or deck info in the benchmark dataset? PresetName and DeckName will be really useful to validate my hypothesis (see #29 (comment)).

Besides, I'm also working on the question: should we use different presets (parameters) in different decks? If so, when? Those features are also helpful to research the question. Here is an initial analysis: https://github.com/open-spaced-repetition/fsrs-when-to-separate-presets

If sharing PresetName and DeckName violates the privacy of AnkiWeb, preset_id or deck_id is OK, too.

from srs-benchmark.

It would need to be anonymised. I can do it, but it will be a while until I can find the time.

from srs-benchmark.

These 2 are the weirdest. The other ones may simply be too noisy, but these two have a clear pattern: R goes up as time passes.

from srs-benchmark.

Hypothesis: The user has two presets with two graduate intervals. And the materials are very different in these two presets.

To validate the hypothesis, we need to know which presets those cards are from.

from srs-benchmark.

A bit unrelated to the current issue, but I wonder if we should change this:

        for small_rating, big_rating in (
            (1, 2),
            (2, 3),
            (3, 4),
            (1, 3),
            (2, 4),
            (1, 4),
        ):
            if small_rating in rating_stability and big_rating in rating_stability:
                # if rating_count[small_rating] > 300 and rating_count[big_rating] > 300:
                #     continue
                if rating_stability[small_rating] > rating_stability[big_rating]:
                    if rating_count[small_rating] > rating_count[big_rating]:
                        rating_stability[big_rating] = rating_stability[small_rating]
                    else:
                        rating_stability[small_rating] = rating_stability[big_rating]

Specifically, these lines:

                    if rating_count[small_rating] > rating_count[big_rating]:
                        rating_stability[big_rating] = rating_stability[small_rating]
                    else:
                        rating_stability[small_rating] = rating_stability[big_rating]

I wonder if we should just simply switch rating_stability[big_rating] and rating_stability[small_rating].
rating_stability[big_rating], rating_stability[small_rating] = rating_stability[small_rating], rating_stability[big_rating]
I will benchmark that and report the results here.

from srs-benchmark.

I tried this:

            if small_rating in rating_stability and big_rating in rating_stability:
                if rating_stability[small_rating] > rating_stability[big_rating]:
                    rating_stability[big_rating], rating_stability[small_rating] = rating_stability[small_rating], rating_stability[big_rating]

It made results worse.

from srs-benchmark.

@dae I also have a request: include both card ID and note ID. I have an idea how to account for reviews of siblings, though it will probably only work "on paper", in the benchmark, and won't be feasible in Anki itself (it requires inserting pseudo-reviews into a card's history). Still, I would appreciate it if you made a dataset with both nid and cid.

from srs-benchmark.

@dae just a reminder about the dataset. I know that changing the way Anki works to make it so that FSRS can access the history of card B while the user is reviewing card A would be too much, but I still want to test whether that would improve the accuracy of FSRS. And it's possible to do that "on paper". For that, the new dataset needs both card ID and note ID. Then we could benchmark how much taking siblings into account affects accuracy. If the results are good, maybe you could consider changing the way Anki works. Allowing arbitrary connections between cards could increase the benefit further, too. I know that Anki doesn't allow to define arbitrary connections between any two cards (and you are probably against it), but it could, in theory, greatly reduce the workload. It could even open up the path for a completely new thing - pre-made decks with pre-defined connections. Domain experts would make decks where they specify which cards are related. Something like this: https://www.justinmath.com/individualized-spaced-repetition-in-hierarchical-knowledge-structures/

from srs-benchmark.

This was sitting in my inbox and not forgotten about, but it's likely going to be a while longer until I've caught up enough to get to this I'm afraid.

from srs-benchmark.

@dae Once you do decide to make a new dataset, please take this into consideration: https://forums.ankiweb.net/t/question-how-difficult-would-it-be-to-give-fsrs-access-to-certain-info/47645
So ideally, the new dataset would have the following new features:

1. PresetName and DeckName
2. Card ID and note ID
3. Number of reviews done before the current review. This number:
   
   I'm not sure how hard it would be to get it during the review, in real time. That may require major changes. If FSRS-6 is ever going to be a thing, it will need to access this number during every review.
4. Hour of the day, in the 24-hour format. Since the graph below exists, I'm assuming this isn't an unreasonable request. Again, if FSRS-6 will ever be developed, it will have to access this number in real time.
   

3 and 4 are necessary for developing FSRS-6, I already have a pretty clear idea how to use them. Of course, this can wait for months or even more than a year. Plus, Jarrett will probably want to take a break for a while, before coming back to developing FSRS. So this can wait until 2025 or even 2026, so feel free to set a reminder for mid-2025 or something.

from srs-benchmark.

I think half of what you're asking was already in the data I provided to @L-M-Sherlock, or can be inferred from it. He processed it to make it smaller, and things like timestamps were lost. When I have time to generate another dataset, I will do a small test run first so we can confirm it has the required info.

from srs-benchmark.