Some weird first forgetting curves

[L-M-Sherlock]

[Expertium]

These 2 are the weirdest. The other ones may simply be too noisy, but these two have a clear pattern: R goes up as time passes.

[L-M-Sherlock]

Hypothesis: The user has two presets with two graduate intervals. And the materials are very different in these two presets.

To validate the hypothesis, we need to know which presets those cards are from.

[Expertium]

A bit unrelated to the current issue, but I wonder if we should change this:

        for small_rating, big_rating in (
            (1, 2),
            (2, 3),
            (3, 4),
            (1, 3),
            (2, 4),
            (1, 4),
        ):
            if small_rating in rating_stability and big_rating in rating_stability:
                # if rating_count[small_rating] > 300 and rating_count[big_rating] > 300:
                #     continue
                if rating_stability[small_rating] > rating_stability[big_rating]:
                    if rating_count[small_rating] > rating_count[big_rating]:
                        rating_stability[big_rating] = rating_stability[small_rating]
                    else:
                        rating_stability[small_rating] = rating_stability[big_rating]

Specifically, these lines:

                    if rating_count[small_rating] > rating_count[big_rating]:
                        rating_stability[big_rating] = rating_stability[small_rating]
                    else:
                        rating_stability[small_rating] = rating_stability[big_rating]

I wonder if we should just simply switch rating_stability[big_rating] and rating_stability[small_rating].
rating_stability[big_rating], rating_stability[small_rating] = rating_stability[small_rating], rating_stability[big_rating]
I will benchmark that and report the results here.

[Expertium]

I tried this:

            if small_rating in rating_stability and big_rating in rating_stability:
                if rating_stability[small_rating] > rating_stability[big_rating]:
                    rating_stability[big_rating], rating_stability[small_rating] = rating_stability[small_rating], rating_stability[big_rating]

It made results worse.

[L-M-Sherlock]

@dae, is it possible to include the preset or deck info in the benchmark dataset? PresetName and DeckName will be really useful to validate my hypothesis (see #29 (comment)).

Besides, I'm also working on the question: should we use different presets (parameters) in different decks? If so, when? Those features are also helpful to research the question. Here is an initial analysis: https://github.com/open-spaced-repetition/fsrs-when-to-separate-presets

If sharing PresetName and DeckName violates the privacy of AnkiWeb, preset_id or deck_id is OK, too.

[dae]

It would need to be anonymised. I can do it, but it will be a while until I can find the time.

[Expertium]

@dae I also have a request: include both card ID and note ID. I have an idea how to account for reviews of siblings, though it will probably only work "on paper", in the benchmark, and won't be feasible in Anki itself (it requires inserting pseudo-reviews into a card's history). Still, I would appreciate it if you made a dataset with both nid and cid.

[Expertium]

@dae just a reminder about the dataset. I know that changing the way Anki works to make it so that FSRS can access the history of card B while the user is reviewing card A would be too much, but I still want to test whether that would improve the accuracy of FSRS. And it's possible to do that "on paper". For that, the new dataset needs both card ID and note ID. Then we could benchmark how much taking siblings into account affects accuracy. If the results are good, maybe you could consider changing the way Anki works. Allowing arbitrary connections between cards could increase the benefit further, too. I know that Anki doesn't allow to define arbitrary connections between any two cards (and you are probably against it), but it could, in theory, greatly reduce the workload. It could even open up the path for a completely new thing - pre-made decks with pre-defined connections. Domain experts would make decks where they specify which cards are related. Something like this: https://www.justinmath.com/individualized-spaced-repetition-in-hierarchical-knowledge-structures/

[dae]

This was sitting in my inbox and not forgotten about, but it's likely going to be a while longer until I've caught up enough to get to this I'm afraid.

[Expertium]

@dae Once you do decide to make a new dataset, please take this into consideration: https://forums.ankiweb.net/t/question-how-difficult-would-it-be-to-give-fsrs-access-to-certain-info/47645
So ideally, the new dataset would have the following new features:

1. PresetName and DeckName
2. Card ID and note ID
3. Number of reviews done before the current review. This number:
   
   I'm not sure how hard it would be to get it during the review, in real time. That may require major changes. If FSRS-6 is ever going to be a thing, it will need to access this number during every review.
4. Hour of the day, in the 24-hour format. Since the graph below exists, I'm assuming this isn't an unreasonable request. Again, if FSRS-6 will ever be developed, it will have to access this number in real time.
   

3 and 4 are necessary for developing FSRS-6, I already have a pretty clear idea how to use them. Of course, this can wait for months or even more than a year. Plus, Jarrett will probably want to take a break for a while, before coming back to developing FSRS. So this can wait until 2025 or even 2026, so feel free to set a reminder for mid-2025 or something.
Tried it, didn't work.

[dae]

I think half of what you're asking was already in the data I provided to @L-M-Sherlock, or can be inferred from it. He processed it to make it smaller, and things like timestamps were lost. When I have time to generate another dataset, I will do a small test run first so we can confirm it has the required info.

[dae]

Circling back to this, I see that I already put the relevant export code in export_revlog_entries_to_protobuf() in the Anki repo. If you can submit a PR that changes that routine to include the info you require, I will be able to run it on AnkiWeb once I've got AnkiWeb up to date.

Please note the info must maintain privacy. Deck/Preset IDs are ok, but no user-provided content like names should be included. And the export process should be relatively straight-forward - if expensive computation is required, that should be done in a separate step.

[Expertium]

Btw @dae, I tried adding the number of hours since day_starts_at and the number of reviews done today before the next review to FSRS to calculate fatigue, but it didn't improve the metrics. So FSRS will still only be using interval lengths and grades.

Regarding preset and deck names, you can either run them through some hash function to scramble them for the sake of privacy, or just give their IDs (actually, do decks/presets have IDs?), either works well.

[L-M-Sherlock]

If you can submit a PR that changes that routine to include the info you require, I will be able to run it on AnkiWeb once I've got AnkiWeb up to date.

Thanks! I will create a PR later. And I plan to export the tree of deck ids which record the information of hierarchical structure.