[Question] Help with a plot for an educational video (on The AnKing)

[gauerle]

Do you have a doubt about using FSRS in Anki? If so, please post on https://forums.ankiweb.net/ instead.

This issue tracker is primarily intended for questions and suggestions about the algorithm itself, rather than its usage in Anki. In addition, a larger community can provide support on the Anki Forums.

Before posting, please read the following:

Research

Enter an [x] character to confirm the points below:

• I have checked the FAQ and could not find an answer to my question
• I have read the wiki and still felt confused
• I have searched for similar existing questions here

Question
I’m currently working on a video about the updates introduced in Anki version 24.11, collaborating with @AnKingMed. To dive deeper into the topic of updates to the forgetting curve, I’ve been considering comparing the formulas used by different algorithms. Would you happen to have a combined plot that compares FSRS-3, FSRS-4, FSRS-4.5, DASH, and ACT-R? I found one in the materials provided by @Expertium, but unfortunately, it lacks proper scaling. If you already have something like this available, it would be very helpful. I've been thinking about something like this:

[Expertium]

You mean this?

I don't think that x values matter, it's the shape that matters.
More importantly, I don't think this is worth putting in the video in the first place. Users don't care about theory, they care about which button they need to press.
Btw, I wrote this for Anking: https://docs.google.com/document/d/1YUmXD3Y2wR-_WDMMMfKCLGtlJCe603e2/edit

I'm also not sure if right now is the best time to make a video. In the next major release FSRS will most likely become the default, so it's probably better to make a video then. But then again, this could be 3+ or even 6+ months away from today, so perhaps releasing a video now is good.

[gauerle]

Yes, I was referring to this graph from your blog. However, while the values on the x-axis aren’t particularly important, it doesn’t display the percentage probability of recall. The goal was to better illustrate the changes from FSRS-3 to FSRS-4.5, especially regarding the “changes to the shape of the forgetting curve.” The script is almost ready. @AnKingMed shared this document with me, and it has been incredibly helpful!

Edit: The video is structured into two parts: a fast lane and a slow lane. This in-depth explanation will be part of the slow lane. The goal here is to provide a deeper dive into the reasons behind certain changes and configurations, for those who are interested in exploring the details.

[Expertium]

IMO, the video shouldn't focus on the math too much and only briefly mention anything math-heavy. It's just not what the average user is looking for. And I plan to make a Reddit post and finish the benchmarking article eventually, it's just that there is still some stuff left to benchmark.

If you really want to compare FSRS curves, here: https://www.desmos.com/calculator/au54ecrpiz. I forgot to mention that x is in days, specifically.

Red = FSRS v3
Green = FSRS v4
Purple = FSRS-4.5 and FSRS-5

(notice that the naming convention isn't exactly the same, LMSherlock changed it around the time of FSRS-4.5)

[gauerle]

This will be a huge help, thank you very much! Yes, we know the video needs to be concise and straight to the point. That’s why we’ve divided it into two sections: the fast lane, which will cover everything the average user needs to do, and the slow lane, for those who want to dive a bit deeper into the reasoning behind some of the changes.

Also, would you like to review the script once it’s finished?

[Expertium]

Also, would you like to review the script once it’s finished?

Yep.

[Expertium]

Btw
ankitects/anki#3616 (comment)

I reacted with "thumbs down" for two reasons:

1. Lack of automatic optimization. It's not just to save 10 seconds per month. It's because a new user won't even realize that he needs to click "Optimize" in the first place. Sadly, this is one of the least likely features to be implemented. Dae is fully commited to making FSRS the default instead of working on whatever syncing issues automatic optimization would create.
2. No robust solution to the "Hard Misuse" problem: users using Hard as "fail". Around 10% of them. This results in insanely long intervals since FSRS treats Hard as "pass".
   A lot of solutions were proposed, the best of them is (IMO) this:
   
   Dae wants to make the two-button mode (Again and Good) the default, which is a much worse solution. It will make people ask "Guys, this blog/video shows Anki with 4 buttons but I only have 2, am I using the wrong Anki?", but more importantly, it doesn't teach users. If a user wants to enable 4 buttons, they would go to (probably) Preferences and enable some setting, and then fall into the same pitfall of not understanding how buttons work. A good solution should teach the user and actually make them realize the intended usage of buttons.

Put these two together, and you get a pretty bleak picture: most users never click "Optimize" because nothing in Anki tells them to do that (and Dae doesn't want to add a pop-up notification either), and the "Hard Misuse" problem is just swept under the rug.

[gauerle]

I completely agree with your points. I really liked the separate implementation for "again" and "hard/good/easy." Another approach could be renaming "hard" to something like "not easy," in addition to adjusting the color coding. For instance, using dark blue for "hard" or something similar. There’s research in psychophysics about the emotional responses triggered by colors—it could be interesting to conduct a study on user perception regarding this.
Another intriguing idea would be adding a second step to access the "hard" and "easy" options. For example, the main buttons could be "fail" and "pass," and then users could refine their choice by selecting "hard," "good," or "easy." It’s a risky suggestion, though. The best approach might be consulting users who were mistakenly using "hard," based on your research, to find a balanced solution. What if we implemented a step-by-step guided tutorial for first-time users when they install the program? That could help clarify things right from the start... Just some suggestions here.

Implementing FSRS as the default algorithm could indeed be harmful to many, uninformed users—essentially, the majority. However, I’m not sure how we could convince Dae to reconsider his stance on this. I imagine he might feel it’s not Anki responsibility to prevent users from falling into the trap of misusing the "hard" option.

And then there’s the issue of pattern matching, as you mentioned in one of your posts. While it might seem harmless at first, I think it’s something quite important to consider in future updates.

Also, we made sure to include a section in the video specifically dedicated to raising awareness among users about these issues.

[Expertium]

Another intriguing idea would be adding a second step to access the "hard" and "easy" options.

That's pretty much what will happen, I assume. Hard and Easy will have to be enabled in Tools -> Preferences.

What if we implemented a step-by-step guided tutorial for first-time users when they install the program?

This was also proposed, but it's too much work that Dae doesn't want.

And then there’s the issue of pattern matching, as you mentioned in one of your posts. While it might seem harmless at first, I think it’s something quite important to consider in future updates.

I would appreciate if you made a separate video about these note types: https://expertium.github.io/Avoid_Pattern_Matching.html

[Expertium]

I just updated the Google docs document to include potential changes in the next upcoming release.

[brishtibheja]

@gauerle requesting something be mentioned in the video: a lot of people think, due to FSRS's complexity, that it might be taking into account how much time you're spending per review. that can make some people really self-conscious so if the Anking video can clarify that, it'd be helpful for some I think.

[Expertium]

Another matter: I really hope that you and Anking will clarify that it's ok to use Hard, just as long as it's used as a passing grade.

https://www.reddit.com/r/Mcat/comments/1hukasv/i_made_have_made_a_huge_mistake_with_anki_fsrs/
This is something that I noticed a while ago, but didn't bother to save the post for later. This post reminded me.
There is a communication problem. We say "Don't use Hard as a failing grade, only use it as a passing grade", people hear "Don't use Hard".

[AnKingMed]

1. Lack of automatic optimization. It's not just to save 10 seconds per month. It's because a new user won't even realize that he needs to click "Optimize" in the first place. Sadly, this is one of the least likely features to be implemented. Dae is fully commited to making FSRS the default instead of working on whatever syncing issues automatic optimization would create.

I don't think this is a reason to not implement FSRS automatically because this feature can always be added in later.

2. No robust solution to the "Hard Misuse" problem: users using Hard as "fail". Around 10% of them. This results in insanely long intervals since FSRS treats Hard as "pass".

I don't think this should be a blocker either. Misusing the hard button has led to Ease hell for years now and will continue to do so. I agree something should be done about it, but don't think it should stop us from implementing a better algorithm

[Expertium]

So turns out that Anki 25.01 is coming. I just updated my Google doc.

[gauerle]

@Expertium, could you share an email address so I can send over the script for you to review and share your suggestions?

[Expertium]

Why not share it here?
Last time Anking was sharing the script on Github.

EDIT: ok, I added suggestions to your file.
EDIT 2: I ended up making 71 deletions/additions/replacements, lol. I hope that's ok.

[Xemorr]

Btw ankitects/anki#3616 (comment)

I reacted with "thumbs down" for two reasons:

1. Lack of automatic optimization. It's not just to save 10 seconds per month. It's because a new user won't even realize that he needs to click "Optimize" in the first place. Sadly, this is one of the least likely features to be implemented. Dae is fully commited to making FSRS the default instead of working on whatever syncing issues automatic optimization would create.

2. No robust solution to the "Hard Misuse" problem: users using Hard as "fail". Around 10% of them. This results in insanely long intervals since FSRS treats Hard as "pass".
   A lot of solutions were proposed, the best of them is (IMO) this:
   ![image](https://private-user-images.githubusercontent.com/83031600/400077331-ba119535-f50f-47a4-8840-19c677f8974f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzY2NDA0MjksIm5iZiI6MTczNjY0MDEyOSwicGF0aCI6Ii84MzAzMTYwMC80MDAwNzczMzEtYmExMTk1MzUtZjUwZi00N2E0LTg4NDAtMTljNjc3Zjg5NzRmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAxMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMTEyVDAwMDIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE2NzhlZGQ4NDJjZWU0Njc5ZWY3ZDViZGRhYTljMjY3ZWZmZmMxZWFhNTIwMTE1ODMzODk0ZDM2YzFkMDgxYTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.fMh2uZXDlnbae0Z5zVpFH7eLROLiqYAwTOPAN0Am-QY)
   Dae wants to make the two-button mode (Again and Good) the default, which is a much worse solution. It will make people ask "Guys, this blog/video shows Anki with 4 buttons but I only have 2, am I using the wrong Anki?", but more importantly, it doesn't _teach_ users. If a user wants to enable 4 buttons, they would go to (probably) Preferences and enable some setting, and then fall into the same pitfall of not understanding how buttons work. A good solution should teach the user and actually make them realize the intended usage of buttons.

Put these two together, and you get a pretty bleak picture: most users never click "Optimize" because nothing in Anki tells them to do that (and Dae doesn't want to add a pop-up notification either), and the "Hard Misuse" problem is just swept under the rug.

I love the suggestion in your screenshot for how to handle multiple buttons, definitely seems like the most graceful way of handling it.

[gauerle]

Here is the last version of the script! Special thanks to @Expertium for the helpful insights =D

https://docs.google.com/document/d/1NyzjaVh6CAPu4raD8z_1uTz_pn7xro9aX68sdOrG9zc/edit?usp=sharing

Let us know if you have any suggestions!

Edit: as Expertium commented below, it's better this way!

[Expertium]

@gauerle I guess you misunderstood me: I meant "share the link to the Google Doc, but set the settings in such a way that when someone else makes an edit, it will appear a a suggestion instead". It would be far more convenient that way. You can make a second copy of the script just for this purpose, if you want to.

Anyway @brishtibheja @user1823 feel free to chime in!

EDIT: in case gauerle doesn't provide a Google Doc link, just use my copy guys: https://docs.google.com/document/d/11Eaq1ydskPuag3SLPC-BXedopLO_OzS3icE0oKr87VU/edit?usp=sharing

[brishtibheja]

share the link to the Google Doc, but set the settings in such a way that when someone else makes an edit, it will appear a a suggestion instead

That's just comment access.

---

I haven't read all of the script, but personally I'd like a better explanation of Daily Load. It doesn't explain how this differs from average reviews which we already include. (I don't understand this either).

Edit: why the script adds expertium's name? i didn't know
him for a long time and most people wouldn't recognise him either. kinda looks cultish to a wider audience imo.

Edit2: i'd appreciate if AnKing links the sort order simulator in the video description. people have asked for it before so I included the link in resource docs

[Expertium]

Edit: why the script adds expertium's name?

I don't see it. Where?

It doesn't explain how this differs from average reviews which we already include. (I don't understand this either).

Daily Load doesn't change if you switch the timeframe (1 month/3 months/etc.).

[brishtibheja]

I don't see it. Where?

The smart fuzz section. (was it u who came up with the name? i think I called it neo-fuzz? lol)

[Expertium]

@gauerle so what about Anki 25.02? Should we even mention it in the script?

[gauerle]

@Expertium AnKing contacted Dae, who said that the 25.01 update would be delayed until February, turning it into 25.02.

[Expertium]

Yeah, I mean: are we going to talk about the upcoming features of 25.02 in the video?

[brishtibheja]

IMO, there's too many dates in the beginning of this script. Most users are concerned with changes made to Anki (and maybe ver. numbers), not with what released when.

Another thing, the script mentioned optimising every time number of reviews doubles. One change that is important however, is that FSRS now prioritises recent reviews more than old reviews. So, if your learning pattern has changed there's a higher chance of FSRS being able to adapt to it despite few additional reviews. Personally experienced this too.

[gauerle]

@Expertium > Yeah, I mean: are we going to talk about the upcoming features of 25.02 in the video?

I believe it's a good decision!

Edit: However, given the uncertainty of the release date, I think it would be wise to focus on the guaranteed new features. According to Dae's response, the February update would only include a few bug fixes and minor new features.

[gauerle]

IMO, there's too many dates in the beginning of this script. Most users are concerned with changes made to Anki (and maybe ver. numbers), not with what released when.

Among all the options, considering the video's progression and the editing plans, addressing the new features chronologically made a lot of sense. This is one of the reasons why we avoid mentioning specific versions and instead refer to months.

Another thing, the script mentioned optimising every time number of reviews doubles. One change that is important however, is that FSRS now prioritises recent reviews more than old reviews. So, if your learning pattern has changed there's a higher chance of FSRS being able to adapt to it despite few additional reviews. Personally ecep

We mentioned this in the second paragraph on the second page. Edit: could you check it out?

[AnKingMed]

Just fyi for everyone - I think I'm going to try and record the video tomorrow

[brishtibheja]

CMRR provides lower DR values for harder decks and higher for easier decks. I think this should be explicitly mentioned. At least one user thought the opposite would be true which lead to some confusions.

Also, with sort orders, I think we should show people what sort order is best for which metric. Either this, or just show the data on screen so interested folks can pause and read it. Descending R is favourite of Expertium but doesn't mean others won't prefer something else. For example, someone might select a order that helps with knowledge/workload balance or that helps them get through the backlog the quickest.

[Expertium]

@gauerle I made some final suggestions, I hope that you will accept them and that AnKing will make a video with those suggestions included.

[gauerle]

Thanks, @Expertium! You're amazing 😄

[Expertium]

@AnKingMed @gauerle Dae said that he will be releasing the new version "in a few days"

So perhaps we should wait a little bit and refine the script once 25.02 comes out.

[AnKingMed]

Too late :) I already recorded it and we're editing now

[Expertium]

Oh boy. Guess I will have to write a comment immediately after your video comes out, to clarify things.

[AnKingMed]

I did say I was recording it: #719 (comment)

I'm out of town for the next month so I can't do anything later. My only time to record is on weekends typically because I work 60+ hours a week

[Expertium]

Unrelated, but I'd love to see a future video about these note types: https://expertium.github.io/Avoid_Pattern_Matching.html

[Expertium]

Btw, Anki 25.02 is out: https://github.com/ankitects/anki/releases/tag/25.02

[Expertium]

So how's it going?