Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reps greater than max reviews + New cards per day? #67

Open
ShootingKing-AM opened this issue Nov 13, 2022 · 8 comments
Open

Reps greater than max reviews + New cards per day? #67

ShootingKing-AM opened this issue Nov 13, 2022 · 8 comments

Comments

@ShootingKing-AM
Copy link

ShootingKing-AM commented Nov 13, 2022

Just wanted to know if there are 100(for example) cards in a deck and my reviews and new card reading is limited by the no of hrs i spent on Anki (for example i can only do 2 cards per day) and given my retention rates as below in image, Can i know by when will i mature these cards ?

What i dont understand is why are my reviews so much higher initially that i definately cant achieve doing in future(and more than the settings i.e New cards per day and Reviews per day)? Am i doing something wrong ? My study plan dosnt give me that much time to spend on anki (but anki-ing is necessary)

Thanks !
image

@yuki-tsubaki
Copy link

I am not sure if I understand your situation, but I have a few guesses why your beginning review count is much higher than 2.

  1. Reviews ≠ cards reviewed. Your reviews are the number of times you will be shown a card to grade.
  2. The 'Maximum number of reviews per day' option limits the number of due (previously 'learned') cards that will be shown that day. This limit (rightfully) does not limit the number of reviews (the number on the graph). This is because
  3. When you have a card that is in a 'learning' or 'relearning' stage, that card can come up an unlimited number of times if it is never marked correct enough times in a row to become a 'learned' card (which is basically a card that functions normally according to the Anki algorithm). Because your learning intervals for your cards (15 1440 4320 in minutes and 1/96 1 3 in days) are long enough to span multiple days, you can get very high numbers of reviews relative to your number added per day and due review limit. Even if you never get any wrong (which your's doesn't as it is at 70 92 92), the number of reviews per day is 4 (3 learning steps for each card and one review each day) . With your settings, I ran a simulation for 9999 days with 1 new card per day. The average number of reviews per day was 5. This makes sense because 1 (see new card for the first time) + 1 (due review) + ~3 (average learning steps for each card taking in to account imperfection) = 5 total.

If you want to ensure exactly 2 reviews per day you would need to have exactly 1 learning step. That way you would see the card once, it would become known, and show up tomorrow as a review (when you would once again add 1 card).

@ShootingKing-AM
Copy link
Author

first and foremost, Thanks for your reply.
If you didn't understand my question please put up some questions where did you not understand.

My sense of reviews means, I open anki -> Study Now -> Can only do 2 cards per day (immaterial of type of card - learned, unlearned, leach, relearned, new etc... etc..)

Your explanation did help me about reviews, I thought that 70 in my settings would account for my first step New -> Learned seems like its not from your info. Then this simulator serves no purpose I guess except for knowing how many days Learned cards would changed to matured cards.

And I don't think leaning -> learned would be unlimited, it would also have some success percentage from which we can calculate the worst no of times I need to 'see' the card to put it in learned state. I thought this simulator took into consideration of that thing as well.

@yuki-tsubaki
Copy link

yuki-tsubaki commented Mar 14, 2023

Learning Cards Change Total Reps per Day

And I don't think leaning -> learned would be unlimited, it would also have some success percentage from which we can calculate the worst no of times I need to 'see' the card to put it in learned state. I thought this simulator took into consideration of that thing as well.
Yes you are absolutely correct, under normal circumstances, this would never be unlimited, and the simulator does take the success rates for each review step into account.

The reason I mentioned that it could be unlimited was to make a clear example of how you could build up a lot of daily reviews. I.e. if you keep adding one day, but you never get any of your cards right three times in a row (the number of learning steps), they will stay as 'learning' cards which are not limited by the 'Maximum reviews per day' setting.

So under this hypothetical situation, you would just be adding taking one 'new' card and adding it to the 'learning' set, but (once again if you were not getting them right) the 'learning' cards would just build up (until you start getting them right so they become 'learned' ('learned' cards are limited by the 'Max reviews per day' setting)).

tl;dr:
reps (the number on the y axis of the graph) ≠ max reviews + new cards
because that equation fails to take into account the reviews of 'learning' cards.
The proper 'equation' would be:
reps ≈ max reviews + new cards + learning card reviews
In more detail (still not a formal equation because it would use a lot of formal mathematics notation, and probably end up being too obtuse):
r = M*(c%) + N + L(s*c%)
where r, M, N, and L are the variables in the previous equation (r would be average reps)
and c% is the success percentage relevant to the set of cards in question
and s being the number of learning steps.

The Effects of Learning Stages

Your explanation did help me about reviews, I thought that 70 in my settings would account for my first step New -> Learned seems like its not from your info. Then this simulator serves no purpose I guess except for knowing how many days Learned cards would changed to matured cards.

You are correct in that the 70 is not New -> Learned, but I think you are misunderstanding what it does.

The 70 says 70% of the time after seeing a new card, after 15 minutes (the interval for the first stage of learning) you mark that you remembered that card.
To put it in terms of what you said earlier, with your settings it is not New -> learned, but rather New -> Learning (step 1 of 3).

If you only had one learning step, say 15 (instead of 15 1440 4320), then the 70% would be the about the New -> Learned (because in this case there is only one learning step between New and Learned).

How Long To Mature

Back to your original question (as I think I am starting to understand what you are asking):

Anki (for example i can only do 2 cards per day) and given my retention rates as below in image, Can i know by when will i mature these cards ?

I am assuming you are using Anki's definition of Mature (which is to have an interval on the card ≥ 21 days).
One question: are you asking how long on average each card will take to mature, or are you asking how long will it take you to learn all the 100 cards in your deck and get all of them mature?

For both of these the answer is yes and there are a bunch of different ways to approach them.

I will give a way to answer both of them with this program (although technically you could get a more if you did some more advanced calculations manually):

Average Time to Mature

To simulate this all you need to do is change your settings to 100 new cards per day and 100 as the max reviews per day. By doing this you are basically running 100 simulations of 1 card at once. By doing it this way we get data to see when, on average, cards are due or reach maturity.

I ran the simulation and here is a screenshot of the results.
Screenshot_2023-03-14_11-49-04
As you can see, 77% of the cards were mature by 26 days, so that is a good estimate of how long it will take most of your cards to mature given the success rates in your simulation.

Estimate of Time to Learn All Cards To Maturity (Using given settings)

This is a simple one to measure, all you need to do is run a long simulation with your setup and see when all of them reach maturity.

I ran it and (although the info pane is not showing in the screen shot) got that it will take around 1000 days with those settings.

Screenshot_2023-03-14_11-45-06

(It was very close to 1000, all of them were mature by day 1000.)

Recommendations for Settings and Anki

If You Must Do Exactly 2 Reps Per Day

Assuming imperfect success %, this is impossible to guarantee from Anki's side. This is because 1 new + 1 review + additional review if you ever get something wrong is strictly greater than 2 if you are imperfect. The only way to ensure it would be to just not do the reps if they are ever greater than 2.

The best you can do from a technical point of view is decrease the learning steps to only 1 (thus new cards will only take 1 rep to get to learned). That stays pretty low in general, but using the first % correct from learning and lapse, a graduating interval of 1 day, a learning and lapse step of 10, and a % correct for everything else as 90% some (maybe a third) days are still over 2.

Screenshot_2023-03-14_12-16-23

Personal Suggestions and Thoughts

I don't know exactly what you are doing, but if possible you should change some things if you want Anki to work well for you.

  • I really don't think Anki functions well with an extremely low ceiling of reviews. Even with the best settings for it, the average number of reps for 1new and 1max review is over 2: it is guaranteed to fail sometime. If you want to use a max review cap to even out your reviews, it will only work well if your average number of reviews is lower than the cap (overall, and preferable locally as well).
  • For most things, Anki flash cards work best when they are small. When you have an enormous card (especially one so large that you could at max do 2 a day) it will be wasteful because if you split it up into smaller things, the easiest parts will be easier (and thus grow faster and get out of your way), while only the harder parts will need to be reviewed more.
  • Additionally one of the key reasons Anki can work well is called Active Recall. There has been a lot of research that shows that people learn the best when they try to remember something first and then check it. If you have an enormous card (large enough that you need to learn it outside of Anki before starting it) you may end up missing out on the benefits of active recall and the efficiency of spaced repetition.
  • This last one may just be my personal preference, but I like to use just two short (less than a day) intervals as my learning steps. I find that it works fine for me to go from 10 minutes to 1 day (although I may be a bad example as I still somehow have retention in the 90s despite 400% ease and 130% modifier. Regardless of weather that specific setup works for you, a lower number of learning steps (which are generally short) and a low graduating interval can be nice because you don't have learning cards building up over days, and you can more easily tune things with the general ease and modifier settings (which seem to work fine for me).
  • You can read the Anki manual about these things (it is even available by clicking the question mark next to something in the options for decks).
  • Finally, if you don't need it, I think you can have a longer maximum interval.

@yuki-tsubaki
Copy link

yuki-tsubaki commented Mar 14, 2023

Also I forgot to mention, but I added some new features and reorganized the menu in my version, so that is why my UI looks different in the screenshots.

@ShootingKing-AM
Copy link
Author

ShootingKing-AM commented Mar 15, 2023

Omg ! thanks for the huge writeup : o and lots of info

I am not actually doing 2 reviews (in my sense, reviews are not only for going through the steps but ANY study(new to >21days Maturity) of cards in any state). 2 reviews is hypothetical example to test this simulator if its going out of range.

I feel that its not useful to just limit the going-through-step-reviews, but there has to be an option to limit all kinds of reviews(in my sense, any thing that's being done after clicking 'Study Now'). Because when planning, one would allocate certain amount of hrs to anki and then figure out max no of cards that I can make and also mature(study) them. I am pursuing some strategic studies, not conventional professional studies (difference being, conventional professional like Medicine has limited amount of well defined syllabus and you need to know everything and recall everything. Strategic -> I have infinite amount of syllabus but limited time hence have to use time judiciously and strategically, am expected to know something of everything as deep as possible. Objectively that 'as deep as possible' would be based off my capability to recall). Hence, if my capability is only of 1000 cards (or 4000 data points - 4 datapoints per card). I need not waste time making 3000 cards, which is off my capability and I anyways would not be able to mature them in a given timeperiod. Hence I thought of using this simulator to figure out how many avg cards can I mature with aforementioned settings and recall percentages in a given timeperiod(say 1year)

How to implement exactly 2 reviews per day (hypothetical but good example) programmatically for simulation purposes? This is actually natural. If I open anki and do 2 reviews(in my sense) today and I have to stop doing anki (since I have to do someother tasks and anki's per day time is done(say 1hr is done)). I would close anki and reopen anki tomorrow, those cards which are remaining yesterday are deferred to today. I expected that kind of deferring would be in this simulator because thats what naturally happens (i guess).

I wrote this Issue on Nov 2022, when I was planning for next year. But since there was no reply till then to now (+20weeks). I did some calculations by myself to figure out an approximate number with max and mins, which after testing 20weeks is quite perfect.

Explained my issue more clearly in expectation that future versions would have such a functionality. Anyways thanks for your interest in trying to solve my issue :3

@yuki-tsubaki
Copy link

yuki-tsubaki commented Mar 16, 2023

Disambiguating 'Reviews'

I feel that its not useful to just limit the going-through-step-reviews, but there has to be an option to limit all kinds of reviews(in my sense, any thing that's being done after clicking 'Study Now').

This simulator (as far as I know) perfectly replicates Anki's builtin functionality for 'limiting reviews'. This is set here in the deck options:

option-limit-reviews

This behavior is further explained in the manual.
Deck Options - Anki Manual

The crucial thing that I think needs to be made more clear is that a 'review' not the same as a 'repetition' (in the context of Anki's algorithms.

A review is short for a 'review card'. You can see the number of due reviews for a deck at any given time by checking the green number (listed as 'Due' on the main screen and 'To Review' on the deck overview screen).
For example, doing the following deck would include 59 reviews.
overview
Image from Studying - Anki Manual

A repetition, on the other hand is a time when are presented with a card, and grade it. In my mind, seemingly yours, and probably most people the word 'review' seems generally applicable to this process, and for most purposes, this is totally fine. Sadly understanding and tuning Anki's 'Maximum reviews/day' option is not one of those purposes.

As an example, in the previous deck (with 59 cards due to review), we know that completing this deck must take at least 74 reviews (59 + 9 + 6 = 74), that it will probably take at a minimum 80 (+6 because they probably had more than one learning step), and that it will probably take even more than because some cards may take a few tries (%correct is not 100).

The Answer You Found Is Correct

2 reviews is hypothetical example to test this simulator if its going out of range.
Both in the simulator, and in the real world (with sane deck options and sub 100% accuracy), your test case will go over, and stay (fairly significantly) two repetition (aka non-technical-usage review) over the limit for as long as you add new cards.

Strategic Studies

I am pursuing some strategic studies...
Those are my favorite kind of studies! And, also the kind of study I know the most about. In general what you are talking about is (to some extent; as it is limited by practicality and reality) possible.
There are two angles from which I approach these kinds of problems.

  1. Domain specific: what do I actually want to learn or accomplish, how should I get there
  2. General optimization: whatever I am learning, how can I increase the efficiency (quantity*quality learned/time spent)

Obviously this is non-exhaustive, and the approaches are not mutually-exclusive.

On the first (in a strict sense), I can't really give any advice as I don't know exactly what you are learning, or your goals. You can research and figure out that for yourself.

On the second (once again strictly), you could look into interval-modifier adjustment (lightly and concisely touched on in the manuall, but you can find information on it in various places, and potentially other SRS algorithms.

Interval Modification

This is useful for solving the problem you talked about (wanting to base strategies on your actual retention). Basically the idea is, if you know how your forgetting curve works with a given interval-modifier, you can calculate an interval-modifier that should lead to you having a given retention rate. (Note: interval-modifier as used here is Anki's interval-modifier * card-ease. I am using it in a more general sense as the factor you multiply the card current interval by to equal the new interval when graded good. From here on, im) In practice this is useful because you can use this data to optimize total learning.
For example, you could compare 10 new cards a day with an im x and 15 new cards a day with an im of x+0.5. (im is inversely proportional to repetitions so you can do more new cards for a given daily workload when you increase it) Basically you try to find the optimal point where end retention * new card rate is the highest (because they don't change at the same rate). (This is mainly what I use for my use case)

Other SRS Algorithms

There are some better algorithms (in general), but that doesn't always mean they will be better for your case. They can be worse or at least even with Anki's normal one when you are doing domain/goal specific optimizations (like one I am doing).
Here is one that uses an algorithm I know is well supported scientifically: open-spaced-repetition/fsrs4anki: A modern Anki custom scheduling based on free spaced repetition scheduler algorithm. (It is somewhat better in general simulations, from what other people have shown (although using any SRS is good enough). It can be (and in my case is) worse if you are doing something not general.

One Thing I am Doing That Might Interest You

In fact, the modification that I made to the simulator on my version was to enable simulating a strategy for optimizing learning.

One of the things I use Anki for is SRS for supplementing immersion based language acquisition. The others being things that fit better in the 'general' SRS model (basically memorizing all the important, useful, or interesting things as work on or learn other things: mathematics, programming, recipes, science, research papers, words, plant identification, keybindings for emacs, etc. ).

As language acquisition is primarily determined by the immersion/input side of things, I take advantage of knowing the purpose of SRS in this: to boost my remembrance of words by giving me artificial practice reading/listening to the sentence from the content I found it in. In doing this, all I need to do is boost that internal interval until it reaches the interval at which I naturally come across that word (so I can acquire it). Because of this, I can 'retire' cards when reach a given interval that I decide is good enough.
In addition to the basic benefit of creating an outlet to the system by suspending cards that are old (thus creating an equilibrium point where my work load stays constant; rather than rising indefinitely), making a cut off point means that I can tune the im to most effectively use that time frame (i.e. spreading it effectively because I don't need to worry about long term effects)).

Simulation of Constant Workload

I assume you are not talking about implementing this directly in Anki (although if you tried something like this IRL it might be best to as the indefinite rollover idea sounds problematic). It sound's quite doable to me. I will probably try doing that tomorrow. (The current implementation is feels a little dirty/stringy to me, so it might not be pretty to implement). Also, just so you know, I am not the maintainer of this project (they seem to be gone?). As such, I can't push my modifications to the official version (until someone comes around that can). That said, I can certainly share the modifications, if nobody shows up here, I might start a proper fork.

.
.
.

Well. That was...long. I need to stop writing so much lol.

@yuki-tsubaki
Copy link

I have been busy, so I haven't had time to make the constant workload option, but I did recently read about the v3 scheduler (which you can enable in the review section of the preferences menu). I haven't tested it so I may be misinterpreting what it does, but it seems they are including multi-day learning cards, normal reviews (like what it counts now), and even new cards in the max reviews calculation. This means it should work exactly as you originally expected if you enable it (in terms of number of cards; it would still work very poorly if you set your limit that low).

@ManabBala
Copy link

Really gained some knowledge from this conversation.
@yuki-tsubaki interested in your version of anki simulator. Where/how can i test it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants