Rendered at 00:53:36 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
andai 1 hours ago [-]
Great work. Have you shared this with the Math Academy folks? I bet they'd love to copy your homework :)
I did a little deep dive into this kind of stuff over the summer. I haven't written it up yet, but I'll share some highlights below.
I realized Anki was optimized for the opposite of the problems I actually had. Anki's optimized to minimize study time. It does this by maximizing difficulty. For me, this means maximizing pain!
I'm not low on time; I'm low on willpower. I need it to be fun and easy! Otherwise, I quit and memory goes to zero. Gwern implies I am typical in this regard; most people who try SRS don't stick with it, even if they want to. So "Minimize Pain" seems like a worthy endeavour here.
So, by necessity, I made the reps easy. By making the reps easy, I realized I had accidentally become much more fluent. [0]
(The "automaticity" you mention, comes from practicing more frequently.) In other words, working hard (suffering more) was giving me worse results. Imagine! (I had a similar experience with complex motor skills, but that's a separate post!)
At some point I realized I could replace my entire flashcard app with a lightly modded Anki: increase target retrievability (to increase frequency), and use latency as the score (easy with a tiny addon, and/or custom card type).
As far as the math goes, I got stuck on generating the cards (past arithmetic). How did you do that? :)
(Also, I'd love to see the knowledge graph! I've been trying to reverse engineer the old one from Khan Academy...)
Speaking of fluency, I think "overlearning" is a misnomer. Our standards are too low and we are chronically "underlearning". Similarly, our "Mastery Learning" is actually Temporary Basic Competence...
andai 1 hours ago [-]
Oh, a worthy postscript: Ebbinghaus (of the Forgetting Curves) also discovered the relearning curves. If you've learned something before, it becomes much easier to learn again later.
This is relevant because you can't necessarily apply Very Frequent Reps to your entire knowledge base. There's just not enough time to maintain fluency in everything.
But occasionally, you will want to be fluent in something, which you were once fluent in. Then, you simply shift it into Tier A, spam reps, and rapidly regain what was lost -- for as long as it's relevant to you.
gmays 37 minutes ago [-]
Thank you! ANd thanks for sharing, that was a good overview and I had a similar experience.
To answer your question, I'm a product guy, so it was easier for me to start with the user app/mobile UX I wanted + constraints rather than starting with the cards.
Then I created a problem view in the dev version of the app that let me see every type of problem to see how it rendered, how input worked, etc.
Then with that plus the system around it (details below) gave me higher confidence in generating the 40,000+ problems across all the topics. It still has room for improvement, but happy with how it came out so far.
So, for the problems, they were programmatically generated, but NOT LLM-generated/trusted, so the process would give me the confidence:
1. Define the problem families explicitly. Each has bounded inupts, known mathematical rule, answer type, constraints & presentation rules.
2. A deterministic compiler generated the problems. So, given the same source definitions/versions itd produces the same corpus. so there's no AI inventing random questions at runtime .
3. Correct answers are computed from the underlying math, not from the rendered text. Integer/rational problems use exact arithmetic. More complex symbolic cases use validation recipes, incl. offline SymPy if needed.
4. LaTeX is just presentation (tried other way, didn't work). Internally the problem is represented as structured math, then laTeX/MathJax derives from that structure, so it doesnt generate a LaTeX string and cross fingers its interpreted correctly.
5. Generation/verification are separate steps. Every generated record has to pass mathematical, domain, schema, answer format & presentation checks before it can enter the corpus. Symbolic cases can be independently verified, for example by differentiating a proposed antiderivative.
6. Whole families are tested, not just samples. Finite spaces can be exhaustively enumerated. Larger spaces get property, boundary, invariant, and regression tests. It also runs corpus-wide audits for malformed questions, duplicate IDs, invalid answers, broken rendering, unreachable answer forms, etc
7. The shipped artifact is tied back to what was verified. Versions/content digests bind source definitions, generated problem, validation result & runtime representation together. If something changes, it has to be revalidated rather than reusing old results.
So, starting out I assumed doing it programmatically would be liability. But rather confidence comes BECAUSE it's programmatic. For a large class of problems deterministic generators + exact/exhaustive validation was easier to audit than tens of thousands of hand written questions. So could leverage AI for all that, just had to define the rules/review the system.
I'm very happy with the outcome, but in terms of process it far exceeded my expectations in what I learned about approaching problems like this.
I'm a very heavy AI coding using (I burn hundreds of billions of tokens a year!) so this was a fun way to validate my approach and test some new ones to build high quality experiences, particularly on mobile which is more of a taste thing. The decade plus of product experience really came in handy in guiding the AI here, which took a lot of iteration on the experience and trying different things. It was a blast, and I was generally surprised at how fast it came together. Thank you again.
35 minutes ago [-]
rahimnathwani 3 days ago [-]
This is great, Gabe.
One small piece of feedback: on my Pixel, the Android keyboard obscures the green button. So when I'm answering a numerical question, I use your number pad to enter the digits, and then use the enter key on my Android keyboard. This feels weird, but I guess with regular use it wouldn't.
gmays 3 hours ago [-]
Thank you, I will look at that bug and get something deployed tonight! I only tested on Mac/iPhone, so I appreciate the feedback.
fn-mote 2 hours ago [-]
Even on iOS the keyboard activates in a very distracting way.
gmays 35 minutes ago [-]
Thanks for letting me know, will get this fixed tonight.
hackermailman 1 hours ago [-]
Expii did something similar but abandoned the AI judge that gave you more practice though I don't know why
tehnoslow 3 days ago [-]
A good project. It’s great that the tasks are generated deterministically, without using AI-generated answers
gmays 3 hours ago [-]
Thanks! Yeah, that was a dealbreaker for me since I didn't want to bother with AI slop. So it took a while and a ton of testing/validation, but happy with how it came out.
AlchemistCamp 3 hours ago [-]
This is a great project to see. I'm also an early fan and customer of Math Academy and am glad to see something like this focused on micro skills. As much as memorization is dismissed, having automatic recall base skills lets you get much further, much faster in harder ones.
gmays 2 hours ago [-]
Thank you! Yeah, Justin Skycak's writing really turned me on to the benefits of automaticity. Once you're aware of it you start to notice everywhere you lack it and how much it'd help in raising your ceiling.
fn-mote 2 hours ago [-]
I did some geometry/trig.
Too much fine print. Trig would be better if it showed the picture of a right triangle with all sides labelled. Giving only the essential information doesn’t help with issues like confusing sine and cosine, which a memorization drill should address.
Basically: add more pictures.
Consider that in applications the names of the variables are different. Not sure that I would change this in a memorization app, but there’s no sign the author is aware of it.
gmays 1 hours ago [-]
Thanks, that's good feedback. I will look at options, thank you.
captn3m0 4 hours ago [-]
I made something similar for my partner a while ago for math drills but this is way better (and more comprehensive). I gave up after 3-5 modules. Kudos.
gmays 3 hours ago [-]
Thank you!
rglover 4 hours ago [-]
Dude, hell yeah! I was just shopping for this exact thing not that long ago [1]. I've never been great with math and wanted something I could practice with away from my laptop. Love that it works in the browser, too. Well done, this is really nice.
I did a little deep dive into this kind of stuff over the summer. I haven't written it up yet, but I'll share some highlights below.
I realized Anki was optimized for the opposite of the problems I actually had. Anki's optimized to minimize study time. It does this by maximizing difficulty. For me, this means maximizing pain!
I'm not low on time; I'm low on willpower. I need it to be fun and easy! Otherwise, I quit and memory goes to zero. Gwern implies I am typical in this regard; most people who try SRS don't stick with it, even if they want to. So "Minimize Pain" seems like a worthy endeavour here.
So, by necessity, I made the reps easy. By making the reps easy, I realized I had accidentally become much more fluent. [0]
(The "automaticity" you mention, comes from practicing more frequently.) In other words, working hard (suffering more) was giving me worse results. Imagine! (I had a similar experience with complex motor skills, but that's a separate post!)
At some point I realized I could replace my entire flashcard app with a lightly modded Anki: increase target retrievability (to increase frequency), and use latency as the score (easy with a tiny addon, and/or custom card type).
As far as the math goes, I got stuck on generating the cards (past arithmetic). How did you do that? :)
(Also, I'd love to see the knowledge graph! I've been trying to reverse engineer the old one from Khan Academy...)
--
[0] The ancients knew this principle well...
https://en.wikipedia.org/wiki/Precision_teaching
Speaking of fluency, I think "overlearning" is a misnomer. Our standards are too low and we are chronically "underlearning". Similarly, our "Mastery Learning" is actually Temporary Basic Competence...
This is relevant because you can't necessarily apply Very Frequent Reps to your entire knowledge base. There's just not enough time to maintain fluency in everything.
But occasionally, you will want to be fluent in something, which you were once fluent in. Then, you simply shift it into Tier A, spam reps, and rapidly regain what was lost -- for as long as it's relevant to you.
To answer your question, I'm a product guy, so it was easier for me to start with the user app/mobile UX I wanted + constraints rather than starting with the cards.
Then I created a problem view in the dev version of the app that let me see every type of problem to see how it rendered, how input worked, etc.
Then with that plus the system around it (details below) gave me higher confidence in generating the 40,000+ problems across all the topics. It still has room for improvement, but happy with how it came out so far.
So, for the problems, they were programmatically generated, but NOT LLM-generated/trusted, so the process would give me the confidence:
1. Define the problem families explicitly. Each has bounded inupts, known mathematical rule, answer type, constraints & presentation rules.
2. A deterministic compiler generated the problems. So, given the same source definitions/versions itd produces the same corpus. so there's no AI inventing random questions at runtime .
3. Correct answers are computed from the underlying math, not from the rendered text. Integer/rational problems use exact arithmetic. More complex symbolic cases use validation recipes, incl. offline SymPy if needed.
4. LaTeX is just presentation (tried other way, didn't work). Internally the problem is represented as structured math, then laTeX/MathJax derives from that structure, so it doesnt generate a LaTeX string and cross fingers its interpreted correctly.
5. Generation/verification are separate steps. Every generated record has to pass mathematical, domain, schema, answer format & presentation checks before it can enter the corpus. Symbolic cases can be independently verified, for example by differentiating a proposed antiderivative.
6. Whole families are tested, not just samples. Finite spaces can be exhaustively enumerated. Larger spaces get property, boundary, invariant, and regression tests. It also runs corpus-wide audits for malformed questions, duplicate IDs, invalid answers, broken rendering, unreachable answer forms, etc
7. The shipped artifact is tied back to what was verified. Versions/content digests bind source definitions, generated problem, validation result & runtime representation together. If something changes, it has to be revalidated rather than reusing old results.
So, starting out I assumed doing it programmatically would be liability. But rather confidence comes BECAUSE it's programmatic. For a large class of problems deterministic generators + exact/exhaustive validation was easier to audit than tens of thousands of hand written questions. So could leverage AI for all that, just had to define the rules/review the system.
I'm very happy with the outcome, but in terms of process it far exceeded my expectations in what I learned about approaching problems like this.
I'm a very heavy AI coding using (I burn hundreds of billions of tokens a year!) so this was a fun way to validate my approach and test some new ones to build high quality experiences, particularly on mobile which is more of a taste thing. The decade plus of product experience really came in handy in guiding the AI here, which took a lot of iteration on the experience and trying different things. It was a blast, and I was generally surprised at how fast it came together. Thank you again.
One small piece of feedback: on my Pixel, the Android keyboard obscures the green button. So when I'm answering a numerical question, I use your number pad to enter the digits, and then use the enter key on my Android keyboard. This feels weird, but I guess with regular use it wouldn't.
Too much fine print. Trig would be better if it showed the picture of a right triangle with all sides labelled. Giving only the essential information doesn’t help with issues like confusing sine and cosine, which a memorization drill should address.
Basically: add more pictures.
Consider that in applications the names of the variables are different. Not sure that I would change this in a memorization app, but there’s no sign the author is aware of it.
[1] https://graybearding.bearblog.dev/how-im-teaching-myself-mat...