Learner Progress Reset

[Daniel-hershel]

Abstract

This document describes a feature that we are calling Full Course Reset. It serves a specific 2U business need but seems like it will likely provide a benefit to the open-source community as well.

Context & Background

Andhra Pradesh is a state in India whose ministry of education has decided to sponsor edX access on behalf of 400k university students. These students will have access to a subscription that will enable them to enroll in unlimited courses for the duration of their license, which will be approximately six months. Andhra Pradesh is granting users these subscriptions along with one mandatory course (chosen from a pool of approximately ten courses) that learners must complete before they may take any other courses.

Should a learner get to the end of their mandatory course and NOT have a passing grade, there is currently no way to enable them to retake the course. When confronted with course failure otherwise, we suggest that learners wait until the next course run to begin again. However, given the six-month duration of the learners’ licenses, there will likely not be any additional course runs for the mandatory selections from the ministry. Learners would be unable to complete their mandatory courses and would have no recourse.

Scope & Approach

We will add a new tab, “Course Reset” in the Learner Information section of the Support Tools MFE (https://github.com/openedx/frontend-app-support-tools).

Clicking the tab will show a new screen, which consists of a table with three columns: Course, Status, and Action. For each course that a learner is enrolled in which allows course reset, a row will be shown:

• Course: Course display name and key.

• Status: The status of the enrollment wrt/ course reset, that is,

  • Course not yet started
  • Course Ended
  • Course reset already completed, and when it was completed and by whom, with comment.
  • Course reset in progress, and when it was started, and by whom, with comment.
  • Course reset available
  • Course reset failed, when it failed, who started it, and comment
  • Learner is passing the course

• Action: If a reset is available, the column will have a button to reset the course. If it is not, it will be either the word Unavailable or a disabled button

Clicking an active Reset button will pop a confirm dialog, with a text box to enter a “reason” or “memo” and confirming the dialog will send a request to the LMS to reset the course for the user.

The Status row will update to reflect that the request is in progress
Every ~ ten seconds we will poll an endpoint to update the status of the reset job and will update the status when the job is complete

Value & Impact

This feature will allow us to complete the terms of our agreement with Andhra Pradesh, and provide a framework for allowing full course learner reset to be a feature other partners can use in the future.

Milestones and/or Epics

• We will create two new DB models, CourseResetCourseOptIn and CourseResetRecord
  • CourseResetCourseOptIn: This model represents a course run that has opted in to the course reset feature.
    • course_id: CourseKey
    • active: bool
    • created: DateTime
    • modified: DateTime
  • CourseResetAudit: This model represents the act of resetting a learner’s course progress
    • course: ForeignKey(CourseResetCourseOptIn)
    • course_enrollment: ForeignKey(CourseEnrollment): FK to the learner’s enrollment, links to User as well as course_id
    • reset_by: ForeignKey(User) User who performed the reset
    • created: DateTime
    • modified: DateTime
    • state: String, one of [in-progress, complete, enqueued, failed]
    • completed_at: DateTime
  • We will create two HTTP endpoints:
    • List a learner’s “resettable” courses and their statuses re:reset
    • Given a user identifier and a course id, perform the reset by enqueueing a celery task that will run the reset
  • We will create a celery task that will run the reset by looping through all “leaf” blocks in the given course and calling reset_student_attempts on each. We will also clear all grades for the learner for the given course, and all completion data for the given course.

Named Release

Redwood

Timeline

We hope to release this feature by the end of march

Proposed By

2U

Additional Info

https://docs.google.com/document/d/15j6yO6bsVB_IOEbRZXGwSd4oBj3yMGatwoqV2xnmMcY/edit?usp=sharing

Relates PRs

Tasks

Preview Give feedback

1. feat: add list endpoint for course reset enabled enrollments edx-platform#34303
   +0
2. feat: add post endpoint for course reset edx-platform#34305
   +0
3. feat: add celery task to reset course progress for learner edx-platform#34350
   product review +1

[github-actions]

Thanks for your submission, @openedx/open-edx-project-managers will review shortly.

[Daniel-hershel]

@arbrandes here's the ticket for learner reset, thanks for your help and let me know if there is anything else to do on our end.

[arbrandes]

@jmakowski1123, FYI.

[jmakowski1123]

Thanks for submitting, @Daniel-hershel . Some questions about the intended behavior of this feature.

1. Will all organizations have the option to add the feature to reset the course, even if the courses are not included in the pool for the Indian universities?
2. If so, when is that configuration decision made? Is it an instance, org or course-level decision?
3. Does the reset button apply to both self-paced and instructor-paced courses?
4. Is there any UI for the learner? What triggers the reset process? Does that request happen off-platform?
5. What permissions logic does the ability to reset a course follow? In other words, who can reset the permissions? Is it only 2U partner support folks? Or is the intent to allow course teams or course/instance administrators to also make the reset?

I have some general concerns about the data implications. Is the data from a learner's prior attempt at a course completely overridden when a course is reset? Does it behave differently from the current experience, when a learner fails a course and then decides to try again during the next course run? What implications might that have (if any) for the data pipeline? cc @bmtcril for any input or concerns here.

Also just a general note, we are not yet supporting the Support Tools MFE in the community releases, and we don't have it on the list to support yet for Redwood.

[jmakowski1123]

cc @ichintanjoshi

[ormsbee]

High level questions:

1. Is there any limit to how many times it can be reset?
2. What are the rules about when a reset is possible from the system level? I get that it's going to be used for students who don't pass after completing the course, but is that a policy restriction or something built into the code?
3. Is doing all this operationally simpler than making a hidden extra run (either regular or a CCX) and enrolling the student in that?

Questions/Comments on the proposal:

CourseResetCourseOptIn: This model represents a course run that has opted in to the course reset feature.

• course_id: CourseKey
• active: bool
• created: DateTime
• modified: DateTime

Adding to @jmakowski1123's questions about who can configure this: If the course team can specify it, it should probably be added to the course advanced settings and get written to this model on save/publish.

FWIW, I think your naming for this issue is clearer than the naming of this model. "Course Learner Progress Reset" doesn't quite roll off the tongue like "Course Reset", but I think it removes a lot of ambiguity.

CourseResetAudit: This model represents the act of resetting a learner’s course progress

• course: ForeignKey(CourseResetCourseOptIn)
• course_enrollment: ForeignKey(CourseEnrollment): FK to the learner’s enrollment, links to User as well as course_id
• reset_by: ForeignKey(User) User who performed the reset
• created: DateTime
• modified: DateTime
• state: String, one of [in-progress, complete, enqueued, failed]
• completed_at: DateTime

Have you folks looked into using django-user-tasks for this?

We will create a celery task that will run the reset by looping through all “leaf” blocks in the given course and calling reset_student_attempts on each. We will also clear all grades for the learner for the given course, and all completion data for the given course.

You'll likely want to reset the container types as well, since they'll contain things like "last saved position", which might be disorienting (e.g. it might start them on the last Unit when they go a "new" Subsection, because it still has position data from their last attempt).

I'm assuming that it's going to be acceptable that courseware_studentmodulehistory is showing outdated historical data.

One thing that you might want to consider for this is to get the module_ids for this user directly from the database by adding a new method to the UserStateClient. This might save a lot of computation time since it's probable that they haven't done a significant portion of the course. It might also save you a little headache in tracking down things where there's an extra layer or two of XBlock nesting (e.g. SplitTestBlock). The big thing you'd have to look out for is that the indexes are badly designed for courseware_studentmodule, so you can't efficiently query by (student_id, course_id) like you'd expect–you'd actually want to to query by student_id and a prefix match on module_id (since we have a composite index for that and all modules in a given course will start with the same prefix). If you go this route, please tag me on the review.

Also, should their user partition assignments be removed so that they get different content when looking at things like randomized blocks from a library?

I don't know how this will interact with external proctoring systems. A lot of things in our system tie to a single enrollment or tuple of (user, block/module).

Is it okay that notes are preserved for the content? (It seems like it should be.)

I think you'll also want to create a new analytics event around this, because it's going to confuse a lot of downstream tracking log analysis code.

[ichintanjoshi]

@jmakowski1123 as per our discussion on meeting, here's what we did for this.

The requirement was to reset a learner's progress or state because of similar issues regarding them not passing it, and also because of giving multiple certs.

The way we did it was via removal of their StudentModule entry for the course, this essentially meant that learner will have no saved state and would interact with the course as if for the first time.

Prior to that and since then I've not done this sort of feature, however as a developer who's testing enrollments and certificates for different users, this sort of feature could be useful at one go. As of now I'd reset the progress via instructor panel or via staff debug info button if the course has less components. It already has Reset learner state/Delete learner state option so I keep using those mostly.

And if in some course where subsections are locked based on grading and students aren't passing, there too I keep using this feature of resetting learner state and grades via the panel.

[jansenk]

@jmakowski1123:

Will all organizations have the option to add the feature to reset the course, even if the courses are not included in the pool for the Indian universities?
If so, when is that configuration decision made? Is it an instance, org or course-level decision?

The current design is to enter "opted-in" courses via the Django admin, which is only available to certain site/instance staff users. The current design is org agnostic, and is simply a list of courses for which course progress can be reset.

Does the reset button apply to both self-paced and instructor-paced courses?

The current design makes no distinction here.
Intended for self paced courses, but implementation doesn’t exclude any courses explicitly

Is there any UI for the learner? What triggers the reset process? Does that request happen off-platform?

No UI for the learner. The current approach has learners reaching out to site learner support staff via email, who will then perform the reset via the support-tools mfe. Learners will receive an email that the course progress has been reset, and they will see in-course that all course progress has been reset

What permissions logic does the ability to reset a course follow? In other words, who can reset the permissions? Is it only 2U partner support folks? Or is the intent to allow course teams or course/instance administrators to also make the reset?

Currently intended to only be platform-level support teams (ie 2U learner support). We're using the existing restrictions around who is able to access the support-tools MFE.

I have some general concerns about the data implications. Is the data from a learner's prior attempt at a course completely overridden when a course is reset? Does it behave differently from the current experience, when a learner fails a course and then decides to try again during the next course run? What implications might that have (if any) for the data pipeline?

Yes, a learner’s data for their “current” attempt at a course run will be completely removed. It will allow a learner to re-attempt a course run in a case where waiting for the next run may not be desired.
I can't speak for the whole data pipeline; there may be implications I'm unaware of, but the functionality of the reset is mostly based on the existing tool available to staff that clears learner's state for a single block. We will essentially be automating the process of calling that existing feature on every block in the course. This feature will additionally clear any grade records and course completion records for the course run. I'm not aware of any negative implications that could have.

[jansenk]

@ormsbee

Is there any limit to how many times it can be reset?

Currently we only allow one reset per learner per course run

What are the rules about when a reset is possible from the system level? I get that it's going to be used for students who don't pass after completing the course, but is that a policy restriction or something built into the code?

Currently it's built into the code. The restrictions are:

• Cannot perform for a course which has not started
• Cannot perform for a course which has ended
• Cannot perform if a learner has a passing grade in the course
• Cannot perform if there is is a course reset which is in progress or has completed for this learner/course

Is doing all this operationally simpler than making a hidden extra run (either regular or a CCX) and enrolling the student in that?

We believe so

If the course team can specify it, it should probably be added to the course advanced settings and get written to this model on save/publish.

Not intended for course team

FWIW, I think your naming for this issue is clearer than the naming of this model. "Course Learner Progress Reset" doesn't quite roll off the tongue like "Course Reset", but I think it removes a lot of ambiguity.

Noted

Have you folks looked into using django-user-tasks for this?

No. Is there any particular reason that Axim has for us to consider this vs our current approach?

You'll likely want to reset the container types as well, since they'll contain things like "last saved position", which might be disorienting (e.g. it might start them on the last Unit when they go a "new" Subsection, because it still has position data from their last attempt).

Thank you, I didn’t realize we saved state for container blocks. We will add this to our project scope.

I'm assuming that it's going to be acceptable that courseware_studentmodulehistory is showing outdated historical data.

Yes

One thing that you might want to consider for this is to get the module_ids for this user directly from the database by adding a new method to the UserStateClient. This might save a lot of computation time since it's probable that they haven't done a significant portion of the course. It might also save you a little headache in tracking down things where there's an extra layer or two of XBlock nesting (e.g. SplitTestBlock). The big thing you'd have to look out for is that the indexes are badly designed for courseware_studentmodule, so you can't efficiently query by (student_id, course_id) like you'd expect–you'd actually want to to query by student_id and a prefix match on module_id (since we have a composite index for that and all modules in a given course will start with the same prefix). If you go this route, please tag me on the review.

Noted.

Also, should their user partition assignments be removed so that they get different content when looking at things like randomized blocks from a library?

Noted. Unclear at the moment how much value this would add, but we can investigate for potential future releases

I don't know how this will interact with external proctoring systems. A lot of things in our system tie to a single enrollment or tuple of (user, block/module).

Currently does not reset proctored exams as it is not intended for use with courses that use proctored exams but could be a future improvement. Our current approach is to have this review/responsibility be on the platform admins who are configuring these courses.

Is it okay that notes are preserved for the content? (It seems like it should be.)

Notes and bookmarks are not removed

I think you'll also want to create a new analytics event around this, because it's going to confuse a lot of downstream tracking log analysis code.

Yes, we are doing that.

CC @Daniel-hershel @jristau1984

[jmakowski1123]

Got it, thanks so much for elucidating on the details @jansenk .

There are a few reasons why we can't support this in the Core Product.

It's functionality that's currently only intended for 2U-specific persona. And while I can see potentially expanding the use case to more generalized persona (like letting course teams or course admins make the reset), it's a pretty significant and hefty action to delete learner data entirely, and I'd prefer to devote more discovery time to fully understanding the implications of this, particularly from a data pipeline perspective, before we support any functionality or code in the core that enables this.

Additionally, it's being built into the Support Tools MFE, which is a 2U-specific MFE that we currently don't support in Redwood, and are unlikely to support in future community releases, given the business-specific nature of that MFE.

If, down the road, you're interested in exploring a more generalized implementation of this functionality, I'd be happy to coordinate the discovery work needed to get that off the ground, and to pull in other stakeholders who have explored their own implementations for added perspective.

[ormsbee]

@jansenk:

Have you folks looked into using django-user-tasks for this?

No. Is there any particular reason that Axim has for us to consider this vs our current approach?

I originally suggested it because I thought you folks might be creating a course-team facing feature, and django-user-tasks gives useful feedback on progress. But since this is only for site admins, I don't think it's relevant.

I don't know how this will interact with external proctoring systems. A lot of things in our system tie to a single enrollment or tuple of (user, block/module).

Currently does not reset proctored exams as it is not intended for use with courses that use proctored exams but could be a future improvement. Our current approach is to have this review/responsibility be on the platform admins who are configuring these courses.

I am concerned that this feature is going to run into a sea of edge cases when interacting with any number of other things that build off student state: FBE, content gating, milestones, user partitioning, teams, schedules, completion, grade reports, etc. Some may break in really small ways, like maybe the mobile app's concept of "pick up where you left off" is something inaccessible. But I believe this is changing underlying assumptions that are widespread enough where it's going to be a long term maintenance nightmare–one where each step of the way is going to feel like it's fixing "just one more thing", but there's always a new CAT-2 gremlin lying around the next corner.

It's also something that will likely break as new things get developed over time, since many won't be aware of this feature at all. Or as use cases expand, because even if you document it in bold letters that it should not be used in combination with features X, Y, and Z, someone will eventually use it in that situation.

I would strongly urge you folks to consider other ways to work around this, like making unadvertised course runs (or even CCX runs), and taking students through those instead.

@jansenk, @Daniel-hershel, @jristau1984: Given @jmakowski1123's response in the previous comment, let's talk about how this development can be shifted into a plugin (or fork). There might be some pieces of general platform functionality that make sense to pull into edx-platform itself (e.g. the ability to enumerate CSM state for a user across the course), but it sounds like most of this should live in its own space.

[jmakowski1123]

@jristau1984
2 days ago
@Jenna
can you please help me understand the delta between real issue and general worry related to data regarding #331 (comment)? We are leveraging an already existing reset mechanism, just applying it to all problems instead of one. It is essentially automating manual actions that can be done today. It is also only available to runs that are manually flagged by an instance admin, it is not widely available.

To help clarify my confusion...
I was under the impression that "company/2U specific" mostly meant changes that are unable to be leveraged by another instance. Some examples in my head include:
Changes that have company specific logos or branding
Changes that call out to other private code
Changes that are counter to the platform's mission

---

Hi Jeremy, I wanted to address your questions in the same space as the proposal, so we can keep track of the conversation in one place.

I'll ask @ormsbee to elucidate on the specifics of the concern about the data pipeline and unforeseen downstream consequences.

As for the 2U specificity, we've gathered at least two sources of input from others across the community that an option to reset a course would be helpful, but it would need to be use-able functionality by course staff/course teams. One approach, maybe we could imagine it as a permission that could be attached to roles in the new R&P framework. My understanding is that it's currently being built in such a way that only site staff who have access to the Support Tools MFE (ie 2U staff) are able to use it. So out the gate, it's not really use-able more broadly without further investment. Please correct me if I'm wrong here.

I wouldn't say it's a hard and fast rule that all new features must be fully feature complete out the gate. But if the first implementation is an "MVP" approach, it would need to be designed in a way that can scale and be easily generalized down the road. I believe Dave has some specific approach ideas to help describe what a platform-wide view/approach to overwriting student status could look like. Happy to keep this conversation going.

[ormsbee]

We are leveraging an already existing reset mechanism, just applying it to all problems instead of one. It is essentially automating manual actions that can be done today. It is also only available to runs that are manually flagged by an instance admin, it is not widely available.

It’s not just about running a known thing over and over again, though even that would pose operational risks, since firing off a bunch of tasks to recalculate grades on the same subsections could introduce race conditions and locking problems.

Problem reset is typically used to get around an acute issue that a student is currently running into in a course. It’s usually very proximate to what the student is already working on. It doesn’t really try to rewind the clock in the way that this feature is doing. This sounds like it should also reset navigation state, re-init scheduling, re-gate content, etc. Just off the top of my head, we have things like:

1. If I have a prerequisite subsection, are those properly reset for this user?
2. The enterprise repo has a number of integration channel apps that can be configured to push data into other systems. I'm not familiar with the specifics here, but I believe at least some of that data takes the form of subsection aggregate completion data. Will those need to be aware of the reset?
3. Does it roll you out of the team that you’re on? A single problem state reset might be used to let a team re-submit something today. If they submitted something and it was graded for the other students, do we detach them from that team and put them in a new one so that their passing teammates can keep the score they earned while their non-passing teammate gets to do it again?

That's not me doing discovery to try to figure out what else might be impacted. That's just an off-the-cuff answer. I'm sure there are more things out there, not the least of which is what it does to the analytics pipeline and reporting.

Now I realize that the answer for any of these issues can be, “we just won’t use it on a course that uses that feature”, but that doesn’t scale. The feature as proposed needs to understand how every other feature that touches user course state works in order to either clear it out or tell the user that they can't use that feature in conjunction with this reset functionality. Every miss will be a bug of some sort or another, and at some point those bugs will turn into feature requests that try to patch up the holes.

The only thing I can think of that would scale would be to decouple the concept of "user's interactions with the course" from the tuple (user, course). So for instance, we could create a new model like LearningContextStudent where there is only one that is active for a person at any given time, but a user could have multiple instances of it. Then when the user gets a new one for a course and the old one is retired, everything works by default. Retrofitting that kind of abstraction into the system and converting everyone to use it would be amazing, but it's a huge amount of work.

A lower effort thing that might get much the same effect is to make another course-type in way that CCX courses work, or extend CCX to work for this use case. That way, other systems are still insulated from the concept of having to reset and rewind progress–they think it's a different course run altogether. This might be aligned with course and run modeling cleanup that @feanil is championing, but we'd have to do more discovery.

I would love to be wrong about this. If it turns out that the plugin implementation is straightforward and it works well, we can fold it into edx-platform at a later date. If you have the time to talk about a long term implementation that addressees the kinds of maintenance/impact issues I listed, I'd be happy to have that discussion, review discovery work, and work with you folks towards getting an implementation of this in edx-platform. But I don't feel comfortable with the proposed approach going to edx-platform directly.

[ormsbee]

@jansenk, @jristau1984: BTW, if there's something you think I'm just not seeing here, I'm happy to meet to try to understand this better.