Convert "Pages & Resources" page to a plugin system #638

bradenmacdonald · 2023-10-17T18:57:29Z

This is a proof-of-concept with a real-world use case for the Frontend Pluggability Summit.

The "Pages & Resources" page of the Course Authoring MFE displays the settings for any "Course Apps" enabled for a particular course. You can read in this ADR more about Course Apps :

The new Course Authoring MFE includes a new UX called "Pages and Resources" for configuring different aspects of the course experience such as progress, wiki, teams, discussions, etc. all from one place. ... This ADR describes how we can use the existing plugin architecture of Open edX to automatically discover such apps and expose them via an API. This API will be used to list the installed apps that are available for a course and to enable/disable these apps using the API.

However, even though Course Apps were specifically designed for this Course Authoring MFE and even though they use a plugin framework on the backend, the frontend is currently hard-coded, and every configurable course app must be built in to frontend-app-course-authoring. Including, for example, 2U's (proprietary?) "Xpert Unit Summaries" configuration UI.

What this PR does: is makes every "Course App" frontend an installable plugin as well. Eight plugins are included in the repo and installed by default (calculator, edxnotes, live, ora_settings, proctoring, progress, teams, wiki), one is still built-in (discussions - because it has a totally different UI than the others), and one is still included in the repo for now but not installed by default (xpert_unit_summary).

A basic plugin with a fairly complex UI:

A plugin showing custom styles (the "reset all units" button has custom SCSS that is part of the plugin):

Technical Details

Plugins will be detected when they are npm installed. For example, to enable the "xpert unit summary" plugin, you have to run the command npm install some-other-repo/plugins/xpert_unit_summary/ --no-save. I put that plugin in a folder called some-other-repo for now to show that it doesn't matter where the plugin is hosted - it can be a totally separate repo/npm-package, and yet it will still install and register with the MFE.

Because we have the course apps functionality on the backend, this particular usage of plugins does not have any need to "list all installed plugins" so you won't find any code for that. Instead, the backend determines which course apps are enabled, and then the MFE just assumes that the corresponding plugin is available and will load it on demand when the user clicks on the app card.

If you did want to be able to list all the installed frontend plugins from the MFE code, it's actually quite easy to do though:

/**
 * Load all installed plugins that match a given regex by scanning the
 * node_modules folder.
 *
 * Note that this is actually done by webpack at build time, so there's almost
 * no runtime overhead to calling this function.
 * 
 * @param {{(): unknown, keys: () => string[]}} loader result of calling require.context()
 * @param {RegExp} regex regular expression that will match the plugins installed in the @openedx-plugins npm namespace
 *                       must contain a capturing group around the plugin ID, like /course-app-(\w+)\/Settings\.jsx$/
 * @returns {{[pluginName: string]: () => Record<string, unknown>}}
 */
function loadPluginContext(loader, regex) {
    // Load a require context. Allows us to get a list of matching plugins without knowing them in advance.
    // See https://webpack.js.org/guides/dependency-management/#requirecontext
    return Object.fromEntries(
        // convert from paths like "./course-app-calculator/Settings.jsx" to plugin names like "calculator"
        loader.keys().map(key => [regex.exec(key)[1], () => loader(key)])
    );
}

export const courseAppSettingsPlugins = loadPluginContext(
    // Due to how webpack works, the following strings need to be static, and we have to duplicate the regex.
    require.context('../node_modules/@openedx-plugins/', true, /course-app-(\w+)\/Settings\.jsx$/),
    /course-app-(\w+)\/Settings\.jsx$/,
);

// Access as e.g. courseAppSettingsPlugins["calculator"]()
// or list all installed plugins with Object.keys(courseAppSettingsPlugins)

How to test it out

If you have the MFE installed and running in development mode already, just check out this branch. You'll need to run npm install within the MFE's root folder then restart the MFE. If you want to test the Xpert Unit Summaries, you'll need to have the corresponding backend plugin installed, and then run npm install plugins/course-apps/xpert_unit_summary/ --no-save to install the frontend plugin into the MFE environment. To make things easier, I used this shim to fake the AI Aside/Xpert plugin and force enable many others.

How would I use this in production?

You'd just have to add some additional step to npm install @openedx-plugins/my-custom-plugin for any custom plugins you have defined when you build the MFE.

openedx-webhooks · 2023-10-17T18:57:35Z

Thanks for the pull request, @bradenmacdonald! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

bradenmacdonald · 2023-10-17T19:03:47Z

plugins/course-apps/calculator/Settings.jsx

+  );
+};
+
+export default CalculatorSettings;


Commentary: This file is mostly unchanged (just moved) but since I changed CalculatorSettings to use useIntl instead of injectIntl, the indentation all changed and git shows it in the diff as a new file.

bradenmacdonald · 2023-10-17T19:04:18Z

plugins/course-apps/edxnotes/Settings.jsx

+    );
+  };
+
+  export default NotesSettings;


Commentary: This file is mostly unchanged (just moved) but since I changed NotesSettings to use useIntl instead of injectIntl, the indentation all changed and git shows it in the diff as a new file. Even though useIntl is way nicer than injectIntl, I didn't change the rest of these plugins, in order to make the diff simpler.

bradenmacdonald · 2023-10-17T19:06:47Z

plugins/course-apps/calculator/package.json

+    "dependencies": {
+    },
+    "devDependencies": {}
+}


In case it's not clear, generally each plugin is just three files:

Settings.jsx which defines the UI (mostly unchanged from how it was before; just updated import paths)

messages.js which contains the strings (unchanged) (optional)

package.json which defines the plugin as an installable module (new)

bradenmacdonald · 2023-10-17T19:14:51Z

src/pages-and-resources/PagesAndResources.jsx

          <PageRoute path={`${path}/:appId/settings`}>
            {
-              ({ match, history }) => {
-                const SettingsComponent = React.lazy(async () => {


Commentary: There was a some kind of bug with this line, where the React.lazy was actually getting re-evaluated many times, and the useEffect of the Xpert Unit Summary plugin was getting called over and over, hammering the backend with requests to populate the redux state. This wasn't actually affecting the current MFE because it was hard-coded to load Xpert Unit Summary settings differently (see removed lines above). But when I changed it to be loaded like the other plugins, this bug occurred. The fix was easy, however: I moved it to a component and put it behind a useMemo hook, so it would only load the state once.

bradenmacdonald · 2023-10-17T19:21:14Z

src/pages-and-resources/PluginSettingsComponent.jsx

+        // if we use a template string here:
+        //     TypeError: Cannot read property 'range' of null with using template strings here.
+        // Ref: https://github.com/babel/babel-eslint/issues/530
+        return await import('@openedx-plugins/course-app-' + appId + '/Settings.jsx'); // eslint-disable-line


Commentary: As you can see, a "Course App Frontend Plugin" is any package that has a name like course-app-______ and includes a Settings.jsx component.

In this proof of concept, all plugins must be installed into a single NPM organization scope (@openedx-plugins/). This is because during the build, webpack has to search for all matching plugins in node_modules. Using an org scope allows it to only have to search that org scope's subfolder (i.e. node_modules/@openedx-plugins/*/ rather than the rootnode_modules folder and all subfolders (which can be huge, as we all know).

I think using the @openedx-plugins/ scope would be a nice clean, performant approach, but it would mean that plugins can't be hosted on the npm registry unless published by Axim. (They could still be installed by git from private repos though). So it may be preferable to remove the org scope restriction to allow more flexible hosting of plugins on npm, even though it could mean slower builds (I didn't test whether it makes much difference or not).

How much of an impact to build-time would loosening the scope have? Would it be any better to make this a tad less dynamic and have the operator provide a list (more likely, a map) of fully qualified module names at build time?

How much of an impact to build-time would loosening the scope have?

I'm not sure. It's possible it makes hardly any difference at all. If we are interested in adopting something like this, I will measure it.

Would it be any better to make this a tad less dynamic and have the operator provide a list (more likely, a map) of fully qualified module names at build time?

If you put the config into an actual .js file that doesn't use dynamic imports at all, sure, you can avoid this entirely. It's definitely simpler. The downside is that that sort of configuration is slightly harder to work with when customizing deployments. I did write a blog post where I recommended that approach at first though, to keep it simple :)

@arbrandes

How much of an impact to build-time would loosening the scope have?

So I actually measured this now and for this course-authoring MFE, using node_modules/@openedx-plugins/course-app-*/Settings.jsx as the pattern takes about 103 seconds to build the MFE and using a looser scope pattern of node_modules/openedx-fe-plugin-course-app-*/Settings.jsx it takes on average 107 seconds. So it actually has an impact of 4 seconds on my machine, which is not trivial.

And without webpack needing to search for those possible plugins to facilitate loading them later (i.e. without the changes in this PR), the build took 97s. So I would say we probably don't want to merge a change that takes the build from 97 to 107 seconds if possible :p.

One way to avoid some of this is to use hard-coded imports instead of dynamic imports, e.g. put something like

const enabledPlugins = { "progress": () => import("path/to/progress/plugin.jsx"), }

this build takes about 100s, 3s longer than normal. In this case webpack is spending 3 seconds building the extra bundles for code splitting and dynamic loading, but not spending any time searching for what plugins/modules might be imported, because they're already listed.

Have you looked into an environment or env.config.js variable for this? i.e. an entry called 'plugins' there can list all the plugins to install, and the code here can iterate over entries in that.

In my testing I'd used webpack's module federation for this, in which case you could pre-build such plugins and then the list of plugins to load could even be provided by runtime config from the MFE config API. I'm not sure if that approach is still in consideration.

@xitij2000 Thanks for the idea. I will look into it, but haven't had time yet.

@xitij2000 Thinking more about this - for this particular use case, it's not really necessary because tehre is already a server-side configuration to choose which of these "course app" plugins are enabled. The MFE now just mirrors the server side and assumes that whatever server-side plugins are installed, there is a corresponding frontend plugin installed.

For other cases, I agree that some config-based enabling would be good. Ideally, you could also pass parameters to each plugin. The MFE runtime config API is probably the best option for this, though it seems quite limited in what data types it provides to the MFE (more like env vars with simple string values than JSON objects with rich configuration).

I don't think the MFE Config API has such a restriction, you can pass any kind of rich object. In fact, the runtime theming configuration relies on it. Even for "environment variables" there is now the 'env.config.js' route that will allow richer configuration.

I agree that the MFE config API is a bit limited. I'd like to see it becoming pluggable and support more than just site configuration as a source. In that case, a backend plugin could inject its own config into it. This is easily doable with hooks and filters.

bradenmacdonald · 2023-10-17T19:41:14Z

some-other-repo/plugins/xpert_unit_summary/settings-modal/SettingsModal.jsx

 import messages from './messages';
 import appInfo from '../appInfo';
 import ResetIcon from './ResetIcon';

+import './SettingsModal.scss';


Commentary: This plugin's SCSS is now loaded when the plugin is displayed, rather than always being included in the MFE's overall styles.

codecov · 2023-11-07T23:13:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (76bb8e8) 89.27% compared to head (decd291) 89.74%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #638      +/-   ##
==========================================
+ Coverage   89.27%   89.74%   +0.46%     
==========================================
  Files         551      517      -34     
  Lines        9738     9062     -676     
  Branches     2099     1907     -192     
==========================================
- Hits         8694     8133     -561     
+ Misses        996      883     -113     
+ Partials       48       46       -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bradenmacdonald · 2023-11-08T19:50:35Z

src/pages-and-resources/__snapshots__/SettingsComponent.test.jsx.snap

@@ -1,3 +0,0 @@
-// Jest Snapshot v1, https://goo.gl/fbAQLP
-
-exports[`SettingsComponent renders LazyLoadedComponent when provided with props 1`] = `[Function]`;


This snapshot test was not implemented properly - the snapshot was just [Function]

bradenmacdonald · 2023-11-08T19:52:03Z

@arbrandes Can you help me get this reviewed?

@xitij2000 FYI - you might want to review this too, if you haven't seen it already.

awais-ansari · 2023-11-15T13:50:32Z

@bradenmacdonald Please rebase this branch with the latest master and resolve the conflict. Other than this PR LGTM.

bradenmacdonald · 2023-11-15T22:42:59Z

Thanks @awais-ansari. Done.

One note before we consider merging this: I have configured the xpert_unit_summary plugin not to be installed by default (so we have an example of a separate plugin). But that means that we have to either make it installed by default or change it so that it gets installed for edx.org use, before we can merge this. The simplest would be if I just make it installed by default, I suppose, which would be like that status quo.

brian-smith-tcril

This looks really great @bradenmacdonald! In my local testing everything worked as expected, and the architecture is quite clear!

In the PR description you mention that

Course Apps were specifically designed for this Course Authoring MFE and even though they use a plugin framework on the backend, the frontend is currently hard-coded, and every configurable course app must be built in to frontend-app-course-authoring

This PR does a wonderful job of addressing that!

I noticed one spot where the hardcoded-ness is still present. I don't think that addressing this is a blocker for merging this PR, but I'd love to hear any thoughts on how to best address this.

https://github.com/openedx/frontend-app-course-authoring/blob/4850302175853bd9a2edec41b99fdcd9be77836e/src/pages-and-resources/data/thunks.js#L16-L28

I'm also very interested to hear your thoughts on how to provide a intuitive experience for operators installing new course apps via tutor plugins. Would it make sense to include the frontend plugin as part of the course app tutor plugin? If so, it'd be really cool to have an example and/or template course app repository showing how all of that fits together.

I also want to thank you for the documentation in the PR description. I'd love to see some of that find a more permanent home (maybe @feanil has thoughts on where that could live). It might be worth making a "document course app frontend plugins" issue on this repo to discuss/track that.

Thanks again for this PR. I'm really excited to see this all coming together.

Here's to a smooth rebase!

bradenmacdonald · 2024-02-13T20:22:10Z

Thanks @brian-smith-tcril ! Sorry I haven't had a chance to reply and rebase yet. Hoping to get that done soon and then will ping you :)

bradenmacdonald · 2024-02-17T00:15:55Z

@brian-smith-tcril OK, I got this updated with master.

hinakhadim · 2024-02-20T11:50:32Z

Hi everyone,
I was trying Plugin based approach for @edx/frontend-component-footer. I am stuck at two errors. If you can help, then its much appreciable.

The debugging resulted that this is originated inside UISlot.

1. You might have mismatching versions of React and the renderer (such as React DOM)
2. You might be breaking the Rules of Hooks
3. You might have more than one copy of React in the same app
See https://reactjs.org/link/invalid-hook-call for tips about how to debug and fix this problem.

Call Stack
 resolveDispatcher
  frontend-component-footer/node_modules/react/cjs/react.development.js:1476:13
 useContext
  frontend-component-footer/node_modules/react/cjs/react.development.js:1484:20
 UISlot
  frontend-component-footer/dist/plugin-template/UISlot.js:160:73
 renderWithHooks
  node_modules/react-dom/cjs/react-dom.development.js:14985:18
 mountIndeterminateComponent
  node_modules/react-dom/cjs/react-dom.development.js:17811:13
 beginWork
  node_modules/react-dom/cjs/react-dom.development.js:19049:16
 HTMLUnknownElement.callCallback
  node_modules/react-dom/cjs/react-dom.development.js:3945:14
 Object.invokeGuardedCallbackDev
  node_modules/react-dom/cjs/react-dom.development.js:3994:16
 invokeGuardedCallback
  node_modules/react-dom/cjs/react-dom.development.js:4056:31
 beginWork$1
  node_modules/react-dom/cjs/react-dom.development.js:23959:7

2.can't able to import packageJson to get the packages name.

Here's the branch for it.
https://github.com/edly-io/frontend-component-footer/tree/hina/footer-ui-plugin

First, I have tried this approach on a dummy footer in a codesandbox just to understand it. Here, the approach works fine with react17.
But when I tried it with locally on footer package (same version react17), it is giving errors I written above.

For injecting plugin dynamically, I import packagesName from package.json and filter packages start with @openedx-plugins/. Then import them and pass to UI context provider for the time being. Don't know is it the right approach or not.

Any insights on this will be appreciated.

bradenmacdonald · 2024-02-20T19:33:39Z

@hinakhadim Can you please open a PR for your changes and ping us on that dedicated PR? The approach you're using is different than what's going on in this PR.

hinakhadim · 2024-02-21T07:43:35Z

@hinakhadim Can you please open a PR for your changes and ping us on that dedicated PR? The approach you're using is different than what's going on in this PR.

Here is the link to the PR. The approach is different as this mfe's backend also follows the plugin based approach. I followed this article.

arbrandes · 2024-02-27T18:01:46Z

How're we looking? @brian-smith-tcril, any objections to merging this?

brian-smith-tcril · 2024-02-27T20:59:42Z

How're we looking? @brian-smith-tcril, any objections to merging this?

arbrandes · 2024-02-28T12:18:14Z

@bradenmacdonald, can we get a last rebase, pretty please?

openedx-webhooks · 2024-02-28T16:50:57Z

@bradenmacdonald 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

brian-smith-tcril · 2024-02-28T16:51:10Z

@arbrandes it was squashable!

bradenmacdonald added 7 commits October 13, 2023 14:58

feat: Make "Pages & Resources" course apps into plugins

b15ade4

feat: move ora_settings

371aa1d

feat: move proctoring

c2fc5c9

feat: move progress

e7af283

feat: move teams

447dfbf

feat: move wiki

2ccc8cf

feat: move Xpert settings

c81182f

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Oct 17, 2023

bradenmacdonald commented Oct 17, 2023

View reviewed changes

bradenmacdonald added 8 commits October 27, 2023 11:47

fix: add webpack.prod.config.js

d1be807

chore: update with latest changes from master

f89c60a

fix: clean up unused parts of package.json files

0a0695b

feat: Add an error message when displaying a Course App Plugin fails

48ba65e

chore: fix various eslint warnings

86b538a

chore: fix jest tests

b385541

chore: update with latest changes from master

dab1291

fix: error preventing "npm ci" from working

6d0481c

feat: better tests for <SettingsComponent>

8d26ef2

bradenmacdonald commented Nov 8, 2023

View reviewed changes

awais-ansari requested a review from arbrandes November 15, 2023 13:39

bradenmacdonald added 2 commits November 15, 2023 14:48

chore: update with latest master

dd45ac1

chore: move xpert_unit_summary into same dir as other plugins

2ab6b2d

bradenmacdonald force-pushed the braden/plugin-test branch from 8a3add6 to 2ab6b2d Compare November 15, 2023 22:48

fix: eslint-import-resolver-webpack is a dev dependency

7257af9

awais-ansari requested a review from davidjoy November 16, 2023 07:21

arbrandes mentioned this pull request Nov 16, 2023

Adolfo's reviews openedx/axim-engineering#696

Closed

This was referenced Dec 13, 2023

feat: slot component for dynamic plugins openedx/frontend-app-communications#183

Closed

feat: slot component for dynamic plugins openedx/frontend-app-communications#184

Closed

chore: update with latest master

10d6f3e

brian-smith-tcril approved these changes Feb 7, 2024

View reviewed changes

brian-smith-tcril mentioned this pull request Feb 14, 2024

Micro-frontend Plugins openedx/platform-roadmap#27

Open

11 tasks

bradenmacdonald added 5 commits February 16, 2024 15:42

Merge branch 'master' into braden/plugin-test

3235066

chore: move learning_assistant to be a plugin too

833845f

feat: for compatibility, install 2U plugins by default

d869816

chore: update with latest master

15ec549

fix: bug with learning_assistant package.json

decd291

brian-smith-tcril merged commit 3c661e1 into openedx:master Feb 28, 2024
6 checks passed

bradenmacdonald deleted the braden/plugin-test branch March 11, 2024 19:15

hinakhadim mentioned this pull request Mar 31, 2024

Footer UI Plugin Approach openedx/frontend-component-footer#405

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert "Pages & Resources" page to a plugin system #638

Convert "Pages & Resources" page to a plugin system #638

bradenmacdonald commented Oct 17, 2023 •

edited

Loading

openedx-webhooks commented Oct 17, 2023 •

edited

Loading

bradenmacdonald Oct 17, 2023

bradenmacdonald Oct 17, 2023

bradenmacdonald Oct 17, 2023 •

edited

Loading

bradenmacdonald Oct 17, 2023

bradenmacdonald Oct 17, 2023

arbrandes Oct 24, 2023

bradenmacdonald Oct 25, 2023

bradenmacdonald Nov 7, 2023

xitij2000 Nov 24, 2023

bradenmacdonald Nov 29, 2023

bradenmacdonald Dec 14, 2023

xitij2000 Dec 15, 2023

bradenmacdonald Oct 17, 2023

codecov bot commented Nov 7, 2023 •

edited

Loading

bradenmacdonald Nov 8, 2023

bradenmacdonald commented Nov 8, 2023

awais-ansari commented Nov 15, 2023

bradenmacdonald commented Nov 15, 2023 •

edited

Loading

brian-smith-tcril left a comment

bradenmacdonald commented Feb 13, 2024

bradenmacdonald commented Feb 17, 2024

hinakhadim commented Feb 20, 2024

bradenmacdonald commented Feb 20, 2024 •

edited

Loading

hinakhadim commented Feb 21, 2024

arbrandes commented Feb 27, 2024

brian-smith-tcril commented Feb 27, 2024

arbrandes commented Feb 28, 2024

openedx-webhooks commented Feb 28, 2024

brian-smith-tcril commented Feb 28, 2024

		@@ -1,3 +0,0 @@
		// Jest Snapshot v1, https://goo.gl/fbAQLP

		exports[`SettingsComponent renders LazyLoadedComponent when provided with props 1`] = `[Function]`;

Convert "Pages & Resources" page to a plugin system #638

Convert "Pages & Resources" page to a plugin system #638

Conversation

bradenmacdonald commented Oct 17, 2023 • edited Loading

Technical Details

How to test it out

How would I use this in production?

openedx-webhooks commented Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bradenmacdonald Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 7, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

bradenmacdonald commented Nov 8, 2023

awais-ansari commented Nov 15, 2023

bradenmacdonald commented Nov 15, 2023 • edited Loading

brian-smith-tcril left a comment

Choose a reason for hiding this comment

bradenmacdonald commented Feb 13, 2024

bradenmacdonald commented Feb 17, 2024

hinakhadim commented Feb 20, 2024

bradenmacdonald commented Feb 20, 2024 • edited Loading

hinakhadim commented Feb 21, 2024

arbrandes commented Feb 27, 2024

brian-smith-tcril commented Feb 27, 2024

arbrandes commented Feb 28, 2024

openedx-webhooks commented Feb 28, 2024

brian-smith-tcril commented Feb 28, 2024

bradenmacdonald commented Oct 17, 2023 •

edited

Loading

openedx-webhooks commented Oct 17, 2023 •

edited

Loading

bradenmacdonald Oct 17, 2023 •

edited

Loading

codecov bot commented Nov 7, 2023 •

edited

Loading

bradenmacdonald commented Nov 15, 2023 •

edited

Loading

bradenmacdonald commented Feb 20, 2024 •

edited

Loading