Plugins for default/factory firmware sketches

[algernon]

We have a number of sketches in the repository (for the Keyboardio Atreus and the Model 100) which double as factory firmware. We should discuss if we want to enable additional plugins in the default firmware, to enable more features in Chrysalis without having to flash the "experimental" firmware we ship with it. It would also reduce the maintenance burden if we didn't ship the experimental firmware with Chrysalis to begin with.

Off the top of my head, the most useful plugins would be:

• OneShot (+ EscapeOneShot & EscapeOneShotConfig)
• Qukeys
• DynamicMacros
• DynamicTapDance (+ TapDance, to activate it)

None of these change the default behaviour of the keyboard, and should be safe to enable. We could also add SpaceCadet, but like the current experimental firmware, ship it disabled on boot, requiring explicit activation. I'm inclined to not ship SpaceCadet, because it's rather useless in the mentioned setup.

At the moment, Chrysalis does not support DynamicMacros or DynamicTapDance, but the dynamic macro support is quite well advanced, and I expect it to be merged sooner rather than later. Based on that, dynamic tapdance support will be considerably faster to implement, so might aswell start shipping it early in the firmware files, so upgrading to a newer Chrysalis would immediately make it available, without having to flash new firmware too.

[obra]

That works for me.

…

On Mon, May 23, 2022 at 12:15 PM Gergely Nagy ***@***.***> wrote: We have a number of sketches in the repository (for the Keyboardio Atreus and the Model 100) which double as factory firmware. We should discuss if we want to enable additional plugins in the default firmware, to enable more features in Chrysalis without having to flash the "experimental" firmware we ship with it. It would also reduce the maintenance burden if we didn't ship the experimental firmware with Chrysalis to begin with. Off the top of my head, the most useful plugins would be: - OneShot (+ EscapeOneShot & EscapeOneShotConfig) - Qukeys - DynamicMacros - DynamicTapDance (+ TapDance, to activate it) None of these change the default behaviour of the keyboard, and should be safe to enable. We could also add SpaceCadet, but like the current experimental firmware, ship it disabled on boot, requiring explicit activation. I'm inclined to *not* ship SpaceCadet, because it's rather useless in the mentioned setup. At the moment, Chrysalis does not support DynamicMacros or DynamicTapDance, but the dynamic macro support is quite well advanced, and I expect it to be merged sooner rather than later. Based on that, dynamic tapdance support will be considerably faster to implement, so might aswell start shipping it early in the firmware files, so upgrading to a newer Chrysalis would immediately make it available, without having to flash new firmware too. — Reply to this email directly, view it on GitHub <#1176>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALC2AK5ICK3BGAABY4JO3VLPKNTANCNFSM5WWZ6BAQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[algernon]

Thinking this a step further: what if we had both the default and the extended (currently experimental) firmware in Kaleidoscope? Perhaps the difference between the two would be an #ifdef? Or what if we dropped the experimental firmware altogether?

Since @obra's fine with moving the previously experimental plugins (Qukeys, OneShot & co, DynamicMacros and DynamicTapDance once that's stable enough) into the default firmware, there's little left for the experimental one. The only device where that'd make sense is the Model01, where the experimentals don't fit into PROGMEM unless we disable other features.

So, I propose an even more radical approach than in the original issue: lets drop experimental firmware, and ship default, Chrysalis-enabled firmware in Kaleidoscope, in examples/Devices, or even in the examples/ dir of the respective hardware plugins, if Arduino IDE finds them there (it should). This firmware would have all the plugins we want as defaults enabled, they'd live in Kaleidoscope, so keeping them up to date would be less of an issue too. For keyboards like the Model01 where some of the experimental plugins don't fit, we just wouldn't include them.

This, in turn, would result in some changes in Chrysalis too, because we'd need to describe what the firmware contains. We also need a way to reliably identify the libraries a particular firmware was built with for Chrysalis. Having them in Kaleidoscope makes that easier, because it's pretty much Kaleidoscope + the Bundle + Arduino is all we need to note down.

Doing the above would make the Model01-Firmware repository obsolete (which we planned to do anyway, see the discussion in keyboardio/Model01-Firmware#112), along with Chrysalis-Firmware-Bundle.

[algernon]

As an additional point: we should look at the number of eeprom + colormap layers we enable, and should up them to 8 (the limit supported by OneShot and secondary actions) where possible. In particular, on the Model 100.

[algernon]

Most of the things mentioned here and in the referenced issues are now on available on the examples/factory-firmware branch.

The branch itself starts off with #1188, #1196, #1197, and #1198, for practical reasons. Then it moves the factory firmwares to the plugin dirs, pulling in default sketches from external sources if necessary. For the Model01, we pull in Model01-Firmware master, for the Keyboardio Atres and Model100, we move the existing factory firmware from examples/ to the plugin dir. For the ErgoDox EZ, Splitography, and Technomancy Atreus keyboards, the branch copies the latest from Chrysalis-Firmware-Bundle's default set.

I wrote a few paragraphs about what I expect from firmware that we'd ship with Chrysalis, that's in docs/.

Then, the factory firmware sketches are all updated to include plugins based on the aforementioned documentation. Other than making clang-format happy, the Model01 firmware is not changed. The thinking here is that the Model01 is out of production, and its default firmware is already stretching the limits of the hardware (size wise, at least), so adding more plugins in there doesn't work, and I don't want to remove any.

For the Keyboardio Atreus, Technomancy Atreus, ErgoDox EZ and Splitography, plugins that make sense are added to the factory sketches.

For the Model100, I made more substantial changes, the biggest of which is that it uses the new DefaultColormap plugin, along with the similarly newly introduced DefaultLEDModeConfig plugin to replace the NumPad plugin. This makes the numpad colors visible and editable in Chrysalis.

The goal of the branch is to obsolete the Model01-Firmware and Chrysalis-Firmware-Bundle repositories, and provide a single sketch for each keyboard supported by Chrysalis, with as much of the "standard set" of plugins enabled as possible and sensible for a particular device.

One thing I need to tackle is embedding some build information in the firmware that can be queried via Focus, and helps us identify the components used to build a particular version of the firmware. I'm thinking about using a tag like chrysalis-0.10.1-firmware, and embedding that in the firmware. On the Chrysalis side, whenever we build firmware we ship, we'll look at the sha1s of the components we care about (Kaleidoscope, KeyboardioHID, and the Arduino Core), and note that down into a file. To figure out the versions used for a particular Chrysalis release, we'd look at the tag in the firmware, search the respective tag in Chrysalis, and look at the file listing the sha1s.

[algernon]

Oh, and either smoke-sketches has to look for examples in the plugin dirs too, or we need another target to smoke-test the hardware plugin examples.

[algernon]

I've been playing with some ideas in my head, and I think I have a very neat one. We can set up a workflow that will compile only the firmware sketches meant for Chrysalis (currently plugins/Kaleidoscope-Hardware-*/examples/FactoryFirmware), collect info about the build environment (arduino-cli core list, git rev-parse HEAD, $GITHUB_RUN_NUMBER), then shove the compiled firmware and the collected env info into a $GITHUB_RUN_NUMBER directory, add that to a (currently hypothetical) Chrysalis-Firmware-Builds repo, push it up, done.

This way every time we commit to Kaleidoscope, the Chrysalis firmware will get rebuilt.

When we are about to release Chrysalis, we'll pull the latest firmware from the aforementioned repo, add it to Chrysalis, then create a chrysalis/v${VERSION} symlink in the same repo that points to the run-number directory we pulled the firmware from (usually the latest). We can probably automate a big part of this, too: when we pull in firmware from the repo, we'll save the build number in a file within the Chrysalis source, so when we do a release, the Chrysalis release workflow can pick the number up, and do the symlink & tag stuff on the external repo.

This makes it trivial to update the firmware for Chrysalis, and we can be certain that its up to date with regards to Kaleidoscope. If need be, we can even add workflows to the Arduino platform bundles, and trigger a rebuild if the bundle changes without Kaleidoscope changing.

If we embed the build number into the firmware, we can track that down to the precise file, we can view the build logs, we can view the version of the platforms, cores, and Kaleidoscope with ease. As the build number is strictly increasing, it makes for a decent version number too.

[obra]

I want to think some about this before it gets implemented. For one thing, we *need* to be tracking arduino core revs and keyboardiohid revs that these are built against. Perhaps we could use the factory-firmware-builder tooling. But either way, please don't implement this one yet.

…

On Tue, Jun 7, 2022 at 7:44 AM Gergely Nagy ***@***.***> wrote: I've been playing with some ideas in my head, and I think I have a very neat one. We can set up a workflow that will compile only the firmware sketches meant for Chrysalis (currently plugins/Kaleidoscope-Hardware-*/examples/FactoryFirmware), collect info about the build environment (arduino-cli core list, git rev-parse HEAD, $GITHUB_RUN_NUMBER), then shove the compiled firmware and the collected env info into a $GITHUB_RUN_NUMBER directory, add that to a (currently hypothetical) Chrysalis-Firmware-Builds repo, push it up, done. This way every time we commit to Kaleidoscope, the Chrysalis firmware will get rebuilt. When we are about to release Chrysalis, we'll pull the latest firmware from the aforementioned repo, add it to Chrysalis, then create a chrysalis/v${VERSION} symlink in the same repo that points to the run-number directory we pulled the firmware from (usually the latest). We can probably automate a big part of this, too: when we pull in firmware from the repo, we'll save the build number in a file within the Chrysalis source, so when we do a release, the Chrysalis release workflow can pick the number up, and do the symlink & tag stuff on the external repo. This makes it trivial to update the firmware for Chrysalis, and we can be certain that its up to date with regards to Kaleidoscope. If need be, we can even add workflows to the Arduino platform bundles, and trigger a rebuild if the bundle changes without Kaleidoscope changing. If we embed the build number into the firmware, we can track that down to the precise file, we can view the build logs, we can view the version of the platforms, cores, and Kaleidoscope with ease. As the build number is strictly increasing, it makes for a decent version number too. — Reply to this email directly, view it on GitHub <#1176 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALC2DOHGBNQZPZYQYNWTLVN5N6NANCNFSM5WWZ6BAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[algernon]

Nah, not going to implement any tooling just yet, that's why I posted the idea first.

For one thing, we need to be tracking arduino core revs and keyboardiohid revs that these are built against.

Both of those are easy to figure out, record, and keep alongside the built files. We can collect everything we need into a single file per build, so if we have a built firmware version, we can just go to Chrysalis-Firmware-Builds/$N/build-env.json, and it'd be something like this:

{
  "buildNumber": "N",
  "dependencies": {
    "keyboardio:gd32": "<sha>",
    "Kaleidoscope": "<sha>",
    "KeyboardioHID": "<sha>"
  }
}

And we can add whatever we want. As long as we can dig out the info during CI build, we can also shove it into a file and keep a record, and we still only need the build number to track it back. We could also embed all that info into the firmware, too.

[algernon]

After a quick chat with Jesse, we decided that best way forward is to decouple the Chrysalis sketches from Kaleidoscope, rather than pushing them into it, because we do not want to be tied to Kaleidoscope's master, sometimes we want to build the firmware against a stable Kaleidoscope release, rather than the latest, and that's more practical when they're in a separate repository.

So I'll be working on this over at the Chrysalis-Firmware-Bundle repository. For now, only on merging the default & experimental firmwares, and making sure they're all right. There is some tooling around that repo, but we can - and will - iterate over that, and make sure it does everything we need it to do.

[algernon]

I have, what I believe, is a reasonable state at Chrysalis-Firmware-Bundle@next.

It includes the Model01-Firmware (as is, no changes), the Model100 firmware copied from Kaleidoscope (no changes), and the Atreus firmware copied from Kaleidoscope aswell (with EscapeOneShot added). There are docs, and some tooling too.

That's about as far as I can take it right now, we'll need to discuss the next steps before I can proceed.

[algernon]

Next steps have been discussed and acted upon, so this issue can be considered done.