Enhancement: handle cases where the number of printable characters is different than the chord output

[jdestgermain]

Is your feature request related to a problem? Please describe.
Users that use international layouts and such can output special characters by for example typing ' + a -> á before the letter, etc. But as a result, the number of chord output characters is LONGER than is actually printed/on the computer. This then causes issues during arpeggiates.

Describe the solution you'd like
I don't know the best way around this yet, but it seems like user customizable "number of printable characters" in a chord output might help? Maybe we could automatically detect when these things are being made in chord outputs (seems really hard). Maybe we could provide some generic actions that "add 1 printable character" or "remove 1 printable character"

Additional context
https://discord.com/channels/861730583092658206/1224105680924049469
https://discord.com/channels/861730583092658206/894760876727472178/1230201601872887819

[slendidev]

I am thinking that there could be an option that you can enable/disable such that when it counts the characters, it would ignore the '

[jdestgermain]

I am posting here a set of Discord messages from Starlight who offered their opinions on how this feature (and a few auxiliary ones) could work.

Starlight 🌌 — Yesterday at 9:48 PM
So I sat down for a good long while thinking about this, and have some suggestions. I’m honestly really anxious about being judged over this, and spent a lot of effort thinking of how CCOS might be built up and tried to write this as cleanly as possible. If there are things that don’t make sense then please be gentle with me. :starEmpatheticHeart:

I’m gonna make a whole bunch of assumptions about the internals. I have no idea how the wear leveling works so I’m just going to assume it’s not something I need to worry about. I can’t remember if the device is completely wiped when you upgrade the firmware, but you’ll probably need to anyway.

I’m going to assume that the basic shape of the relevant data blocks are something like this:

typedef struct {
    uint16_t refCount;
    char flags[some_number];
    char* output;
} StringIntern;

typedef struct {
    char key[MAX_KEY_LEN]; # for hash collisions
    StringIntern* value;
} ChordMapEntry;

I’m just assuming the size of the ref count based on the maximum number of stored chords, but if you think that being able to fill the entire addressable library space with chords that output the exact same string is a bit much, then it could instead be packed in with the flags and use a smaller number of bits. :starGiggle: I suppose even a single byte is realistically fine.
As for the backspacing, this is only going to be relevant for arpeggiation and spurring I think?

You probably already have one or more bytes of flags in the interned string struct-- I’d extend it with another byte. And it’s easier/safer to wipe the library when upgrading rather than like trying to rebuild the string pool in-place or something. Spend 7 or 8 bits on an unsigned integer value (depending on if you want a spare bit for another flag), let’s call it the word length. If it's zero, just use the regular deletion method and step along the length of the arpeggiation/spurring string buffer backspacing each time until you get to NUL. If it isn’t zero, backspace that many times instead. (Alternatively you could by default set word length = input characters when creating the chord but I think that’s probably not good choice.) Mention in the manual that 127/255 characters is the maximum length of chords that will behave correctly for arpeggiation/spurring; hopefully that wouldn't be considered restrictive. (If there is already a maximum size for the chord output or for the character deletion counter then adapt to those limitations.) You can say “over 100” or “over 200” in the manual if you want leeway for arpeggiation without having to be very exacting about it.

Arpeggiation is probably going to be messy if you use a language that requires an IME, and even without that, for a lot of languages you’d probably need to be able to user input custom arpeggiation functions. I guess one option there would be something like providing a domain specific language for arpeggiation functions that the Manager compiles into something that can be inserted into and run in CCOS. Or you could just some big lookup tables, but that is more limited in how useful it’s going to be. Since (most) arpeggiation manipulates the length of the buffer you’ll need to use word lengths there too.
So as for using word lengths in a good way in the Manager, it’s a little more involved. To start with, add another text field when creating or editing chords next to the chord input and what CCOS sees as the output, where you can type the expected output including all your IME trigger keys or whatever. The keycodes input are recorded as well and shown in the other, existing field (within the limitations of keycode capture in the browser). The Manager counts how many characters are in the resulting output text field and updates a number text field next to the chord which shows how many characters long the output is. The user can adjust that number if they want to for some reason. If the number of keycodes recorded match the number of output characters, show the output length in the field but set word length to zero when writing. The word length is then recorded to the chord metadata when it’s saved.

Doing it that way means that you can’t read back the output characters when loading the library. If that was something you wanted then for that to work it would probably be the easiest to actually add another char* to the StringIntern struct where the UTF-8 output of the chord is recorded... The other option is maybe to play back the output from the CCOS somehow, stripping browser reserved keycodes, but I feel very unsure about how that would work.
Finally, spend two bits on flags in the CCOS settings:
One flag for whether or not to by default add a space at the end of the chord, defaulting to true. If true, it adds a space to the end of every chord regardless of what you input; if false, it doesn’t. The flag only applies to creating new chords, not to what the stored output is for the existing ones.
The Manager has checkboxes for each chord entry. It checks if the final character of the output string is a space character. If it is, it hides that character and checks the box; if it isn’t, it unchecks the box. If the user then checks or unchecks the box, it adds or removes a space at the end. If there’s more than one space character at the end, the others are listed exactly as today. (You could maybe add an option in the Manager to rewrite the whole library to add or remove a final space character, if that’s something people want.)
One flag that enables asking the user after each impulse chord how many characters were output. If the user just sends another Enter, set it to zero. I’d have this as optional so it doesn’t add a slowdown for people who don’t need this feature.

Hopefully there’s space for all that including the GTM functionality and serial functions.
While we’re at it, I’d suggest if possible to spend n bits on a selector you add to the hash map key for which language a chord belongs to, so that you can have essentially 2ⁿ flexibly sized chord libraries separated by language. Put a selector in the GTM and in the Manager to choose which language is currently active and record any impulse chords to that language. This selector only needs to be stored in volatile memory, so you’ll avoid rewriting it over and over. Add an option in the Manager to choose which language library to browse, as well as a way to move a chord from one language to another (effectively deleting the entry and making a new one). I’d like 4 or more language options personally, but even just 2 would be helpful.

[starlightsys]

Really interested to see how much of it is going to be useful 😁