MWS: Async rewrite

[Arlen22]

It was kind of fun. This took me about 12 hours, not counting interruptions.

It's not much of a rewrite, more just converting every database call (and everything upstream of every database call) to async/await.

If you haven't used VSCode, you're definitely going to want to explore the code using it to fully appreciate the little improvements that typing can bring.

I'm still working out a few bugs surrounding the somewhat confusing authenticatedUser property. But I'm too tired and I need to get some sleep and take a fresh look at it later.

[github-actions]

Confirmed: Arlen22 has already signed the Contributor License Agreement (see contributing.md)

[netlify]

❌ Deploy Preview for tiddlywiki-previews failed.

Name	Link
🔨 Latest commit	4026a40
🔍 Latest deploy log	https://app.netlify.com/sites/tiddlywiki-previews/deploys/6785f8b40a318d0008b72b3f

[Arlen22]

Before I forget, this adds a NODE_DEV_PATH env variable which when set causes stack traces to print the actual file location instead of the tiddler name. This way you can just click on the line in the stack trace and it takes you directly to the file.

[Arlen22]

It also allows startup and commander code to handle startups and commands that return promises while still maintaining backward compatibility.

[pmario]

Before I forget, this adds a NODE_DEV_PATH env variable which when set causes stack traces to print the actual file location instead of the tiddler name. This way you can just click on the line in the stack trace and it takes you directly to the file.

IMO this one is interesting in general. Can we have this one as a separated PR, or is this PR needed for that feature?

[pmario]

It also allows startup and commander code to handle startups and commands that return promises while still maintaining backward compatibility.

I think that's a "side effect" you wanted for a long time. -- "Sneaking in such functionality" does not work all too well, if this PR is rejected, the side effect is gone.

So is it possible to have this one as a standard PR, with only one goal

[Arlen22]

It also allows startup and commander code to handle startups and commands that return promises while still maintaining backward compatibility.

I think that's a "side effect" you wanted for a long time. -- "Sneaking in such functionality" does not work all too well, if this PR is rejected, the side effect is gone.

I don't remember ever wanting that in those particular scenarios. I've only ever needed it in relation to the file system loading where no callback option was possible. And if I was trying to be sneaky about it I wouldn't put it by itself in the third comment of the PR. But now my lack of sleep is clearly showing!

So is it possible to have this one as a standard PR, with only one goal.

I wanted to make as few changes as possible to avoid cluttering up the git diffs and to make it easy to understand what would have to be changed to convert to async.

In order to do that it was easier to add the async keyword to startup and command functions and then add a bit of code behind the scenes to handle it rather than invoking callback hell while attempting to demonstrate how simple async/await can be.

It wouldn't be hard to convert the startups and commands to use the callback option that is already there. You basically just wrap the code in an async IIFE:

(async () =>  {
  // async function handled here
})().then(() => callback(), err => callback(err))

[Arlen22]

Before I forget, this adds a NODE_DEV_PATH env variable which when set causes stack traces to print the actual file location instead of the tiddler name. This way you can just click on the line in the stack trace and it takes you directly to the file.

IMO this one is interesting in general. Can we have this one as a separated PR, or is this PR needed for that feature?

@pmario Sure. I've opened #8889 for that. Some further details are there.

[Arlen22]

@pmario, is there a style guide anywhere that I can refer to? Is there some kind of prettier spec or eslint rule that's supposed to be applied?

I tried using eslint format, but the entire mws plugin is inconsistently formatted to begin with which just cluttered up the diff. II tried to do whatever caused the least amount of changes in the git diff so I tried to preserve the messed up formatting wherever possible.

That being said, I didn't do as well with sql-tiddler-database.js and the git diff there is particularly bad. I'll see if I can redo it without changing any formatting.

[pmario]

@pmario, is there a style guide anywhere that I can refer to? Is there some kind of prettier spec or eslint rule that's supposed to be applied?

There is an eslint-config.js in the main repo, that should be configured in a way that fits TW style-guide.
It seems in this PR you added some other stuff, that may interfere with that configuration.

[pmario]

I tried using eslint format, but the entire mws plugin is inconsistently formatted to begin with which just cluttered up the diff. II tried to do whatever caused the least amount of changes in the git diff so I tried to preserve the messed up formatting wherever possible.

Auto-formatting should be switched off. At the moment we still have the "function() wrapper" active. All the auto formatters mess up the indentation, because they do want to indent 1 level, that is not necessary.

There is a consensus, that this wrapper function will go away, because it is redundant in 99.9% of the cases. But that's not done yet.

[Arlen22]

@pmario I redid the store folder using an eslint rule, which was much cleaner and preserved the original formatting pretty exactly, whatever that was, which makes the diff cleaner. Somehow it still managed to switch single quotes to double quotes. No idea. It feels like git is running something after I commit but I have no idea.

Sorry, I'm just trying to propose a change to how the MWS plugin is written: one that I feel is very important, and I'm trying hard to communicate exactly what the change would entail as cleanly as I possibly can. I really don't want to fix other people's formatting when I'm not a primary contributor here.

I did notice that eslint has an indent rule with an IIFE option. You'll have to play around with it though because it has some funny defaults.

[pmario]

Indentation problem :/

[Arlen22]

@pmario, Per #8885 (comment), I added a simple worker thread to handle the SQLite calls, but then I realized that the current sync setup is the only thing keeping transactions in order, so it's broken until I figure that out.

[pmario]

@Arlen22 ... Can this be tested? If yes how? -- Did you try Deno2 or Bun to run .ts directly, without the whole transpiler overhead?

[Arlen22]

Not yet. I'm still converting the code.

[Arlen22]

@pmario, in order to run it you need to run the following code:

npm install # there are some new deps
npm run prisma:generate # generate the prisma client code
npm run mws:create # delete and create the sqlite db
npm run arlen # this rebuilds the typescript code and starts the server

[Arlen22]

I'm still working on transactions. But for a proof of concept, I'd say I've proved about enough. It works... but there is so much type weirdness going on throughout the codebase that I'm not sure exactly what's working and what's broken without testing every last code path.

Normally when I'm working with Typescript everything is straightforward. All the types are right in the tooltip, and I get errors whenever I mess something up. Obviously that's not happening here, although there are some things that Typescript is catching even in the Javascript modules.

I recognize that MWS is an amalgamation of the existing server module with a bunch of extra things bolted onto it. I'm not sure if that's good, but I guess this is in the early enough stage that it's not a big deal. I'm sure someone is going to be doing some more passes on MWS and straighten everything out.

That being said, I'm so glad to see this plugin finally making it into the plugin folder. We've needed this for a long time. I'm looking forward to it being finished and being able to use it for all kinds of stuff. I don't know what it will look like exactly, but I've always felt like there's so much untapped potential in the multiwiki concept.

Anyway, I think that's all I'll be doing with this particular proof of concept for the moment. The existing codebase isn't clean enough for me to be able to convert it myself. I really think in Typescript. I would seriously love to use Typescript in TiddlyWiki, but last I heard that's not happening.

I'll leave it open for now and I might come back to it over the next few weeks, I'm not sure.

[Arlen22]

After studying SQLite some more, asking Prisma what they're doing, and some other stuff, I'm starting to realize that the MWS plugin cannot be converted to async in its current state. Why? Transactions! You cannot simply convert a synchronous code path using one SQLite file handle into an async code path handling multiple concurrent requests still using that one SQLite file handle.

So the entire request architecture would need to be changed to only refer to the state variable, for instance, never the $tw global, and the store of each request would need to be assigned from a pool of open SQLite handles and carefully managed to make absolutely certain that no database access can occur outside of those defined paramaters.

Are there ways this could be done quite easily? Absolutely. Would it require a complete rewrite? Definitely. Would it introduce unneeded complexity? Probably. Do our expected use cases actually need it? I don't know, do they? If not, then it's not worth it.

If I'm expecting to use it for a public service like TiddlySpace or something similar, I would have to change a bunch of things because more likely than not I would serve a static HTML instead of rendering the admin wiki for every request. The database access paths which the wiki itself uses are only like 6 simple request paths. Everything else is administration stuff.

A public service like tiddlyspace would not use any part of the MWS plugin:

• It would need a full user management solution with oauth, email support, public signup, onboarding, etc. Because there is so much configuration that goes into this, it's well outside the scope of anything TiddlyWiki could be looking at.
• Recipe/bag management is more or less static. A website like tiddlyspace would define a recipe and set of bags statically which would be setup for each user automatically. Everything would be very static, not requiring UI based management at all. It would probably just be a few lines in a master config somewhere.
• File uploads definitely aren't getting stored on the app server, they'll be going in a separate file sharing service that is designed to handle files. Normally nodejs app servers are scaled horizontally to handle volume, and the file system is ephemeral.
• The index file would definitely be served statically. Obviously it would be rendered dynamically, but I would probably just use a tw --listen to make changes and then run a build command and verify the changes. If we need to include user tiddlers in the index file, it would be done using $tw.preloadTiddlers.

So if you don't need all of these optimizations, a sqlite database should also work just fine. It'll be easy, simple, and work with your existing file system backup solutions.

[Arlen22]

That being said, I still think it should be written in typescript because of the number of bugs I was catching using it. I still can't figure how @Jermolene or anyone else working on the core actually keeps everything straight. All those util functions???

You can add types without changing anything about the module loader system. I think it would work exactly like it is, you just have to know how to declare types into the global system, which isn't as hard as you'd think.