How to create a pure, synchronous and serializable parser?

[jindong-zhannng]

Hi team. I wanna export my DSL parser as a tiny core lib for other my applications. But I am not sure what are the minimal steps.

Firstly I did it like this:

createFormMLServices(EmptyFileSystem).FormML.parser.LangiumParser

It worked. But later I faced problem when I wanna serialize the parsed AST with services.FormML.serializer.JsonSerializer. It reported Ast is not included in any document.

After that I followed langium-lox way to build a real document:

const uuid = v4();
const uri = URI.parse(`memory:///${uuid}.lox`);
const document = services.shared.workspace.LangiumDocumentFactory.fromString(program, uri);
services.shared.workspace.LangiumDocuments.addDocument(document);
await services.shared.workspace.DocumentBuilder.build([document]);
const ast = document.parseResult.value;

Serialization succeeds now. The only flaw is DocumentBuilder.build step is async, then it makes my whole parser to be also async. It doesn't make sense for my use case. And for my purpose, just parsing string to AST, building a real document seems too heavy and strange.

Any suggestions from you?

[msujew]

Hey @jindong-zhannng,

If you just need an AST (without cross reference resolution, diagnostics, etc.), you can just invoke the parser:

const parseResult = services.parser.Parser.parse(program);

The parser is sync. Note that in order to resolve cross references, the async DocumentBuilder API needs to be used.

[jindong-zhannng]

Ok, "cross references" is needed by my grammar, but resolving seems not necessary as I can do it manually according to parsed AST. I will try it firstly.

By the way, I don't quite understand why the DocumentBuilder has to be async when all things are in memory, I mean, building documents from string. This greatly limits the available scenarios for these apis.

[msujew]

The document builder needs to be async so that we can interrupt it at any time and restart it. This is necessary for the language server to stay responsive in case any changes are made by the user in the editor.

Note that I'm a bit confused by the limiting part. In what context are you working that you're not allowed to await an async function call?

[jindong-zhannng]

For example, resolving dsl in a class's constructor. Because langium api is async, I can't finish all initialization tasks only in constructor, I have to expose an async init() api additionally. This is a case how lower-level api affects the design of upper-level API.

In addition, I am also building a React custom hook to resolving dsl for front-end projects. Because underlying api is async, my hook's api can not be straightforward. If you are familiar with React, you could know that in order to handle this situation, we may have to add some extra status such as isInitializing and related logics. This introduces additional complexity into components.

[jindong-zhannng]

The document builder needs to be async so that we can interrupt it at any time and restart it.

If the purpose of async is only to interrupt at any time, would it be better to use a generator (aka, function*) instead of async function? It can yield during execution without causing the outer function lose control of it.

[msujew]

If the purpose of async is only to interrupt at any time, would it be better to use a generator (aka, function*) instead of async function? It can yield during execution without causing the outer function lose control of it.

I'll have to look deeper into it. If I can find a way to make this work without overcomplicating the API surface of our services, it might be worthwhile. The benefit of using async/await is that it keeps the API surface very simple for most use cases.

For example, resolving dsl in a class's constructor. Because langium api is async, I can't finish all initialization tasks only in constructor, I have to expose an async init() api additionally. This is a case how lower-level api affects the design of upper-level API.

In addition, I am also building a React custom hook to resolving dsl for front-end projects. Because underlying api is async, my hook's api can not be straightforward. If you are familiar with React, you could know that in order to handle this situation, we may have to add some extra status such as isInitializing and related logics. This introduces additional complexity into components.

I see. We're using Langium usually either inside of node - as a language server for vscode or inside of a RESTish server - or in browser inside of a webworker. That's why we haven't encountered issues with that yet. However, I can imagine that it makes writing React components a bit more difficult.