How to Prevent Nesting

[Malkuthe]

Hi there.

To preface this, I've been reading a couple of the discussions on here and I think the advice that's been given thus far has mostly been not to write an extension and use generic directives instead. However, I thought it would be a fun challenge to try and write one, regardless. I'm doing this mostly as a programming exercise and, unfortunately, I seem to have hit something of a stumbling block.

The extension I've been working on recently provides the following syntax: [:<type>]{<attributes>}(<label>). My primary area of concern is the label. In order to allow markdown inside the label, I've wrapped it in contentType: text.

As my extension hooks into text as well, what happens when I have valid syntax inside the label is that there ends up being a valid nested element inside the label.

Let's say we have [:a]{b}(c). Purely for illustration, let's say this produces the output:

<type>a</type>
<attributes>b</attributes>
<label>c</label>

If, for instance, I have [:a]{b}(c_[:d]{e}(f)_), then it would produce the following:

<type>a</type>
<attributes>b</attributes>
<label>c_
    <type>d</type>
    <attributes>e</attributes>
    <label>f<label>
_</label>

This is unintended behavior. I know I could probably just not allow markdown inside the label, but I was just wondering if there was any way to disable a particular tokenizer inside another tokenizer?

I'm sorry if the terminology is confusing. I'm not sure if I'm using terms correctly as I'm still relatively unfamiliar.

[wooorm]

oh no, sorry, I missed this! How did you fare?

Ok, as I understand it, you have some text construct, which allows more text inside of it, but you don‘t want these constructs nested?
Hmm, complex.

Why not use directives?

Why disallow nesting?

Links are similar, but they are [ and then ](whatever). You could similarly parse starts and ends, but your end is quite a common marker.

Alternatively, you could do this when compiling to html? Set a toggle when you enter your customConstruct and turn that off when exiting. When you’re entering but already in one, do something different.

[Malkuthe]

oh no, sorry, I missed this!

Hehe. That's alright! This is mostly a for-fun thing so there's no urgency about it.

Why not use directives?

Mostly because I'm doing this as a programming exercise and I wanted to give it a crack building something from scratch myself. It's been fun so far! This was the only snag I hit.

Why disallow nesting?

Primarily because something would break if they were allowed to nest.

Alternatively, you could do this when compiling to html? Set a toggle when you enter your customConstruct and turn that off when exiting. When you’re entering but already in one, do something different.

Yeah, I've been considering doing this since my other efforts have mostly proven futile. I was just wondering if there were any way to do it during the tokenizing process. As I understand, after a bit of messing around, the main text construct is tokenized first and then there's a second pass after that which tokenizes the text inside of it and that's where the problem comes in since I can't touch that second pass.

[wooorm]

yeah, there’s several passes. First for “document” (blockquotes and lists), then “flow” (ATX headings, code, content), then “content” (definitions, paragraphs, setext headings), and finally “text” (inline stuff).

In commonmark, that’s all that’s needed. But for example, to parse GFM tasklists (- [x] this is checked) it’s needed to know that this is the head paragraph of a head content in a head list item.

That’s different, but somewhat similar to your question: more knowledge about what’s going on.
I’m not sure how to expose this nicely, and fast, though. That seems like an edge case. And your example might be too. I’m not sure how often it’s needed.

There is a way to turn off constructs. But it works on the whole instead of in certain contexts. As you’re hacking, that might be of interest

[Malkuthe]

Mhm. I imagine this might be an edge case sort of thing. I can probably just deal with it, for the most part. xD

There is a way to turn off constructs. But it works on the whole instead of in certain contexts. As you’re hacking, that might be of interest

I was actually wondering if it were possible to do something like:

effects.enter("chunkText", {
  contentType: "text", 
  disable: {
    null: ["customConstruct"]
  }
}

But I imagine that's probably not doable.

[wooorm]

I mean, it could be made to work. But I’m not sure it’s wise. My hunch is that it’s complex?

My idea would be more of exposing a “stack” of token types that aren‘t currently being parsed: ['document', 'flow', 'content', 'paragraph', 'myCustomDirective'] or so. Then you could check if you’re already in myCustomConstruct or not?

[Malkuthe]

I mean, it could be made to work. But I’m not sure it’s wise. My hunch is that it’s complex?

That's what I was thinking, too. Doing it that way might be needlessly complex considering the way things flow currently.

My idea would be more of exposing a “stack” of token types that aren‘t currently being parsed: ['document', 'flow', 'content', 'paragraph', 'myCustomDirective'] or so. Then you could check if you’re already in myCustomConstruct or not?

That's definitely an interesting idea!