Custom DSL support with instaparse

[amiika]

How cool it would be to able to parse any string to yaml using ebnf in yamlscript ... or script your own scripting language with yamlscript?

Adding support for instaparse would do it. The parsetree from the instaparser could be easily transformed to YAML for further processing in yamlscript.

Here's a simple example:

(ns noob.instayaml
  (:require [instaparse.core :as insta]
            [clj-yaml.core :as yaml]
            [clojure.walk :as walk]
            [clojure.java.io :as io]))

(defn vector-to-map [data]
  (walk/postwalk
   (fn [x]
     (if (and (vector? x) (keyword? (first x)))
       {(first x) (if (= 2 (count x)) (second x) (vec (rest x)))}
       x))
   data))

(defn vector-to-yaml-string [parsed-result]
  (yaml/generate-string (vector-to-map parsed-result) :dumper-options {:flow-style :block}))

(defn instayaml [expr input]
  (let [parsed ((insta/parser expr) input)]
    (vector-to-yaml-string parsed)))

(defn instayaml-to-file [expr input filename]
  (with-open [writer (io/writer filename)]
    (.write writer (instayaml expr input))))

(print (instayaml 
    "Lambda  = Params <'=>'> Expression
    Params   = <'('> VarList <')'>
    <VarList>  = Var (<','> Var)*
    Expression = Term (Op Term)*
    Term    = Var | Number | '(' Expression ')'
    Op       = '*' | '/' | '+' | '-' | '>' | '<' | '==' | '&&' | '||'
    <Var>      = #'[a-zA-Z_][a-zA-Z0-9_]*'
    Number   = #'[0-9]+'"
        
    "(foo,bar,zyx)=>foo+bar>zyx"))
                          

(instayaml-to-file "S = VAL | EXPR | PAR
                   PAR = <'('> S <')'>
                   <EXPR> = S OP S
                   <VAL> = #'[0-9]+'
                   OP = '+' | '-' | '*' | '/'"

                   "1+(5/3)*2" 
                   
                   "output.yaml")

This prints "(foo,bar,zyx)=>foo+bar>zyx" as:

Lambda:
- Params:
  - foo
  - bar
  - zyx
- Expression:
  - Term: foo
  - Op: +
  - Term: bar
  - Op: '>'
  - Term: zyx

And second example turns "1+(5/3)*2" to output.yaml file:

S:
- S:
  - S: '1'
  - OP: +
  - S:
      PAR:
        S:
        - S: '5'
        - OP: /
        - S: '3'
- OP: '*'
- S: '2'

[ingydotnet]

This is a cool idea, but we should do it by fixing ys to use arbitrary clojure libraries.

I created this repo https://github.com/ingydotnet/instayaml?tab=readme-ov-file to start playing around with getting this to work.

Please take a look. It's currently failing because of a hiccup in ys pod loading. Looking into that now.

Also, Instaparse might be useful for replacing the YS custom parser in yamlscript.ysreader.clj.

[ingydotnet]

@delonnewman ^^

[ingydotnet]

Some related links

• external-libs for database queries #193 (comment)
• https://github.com/Engelberg/instaparse
• https://github.com/babashka/pod-registry/blob/master/examples/tools-deps-native.clj
• https://github.com/babashka/babashka/blob/43584284641918334bb1c47ede04d9c5a74e4763/src/babashka/main.clj#L891-L937
• https://github.com/babashka/sci/blob/7bc5c4f1e35fb18a85005c47d7659213ff7f64e9/src/sci/impl/namespaces.cljc#L1800-L1822
• https://book.babashka.org/#_pod_registry
• https://github.com/borkdude/deps.clj
• https://github.com/babashka/instaparse-bb/tree/main
• https://github.com/babashka/pod-babashka-instaparse
• https://github.com/babashka/tools-deps-native
• https://github.com/babashka/babashka?tab=readme-ov-file#differences-with-clojure
• https://github.com/babashka/pods/blob/47e55fe5e728578ff4dbf7d2a2caf00efea87b1e/src/babashka/pods/impl.clj#L63-L69

[ingydotnet]

Please take a look. It's currently failing because of a hiccup in ys pod loading. Looking into that now.

Fixed the YS bug on the [devel(https://github.com/yaml/yamlscript/tree/devel) branch.

I'll release that fix in YS v0.1.91. Until then you can checkout that branch and run make install-ys.

[ingydotnet]

Cleaned up everything in step06: https://github.com/ingydotnet/instayaml/blob/main/step06/ReadMe.md

I'll add first class Pod loading to the use command for 0.1.91, and adjust in step07.

That's about as far as we can go with instaparse since the original won't work with ys; just the pod version.

Here's the Pod Registry: https://github.com/babashka/pod-registry
Most should work with ys (0.1.91).

Note that we can use instaparse internally in the ys compiler code. (we can use all of clojure there).

We really want to be able to use non-Pod external libs and that will take a bit more work.
Then these libraries should work with ys: https://babashka.org/toolbox/