A macro to generating AST with python grammar

[youknowone]

Inspired by https://docs.rs/pmutil/latest/pmutil/macro.smart_quote.html

e.g.

ExprKind::Tuple(ast::ExprTuple {
    elts: names
        .iter()
        .map(|&name| {
            create_expr(ExprKind::Constant(
                ast::ExprConstant {
                    value: Constant::Str(name.to_string()),
                    kind: None,
                },
            ))
        })
        .collect(),
    ctx: ExprContext::Load,
}))

can be rewritten to:

&python_ast! {
    Vars { names },
    "(*names)"
}

The grammar will be limited to legal python codes to leverage python parser.

[MichaReiser]

I'm struggling to understand what's happening in

&python_ast! {
    Vars { names },
    "(*names)"
}

even with the example. I'm also vary of macros because most Rust tooling breaks down (formatting, autocompletion) and can be difficult to understand if you haven't used the macro before.

Have you considered alternative mutation APIs? E.g. Rome has two APIs:

• SyntaxFactory::new_if_stmt(condition, body) for the creation of nodes
• Node::with_X(x) -> Node methods for returning a new copy with X set to a specific value.

This allows, in combination, to write SyntaxFactory::new_if_stmt(condition, body).with_orelse(orelse).

[youknowone]

It turns a python snippet to python ast with given values. It has very different user experience to factories.

Starting from simpler example

&python_ast! {
    "def new_function(a, b, c): pass"
}

will be parsed to

FunctionDef(
      name='new_function',
      args=arguments(
        posonlyargs=[],
        args=[
          arg(arg='a'),
          arg(arg='b'),
          arg(arg='c')],
        kwonlyargs=[],
        kw_defaults=[],
        defaults=[]),
      body=[
        Pass()],
      decorator_list=[])

The factory will be somewhere between it, but it will be more close to the latter.

Simple value example

let new_function_name = Identifier::new("overriding_name");
&python_ast! {
    Vars { new_function_name },
    "def new_function_name(a, b, c): pass"
}

will turned into

FunctionDef(
      name='name',
      args=arguments(
        posonlyargs=[],
        args=[
          arg(arg='a'),
          arg(arg='b'),
          arg(arg='c')],
        kwonlyargs=[],
        kw_defaults=[],
        defaults=[]),
      body=[
        Pass()],
      decorator_list=[])

By looking in every node and checking its identifier name new_function_name is same as the given Vars list.

More complex one, similar to the original example.

let args: Vec<_> = ["a", "b", "c"].iter().map(|name| Identifier::new(name));
let values = HashMap::new();
values.insert("a", 10);
values.insert("b", 20);
values.insert("c", 30);

&python_ast! {
    Vars { args, values },
    r#"
    def new_function(*args):
        return { **values }
    #"
}

will be originally parsed to:

    FunctionDef(
      name='new_function',
      args=arguments(
        posonlyargs=[],
        args=[],
        vararg=arg(arg='args'),
        kwonlyargs=[],
        kw_defaults=[],
        defaults=[]),
      body=[
        Return(
          value=Dict(
            keys=[
              None],
            values=[
              Name(id='values', ctx=Load())]))],
      decorator_list=[])],

and then folded to

    FunctionDef(
      name='new_function',
      args=arguments(
        posonlyargs=[
          arg(arg='a'),
          arg(arg='b'),
          arg(arg='c')],
        args=[],
        kwonlyargs=[],
        kw_defaults=[],
        defaults=[]),
      body=[
        Return(
          value=Dict(
            keys=[
              Constant(value='a'),
              Constant(value='b'),
              Constant(value='c')],
            values=[
              Constant(value=10),
              Constant(value=10),
              Constant(value=10)]))],
      decorator_list=[])],

More with comprehension

let args: Vec<_> = ["a", "b", "c"].iter().map(str::to_owned).collect();

&python_ast! {
    Vars { args },
    r#"
    def new_function(*args):
        return [f"prefixed_{arg}" for arg in args]
    #"
}

originally turns into

    FunctionDef(
      name='new_function',
      args=arguments(
        posonlyargs=[],
        args=[],
        vararg=arg(arg='args'),
        kwonlyargs=[],
        kw_defaults=[],
        defaults=[]),
      body=[
        Return(
          value=ListComp(
            elt=JoinedStr(
              values=[
                Constant(value='prefixed_'),
                FormattedValue(
                  value=Name(id='arg', ctx=Load()),
                  conversion=-1)]),
            generators=[
              comprehension(
                target=Name(id='arg', ctx=Store()),
                iter=Name(id='args', ctx=Load()),
                ifs=[],
                is_async=0)]))],
      decorator_list=[])],

and then folded to

    FunctionDef(
      name='new_function',
      args=arguments(
        posonlyargs=[
          arg(arg='a'),
          arg(arg='b'),
          arg(arg='c')],
        args=[],
        kwonlyargs=[],
        kw_defaults=[],
        defaults=[]),
      body=[
        Return(
          value=List(
            elts=[
              Constant(value='prefixed_a'),
              Constant(value='prefixed_b'),
              Constant(value='prefixed_c')],
            ctx=Load()))],
      decorator_list=[])],

[youknowone]

I'm also vary of macros because most Rust tooling breaks down (formatting, autocompletion) and can be difficult to understand if you haven't used the macro before.

Unlike other macros, the linked smart_quote! doesn't break tools due to the similar restriction - it must be always a valid rust code.

One good thing about it is python_ast! doesn't include any rust code inside. It only requires python code and binding list of variables.