Experiment: Playing with currying, chaining and underscores

[c42f]

A super hacky, quick implementation of "pipefirst / pipelast" ideas inspired by

https://discourse.julialang.org/t/fixing-the-piping-chaining-issue

Tiny demo:

julia> 1:10 />> filter(isodd) />> map(z->z^2)
5-element Vector{Int64}:
  1
  9
 25
 49
 81

The meaning of />> is roughly "pipelast", so the above expression is like x |> (y->filter(isodd, y)) |> (y->map(z->z^2, y)) though with different lowering. Similarly /> is "pipefirst".

[Note: in the original version of this PR, />> was spelled \> but this is visually confusing.]

Implementation

Parse chains of /> and />> at the rough same precedence as |>, treating them as a currying operator for function calls such that the succeeding function call becomes curried with the only free argument as either the first or last, respectively. Thus, the following

x  />  f(y)  />>  g(z)

is parsed as

(chain x (/> (call f y)) (/>> (call g z)))

and lowered to

chain(x, fixbutfirst(f, y), fixbutlast(g, z))

Also, this can be used without data on the left hand side to create a first class data pipeline:

h = />  f(y)  />>  g(z)

Lowers to

compose_chain(fixbutfirst(f, y), fixbutlast(g, z))

Additionally, add lowering of underscore as strictly tight-binding placeholder syntax. (Super hacky - more forms should be allowed! This is just for experimentation).

---

Note, I don't really expect to ever merge this, given how difficult JuliaLang/julia#24990 has proven 😬

It's just an experiment for people to play with.

You can try this by checking out this branch and using

JuliaSyntax.enable_in_core!(freeze_world_age=false)

(With this setup, you can also edit JuliaSyntax.jl to play with different lowering scenarios and see them almost immediately in the REPL using Revise. (May need to run a command twice for Revise to pick things up.))

[CameronBieganek]

Is this intended to implement the original proposal in that Discourse thread, or a variation on that theme? To me this looks a bit different from the original proposal.

[c42f]

It was meant to implement the flavor of the original proposal, more or less (but I've also added some underscore processing which is clearly unrelated).

Think of this as a playground - we can change things and implement multiple syntaxes at the same time to get some idea of how different things play out.

I found the suggested parsing/precedence of /> in the original proposal confusing so the choice to use pipe precedence is a big departure. Unsure if that's a difference in usability though, or just a technicality.

I probably need to read the original proposal more carefully too 😅

[c42f]

Noting that this syntax can be used without data on the left hand side to create a first class data pipeline, I've upgraded this to allow syntax such as

h = />  f(y)  \>  g(z)

Lowering to

compose_chain(fixbutfirst(f, y), fixbutlast(g, z))

Also I've cleaned up the lowering a bit such that we're creating FixButLast and FixButFirst instances here rather than a pile of anonymous functions.

[CameronBieganek]

Here's my comparison of this PR to the original Discourse proposal:

https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/178

TLDR:
This PR behaves very similarly to having a "front pipe" and a "back pipe". I think it is semantically cleaner to just have a "front pipe" and a "back pipe" than to have "fix every argument but the first" and "fix every argument but the last" with an implicit chain operation thrown in. All that being said, I would still prefer having underscore currying over having front and back pipes. :)

*By "front pipe" and "back pipe", I mean having the following syntax transformations in the parser:

# front pipe:
a \> f(b, c)
# parses as
f(a, b, c)

# back pipe:
c \>> f(a, b)
# parses as
f(a, b, c)

[c42f]

Thanks!

# front pipe:
a \> f(b, c)
# parses as
f(a, b, c)

A quibble: we wouldn't parse it this way because an important feature of the parser is to reflect the surface syntax such that macros can inspect and manipulate it. So the parser should emit a representation of /> distinct from normal function call.

However, we could lower it to f(a,b,c) rather than lowering it to FixButFirst.

[CameronBieganek]

A quibble: we wouldn't parse it this way because an important feature of the parser is to reflect the surface syntax such that macros can inspect and manipulate it. So the parser should emit a representation of /> distinct from normal function call.

Good point, thanks. That makes sense!

[c42f]

So, it turns out that the lowering adopted here can provide fundamentally more efficient data pipelines than normal piping by allowing rewrite rules to be added as methods to chain. I feel this could be a big deal. Maybe even enough of a sweetener to make some variant of this get into Base... we will see (I give it a 20% chance hehe)

The latest commit shows how \> map \> reduce chains can be rewritten into a call to mapreduce with a simple extension of the chain function. I guess this would also apply to functions like groupreduce from SplitApplyCombine.jl. SplitApplyCombine has a whole menagerie of such functions groupsum groupprod etc etc which could just be deleted if this approach works well enough.

[c42f]

more efficient data pipelines than normal piping by allowing rewrite rules to be added as methods to chain

Though rewrite rules are a symbolic simplification step which is not ideal in some ways. Transducers are still fundamentally more composable than a pile of rewrite rules. I wonder whether we could get them in here somehow.

[CameronBieganek]

So, it turns out that the lowering adopted here can provide fundamentally more efficient data pipelines than normal piping by allowing rewrite rules to be added as methods to chain.

One could imagine the underscore currying lowering to a FixArgs type. Then perhaps similar things could be done with FixArgs. However, I'm not sure if it would actually be possible, since |> is just a regular operator/function with no special lowering. So an expression like this,

x |> map(sqrt, _) |> reduce(+, _)

would be equivalent to something like this,

FixArgs(reduce, (1, ), +)(
    FixArgs(map, (1, ), sqrt)(x)
)

so there doesn't seem to be a way to intercept the FixArgs(map, ...) and FixArgs(reduce, ...) and convert them to a mapreduce.

Transducers are still fundamentally more composable than a pile of rewrite rules. I wonder whether we could get them in here somehow.

Yeah, perhaps transducers are the real way to make piping in Base more powerful. (Well, Base piping is already powerful, it's just the syntax with a bunch of anonymous functions that is annoying.) Although it seems that the usage of |> in Transducers.jl is kind of a pun. I've only recently started to look into transducers, but my understanding is that the correct way to combine transducers is composition, since a transducer is a function. Transducers in Clojure are combined with compose. Of course, with a little digging, I see that Transducers.jl does provide both a compose function and an opcompose function. I need to do more studying of transducers, both in Clojure and in Transducers.jl.

Anyhow, maybe the moral of the story is that if you use transducers, then you don't even need piping, you just need function composition. It would be cool if in Julia 2.0 map(f) and filter(f) returned transducers like they do in Clojure.

[adienes]

@c42f I have another proposal I'd like to try, which is essentially to move Chain.jl into real syntax by interpreting chain ... end as @chain begin ... end

I might try to give it a shot in a fork myself because that seems like a cool learning experience, but I hope it's ok if I can ask you questions on the way :)

[c42f]

@adienes go for it if you like, I think it's a good learning experience :-)

For a new syntax to be really useful it's got to be somehow significantly better than the associated macro version (eg, somehow more composable or succinct). Unsure this would be the case for @chain? But still, implementing it could be fun and shouldn't be too hard!

[c42f]

I've updated this to use />> for piping into the last slot - that seems much more visually obvious as pointed out by @CameronBieganek further up the thread. I'm using />> rather than \>> because I think that's slightly easier to type.

I've also merged the changes from #199 in here so we have something a bit more coherent in terms of underscore lowering. And made function calls to the right of /> automatically apply that version of underscore lowering to their arguments. Thus, we can have things like

julia> 1:10 />> filter(isodd) />> map(_^2)
5-element Vector{Int64}:
  1
  9
 25
 49
 81

And

julia> data = [(a=i,b=2i) for i=1:10];

julia> data />> filter(isodd(_.a^2))
5-element Vector{NamedTuple{(:a, :b), Tuple{Int64, Int64}}}:
 (a = 1, b = 2)
 (a = 3, b = 6)
 (a = 5, b = 10)
 (a = 7, b = 14)
 (a = 9, b = 18)

julia> data />> filter(isodd(_.a^2)) />> sum(_.b)
50

[adienes]

is #199 complementary, incompatible, or orthogonal to this proposal? I must admit I am not at all a fan of the aesthetics of piping into a headless |> ->(...) compared to /> or />> (although I know they have different semantics)

[c42f]

It's largely complimentary and I've merged the changes from that PR in here as well to see how they'd play out together.

[ShalokShalom]

@c42f Thanks a lot for this.

As a heads-up: />> has no ligature, and I would recommend to a) pick another combination, or b) open up an issue with Fira Code, and the other ligature based fonts.

Thanks a lot

For inspiration, you could also be interested in DataPipes, if you dont know it yet. :)