Excessive compile time

[ToddAndMargo]

Hi All,

Fedora 33, x64
rakudo-pkg-2020.12-01.x86_64

Something I would really, really like fixed: the excessive compile time not found it Perl 5.

$ ls -al GetUpdates.pl6
-rwxr-xr-x. 1 todd users 356600 Jan 30 01:34 GetUpdates.pl6

$ raku -c --stagestats GetUpdates.pl6
Stage start : 0.000
Stage parse : 17.851
Stage syntaxcheck: Syntax OK

17.851 seconds. Molasses slow

Here is Perl 5

$ ls -al GetUpdates.pl
-rwxrwxrwx. 1 todd users 208173 Apr 26 2019 GetUpdates.pl

This is the Perl 5 code the above Raku was ported from:
$ time perl -c GetUpdates.pl
GetUpdates.pl syntax OK

real 0m0.305s
user 0m0.243s
sys 0m0.029s

0.75 seconds. Lightning fast

Many thanks,
-T

[lizmat]

Most of the compile time is caused by the fact that grammars have not been optimized. To get faster compile times, we need to make grammars faster. That could be a GSoC project!

[JJ]

Who're the grammar experts? Could we contact them and ask for a project?

[lizmat]

Sadly, the only grammar expert left is jnthn, afaik. All others have not been active in Raku for years. :-(

[JJ]

Good thing about this projects, is that they teach the student as well as the co-mentors. So maybe helping @jnthn make a proposal and co-mentor with someone would be a good idea.

[lizmat]

I would not be against that. Maybe we could even convince @pmichaud to become a mentor :-)

[jnthn]

Most of the compile time is caused by the fact that grammars have not been optimized.

[Citation needed]. :-) They're certainly a significant factor. However, my work on the new RakuAST-based compiler frontend leads me to believe there are significant costs aside from parsing that we might also be able to reduce. Of note, anything BEGIN-time currently carries a high cost, and role bodies are effectively BEGIN-time constructs, so we likely pay something like 50% more compile time on code that happens to be within a role. There's also no use of our parallel processing capabilities in the current compiler architecture (which is fair enough; they didn't exist at the time it was designed). My intuition is that the current sink-marking code is also costly, but I don't have any data to back that up.

To get faster compile times, we need to make grammars faster.

Not to mention that faster grammars will have benefits for all code using them too.

That could be a GSoC project!

Perhaps so; I'm not immediately sure how one would define such a project. Any significant wins will probably need significant redesigns of parts of the grammar engine.

Sadly, the only grammar expert left is jnthn, afaik.

Well, expert on the current implementation perhaps. I'd not claim any particular expertise with regards to parsing in general; compilers and runtimes is a broad field, and to date implementation of parsing/lexing isn't an area I've particularly focused on (and yes, I know, I did the parse engine for the Comma IDE, and that was a from-scratch grammar compiler, but even that owes most of its approach to the current NQP engine, and in some ways is notably worse, saved mostly by being able to lex incrementally, and by having the JVM's JIT compiler go to work on it).

There's literature on scannerless parsers (for example, this) that is likely to be highly relevant for the high-level design of a better grammar engine. At a lower level, there's a good amount of work done on how to JIT the regular parts (useful for LTM, which is all about regular prefixes). There will also be some Raku-specific bits to think about in terms of the integration with action methods, and cheating when you know where won't be any to call (I suspect new-disp can help with that). I guess in short, I know enough to know how much background reading and thinking I'd have ahead of me if I were to work on significantly better grammar performance. :-)

Good thing about this projects, is that they teach the student as well as the co-mentors. So maybe helping @jnthn make a proposal and co-mentor with someone would be a good idea.

In principle, I'd really like this; learning new compiler stuff is -Ofun for me. In practice, I currently am working on:

• new-disp - the huge dispatch overhaul
• RakuAST and a new compiler frontend to go with it
• Partial escape analysis (I've a MoarVM branch with a bunch of work in this direction, which I'm keen to bring to completion, but waits on some of the new-disp reworkings of the callstack to land)

And if all of those are shipped and running in production by the end of 2021, I'll feel like I've had a productive Raku year. I struggle to see myself having the the bandwidth to be thinking seriously about the grammar engine by this summer, however (of course, I'm happy to provide hints and input to anybody who does have the time, interest, and epic patience needed to have a go at it).

[MasterDuke17]

There has been discussion in the past with TimToady about "passing fates" (e.g., https://colabti.org/irclogger/irclogger_log/perl6-dev?date=2019-07-02#l79), would that be relevant to this?

[jnthn]

Yes, certainly relevant. The idea there is to save work when one protoregex calls another, by using the result calculated by the outer one. Another potential saving is to not reparse things imperatively when LTM has done the work and we can get away with it.

[MasterDuke17]

This thread might also have some ideas https://perl.markmail.org/message/3wiz26ik6jqsmz27

[coke]

I don't see this mentioned in the thread: Easiest way to get startup speedups is to move your code from your script into a module, and load the module from the script. Then you get the advantages of precompilation.

[ToddAndMargo]

On 8/28/22 12:47, Will Coleda wrote: I don't see this mentioned in the thread: Easiest way to get startup speedups is to move your code from your script into a module, and load the module from the script. Then you get the advantages of precompilation.

That is a workaround. It does not help when troubleshoot and multiple runs are necessary. I am after a fix. There is no reason why Perl 5 should literally compile 100 times faster than Raku (Perl 6)