chore: Try replace callstack with a linked list

[jfecher]

Description

Problem*

Resolves

Summary*

Test to see memory usage change from replacing the callstack with a (shared) linked list.

Additional Context

Surprisingly, I had trouble finding an existing crate which provides a shared linked list. I ended up writing my own. (std::collections::LinkedList boxes each item so it isn't shared). Callstacks come up several times in parallel iterators unfortunately so I had to use Arc over Rc.

Still need to reverse the display order of callstacks now that they're stored in reverse order due to being lists.

Documentation*

Check one:

• No documentation needed.
• Documentation included in this PR.
• [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

• I have tested the changes locally.
• I have formatted the changes with Prettier and/or cargo fmt on default settings.

[github-actions]

Compilation Sample

Program	Compilation Time	%
sha256_regression	0m1.423s	-4%
regression_4709	0m1.540s	-4%
ram_blowup_regression	0m16.817s	-3%
private-kernel-tail	0m1.317s	8%
private-kernel-reset	0m9.096s	-5%
private-kernel-inner	0m2.356s	-12%
parity-root	0m0.940s	6%
noir-contracts	2m36.021s	-8%

[github-actions]

Peak Memory Sample

Program	Peak Memory	%
keccak256	78.17M	-4%
workspace	121.79M	-1%
regression_4709	294.72M	-11%
ram_blowup_regression	2.44G	0%
private-kernel-tail	208.76M	-10%
private-kernel-reset	861.44M	100%
private-kernel-inner	307.68M	-9%
parity-root	174.42M	-13%

[jfecher]

regression_4709 300.69M -10%

Maybe not big enough to warrant merging? Wish we could avoid the Arc, otherwise I'd be more fine with merging.

Also ram_blowup_regression is 2.55G on other PRs but is showing as 0% changed here. Should be roughly -4%

[TomAFrench]

Let's keep this PR open until #6750 is landed to see what the effects are on the protocol circuits.

[jfecher]

Comparing this to #6753, the peak memory usage reduction is much closer to optimal, -4% and -10% compared to a maximum of -5% and -15% respectively for those tests.

[TomAFrench]

The report isn't showing this correctly but it's cutting the memory usage of private-kernel-reset by ~400MB.

[TomAFrench]

Approving but not going to merge as it was left in draft.

[TomAFrench]

Still need to reverse the display order of callstacks now that they're stored in reverse order due to being lists.

Ah should have read the OP more closely.

[jfecher]

Reversed iteration direction of the call stacks so this should be ready for review / merge now.
Going to see how much this change impacts runtime performance.

[TomAFrench]

LGTM

[aakoshh]

Both push_back and pop_front operate on the head; I understand what you meant by push_back actually being a "push_front", but wouldn't it be better to name both of these head operations to be the same? I thought pop_front will go to the opposite end from push_back in an O(n) operation. Why not just push and pop?

[jfecher]

Yeah, that's my mistake. They were originally both *_back to make it more easy for me to migrate existing code, then I changed pop_front to be the front to be more accurate but thought renaming the public push_back to front as well could be confusing to CallStack users who want to add something to the conceptual "back" of the structure, unaware that the structure is essentially reversed internally.

I'd be for renaming it back to pop_back.

[aakoshh]

My all time favourite introduction to Rust has some hilarious discussion about linked lists that might explain why there aren't many implementations around: https://rust-unofficial.github.io/too-many-lists/index.html :)