[BUG] Performance degradation when opening a project in Phoenix as the number of traces grows

[axiomofjoy]

Loading the traces list for a project (/projects/{id}) becomes increasingly slow relative to the number of traces, though the projects (plural) page and individual trace views remain fast. The issue persists despite traces being paginated and deleted weekly. Some traces contain a large number of spans (>100) and long-context spans, which may contribute to the slowdown.

User is self-hosting Phoenix in Kubernetes. The database is Postgres in Amazon Aurora with db.r7g.xlarge instances. Currently on Phoenix 7.9.0.

[dosubot]

I found 1 similar issue that might be helpful:

• Trace lazy loading [ENHANCEMENT] - This issue was addressed by implementing UI lazy loading of spans, which may help improve performance when handling a large number of traces.

[axiomofjoy]

Related to #4006

[BenMcH]

This has been observed on projects that I've used phoenix as well. We've run phoenix in fargate instances on AWS and in container apps on Azure, both backed by hosted/managed Postgres instances in their respective clouds.