Validator slot falling behind when enabled nft_ingester.

[muhitrhn]

I've tried with solana v1.16.15 and v1.16.16, the validator gains slot slowly when the ingester is not active. But as soon as I turn the ingester on, validator starts falling behind on slots by a lot every minute. I run ingester, postgres, redis in the same server so I guess that can be the reason but shouldn't it still work properly with all of these on the same server?

Device Configuration

CPU AMD EPYC 7443P @ 4.04GHz (24 cores)
RAM - 512 GB
Disk - 2 X 4 TB NVMe
NICs - 10 Gbit/s + 1 Gbit/s
OS - Ubuntu 20.04

[austbot]

how are you deploying this, does your validator plugin point to a redis instance with streams on?

[muhitrhn]

Yes the validator plugin points to the redis instance running on the same server as the validator with plugin.

[austbot]

What is the latency of your redis instance, and if this is all on the same server then likley its out of resources, unless its localnet. Devnet,Testnet and Mainnet require more resources

[muhitrhn]

Not sure about the redis instance latency but later I used 8 Cores @ 3.4 GHz, 64 GB RAM, 2 x 500 GB SSD, 10 Gbps for the redis + db and 24 Cores @ 2.85 GHz, 256 GB RAM, 2 x 1.9 TB NVME, 10 Gbps for the validator + plugin. But the issue was still the same, couldn't catchup.

[taoyouxian]

I've tried with solana v1.16.15 and v1.16.16, the validator gains slot slowly when the ingester is not active. But as soon as I turn the ingester on, validator starts falling behind on slots by a lot every minute. I run ingester, postgres, redis in the same server so I guess that can be the reason but shouldn't it still work properly with all of these on the same server?

Device Configuration

CPU AMD EPYC 7443P @ 4.04GHz (24 cores) RAM - 512 GB Disk - 2 X 4 TB NVMe NICs - 10 Gbit/s + 1 Gbit/s OS - Ubuntu 20.04

Have you finished the problem, how do you run Redis with the correct configs? I developed locally but failed.