How to avoid large batch been killed

[lxrswdd]

I have 5176 images with 5176 prompts

(dynamicrafter) xiangrui@ward-lambda01:~/video_generators/DynamiCrafter$ sh scripts/run_1024.sh 1024
@DynamiCrafter cond-Inference: 2024-09-18-15-48-40
Global seed set to 666
AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.
>>> model checkpoint loaded.
Inference with 16 frames
Killed
(dynamicrafter) xiangrui@ward-lambda01:~/video_generators/DynamiCrafter$ 

The model worked well with 8 samples but once I replaced the default prompts with my 5000+ prompts. The task is killed.

[Doubiiu]

Hi. It may be due to the memory leakage issue, as we load all input data into memory at the same time. I think a solution is to use a data loader to fetch the input data by batch.