Fix broken AsyncInferenceClient on [DONE] signal (#2458) #6395

Job	Run time
build-windows (3.8)	5m 1s
build-ubuntu (3.8, Repository only)	1m 42s
build-ubuntu (3.8, Everything else)	2m 38s
build-ubuntu (3.8, torch)	1m 4s
build-ubuntu (3.11, Repository only)	2m 4s
build-ubuntu (3.11, Everything else)	4m 16s
build-ubuntu (3.11, torch)	1m 9s
build-ubuntu (3.11, lfs)	36s
build-ubuntu (3.8, fastai)	1m 28s
build-ubuntu (3.10, fastai)	1m 11s
build-ubuntu (3.8, tensorflow)	1m 0s
build-ubuntu (3.10, tensorflow)	53s
	23m 2s

Provide feedback