Detokenizer fixes (#8039) #14194

Job	Run time
Push Docker image to Docker Hub (light, .devops/llama-cli.Dockerfile, linux/amd64,linux/arm64)	11m 59s
Push Docker image to Docker Hub (server, .devops/llama-server.Dockerfile, linux/amd64,linux/arm64)	14m 8s
Push Docker image to Docker Hub (full, .devops/full.Dockerfile, linux/amd64,linux/arm64)	50m 45s
Push Docker image to Docker Hub (light-cuda, .devops/llama-cli-cuda.Dockerfile, linux/amd64)	42m 33s
Push Docker image to Docker Hub (server-cuda, .devops/llama-server-cuda.Dockerfile, linux/amd64)	42m 44s
Push Docker image to Docker Hub (full-cuda, .devops/full-cuda.Dockerfile, linux/amd64)	1h 56m 41s
Push Docker image to Docker Hub (light-rocm, .devops/llama-cli-rocm.Dockerfile, linux/amd64,linux...	40m 11s
Push Docker image to Docker Hub (server-rocm, .devops/llama-server-rocm.Dockerfile, linux/amd64,l...	40m 35s
Push Docker image to Docker Hub (light-intel, .devops/llama-cli-intel.Dockerfile, linux/amd64)	14m 17s
Push Docker image to Docker Hub (server-intel, .devops/llama-server-intel.Dockerfile, linux/amd64)	15m 14s
	6h 29m 7s

Provide feedback