tokenizer : BPE fixes (#7530) #13835

Job	Run time
Push Docker image to Docker Hub (light, .devops/llama-cli.Dockerfile, linux/amd64,linux/arm64)	10m 40s
Push Docker image to Docker Hub (server, .devops/llama-server.Dockerfile, linux/amd64,linux/arm64)	13m 1s
Push Docker image to Docker Hub (full, .devops/full.Dockerfile, linux/amd64,linux/arm64)	34m 57s
Push Docker image to Docker Hub (light-cuda, .devops/llama-cli-cuda.Dockerfile, linux/amd64)	24m 52s
Push Docker image to Docker Hub (server-cuda, .devops/llama-server-cuda.Dockerfile, linux/amd64)	48m 37s
Push Docker image to Docker Hub (full-cuda, .devops/full-cuda.Dockerfile, linux/amd64)	1h 6m 35s
Push Docker image to Docker Hub (light-rocm, .devops/llama-cli-rocm.Dockerfile, linux/amd64,linux...	34m 10s
Push Docker image to Docker Hub (server-rocm, .devops/llama-server-rocm.Dockerfile, linux/amd64,l...	34m 40s
Push Docker image to Docker Hub (full-rocm, .devops/full-rocm.Dockerfile, linux/amd64,linux/arm64)	38m 28s
Push Docker image to Docker Hub (light-intel, .devops/llama-cli-intel.Dockerfile, linux/amd64)	14m 0s
Push Docker image to Docker Hub (server-intel, .devops/llama-server-intel.Dockerfile, linux/amd64)	14m 28s
	5h 34m 28s

Provide feedback