test: add vlm to token in & out example

[mickqian]

Motivation

1. Support vlm in skip_tokenizer_init, ref [Feature] Support token-in-token-out for Vision LM #3871
2. Add docs for offline token in & token out, ref [Feature] Add docs for Offline Engine token-in token-out #2968

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhaochenyang20]

(sglang) chayenne@lmsys:~/token-in-token-out/sglang/examples/runtime/engine$ python token_in_token_out_vlm.py 
INFO 03-03 05:48:20 __init__.py:190] Automatically detected platform cuda.
parser
usage: token_in_token_out_vlm.py [-h] --model-path MODEL_PATH [--tokenizer-path TOKENIZER_PATH] [--host HOST] [--port PORT] [--tokenizer-mode {auto,slow}]
                                 [--skip-tokenizer-init] [--load-format {auto,pt,safetensors,npcache,dummy,gguf,bitsandbytes,layered}] [--trust-remote-code]
                                 [--dtype {auto,half,float16,bfloat16,float,float32}] [--kv-cache-dtype {auto,fp8_e5m2,fp8_e4m3}]
                                 [--quantization-param-path QUANTIZATION_PARAM_PATH]
                                 [--quantization {awq,fp8,gptq,marlin,gptq_marlin,awq_marlin,bitsandbytes,gguf,modelopt,w8a8_int8}]
                                 [--context-length CONTEXT_LENGTH] [--device {cuda,xpu,hpu,cpu}] [--served-model-name SERVED_MODEL_NAME]
                                 [--chat-template CHAT_TEMPLATE] [--is-embedding] [--revision REVISION] [--mem-fraction-static MEM_FRACTION_STATIC]
                                 [--max-running-requests MAX_RUNNING_REQUESTS] [--max-total-tokens MAX_TOTAL_TOKENS]
                                 [--chunked-prefill-size CHUNKED_PREFILL_SIZE] [--max-prefill-tokens MAX_PREFILL_TOKENS]
                                 [--schedule-policy {lpm,random,fcfs,dfs-weight}] [--schedule-conservativeness SCHEDULE_CONSERVATIVENESS]
                                 [--cpu-offload-gb CPU_OFFLOAD_GB] [--prefill-only-one-req PREFILL_ONLY_ONE_REQ]
                                 [--tensor-parallel-size TENSOR_PARALLEL_SIZE] [--stream-interval STREAM_INTERVAL] [--stream-output]
                                 [--random-seed RANDOM_SEED] [--constrained-json-whitespace-pattern CONSTRAINED_JSON_WHITESPACE_PATTERN]
                                 [--watchdog-timeout WATCHDOG_TIMEOUT] [--dist-timeout DIST_TIMEOUT] [--download-dir DOWNLOAD_DIR]
                                 [--base-gpu-id BASE_GPU_ID] [--gpu-id-step GPU_ID_STEP] [--log-level LOG_LEVEL] [--log-level-http LOG_LEVEL_HTTP]
                                 [--log-requests] [--show-time-cost] [--enable-metrics] [--decode-log-interval DECODE_LOG_INTERVAL] [--api-key API_KEY]
                                 [--file-storage-pth FILE_STORAGE_PTH] [--enable-cache-report] [--data-parallel-size DATA_PARALLEL_SIZE]
                                 [--load-balance-method {round_robin,shortest_queue}] [--expert-parallel-size EXPERT_PARALLEL_SIZE]
                                 [--dist-init-addr DIST_INIT_ADDR] [--nnodes NNODES] [--node-rank NODE_RANK]
                                 [--json-model-override-args JSON_MODEL_OVERRIDE_ARGS] [--lora-paths [LORA_PATHS ...]]
                                 [--max-loras-per-batch MAX_LORAS_PER_BATCH] [--lora-backend LORA_BACKEND]
                                 [--attention-backend {flashinfer,triton,torch_native}] [--sampling-backend {flashinfer,pytorch}]
                                 [--grammar-backend {xgrammar,outlines,llguidance}] [--enable-flashinfer-mla] [--flashinfer-mla-disable-ragged]
                                 [--speculative-algorithm {EAGLE,NEXTN}] [--speculative-draft-model-path SPECULATIVE_DRAFT_MODEL_PATH]
                                 [--speculative-num-steps SPECULATIVE_NUM_STEPS] [--speculative-eagle-topk {1,2,4,8}]
                                 [--speculative-num-draft-tokens SPECULATIVE_NUM_DRAFT_TOKENS] [--speculative-token-map SPECULATIVE_TOKEN_MAP]
                                 [--enable-double-sparsity] [--ds-channel-config-path DS_CHANNEL_CONFIG_PATH] [--ds-heavy-channel-num DS_HEAVY_CHANNEL_NUM]
                                 [--ds-heavy-token-num DS_HEAVY_TOKEN_NUM] [--ds-heavy-channel-type DS_HEAVY_CHANNEL_TYPE]
                                 [--ds-sparse-decode-threshold DS_SPARSE_DECODE_THRESHOLD] [--disable-radix-cache] [--disable-jump-forward]
                                 [--disable-cuda-graph] [--disable-cuda-graph-padding] [--enable-nccl-nvls] [--disable-outlines-disk-cache]
                                 [--disable-custom-all-reduce] [--disable-mla] [--disable-overlap-schedule] [--enable-mixed-chunk] [--enable-dp-attention]
                                 [--enable-ep-moe] [--enable-torch-compile] [--torch-compile-max-bs TORCH_COMPILE_MAX_BS]
                                 [--cuda-graph-max-bs CUDA_GRAPH_MAX_BS] [--cuda-graph-bs CUDA_GRAPH_BS [CUDA_GRAPH_BS ...]]
                                 [--torchao-config TORCHAO_CONFIG] [--enable-nan-detection] [--enable-p2p-check] [--triton-attention-reduce-in-fp32]
                                 [--triton-attention-num-kv-splits TRITON_ATTENTION_NUM_KV_SPLITS]
                                 [--num-continuous-decode-steps NUM_CONTINUOUS_DECODE_STEPS] [--delete-ckpt-after-loading] [--enable-memory-saver]
                                 [--allow-auto-truncate] [--enable-custom-logit-processor] [--tool-call-parser {qwen25,mistral,llama3}]
                                 [--enable-hierarchical-cache]
token_in_token_out_vlm.py: error: the following arguments are required: --model-path

Hey Mick, could you fix the bugs here?

[mickqian]

@zhaochenyang20 fixed

[zhaochenyang20]

@mickqian Fix the conflicts and rebase plz

[mickqian]

After rebasing, the stream test of skip_tokenizer_init needs some additional work, while the none-stream version is ready.

[zhaochenyang20]

Hey mick. Could you try to simplify your examples:

https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/token_in_token_out_llm.py

Following this and name it token_in_token_out_vlm.py?

[zhaochenyang20]

@mickqian Thanks！