[Streamer] Fix UTF-8 handling in streamer (mlc-ai#2978)

This PR fixes a bug in the streamer handling for UTF-8 characters.
Prior to this PR, the streamer has an assumption that a replacement
character (`�`) always correspond to an entire token. However, for
the Qwen2 model tokenizer, some token can be ` �` if decoded directly,
which breaks the assumption and leads to incorrect result generated
by the streamer.

This PR fixes this issue with a safer behavior that does not rely
on such an assumption.

cpp/tokenizers/streamer.cc

@@ -64,7 +64,8 @@ std::string TextStreamerObj::Put(const std::vector<int32_t>& delta_tokens) {
                  0) {
         new_pending_tokens.push_back(pending_tokens_.back());
         pending_tokens_.pop_back();
-        validated_str = validated_str.substr(0, validated_str.length() - 3);
+        all_tokens.pop_back();
+        validated_str = tokenizer_->Decode(all_tokens).substr(prefix_str.length());
       }
     } else {
       // Case 2. prefix_str is not a prefix of `full_str`.