Header could not send some mystery charactor

[BenderBlog]

Package

dio

Version

5.3.2

Operating-System

iOS

Output of flutter doctor -v

superbart@superbart-macmini ~ % flutter doctor -v
[✓] Flutter (Channel stable, 3.13.2, on macOS 13.5.1 22G90 darwin-arm64, locale
    zh-Hans-CN)
    • Flutter version 3.13.2 on channel stable at
      /Users/superbart/Downloads/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ff5b5b5fa6 (2 weeks ago), 2023-08-24 08:12:28 -0500
    • Engine revision b20183e040
    • Dart version 3.1.0
    • DevTools version 2.25.0
    • Pub download mirror https://pub.flutter-io.cn
    • Flutter download mirror https://storage.flutter-io.cn

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/superbart/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /Applications/Android
      Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      17.0.6+0-17.0.6b829.9-10027231)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E300c
    • CocoaPods version 1.12.1

[✗] Chrome - develop for the web (Cannot find Chrome executable at
    /Applications/Google Chrome.app/Contents/MacOS/Google Chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.

[✓] Android Studio (version 2022.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app
    • Java version OpenJDK Runtime Environment (build
      17.0.6+0-17.0.6b829.9-10027231)

[✓] VS Code (version 1.81.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.72.0

[✓] Connected device (2 available)
    • iPhone 14 (mobile) • A24E37B7-ED9D-4CBE-8FA7-EC0843B7DC44 • ios          •
      com.apple.CoreSimulator.SimRuntime.iOS-16-4 (simulator)
    • macOS (desktop)    • macos                                • darwin-arm64 •
      macOS 13.5.1 22G90 darwin-arm64

[!] Network resources             
    ✗ A network error occurred while checking "https://github.com/": Operation
      timed out

! Doctor found issues in 2 categories.

Dart Version

3.13.2

Steps to Reproduce

1. Put some unusual charactor in the header.
2. Post the request.

example code

/// This is the cookies from previous login request.
/// Notice that the server is pretty old (2008 or older), it encoded data in GBK.
/// And cookies contains **Chinese charactors**, pretty old school :P
/// Since dart will misunderstand the GBK encoded charactors, it will produce some 
/// unusual chars, in my case, ä.
List<String> cookie = loginResponse.headers[HttpHeaders.setCookieHeader] ?? [];

/// Now I want to post with the cookie to fetch the data.
dio.get("http://wlsy.xidian.edu.cn/${loginResponse.headers["Location"]![0]}",
      options: Options(
      headers: {
          HttpHeaders.cookieHeader: loginResponse.headers[HttpHeaders.setCookieHeader],
      },
   ),
).then((value) => print(value.data));

cookie example

especially

PhyEws_StuName=任�涵; path=/, PhyEws_StuType=1; path=/, .

这里再用中文解释一下我的问题：
这个服务器使用 GBK 编码，其返回的 Cookie 包含 GBK 编码的中文字符。Dart 使用 UTF-16 解码会出现不合适的字符，然后当我想用这个 Cookie POST 请求的时候，其报错会引出“非法字符错误”。
顺便，我直接用中文发过去，照样报错。
根据 Cookie 规范，其只能是 ASCII 字符，但对于老服务器，有时候不会是这样。所以请求各位能考虑一下适配这样不太合适的需求吗，让 Headers 放行这样不正常的字符。

Expected Result

Well, it sends

Actual Result

[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: DioException [unknown]: null
Error: FormatException: Invalid HTTP header field value: "PhyEws_StuName=任�涵; path=/" (at character 16)

PhyEws_StuName=任�涵; path=/
^
#0 DioMixin.fetch. (package:dio/src/dio_mixin.dart:507:7)
#1 _RootZone.runUnary (dart:async/zone.dart:1661:54)
#2 _FutureListener.handleError (dart:async/future_impl.dart:174:22)
#3 Future._propagateToListeners.handleError (dart:async/future_impl.dart:852:47)
#4 Future._propagateToListeners (dart:async/future_impl.dart:873:13)
#5 Future._completeError (dart:async/future_impl.dart:649:5)
#6 _SyncCompleter._completeError (dart:async/future_impl.dart:60:12)
#7 _Completer.completeError (dart:async/future_impl.dart:26:5)
#8 Future.any.onError (dart:async/future.dart:620:45)
#9 _RootZone.runBinary (dart:async/zone.dart:1666:54)
#10 _FutureListener.handleError (dart:async/<…>

[DouglasValerio]

@BenderBlog
This seems to be the expected behavior on the HTTP protocol: LINK

I had a similar issue a couple of days ago, and ended up having to place a ReGex guard on my headers to keep it running.
I'm gathering that just removing/replacing/mapping the chars is not an option in your case, right?

[AlexV525]

I'm not sure if we can do something to help the circumstances. Header fields will eventually parsed by the parser embedded in the Dart, there seems to be no way to bypass the routine at this moment.

Could you evaluate the possibility of sending a raw request based on a HttpRequest with your headers?

[BenderBlog]

@BenderBlog This seems to be the expected behavior on the HTTP protocol: LINK

I had a similar issue a couple of days ago, and ended up having to place a ReGex guard on my headers to keep it running. I'm gathering that just removing/replacing/mapping the chars is not an option in your case, right?

You mean changing the dart's implementation on headers, like this? dart-lang/sdk#42902
Or could you explain more about ReGex, did you changed the dart sdk code related to http?
And yes, it is expected behaviour.

[DouglasValerio]

@BenderBlog This seems to be the expected behavior on the HTTP protocol: LINK
I had a similar issue a couple of days ago, and ended up having to place a ReGex guard on my headers to keep it running. I'm gathering that just removing/replacing/mapping the chars is not an option in your case, right?

You mean changing the dart's implementation on headers, like this? dart-lang/sdk#42902 Or could you explain more about ReGex, did you changed the dart sdk code related to http? And yes, it is expected behaviour.

We put the headers object through a ReGex filter, before adding it to the request.
Something along this lines:

final cleanHeaders = oldHeaders.map((key, value) => value is String ? MapEntry(key, _removeNonAsciiChars(value)) : MapEntry(key, value));

The implementation of _removeNonAsciiChars is very particular to our set of business rules, once we can get rid of some non-ascii chars, but still need to have some form of reconstructing the 'lost' info.

For instance, we register the public name of the wifi network from which the user is making the request:

{
'xWifiName':'Wifi do Paço'
}

The names tend to contain a lot of non-ascii chars, since the app is used by Portuguese-speaking customers.
In this case, after we put the map above through the filter, we get this:

{
'xWifiName':'Wifi do Pa_231_o'
}

Where 231 is the ASCII code for the ç char
This won´t cause any issues during the request but will keep our ability to reconstruct the info if ever needed.

At last, my opinion on this matter is: This kind of filter/treatment should not be part of dart-lang SDK or even the Dio Package.
And why is that?
Any char-specific treatment will incur some form of information mutation on the sent header, which of course will carry the opinion of those who come up with the implementation, which will lead to some other type of unforseable issues down the road.
Maybe docs should be more explicit on stating that chars not compliant with RULE will cause a FormatException, but ultimately, the users of the Dio Package are the ones responsible for cleaning the headers before sending them on a request.

[AlexV525]

Based on #1959 (comment) from @DouglasValerio, you can try to submit a proposal to the Dart team, and see if they can handle this better. Closing as there is nothing Dio can do at this moment.

[BenderBlog]

OK I have some solution right here.

You can encode the data with UrlEncoder, like Uri.encodeFull. GBK and UTF transfer is another story...

[burnermanx]

Retrofit 2.10 added support for non-ascii characters on headers because is supported or even required for some services.

https://github.com/square/retrofit/blob/trunk/CHANGELOG.md#2100---2024-03-18

If you need to send device model on headers for some pruporse, it's not possible to send "iPhone XR", because Apple use a small R on model name witch isn't a ASCII character and needs a special treatment just for this case.

It tooks months to figure out what was happening, because this problem was affecting only iPhone XR devices.