Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow content compression with gzip #137

Closed
fvisticot opened this issue Jan 12, 2019 · 7 comments
Closed

Allow content compression with gzip #137

fvisticot opened this issue Jan 12, 2019 · 7 comments

Comments

@fvisticot
Copy link

Is it possible to compress POST request with gzip ?

@wendux
Copy link
Contributor

wendux commented Feb 12, 2019

Not yet!

@wendux wendux closed this as completed Feb 12, 2019
@fvisticot
Copy link
Author

Any news regarding this feature ?
It would be very interesting for every mobile developer to improve application speed !

edwardez added a commit to edwardez/BangumiN that referenced this issue Apr 17, 2019
Previously bgm.tv is used because it's behind cdn and, theoretically, can provide a better user experience. However it looks like cloudflare is blocking non-standard way to access html files. On iOS munin is able to download and parse files, however on Android(Pixel, Android 9) munin is blocked by cloudflare(error: https://support.cloudflare.com/hc/en-us/articles/200171806-Error-1010-The-owner-of-this-website-has-banned-your-access-based-on-your-browser-s-signature).

Changing user agent/adding browser-like headers don't work. Cloudflare must be using some super advanced machine learning algorithms to identify browser/non-browser...

Workarounds I can think of
1. Use bangumi.tv to authorize user and parsing html(bangumi.tv is not officially documented as the oauth server but it works. Well, we are now officially entering the uncharted territory)
2. Let user directly enter emails/password in our app, and munin will handle login for them(this should always work but it might freaks users out since we are asking account/password directly)

We opt for 1 for now.

Side note: I suspect this is related to DIO, some preliminary experiments showed dart plain http client is not blocked by cf

Side note2: it seems like DIO currently doesn't support gzip, cfug/dio#137 while plain http client does. Maybe we should stick with old-school dart http client...
@fzyzcjy
Copy link

fzyzcjy commented Mar 18, 2021

Any news after 2 years? Thanks! @wendux

@omidraha
Copy link

Is it possible now?
How about receiving json response data as gzip?
#307

@kmtong
Copy link

kmtong commented Mar 10, 2022

It can be simply implemented as requestEncoder:

    var dio = Dio(BaseOptions(
      requestEncoder: gzipEncoder,
    ));

gzipEncoder is a function for content encoding.

// gzip request
List<int> gzipEncoder(String request, RequestOptions options) {
  options.headers.putIfAbsent("Content-Encoding", () => "gzip");
  return gzip.encode(utf8.encode(request));
}

@hurelhuyag
Copy link

I tried this solution. But this throws FormatException. bad data.

This is the code I tried.

final baseOptions = BaseOptions(
      headers: {
        'accept-encoding': 'gzip',
        'content-encoding': 'gzip',
      },
      requestEncoder: (request, options) {
        final encoding = options.headers["content-encoding"];
        switch (encoding) {
          case null:
            return utf8.encode(request);
          case "gzip":
            return gzip.encode(utf8.encode(request));
          default:
            throw Exception("Unsupported encoding /$encoding/ used in request body");
        }
      },
      responseDecoder: (responseBytes, options, responseBody) {
        final encoding = options.headers["content-encoding"];
        switch (encoding) {
          case null:
            return utf8.decode(responseBytes);
          case "gzip":
            return utf8.decode(gzip.decode(responseBytes));
          default:
            throw Exception("unsupported encoding /$encoding/ used in response body");
        }
      },
    );
    final dio = Dio(baseOptions);

@Sangamesh-AT
Copy link

This worked for me! And I'm guessing that Dio automatically handles the decompression when the response is received in gzip format!

@lazySingleton
Dio dio(@Named('BaseUrl') String url) {
  final gZipCodec = GZipCodec();
  final dio = Dio(
    BaseOptions(
      baseUrl: url,
      contentType: Headers.jsonContentType,
      connectTimeout: const Duration(seconds: 120),
      receiveTimeout: const Duration(seconds: 120),
      sendTimeout: const Duration(seconds: 120),
    ),
  );

  dio.interceptors.add(InterceptorsWrapper(
    onRequest: (options, handler) {
      final jsonData = jsonEncode(options.data);
      final compressedData = gZipCodec.encode(utf8.encode(jsonData));
      options.data = compressedData;
      options.headers['Content-Encoding'] = 'gzip';
      options.headers['Accept-Encoding'] = 'gzip';
      options.headers['Content-Type'] = 'application/json';
      return handler.next(options);
    },
    onResponse: (response, handler) {
      return handler.next(response);
    },
  ));

  return dio;
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants