Keep authorization headers on redirect

[dmvianna]

Many SaaS vendors provide REST APIs behind URLs that redirect traffic. Dayforce and MYOB are examples. This presents a problem for Airbyte users: Airbyte relies on the Python library requests as a HTTP handler, and requests, on receiving a HTTP 302 redirect code will issue another HTTP request to the new target URL without the Authentication headers. This will invariably result in a HTTP 401 Unauthorized response in these platforms.

The requests core developers chose this behaviour in response to a security vulnerability, which is understandable from their point of view. However this prevents users of the above vendors from using the Airbyte platform to communicate with them.

I would like to request Airbyte to add a feature where an user could optionally keep the authorization headers on redirect. This is the approach both curl and postman have adopted.

Below some runnable code illustrating the problem on the public Dayforce platform:

dayforce_redirect.sh

access_token=$(
  curl --location --request POST \
       "https://dfidtst.np.dayforcehcm.com/connect/token" \
       --data "grant_type=password&companyId=ddn&username=DFWSTest&password=DFWSTest&client_id=Dayforce.HCMAnywhere.Client" \
    --header "Content-Type: application/x-www-form-urlencoded" \
      | jq --raw-output .access_token
            )

# replace `--location` with `--location-trusted` below.
curl \
  --location \
  --request GET \
  "https://test.dayforcehcm.com/api/ddn/v1/ClientMetadata" \
  --header "Authorization: Bearer ${access_token}" \
  --include

This will demonstrate a HTTP 302 Redirect follow by HTTP 401 Unauthorized, with no payload. That's what Airbyte currently does. Now, if we replace --location in the second request with --location-trusted,

 --location-trusted
     (HTTP) Like -L, --location, but  will allow sending the name
     + password to all hosts that  the site may redirect to. This
     may  or may  not introduce  a  security breach  if the  site
     redirects  you  to  a  site  to which  you  will  send  your
     authentication info (which is plaintext  in the case of HTTP
     Basic authentication).

then we should get a HTTP 302 Redirect followed by HTTP 200 OK with some JSON payload.

{"ServiceVersion":"65.1.20.03713","ServiceUri":"https://ustest232-services.dayforcehcm.com/api"}

How to fix that in requests?
dayforce_redirect.py

#!/usr/bin/env python

import requests
from requests import Request, Session
import json

auth_url = "https://dfidtst.np.dayforcehcm.com/connect/token"
api_base = "test"
api_redirect = "ustest232-services"

api_url = f"https://{api_base}.dayforcehcm.com/api/ddn/v1/ClientMetadata"

class DayforceSession(requests.Session):

    AUTH_DOMAINS = ['dayforcehcm.com']

    def __init__(self):
        """Create Dayforce Session that preserves headers when redirecting"""
        super(DayforceSession, self).__init__()  # Python 2 and 3

    def rebuild_auth(self, prepared_request, response):
        """Keep headers upon redirect as long as we are on any of self.AUTH_DOMAINS"""
        headers = prepared_request.headers
        url = prepared_request.url
        if 'Authorization' in headers:
            original_parsed = requests.utils.urlparse(response.request.url)
            redirect_parsed = requests.utils.urlparse(url)
            original_domain = '.'.join(original_parsed.hostname.split('.')[-2:])
            redirect_domain = '.'.join(redirect_parsed.hostname.split('.')[-2:])
            if (
                    original_domain != redirect_domain and
                    redirect_domain not in self.AUTH_DOMAINS and
                    original_domain not in self.AUTH_DOMAINS):
                del headers['Authorization']
                
s = DayforceSession()

# Authentication

data = {
    "grant_type" : "password",
    "companyId" : "ddn",
    "username" : "DFWSTest",
    "password" : "DFWSTest",
    "client_id" : "Dayforce.HCMAnywhere.Client"
}

token_resp = requests.post(auth_url, data=data)
access_token = json.loads(token_resp.text)["access_token"]

# Request data to REST API

s.headers['Authorization'] =  f"Bearer {access_token}"
api_resp = s.get(api_url)

print(api_resp.history)
print(api_resp)
print(api_resp.text)
print(s.headers['User-Agent'])

This is of course a minimum example. But it illustrates that it is possible to just subclass Session and even explicitly refer to what domains we are happy to be redirected. I of course defer to Airbyte developers for implementation.