Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(insecure_transport) OAuth 2 MUST utilize https. #44

Open
liquanyong305 opened this issue Dec 30, 2021 · 17 comments
Open

(insecure_transport) OAuth 2 MUST utilize https. #44

liquanyong305 opened this issue Dec 30, 2021 · 17 comments

Comments

@liquanyong305
Copy link

liquanyong305 commented Dec 30, 2021 •
edited
Loading

hello everyone:
Even if I login using the https protocol, it still shows the error ' (insecure_transport) OAuth 2 MUST utilize https.' after the authentication.
ckan version: 2.8.9

I use fiware IDM authentication. After the authentication, I checked the cookies and found that the expires of ckan is one year earlier than the current date.
image

I'm stuck because I can't login.

Thanks a lot for the work you have done on this plugin.
@jlphillipsphd
Copy link

I was able to fix this particular issue by setting the environment variable:

export OAUTHLIB_INSECURE_TRANSPORT=True

Reference (see Additional Notes at the bottom of the page): https://github.com/conwetlab/ckanext-oauth2/wiki/Activating-and-Installing

However, this only got me one more step, and led to a new error after attempting to log in:
image

I am running behind an Apache2 reverse proxy for SSL termination (on port 443) while my CKAN instance is plain http (at http://localhost:5000/) set up using docker-compose. I add the variable above to my docker-compose.yml to make it get to this stage.

In the logs I just see the following:

docker logs ckan

2022-01-08 22:48:02,092 DEBUG [ckanext.oauth2.plugin] identify
2022-01-08 22:48:02,093 WARNI [ckanext.oauth2.plugin] The user is not currently logged...
2022-01-08 22:48:02,094 DEBUG [ckanext.oauth2.controller] login
2022-01-08 22:48:02,096 DEBUG [ckanext.oauth2.oauth2] Challenge: Redirecting challenge to page https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=[hidden]&redirect_uri=https%3A%2F%2F[hidden]%2Foauth2%2Fcallback&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile&state=[hidden]
2022-01-08 22:48:02,097 INFO  [ckan.lib.base]  /user/login render time 0.010 seconds
2022-01-08 22:48:04,543 DEBUG [ckanext.oauth2.plugin] identify
2022-01-08 22:48:04,544 WARNI [ckanext.oauth2.plugin] The user is not currently logged...
/usr/lib/ckan/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'accounts.google.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/usr/lib/ckan/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.googleapis.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
2022-01-08 22:48:04,760 INFO  [ckan.lib.base]  /oauth2/callback render time 0.221 seconds
2022-01-08 22:48:04,878 DEBUG [ckanext.oauth2.plugin] identify
2022-01-08 22:48:04,879 WARNI [ckanext.oauth2.plugin] The user is not currently logged...
2022-01-08 22:48:04,960 INFO  [ckan.config.middleware.flask_app]  200 / render time 0.085 seconds
2022-01-08 22:48:05,096 DEBUG [ckanext.oauth2.plugin] identify
2022-01-08 22:48:05,101 WARNI [ckanext.oauth2.plugin] The user is not currently logged...

I don't know what the issue is at this point now though... any thoughts?

@aitormagan
Copy link
Contributor

aitormagan commented Jan 8, 2022 via email

@jlphillipsphd
Copy link

Thanks for getting back with us @aitormagan !

Here are my environment settings in docker-compose.yml:

      - CKAN_OAUTH2_AUTHORIZATION_ENDPOINT=https://accounts.google.com/o/oauth2/auth
      - CKAN_OAUTH2_TOKEN_ENDPOINT=https://accounts.google.com/o/oauth2/token
      - CKAN_OAUTH2_PROFILE_API_URL=https://www.googleapis.com/oauth2/v2/userinfo
      - CKAN_OAUTH2_CLIENT_ID=[hidden]
      - CKAN_OAUTH2_CLIENT_SECRET=[hidden]
      - CKAN_OAUTH2_SCOPE=https://www.googleapis.com/auth/userinfo.profile
      - CKAN_OAUTH2_PROFILE_API_USER_FIELD=email
      - CKAN_OAUTH2_PROFILE_API_FULLNAME_FIELD=name
      - CKAN_OAUTH2_PROFILE_API_MAIL_FIELD=email
      - CKAN_OAUTH2_AUTHORIZATION_HEADER=Authorization
      - OAUTHLIB_INSECURE_TRANSPORT=True

Note that I -do- see the transfer to the Google login screen, then I can select my account and see it come back to the ckan site. Is there something I am missing about what these settings need to be?

I also was using ckan v2.9.4 before, but decided to downgrade to a fresh installation of v2.8.9 since the docs for the extension don't mention explicit compatibility with 2.9. I still have the same issue, but the logs look slightly different:

2022-01-08 23:40:37,555 WARNI [ckanext.oauth2.plugin] The user is not currently logged...
2022-01-08 23:40:37,556 DEBUG [ckanext.oauth2.controller] login
2022-01-08 23:40:37,557 DEBUG [ckanext.oauth2.oauth2] Challenge: Redirecting challenge to page https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=[hidden]&redirect_uri=https%3A%2F%2F[hidden]%2Foauth2%2Fcallback&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile&state=eyJjYW1lX2Zyb20iOiAiL2Rhc2hib2FyZCJ9
2022-01-08 23:40:37,558 INFO  [ckan.lib.base]  /user/login render time 0.009 seconds
2022-01-08 23:40:40,428 DEBUG [ckanext.oauth2.plugin] identify
2022-01-08 23:40:40,429 WARNI [ckanext.oauth2.plugin] The user is not currently logged...
/usr/lib/ckan/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/ckan/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
2022-01-08 23:40:40,567 INFO  [ckan.lib.base]  /oauth2/callback render time 0.142 seconds
2022-01-08 23:40:40,746 DEBUG [ckanext.oauth2.plugin] identify
2022-01-08 23:40:40,747 WARNI [ckanext.oauth2.plugin] The user is not currently logged...
2022-01-08 23:40:41,454 DEBUG [ckanext.oauth2.plugin] identify
2022-01-08 23:40:41,455 WARNI [ckanext.oauth2.plugin] The user is not currently logged...

Maybe it's just the way I am running things though because I am running my reverse proxy (apache) entirely outside of docker/docker-compose... the docs for CKAN didn't include any info on how to use apache/nginx although I -do- see the config files packaged along with the docker-compose setup (apache.conf and apache.wsgi). So, this begs the question of how to set this up using docker-compose? Either it's working and not documented or just not working yet?

I guess the extension documentation assumes that apache and/or nginx are running in the -same environment- as the ckan? If one uses a docker-based deployment then this isn't the case.

Sorry to derail from the main issue the OP is experiencing... it might be the same issue based on what I saw before, but my setup seems to be very different from what the extension assumes so I'll do some more digging on how HTTPS is supposed to be set up with a docker-compose deployments instead of posting more here.

@aitormagan
Copy link
Contributor

aitormagan commented Jan 9, 2022 via email

@jlphillipsphd
Copy link

Thanks again for the pointers: I am trying to narrow it down.

Here is my updated config:

      - CKAN_OAUTH2_AUTHORIZATION_ENDPOINT=https://accounts.google.com/o/oauth2/auth
      - CKAN_OAUTH2_TOKEN_ENDPOINT=https://accounts.google.com/o/oauth2/token
      - CKAN_OAUTH2_PROFILE_API_URL=https://www.googleapis.com/oauth2/v1/userinfo
      - CKAN_OAUTH2_CLIENT_ID=[hidden]
      - CKAN_OAUTH2_CLIENT_SECRET=[hidden]
      - CKAN_OAUTH2_SCOPE=https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/plus.me
      - CKAN_OAUTH2_PROFILE_API_USER_FIELD=email
      - CKAN_OAUTH2_PROFILE_API_FULLNAME_FIELD=name
      - CKAN_OAUTH2_PROFILE_API_MAIL_FIELD=email
      - CKAN_OAUTH2_AUTHORIZATION_HEADER=Authorization
      - OAUTHLIB_INSECURE_TRANSPORT=True

However, this gives me the following error now:
image

I've checked over the OAuth2 client in the GCP dashboard and approved everything I could think of... not sure why I would see this - everything looks approved.

@jlphillipsphd
Copy link

Also, no indication of a failure in the apache logs... unless you see something I don't?

X.X.X.X - - [09/Jan/2022:19:28:29 +0000] "GET / HTTP/1.1" 200 4272 "https://[hidden]/" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
X.X.X.X - - [09/Jan/2022:19:28:29 +0000] "GET /api/i18n/en HTTP/1.1" 200 516 "https://[hidden]/" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
X.X.X.X - - [09/Jan/2022:19:28:31 +0000] "GET /user/login HTTP/1.1" 302 1804 "https://[hidden]/" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
X.X.X.X - - [09/Jan/2022:19:28:34 +0000] "GET /oauth2/callback?state=eyJjYW1lX2Zyb20iOiAiL2Rhc2hib2FyZCJ9&code=4%2F0AX4XfWgz2FqQlw2MoiwkdQNhNaDHs7HC5RQMLz3-C00GaC5ie2qlNjeMODHxqpx7rIYA8Q&scope=email+profile+openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&authuser=0&prompt=none HTTP/1.1" 302 417 "https://accounts.google.com/o/oauth2/auth/oauthchooseaccount?response_type=code&client_id=[hidden]&redirect_uri=https%3A%2F%2F[hidden]%2Foauth2%2Fcallback&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fplus.me&state=eyJjYW1lX2Zyb20iOiAiL2Rhc2hib2FyZCJ9&flowName=GeneralOAuthFlow" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
X.X.X.X - - [09/Jan/2022:19:28:34 +0000] "GET / HTTP/1.1" 200 4032 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
X.X.X.X - - [09/Jan/2022:19:28:35 +0000] "GET /api/i18n/en HTTP/1.1" 200 378 "https://[hidden]/" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"

@aitormagan
Copy link
Contributor

aitormagan commented Jan 9, 2022 via email

@jlphillipsphd
Copy link

I finally got it working!!!

Step 1 - once I had all other settings for scope and domains correct in the GCP console, I had to regenerate the client secret for my OAuth credential. (I think this was the main culprit and not sure why - some StackOverflow searches led me to see this had happened to others as well but couldn't find a reason -why- it is needed).

Step 2 - I used docker-compose down, changed to the new client secret, and then docker-compose up -d. This led me to this message:
image

OK, so I did another docker-compose down and then changed that setting to match what was indicated by that return value:

      - CKAN_OAUTH2_SCOPE=openid https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile

After bringing it back up (docker-compose up -d): It was working!!! 😄

Looks like the insecure transport option -is- required when behind a reverse proxy and the ckan is running on plain http. This does make sense because the ckan is talking to Apache via http even if Apache is sending that out to the world over https. I also have Apache set up to rewrite all http URLs to https URLs, so I don't think this is really a security concern (so long as the local network and ckan port are not accessible from the outside - which is my current setup).

It seems the plugin documentation may need to be updated to match the scopes info above?

@jlphillipsphd
Copy link

Only other thing I would like to add: I only had success using ckan 2.8.9. When I tried 2.9.4, then I could log in, but the user's profile page resulted in an internal server error. I have had no troubles at all when making profile changes, making an oauth user a sysadmin, adding organizations/groups, or creating editing data sets when using 2.8.9.

@aitormagan
Copy link
Contributor

aitormagan commented Jan 9, 2022 via email

@jlphillipsphd
Copy link

Thanks so much for your help @aitormagan !

@liquanyong305 , I hope the insecure option worked for your situation! I think Google OAuth2 requires https termination, so you may need a reverse proxy or load balancer that supports SSL termination in front of your deployment to make it work in the end.

@liquanyong305
Copy link
Author

@jlphillipsphd
thank you very much for your help, when I set insecure option, the problem was solved,
but the following problem occurs:
image

@jlphillipsphd
Copy link

It might be specific to the fiware setup: I was using google instead and I don't remember seeing that particular error.
I don't have the ability to check fiware here.

@liquanyong305
Copy link
Author

@jlphillipsphd
thank you very much for your help.

@Jianbo-jia
Copy link

@jlphillipsphd hi,I use google same with you. but I got error below
image
if set CKAN_OAUTH2_SCOPE = https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/plus.me the error same with you.
I mod CKAN_OAUTH2_SCOPE, and restart again got err(you config is success.) I am confused, can you give me some suggest. thank you very much.

@jlphillipsphd
Copy link

I haven't observed that particular error before, so I am only guessing: Did you add that email to the list of testing users in the GCP console? It's found on the "Oauth consent screen" settings for the project.
image

@TheoStefou
Copy link

I finally got it working!!!

Step 1 - once I had all other settings for scope and domains correct in the GCP console, I had to regenerate the client secret for my OAuth credential. (I think this was the main culprit and not sure why - some StackOverflow searches led me to see this had happened to others as well but couldn't find a reason -why- it is needed).

Step 2 - I used docker-compose down, changed to the new client secret, and then docker-compose up -d. This led me to this message: image

OK, so I did another docker-compose down and then changed that setting to match what was indicated by that return value:

      - CKAN_OAUTH2_SCOPE=openid https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile

After bringing it back up (docker-compose up -d): It was working!!! 😄

Looks like the insecure transport option -is- required when behind a reverse proxy and the ckan is running on plain http. This does make sense because the ckan is talking to Apache via http even if Apache is sending that out to the world over https. I also have Apache set up to rewrite all http URLs to https URLs, so I don't think this is really a security concern (so long as the local network and ckan port are not accessible from the outside - which is my current setup).

It seems the plugin documentation may need to be updated to match the scopes info above?

For anyone else that might be having trouble with this (as I did for a few hours), in scopes, openid should be the first (leading) token in the list, otherwise you will still get this error message.

@frafra frafra mentioned this issue Aug 14, 2022
Closed
@jat255 jat255 mentioned this issue Nov 12, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@aitormagan @jlphillipsphd @liquanyong305 @TheoStefou @Jianbo-jia