Update pip package + ckan 2.9 support

[mdantonio]

I'm installing this extention on a dockerized ckan 2.7 with nginx as reverse proxy to enable https

Everything is working fine except for the redirect uri built as:

http://myhost/oauth2/callback (schema is wrong, should be https)

By inspecting the source code I found that the redirection uri is built from request.host_url:

    def _redirect_uri(self, request):
        return ''.join([request.host_url, constants.REDIRECT_URL])

And since nginx forwards requests to the ckan container via http, the schema found into the flask request object is http and not https (=> uri is wrongly built)

This is my nginx configuration:

[...]
proxy_pass http://ckan:5000;
proxy_set_header   X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_set_header   X-Forwarded-Proto $scheme;
[...]

Already tried to force the schema passed as X-Forwarded-Proto but without any result.

In the meanwhile I found that recently you modified how the request uri is build:

New implementation from master:

self.redirect_uri = urljoin(urljoin(config.get('ckan.site_url', 'http://localhost:5000'), config.get('ckan.root_path')), constants.REDIRECT_URL)

And since this implementation is based on site_url, I think that this version should work fine with my configuration but i'm unable to test it.

Could you include this fix to the pypi package?
I tried to install the extention directly from git, but it does not work:

pip install --upgrade pip install git+git://github.com/conwetlab/ckanext-oauth2.git@master

Downloading/unpacking git+git://github.com/conwetlab/ckanext-oauth2.git@master
  Cloning git://github.com/conwetlab/ckanext-oauth2.git (to master) to /tmp/pip-Qo4L9v-build
  Running setup.py (path:/tmp/pip-Qo4L9v-build/setup.py) egg_info for package from git+git://github.com/conwetlab/ckanext-oauth2.git@master
Downloading/unpacking pip from https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl#sha256=717cdffb2833be8409433a93746744b59505f42146e8d37de6c62b430e25d6d7
  Downloading pip-10.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded
Downloading/unpacking install
  Could not find any downloads that satisfy the requirement install
Cleaning up...
No distributions at all found for install

As final question: do you think that you will support the latest ckan versions (2.8 and 2.9) in a near future?

Thank you very much for your work with this ext

[aarranz]

Hi @mdantonio!

We were waiting some feedback from #12, because we don't have an environment using keycloak to test. Anyway we were using this version for a while and we didn't see any problem, so I just released version 0.5.0. Any potential problem with keycloack will be fixed as soon as it get reported.

This new version should work with CKAN 2.8. We expect to work on providing CKAN 2.9 support before it is released, but I cannot warranty this at this point.

[mdantonio]

Thank you @aarranz it works

redirect_uri is now correct and it is accepted by my oauth2 server who correctly calls the callback uri via https after the autorization

ckan login -> ouath2server -> https://myhost/oauth2/callback

But once back on ckan the callback endpoint makes a new redirect to http

https://myhost/oauth2/callback -> https://myhost/

And I receive the following error:

(insecure_transport) OAuth 2 MUST utilize https.

Maybe another point in the code to be fixed? I tried to check again the implementation but i don't find where this redirection is executed

This new version should work with CKAN 2.8. We expect to work on providing CKAN 2.9 support before it is released, but I cannot warranty this at this point.

Thank you, i will try with 2.8 as soon as possible

[aitormagan]

As stated in the documentation, this extension only works when HTTPs is enabled. You can enable HTTPs in your CKAN instance by following the steps included in this guide: https://github.com/conwetlab/ckanext-oauth2/wiki/Starting-CKAN-over-HTTPs.

There is an option for the extension to work without HTTPs. Just modify the envvars file as stated in https://github.com/conwetlab/ckanext-oauth2/wiki/Activating-and-Installing.

Please, let us know if you have any other questions :)

BR
Aitor

[mdantonio]

Https is enabled and fully working (nginx proxy + letsencrypt certificate)

With version 0.4.0 the request_uri was wrongly built, but this problem is fixed on 0.5.0 (released yesterday by @aarranz, thank you)

Now a further problem affects the redirect after the callback

• ckan login -> oauth2 server: OK (up to yesterday was wrong due to redirect_uri build on http)
• ouauth2 server -> ckan callback: OK
• ckan callback -> ckan homepage: this automatic redirect is switching from https to http => fails

Hope that I clarified the problem

[aitormagan]

I know what you mean, but as I stated yesterday, enabling HTTPs in Nginx is not enough. Please, try to follow the steps indicated in the following guide to check if it works: https://github.com/conwetlab/ckanext-oauth2/wiki/Starting-CKAN-over-HTTPs-using-Apache-and-Nginx

BR
Aitor

[mdantonio]

Ah, thank you... now i understood what you mean

but I'm unable to follow this guide since i have no apache running.

I'm running a dockerized version obtained here:
https://github.com/ckan/ckan/tree/2.7/contrib/docker

Ckan is running with this command:

/usr/lib/ckan/default/bin/python2 /usr/local/bin/ckan-paster serve /etc/ckan/default/ckan.ini

And I think that it is directly served by Flask

in /etc/ckan/default/ i have ckan.ini and who.ini and nothing else (in particular i haven't any apache.wsgi)

[aitormagan]

If your CKAN instance is running only with nginx, you can follow some of the steps stated here: https://github.com/conwetlab/ckanext-oauth2/wiki/Starting-CKAN-over-HTTPs-using-Nginx

[aarranz]

Hi all, In my opinion, the problem is not related to configuring the ckan portal to use https, but with problems validating the certificate of the oauth2 server. Are you using also let's encrypt for securing the oauth2 server? Have you installed root certificates (they are installed through the `ca-certificates` package)? Best regards. El mar., 3 jul. 2018 11:01, Mattia <[email protected]> escribió:

…

Ah, thank you... now i understood what you mean but I'm unable to follow this guide since i have no apache running. I'm running a dockerized version obtained here: https://github.com/ckan/ckan/tree/2.7/contrib/docker Ckan is running with this command: /usr/lib/ckan/default/bin/python2 /usr/local/bin/ckan-paster serve /etc/ckan/default/ckan.ini And I think that it is directly served by Flask in /etc/ckan/default/ i have ckan.ini and who.ini and nothing else (in particular i haven't any apache.wsgi) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzV8YJYKhT3V6MnIubvGB1QmYBSMl70ks5uCzL2gaJpZM4U-3cj> .

[mdantonio]

@aitormagan my nginx proxy is external to the ckan container and configured with a proxy_pass. I mainly followed this configuration:

https://github.com/ckan/ckan/wiki/SSL

the docker build do not leverage on uwsgi nor physical socket, it simply expose a port. I can forward https requests from nginx to the ckan container via proxy pass and add proxy_set_header to propagate information. I tried to add configurations like proxy_set_header X-Forwarded-Proto https; and proxy_redirect http://ckan:5000 https://myhostname but without any result.

A similar problem was affecting the building of redirect_uri (built from request.host_url instead of config.ckan.site_url).

The redirect from /oauth2/callback to homepage how is built?

I were expecting that once successfully come back to ckan on the https callback, further redirects should work well (very very basicly it should simply redirect to the config.ckan.site_url)

@aarranz i'm testing this configuration by using google as oauth2 server by following the configuration reported here https://github.com/conwetlab/ckanext-oauth2/wiki/Activating-and-Installing

[aitormagan]

The redirection is made in the oauth2 module in the redirect_from_callback function. This function gets the state (which is JSON encoded in Base64) from the callback call and redirects the user to the page where the login attempt started. The thing is that no protocol is involved there as it is a relative redirection... 😢

Please, can you provide me with the details of the OAuth server you are using? (not the tokens, but the server so I can test it by myself in order to provide you with extra help... :) )

P.S.: As far as I can remember the (insecure_transport) OAuth 2 MUST utilize https error occurs when the OAuth2 server is not using HTTPs. Is that your case?

[mdantonio]

Please, can you provide me with the details of the OAuth server you are using?

I'm testing the workflow by using google, i get the configuration from https://github.com/conwetlab/ckanext-oauth2/wiki/Activating-and-Installing

here the details of my configuration:

ckan.plugins = oauth2 stats text_view image_view recline_view

## OAuth2 configuration
ckan.oauth2.logout_url = /user/logged_out
ckan.oauth2.authorization_endpoint = https://accounts.google.com/o/oauth2/auth
ckan.oauth2.token_endpoint = https://accounts.google.com/o/oauth2/token
ckan.oauth2.profile_api_url = https://www.googleapis.com/oauth2/v1/userinfo

ckan.oauth2.client_id = ...............apps.googleusercontent.com
ckan.oauth2.client_secret = ............

ckan.oauth2.scope = openid email profile
ckan.oauth2.rememberer_name = auth_tkt
ckan.oauth2.profile_api_user_field = email

ckan.oauth2.profile_api_fullname_field = name
ckan.oauth2.profile_api_mail_field = email
ckan.oauth2.authorization_header = Authorization

[mdantonio]

when i back to ckan from google this endpoint is called:

https://myhostname/oauth2/callback?state=eyJjYW1lX2Zyb20iOiAiL2Rhc2hib2FyZCJ9&code=.......&authuser=0&session_state=...&prompt=consent

Request initiator is /accounts/static/etcetc

status = 302

Note: by decoding the state i obtain: {"came_from": "/dashboard"}

After that i see a request for http://myhostname/
(should not be http://myhostname/dashboard ?)

Request initiator: /oauth2/callback/etcetc

status = 301

After that i see a request for https://myhostname/

Request initiator: http://myhostname/

status = 200

So from the callback the relative redirect brings to http and nginx with a 301 redirects it to https

I fear the problem is the nginx proxy_pass who does not allow the extension to detect something required to make oauth2 working

But i don't understand what is missing, this is my proxy pass configuration:

    location / {
        proxy_pass http://ckan:5000;

        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;

        proxy_cache_bypass $cookie_auth_tkt;
        proxy_no_cache     $cookie_auth_tkt;
        proxy_cache_valid  30m;
        proxy_cache_key    $host$scheme$proxy_host$request_uri;
        
        proxy_set_header   X-Forwarded-Proto $scheme;
        # proxy_set_header   X-Forwarded-Proto https;
    }

[aitormagan]

Hi,

As far as I understand the way the extension works, the problem may be you are using paster to launch the CKAN instance, but the extension needs a HTTPs server to work and paster does not provide this option (at least I haven't found that on the Internet). The only idea that comes to my mind is to define the environment variable OAUTHLIB_INSECURE_TRANSPORT with value True before starting the paster process. This will skip the error you mentioned, but will decrease the security level of your application.

export OAUTHLIB_INSECURE_TRANSPORT=True

Anyway, paster should only be used for developing purposes. CKAN can works both with Apache2 or Nginx directly. You can configure your Nginx instance to serve CKAN directly by following this guide: https://github.com/conwetlab/ckanext-oauth2/wiki/Starting-CKAN-over-HTTPs-using-Nginx.

Please, let us know if one of the solutions provided here works for you.

BR
Aitor

[mdantonio]

@aitormagan it works, thank you! Oauth2 is fully working now

I added OAUTHLIB_INSECURE_TRANSPORT=True to docker-compose environment

Anyway, paster should only be used for developing purposes.

I really understand that. In a near future i will modify the default Dockerfile obtained from ckan repository to replace paster with Nginx and remove the OAUTHLIB_INSECURE_TRANSPORT

Thank you again (feel free to close the issue)

[aarranz]

Great!

Closing this ticket... Only a last thing, we have being testing the plugin and the released version (v0.5.0) were not working on CKAN 2.8 (although the we are working on it and will be available on the following days).

[mdantonio]

the released version (v0.5.0) were not working on CKAN 2.8

Ok thank you

[ansh1221]

Hi @aarranz @aitormagan @mdantonio I am facing the following error while using nginx + apache + ckan

Have added the variable : export OAUTHLIB_INSECURE_TRANSPORT=True in docker as well as envvars
and also proxy_set_header X-Forwarded-Proto $scheme; in nginx. But still I get the error. Can you please suggest me in it.