Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable unlisted_choice in LEAP hub #3047

Merged
merged 3 commits into from
Aug 29, 2023

Conversation

yuvipanda
Copy link
Member

Ref #2146

Just brings in @GeorgianaElena's work in jupyterhub/kubespawner#769

@yuvipanda yuvipanda requested a review from a team as a code owner August 29, 2023 06:08
@github-actions
Copy link

github-actions bot commented Aug 29, 2023

Merging this PR will trigger the following deployment actions.

Support and Staging deployments

Cloud Provider Cluster Name Upgrade Support? Reason for Support Redeploy Upgrade Staging? Reason for Staging Redeploy
gcp 2i2c-uk No Yes Core infrastructure has been modified
aws nasa-veda No Yes Core infrastructure has been modified
gcp callysto No Yes Core infrastructure has been modified
aws carbonplan No Yes Core infrastructure has been modified
aws ubc-eoas No Yes Core infrastructure has been modified
gcp pangeo-hubs No Yes Core infrastructure has been modified
kubeconfig utoronto No Yes Core infrastructure has been modified
aws smithsonian No Yes Core infrastructure has been modified
aws jupyter-meets-the-earth No Yes Core infrastructure has been modified
gcp meom-ige No Yes Core infrastructure has been modified
aws catalystproject-africa No Yes Core infrastructure has been modified
gcp linked-earth No Yes Core infrastructure has been modified
aws openscapes No Yes Core infrastructure has been modified
aws gridsst No Yes Core infrastructure has been modified
gcp 2i2c No Yes Core infrastructure has been modified
gcp qcl No Yes Core infrastructure has been modified
gcp catalystproject-latam No Yes Core infrastructure has been modified
gcp cloudbank No Yes Core infrastructure has been modified
gcp m2lines No Yes Core infrastructure has been modified
gcp awi-ciroh No Yes Core infrastructure has been modified
aws victor No Yes Core infrastructure has been modified
aws nasa-cryo No Yes Core infrastructure has been modified
gcp leap No Yes Core infrastructure has been modified
aws nasa-ghg No Yes Core infrastructure has been modified
aws 2i2c-aws-us No Yes Core infrastructure has been modified

Production deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
gcp 2i2c-uk lis Core infrastructure has been modified
aws nasa-veda prod Core infrastructure has been modified
gcp callysto prod Core infrastructure has been modified
aws carbonplan prod Core infrastructure has been modified
aws ubc-eoas prod Core infrastructure has been modified
gcp pangeo-hubs prod Core infrastructure has been modified
gcp pangeo-hubs coessing Core infrastructure has been modified
kubeconfig utoronto prod Core infrastructure has been modified
kubeconfig utoronto r-prod Core infrastructure has been modified
aws smithsonian prod Core infrastructure has been modified
aws jupyter-meets-the-earth prod Core infrastructure has been modified
gcp meom-ige prod Core infrastructure has been modified
gcp linked-earth prod Core infrastructure has been modified
aws openscapes prod Core infrastructure has been modified
aws gridsst prod Core infrastructure has been modified
gcp 2i2c hackanexoplanet Core infrastructure has been modified
gcp 2i2c imagebuilding-demo Core infrastructure has been modified
gcp 2i2c demo Core infrastructure has been modified
gcp 2i2c ohw Core infrastructure has been modified
gcp 2i2c pfw Core infrastructure has been modified
gcp 2i2c aup Core infrastructure has been modified
gcp 2i2c temple Core infrastructure has been modified
gcp 2i2c ucmerced Core infrastructure has been modified
gcp 2i2c climatematch Core infrastructure has been modified
gcp 2i2c neurohackademy Core infrastructure has been modified
gcp 2i2c mtu Core infrastructure has been modified
gcp qcl prod Core infrastructure has been modified
gcp catalystproject-latam unitefa-conicet Core infrastructure has been modified
gcp cloudbank bcc Core infrastructure has been modified
gcp cloudbank ccsf Core infrastructure has been modified
gcp cloudbank csm Core infrastructure has been modified
gcp cloudbank dvc Core infrastructure has been modified
gcp cloudbank elcamino Core infrastructure has been modified
gcp cloudbank evc Core infrastructure has been modified
gcp cloudbank glendale Core infrastructure has been modified
gcp cloudbank howard Core infrastructure has been modified
gcp cloudbank miracosta Core infrastructure has been modified
gcp cloudbank skyline Core infrastructure has been modified
gcp cloudbank demo Core infrastructure has been modified
gcp cloudbank fresno Core infrastructure has been modified
gcp cloudbank humboldt Core infrastructure has been modified
gcp cloudbank laney Core infrastructure has been modified
gcp cloudbank sbcc Core infrastructure has been modified
gcp cloudbank lacc Core infrastructure has been modified
gcp cloudbank lamission Core infrastructure has been modified
gcp cloudbank mills Core infrastructure has been modified
gcp cloudbank mission Core infrastructure has been modified
gcp cloudbank norco Core infrastructure has been modified
gcp cloudbank palomar Core infrastructure has been modified
gcp cloudbank pasadena Core infrastructure has been modified
gcp cloudbank sjcc Core infrastructure has been modified
gcp cloudbank sacramento Core infrastructure has been modified
gcp cloudbank srjc Core infrastructure has been modified
gcp cloudbank saddleback Core infrastructure has been modified
gcp cloudbank santiago Core infrastructure has been modified
gcp cloudbank sjsu Core infrastructure has been modified
gcp cloudbank tuskegee Core infrastructure has been modified
gcp cloudbank wlac Core infrastructure has been modified
gcp cloudbank csulb Core infrastructure has been modified
gcp m2lines prod Core infrastructure has been modified
gcp awi-ciroh prod Core infrastructure has been modified
aws victor prod Core infrastructure has been modified
aws nasa-cryo prod Core infrastructure has been modified
gcp leap prod Core infrastructure has been modified
aws nasa-ghg prod Core infrastructure has been modified
aws 2i2c-aws-us researchdelight Core infrastructure has been modified
aws 2i2c-aws-us ncar-cisl Core infrastructure has been modified
aws 2i2c-aws-us go-bgc Core infrastructure has been modified
aws 2i2c-aws-us itcoocean Core infrastructure has been modified
aws 2i2c-aws-us cosmicds Core infrastructure has been modified

@yuvipanda yuvipanda force-pushed the unlisted-choice-take-2 branch from 3d256f9 to 8d0d3d1 Compare August 29, 2023 06:11
@yuvipanda
Copy link
Member Author

This unfortunately still has the problem previously reported, where once you use an image as the other choice, you can't use any other image :( This shouldn't be deployed to the LEAP prod hub until we figure that out.

@consideRatio
Copy link
Contributor

This unfortunately still has the problem previously reported, where once you use an image as the other choice, you can't use any other image :(

I reproduce this once before in staging.leap.2i2c.cloud earlier today, but I've not been able to reproduce it later or with a local k3s cluster with jupyterhub 4.0.2 + kubespawner's main branch.

@GeorgianaElena
Copy link
Member

I've just pushed a commit that I think fixes it. I've manually deployed it to staging https://staging.leap.2i2c.cloud and seems to be working.

It looks like it was the same bug, with the same fix as the one in jupyterhub/kubespawner#766. But because we were using a callable profile_list defined by us, that fix was not applying for our case.

@yuvipanda
Copy link
Member Author

Yesss, I can confirm that fixes it @GeorgianaElena!!!! \o/

@yuvipanda yuvipanda merged commit ec38063 into 2i2c-org:master Aug 29, 2023
@github-actions
Copy link

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/6013781012

@consideRatio
Copy link
Contributor

Nice work @GeorgianaElena!!!

I think there is another fix to make in kubespawner also. I think this logic shouldn't just take the value returned by profile_list, but should make a copy of the value returned by a profile_list callable.

        if callable(self.profile_list):
            profile_list = await maybe_future(self.profile_list(self))
        else:
            # Use a copy of the self.profile_list dict,
            # otherwise we might unintentionally modify it
            profile_list = copy.deepcopy(self.profile_list)

        profile_list = self._populate_profile_list_defaults(profile_list)

This upstream fix isn't a problem if the callable profile_list returns a copy at all times as done in the leap hub - but we don't return a copy at all times, for example when we return this:

            # Only apply to GitHub Authenticator
            if not isinstance(spawner.authenticator, GitHubOAuthenticator):
                return original_profile_list

@consideRatio
Copy link
Contributor

consideRatio commented Aug 29, 2023

I'll submit a kubespawner PR to make a copy of the returned value of a profile_list callable, and then we should be fine for all hubs - but we will need the fix in b62a46a as well I think.

@yuvipanda
Copy link
Member Author

@consideRatio I just looked into that, and I don't think that works - the problem is in

original_profile_list = c.KubeSpawner.profile_list
, where we are taking the original value of profile_list, and modifying that is the problem. In the kubespawner code, the value returned by the callable isn't re-assigned to self.profile_list anywhere I can see, so is probably fine.

I think we should provide guidance that says "if you are using profile_list as a callable, make sure to make a copy!".

But curious to see the outcome of your investigation too. Perhaps I'm wrong.

@consideRatio
Copy link
Contributor

consideRatio commented Aug 29, 2023

@consideRatio I just looked into that, and I don't think that works - the problem is in

I agree that the problem we have for the leap hub is the specific one patched by @GeorgianaElena, and would be a problem even if we patched what I consider patching in KubeSpawner.

But, I think there is another bug, and I think we are exposed to it because we break the advice of "if you are using profile_list as a callable, make sure to make a copy!".

  1. We end up with a callable profile_list
  2. We end up storing a single reference to what the callable profile_list will return
    EDIT: our callable function now only sometimes makes a copy, not always - so the bug is only partially avoided
  3. KubeSpawner doesn't make a copy of the return value of a callable profile_list as it does for a non-callable dictionary config value

@yuvipanda
Copy link
Member Author

Hmm, I don't fully understand so am curious to see your PR @consideRatio.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Status: Done 🎉
Development

Successfully merging this pull request may close these issues.

3 participants