| Message ID | 20250401010000.764234-1-cel@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | Simplify OCI configuration menu | expand |
On Mon, Mar 31, 2025 at 08:59:29PM -0400, cel@kernel.org wrote: > From: Chuck Lever <chuck.lever@oracle.com> > I've come up with some simplifications that can apply to all of > the providers, but I'd like to start with OCI because Reviewed-by: Luis Chamberlain <mcgrof@kernel.org> > 26 files changed, 1472 insertions(+), 556 deletions(-) That's huge, for all this! I know AWS supports a simple .aws/credentials and I think OCI might too support something similar. If so that might beg the question if eventually we should transition to just that to help simplify things on bring up even further. That should enable users to quickly "fly" on enablement. Other than that, any chance you might be up to add yourself as the cloud / terraform maintainer under MAINTAINERS ? Luis
On Wed, Apr 02, 2025 at 12:21:06PM -0700, Luis Chamberlain wrote: > On Mon, Mar 31, 2025 at 08:59:29PM -0400, cel@kernel.org wrote: > > From: Chuck Lever <chuck.lever@oracle.com> > > I've come up with some simplifications that can apply to all of > > the providers, but I'd like to start with OCI because > > Reviewed-by: Luis Chamberlain <mcgrof@kernel.org> > > > 26 files changed, 1472 insertions(+), 556 deletions(-) > > That's huge, for all this! I meant: thanks for all this! Luis
On 4/2/25 3:21 PM, Luis Chamberlain wrote: > On Mon, Mar 31, 2025 at 08:59:29PM -0400, cel@kernel.org wrote: >> From: Chuck Lever <chuck.lever@oracle.com> >> I've come up with some simplifications that can apply to all of >> the providers, but I'd like to start with OCI because > > Reviewed-by: Luis Chamberlain <mcgrof@kernel.org> Thanks for taking a look! I guess my basic concern here is that I'm changing the name of some Kconfig variables, which would break backwards compatibility. If you and Chandan feel that is not a roadblock, then I will start committing these and the similar patches I have for the other three major providers. >> 26 files changed, 1472 insertions(+), 556 deletions(-) > > That's huge, for all this! I know AWS supports a simple .aws/credentials > and I think OCI might too support something similar. If so that might beg > the question if eventually we should transition to just that to help > simplify things on bring up even further. That should enable users to quickly > "fly" on enablement. The varying way that the providers handle authentication is one of the triggers that started me down this road. I would like something that is easy to bring up and matches the authentication that each provider's CLI tool does. I'm still on the fence about whether keeping the secrets in the kdevops .config or in environment variables or in a ~/.{aws,oci,azure} profile is the better approach. I was thinking of enabling all these possibilities and let users choose which one suits their security requirements... The values of environment variables, I will note, are leaked into the Buildbot log. The best answer might therefore be ~/.oci/profile etc. > Other than that, any chance you might be up to add yourself as the > cloud / terraform maintainer under MAINTAINERS ? I can do that. I promise I will hang out for a beer next time we meet f2f!
On Wed, Apr 02, 2025 at 03:38:04PM -0400, Chuck Lever wrote: > On 4/2/25 3:21 PM, Luis Chamberlain wrote: > > On Mon, Mar 31, 2025 at 08:59:29PM -0400, cel@kernel.org wrote: > >> From: Chuck Lever <chuck.lever@oracle.com> > >> I've come up with some simplifications that can apply to all of > >> the providers, but I'd like to start with OCI because > > > > Reviewed-by: Luis Chamberlain <mcgrof@kernel.org> > > Thanks for taking a look! > > I guess my basic concern here is that I'm changing the name of some > Kconfig variables, which would break backwards compatibility. I've addressed concerns like that befored by adding a respective kconfig value and mapping the old one to the new shiny one with an XXX to annotate we should remove the older one later. But if for cloud I suspect most users might stick to that version of kdevops and have no need to upgrade too, and so I am not sure if its needed as much as the guest / virtualization use case. > If you and Chandan feel that is not a roadblock, then I will start > committing these and the similar patches I have for the other three > major providers. My use cases of cloud are all completely ephemeral and stuck to one version of kdevops in particular because of the cost, and so I am careful to always 'make destroy' as soon as I'm done otheriwse you can economically bleed. And so I think economics drives kdevops cloud instances to be super ephemeral. So for *me*, I say go for it. But I am not sure if other folks are using kdevops for the cloud in other ways, but certainly Chandan would have good sensing on that. > The varying way that the providers handle authentication is one of > the triggers that started me down this road. I would like something > that is easy to bring up and matches the authentication that each > provider's CLI tool does. Definitely. > I'm still on the fence about whether keeping the secrets in the > kdevops .config Secrets don't have to be part of an output .config, they should not. The reasin being that we want to support using cloud for CI too and in doing so we don't want any secrets leaked out on .config as we automatically upload to kdevops-results-archive the .config used so to enable reproducible CI builds / testing. If you look at kdevops-results-archive the .configs are never sensitive to the environment or host. Its just generic info. We support some secrets for enterprise releases like SUSE, but I forget what was done for RHEL. At least for SUSE the expectations is that you fill in a yaml optional secret file readin for extra vars. Even if we had a new "output private_yaml" that kconfig symbol would still be part of .config, and so if we wantd to add a "output private_yaml" we'd have to negate the entry being put on .config but not sure if that may mess with Kconfig's dep mechanisms. > or in environment variables or in a ~/.{aws,oci,azure} The nice thing about ~/.{aws,oci,azure} is its out of the way and we just infer you have it, we never have any of that added to .config. > profile is the better approach. I was thinking of enabling all these > possibilities and let users choose which one suits their security > requirements... Makes sense. We know AWS supports it, OCI I have heard does too, but not sure about azure. > The values of environment variables, I will note, are leaked into the > Buildbot log. Oh no. > The best answer might therefore be ~/.oci/profile etc. I think that's a clean simple solution to the problem, if we can support it. > > Other than that, any chance you might be up to add yourself as the > > cloud / terraform maintainer under MAINTAINERS ? > > I can do that. > > I promise I will hang out for a beer next time we meet f2f! Great! Luis
From: Chuck Lever <chuck.lever@oracle.com> Over the past month or two I've had to set up kdevops with all four major public cloud providers in order to test the new volume_group role on each of them. There are a lot of similarities amongst the providers, and one of them is that set-up is probably much more complicated than necessary for all of them. I've come up with some simplifications that can apply to all of the providers, but I'd like to start with OCI because it looks to me like it's the one that has the most users currently. Let's figure out what OCI configuration needs to look like (ie, what changes its users will tolerate!) and make the other providers work the same way. This series features a number of changes: - The configuration is split into several submenus, just like the cloud console, to help humans navigate the complexity - A number of internal simplifications will help us keep up with changes and new features in the provider. In particular, splitting things into smaller files will make it simpler to add scripts to poll provider feature changes and adjust the kdevops menus accordingly - Free-form strings and "copy this OCID from the console" are replaced with "choose from this menu of known-working options". This includes selection of region, availability domain, shape, and source image - I've added an option to provision a VCN instead of assuming a pre-existing one, similar to how the other providers work Some of this stuff will make immediate sense, and some might be controversial. All comments are welcome. https://github.com/chucklever/kdevops branch => terraform-oci Chuck Lever (31): terraform/OCI: Remove terraform_oci_instance_display_name terraform/OCI: One default value to rule them terraform/OCI: Add an "Identity & Access" submenu terraform/OCI: Add a "Resource location" submenu terraform/OCI: Add a "Compute" Kconfig submenu terraform/OCI: Add a "Storage" Kconfig submenu terraform/OCI: Add a "Networking" Kconfig submenu terraform/OCI: Use "output yaml" for the tenancy OCID terraform/OCI: Use "output yaml" for the user OCID terraform/OCI: Use "output yaml" for the signing key terraform/OCI: Use "output yaml" for the fingerprint terraform/OCI: Add a Region selector terraform/OCI: Add an availability domain selector terraform/OCI: Select your compartment by name instead of by OCID terraform/OCI: Use "output yaml" for the instance shape setting terraform/OCI: Add a shape selector for Flex shapes terraform/OCI: Use "output yaml" for the OCPUs setting terraform/OCI: Use "output yaml" for the memory_in_gbs setting terraform/OCI: Add a shape family selector terraform/OCI: Add a bare metal shape selector terraform/OCI: Use "output yaml" for the source image setting terraform/OCI: Simplify image selection terraform/OCI: Remove TERRAFORM_OCI_VOLUMES_ENABLE_EXTRA terraform/OCI: Use "output yaml" for the assign_public_ip" setting terraform/OCI: Use "output yaml" for the subnet_ocid setting terraform/OCI: Add a default VCN terraform/OCI: Add a Kconfig switch to create a VCN on the fly terraform/OCI: Run "terraform fmt" on provider.tf terraform/OCI: Run "terraform fmt" on main.tf terraform/OCI: Nit: alphabetize vars.tf terraform/OCI: Update the OCI section of docs/kdevops-terraform.md docs/kdevops-terraform.md | 117 +++---- playbooks/roles/gen_tfvars/defaults/main.yml | 20 +- .../templates/oci/terraform.tfvars.j2 | 20 +- scripts/terraform.Makefile | 31 -- terraform/oci/Kconfig | 289 +----------------- terraform/oci/kconfigs/Kconfig.compute | 72 +++++ terraform/oci/kconfigs/Kconfig.identity | 35 +++ terraform/oci/kconfigs/Kconfig.location | 52 ++++ terraform/oci/kconfigs/Kconfig.network | 39 +++ terraform/oci/kconfigs/Kconfig.storage | 137 +++++++++ .../oci/kconfigs/images/Kconfig.OracleLinux | 237 ++++++++++++++ terraform/oci/kconfigs/images/Kconfig.Ubuntu | 162 ++++++++++ terraform/oci/kconfigs/images/Kconfig.custom | 20 ++ .../kconfigs/regions/Kconfig.ap-hyderabad-1 | 28 ++ .../oci/kconfigs/regions/Kconfig.ap-mumbai-1 | 27 ++ .../kconfigs/regions/Kconfig.ca-montreal-1 | 28 ++ .../oci/kconfigs/regions/Kconfig.ca-toronto-1 | 28 ++ .../oci/kconfigs/regions/Kconfig.us-ashburn-1 | 36 +++ .../oci/kconfigs/regions/Kconfig.us-chicago-1 | 36 +++ .../oci/kconfigs/regions/Kconfig.us-phoenix-1 | 36 +++ .../oci/kconfigs/regions/Kconfig.us-sanjose-1 | 28 ++ terraform/oci/kconfigs/shapes/Kconfig.bm | 83 +++++ terraform/oci/kconfigs/shapes/Kconfig.flex | 104 +++++++ terraform/oci/main.tf | 195 +++++++++--- terraform/oci/provider.tf | 10 +- terraform/oci/vars.tf | 158 +++++----- 26 files changed, 1472 insertions(+), 556 deletions(-) create mode 100644 terraform/oci/kconfigs/Kconfig.compute create mode 100644 terraform/oci/kconfigs/Kconfig.identity create mode 100644 terraform/oci/kconfigs/Kconfig.location create mode 100644 terraform/oci/kconfigs/Kconfig.network create mode 100644 terraform/oci/kconfigs/Kconfig.storage create mode 100644 terraform/oci/kconfigs/images/Kconfig.OracleLinux create mode 100644 terraform/oci/kconfigs/images/Kconfig.Ubuntu create mode 100644 terraform/oci/kconfigs/images/Kconfig.custom create mode 100644 terraform/oci/kconfigs/regions/Kconfig.ap-hyderabad-1 create mode 100644 terraform/oci/kconfigs/regions/Kconfig.ap-mumbai-1 create mode 100644 terraform/oci/kconfigs/regions/Kconfig.ca-montreal-1 create mode 100644 terraform/oci/kconfigs/regions/Kconfig.ca-toronto-1 create mode 100644 terraform/oci/kconfigs/regions/Kconfig.us-ashburn-1 create mode 100644 terraform/oci/kconfigs/regions/Kconfig.us-chicago-1 create mode 100644 terraform/oci/kconfigs/regions/Kconfig.us-phoenix-1 create mode 100644 terraform/oci/kconfigs/regions/Kconfig.us-sanjose-1 create mode 100644 terraform/oci/kconfigs/shapes/Kconfig.bm create mode 100644 terraform/oci/kconfigs/shapes/Kconfig.flex