Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARO-13400] Old E2E pipeline uses MiWi clusters #4056

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

[ARO-13400] Old E2E pipeline uses MiWi clusters #4056

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
4ccd6fe
E2E tests with miwi clusters
bitoku Nov 13, 2024
bbe0738
go fmt
mrWinston Feb 6, 2025
ba659a8
Update pkg/util/cluster/cluster.go
mrWinston Feb 6, 2025
06d87b7
remove unneccessary comments
mrWinston Feb 6, 2025
5074790
Update pkg/util/cluster/cluster.go
mrWinston Feb 10, 2025
9ebe2fc
rename
mrWinston Feb 10, 2025
17e8e46
Merge branch 'master' into poc/hack-miwi
mrWinston Feb 10, 2025
c08b52f
don't create an app when miwi is enabled
mrWinston Feb 10, 2025
e7acaca
remove early exit from checkAccessv2
mrWinston Feb 10, 2025
452fba5
comment fixing
mrWinston Feb 11, 2025
d918149
make sure create_miwi_env_file behaviour stays the same in dev env
mrWinston Feb 11, 2025
84f0863
uppercase env vars in cluster_config
mrWinston Feb 11, 2025
8d683fb
Merge branch 'master' into poc/hack-miwi
mrWinston Feb 14, 2025
9b84b07
fix formatting, see what renaming does to a pipe
mrWinston Feb 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion .pipelines/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,20 @@ jobs:
displayName: Enable regression tests in CI
condition: in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI')

- script: |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it easy for us to rename this pipeline/azp run target to something like /azp run e2e-wimi or something similar rather than /azp run e2e? We can handle it in a future PR as well, if that's easier.

This was raised as something that could cause confusion for contributors who want to re-run CI e2e as normal but end up kicking off a MIWI e2e instead.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm looking into that. currently i'm unsure how to rename a pipeline.

. secrets/env

# Retrieve the kubeconfig
hack/get-admin-aks-kubeconfig.sh > aks.kubeconfig

if [ -f aks.kubeconfig ]; then
echo "Kubeconfig retrieved successfully."
else
echo "Failed to retrieve Kubeconfig."
exit 1
fi
displayName: Get Kubeconfig for AKS Cluster

- script: |
az account set -s $AZURE_SUBSCRIPTION_ID
SECRET_SA_ACCOUNT_NAME=$(SECRET_SA_ACCOUNT_NAME) make secrets
Expand All @@ -66,12 +80,22 @@ jobs:
export CI=true
. ./hack/e2e/run-rp-and-e2e.sh
deploy_e2e_db

export SKIP_MIWI_ROLE_ASSIGNMENT=true
. ./hack/devtools/local_dev_env.sh
create_miwi_env_file
displayName: Setup (Azure)

- script: |
export CI=true
# Tell the E2E binary to run the MIMO tests
export ARO_E2E_MIMO=true
export OS_CLUSTER_VERSION=4.15.35
export USE_WI=true
export ARO_INSTALL_VIA_HIVE="true"
export ARO_ADOPT_BY_HIVE="true"
export HIVE_KUBE_CONFIG_PATH="$(realpath ./aks.kubeconfig)"

. ./env
. secrets/env
. ./hack/e2e/run-rp-and-e2e.sh

Expand All @@ -86,6 +110,8 @@ jobs:
run_selenium
validate_selenium_running

update_role_sets

mrWinston marked this conversation as resolved.
Show resolved Hide resolved
run_mimo_actuator
validate_mimo_actuator_running

Expand All @@ -94,6 +120,8 @@ jobs:

register_sub

# cluster is cleaned up in a later step. Keep it around for now to enable must-gather collection on error
export E2E_DELETE_CLUSTER=false
mrWinston marked this conversation as resolved.
Show resolved Hide resolved
make test-e2e -o e2e.test
displayName: Execute Tests
target: container
Expand Down
9 changes: 9 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ require (
github.com/prometheus/common v0.51.1
github.com/serge1peshcoff/selenium-go-conditions v0.0.0-20170824121757-5afbdb74596b
github.com/sirupsen/logrus v1.9.3
github.com/spf13/viper v1.20.0-alpha.6
github.com/stretchr/testify v1.10.0
github.com/tebeka/selenium v0.9.9
github.com/ugorji/go/codec v1.2.12
Expand Down Expand Up @@ -164,6 +165,7 @@ require (
github.com/go-openapi/validate v0.24.0 // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
github.com/go-viper/mapstructure/v2 v2.0.0 // indirect
github.com/godbus/dbus/v5 v5.1.1-0.20230522191255-76236955d466 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v5 v5.2.1 // indirect
Expand Down Expand Up @@ -222,6 +224,7 @@ require (
github.com/opencontainers/selinux v1.11.0 // indirect
github.com/openshift/custom-resource-status v1.1.3-0.20220503160415-f2fdb4999d87 // indirect
github.com/ostreedev/ostree-go v0.0.0-20210805093236-719684c64e4f // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/sftp v1.13.6 // indirect
Expand All @@ -232,14 +235,19 @@ require (
github.com/rivo/uniseg v0.4.7 // indirect
github.com/robfig/cron v1.2.0 // indirect
github.com/russross/blackfriday v1.6.0 // indirect
github.com/sagikazarmark/locafero v0.6.0 // indirect
github.com/secure-systems-lab/go-securesystemslib v0.8.0 // indirect
github.com/sigstore/fulcio v1.4.5 // indirect
github.com/sigstore/rekor v1.3.6 // indirect
github.com/sigstore/sigstore v1.8.3 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/spf13/cobra v1.8.0 // indirect
github.com/spf13/pflag v1.0.6-0.20210604193023-d5e0c0615ace // indirect
github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6 // indirect
github.com/stoewer/go-strcase v1.2.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/sylabs/sif/v2 v2.16.0 // indirect
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 // indirect
github.com/tchap/go-patricia/v2 v2.3.1 // indirect
Expand All @@ -257,6 +265,7 @@ require (
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
go.starlark.net v0.0.0-20220328144851-d1966c6b9fcd // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/mod v0.17.0 // indirect
golang.org/x/sys v0.29.0 // indirect
golang.org/x/term v0.28.0 // indirect
Expand Down
22 changes: 20 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,8 +228,8 @@ github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f h1:Wl78ApPPB2
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f/go.mod h1:OSYXu++VVOHnXeitef/D8n/6y4QV8uLHSFXX4NeXMGc=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/frankban/quicktest v1.14.5 h1:dfYrrRyLtiqT9GyKXgdh+k4inNeTvmGbuSgZ3lx3GhA=
github.com/frankban/quicktest v1.14.5/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
Expand Down Expand Up @@ -289,6 +289,8 @@ github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1v
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
github.com/go-test/deep v1.1.0 h1:WOcxcdHcvdgThNXjw0t76K42FXTU7HpNQWHpA2HHNlg=
github.com/go-test/deep v1.1.0/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/go-viper/mapstructure/v2 v2.0.0 h1:dhn8MZ1gZ0mzeodTG3jt5Vj/o87xZKuNAprG2mQfMfc=
github.com/go-viper/mapstructure/v2 v2.0.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/godbus/dbus/v5 v5.0.3/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/godbus/dbus/v5 v5.1.0/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/godbus/dbus/v5 v5.1.1-0.20230522191255-76236955d466 h1:sQspH8M4niEijh3PFscJRLDnkL547IeP7kpPe3uUhEg=
Expand Down Expand Up @@ -535,6 +537,8 @@ github.com/openshift/machine-config-operator v0.0.1-0.20230908201248-46b93e64dea
github.com/openshift/machine-config-operator v0.0.1-0.20230908201248-46b93e64dea6/go.mod h1:zXXyi68fZLTGiIamAIwETIJdts9FCdyi9OCpnxh0N84=
github.com/ostreedev/ostree-go v0.0.0-20210805093236-719684c64e4f h1:/UDgs8FGMqwnHagNDPGOlts35QkhAZ8by3DR7nMih7M=
github.com/ostreedev/ostree-go v0.0.0-20210805093236-719684c64e4f/go.mod h1:J6OG6YJVEWopen4avK3VNQSnALmmjvniMmni/YFYAwc=
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pin/tftp v2.1.0+incompatible/go.mod h1:xVpZOMCXTy+A5QMjEVN0Glwa1sUvaJhFXbr/aAxuxGY=
Expand Down Expand Up @@ -576,6 +580,8 @@ github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99
github.com/russross/blackfriday v1.6.0 h1:KqfZb0pUVN2lYqZUYRddxF4OR8ZMURnJIG5Y3VRLtww=
github.com/russross/blackfriday v1.6.0/go.mod h1:ti0ldHuxg49ri4ksnFxlkCfN+hvslNlmVHqNRXXJNAY=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.6.0 h1:ON7AQg37yzcRPU69mt7gwhFEBwxI6P9T4Qu3N51bwOk=
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a library we're already using?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added viper as a dependency to help managing the different envioronment variables as a config struct ( see ClusterConfig definition and viper usage in NewClusterConfigFromEnv)

All the addiitonal dependencies are pulled in as dependencies of viper itself.

github.com/sagikazarmark/locafero v0.6.0/go.mod h1:77OmuIc6VTraTXKXIs/uvUxKGUXjE1GbemJYHqdNjX0=
github.com/sebdah/goldie/v2 v2.5.3 h1:9ES/mNN+HNUbNWpVAlrzuZ7jE+Nrczbj8uFRjM7624Y=
github.com/sebdah/goldie/v2 v2.5.3/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
github.com/seccomp/libseccomp-golang v0.10.0 h1:aA4bp+/Zzi0BnWZ2F1wgNBs5gTpm+na2rWM6M9YjLpY=
Expand All @@ -595,12 +601,20 @@ github.com/sigstore/sigstore v1.8.3 h1:G7LVXqL+ekgYtYdksBks9B38dPoIsbscjQJX/MGWk
github.com/sigstore/sigstore v1.8.3/go.mod h1:mqbTEariiGA94cn6G3xnDiV6BD8eSLdL/eA7bvJ0fVs=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
github.com/spf13/afero v1.3.3/go.mod h1:5KUK8ByomD5Ti5Artl0RtHeI5pTF7MIDuXL3yY520V4=
github.com/spf13/afero v1.9.2/go.mod h1:iUV7ddyEEZPO5gA3zD4fJt6iStLlL+Lg4m2cihcDf8Y=
github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8=
github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY=
github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0=
github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0=
github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho=
github.com/spf13/pflag v1.0.6-0.20210604193023-d5e0c0615ace h1:9PNP1jnUjRhfmGMlkXHjYPishpcw4jpSt/V/xYY3FMA=
github.com/spf13/pflag v1.0.6-0.20210604193023-d5e0c0615ace/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.20.0-alpha.6 h1:f65Cr/+2qk4GfHC0xqT/isoupQppwN5+VLRztUGTDbY=
github.com/spf13/viper v1.20.0-alpha.6/go.mod h1:CGBZzv0c9fOUASm6rfus4wdeIjR/04NOLq1P4KRhX3k=
github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0=
github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6 h1:pnnLyeX7o/5aX8qUQ69P/mLojDqwda8hFOCBTmP/6hw=
github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6/go.mod h1:39R/xuhNgVhi+K0/zst4TLrJrVmbm6LVgl4A0+ZFS5M=
Expand All @@ -610,8 +624,12 @@ github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpE
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/sylabs/sif/v2 v2.16.0 h1:2eqaBaQQsn5DZTzm3QZm0HupZQEjNXfxRnCmtyCihEU=
github.com/sylabs/sif/v2 v2.16.0/go.mod h1:d5TxgD/mhMUU3kWLmZmWJQ99Wg0asaTP0bq3ezR1xpg=
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 h1:kdXcSzyDtseVEc4yCz2qF8ZrQvIDBJLl4S1c3GCXmoI=
Expand Down
45 changes: 5 additions & 40 deletions hack/cluster/cluster.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,9 @@ import (
"github.com/davecgh/go-spew/spew"
"github.com/sirupsen/logrus"

"github.com/Azure/ARO-RP/pkg/env"
"github.com/Azure/ARO-RP/pkg/util/cluster"
msgraph_errors "github.com/Azure/ARO-RP/pkg/util/graph/graphsdk/models/odataerrors"
utillog "github.com/Azure/ARO-RP/pkg/util/log"
"github.com/Azure/ARO-RP/pkg/util/version"
)

const (
Expand All @@ -28,55 +26,22 @@ func run(ctx context.Context, log *logrus.Entry) error {
return fmt.Errorf("usage: CLUSTER=x %s {create,delete}", os.Args[0])
}

if err := env.ValidateVars(Cluster); err != nil {
return err
}

env, err := env.NewCore(ctx, log, env.COMPONENT_TOOLING)
conf, err := cluster.NewClusterConfigFromEnv()
if err != nil {
return err
}

vnetResourceGroup := os.Getenv("RESOURCEGROUP") // TODO: remove this when we deploy and peer a vnet per cluster create
if os.Getenv("CI") != "" {
vnetResourceGroup = os.Getenv(Cluster)
}
clusterName := os.Getenv(Cluster)

osClusterVersion := os.Getenv("OS_CLUSTER_VERSION")
if osClusterVersion == "" {
osClusterVersion = version.DefaultInstallStream.Version.String()
log.Infof("using default cluster version %s", osClusterVersion)
} else {
log.Infof("using specified cluster version %s", osClusterVersion)
}

masterVmSize := os.Getenv("MASTER_VM_SIZE")
if masterVmSize == "" {
masterVmSize = cluster.DefaultMasterVmSize.String()
log.Infof("using default master VM size %s", masterVmSize)
} else {
log.Infof("using specified master VM size %s", masterVmSize)
}

workerVmSize := os.Getenv("WORKER_VM_SIZE")
if workerVmSize == "" {
workerVmSize = cluster.DefaultWorkerVmSize.String()
log.Infof("using default worker VM size %s", workerVmSize)
} else {
log.Infof("using specified worker VM size %s", workerVmSize)
}

c, err := cluster.New(log, env, os.Getenv("CI") != "")
c, err := cluster.New(log, conf)
if err != nil {
return err
}

spew.Dump(c.Config)
switch strings.ToLower(os.Args[1]) {
case "create":
return c.Create(ctx, vnetResourceGroup, clusterName, osClusterVersion, masterVmSize, workerVmSize)
return c.Create(ctx)
case "delete":
return c.Delete(ctx, vnetResourceGroup, clusterName)
return c.Delete(ctx, conf.VnetResourceGroup, conf.ClusterName)
default:
return fmt.Errorf("invalid command %s", os.Args[1])
}
Expand Down
Loading