Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: replace helm with go service templates #289

Merged
merged 107 commits into from
Jul 21, 2024
Merged
Show file tree
Hide file tree
Changes from 102 commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
d8a3358
feat: initial run at service-templates
shreddedbacon Dec 8, 2022
53addd9
feat: initial run at service-templates
shreddedbacon Dec 8, 2022
d6ac435
feat: add inpod cronjob detection
shreddedbacon Dec 8, 2022
0afa589
chore: add cronjob to lagoonyml
shreddedbacon Dec 8, 2022
14a3d45
refactor: add cronjob to service calculations
shreddedbacon Dec 8, 2022
8338f72
refactor: add cronjob to service calculations
shreddedbacon Dec 8, 2022
de30e41
refactor: use diff in test outputs to show changes, and support cronj…
shreddedbacon Dec 8, 2022
9717fda
chore: use same lessthan logic in cronjob check as old bash
shreddedbacon Dec 8, 2022
1c3cfd8
chore: shorten name of environment for tests
shreddedbacon Dec 8, 2022
3bb037d
fix: kind in templates
shreddedbacon Dec 8, 2022
0558aae
chore: add support for generating spot instance values
shreddedbacon Dec 8, 2022
ff3355f
chore: update tests for deployment with spot instances
shreddedbacon Dec 8, 2022
6a6feab
test: add test for lagoon.yml cronjob
shreddedbacon Dec 8, 2022
b946772
refactor: move polysite handling into lagoon unmarshal function
shreddedbacon Dec 8, 2022
2635558
test: add polysite unmarshal test
shreddedbacon Dec 8, 2022
2d124f2
chore: add more templating
shreddedbacon Dec 13, 2022
e66452a
chore: merge main
shreddedbacon Jun 27, 2023
cb3931f
chore: tidy up servicetypes a bit and add more default values
shreddedbacon Jul 21, 2023
a7e705d
chore: update templating engines some more
shreddedbacon Jul 21, 2023
7a98bd6
chore: remove unneeded helmchart templates
shreddedbacon Jul 21, 2023
a98ee4f
chore: update generator and helpers with additional supporting functi…
shreddedbacon Jul 21, 2023
0b41761
chore: add the command to run the template generation
shreddedbacon Jul 21, 2023
5323296
chore: update bash script to execute the templating command now and r…
shreddedbacon Jul 21, 2023
95bc35e
chore: add cronjob templating
shreddedbacon Jul 24, 2023
fdba50b
chore: remove old templating code from bash, use build-deploy-tool now
shreddedbacon Jul 24, 2023
f9a87cf
chore: first service should be named the same as the default service …
shreddedbacon Jul 24, 2023
1634ff7
chore: update ingress template to support additional services
shreddedbacon Jul 26, 2023
587d49e
chore: fix runasuser and fsgroup
shreddedbacon Jul 26, 2023
11db8e3
chore: remove unused options on routes
shreddedbacon Jul 26, 2023
3cc527e
chore: fix runasuser and fsgroup
shreddedbacon Jul 26, 2023
addfe98
chore: combine additional ports into one service object and fix tests
shreddedbacon Jul 26, 2023
608868e
chore: only create templates if there is anything to create
shreddedbacon Jul 27, 2023
66acfa4
chore: support configmapsha
shreddedbacon Jul 30, 2023
fd56129
chore: use protocol in service names instead for shorter names
shreddedbacon Jul 30, 2023
3e95f53
chore: export configmapsha in builds for generator to consume
shreddedbacon Jul 30, 2023
d373366
test: update cases
shreddedbacon Jul 30, 2023
d03d431
chore: merge main and fix conflicts
shreddedbacon Nov 9, 2023
504b20c
chore: minor fixes and backport imagecache injection
shreddedbacon Nov 10, 2023
b14e5a0
chore: add worker servicetype config
shreddedbacon Nov 13, 2023
9e1f5fd
chore: merge main fix conflict
shreddedbacon Nov 27, 2023
00644cb
feat: check duplicate cronjobs and return error if found
shreddedbacon Nov 28, 2023
076bc39
chore: merge main and fix conflicts and tests
shreddedbacon Jan 8, 2024
a7652bc
test: fix tests after merge
shreddedbacon Jan 8, 2024
ef3f3f9
refactor: split service templates down individually
shreddedbacon Jan 9, 2024
51a15c2
chore: remove ioutil
shreddedbacon Jan 9, 2024
9fe290b
refactor: add identify services command
shreddedbacon Jan 9, 2024
52f4ee5
chore: set paths for testing from root
shreddedbacon Jan 10, 2024
e5448e5
chore: set paths for testing from root
shreddedbacon Jan 10, 2024
8e9c5b5
chore: delete scripts due to merging in main bash
shreddedbacon Jan 12, 2024
8e38c5a
refactor: extend testdata for missing values
shreddedbacon Jan 12, 2024
64fa812
refactor: generator, add missing values and create image build data
shreddedbacon Jan 12, 2024
9e30a68
refactor: support additional lagoonyaml values
shreddedbacon Jan 12, 2024
ff23a15
chore: update test results with latest fixes
shreddedbacon Jan 12, 2024
364238e
chore: update test results with latest fixes
shreddedbacon Jan 12, 2024
5070fac
refactor: main commands to use refactoring done with generator
shreddedbacon Jan 12, 2024
c25ea64
chore: remove legacy helmcharts
shreddedbacon Jan 12, 2024
76199eb
refactor: additional refactoring to support additional features from …
shreddedbacon Jan 12, 2024
6ba6533
refactor: use build-deploy-tool for image build calculation and servi…
shreddedbacon Jan 12, 2024
7644928
test: fix test name
shreddedbacon Jan 12, 2024
47ee6f8
test: github actions sets CI variable, this needs to be disabled in t…
shreddedbacon Jan 15, 2024
3e31f77
chore: merge branch 'main' into service-templates
shreddedbacon Jan 24, 2024
59329ee
fix: add missing container information for other templates
shreddedbacon Jan 29, 2024
235c42a
fix: commands for identify, and task runner to use new lagoonyml loca…
shreddedbacon Jan 29, 2024
dc08f47
fix: jq defaults for new image builds
shreddedbacon Jan 29, 2024
eff8f57
fix: postgres backup variable names for user,pass, and db
shreddedbacon Jan 31, 2024
78e97fe
chore: remove unused backup-generation script
shreddedbacon Feb 12, 2024
48280ca
refactor: handle dynamic secret generation in generator
shreddedbacon Feb 12, 2024
522f648
refactor: consume container-registries from .lagoon.yml and provide t…
shreddedbacon Feb 13, 2024
6d22fbc
refactor: modify the bash so that any remaining calls to .lagoon.yml …
shreddedbacon Feb 13, 2024
11189bf
chore: merge main and fix conflicts
shreddedbacon Feb 13, 2024
86288a3
test: fix after merge conflicts
shreddedbacon Feb 13, 2024
7c8d358
fix: container registry login iterator, additional formatting
shreddedbacon Feb 14, 2024
23bb571
refactor: move container registry logins to login section
shreddedbacon Feb 14, 2024
b916434
refactor: reduce usage of set -+ x
shreddedbacon Feb 14, 2024
b50e6f8
chore: merge branch 'main' into service-templates
shreddedbacon Feb 14, 2024
e552506
chore: merge main, fix conflicts and update test reference paths
shreddedbacon Feb 29, 2024
35f12f9
chore: clean up unused files
shreddedbacon Mar 5, 2024
120e594
feat: add support for dynamic dbaas secret loading
shreddedbacon Mar 6, 2024
7f76570
chore: merge branch 'main' into service-templates
shreddedbacon Mar 8, 2024
3dae278
test: add services test for dbaas detection and templating
shreddedbacon Mar 8, 2024
03e33a3
chore: backport #310 container-registries refactoring
shreddedbacon Apr 29, 2024
805d29c
refactor: dont use imagecache if image appears to not be in dockerhub
shreddedbacon May 14, 2024
7abc185
chore: merge main fix conflicts
shreddedbacon May 14, 2024
65b5317
chore: merge main and fix conflicts, update redis backup cmd
shreddedbacon May 24, 2024
14534b9
chore: fix redis template test for backup cmd
shreddedbacon May 24, 2024
fb6924b
chore: remove labels from cronjobs that deployments use as selectors
shreddedbacon May 24, 2024
5c90dbb
chore: apply other templates individually
shreddedbacon May 29, 2024
27cc3e6
chore: merge main and fix conflicts
shreddedbacon Jun 3, 2024
194aabd
chore: merge main and fix conflicts
shreddedbacon Jun 20, 2024
e03b200
fix: template creation for dbaas and other fmt fixes
shreddedbacon Jun 20, 2024
693e226
fix: image reference checks
shreddedbacon Jun 21, 2024
8bca886
fix: long branch domain truncating hash
shreddedbacon Jun 25, 2024
bf7b8da
fix: network policy
shreddedbacon Jun 27, 2024
5371cd3
fix: imagepullpolicy, recreate strategy for redis, and empty seccontext
shreddedbacon Jun 27, 2024
5bd18c0
fix: add missing SERVICE_NAME variable to deployments
shreddedbacon Jun 27, 2024
edd966e
chore: backport #329
shreddedbacon Jul 3, 2024
7ea72ba
chore: merge main fix conflicts
shreddedbacon Jul 3, 2024
e0d8155
chore: merge main and fix conflicts
shreddedbacon Jul 14, 2024
333784d
fix: set enableservicelinks to be correct for -singles, adjust postgr…
shreddedbacon Jul 15, 2024
651d718
fix: opensearch/elasticsearch liveness and readiness probes
shreddedbacon Jul 15, 2024
1b69bf8
fix: redis liveness and memory
shreddedbacon Jul 15, 2024
ba199e0
fix: solr recreate strategy and memory
shreddedbacon Jul 15, 2024
b4a582a
fix: adjust other memory to match legacy helm values
shreddedbacon Jul 15, 2024
7dc6df4
fix: use http for default port names in services, other test data fixes
shreddedbacon Jul 15, 2024
70f2474
fix: isolation-network-policy labels
shreddedbacon Jul 15, 2024
3922f93
refactor: select lagoon core version if variable is present
shreddedbacon Jul 15, 2024
7e775fd
refactor: use build-deploy-tool labels on ingress, not helm
shreddedbacon Jul 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
.idea
test-resources/output
testoutput
1 change: 0 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,6 @@ WORKDIR /kubectl-build-deploy/git
COPY legacy/docker-entrypoint.sh /lagoon/entrypoints/100-docker-entrypoint.sh
COPY legacy/build-deploy.sh /kubectl-build-deploy/build-deploy.sh
COPY legacy/build-deploy-docker-compose.sh /kubectl-build-deploy/build-deploy-docker-compose.sh
COPY legacy/rootless.values.yaml /kubectl-build-deploy/rootless.values.yaml

COPY legacy/scripts /kubectl-build-deploy/scripts

Expand Down
4 changes: 0 additions & 4 deletions cmd/config_fastly.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,6 @@ import (
"github.com/uselagoon/build-deploy-tool/internal/lagoon"
)

var (
domainName string
)

var fastlyConfigGeneration = &cobra.Command{
Use: "fastly",
Aliases: []string{"f"},
Expand Down
1 change: 1 addition & 0 deletions cmd/config_fastly_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ func TestGenerateFastlyConfig(t *testing.T) {
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
helpers.UnsetEnvVars(nil) //unset variables before running tests
// set the environment variables from args
err := os.Setenv("LAGOON_FASTLY_NOCACHE_SERVICE_ID", tt.args.cacheNoCache)
if err != nil {
Expand Down
34 changes: 5 additions & 29 deletions cmd/identify_dbaas_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,30 +13,6 @@ import (

// these tests uses the same files as the dbaas templates
func TestIdentifyDBaaSConsumers(t *testing.T) {
type args struct {
name string
alertContact string
statusPageID string
projectName string
environmentName string
branch string
prNumber string
prHeadBranch string
prBaseBranch string
environmentType string
buildType string
activeEnvironment string
standbyEnvironment string
cacheNoCache string
serviceID string
secretPrefix string
projectVars string
envVars string
lagoonVersion string
lagoonYAML string
valuesFilePath string
templatePath string
}
tests := []struct {
name string
args testdata.TestData
Expand All @@ -52,7 +28,7 @@ func TestIdentifyDBaaSConsumers(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/complex/lagoon.yml",
LagoonYAML: "internal/testdata/complex/lagoon.yml",
}, true),
templatePath: "testdata/output",
want: []string{
Expand All @@ -66,7 +42,7 @@ func TestIdentifyDBaaSConsumers(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/complex/lagoon.yml",
LagoonYAML: "internal/testdata/complex/lagoon.yml",
ProjectVariables: []lagoon.EnvironmentVariable{
{Name: "LAGOON_SERVICE_TYPES", Value: "mariadb:mariadb-shared", Scope: "build"},
},
Expand All @@ -83,7 +59,7 @@ func TestIdentifyDBaaSConsumers(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/complex/lagoon.yml",
LagoonYAML: "internal/testdata/complex/lagoon.yml",
ProjectVariables: []lagoon.EnvironmentVariable{
{Name: "LAGOON_DBAAS_ENVIRONMENT_TYPES", Value: "mariadb:development2", Scope: "build"},
},
Expand All @@ -100,7 +76,7 @@ func TestIdentifyDBaaSConsumers(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/complex/lagoon.yml",
LagoonYAML: "internal/testdata/complex/lagoon.yml",
ProjectVariables: []lagoon.EnvironmentVariable{
{Name: "LAGOON_SERVICE_TYPES", Value: "mariadb:mariadb-shared", Scope: "build"},
},
Expand All @@ -117,7 +93,7 @@ func TestIdentifyDBaaSConsumers(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/complex/lagoon.multidb.yml",
LagoonYAML: "internal/testdata/complex/lagoon.multidb.yml",
}, true),
templatePath: "testdata/output",
want: []string{
Expand Down
2 changes: 1 addition & 1 deletion cmd/identify_feature.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ func IdentifyFeatureFlag(g generator.GeneratorInput, name string) (string, error
if forceFlagVar != "" {
return forceFlagVar, nil
}
featureFlagVar, _ := lagoon.GetLagoonVariable(fmt.Sprintf("%s%s", "LAGOON_FEATURE_FLAG_", name), []string{"build", "global"}, *lagoonBuild.LagoonEnvironmentVariables)
featureFlagVar, _ := lagoon.GetLagoonVariable(fmt.Sprintf("%s%s", "LAGOON_FEATURE_FLAG_", name), []string{"build", "global"}, lagoonBuild.BuildValues.EnvironmentVariables)
if featureFlagVar != nil {
return featureFlagVar.Value, nil
}
Expand Down
48 changes: 14 additions & 34 deletions cmd/identify_feature_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,33 +7,12 @@ import (
"github.com/uselagoon/build-deploy-tool/internal/helpers"
"github.com/uselagoon/build-deploy-tool/internal/lagoon"
"github.com/uselagoon/build-deploy-tool/internal/testdata"

// changes the testing to source from root so paths to test resources must be defined from repo root
_ "github.com/uselagoon/build-deploy-tool/internal/testing"
)

func TestIdentifyFeatureFlag(t *testing.T) {
type args struct {
name string
alertContact string
statusPageID string
projectName string
environmentName string
branch string
prNumber string
prHeadBranch string
prBaseBranch string
environmentType string
buildType string
activeEnvironment string
standbyEnvironment string
cacheNoCache string
serviceID string
secretPrefix string
projectVars string
envVars string
lagoonVersion string
lagoonYAML string
valuesFilePath string
templatePath string
}
tests := []struct {
name string
args testdata.TestData
Expand All @@ -51,7 +30,7 @@ func TestIdentifyFeatureFlag(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/node/lagoon.yml",
LagoonYAML: "internal/testdata/node/lagoon.yml",
ProjectVariables: []lagoon.EnvironmentVariable{
{
Name: "LAGOON_FEATURE_FLAG_ROOTLESS_WORKLOAD",
Expand All @@ -60,7 +39,7 @@ func TestIdentifyFeatureFlag(t *testing.T) {
},
},
}, true),
templatePath: "testdata/output",
templatePath: "testoutput",
want: "enabled",
},
{
Expand All @@ -71,7 +50,7 @@ func TestIdentifyFeatureFlag(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/node/lagoon.yml",
LagoonYAML: "internal/testdata/node/lagoon.yml",
EnvVariables: []lagoon.EnvironmentVariable{
{
Name: "LAGOON_FEATURE_FLAG_ROOTLESS_WORKLOAD",
Expand All @@ -80,7 +59,7 @@ func TestIdentifyFeatureFlag(t *testing.T) {
},
},
}, true),
templatePath: "testdata/output",
templatePath: "testoutput",
want: "enabled",
},
{
Expand All @@ -91,9 +70,9 @@ func TestIdentifyFeatureFlag(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/node/lagoon.yml",
LagoonYAML: "internal/testdata/node/lagoon.yml",
}, true),
templatePath: "testdata/output",
templatePath: "testoutput",
vars: []helpers.EnvironmentVariable{
{
Name: "LAGOON_FEATURE_FLAG_FORCE_ROOTLESS_WORKLOAD",
Expand All @@ -110,9 +89,9 @@ func TestIdentifyFeatureFlag(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/node/lagoon.yml",
LagoonYAML: "internal/testdata/node/lagoon.yml",
}, true),
templatePath: "testdata/output",
templatePath: "testoutput",
vars: []helpers.EnvironmentVariable{
{
Name: "LAGOON_FEATURE_FLAG_FORCE_ROOTLESS_WORKLOAD",
Expand All @@ -133,7 +112,7 @@ func TestIdentifyFeatureFlag(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/node/lagoon.yml",
LagoonYAML: "internal/testdata/node/lagoon.yml",
ProjectVariables: []lagoon.EnvironmentVariable{
{
Name: "LAGOON_FEATURE_FLAG_ROOTLESS_WORKLOAD",
Expand Down Expand Up @@ -162,7 +141,7 @@ func TestIdentifyFeatureFlag(t *testing.T) {
ProjectName: "example-project",
EnvironmentName: "main",
Branch: "main",
LagoonYAML: "../internal/testdata/node/lagoon.yml",
LagoonYAML: "internal/testdata/node/lagoon.yml",
ProjectVariables: []lagoon.EnvironmentVariable{
{
Name: "LAGOON_FEATURE_FLAG_ROOTLESS_WORKLOAD",
Expand All @@ -182,6 +161,7 @@ func TestIdentifyFeatureFlag(t *testing.T) {
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
helpers.UnsetEnvVars(tt.vars) //unset variables before running tests
// set the environment variables from args
savedTemplates := tt.templatePath
generator, err := testdata.SetupEnvironment(*rootCmd, savedTemplates, tt.args)
Expand Down
78 changes: 78 additions & 0 deletions cmd/identify_imagebuild.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
package cmd

import (
"encoding/json"
"fmt"

"github.com/spf13/cobra"
generator "github.com/uselagoon/build-deploy-tool/internal/generator"
)

var imageBuildIdentify = &cobra.Command{
Use: "image-builds",
Aliases: []string{"image-build", "img-build", "ib"},
Short: "Identify the configuration for building images for a Lagoon build",
RunE: func(cmd *cobra.Command, args []string) error {
gen, err := generator.GenerateInput(*rootCmd, false)
if err != nil {
return err
}
out, err := ImageBuildConfigurationIdentification(gen)
if err != nil {
return err
}
bc, err := json.Marshal(out)
if err != nil {
return err
}
fmt.Println(string(bc))
return nil
},
}

type imageBuild struct {
BuildKit bool `json:"buildKit"`
Images []imageBuilds `json:"images"`
BuildArguments map[string]string `json:"buildArguments"`
ContainerRegistries []generator.ContainerRegistry `json:"containerRegistries,omitempty"`
}

type imageBuilds struct {
Name string `json:"name"`
ImageBuild generator.ImageBuild `json:"imageBuild"`
}

// ImageBuildConfigurationIdentification takes the output of the generator and turns it into a JSON payload
// that can be used by the legacy bash to build container images. This payload contains the buildkit flag if it was as part of the build
// but it also provides all the container contexts and dockerfile paths that can be passed to the build command
// this includes the:
// * temporary image name (namespace-service)
// * the push image name (registry/project/environment/service:latest)
// * the pull through image, if there is no dockerfile
// * eventually other information like build args etc
func ImageBuildConfigurationIdentification(g generator.GeneratorInput) (imageBuild, error) {

lServices := imageBuild{}
lagoonBuild, err := generator.NewGenerator(
g,
)
if err != nil {
return lServices, err
}
lServices.BuildKit = lagoonBuild.BuildValues.DockerBuildKit
lServices.BuildArguments = lagoonBuild.BuildValues.ImageBuildArguments
for _, service := range lagoonBuild.BuildValues.Services {
if service.ImageBuild != nil {
lServices.Images = append(lServices.Images, imageBuilds{
Name: service.Name,
ImageBuild: *service.ImageBuild,
})
}
}
lServices.ContainerRegistries = lagoonBuild.BuildValues.ContainerRegistry
return lServices, nil
}

func init() {
identifyCmd.AddCommand(imageBuildIdentify)
}
Loading