Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add asset package creation #1085

Closed
wants to merge 36 commits into from
Closed
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
37038bc
initial create-asset-package implementation
dwillist Feb 19, 2021
7e14c8e
update command name, formatting changes
dwillist Feb 19, 2021
91c3f79
feedback updates
dwillist Feb 19, 2021
d66afe7
add parallel downloads
dwillist Feb 22, 2021
9aac982
add asset cache acceptance test
dwillist Feb 23, 2021
a697f8f
add windows testing and support
dwillist Feb 26, 2021
c3026b3
remove test focus
dwillist Feb 26, 2021
d9b34c9
refactor part 1
dwillist Feb 26, 2021
9e43ddd
refactoring
dwillist Mar 1, 2021
bd14cf1
rebase to current main
dwillist Mar 18, 2021
32046be
passing tests
dwillist Apr 8, 2021
95f7149
formatting
dwillist Apr 9, 2021
559f1b6
naming and formatting
dwillist Apr 9, 2021
de288b6
update windows tests
dwillist Apr 14, 2021
b74783b
update windows paths for tests
dwillist Apr 14, 2021
1ce7b7b
merge conflicts
dwillist Apr 14, 2021
8cd64ab
formatting
dwillist Apr 15, 2021
50392ce
add build integration test
dwillist Apr 15, 2021
faa3452
formatting update
dwillist Apr 15, 2021
0b7da6f
windows test updates
dwillist Apr 15, 2021
656bead
accceptance test using builder
dwillist Apr 16, 2021
69c8770
remove & test remaining panics
dwillist Apr 19, 2021
ab47504
TODO cleanup
dwillist Apr 19, 2021
5600f80
asset package renames
dwillist Apr 19, 2021
5f12f2d
image fetcher interface rename
dwillist Apr 19, 2021
a17d448
nameing & commenting updates
dwillist Apr 20, 2021
62b995e
format updates
dwillist Apr 20, 2021
336fd97
require buildpack api 0.8 to create assets packages
dwillist Apr 21, 2021
d628689
formatting updates
dwillist Apr 21, 2021
8084466
remove API checks, these will be added back in when next lifecycle is…
dwillist Apr 22, 2021
89afd7a
Merge branch 'main' into asset_cache
dwillist Apr 26, 2021
30d9fda
resolve merge conflicts
dwillist May 24, 2021
ced5d7a
wip acceptance refactoring
dwillist May 28, 2021
9208e55
refactor acceptance tests
dwillist May 28, 2021
46bba8f
merge updates
dwillist Jun 2, 2021
d341285
Merge branch 'main' into asset_cache
dfreilich Jun 15, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
548 changes: 548 additions & 0 deletions acceptance/acceptance_test.go

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions acceptance/buildpacks/archive_buildpack.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ const (
defaultMode = 0755
)

func NewArchiveBuildpack(name string) archiveBuildpack {
return archiveBuildpack{
name: name,
}
}

type archiveBuildpack struct {
name string
}
Expand Down
6 changes: 6 additions & 0 deletions acceptance/buildpacks/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ func WithBuildpackAPIVersion(apiVersion string) func(b *BuildpackManager) {
}
}

func WithBuildpackSource(source string) func(b *BuildpackManager) {
return func(b *BuildpackManager) {
b.sourceDir = source
}
}

func NewBuildpackManager(t *testing.T, assert testhelpers.AssertionManager, modifiers ...BuildpackManagerModifier) BuildpackManager {
m := BuildpackManager{
testObject: t,
Expand Down
4 changes: 4 additions & 0 deletions acceptance/invoke/pack.go
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ type Feature int
const (
InspectRemoteImage = iota
BuilderNoDuplicateLayers
AssetPackages
)

var featureTests = map[Feature]func(i *PackInvoker) bool{
Expand All @@ -229,6 +230,9 @@ var featureTests = map[Feature]func(i *PackInvoker) bool{
BuilderNoDuplicateLayers: func(i *PackInvoker) bool {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an aside, this can be taken out, but this probably isn't the place to do that.

return i.laterThan("0.18.0")
},
AssetPackages: func(i *PackInvoker) bool {
return i.laterThan("0.18.1")
},
}

func (i *PackInvoker) SupportsFeature(f Feature) bool {
Expand Down
19 changes: 19 additions & 0 deletions acceptance/invoke/pack_fixtures.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,25 @@ func (m PackFixtureManager) TemplateFixtureToFile(name string, destination *os.F
m.assert.Nil(err)
}

func (m PackFixtureManager) TemplateFile(file *os.File, data map[string]interface{}) {
m.testObject.Helper()

outputTemplate, err := ioutil.ReadAll(file)
m.assert.Nil(err)

_, err = file.Seek(0, 0)
m.assert.Nil(err)

err = file.Truncate(0)
m.assert.Nil(err)
Comment on lines +88 to +92
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do these do, together?

Copy link
Contributor Author

@dwillist dwillist May 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They delete all the files contents and reset the offset in the file to 0. In general the operations in the TemplateFile function are:

  1. read the file's contents into memory
  2. delete them
  3. re-write out the newly templated contents to the file.

Note this is only done on files that have been moved into a tmp directory for testing


str := m.fillTemplate(outputTemplate, data)
fmt.Printf("filled template for %s\n", file.Name())
fmt.Println(str)
_, err = io.WriteString(file, m.fillTemplate(outputTemplate, data))
m.assert.Nil(err)
}

func (m PackFixtureManager) fillTemplate(templateContents []byte, data map[string]interface{}) string {
tpl, err := template.New("").
Funcs(template.FuncMap{
Expand Down
1 change: 1 addition & 0 deletions acceptance/testdata/mock_assets/assetA.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A pretty cool asset Ayyyy.
1 change: 1 addition & 0 deletions acceptance/testdata/mock_assets/assetB.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Just another asset.
1 change: 1 addition & 0 deletions acceptance/testdata/mock_assets/assetC.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Just another-nother asset.
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
api = "0.8"

[buildpack]
id = "all-assets/asset-buildpack"
version = "all-assets-builapck-version"
name = "All Assets Buildpack Buildpack"

[[stacks]]
id = "pack.test.stack"

[[assets]]
id = "asset A"
sha256 = "797b3f6bf2b2c10a8299d51dfdbcfed329d3c133fdc7e695beddbe8f70b49da9"
name = "A Asset"
stacks = ["pack.test.stack"]
uri = "{{ .assetAURI }}"
version = "1.2.3"

[[assets]]
id = "asset C"
sha256 = "339b5181dac7c2d01cbbd6fff90f541f0350864f03a02f05faf852baa2826064"
name = "C Asset"
stacks = ["pack.test.stack"]
uri = "{{ .assetCURI }}"
version = "1.3.5"

[[assets]]
id = "asset B"
name = "B Asset"
sha256 = "61eea2ec4053ca25b9bd5d7bebaba48ee5398569aa1da5bc3541cbab1d09b86b"
stacks = ["pack.test.stack"]
uri = "{{ .assetBURI }}"
version = "4.5.6"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a line

Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
api = "0.2"

[buildpack]
id = "simple/nested"
version = "nested-version"
name = "Nested Asset Buildpack"

[[order]]
[[order.group]]
id = "simple/asset-buildpack"
version = "simple-asset-buildpack-version"
[[order.group]]
id = "second-simple/asset-buildpack"
version = "second-simple-asset-buildpack-version"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a line

Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
api = "0.2"

[buildpack]
id = "second-simple/asset-buildpack"
version = "second-simple-asset-buildpack-version"
name = "Second Simple Asset Buildpack Buildpack"

[[stacks]]
id = "pack.test.stack"


[[assets]]
id = "asset C"
sha256 = "339b5181dac7c2d01cbbd6fff90f541f0350864f03a02f05faf852baa2826064"
name = "C Asset"
stacks = ["pack.test.stack"]
uri = "{{ .assetCURI }}"
version = "1.3.5"

[[assets]]
id = "asset B repeat"
name = "B Asset Repeat"
sha256 = "61eea2ec4053ca25b9bd5d7bebaba48ee5398569aa1da5bc3541cbab1d09b86b"
stacks = ["pack.test.stack"]
uri = "{{ .assetBURI }}"
version = "4.5.6"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add a line

Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#!/usr/bin/env bash

echo "---> Build: Simple Asset Layers Buildpack"

set -o errexit
set -o nounset
set -o pipefail

launch_dir=$1

## makes a launch layer
echo "making launch layer"

# Add color line, to test for --no-color
echo "Color: Styled"

mkdir "$launch_dir/launch-layer"
echo "Launch Dep Contents" > "$launch_dir/launch-layer/launch-dep"
ln -snf "$launch_dir/launch-layer" launch-deps
echo "launch = true" > "$launch_dir/launch-layer.toml"

## checks if assetsA is available
if [[ -f "/cnb/assets/797b3f6bf2b2c10a8299d51dfdbcfed329d3c133fdc7e695beddbe8f70b49da9" ]]; then
echo "Asset A exists!"
cat "/cnb/assets/797b3f6bf2b2c10a8299d51dfdbcfed329d3c133fdc7e695beddbe8f70b49da9"
printf "\n"
fi

if [[ -f "/cnb/assets/61eea2ec4053ca25b9bd5d7bebaba48ee5398569aa1da5bc3541cbab1d09b86b" ]]; then
echo "Asset B exists!"
cat "/cnb/assets/61eea2ec4053ca25b9bd5d7bebaba48ee5398569aa1da5bc3541cbab1d09b86b"
printf "\n"
fi

## makes a cached launch layer
if [[ ! -f "$launch_dir/cached-launch-layer.toml" ]]; then
echo "making cached launch layer"
mkdir "$launch_dir/cached-launch-layer"
echo "Cached Dep Contents" > "$launch_dir/cached-launch-layer/cached-dep"
ln -snf "$launch_dir/cached-launch-layer" cached-deps
echo "launch = true" > "$launch_dir/cached-launch-layer.toml"
echo "cache = true" >> "$launch_dir/cached-launch-layer.toml"
else
echo "reusing cached launch layer"
ln -snf "$launch_dir/cached-launch-layer" cached-deps
fi

## adds a process
cat <<EOF > "$launch_dir/launch.toml"
[[processes]]
type = "web"
command = "./run"
args = ["8080"]

[[processes]]
type = "hello"
command = "echo"
args = ["hello", "world"]
direct = true
EOF

echo "---> Done"
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
@echo off
echo --- Build: Simple Asset Layers Buildpack

set launch_dir=%1

:: makes a launch layer
echo making launch layer %launch_dir%\launch-layer
mkdir %launch_dir%\launch-layer
echo Launch Dep Contents > "%launch_dir%\launch-layer\launch-dep
mklink /j launch-deps %launch_dir%\launch-layer
echo launch = true > %launch_dir%\launch-layer.toml

:: makes a cached launch layer
if not exist %launch_dir%\cached-launch-layer.toml (
echo making cached launch layer %launch_dir%\cached-launch-layer
mkdir %launch_dir%\cached-launch-layer
echo Cached Dep Contents > %launch_dir%\cached-launch-layer\cached-dep
mklink /j cached-deps %launch_dir%\cached-launch-layer
echo launch = true > %launch_dir%\cached-launch-layer.toml
echo cache = true >> %launch_dir%\cached-launch-layer.toml
) else (
echo reusing cached launch layer %launch_dir%\cached-launch-layer
mklink /j cached-deps %launch_dir%\cached-launch-layer
)

:: check if assets are locally available
if exist "/cnb/assets/797b3f6bf2b2c10a8299d51dfdbcfed329d3c133fdc7e695beddbe8f70b49da9" (
echo "Asset A exists!"
type \cnb\assets\797b3f6bf2b2c10a8299d51dfdbcfed329d3c133fdc7e695beddbe8f70b49da9
)

if exist "/cnb/assets/61eea2ec4053ca25b9bd5d7bebaba48ee5398569aa1da5bc3541cbab1d09b86b" (
echo "Asset B exists!"
type \cnb\assets\61eea2ec4053ca25b9bd5d7bebaba48ee5398569aa1da5bc3541cbab1d09b86b
)

:: adds a process
(
echo [[processes]]
echo type = "web"
echo command = '.\run'
echo args = ["8080"]
echo.
echo [[processes]]
echo type = "hello"
echo command = "cmd"
echo args = ["/c", "echo hello world"]
echo direct = true
) > %launch_dir%\launch.toml

echo --- Done
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env bash

## always detect
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
@echo off
:: always detect
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
api = "0.2"

[buildpack]
id = "simple/asset-buildpack"
version = "simple-asset-buildpack-version"
name = "Simple Asset Buildpack Buildpack"

[[stacks]]
id = "pack.test.stack"

[[assets]]
id = "asset A"
sha256 = "797b3f6bf2b2c10a8299d51dfdbcfed329d3c133fdc7e695beddbe8f70b49da9"
name = "A Asset"
stacks = ["pack.test.stack"]
uri = "{{ .assetAURI }}"
version = "1.2.3"

[[assets]]
id = "asset B"
name = "B Asset"
sha256 = "61eea2ec4053ca25b9bd5d7bebaba48ee5398569aa1da5bc3541cbab1d09b86b"
stacks = ["pack.test.stack"]
uri = "{{ .assetBURI }}"
version = "4.5.6"
24 changes: 24 additions & 0 deletions acceptance/testdata/pack_fixtures/builder_with_assets.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[[buildpacks]]
image = "{{.simple_buildpack_with_assets}}"

[lifecycle]
{{- if .lifecycle_uri}}
uri = "{{.lifecycle_uri}}"
{{- end}}
{{- if .lifecycle_version}}
version = "{{.lifecycle_version}}"
{{- end}}

[[order]]
[[order.group]]
id = "simple/asset-buildpack"

[stack]
id = "pack.test.stack"
build-image = "pack-test/build"
run-image = "pack-test/run"
run-image-mirrors = ["{{.run_image_mirror}}"]

[assets]
[[assets.package]]
image = "{{.asset_package_image_name}}"
5 changes: 5 additions & 0 deletions acceptance/testdata/pack_fixtures/generic_package.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
[buildpack]
uri = "{{ .buildpack_uri }}"

[platform]
os = "{{ .OS }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add a line

Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[buildpack]
uri = "nested-assets-buildpack.tgz"

[[dependencies]]
image = "{{.simple_buildpack}}"

[[dependencies]]
image = "{{.second_simple_buildpack}}"

[platform]
os = "{{ .OS }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add a line

Loading