Some Lagoon features can be controlled by setting feature flags. +This is designed to assist users and administrators to roll out new platform features in a controlled manner.
+The following environment variables can be set on an environment or project to toggle feature flags.
+| Environment Variable Name | +Active scope | +Version introduced | +Version removed | +Default Value | +Description | +
|---|---|---|---|---|---|
LAGOON_FEATURE_FLAG_ROOTLESS_WORKLOAD |
+global |
+2.2.0 | +- | +disabled |
+Set to enabled to set a non-root pod security context on the pods in this environment or project.This flag will eventually be deprecated, at which point non-root workloads will be enforced. |
+
LAGOON_FEATURE_FLAG_ISOLATION_NETWORK_POLICY |
+global |
+2.2.0 | +- | +disabled |
+Set to enabled to add a default namespace isolation network policy to each environment on deployment.This flag will eventually be deprecated, at which point the namespace isolation network policy will be enforced. NOTE: enabling and then disabling this feature will not remove any existing network policy from previous deployments. Those must be removed manually. |
+
Feature flags may also be controlled at the cluster level. There is support for this in the lagoon-build-deploy chart.
+For each feature flag there are two flavours of values which can be set: default and force.
default controls the default policy for environments deployed to the cluster, but can be overridden at the project or environment level by the environment variables documented above.force also controls the policy for environments deployed to the cluster, but cannot be overridden by the environment variables documented above.Direct API interactions in Lagoon are done via GraphQL.
+In order to authenticate with the API, we need a JWT (JSON Web Token) that allows us to use the GraphQL API as admin. To generate this token, open the terminal of the storage-calculator pod via your Kubernetes UI, or via kubectl and run the following command:
This will return a long string which is the JWT token. Make a note of this, as we will need it to send queries.
+We also need the URL of the API endpoint, which can be found under "Ingresses" in your Kubernetes UI or via kubectl on the command line. Make a note of this endpoint URL, which we will also need.
+To compose and send GraphQL queries, we recommend GraphiQL.app, a desktop GraphQL client with features such as autocomplete. To continue with the next steps, install and start the app.
+Under "GraphQL Endpoint", enter the API endpoint URL with /graphql on the end. Then click on "Edit HTTP Headers" and add a new header:
AuthorizationBearer [JWT token] (make sure that the JWT token has no spaces, as this would not work)Press ESC to close the HTTP header overlay and now we are ready to send the first GraphQL request!
+
Enter this in the left panel
+ +
And press the ▶️ button (or press CTRL+ENTER).
+If all went well, your first GraphQL response should appear shortly afterwards in the right pane.
+Let's create the first project for Lagoon to deploy! For this we'll use the queries from the GraphQL query template in create-project.gql.
For each of the queries (the blocks starting with mutation {), fill in all of the empty fields marked by TODO comments and run the queries in GraphiQL.app. This will create one of each of the following two objects:
kubernetes : The Kubernetes (or Openshift) cluster to which Lagoon should deploy. Lagoon is not only capable of deploying to its own Kubernetes cluster, but also to any Kubernetes cluster anywhere in the world.project : The Lagoon project to be deployed, which is a Git repository with a .lagoon.yml configuration file committed in the root.In Lagoon, each developer authenticates via their SSH key(s). This determines their access to:
+To allow access to the project, we first need to add a new group to the API:
+mutation {
+ addGroup (
+ input: {
+ # TODO: Enter the name for your new group.
+ name: ""
+ }
+ ) {
+ id
+ name
+ }
+}
+Then we need to add a new user to the API:
+mutation {
+ addUser(
+ input: {
+ email: "michael.schmid@example.com"
+ firstName: "Michael"
+ lastName: "Schmid"
+ comment: "CTO"
+ }
+ ) {
+ # TODO: Make a note of the user ID that is returned.
+ id
+ }
+}
+Then we can add an SSH public key for the user to the API:
+mutation {
+ addSshKey(
+ input: {
+ # TODO: Fill in the name field.
+ # This is a non-unique identifier for the SSH key.
+ name: ""
+ # TODO: Fill in the keyValue field.
+ # This is the actual SSH public key (without the type at the beginning and without the comment at the end, ex. `AAAAB3NzaC1yc2EAAAADAQ...3QjzIOtdQERGZuMsi0p`).
+ keyValue: ""
+ # TODO: Fill in the keyType field.
+ # Valid values are either SSH_RSA, SSH_ED25519, ECDSA_SHA2_NISTP256/384/521
+ keyType: SSH_RSA
+ user: {
+ # TODO: Fill in the userId field.
+ # This is the user ID that we noted from the addUser query.
+ id:"0",
+ email:"michael.schmid@example.com"
+ }
+ }
+ ) {
+ id
+ }
+}
+After we add the key, we need to add the user to a group:
+mutation {
+ addUserToGroup (
+ input: {
+ user: {
+ #TODO: Enter the email address of the user.
+ email: ""
+ }
+ group: {
+ #TODO: Enter the name of the group you want to add the user to.
+ name: ""
+ }
+ #TODO: Enter the role of the user.
+ role: OWNER
+
+ }
+ ) {
+ id
+ name
+ }
+}
+After running one or more of these kinds of queries, the user will be granted access to create tokens via SSH, access containers and more.
+If you want to know what is going on during a deployment, we suggest configuring notifications for your project, which provide:
+As notifications can be quite different in terms of the information they need, each notification type has its own mutation.
+As with users, we first add the notification:
+mutation {
+ addNotificationSlack(
+ input: {
+ # TODO: Fill in the name field.
+ # This is your own identifier for the notification.
+ name: ""
+ # TODO: Fill in the channel field.
+ # This is the channel for the message to be sent to.
+ channel: ""
+ # TODO: Fill in the webhook field.
+ # This is the URL of the webhook where messages should be sent, this is usually provided by the chat system to you.
+ webhook: ""
+ }
+ ) {
+ id
+ }
+}
+After the notification is created, we can now assign it to our project:
+mutation {
+ addNotificationToProject(
+ input: {
+ notificationType: SLACK
+ # TODO: Fill in the project field.
+ # This is the project name.
+ project: ""
+ # TODO: Fill in the notification field.
+ # This is the notification name.
+ notificationName: ""
+ # TODO: OPTIONAL
+ # The kind notification class you're interested in defaults to DEPLOYMENT
+ contentType: DEPLOYMENT/PROBLEM
+ # TODO: OPTIONAL
+ # Related to contentType PROBLEM, we can set the threshold for the kinds of problems
+ # we'd like to be notified about
+ notificationSeverityThreshold "NONE/UNKNOWN/NEGLIGIBLE/LOW/MEDIUM/HIGH/CRITICAL
+ }
+ ) {
+ id
+ }
+}
+Now for every deployment you will receive messages in your defined channel.
+Note
+In Lagoon, both addKubernetes and addOpenshift can be used for both Kubernetes and OpenShift targets - either will work interchangeably.
The cluster to which Lagoon should deploy.
+mutation {
+ addKubernetes(
+ input: {
+ # TODO: Fill in the name field.
+ # This is the unique identifier of the cluster.
+ name: ""
+ # TODO: Fill in consoleUrl field.
+ # This is the URL of the Kubernetes cluster
+ consoleUrl: ""
+ # TODO: Fill in the token field.
+ # This is the token of the `lagoon` service account created in this cluster (this is the same token that we also used during installation of Lagoon).
+ token: ""
+ }
+ ) {
+ name
+ id
+ }
+}
+This query will add a group to a project. Users of that group will be able to access the project. They will be able to make changes, based on their role in that group.
+mutation {
+ addGroupsToProject (
+ input: {
+ project: {
+ #TODO: Enter the name of the project.
+ name: ""
+ }
+ groups: {
+ #TODO: Enter the name of the group that will be added to the project.
+ name: ""
+ }
+ }
+ ) {
+ id
+ }
+}
+This query adds a new Lagoon project to be deployed, which is a Git repository with a .lagoon.yml configuration file committed in the root.
If you omit the privateKey field, a new SSH key for the project will be generated automatically.
If you would like to reuse a key from another project. you will need to supply the key in the addProject mutation.
mutation {
+ addProject(
+ input: {
+ # TODO: Fill in the name field.
+ # This is the project name.
+ name: ""
+ # TODO: Fill in the private key field (replace newlines with '\n').
+ # This is the private key for a project, which is used to access the Git code.
+ privateKey: ""
+ # TODO: Fill in the Kubernetes field.
+ # This is the ID of the Kubernetes or OpenShift to assign to the project.
+ kubernetes: 0
+ # TODO: Fill in the name field.
+ # This is the project name.
+ gitUrl: ""
+ # TODO: Fill in the branches to be deployed.
+ branches: ""
+ # TODO: Define the production environment.
+ productionEnvironment: ""
+ }
+ ) {
+ name
+ kubernetes {
+ name
+ id
+ }
+ gitUrl
+ activeSystemsDeploy
+ activeSystemsRemove
+ branches
+ pullrequests
+ }
+}
+This is a good query to see an overview of all projects, clusters and groups that exist within our Lagoon.
+query {
+ allProjects {
+ name
+ gitUrl
+ }
+ allKubernetes {
+ name
+ id
+ }
+ allGroups{
+ id
+ name
+ members {
+ # This will display the users in this group.
+ user {
+ id
+ firstName
+ lastName
+ }
+ role
+ }
+ groups {
+ id
+ name
+ }
+ }
+}
+If you want a detailed look at a single project, this query has been proven quite good:
+query {
+ projectByName(
+ # TODO: Fill in the project name.
+ name: ""
+ ) {
+ id
+ branches
+ gitUrl
+ pullrequests
+ productionEnvironment
+ notifications(type: SLACK) {
+ ... on NotificationSlack {
+ name
+ channel
+ webhook
+ id
+ }
+ }
+ environments {
+ name
+ deployType
+ environmentType
+ }
+ kubernetes {
+ id
+ }
+ }
+}
+Don't remember the name of a project, but know the Git URL? Search no longer, there is a GraphQL query for that:
+query {
+ projectByGitUrl(gitUrl: "git@server.com:org/repo.git") {
+ name
+ }
+}
+The Lagoon GraphQL API can not only display objects and create objects, it also has the capability to update existing objects, using a patch object.
+Update the branches to deploy within a project:
+mutation {
+ updateProject(
+ input: { id: 109, patch: { branches: "^(prod|stage|dev|update)$" } }
+ ) {
+ id
+ }
+}
+Update the production environment within a project:
+Warning
+This requires a redeploy in order for the changes to be reflected in the containers.
+ mutation {
+ updateProject(
+ input: { id: 109, patch: { productionEnvironment: "main" } }
+ ) {
+ id
+ }
+}
+You can also combine multiple changes at once:
+mutation {
+ updateProject(
+ input: {
+ id: 109
+ patch: {
+ productionEnvironment: "main"
+ branches: "^(prod|stage|dev|update)$"
+ }
+ }
+ ) {
+ id
+ }
+}
+You can also use the Lagoon GraphQL API to delete an environment. You'll need to know the project name and the environment name in order to run the command.
+mutation {
+ deleteEnvironment(
+ input: {
+ # TODO: Fill in the name field.
+ # This is the environment name.
+ name:""
+ # TODO: Fill in the project field.
+ # This is the project name.
+ project:""
+ execute:true
+ }
+ )
+}
+Want to see what groups and users have access to a project? Want to know what their roles are? Do I have a query for you! Using the query below you can search for a project and display the groups, users, and roles that are assigned to that project.
+query search{
+ projectByName(
+ #TODO: Enter the name of the project.
+ name: ""
+ ) {
+ id,
+ branches,
+ productionEnvironment,
+ pullrequests,
+ gitUrl,
+ kubernetes {
+ id
+ },
+ groups{
+ id
+ name
+ groups {
+ id
+ name
+ }
+ members {
+ role
+ user {
+ id
+ email
+ }
+ }
+ }
+ }
+}
+Project metadata can be assigned using arbitrary key/value pairs. Projects can then be queried by the associated metadata; for example you may categorize projects by type of software, version number, or any other categorization you may wish to query on later.
+Updates to metadata expect a key/value pair. It operates as an UPSERT, meaning if a key already exists the value will be updated, otherwise inserted.
You may have any number of k/v pairs stored against a project.
+mutation {
+ updateProjectMetadata(
+ input: { id: 1, patch: { key: "type", value: "saas" } }
+ ) {
+ id
+ metadata
+ }
+}
+Queries may be by key only (e.g return all projects where a specific key exists) or both key and value where both key and value must match.
All projects that have the version tag:
query projectsByMetadata {
+ projectsByMetadata(metadata: [{key: "version"] ) {
+ id
+ name
+ }
+}
+All projects that have the version tag, specifically version 8:
query projectsByMetadata {
+ projectsByMetadata(metadata: [{key: "version", value: "8"] ) {
+ id
+ name
+ }
+}
+Metadata can be removed on a per-key basis. Other metadata key/value pairs will persist.
+ + + + + + + + + +Version 1.0 of Lagoon changed how you access your projects! Access to your project is handled via groups, with projects assigned to one or multiple groups. Users are added to groups with a role. Groups can also be nested within sub-groups. This change provides a lot more flexibility and the possibility to recreate real world teams within Lagoon.
+When assigning a user to a group, you need to provide a group role for that user inside this group. Each one of the 5 current existing group roles gives the user different permissions to the group and projects assigned to the group. Here are the platform-wide roles and the group roles that are currently found in Lagoon:
+The platform-wide admin has access to everything across all of Lagoon. That includes dangerous mutations like deleting all projects. Use very, very, very carefully.
+The platform-wide owner has access to every Lagoon group, like the group owner role, and can be used if you need a user that needs access to everything but you don't want to assign the user to every group.
+The owner role can do everything within a group and its associated projects. They can add and manage users of a group. Be careful with this role, as it can delete projects and production environments!
+The maintainer role can do everything within a group and its associated projects except deleting the project itself or the production environment. They can add and manage users of a group.
+The developer role has SSH access only to development environments. This role cannot access, update or delete the production environment. They can run a sync task with the production environment as a source, but not as the destination. Cannot manage users of a group.
+IMPORTANT
+This role does not prevent the deployment of the production environment as a deployment is triggered via a Git push! You need to make sure that your Git server prevents these users from pushing into the branch defined as production environment.
+The reporter role has view access only. They cannot access any environments via SSH or make modifications to them. They can run cache-clear tasks. This role is mostly used for stakeholders to have access to Lagoon UI and logging.
+The guest role has the same privileges as the reporter role listed above.
+Here is a table that lists the roles and the access they have:
+| Name | +Resource | +Scope | +Attributes | +
|---|---|---|---|
| addSshKey | +ssh_key | +add | +userID | +
| updateSshKey | +ssh_key | +update | +userID | +
| deleteSshKey | +ssh_key | +delete | +userID | +
| getUserSshKeys | +ssh_key | +view:user | +userID | +
| updateUser | +user | +update | +userID | +
| deleteUser | +user | +delete | +userID | +
| Name | +Resource | +Scope | +Attributes | +
|---|---|---|---|
| getBackupsByEnvironmentId | +deployment | +view | +projectID | +
| getEnvironmentsByProjectId | +environment | +view | +projectID | +
| getEnvironmentServicesByEnvironmentId | +environment | +view | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:view | +projectID | +
| addGroup | +group | +add | ++ |
| getOpenshiftByProjectId | +openshift | +view | +projectID | +
| addProject | +project | +add | ++ |
| getProjectByEnvironmentId | +project | +view | +projectID | +
| getProjectByGitUrl | +project | +view | +projectID | +
| getProjectByName | +project | +view | +projectID | +
| addRestore | +restore | +add | +projectID | +
| updateRestore | +restore | +update | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:development | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:production | +projectID | +
| taskDrushCron | +task | +drushCron:development | +projectID | +
| taskDrushCron | +task | +drushCron:production | +projectID | +
| getFilesByTaskId | +task | +view | +projectID | +
| getTasksByEnvironmentId | +task | +view | +projectID | +
| getTaskByRemoteId | +task | +view | +projectID | +
| getTaskById | +task | +view | +projectID | +
| addUser | +user | +add | ++ |
| Name | +Resource | +Scope | +Attributes | +
|---|---|---|---|
| addBackup | +backup | +add | +projectID | +
| getBackupsByEnvironmentId | +backup | +view | +projectID | +
| addEnvVariable (to Environment) | +env_var | +environment:add:development | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:development | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:development | +projectID | +
| updateEnvironment | +environment | +update:development | +projectID | +
| deleteEnvironment | +environment | +delete:development | +projectID | +
| addDeployment | +environment | +deploy:development | +projectID | +
| setEnvironmentServices | +environment | +update:development | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:development | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:development | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:development | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:development | +projectID | +
| userCanSshToEnvironment | +environment | +ssh:development | +projectID | +
| getNotificationsByProjectId | +notification | +view | +projectID | +
| addTask | +task | +add:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:production | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:development | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:development | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:development | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:production | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:development | +projectID | +
| deleteTask | +task | +delete | +projectID | +
| updateTask | +task | +update | +projectID | +
| uploadFilesForTask | +task | +update | +projectID | +
| deleteFilesForTask | +task | +delete | +projectID | +
| getBackupsByEnvironmentId | +deployment | +view | +projectID | +
| getEnvironmentsByProjectId | +environment | +view | +projectID | +
| getEnvironmentServicesBy EnvironmentId |
+environment | +view | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:view | +projectID | +
| addGroup | +group | +add | ++ |
| getOpenshiftByProjectId | +openshift | +view | +projectID | +
| addProject | +project | +add | ++ |
| getProjectByEnvironmentId | +project | +view | +projectID | +
| getProjectByGitUrl | +project | +view | +projectID | +
| getProjectByName | +project | +view | +projectID | +
| addRestore | +restore | +add | +projectID | +
| updateRestore | +restore | +update | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:development | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:production | +projectID | +
| taskDrushCron | +task | +drushCron:development | +projectID | +
| taskDrushCron | +task | +drushCron:production | +projectID | +
| getFilesByTaskId | +task | +view | +projectID | +
| getTasksByEnvironmentId | +task | +view | +projectID | +
| getTaskByRemoteId | +task | +view | +projectID | +
| getTaskById | +task | +view | +projectID | +
| addUser | +user | +add | ++ |
| Name | +Resource | +Scope | +Attributes | +
|---|---|---|---|
| deleteBackup | +backup | +delete | +projectID | +
| addEnvVariable (to Project) | +env_var | +project:add | +projectID | +
| addEnvVariable (to Environment) | +env_var | +environment:add:production | +projectID | +
| deleteEnvVariable | +env_var | +delete | +projectID | +
| deleteEnvVariable (from Project) | +env_var | +project:delete | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:viewValue | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:production | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:production | +projectID | +
| updateEnvironment | +environment | +update:production | +projectID | +
| addDeployment | +environment | +deploy:production | +projectID | +
| deleteDeployment | +deployment | +delete | +projectID | +
| updateDeployment | +deployment | +update | +projectID | +
| setEnvironmentServices | +environment | +update:production | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:production | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:production | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:production | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:production | +projectID | +
| userCanSshToEnvironment | +environment | +ssh:production | +projectID | +
| updateGroup | +group | +update | +groupID | +
| deleteGroup | +group | +delete | +groupID | +
| addUserToGroup | +group | +addUser | +groupID | +
| removeUserFromGroup | +group | +removeUser | +groupID | +
| addNotificationToProject | +project | +addNotification | +projectID | +
| removeNotificationFromProject | +project | +removeNotification | +projectID | +
| updateProject | +project | +update | +projectID | +
| addGroupsToProject | +project | +addGroup | +projectID | +
| removeGroupsFromProject | +project | +removeGroup | +projectID | +
| addTask | +task | +add:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:production | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:production | +projectID | +
| addBackup | +backup | +add | +projectID | +
| getBackupsByEnvironmentId | +backup | +view | +projectID | +
| addEnvVariable (to Environment) | +env_var | +environment:add:development | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:development | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:development | +projectID | +
| updateEnvironment | +environment | +update:development | +projectID | +
| deleteEnvironment | +environment | +delete:development | +projectID | +
| addDeployment | +environment | +deploy:development | +projectID | +
| setEnvironmentServices | +environment | +update:development | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:development | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:development | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:development | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:development | +projectID | +
| getNotificationsByProjectId | +notification | +view | +projectID | +
| addTask | +task | +add:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:production | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:development | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:development | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:development | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:production | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:development | +projectID | +
| deleteTask | +task | +delete | +projectID | +
| updateTask | +task | +update | +projectID | +
| uploadFilesForTask | +task | +update | +projectID | +
| deleteFilesForTask | +task | +delete | +projectID | +
| getBackupsByEnvironmentId | +deployment | +view | +projectID | +
| getEnvironmentsByProjectId | +environment | +view | +projectID | +
| getEnvironmentServicesBy EnvironmentId |
+environment | +view | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:view | +projectID | +
| addGroup | +group | +add | ++ |
| getOpenshiftByProjectId | +openshift | +view | +projectID | +
| addProject | +project | +add | ++ |
| getProjectByEnvironmentId | +project | +view | +projectID | +
| getProjectByGitUrl | +project | +view | +projectID | +
| getProjectByName | +project | +view | +projectID | +
| addRestore | +restore | +add | +projectID | +
| updateRestore | +restore | +update | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:development | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:production | +projectID | +
| taskDrushCron | +task | +drushCron:development | +projectID | +
| taskDrushCron | +task | +drushCron:production | +projectID | +
| getFilesByTaskId | +task | +view | +projectID | +
| getTasksByEnvironmentId | +task | +view | +projectID | +
| getTaskByRemoteId | +task | +view | +projectID | +
| getTaskById | +task | +view | +projectID | +
| addUser | +user | +add | ++ |
| Name | +Resource | +Scope | +Attributes | +
|---|---|---|---|
| deleteEnvironment | +environment | +delete:production | +projectID | +
| deleteProject | +project | +delete | +projectID | +
| getProjectByEnvironmentId | +project | +viewPrivateKey | +projectID | +
| getProjectByGitUrl | +project | +viewPrivateKey | +projectID | +
| getProjectByName | +project | +viewPrivateKey | +projectID | +
| deleteBackup | +backup | +delete | +projectID | +
| addEnvVariable (to Project) | +env_var | +project:add | +projectID | +
| addEnvVariable (to Environment) | +env_var | +environment:add:production | +projectID | +
| deleteEnvVariable | +env_var | +delete | +projectID | +
| deleteEnvVariable (from Project) | +env_var | +project:delete | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:viewValue | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:production | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:production | +projectID | +
| updateEnvironment | +environment | +update:production | +projectID | +
| addDeployment | +environment | +deploy:production | +projectID | +
| deleteDeployment | +deployment | +delete | +projectID | +
| updateDeployment | +deployment | +update | +projectID | +
| setEnvironmentServices | +environment | +update:production | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:production | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:production | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:production | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:production | +projectID | +
| updateGroup | +group | +update | +groupID | +
| deleteGroup | +group | +delete | +groupID | +
| addUserToGroup | +group | +addUser | +groupID | +
| removeUserFromGroup | +group | +removeUser | +groupID | +
| addNotificationToProject | +project | +addNotification | +projectID | +
| removeNotificationFromProject | +project | +removeNotification | +projectID | +
| updateProject | +project | +update | +projectID | +
| addGroupsToProject | +project | +addGroup | +projectID | +
| removeGroupsFromProject | +project | +removeGroup | +projectID | +
| addTask | +task | +add:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:production | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:production | +projectID | +
| addBackup | +backup | +add | +projectID | +
| getBackupsByEnvironmentId | +backup | +view | +projectID | +
| addEnvVariable (to Environment) | +env_var | +environment:add:development | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:development | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:development | +projectID | +
| updateEnvironment | +environment | +update:development | +projectID | +
| deleteEnvironment | +environment | +delete:development | +projectID | +
| addDeployment | +environment | +deploy:development | +projectID | +
| setEnvironmentServices | +environment | +update:development | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:development | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:development | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:development | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:development | +projectID | +
| getNotificationsByProjectId | +notification | +view | +projectID | +
| addTask | +task | +add:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:production | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:development | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:development | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:development | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:production | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:development | +projectID | +
| deleteTask | +task | +delete | +projectID | +
| updateTask | +task | +update | +projectID | +
| uploadFilesForTask | +task | +update | +projectID | +
| deleteFilesForTask | +task | +delete | +projectID | +
| getBackupsByEnvironmentId | +deployment | +view | +projectID | +
| getEnvironmentsByProjectId | +environment | +view | +projectID | +
| getEnvironmentServices ByEnvironmentId |
+environment | +view | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:view | +projectID | +
| addGroup | +group | +add | ++ |
| getOpenshiftByProjectId | +openshift | +view | +projectID | +
| addProject | +project | +add | ++ |
| getProjectByEnvironmentId | +project | +view | +projectID | +
| getProjectByGitUrl | +project | +view | +projectID | +
| getProjectByName | +project | +view | +projectID | +
| addRestore | +restore | +add | +projectID | +
| updateRestore | +restore | +update | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:development | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:production | +projectID | +
| taskDrushCron | +task | +drushCron:development | +projectID | +
| taskDrushCron | +task | +drushCron:production | +projectID | +
| getFilesByTaskId | +task | +view | +projectID | +
| getTasksByEnvironmentId | +task | +view | +projectID | +
| getTaskByRemoteId | +task | +view | +projectID | +
| getTaskById | +task | +view | +projectID | +
| addUser | +user | +add | ++ |
| Name | +Resource | +Scope | +Attributes | +
|---|---|---|---|
| addOrUpdateEnvironment Storage |
+environment | +storage | ++ |
| addNotificationSlack | +notification | +add | ++ |
| updateNotificationSlack | +notification | +update | ++ |
| deleteNotificationSlack | +notification | +delete | ++ |
| addKubernetes | +kubernetes | +add | ++ |
| updateKubernetes | +kubernetes | +update | ++ |
| deleteKubernetes | +kubernetes | +delete | ++ |
| deleteAllKubernetes | +kubernetes | +deleteAll | ++ |
| getAllOpenshifts | +openshift | +viewAll | ++ |
| getAllProjects | +project | +viewAll | ++ |
| addSshKey | +ssh_key | +add | +userID | +
| updateSshKey | +ssh_key | +update | +userID | +
| deleteSshKey | +ssh_key | +delete | +userID | +
| getUserSshKeys | +ssh_key | +view:user | +userID | +
| updateUser | +user | +update | +userID | +
| deleteUser | +user | +delete | +userID | +
| deleteEnvironment | +environment | +delete:production | +projectID | +
| deleteProject | +project | +delete | +projectID | +
| getProjectByEnvironmentId | +project | +viewPrivateKey | +projectID | +
| getProjectByGitUrl | +project | +viewPrivateKey | +projectID | +
| getProjectByName | +project | +viewPrivateKey | +projectID | +
| deleteBackup | +backup | +delete | +projectID | +
| addEnvVariable (to Project) | +env_var | +project:add | +projectID | +
| addEnvVariable (to Environment) | +env_var | +environment:add:production | +projectID | +
| deleteEnvVariable | +env_var | +delete | +projectID | +
| deleteEnvVariable (from Project) | +env_var | +project:delete | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:viewValue | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:production | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:production | +projectID | +
| updateEnvironment | +environment | +update:production | +projectID | +
| allEnvironments | +environment | +viewAll | ++ |
| getEnvironmentStorageMonthBy EnvironmentId |
+environment | +storage | ++ |
| getEnvironmentHoursMonthBy EnvironmentId |
+environment | +storage | ++ |
| getEnvironmentHitsMonthBy EnvironmentId |
+environment | +storage | ++ |
| addOrUpdateEnvironment Storage |
+environment | +storage | ++ |
| addDeployment | +environment | +deploy:production | +projectID | +
| deleteDeployment | +deployment | +delete | +projectID | +
| updateDeployment | +deployment | +update | +projectID | +
| setEnvironmentServices | +environment | +update:production | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:production | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:production | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:production | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:production | +projectID | +
| updateGroup | +group | +update | +groupID | +
| deleteGroup | +group | +delete | +groupID | +
| addUserToGroup | +group | +addUser | +groupID | +
| removeUserFromGroup | +group | +removeUser | +groupID | +
| addNotificationToProject | +project | +addNotification | +projectID | +
| removeNotificationFromProject | +project | +removeNotification | +projectID | +
| updateProject | +project | +update | +projectID | +
| addGroupsToProject | +project | +addGroup | +projectID | +
| removeGroupsFromProject | +project | +removeGroup | +projectID | +
| addTask | +task | +add:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:production | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:production | +projectID | +
| addBackup | +backup | +add | +projectID | +
| getBackupsByEnvironmentId | +backup | +view | +projectID | +
| addEnvVariable (to Environment) | +env_var | +environment:add:development | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:development | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:development | +projectID | +
| updateEnvironment | +environment | +update:development | +projectID | +
| deleteEnvironment | +environment | +delete:development | +projectID | +
| addDeployment | +environment | +deploy:development | +projectID | +
| setEnvironmentServices | +environment | +update:development | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:development | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:development | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:development | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:development | +projectID | +
| getNotificationsByProjectId | +notification | +view | +projectID | +
| addTask | +task | +add:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:production | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:development | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:development | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:development | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:production | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:development | +projectID | +
| deleteTask | +task | +delete | +projectID | +
| updateTask | +task | +update | +projectID | +
| uploadFilesForTask | +task | +update | +projectID | +
| deleteFilesForTask | +task | +delete | +projectID | +
| getBackupsByEnvironmentId | +deployment | +view | +projectID | +
| getEnvironmentsByProjectId | +environment | +view | +projectID | +
| getEnvironmentServices ByEnvironmentId |
+environment | +view | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:view | +projectID | +
| addGroup | +group | +add | ++ |
| getOpenshiftByProjectId | +openshift | +view | +projectID | +
| addProject | +project | +add | ++ |
| getProjectByEnvironmentId | +project | +view | +projectID | +
| getProjectByGitUrl | +project | +view | +projectID | +
| getProjectByName | +project | +view | +projectID | +
| addRestore | +restore | +add | +projectID | +
| updateRestore | +restore | +update | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:development | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:production | +projectID | +
| taskDrushCron | +task | +drushCron:development | +projectID | +
| taskDrushCron | +task | +drushCron:production | +projectID | +
| getFilesByTaskId | +task | +view | +projectID | +
| getTasksByEnvironmentId | +task | +view | +projectID | +
| getTaskByRemoteId | +task | +view | +projectID | +
| getTaskById | +task | +view | +projectID | +
| addUser | +user | +add | ++ |
| Name | +Resource | +Scope | +Attributes | +
|---|---|---|---|
| deleteAllBackups | +backup | +deleteAll | ++ |
| deleteAllEnvironments | +environment | +deleteAll | ++ |
| getEnvironmentStorageMonthBy EnvironmentId |
+environment | +storage | ++ |
| getEnvironmentHoursMonthBy EnvironmentId |
+environment | +storage | ++ |
| getEnvironmentHitsMonthBy EnvironmentId |
+environment | +storage | ++ |
| deleteAllGroups | +group | +deleteAll | ++ |
| deleteAllNotificationSlacks | +notification | +deleteAll | ++ |
| removeAllNotificationsFrom AllProjects |
+notification | +removeAll | ++ |
| getAllOpenshifts | +openshift | +viewAll | ++ |
| deleteAllProjects | +project | +deleteAll | ++ |
| deleteAllSshKeys | +ssh_key | +deleteAll | ++ |
| removeAllSshKeysFromAllUsers | +ssh_key | +removeAll | ++ |
| deleteAllUsers | +user | +deleteAll | ++ |
| addOrUpdateEnvironment Storage |
+environment | +storage | ++ |
| addNotificationSlack | +notification | +add | ++ |
| updateNotificationSlack | +notification | +update | ++ |
| deleteNotificationSlack | +notification | +delete | ++ |
| addKubernetes | +kubernetes | +add | ++ |
| updateKubernetes | +kubernetes | +update | ++ |
| deleteKubernetes | +kubernetes | +delete | ++ |
| deleteAllKubernetes | +kubernetes | +deleteAll | ++ |
| getAllProjects | +project | +viewAll | ++ |
| addSshKey | +ssh_key | +add | +userID | +
| updateSshKey | +ssh_key | +update | +userID | +
| deleteSshKey | +ssh_key | +delete | +userID | +
| getUserSshKeys | +ssh_key | +view:user | +userID | +
| updateUser | +user | +update | +userID | +
| deleteUser | +user | +delete | +userID | +
| deleteEnvironment | +environment | +delete:production | +projectID | +
| deleteProject | +project | +delete | +projectID | +
| getProjectByEnvironmentId | +project | +viewPrivateKey | +projectID | +
| getProjectByGitUrl | +project | +viewPrivateKey | +projectID | +
| getProjectByName | +project | +viewPrivateKey | +projectID | +
| deleteBackup | +backup | +delete | +projectID | +
| addEnvVariable (to Project) | +env_var | +project:add | +projectID | +
| addEnvVariable (to Environment) |
+env_var | +environment:add:production | +projectID | +
| deleteEnvVariable | +env_var | +delete | +projectID | +
| deleteEnvVariable (from Project) | +env_var | +project:delete | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:viewValue | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:production | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:production | +projectID | +
| updateEnvironment | +environment | +update:production | +projectID | +
| addDeployment | +environment | +deploy:production | +projectID | +
| deleteDeployment | +deployment | +delete | +projectID | +
| updateDeployment | +deployment | +update | +projectID | +
| setEnvironmentServices | +environment | +update:production | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:production | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:production | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:production | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:production | +projectID | +
| updateGroup | +group | +update | +groupID | +
| deleteGroup | +group | +delete | +groupID | +
| addUserToGroup | +group | +addUser | +groupID | +
| removeUserFromGroup | +group | +removeUser | +groupID | +
| addNotificationToProject | +project | +addNotification | +projectID | +
| removeNotificationFromProject | +project | +removeNotification | +projectID | +
| updateProject | +project | +update | +projectID | +
| addGroupsToProject | +project | +addGroup | +projectID | +
| removeGroupsFromProject | +project | +removeGroup | +projectID | +
| addTask | +task | +add:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:production | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:production | +projectID | +
| addBackup | +backup | +add | +projectID | +
| getBackupsByEnvironmentId | +backup | +view | +projectID | +
| addEnvVariable (to Environment) |
+env_var | +environment:add:development | +projectID | +
| deleteEnvVariable (from Environment) | +env_var | +environment:delete:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:viewValue:development | +projectID | +
| addOrUpdateEnvironment | +environment | +addOrUpdate:development | +projectID | +
| updateEnvironment | +environment | +update:development | +projectID | +
| deleteEnvironment | +environment | +delete:development | +projectID | +
| addDeployment | +environment | +deploy:development | +projectID | +
| setEnvironmentServices | +environment | +update:development | +projectID | +
| deployEnvironmentLatest | +environment | +deploy:development | +projectID | +
| deployEnvironmentBranch | +environment | +deploy:development | +projectID | +
| deployEnvironmentPullrequest | +environment | +deploy:development | +projectID | +
| deployEnvironmentPromote | +environment | +deploy:development | +projectID | +
| getNotificationsByProjectId | +notification | +view | +projectID | +
| addTask | +task | +add:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:development | +projectID | +
| taskDrushArchiveDump | +task | +drushArchiveDump:production | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:development | +projectID | +
| taskDrushSqlDump | +task | +drushSqlDump:production | +projectID | +
| taskDrushUserLogin | +task | +drushUserLogin:destination:development | +environmentID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:development | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:source:production | +projectID | +
| taskDrushSqlSync | +task | +drushSqlSync:destination:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:development | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:source:production | +projectID | +
| taskDrushRsyncFiles | +task | +drushRsync:destination:development | +projectID | +
| deleteTask | +task | +delete | +projectID | +
| updateTask | +task | +update | +projectID | +
| uploadFilesForTask | +task | +update | +projectID | +
| deleteFilesForTask | +task | +delete | +projectID | +
| getBackupsByEnvironmentId | +deployment | +view | +projectID | +
| getEnvironmentsByProjectId | +environment | +view | +projectID | +
| getEnvironmentServices ByEnvironmentId |
+environment | +view | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:development | +projectID | +
| getEnvVarsByEnvironmentId | +env_var | +environment:view:production | +projectID | +
| getEnvVarsByProjectId | +env_var | +project:view | +projectID | +
| addGroup | +group | +add | ++ |
| getOpenshiftByProjectId | +openshift | +view | +projectID | +
| addProject | +project | +add | ++ |
| getProjectByEnvironmentId | +project | +view | +projectID | +
| getProjectByGitUrl | +project | +view | +projectID | +
| getProjectByName | +project | +view | +projectID | +
| addRestore | +restore | +add | +projectID | +
| updateRestore | +restore | +update | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:development | +projectID | +
| taskDrushCacheClear | +task | +drushCacheClear:production | +projectID | +
| taskDrushCron | +task | +drushCron:development | +projectID | +
| taskDrushCron | +task | +drushCron:production | +projectID | +
| getFilesByTaskId | +task | +view | +projectID | +
| getTasksByEnvironmentId | +task | +view | +projectID | +
| getTaskByRemoteId | +task | +view | +projectID | +
| getTaskById | +task | +view | +projectID | +
| addUser | +user | +add | ++ |
Harbor-Core requires a configuration file to start, which is located at /etc/core/app.conf within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
The configmap from which this config file is generated is stored within Lagoon in the services/harbor-core/harbor-core.yml file. Any changes made to this configmap will be persisted across container restarts.
_REDIS_URLharbor-redis:6379,100,._REDIS_URL_REGredis://harbor-redis:6379/2.ADMIRAL_URLNA.CFG_EXPIRATION5.CHART_CACHE_DRIVERredis.CLAIR_ADAPTER_URLhttp://harbor-trivy:8080.CLAIR_DBpostgres.CLAIR_DB_HOSTharbor-database.CLAIR_DB_PASSWORDtest123 when run locally or during CI testing.CLAIR_DB_PORT5432.CLAIR_DB_SSLMODEdisable.CLAIR_DB_USERNAMEpostgres.CLAIR_HEALTH_CHECK_SERVER_URLhttp://harbor-trivy:8080CLAIR_URLhttp://harbor-trivy:6060.CONFIG_PATH/etc/core/app.conf.CORE_SECRETsecret123 when Harbor is run locally or during CI testing.CORE_URLhttp://harbor-core:8080.DATABASE_TYPEpostgresql.HARBOR_ADMIN_PASSWORDadmin user.admin when run locally or during CI testing.HARBOR_NGINX_ENDPOINThttp://harbor-nginx:8080 when run locally or during CI testing.HTTP_PROXYHTTPS_PROXYJOBSERVICE_SECRETsecret123 when Harbor is run locally or during CI testing.JOBSERVICE_URLhttp://harbor-jobservice:8080.LOG_LEVELerror.NO_PROXYharbor-core,harbor-jobservice,harbor-database,harbor-trivy,harborregistry,harbor-portal,127.0.0.1,localhost,.local,.internal.PORTAL_URLhttp://harbor-portal:8080.POSTGRESQL_DATABASEregistry.POSTGRESQL_HOSTharbor-database.POSTGRESQL_MAX_IDLE_CONNS50.POSTGRESQL_MAX_OPEN_CONNS100.POSTGRESQL_PASSWORDPOSTGRESQL_PORT5432.POSTGRESQL_USERNAMEpostgres.POSTGRESQL_SSLMODEdisable.REGISTRY_HTTP_SECRETsecret123 when Harbor is run locally or during CI testing.REGISTRY_STORAGE_PROVIDER_NAMEs3.REGISTRY_URLhttp://harborregistry:5000.REGISTRYCTL_URLhttp://harborregistryctl:8080.ROBOT_TOKEN_DURATION999.SYNC_REGISTRYfalse.TOKEN_SERVICE_URLhttp://harbor-core:8080/service/token.TRIVY_ADAPTER_URLhttp://harbor-trivy:8080.WITH_CHARTMUSEUMfalse.WITH_CLAIRtrue.WITH_NOTARYfalse.WITH_TRIVYtrue.Harbor-Database requires specific environment variables to be set in order to start, which are stored within secrets as described in the services/harbor-database/harbor-core.yml file.
POSTGRES_DBpostgres.POSTGRES_PASSWORDtest123.POSTGRES_USERpostgres.Harbor-Jobservice requires a configuration file to start, which is located at /etc/jobservice/config.yml within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
The configmap from which this config file is generated is stored within Lagoon in the services/harbor-jobservice/harbor-jobservice.yml file. Any changes made to this configmap will be persisted across container restarts.
CORE_URLharbor-jobservice where harbor-core can be reached.http://harbor-core:8080.CORE_SECRETharbor-core.secret123 when Harbor is run locally or during CI testing.HTTP_PROXYHTTPS_PROXYJOBSERVICE_SECRETharbor-jobservice.secret123 when Harbor is run locally or during CI testing.LOG_LEVELerror.debug to enable very verbose logging.NO_PROXYharbor-core,harbor-jobservice,harbor-database,harbor-trivy,harborregistry,harbor-portal,127.0.0.1,localhost,.local,.internal.REGISTRY_CONTROLLER_URLharborregistryctl service.http://harborregistryctl:8080SCANNER_LOG_LEVELerror.debug to enable very verbose logging.SCANNER_STORE_REDIS_URLharbor-trivy how to connect to its Redis store.redis://harbor-redis:6379/4.Harbor-Trivy is configured via specific environment variables and does not use a config file.
+SCANNER_LOG_LEVELerror.debug to enable very verbose logging.SCANNER_STORE_REDIS_URLredis://harbor-redis:6379/4.SCANNER_JOB_QUEUE_REDIS_URLredis://harbor-redis:6379/4.SCANNER_TRIVY_VULN_TYPEos,libraryHarborRegistry requires a configuration file to start, which is located at /etc/registry/config.yml within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
This config file is stored within the services/harborregistry/harborregistry.yml file and loaded into the container as /etc/registry/pre-config.yml.
A custom container entrypoint, services/harborregistry/entrypoint.sh, then transposes provided environment variables into this config file and saves the results as /etc/registry/config.yml.
CORE_SECRETharbor-core.secret123 when Harbor is run locally or during CI testing.HARBOR_NGINX_ENDPOINTharborregistry where its NGINX ingress controller, harbor-nginx, is running in order to construct proper push and pull instructions in the UI, among other things.http://harbor-nginx:8080 when run locally or during CI testing.JOBSERVICE_SECRETharbor-jobservice.secret123 when Harbor is run locally or during CI testing.REGISTRY_HTTP_SECRETharborregistry.secret123 when Harbor is run locally or during CI testing.REGISTRY_REDIS_PASSWORDharborregistryctl the password that should be used to connect to Redis.HarborRegistryCtl requires a configuration file to start, which is located at /etc/registryctl/config.yml within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
The configmap from which this config file is generated is stored within Lagoon in the services/harborregistryctl/harborregistry.yml file. Any changes made to this configmap will be persisted across container restarts.
CORE_SECRETharbor-core.secret123 when Harbor is run locally or during CI testing.JOBSERVICE_SECRETharbor-jobservice.secret123 when Harbor is run locally or during CI testing.REGISTRY_HTTP_SECRETharborregistry.secret123 when Harbor is run locally or during CI testing.REGISTRY_REDIS_PASSWORDharborregistryctl the password that should be used to connect to Redis.Lagoon supports running Harbor locally, and it is automatically used for hosting all Kubernetes-based builds (any time the project's activeSystemsDeploy value is set to lagoon_kubernetesBuildDeploy). When Harbor is ran locally, it makes use of MinIO as a storage backend, which is an AWS S3 compatible local storage solution.
Harbor is composed of multiple containers, which all require different settings in order for them to run successfully.
+The following environment variables are required to be set in order for Harbor to properly start:
+HARBOR_REGISTRY_STORAGE_AMAZON_BUCKETharbor-images when Lagoon is run locally or during CI testing.HARBOR_REGISTRY_STORAGE_AMAZON_REGIONus-east-1 when Lagoon is run locally or during CI testing.REGISTRY_STORAGE_S3_ACCESSKEYREGISTRY_STORAGE_S3_SECRETKEYThe following environment variables can be set if required:
+HARBOR_REGISTRY_STORAGE_AMAZON_ENDPOINThttps://s3.amazonaws.com when this variable is not set.The following containers make use of configuration files:
+ +The following containers do not require configuration files to run:
+Harbor is used as the default package repository for Lagoon when deploying to Kubernetes infrastructure. Harbor provides a Docker registry and a container security scanning solution provided by Trivy.
+Note
+When running Lagoon locally, the configuration for Harbor is handled entirely automagically.
+If you are running Lagoon locally, you can access that UI at localhost:8084. The username is admin and the password is admin.
Note
+If you are hosting a site with a provider (such as amazee.io), they may not allow customer access to the Harbor UI.
+Once logged in, the first screen is a list of all repositories your user has access to. Each "repository" in Harbor correlates to a project in Lagoon.
+
Within each Harbor repository, you'll see a list of container images from all environments with a single Lagoon project.
+
From here, you can drill down into an individual container in order to see its details, including an overview of its security scan results.
+
Harbor comes with a built-in security scanning solution provided by the Trivy service. This service analyzes a specified container image for any installed packages, and collects the version numbers of those installed packages. The Trivy service then searches the National Vulnerability Database for any CVEs (common vulnerabilities and exposures) affecting those package versions. Trivy is also library aware, so it will scan any Composer files or other package library definition files and report any vulnerabilities found within those package versions. These vulnerabilities are then reported within Harbor for each individual container.
+An example of a security scan in Harbor, showing applicable vulnerabilities for a scanned container:
+
Lagoon broadly classifies three levels in the application stack:
+The core building blocks of any Lagoon project, these are usually provided by Lagoon-specific images.
+These take those base images, and add in the necessary logic, tools and packages needed to serve a website, or drive an application.
+Usually built on top of Frameworks, this is the layer that content editors or developers will interact with to shape the finished product.
+When we reference any repositories for use on Lagoon, we usually refer to them in three ways:
+These are fully-functional, cloneable starter repositories, maintained and updated regularly, ready to be extended and used with little customization.
+These are fully functional repositories, maintained and updated regularly, but may require some effort to make work for your individual project.
+These are repositories that have been built as a demonstration, and are usable for some of the concepts within, but aren't routinely maintained or updated.
+For a more complete list, check out out our GitHub repository: https://www.github.com/lagoon-examples and our website https://lagoon.sh/application/
+ + + + + + + + +Lagoon provides Node.js images that are based on the official Node Alpine images.
+More information on how to adapt your project to run on Lagoon can be found in our Node.js Docker Images section.
+ + + + + + + + +lagoon.yml#Project- and environment-level configuration for Lagoon is provided in the .lagoon.yml file in your repository.
See lagoon-yml.md.
docker-compose.yml#Service-level configuration for Lagoon in provided in the docker-compose.yml file in your repository. In particular, the lagoon.type and associated service labels are documented in the individual services.
Lagoon has the ability to provision storage for most services - the built-in Lagoon service types have a -persistent variant that can add in the necessary PVCs, volumes, etc. We have updated our examples to reflect this configuration locally.
Lagoon has configurations available for:
+Lagoon also has the capability to utilize the dbaas-operator to automatically provision these databases using an underlying managed database service (i.e. RDS, Google Cloud Databases, Azure Database). This will happen automatically when these services are provisioned and configured for your cluster. If these are not available, a pod will be provisioned as a fallback.
+Lagoon supports Redis as a cache backend. In production, some users provision a managed Redis service for their production environments to help them scale.
+Lagoon supports Elasticsearch, Solr and OpenSearch as search providers. External search providers can also be configured if required.
+Lagoon auto-generates routes for services that have ingress requirements. Custom routes can be provided in the .lagoon.yml on a per-service basis.
Lagoon makes heavy use of environment variables, at build and runtime. Where these are used to provide critical configuration for your application (e.g. database config/credentials) - it is important that the local and Lagoon versions are named similarly.
+ + + + + + + + + +Even if Lagoon doesn't have a base image for your particular application, framework or language, Lagoon can still build it!
+Extending on, or inheriting from the commons image, Lagoon can run almost any workload.
+This brief example shows how to build a Hugo website and serve it as static files in an NGINX image. The commons image is used to add Hugo, copy the site in, and build it. The NGINX image is then used to serve the site, with the addition of a customized NGINX config.
+ + + + + + + + + + +Lagoon supports a wide range of PHP-based applications, such as Drupal, Laravel, Wordpress, Magento and Symfony.
+More information on how to adapt your PHP project to run on Lagoon can be found in our PHP-cli Docker Images and PHP-FPM Docker Images sections.
+ + + + + + + + +Lagoon provides images for Python 3.7 and above that can be used to build web apps in a wide range of Python-based frameworks and applications.
+More information on how to adapt your Python project to run on Lagoon can be found in our Python Docker Images section.
+ + + + + + + + +We provide images for Ruby 3.0 and above, built on the official Ruby alpine Docker images.
+Below we assume that you're attempting to get a Rails app deployed on Lagoon, although most of the details described are really framework-neutral.
+The Ruby on Rails example in the Lagoon examples repository is instructive here.
+In the docker-compose.yml we set up a service named ruby, which is the primary service that will be processing any dynamic requests.
If you look at the dockerfile specified for the ruby service, you'll see that we're exposing port 3000. The nginx service will direct any requests for non-static assets to the ruby service on this port (see the nginx configuration file for more details).
The Lagoon logging infrastructure is described in the docs here. Essentially, in order to make use of the infrastructure, logs need to be sent via a UDP message to udp://application-logs.lagoon.svc:5140.
In our Rails example, we're importing the logstash-logger gem, and then in our config/application.rb we're initializing it with the following:
if ENV.has_key?('LAGOON_PROJECT') && ENV.has_key?('LAGOON_ENVIRONMENT') then
+ lagoon_namespace = ENV['LAGOON_PROJECT'] + "-" + ENV['LAGOON_ENVIRONMENT']
+ LogStashLogger.configure do |config|
+ config.customize_event do |event|
+ event["type"] = lagoon_namespace
+ end
+ end
+
+ config.logstash.host = 'application-logs.lagoon.svc'
+ config.logstash.type = :udp
+ config.logstash.port = 5140
+ end
+The example uses our PostgreSQL image (see the docker-compose.yml file). Configuring database access in Rails for Lagoon is very straightforward. Since Lagoon injects the database host, name, and credentials as environment variables, we can change our config/database.yml to be aware of these env vars, and consume them if they exist.
default: &default
+ adapter: postgresql
+ encoding: unicode
+ pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
+ username: <%= ENV.fetch("POSTGRES_USERNAME") { "drupal" } %>
+ password: <%= ENV.fetch("POSTGRES_PASSWORD") { "drupal" } %>
+ host: <%= ENV.fetch("POSTGRES_HOST") { "postgres" } %>
+ database: <%= ENV.fetch("('POSTGRES_DATABASE'") { "drupal" } %>
+The WordPress template is configured to use Composer to install WordPress, its dependencies, and themes.
+The WordPress template is based on the https://github.com/roots/bedrock boilerplate, but extended to match a standardized Lagoon deployment pattern.
+The template uses Composer to install WordPress and its themes.
+Lagoon can support MariaDB and PostgreSQL databases, but as support for PostgreSQL is limited in WordPress, it isn't recommended for use.
+Lagoon doesn't have a built-in configuration for WordPress - instead, the template comes with a starting nginx.conf - please contribute any improvements you may find!
+The Lagoon template installs wp-cli into the cli image to manage your WordPress install.
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+Examples of behavior that contributes to creating a positive environment include:
+Examples of unacceptable behavior by participants include:
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at uselagoon@amazee.io. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4.
+ + + + + + + + +Our official community meeting space is the Lagoon Discord.
+We’re starting this community as a place for all Lagoon users to collaborate, solve problems, share ideas, and contribute back to the Lagoon project. We’re working to consolidate our community as it’s currently spread out over Slack and various other places. We also wanted to invite all of our users and customers to join so that everyone can benefit from the community, no matter how they’re using Lagoon.
+Please remember that this is not to replace your current support channels - those will remain the same. This is a place to connect with other users as well as the Lagoon maintainers.
+We ask that all community members review our Participation and Moderation Guidelines, as well as the Code of Conduct.
+In addition to our Zoom Community Hours, we'll also be hosting Community Hours on Discord in 2023!
+ + + + + + + + +These guidelines have been adapted from Drupal Diversity & Inclusion’s Moderation Guidelines.
+In Lagoon spaces, strive to promote understanding, empathy, and increase personal awareness of all people. This includes people from across the Drupal Community and the greater Technical Community, even those you may personally disagree with.
+If kicked from the Discord, the kicked user can send a private message (PM) to the kicker or another Moderator, if desired, for re-admittance. If a disruptive person is engaging in what appears to be intentionally inflammatory, bullying, or harassing behavior provoking hostile responses (or acting in a hostile manner), kicking is faster and easier than trying to placate a disruptive person whose behavior is causing distress to other channel members.
+The kick is not a ban. There are times when disruptive or triggering comments and statements are genuine and break the lines of communication between two parties. By speaking with a Moderator, the (potentially) disruptive person can be coached on using more sensitive, inclusive, and diverse-aware language, and on engaging in a more constructive manner.
+Tier One Response
+User is welcomed in the channel, asked to read some scroll back, and given a link to participation guidelines.
+Tier Two Response
+User is gently reminded in channel to keep posts on topic, and/or of participation guidelines.
+Tier Three Response
+User is PM’d by available Moderator to explain the problem(s) with their posts and given suggestions of what to do differently.
+Tier Four Response
+If behavior continues, User is kicked for no less than 24 hours from the Discord.
+Intentionally disruptive individuals get kicked, not tiered. Repeated offenses will result in a ban.
+ + + + + + + + +We ask that all members of our community, in any spaces, virtual or physical, adhere to our Code of Conduct.
+These guidelines have been adapted from Drupal Diversity & Inclusion’s Participation Guidelines.
+Additional considerations for in-person Lagoon spaces
+The Lagoon team reserves the right to terminate anyone’s access to the Lagoon spaces.
+ + + + + + + + +1 . Ensure the dev script at services/api/package.json includes the following:
2 . Update docker-compose.yml to map the dist folder and expose the 9229 port:
api:
+ image: ${IMAGE_REPO:-lagoon}/api
+ command: yarn run dev
+ volumes:
+ - ./services/api/src:/app/services/api/src
+ - ./services/api/dist:/app/services/api/dist
+ depends_on:
+ - api-db
+ - local-api-data-watcher-pusher
+ - keycloak
+ ports:
+ - '3000:3000'
+ - '9229:9229'
+3 . Add the following to .vscode/launch.json:
{
+ // Use IntelliSense to learn about possible attributes.
+ // Hover to view descriptions of existing attributes.
+ // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387.
+ "version": "0.2.0",
+ "configurations": [
+ {
+ "name": "Docker: Attach to Node",
+ "type": "node",
+ "request": "attach",
+ "port": 9229,
+ "address": "localhost",
+ "outFiles": ["${workspaceRoot}/app/services/api/dist/**/*.js"],
+ "localRoot": "${workspaceFolder}/services/api",
+ "remoteRoot": "/app/services/api",
+ "sourceMaps": true,
+ "protocol": "inspector"
+ }
+ ]
+}
+4 . Rebuild/restart the containers:
+ +5 . Restart VScode.
+ + + + + + + + +Development of Lagoon locally can now be performed on a local Kubernetes cluster, or via Docker Compose (as a fallback).
+Note
+The full Lagoon stack relies on a range of upstream projects which are currently incompatible with ARM-based architectures, such as the the M1/M2 Apple Silicon-based machines. For this reason, running or developing lagoon-core or lagoon-remote locally on these architectures is not currently supported. See https://github.com/uselagoon/lagoon/issues/3189 for more information.
Docker must be installed to build and run Lagoon locally.
+Please check the official docs for how to install Docker.
+Docker Compose is included in Docker for Mac installations. For Linux installations see the directions here.
+You will need to update your insecure registries in Docker. Read the instructions here on how to do that. We suggest adding the entire local IPv4 Private Address Spaces to avoid unnecessary reconfiguration between Kubernetes and Docker Compose. e.g. "insecure-registries" : ["172.16.0.0/12","192.168.0.0/16"],
Running a Lagoon, Kubernetes, or Docker cluster on your local machine consumes a lot of resources. We recommend that you give your Docker host a minimum of 8 CPU cores and 12GB RAM.
+Warning
+Only consider building Lagoon this way if you intend to develop features or functionality for it, or want to debug internal processes. We will also be providing instruction to install Lagoon without building it (i.e. by using the published releases).
+We're using make (see the Makefile) in order to build the needed Docker images, configure Kubernetes and run tests.
We have provided a number of routines in the Makefile to cover most local development scenarios. Here we will run through a complete process.
+-j8 tells make to run 8 tasks in parallel to speed the build up. Adjust as necessary.SCAN_IMAGES=false as a default to not scan the built images for vulnerabilities. If set to true, a scan.txt file will be created in the project root with the scan output.Warning
+There are a lot of tests configured to run by default - please consider only testing locally the minimum that you need to ensure functionality. This can be done by specifying or removing tests from the TESTS variable in the Makefile.
This process will:
+kind, kubectl, helm, jq.CHARTS_TREEISH variable in the Makefile to control which branch if needed).lagoon-test-suite-* pod, and can be accessed one test per container.Ideally, all of the tests pass and it's all done!
+The test routine creates a local Kubeconfig file (called kubeconfig.kind.lagoon in the root of the project, that can be used with a Kubernetes dashboard, viewer or CLI tool to access the local cluster. We use tools like Lens, Octant, kubectl or Portainer in our workflows. Lagoon Core, Remote and Tests all build in the Lagoon namespace, and each environment creates its own namespace to run, so make sure to use the correct context when inspecting.
In order to use kubectl with the local cluster, you will need to use the correct Kubeconfig. This can be done for every command or it can be added to your preferred tool:
+ +The Helm charts used to build the local Lagoon are cloned into a local folder and symlinked to lagoon-charts.kind.lagoon where you can see the configuration. We'll cover how to make easy modifications later in this documentation.
The Makefile includes a few simple routines that will make interacting with the installed Lagoon simpler:
+ +This will create local ports to expose the UI (6060), API (7070) and Keycloak (8080). Note that this logs to stdout, so it should be performed in a secondary terminal/window.
This will retrieve the necessary credentials to interact with the Lagoon.
+This will re-push the images listed in KIND_SERVICES with the correct tag, and redeploy the lagoon-core chart. This is useful for testing small changes to Lagoon services, but does not support "live" development. You will need to rebuild these images locally first, e.g rm build/api && make build/api.
This will build the typescript services, using your locally installed Node.js (it should be >16.0). It will then:
+nodemonwatching the code for changesgit clean -dfx as the dist folders are ignored by Git.This will create a standalone OpenDistro for Elasticsearch cluster in your local Docker, and configure Lagoon to dispatch all logs (Lagoon and project) to it, using the configuration in lagoon-logging.
+ +This will re-run a suite of tests (defined in the TESTS variable) against the existing cluster. It will re-push the images needed for tests (tests, local-git, and the data-watcher-pusher). You can specify tests to run by passing the TESTS variable inline.
If updating a test configuration, the tests image will need to be rebuilt and pushed, e.g rm build/tests && make build/tests && make kind/push-images IMAGES='tests' && make kind/retest TESTS='[api]'
This will push all the images up to the image registry. Specifying IMAGES will tag and push specific images.
This will remove the KinD Lagoon cluster from your local Docker.
+The Lagoon test uses Ansible to run the test suite. Each range of tests for a specific function has been split into its own routine. If you are performing development work locally, select which tests to run, and update the $TESTS variable in the Makefile to reduce the concurrent tests running.
The configuration for these tests is held in three services:
+tests is the Ansible test services themselves. The local testing routine runs each individual test as a separate container within a test-suite pod. These are listed below.local-git is a Git server hosted in the cluster that holds the source files for the tests. Ansible pulls and pushes to this repository throughout the testsapi-data-watcher-pusher is a set of GraphQL mutations that pre-populates local Lagoon with the necessary Kubernetes configuration, test user accounts and SSH keys, and the necessary groups and notifications. Note that this will wipe local projects and environments on each run.The individual routines relevant to Kubernetes are:
+active-standby-kubernetes runs tests to check active/standby in Kubernetes.api runs tests for the API - branch/PR deployment, promotion.bitbucket, gitlab and github run tests for the specific SCM providers.drupal-php74 runs a single-pod MariaDB, MariaDB DBaaS and a Drush-specific test for a Drupal 8/9 project (drupal-php73 doesn't do the Drush test).drupal-postgres runs a single-pod PostgreSQL and a PostgreSQL DBaaS test for a Drupal 8 project.elasticsearch runs a simple NGINX proxy to an Elasticsearch single-pod.features-variables runs tests that utilize variables in Lagoon.features-kubernetes runs a range of standard Lagoon tests, specific to Kubernetes.features-kubernetes-2 runs more advanced kubernetes-specific tests - covering multi-project and subfolder configurations.nginx, node and python run basic tests against those project types.node-mongodb runs a single-pod MongoDB test and a MongoDB DBaaS test against a Node.js app.Most services are written in Node.js. As many of these services share similar Node.js code and Node.js packages, we're using a feature of Yarn, called Yarn workspaces. Yarn workspaces need a package.json in the project's root directory that defines the workspaces.
The development of the services can happen directly within Docker. Each container for each service is set up in a way that its source code is mounted into the running container (see docker-compose.yml). Node.js itself is watching the code via nodemon , and restarts the Node.js process automatically on a change.
The services not only share many Node.js packages, but also share actual custom code. This code is within node-packages/lagoon-commons. It will be automatically symlinked by Yarn workspaces. Additionally, the nodemon of the services is set up in a way that it checks for changes in node-packages and will restart the node process automatically.
Rebuild the images via:
+ +node_modules content when I try to build / run a Node.js based image#Make sure to run yarn in Lagoon's root directory, since some services have common dependencies managed by yarn workspaces.
nip.io domains#Error response from daemon: Get https://registry.172.18.0.2.nip.io:32080/v2/: dial tcp: lookup registry.172.18.0.2.nip.io: no such host
+This can happen if your local resolver filters private IPs from results. You can work around this by editing /etc/resolv.conf and adding a line like nameserver 8.8.8.8 at the top to use a public resolver that doesn't filter results.
Here are some development scenarios and useful workflows for getting things done.
+tests/tests/features-variables.yaml and add a test case.tests image.tests image into the cluster registry.We really value anything that you can offer us!
+We've made building and viewing the documentation really straightforward, and the team is always ready to help out with reviews or pointers.
+We use mkdocs with the excellent Material theme.
+From the root of the Lagoon repository (you'll need Docker), run:
+docker run --rm -it -p 127.0.0.1:8000:8000 -v ${PWD}:/docs ghcr.io/amazeeio/mkdocs-material
+This will start a development server on http://127.0.0.1:8000, configured to live-reload on any updates.
+The customized Docker image contains all the necessary extensions.
+Alternatively, to run the mkdocs package locally, you'll need to install mkdocs, and then install all of the necessary plugins.
Each documentation page also has an "edit" pencil in the top right, that will take you to the correct page in the Git repository.
+Feel free to contribute here, too - you can always use the built-in github.dev web-based editor. It's got basic Markdown previews, but none of the mkdocs loveliness.
+We use the Deploy MkDocs GitHub Action to build all main branch pushes, and trigger a deployment of the gh-pages branch.
Lagoon has a number of moving parts, making releases quite complicated!
+lagoon-ui and build-deploy-tool in the format core-v2.MINOR.PATCH. Note that there are no other tags or releases on build-deploy-tool, but lagoon-ui also has it's own semver releases that are based on it's features.In the chart.yaml for the lagoon-core and lagoon-test charts, update the following fields:
# This is the chart version. This version number should be incremented each
+# time you make changes to the chart and its templates, including the app
+# version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 1.28.0
+
+# This is the version number of the application being deployed. This version
+# number should be incremented each time you make changes to the application.
+# Versions are not expected to follow Semantic Versioning. They should reflect
+# the version the application is using.
+appVersion: v2.14.2
+
+# This section is used to collect a changelog for artifacthub.io
+# It should be started afresh for each release
+# Valid supported kinds are added, changed, deprecated, removed, fixed and security
+annotations:
+ artifacthub.io/changes: |
+ - kind: changed
+ description: update Lagoon appVersion to v2.14.2
+Create a PR for this chart release, and the Github Actions suite will undertake a full suite of tests:
+Usually, failures in the lint and test charts are well explained (missing/misconfigured chart settings). If a single Lagoon test failes, it may just need re-running. If multiple failures occur, they will need investigating.
+Once those tests have all passed successfully, you can proceed with creating the releases:
+Lagoon remote has a release cycle separate to Lagoon Core, and as such, can be released anytime that a dependency sub-chart or service is updated.
+ + + + + + + + +All of our tests are written with Ansible and mostly follow this approach:
+tests/files) into this Git repository.Lagoon is mostly tested in 3 different ways:
+During local development, the best way to test is locally. All tests are started via make. Make will download and build all the required dependencies.
This will run all defined tests. If you only want to run a subset of the tests, run make tests-list to see all existing tests and run them individually.
For example, make tests/node will run the Node.js Docker images tests.
In order to actually see what is happening inside the microservices, we can use make logs:
Or only for a specific service:
+ +In order to test pull requests that are created against Lagoon, we have a fully automatic integration test running on a dedicated Jenkins instance: https://ci.lagoon.sh. It is defined inside the .Jenkinsfile, and runs automatically for every pull request that is opened.
This will build all images, start a Kubernetes cluster and run a series of tests.
+The tests can be found here:
+ + + + + + + + + + + +We gladly welcome any and all contributions to Lagoon!
+Lagoon benefits from any kind of contribution - whether it's a bugfix, new feature, documentation update, or simply some queue maintenance - we're happy that you want to help
+There's a whole section on how to get Lagoon running on your local machine using KinD over at Developing Lagoon. This documentation is still very WIP - but there are a lot of Makefile routines to help you out.
+We've got another section that outlines how to install Lagoon from Helm charts at Installing Lagoon Into Existing Kubernetes Cluster - we'd love to get this process as slick as possible!
+Right now one of our biggest needs is putting together examples of Lagoon working with various content management systems, etc, other than Drupal.
+If you can spin up an open source CMS or framework that we don’t currently have as a Docker Compose stack, send us a PR. Look at the existing examples at https://github.com/uselagoon/lagoon-examples for tips, pointers and starter issues.
+One small catch – wherever possible, we’d like them to be built using our base Docker Hub images https://hub.docker.com/u/uselagoon – if we don’t have a suitable image, or our images need modifying – throw us a PR (if you can) or create an issue (so someone else can) at https://github.com/uselagoon/lagoon-images.
+Help us improve our existing examples, if you can - are we following best practices, is there something we’re doing that doesn’t make sense?
+Bonus points for anyone that helps contribute to tests for any of these examples – we’ve got some example tests in a couple of the projects you can use for guidance – https://github.com/amazeeio/drupal-example-simple/blob/8.x/TESTING_dockercompose.md. The testing framework we’re using is Leia, from the excellent team behind Lando.
+Help us to document our other examples better – we’re not expecting a full manuscript, but tidy-ups, links to helpful resources and clarifying statements are all super-awesome.
+If you have any questions, reach out to us on Discord!
+We take security very seriously. If you discover a security issue or think you found one, please bring it to the maintainers' attention.
+Danger
+Please send your findings to security@amazee.io. Please DO NOT file a GitHub issue for them.
+Security reports are greatly appreciated and will receive public karma and swag! We're also working on a Bug Bounty system.
+We're always interested in fixing issues, therefore issue reports are very welcome. Please make sure to check that your issue does not already exist in the issue queue.
+Cool! Create an issue and we're happy to look over it. We can't guarantee that it will be implemented. But we are always interested in hearing ideas of what we could bring to Lagoon.
+Another good way is also to talk to us via Discord about your idea. Join today!
+Epic! Please send us a pull request for it, we will do our best to review it and merge it if possible.
+ + + + + + + + +The Lagoon commons Docker image. Based on the official Alpine images.
This image has no functionality itself, but is instead a base image, intended to be extended and utilized to build other images. All the alpine-based images in Lagoon inherit components from commons.
+docker-sleep - standardized one-hour sleepfix-permissions - automatically fixes permissions on a given directory to all group read-writewait-for - a small script to ensure that services are up and running in the correct order - based off https://github.com/eficode/wait-forentrypoint-readiness - checks to make sure that long-running entrypoints have completedentrypoints - a script to source all entrypoints under /lagoon/entrypoints/* in an alphabetical/numerical orderThe list of default entrypoints in this image is found at https://github.com/uselagoon/lagoon-images/tree/main/images/commons/lagoon/entrypoints. Subsequent downstream images will also contribute entrypoints under /lagoon that are run in the eventual image.
MariaDB is the open source successor to MySQL.
+The Lagoon MariaDB image Dockerfile. Based on the official packages mariadb and mariadb-client provided by the the upstream Alpine image.
This Dockerfile is intended to be used to set up a standalone MariaDB database server.
+uselagoon/mariadb-10.4uselagoon/mariadb-10.5uselagoon/mariadb-10.6uselagoon/mariadb-10.11Info
+As these images are not built from the upstream MariaDB images, their support follows a different cycle - and will only receive updates as long as the underlying Alpine images receive support - see https://alpinelinux.org/releases/ for more information. In practice, most MariaDB users will only be running these containers locally - the production instances will use the Managed Cloud Databases provided by the DBaaS Operator
+The default exposed port of MariaDB containers is port 3306.
To allow Lagoon to select the best way to run the MariaDB container, use lagoon.type: mariadb - this allows the DBaaS operator to provision a cloud database if available in the cluster. Use lagoon.type: mariadb-single to specifically request MariaDB in a container. Persistent storage is always provisioned for MariaDB containers at /var/lib/mysql.
This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user.readiness-probe.sh script to check when MariaDB container is ready.docker-compose.yml snippet# mariadb:
+ image: uselagoon/mariadb-10.6-drupal:latest
+ labels:
+ # tells Lagoon this is a MariaDB database
+ lagoon.type: mariadb
+ ports:
+ # exposes the port 3306 with a random local port, find it with `docker-compose port mariadb 3306`
+ - "3306"
+ volumes:
+ # mounts a named volume at the default path for MariaDB
+ - db:/var/lib/mysql
+mysqltuner.pl - Perl script useful for database parameter tuning.mysql-backup.sh - Script for automating the daily MySQL backups on development environment.pwgen - Utility to generate random and complex passwords.my.cnf configuration file#The image ships a default MariaDB configuration file, optimized to work on +Lagoon. Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| MARIADB_DATABASE | +lagoon | +Database name created at startup. | +
| MARIADB_USER | +lagoon | +Default user created at startup. | +
| MARIADB_PASSWORD | +lagoon | +Password of default user created at startup. | +
| MARIADB_ROOT_PASSWORD | +Lag00n | +MariaDB root user's password. | +
| MARIADB_CHARSET | +utf8mb4 | +Set the server charset. | +
| MARIADB_COLLATION | +utf8mb4_bin | +Set server collation. | +
| MARIADB_MAX_ALLOWED_PACKET | +64M | +Set the max_allowed_packet size. |
+
| MARIADB_INNODB_BUFFER_POOL_SIZE | +256M | +Set the MariaDB InnoDB buffer pool size. | +
| MARIADB_INNODB_BUFFER_POOL_INSTANCES | +1 | +Number of InnoDB buffer pool instances. | +
| MARIADB_INNODB_LOG_FILE_SIZE | +64M | +Size of InnoDB log file. | +
| MARIADB_LOG_SLOW | +(not set) | +Variable to control the save of slow queries. | +
| MARIADB_LOG_QUERIES | +(not set) | +Variable to control the save of ALL queries. | +
| BACKUPS_DIR | +/var/lib/mysql/backup | +Default path for databases backups. | +
| MARIADB_DATA_DIR | +/var/lib/mysql | +Path of the MariaDB data dir, be careful, changing this can occur data loss! | +
| MARIADB_COPY_DATA_DIR_SOURCE | +(not set) | +Path which the entrypoint script of mariadb will use to copy into the defined MARIADB_DATA_DIR, this can be used for prepopulating the MariaDB with a database. The scripts expects actual MariaDB data files and not a sql file! Plus it only copies data if the destination does not already have a mysql datadir in it. |
+
If the LAGOON_ENVIRONMENT_TYPE variable is set to production, performances
+are set accordingly by using MARIADB_INNODB_BUFFER_POOL_SIZE=1024 and
+MARIADB_INNODB_LOG_FILE_SIZE=256.
++MongoDB is a general purpose, document-based, distributed database built for modern application developers and for the cloud era. MongoDB is a document database, which means it stores data in JSON-like documents.
++
+- from mongodb.com
+
4.0 Dockerfile - uselagoon/mongo-4
This Dockerfile is intended to be used to set up a standalone MongoDB database server.
+This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user, and therefore also on Kubernetes or OpenShift.The Lagoon nginx image Dockerfile. Based on the official openresty/openresty images.
This Dockerfile is intended to be used as a base for any web servers within Lagoon.
+The default exposed port of NGINX containers is port 8080.
This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user./etc/nginx/* are parsed through envplate with a container-entrypoint.NGINX configuration (static-files.conf)#Warning
+By default NGINX only serves static files - this can be used for static sites that don't require a database or PHP components: for example, static site generators like Hugo, Jekyll or Gatsby.
If you need PHP, have a look at the php-fpm image and use nginx and php-fpm in tandem.
Build the content during the build process and inject it into the nginx container.
redirects-map.conf#In order to create redirects, we have redirects-map.conf in place. This helps you to redirect marketing domains to sub-sites or do non-www to www redirects. If you have a lot of redirects, we suggest having redirects-map.conf stored next to your code for easier maintainability.
Note
+If you only have a few redirects, there's a handy trick to create the redirects with a RUN command in your nginx.dockerfile.
Here's an example showing how to redirect www.example.com to example.com and preserve the request:
RUN echo "~^www.example.com http://example.com\$request_uri;" >> /etc/nginx/redirects-map.conf
+To get more details about the various types of redirects that can be achieved, see the documentation within the redirects-map.conf directly.
After you put the redirects-map.conf in place, you also need to include it in your nginx.dockerfile in order to get the configuration file into your build.
Basic authentication is enabled automatically when the BASIC_AUTH_USERNAME
+and BASIC_AUTH_PASSWORD environment
+variables are set.
Warning
+Automatic basic auth configuration is provided for convenience. It should not be considered a secure method of protecting your website or private data.
+Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| BASIC_AUTH | +restricted | +Set to off to disable basic authentication. |
+
| BASIC_AUTH_USERNAME | +(not set) | +Username for basic authentication. | +
| BASIC_AUTH_PASSWORD | +(not set) | +Password for basic authentication (unencrypted). | +
| FAST_HEALTH_CHECK | +(not set) | +Set to true to redirect GET requests from certain user agents (StatusCake, Pingdom, Site25x7, Uptime, nagios) to the lightweight Lagoon service healthcheck. |
+
The Lagoon Node.js Docker image. Based on the official Node Alpine images.
We ship 2 versions of Node.js images: the normal node:version image and the node:version-builder.
The builder variant of those images comes with additional tooling that is needed when you build Node.js apps (such as the build libraries, npm and Yarn). For a full list check out their Dockerfile.
+uselagoon/node-12uselagoon/node-14uselagoon/node-16uselagoon/node-18uselagoon/node-20Tip
+We stop updating EOL Node.js images usually with the Lagoon release that comes after the officially communicated EOL date: https://nodejs.org/en/about/releases/.
+The default exposed port of Node.js containers is port 3000.
Persistent storage is configurable in Lagoon, using the lagoon.type: node-persistent. See the docs for more info
Use the following labels in your docker-compose.yml file to configure it:
lagoon.persistent = use this to define the path in the container to use as persistent storage - e.g. /app/files.lagoon.persistent.size = this to tell Lagoon how much storage to assign this path.lagoon.persistent.name = (optional) use this to tell Lagoon to use the storage defined in another named service.docker-compose.yml snippet# node:
+ build:
+ # this configures a build from a Dockerfile in the root folder
+ context: .
+ dockerfile: Dockerfile
+ labels:
+ # tells Lagoon this is a node service, configured with 500MB of persistent storage at /app/files
+ lagoon.type: node-persistent
+ lagoon.persistent: /app/files
+ lagoon.persistent.size: 500Mi
+ ports:
+ # local development only
+ # this exposes the port 3000 with a random local port
+ # find it with `docker-compose port node 3000`
+ - "3000"
+ volumes:
+ # local development only
+ # mounts a named volume (files) at the defined path for this service to replicate production
+ - files:/app/files
+++OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
++
+- from https://opensearch.org/
+
uselagoon/opensearch-2Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| OPENSEARCH_JAVA_OPTS | +-Xms512m -Xmx512m | +Sets the memory usage of the OpenSearch container. Both values need be the same value or OpenSearch will not start cleanly. | +
On Linux-based systems, the start of the OpenSearch container may fail due to a low vm.max_map_count setting.
The Lagoon php-cli Docker image. Based on Lagoon php-fpm image, it has all the needed command line tools for daily operations.
Containers (or pods) started from cli images are responsible for building code for Composer or Node.js based projects.
The image also contains database clis for both MariaDB and PostgreSQL.
Info
+This Dockerfile is intended to be used as a base for any cli needs within Lagoon.
uselagoon/php-8.0-cliuselagoon/php-8.1-cliuselagoon/php-8.2-cliAll PHP versions use their own Dockerfiles.
+This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user.COMPOSER_ALLOW_SUPERUSER=1 removes warning about use of Composer as root.80-shell-timeout.sh script checks if containers are running in a Kubernetes environment and then set a 10 minutes timeout to idle cli pods.cli containers use an SSH key injected by Lagoon or defined into SSH_PRIVATE_KEYenvironment variable.The included CLI tools are:
+composer version 1.9.0 (changeable via COMPOSER_VERSION and COMPOSER_HASH_SHA256)node.js verison 17 (as of Mar 2022)npmyarnmariadb-clientpostgresql-clientBy default this image ships with the nodejs-current package (v17 as of Mar 2022). If you need another version you can remove the current version and install the one of your choice. For example, to install Node.js 16, modify your dockerfile to include:
Some options are configurable via environment +variables. The php-fpm +environment variables also apply.
+| Name | +Default | +Description | +
|---|---|---|
| MARIADB_MAX_ALLOWED_PACKET | +64M | +Controls the max allowed packet for the MySql client. | +
The Lagoon php-fpm Docker image. Based on the official PHP Alpine images.
++PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation with some additional features useful for sites of any size, especially busier sites.
++
+- from https://php-fpm.org/
+FastCGI is a way of having server scripts execute time-consuming code just once instead of every time the script is loaded, reducing overhead.
+
Info
+This Dockerfile is intended to be used as a base for any PHP needs within Lagoon. This image itself does not create a web server, rather a php-fpm fastcgi listener. You may need to adapt the php-fpm pool config.
uselagoon/php-7.3-fpmuselagoon/php-7.4-fpmuselagoon/php-8.0-fpmuselagoon/php-8.1-fpmuselagoon/php-8.2-fpmAll PHP versions use their own Dockerfiles.
+Tip
+We stop updating End of Life (EOL) PHP images usually with the Lagoon release that comes after the officially communicated EOL date: https://www.php.net/supported-versions.php. Previous published versions will remain available.
+This image is prepared to be used on Lagoon. There are therefore some things are already done:
+fix-permissions, so this image will work with a random user./usr/local/etc/php/php.ini and /usr/local/etc/php-fpm.conf, plus all files within /usr/local/etc/php-fpm.d/ , are parsed through envplate with a container-entrypoint.PHP extensions.The included PHP config contains sensible values that will make the creation of PHP pools config easier. Here is a list of some of these. Check /usr/local/etc/php.ini, /usr/local/etc/php-fpm.conf for all of them:
| Value | +Details | +
|---|---|
max_execution_time = 900 |
+Changeable via PHP_MAX_EXECUTION_TIME. |
+
realpath_cache_size = 256k |
+For handling big PHP projects. | +
memory_limit = 400M |
+For big PHP projects (changeable via PHP_MEMORY_LIMIT). |
+
opcache.memory_consumption = 265 |
+For big PHP projects. | +
opcache.enable_file_override = 1 and opcache.huge_code_pages = 1 |
+For faster PHP. | +
display_errors = Off and display_startup_errors = Off |
+For sensible production values (changeable via PHP_DISPLAY_ERRORS and PHP_DISPLAY_STARTUP_ERRORS). |
+
upload_max_filesize = 2048M |
+For big file uploads. | +
apc.shm_size = 32m and apc.enabled = 1 |
+Changeable via PHP_APC_SHM_SIZE and PHP_APC_ENABLED. |
+
Also, php-fpm error logging happens in stderr.
💡 If you don't like any of these configs, you have three possibilities:
+fpm-pool config and set via php_admin_value and php_admin_flag.this documentation for Running PHP as an Apache module. This documentation refers to Apache, but it is also the case for php-fpm).Important:
+1. If you want to provide your own php-fpm pool, overwrite the file /usr/local/etc/php-fpm.d/www.conf with your own config, or rename this file if you want it to have another name. If you don't do that, the provided pool will be started!
+ 2. PHP values with the PHP_INI_SYSTEM changeable mode cannot be changed via an fpm-pool config. They need to be changed either via already provided environment variables or:
+3. Provide your own php.ini or php-fpm.conf file (this is the least preferred method).
This image is shipped with an fpm-pool config (php-fpm.d/www.conf) that creates an fpm-pool and listens on port 9000. This is because we try to provide an image which already covers most needs for PHP, so you don't need to create your own. You are welcome to do so if you like, though!
Here a short description of what this file does:
+dynamic and creates between 2-50 children.php-fpm pool children after 500 requests to prevent memory leaks.pong when making a fastcgi request to /ping (good for automated testing to check if the pool started).catch_workers_output = yes to see PHP errors.clear_env = no to be able to inject PHP environment variables via regular Docker environment variables.Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| NEWRELIC_ENABLED | +false | +Enable NewRelic performance monitoring, needs NEWRELIC_LICENSE be configured. |
+
| NEWRELIC_LICENSE | +(not set) | +NewRelic license to be used. Important: NEWRELIC_ENABLED needs to be set totrue in order for NewRelic to be enabled. |
+
| NEWRELIC_BROWSER_MONITORING_ENABLED | +true | +This enables auto-insertion of the JavaScript fragments for NewRelic browser monitoring. Important: NEWRELIC_ENABLED needs to be set totrue in order for NewRelic to be enabled. |
+
| NEWRELIC_DISTRIBUTED_TRACING_ENABLED | +false | +This enables distributed tracing. Important: NEWRELIC_ENABLED needs to be set totrue in order for NewRelic to be enabled. |
+
| PHP_APC_ENABLED | +1 | +Can be set to 0 to disable APC. |
+
| PHP_APC_SHM_SIZE | +32m | +The size of each shared memory segment given. | +
| PHP_DISPLAY_ERRORS | +Off | +Configures whether errors are printed or hidden. See php.net. | +
| PHP_DISPLAY_STARTUP_ERRORS | +Off | +Configures whether startup errors are printed or hidden. See php.net. | +
| PHP_ERROR_REPORTING | +Production E_ALL & ~E_DEPRECATED & ~E_STRICT Development: E_ALL & ~E_DEPRECATED & ~E_STRICT & ~E_NOTICE |
+The desired logging level you'd like PHP to use. See php.net. | +
| PHP_FPM_PM_MAX_CHILDREN | +50 | +The the maximum number of child processes. See php.net. | +
| PHP_FPM_PM_MAX_REQUESTS | +500 | +The number of requests each child process should execute before re-spawning. See php.net. | +
| PHP_FPM_PM_MAX_SPARE_SERVERS | +2 | +The desired maximum number of idle server processes. See php.net. | +
| PHP_FPM_PM_MIN_SPARE_SERVERS | +2 | +The desired minimum number of idle server processes. See php.net. | +
| PHP_FPM_PM_PROCESS_IDLE_TIMEOUT | +60s | +The number of seconds after which an idle process will be killed. See php.net. | +
| PHP_FPM_PM_START_SERVERS | +2 | +The number of child processes created on startup. See php.net. | +
| PHP_MAX_EXECUTION_TIME | +900 | +Maximum execution time of each script, in seconds. See php.net. | +
| PHP_MAX_FILE_UPLOADS | +20 | +The maximum number of files allowed to be uploaded simultaneously. See php.net. | +
| PHP_MAX_INPUT_VARS | +2000 | +How many input variables will be accepted. See php.net. | +
| PHP_MEMORY_LIMIT | +400M | +Maximum amount of memory a script may consume. See php.net. | +
| XDEBUG_ENABLE | +(not set) | +Set to true to enable xdebug extension. |
+
| BLACKFIRE_ENABLED | +(not set) | +Set to true to enable blackfire extension. |
+
| BLACKFIRE_SERVER_ID | +(not set) | +Set to Blackfire Server ID provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true. |
+
| BLACKFIRE_SERVER_TOKEN | +(not set) | +Set to Blackfire Server Token provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true. |
+
| BLACKFIRE_LOG_LEVEL | +3 | +Change the log level of the blackfire agent. Available values: log verbosity level (4: debug, 3: info, 2: warning, 1: error) See blackfire.io. |
+
The Lagoon PostgreSQL Docker image. Based on the official PostgreSQL Alpine images.
+uselagoon/postgres-11uselagoon/postgres-12uselagoon/postgres-13uselagoon/postgres-14uselagoon/postgres-15Tip
+We stop updating EOL PostgreSQL images usually with the Lagoon release that comes after the officially communicated EOL date: https://www.postgresql.org/support/versioning
+The default exposed port of Postgres containers is port 5432.
To allow Lagoon to select the best way to run the Postgres container, use lagoon.type: postgres - this allows DBaaS operator to provision a cloud database if available in the cluster. Use lagoon.type: postgres-single to specifically request Postgres in a container. Persistent storage is always provisioned for postgres containers at /var/lib/postgresql/data.
docker-compose.yml snippet#postgres:
+ image: uselagoon/postgres-14-drupal:latest
+ labels:
+ # tells Lagoon this is a Postgres database
+ lagoon.type: postgres
+ ports:
+ # exposes the port 5432 with a random local port
+ # find it with `docker-compose port postgres 5432`
+ - "5432"
+ volumes:
+ # mounts a named volume at the default path for Postgres
+ - db:/var/lib/postgresql/data
+If you have SQL statements that need to be run immediately after container startup to initialize the database, you can place those .sql files in the container's docker-entrypoint-initdb.d directory. Any .sql files contained in that directory are run automatically at startup, as part of bringing the PostgreSQL container up.
Warning
+These scripts are only run if the container is started with an empty database.
+The Lagoon python Docker image. Based on the official Python Alpine images.
uselagoon/python-2.7uselagoon/python-3.7uselagoon/python-3.8uselagoon/python-3.9uselagoon/python-3.10uselagoon/python-3.11Tip
+We stop updating and publishing EOL Python images usually with the Lagoon release that comes after the officially communicated EOL date: https://devguide.python.org/versions/#versions. Previous published versions will remain available.
+The default exposed port of Python containers is port 8800.
Persistent storage is configurable in Lagoon, using the lagoon.type: python-persistent. See the docs for more info
Use the following labels in your docker-compose.yml file to configure it:
+lagoon.persistent = use this to define the path in the container to use as persistent storage - e.g. /app/files
+lagoon.persistent.size = this to tell Lagoon how much storage to assign this path
If you have multiple services that share the same storage, use this
+lagoon.persistent.name = (optional) use this to tell Lagoon to use the storage defined in another named service
docker-compose.yml snippet#python:
+ build:
+ # this configures a build from a Dockerfile in the root folder
+ context: .
+ dockerfile: Dockerfile
+ labels:
+ # tells Lagoon this is a python service, configured with 500MB of persistent storage at /app/files
+ lagoon.type: python-persistent
+ lagoon.persistent: /app/files
+ lagoon.persistent.size: 500Mi
+ ports:
+ # local development only
+ # this exposes the port 8800 with a random local port
+ # find it with `docker-compose port python 8800`
+ - "8800"
+ volumes:
+ # local development only
+ # mounts a named volume (files) at the defined path for this service to replicate production
+ - files:/app/files
+The Lagoon RabbitMQ Dockerfile with management plugin installed. Based on the official rabbitmq:3-management image at docker-hub.
This Dockerfile is intended to be used to set up a standalone RabbitMQ queue broker, as well as a base image to set up a cluster with high availability queue support by default (Mirrored queues).
+By default, the RabbitMQ broker is started as single node. If you want to start a cluster, you need to use the rabbitmq-cluster Docker image, based on rabbitmq image plus the rabbitmq_peer_discovery_k8s plugin.
uselagoon/rabbitmqThis image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user./etc/rabbitmq/definitions.json is parsed through envplate with a container-entrypoint.policymust exist.In the definitions.json schema file, minimal entities are defined to make the
container run: virtualhost (vhost), username , and password to access management
UI, permissions , and policies.
By default, a policy called lagoon-ha is created at startup, but it is not active because it doesn't match any queue's name pattern (see default Environment Variables).
"policies":[
+ {"vhost":"${RABBITMQ_DEFAULT_VHOST}","name":"lagoon-ha","pattern":"${RABBITMQ_DEFAULT_HA_PATTERN}", "definition":{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic","ha-sync-batch-size":5}}
+ ]
+By default, the ha-mode is set to exactly which controls the exact number of mirroring nodes for a queue (mirrors). The number of nodes is controller by ha-params.
For further information and custom configuration, please refer to official RabbitMQ documentation.
+Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| RABBITMQ_DEFAULT_USER | +guest | +Username for management UI access. | +
| RABBITMQ_DEFAULT_PASS | +guest | +Password for management UI access. | +
| RABBITMQ_DEFAULT_VHOST | +/ | +RabbitMQ main virtualhost. | +
| RABBITMQ_DEFAULT_HA_PATTERN | +^$ | +Regular expression to match for mirrored queues. | +
Lagoon Redis image Dockerfile, based on offical redis:alpine image.
This Dockerfile is intended to be used to set up a standalone Redis ephemeral server by default.
+uselagoon/redis-5 or uselagoon/redis-5-persistentuselagoon/redis-6 or uselagoon/redis-6-persistentuselagoon/redis-7 or uselagoon/redis-7-persistentThere are 2 different flavors of Redis Images: Ephemeral and Persistent.
+The ephemeral image is intended to be used as an in-memory cache for applications and will not retain data across container restarts.
+When being used as an in-memory (RAM) cache, the first thing you might want to tune if you have large caches is to adapt the MAXMEMORY variable. This variable controls the maximum amount of memory (RAM) which redis will use to store cached items.
The persistent Redis image will persist data across container restarts and can be used for queues or application data that will need persistence.
+We don't typically suggest using a persistent Redis for in-memory cache scenarios as this might have unintended side-effects on your application while a Redis container is restarting and loading data from disk.
+This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissionsso this image will work with a random user./etc/redis/* are templated using envplate via a container-entrypoint.redis.conf configuration file#The image ships a default Redis configuration file, optimized to work on Lagoon.
+Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| DATABASES | +-1 | +Default number of databases created at startup. | +
| LOGLEVEL | +notice | +Define the level of logs. | +
| MAXMEMORY | +100mb | +Maximum amount of memory. | +
| MAXMEMORYPOLICY | +allkeys-lru | +The policy to use when evicting keys if Redis reaches its maximum memory usage. | +
| REDIS_PASSWORD | +disabled | +Enables authentication feature. | +
By building on the base image you can include custom configuration. +See https://raw.githubusercontent.com/antirez/redis/4.0/redis.conf for full documentation of the Redis configuration file.
+Based on the Lagoon redis image, the Lagoon redis-persistent Docker image is intended for use when the Redis service must be utilized in persistent mode (ie. with a persistent volume where keys will be saved to disk).
It differs from redis only with the FLAVOR environment variable, which will use the respective Redis configuration according to the version of redis in use.
The Lagoon Redis images all come pre-loaded with the redis-cli command, which allows for querying the Redis service for information and setting config values dynamically. To use this utility, you can simply SSH into your Redis pod by using the instructions [here] (../using-lagoon-advanced/ssh.md) with redis as the pod value then run it from the terminal once you've connected.
By default, the Lagoon redis images are set to use the allkeys-lru policy. This policy will alow ANY keys stored in Redis to be evicted if/when the Redis service hits its maxmemory limit according to when the key was least recently used.
For typical installations, this is the ideal configuration, as Drupal may not set a TTL value for each key cached in Redis. If the maxmemory-policy is set to something like volatile-lru and Drupal doesn't provide these TTL tags, this would result in the Redis container filling up, being totally unable to evict ANY keys, and ceasing to accept new cache keys at all.
More information on Redis' maxmemory policies can be found in Redis' official documentation.
+Proceed with Caution
+Changing this setting can lead to Redis becoming completely full and cause outages as a result.
+maxmemory value#Finding the optimal amount of memory to give Redis can be quite the difficult task. Before attempting to tune your Redis cache's memory size, it is prudent to let it run normally for as long as practical, with at least a day of typical usage being the ideal minimum timeframe.
+There are a few high level things you can look at when tuning these memory values:
+50%, you might consider lowering the maxmemory value by 25%.50% and 75%, things are running just fine.75%, then it's worth looking at other variables to see if maxmemory needs to be increased.95% is a fairly good indicator that your redis needs a higher maxmemory setting.The following commands can be used to view information about the Redis service:
+redis-cli inforedis-cli info memoryredis-cli info keyspaceredis-cli info statsIt is also possible to set values for the Redis service dynamically without a restart of the Redis service. It is important to note that these dynamically set values will not persist if the pod is restarted (which can happen as a result of a deployment, maintenance, or even just being shuffled from one node to another).
+maxmemory config value dynamically to 500mb: config set maxmemory 500mbmaxmemory-policy config value dynamically to volatile-lru: config set maxmemory-policy volatile-lruThe Lagoon ruby Docker image. Based on the official Python Alpine images.
uselagoon/ruby-3.0uselagoon/ruby-3.1uselagoon/ruby-3.2Tip
+We stop updating and publishing EOL Ruby images usually with the Lagoon release that comes after the officially communicated EOL date: https://www.ruby-lang.org/en/downloads/releases/. Previous versions will remain available.
+The default exposed port of ruby containers is port 3000.
Lagoon has no "pre-defined" type for Ruby services, they should be configured with the lagoon.type: generic and a port set with lagoon.port: 3000
docker-compose.yml snippet#ruby:
+ build:
+ # this configures a build from a Dockerfile in the root folder
+ context: .
+ dockerfile: Dockerfile
+ labels:
+ # tells Lagoon this is a generic service, configured to expose port 3000
+ lagoon.type: generic
+ lagoon.port: 3000
+ ports:
+ # local development only
+ # this exposes the port 3000 with a random local port
+ # find it with `docker-compose port ruby 3000`
+ - "3000"
+The Lagoon Solr image Dockerfile. Based on the official solr:<version>-alpine images.
This Dockerfile is intended to be used to set up a standalone Solr server with an initial core mycore.
uselagoon/solr-7uselagoon/solr-8This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user.10-solr-port.sh script to fix and check Solr port.20-solr-datadir.sh script to check if Solr config is compliant for Lagoon. This sets directory paths, and configures the correct lock type.Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| SOLR_JAVA_MEM | +512M | +Default Java HEAP size (ie. SOLR_JAVA_MEM="-Xms10g -Xmx10g"). |
+
| SOLR_DATA_DIR | +/var/solr | +Path of the solr data dir. Be careful, changing this can cause data loss! | +
| SOLR_COPY_DATA_DIR_SOURCE | +(not set) | +Path which the entrypoint script of solr will use to copy into the defined SOLR_DATA_DIR, this can be used for prepopulating the Solr with a core. The scripts expects actual Solr data files! Plus it only copies data if the destination does not already have a solr core in it. |
+
The Lagoon Varnish Docker images. Based on the official Varnish package
uselagoon/varnish-5uselagoon/varnish-6uselagoon/varnish-7vbox-dynamic - Dynamic backends from DNS lookups and service discovery from SRV records.vbox-bodyaccess - Varnish vmod that lets you access the request body.This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user.default.vcl configuration file#The image ships a default vcl configuration file, optimized to work on Lagoon. Some options are configurable via environments variables (see Environment Variables).
Some options are configurable via environment +variables.
+| Environment Variable | +Default | +Description | +
|---|---|---|
| VARNISH_BACKEND_HOST | +NGINX | +Default backend host. | +
| VARNISH_BACKEND_PORT | +8080 | +Default listening Varnish port. | +
| VARNISH_SECRET | +lagoon_default_secret | +Varnish secret used to connect to management. | +
| LIBVMOD_DYNAMIC_VERSION | +5.2 | +Default version of vmod-dynamic module. |
+
| LIBVMOD_BODYACCESS_VERSION | +5.0 | +Default version of vmod-bodyaccess module. |
+
| HTTP_RESP_HDR_LEN | +8k | +Maximum length of any HTTP backend response header. | +
| HTTP_RESP_SIZE | +32k | +Maximum number of bytes of HTTP backend response we will deal with. | +
| NUKE_LIMIT | +150 | +Maximum number of objects we attempt to nuke in order to make space for an object body. | +
| CACHE_TYPE | +malloc | +Type of varnish cache. | +
| CACHE_SIZE | +100M | +Cache size. | +
| LISTEN | +8080 | +Default backend server port. | +
| MANAGEMENT_LISTEN | +6082 | +Default management listening port. | +
Unfortunately, Drush 9 does not provide the ability to inject dynamic site aliases like Drush 8 did. We are working with the Drush team to implement this again. In the meantime, we have a workaround that allows you to use Drush 9 with Lagoon.
+Drush 9 provides a new command, drush site:alias-convert , which can convert Drush 8-style site aliases over to the Drush 9 YAML site alias style. This will create a on- time export of the site aliases currently existing in Lagoon, and save them in /app/drush/sites . These are then used when running a command like drush sa.
In order to be able to use drush site:alias-convert , you need to do the following:
aliases.drushrc.php inside the drush folder to lagoon.aliases.drushrc.php.You can now convert your Drush aliases by running the following command in your project using the cli container:
It's good practice to commit the resulting YAML files into your Git repository, so that they are in place for your fellow developers.
+In Drush 9, all site aliases are prefixed with a group. In our case, this is lagoon. You can show all site aliases with their prefix via:
and to use them:
+ +If a new environment in Lagoon has been created, you can run drush site:alias-convert to update the site aliases file. If running this command does not update lagoon.site.yml, try deleting lagoon.site.yml first, and then re-run drush site:alias-convert.
rsync from local to remote environments#If you would like to sync files from a local environment to a remote environment, you need to pass additional parameters:
+drush rsync @self:%files @lagoon.main:%files -- --omit-dir-times --no-perms --no-group --no-owner --chmod=ugo=rwX
+This also applies to syncing one remote environment to another, if you're not using the Lagoon tasks UI to copy files between environments.
+For example, if you wanted to sync the files from @lagoon.main to @lagoon.dev , and ran drush rsync @lagoon.main @lagoon.dev locally, without the extra parameters, you would probably run into a "Cannot specify two remote aliases" error.
To resolve this, you would first need to SSH into your destination environment drush @lagoon.dev ssh, and then execute the rsync command with parameters similar to the above:
drush rsync @lagoon.main:%files @self:%files -- --omit-dir-times --no-perms --no-group --no-owner --chmod=ugo=rwX
+This is not necessary if you rsync from a remote to a local environment.
Also, we're working with the Drush maintainers to find a way to inject this automatically.
+ + + + + + + + +
In order to make your first deployment a successful one, please make sure that your Drupal Project is Lagoonized and you have set up the project in Lagoon. If not, don't worry! Follow the Step-by-Step Guide which show you how this works.
+With Lagoon, you create a new deployment by pushing into a branch that is configured to be deployed.
+If you don't have any new code to push, don't worry, you can run
+ +This will trigger a push, and the Git hosting will inform Lagoon about this push via the configured webhook.
+If all is correct, you will see a notification in your configured chat system (this is configured by your friendly Lagoon administrator):
+
This tells you that Lagoon has just started to deploy your code. Depending on the size of the codebase and amount of containers, this will take a couple of seconds. Just relax. If you'd like to know what's happening now, check out the Build and Deploy Process of Lagoon.
+You can also check your Lagoon UI to see the progress of any deployment (your Lagoon administrator has the info).
+Depending on the post-rollout tasks defined in .lagoon.yml , you might have run some tasks like drush updb or drush cr. These Drush tasks depend on a database existing within the environment, which obviously does not exist yet. Let's fix that! Keep reading.
With full Drush site alias support in Lagoon, you can synchronize a local database with the remote Lagoon environment.
+Warning
+You may have to tell pygmy about your public keys before the next step.
+If you get an error like Permission denied (publickey), check out the documentation here: pygmy - adding ssh keys
First let's make sure that you can see the Drush site aliases:
+ +This should return your just deployed environment (let's assume you just pushed into develop):
With this we can now synchronize the local database (which is represented in Drush via the site alias @self) with the remote one (@develop):
You should see something like:
+[drupal-example]cli-drupal:/app$ drush sql-sync @self @develop
+You will destroy data in ssh.lagoon.amazeeio.cloud/drupal and replace with data from drupal.
+Do you really want to continue? (y/n): y
+Starting to dump database on Source. [ok]
+Database dump saved to /home/drush-backups/drupal/20180227075813/drupal_20180227_075815.sql.gz [success]
+Starting to discover temporary files directory on Destination. [ok]
+You will delete files in drupal-example-develop@ssh.lagoon.amazeeio.cloud:/tmp/drupal_20180227_075815.sql.gz and replace with data from /home/drush-backups/drupal/20180227075813/drupal_20180227_075815.sql.gz
+Do you really want to continue? (y/n): y
+Copying dump file from Source to Destination. [ok]
+Starting to import dump file onto Destination database.
+Now let's try another deployment, again an empty push:
+ +This time all should be green:
+
Click on the links in the notification, and you should see your Drupal site loaded in all its beauty! It will probably not have images yet, which we will handle in Step 6.
+If it is still failing, check the logs link for more information.
+You probably guessed it: we can do it with Drush:
+ +It should show you something like:
+[drupal-example]cli-drupal:/app$ drush rsync @self:%files @develop:%files
+You will delete files in drupal-example-develop@ssh.lagoon.amazeeio.cloud:/app/web/sites/default/files and replace with data from /app/web/sites/default/files/
+Do you really want to continue? (y/n): y
+In some cases, though, it might not look correct, like here:
+[drupal-example]cli-drupal:/app$ drush rsync @self:%files @develop:%files
+You will delete files in drupal-example-develop@ssh.lagoon.amazeeio.cloud:'/app/web/%files' and replace with data from '/app/web/%files'/
+Do you really want to continue? (y/n):
+The reason for that is that the Drupal cannot resolve the path of the files directory. This most probably has to do that the Drupal is not fully configured or has a missing database. For a workaround you can use drush rsync @self:sites/default/files @develop:sites/default/files, but we suggest that you actually check your local and remote Drupal (you can test with drush status to see if the files directory is correctly configured).
As soon as Lagoon is done building and deploying it will send a second notification to the chat system, like so:
+
This tells you:
+That's it! We hope that wasn't too hard - making devOps accessible is what we are striving for.
+That's the beauty of Lagoon: it's exactly the same: Push the branch name you defined to be your production branch and that one will be deployed.
+Did the deployment fail? Oh no! But we're here to help:
+logs link in the error notification. It will tell you where in the deployment process the failure happened.Lagoon was built to host Drupal sites (no, seriously, it was - at least initially!)
+In this section you'll find more information on the various services that have been customised for use with Drupal.
+drupal_integrations Drupal scaffolding package#The drupal_integrations package, available on pacakagist extends Drupal's core-composer-scaffold for use on Lagoon. It also provides additional Drush command drush la to retreive the Drush aliases for your Lagoon project.
lagoon-logs Drupal module#The lagoon_logs module, availalble on drupal.org provides zero-configuration logging for Drupal on Lagoon.
7, 8 or 9 siteUse Composer to get the latest version of the module:
+ +You will need to enable the following modules:
+fastlyfastlypurgerhttp_cache_control (2.x)purgepurge_ui (technically optional, but this is really handy to have enabled on production)purge_processor_lateruntimepurge_processor_cronpurge_queuer_coretagspurge_drush (useful for purge via Drush, here is a list of commands)Configure the Fastly service ID and API token. You can use runtime environment variables, or you can edit the settings form found at /admin/config/services/fastly:
FASTLY_API_TOKENFASTLY_API_SERVICEA site ID is required, the module will generate one for you when you first install it. The idea behind the site ID is that it is a unique string which is appended as a cache tag on all requests. Thus, you are able to purge a single site from Fastly, even though multiple sites may flow through the same service in Fastly.
A 4 character cache tag is plenty for most sites, a 5 character cache tag is likely better for sites with millions of entities (to reduce cache tag collisions).
Set the options to what makes sense for your site. Minimum 1 hour (3600), maximum 1 week 604800). Generally something like the following will be fine:
Optionally configure the webhooks (so you can ping Slack for instance when a cache purge is sent).
+Visit the purge page /admin/config/development/performance/purge
Set up the following options:
+What this means is that we will be using Drupal's built in core tag queuer (add tags to the queue), the queue will be stored in the database (default), and the queue will be processed by
+In order for the cron processor to run, you need to ensure that cron is running on your site. Ideally every minute. You can manually run it in your cli pod, to ensure that purge_processor_cron_cron() is being executed without errors.
[drupal8]production@cli-drupal:/app$ drush cron -v
+ ...
+ [notice] Starting execution of purge_processor_cron_cron(), execution of node_cron() took 21.16ms.
+The Late runtime processor will run in hook_exit() for every page load, this can be useful to process the purges nearly as quickly as they come into the queue.
By having both, you guarantee that purges happen as soon as possible.
+Out of the box, Drupal does not have the power to set different cache lifetimes in the browser vs in Fastly. So if you do set long cache lifetimes in Drupal, often end users will not see them if their browser has cached the page. If you install the 2.x version of the HTTP Cache Control module, this will give you a lot more flexibility on what caches and for how long.
For most sites, a sensible default could be
+Note
+This relies on your site having accurate cache tags represented for all the content that exists on the page.
+Use this function: (works in Linux and Mac OSX)
+function curlf() { curl -sLIXGET -H 'Fastly-Debug:1' "$@" | grep -iE 'X-Cache|Cache-Control|Set-Cookie|X-Varnish|X-Hits|Vary|Fastly-Debug|X-Served|surrogate-control|surrogate-key' }
+$ curlf https://www.example-site-fastly.com
+cache-control: max-age=601, public, s-maxage=2764800
+surrogate-control: max-age=2764800, public, stale-while-revalidate=3600, stale-if-error=3600
+fastly-debug-path: (D cache-wlg10427-WLG 1612906144) (F cache-wlg10426-WLG 1612906141) (D cache-fra19179-FRA 1612906141) (F cache-fra19122-FRA 1612906141)
+fastly-debug-ttl: (H cache-wlg10427-WLG - - 3) (M cache-fra19179-FRA - - 0)
+fastly-debug-digest: 1118d9fefc8a514ca49d49cb6ece04649e1acf1663398212650bb462ba84c381
+x-served-by: cache-fra19179-FRA, cache-wlg10427-WLG
+x-cache: MISS, HIT
+x-cache-hits: 0, 1
+vary: Cookie, Accept-Encoding
+From the above headers we can see that:
+2764800 seconds)If you ever want to remove a specific page from cache manually, there are ways to do this.
+For a single page, you do not need any authentication:
+ +For cache tags, you need to supply your API token for authentication:
+curl -XPOST -H "Fastly-Key:<Fastly API Key>" https://api.fastly.com/service/<serviceID>/purge/<surrogatekey>
+You can always find what your site ID cache tag is by using PHP
+php > var_dump(substr(base64_encode(md5('bananasite', true)), 0, 4));
+string(4) "DTRk"
+So you can purge your entire site from Fastly fairly easily.
+We configure Fastly to send the actual client IP back on the HTTP header True-Client-IP, you can make Drupal respect this header with the following changes in settings.php:
$settings['reverse_proxy'] = TRUE;
+$settings['reverse_proxy_header'] = 'HTTP_TRUE_CLIENT_IP';
+ fastly:
+ fastly:purge:all (fpall) Purge whole service.
+ fastly:purge:key (fpkey) Purge cache by key.
+ fastly:purge:url (fpurl) Purge cache by Url.
+settings.php:Note
+This document assumes the following:
+- You are using Docker.
+- You are using a standard Amazee/Lagoon project with a docker-compose.yml file.
- You are on a Mac - it should work for other operating systems but folder structure and some configuration settings may be different.
+/core/phpunit.xml.dist file to /core/phpunit.xmlEdit* /core/phpunit.xml and fill in the following variables:
mysql://drupal:drupal@mariadb:3306/drupal#db<PROJECT_URL>+Docker for Mac
Add a new CLI interpreter:
+... and then +<DOCKER>./docker-compose.ymlcliConnect to existing container ('docker-compose exec')<ROOT_PATH>/app
Add Remote Interpreter:
++ and select PHPUnit by Remote Interpreter<CLI_INTERPRETER><PROJECT_ROOT> -> /appUse Composer autoloader/app/vendor/autoload.php/app/web/core/phpunit.xml
1. Test scope: Defined in the configuration file
2. Interpreter: <CLI_INTERPRETER>
Note
+If you are not on a Mac, this may vary.
+$ docker-compose up -dNow you have the above configuration set up it should be as straightforward as going to the test you want to run and pressing the green arrow!
+Once you press this PhpStorm will use Docker to enter the CLI container, then start running PHPUnit based upon the config.
+
Learn about MariaDB with Drupal
+Documentation on the MariaDB-Drupal image.
+Documentation on the plain MariaDB image (the MariaDB-Drupal image is built on this).
+Learn about Redis with Drupal.
+Documentation on the Redis-persistent image.
+Documentation on the Solr-Drupal image.
+Documentation on the plain Solr image (the Solr-Drupal image is built on this).
+Learn about Varnish with Drupal
+Documentation on the Varnish-Drupal image.
+Documentation on the plain Varnish image (the Varnish-Drupal image is built on this).
+ + + + + + + + +The Lagoon mariadb-drupal Docker image Dockerfile is a customized mariadb image to use within Drupal projects in Lagoon. It differs from the mariadb image only for initial database setup, made by some environment variables:
| Environment Variable | +Default | +Description | +
|---|---|---|
MARIADB_DATABASE |
+drupal | +Drupal database created at startup. | +
MARIADB_USER |
+drupal | +Default user created at startup. | +
MARIADB_PASSWORD |
+drupal | +Password of default user created at startup. | +
If the LAGOON_ENVIRONMENT_TYPE variable is set to production, performances are set accordingly by using MARIADB_INNODB_BUFFER_POOL_SIZE=1024 and MARIADB_INNODB_LOG_FILE_SIZE=256.
During the course of development, it may be necessary to enable either query logging or slow query logging. To do so, set the environment variables MARIADB_LOG_SLOW or MARIADB_LOG_QUERIES. This can be done in docker-compose.yml.
If you would like to connect to your MySQL database inside the Docker container with an external tool like Sequel Pro, MySQL Workbench, HeidiSQL, DBeaver, plain old mysql-cli or anything else, here's how to get the IP and port info.
By default, Docker assigns a randomly published port for MySQL during each container start. This is done to prevent port collisions.
+To get the published port via docker:
Run: docker port [container_name].
Or via docker-compose inside a Drupal repository:
Run: docker-compose port [service_name] [interal_port].
During development, if you are using an external database tool, it may become cumbersome to continually check and set the MySQL connection port.
+To set a static port, edit your service definition in your docker-compose.yml.
mariadb:
+ ...
+ ports:
+ - "33772:3306" # Exposes port 3306 with a 33772 on the host port. Note by doing this you are responsible for managing port collisions`.
+Warning
+By setting a static port you become responsible for managing port collisions.
+Now you can use these details to connect to whatever database management tool you'd like.
+| + | Linux | +OS X | +
|---|---|---|
| IP/Host | +IP from container | +docker.amazee.io |
+
| Port | +Published port from container | +Published port from container | +
| Username | +drupal |
+drupal |
+
| Password | +drupal |
+drupal |
+
| Database | +drupal |
+drupal |
+
The Lagoon nginx-drupal Docker image. Optimized to work with Drupal. Based on Lagoon nginx image.
This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user.drupal.conf 's configuration file as clean and customizable as possible, we added include directives in the main sections of the file:server, location /, location @drupal and location @php.Drupal.conf customization.drupal.conf)#The image includes a full NGINX working configuration for Drupal 7, 8 and 9. It includes some extra functionalities like:
+humanstxt Drupal module.robotstxt Drupal module.vagrant directory for local development.Drupal.conf customization#The drupal.conf file is a customized version of the nginx configuration file, optimized for Drupal. Customers have different ways of customizing it:
*.conf files.The drupal.conf file is divided into several sections. The sections we've included in our customizations are:
serverlocation /location @drupallocation @php.For each of this section, there are two includes:
+*_prepend.conf*_append.confHere what the location @drupal section looks like:
location @drupal {
+ include /etc/nginx/conf.d/drupal/location_drupal_prepend*.conf;
+
+ include /etc/nginx/fastcgi.conf;
+ fastcgi_param SCRIPT_NAME /index.php;
+ fastcgi_param SCRIPT_FILENAME $realpath_root/index.php;
+ fastcgi_pass ${NGINX_FASTCGI_PASS:-php}:9000;
+
+ include /etc/nginx/conf.d/drupal/location_drupal_append*.conf;
+}
+This configuration allows customers to create files called location_drupal_prepend.conf and location_drupal_append.conf, where they can put all the configuration they want to insert before and after the other statements.
Those files, once created, MUST exist in the nginx container, so add them to Dockerfile.nginx like so:
COPY location_drupal_prepend.conf /etc/nginx/conf.d/drupal/location_drupal_prepend.conf
+RUN fix-permissions /etc/nginx/conf.d/drupal/location_drupal_prepend.conf
+If you're using the core Statistics module, you may run into an issue that needs a quick configuration change.
+With the default NGINX configuration, the request to the tracking endpoint /core/modules/statistics/statistics.php is denied (404).
This is related to the default NGINX configuration:
+ +To fix the issue, we instead define a specific location rule and inject this as a location prepend configuration:
+## Allow access to to the statistics endpoint.
+location ~* ^(/core/modules/statistics/statistics.php) {
+ try_files /dev/null @php;
+}
+And copy this during the NGINX container build:
+ + + + + + + + + +The Lagoon php-cli-drupal Docker image is optimized to work with Drupal. It is based on the Lagoon php-cli image, and has all the command line tools needed for the daily maintenance of a Drupal website:
drushdrupal consoledrush launcher (which will fallback to Drush 8 if there is no site installed Drush found)uselagoon/php-7.4-cli-drupaluselagoon/php-8.0-cli-drupaluselagoon/php-8.1-cli-drupalAll PHP versions use their own Dockerfiles.
+This image is prepared to be used on Lagoon. There are therefore some things already done:
+fix-permissions, so this image will work with a random user.We recommend using Redis for internal caching. Add the Redis service to docker-compose.yaml.
redis:
+ image: uselagoon/redis-5
+ labels:
+ lagoon.type: redis
+ << : *default-user # Uses the defined user from top.
+ environment:
+ << : *default-environment
+Also, to configure Redis, add the following to your settings.php.
if(getenv('LAGOON')){
+ $conf['redis_client_interface'] = 'PhpRedis';
+ $conf['redis_client_host'] = 'redis';
+ $conf['lock_inc'] = 'sites/all/modules/contrib/redis/redis.lock.inc';
+ $conf['path_inc'] = 'sites/all/modules/contrib/redis/redis.path.inc';
+ $conf['cache_backends'][] = 'sites/all/modules/contrib/redis/redis.autoload.inc';
+ $conf['cache_default_class'] = 'Redis_Cache';
+ $conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
+ $conf['cache_class_cache_field'] = 'DrupalDatabaseCache';
+ }
+Depending on file system structure, the module paths may need to be updated.
+The Drupal 8 config is largely stock. Notably, Redis is disabled while Drupal is being installed.
+if (getenv('LAGOON')){
+ $settings['redis.connection']['interface'] = 'PhpRedis';
+ $settings['redis.connection']['host'] = getenv('REDIS_HOST') ?: 'redis';
+ $settings['redis.connection']['port'] = getenv('REDIS_SERVICE_PORT') ?: '6379';
+ $settings['cache_prefix']['default'] = getenv('LAGOON_PROJECT') . '_' . getenv('LAGOON_GIT_SAFE_BRANCH');
+
+ // Do not set the cache during installations of Drupal.
+ if (!drupal_installation_attempted() && extension_loaded('redis')) {
+ $settings['cache']['default'] = 'cache.backend.redis';
+
+ // And allows to use it without the Redis module being enabled.
+ $class_loader->addPsr4('Drupal\\redis\\', 'modules/contrib/redis/src');
+
+ $settings['bootstrap_container_definition'] = [
+ 'parameters' => [],
+ 'services' => [
+ 'redis.factory' => [
+ 'class' => 'Drupal\redis\ClientFactory',
+ ],
+ 'cache.backend.redis' => [
+ 'class' => 'Drupal\redis\Cache\CacheBackendFactory',
+ 'arguments' => ['@redis.factory', '@cache_tags_provider.container', '@serialization.phpserialize'],
+ ],
+ 'cache.container' => [
+ 'class' => '\Drupal\redis\Cache\PhpRedis',
+ 'factory' => ['@cache.backend.redis', 'get'],
+ 'arguments' => ['container'],
+ ],
+ 'cache_tags_provider.container' => [
+ 'class' => 'Drupal\redis\Cache\RedisCacheTagsChecksum',
+ 'arguments' => ['@redis.factory'],
+ ],
+ 'serialization.phpserialize' => [
+ 'class' => 'Drupal\Component\Serialization\PhpSerialize',
+ ],
+ ],
+ ];
+ }
+}
+Redis can also be configured as a persistent backend.
+redis:
+ image: uselagoon/redis-5-persistent
+ labels:
+ lagoon.type: redis-persistent
+ environment:
+ << : *default-environment
+Environment variables are meant to store some common information about Redis.
+| Environment Variable | +Default | +Description | +
|---|---|---|
LOGLEVEL |
+notice |
+Redis loglevel | +
DATABASES |
+1 |
+Number of databases | +
MAXMEMORY |
+100mb |
+Maximum memory usage of Redis | +
Here is a snippet to implement a Redis failover in case of the Redis container not being available (for example, during maintenance)
+The following is inserted into Drupal's active settings.php file.
if (getenv('LAGOON')) {
+ $contrib_path = is_dir('sites/all/modules/contrib') ? 'sites/all/modules/contrib' : 'sites/all/modules';
+ $redis = DRUPAL_ROOT . '/sites/all/modules/contrib/redis';
+
+ if (file_exists("$redis/redis.module")) {
+ require_once "$redis/redis.module";
+ $conf['redis_client_host'] = getenv('REDIS_HOST') ?: 'redis';
+ $conf['redis_client_port'] = getenv('REDIS_SERVICE_PORT') ?: 6379;
+ $conf['cache_prefix'] = getenv('REDIS_CACHE_PREFIX') ?: getenv('LAGOON_PROJECT') . '_' . getenv('LAGOON_GIT_SAFE_BRANCH');
+ try {
+ // Ensure that there is a connection to redis.
+ $client = Redis_Client::getClient();
+ $response = $client->ping();
+ if (!$response) {
+ throw new \Exception('Redis could be reached but is not responding correctly.');
+ }
+ $conf['redis_client_interface'] = 'PhpRedis';
+ $conf['lock_inc'] = $contrib_path . '/redis/redis.lock.inc';
+ $conf['path_inc'] = $contrib_path . '/redis/redis.path.inc';
+ $conf['cache_backends'][] = $contrib_path . '/redis/redis.autoload.inc';
+ $conf['cache_default_class'] = 'Redis_Cache';
+ } catch (\Exception $e) {
+ // Redis is not available for this request we should not configure the
+ // redis backend and ensure no cache is used. This will retry next
+ // request.
+ if (!class_exists('DrupalFakeCache')) {
+ $conf['cache_backends'][] = 'includes/cache-install.inc';
+ }
+ $conf['cache_default_class'] = 'DrupalFakeCache';
+ }
+ }
+}
+For Solr 5.5, 6.6 and 7.7, we ship the default schema files provided by the search_api_solr Drupal module. Add the Solr version you would like to use in your docker-compose.yml file, following our example.
To implement schema customizations for Solr in your project, look to how Lagoon creates our standard images.
+solr section of your docker-compose.yml file, replace image: amazeeio/solr:7.7 with:.lagoon/solr.solr.dockerfile.FROM amazeeio/solr:7.7
+
+COPY .lagoon/solr /solr-conf/conf
+
+RUN precreate-core drupal /solr-conf
+
+CMD ["solr-foreground"]
+The goal is to have your Solr configuration files exist at /solr-conf/conf in the image you are building.
To implement multiple cores, you will also need to ship your own Solr schema as above. The only change needed is to the CMD of the Dockerfile - repeat the pattern of precreate-core corename /solr-conf/ ; for each core you require.
We suggest using Drupal with a Varnish reverse proxy. Lagoon provides a varnish-drupal Docker image that has Varnish already configured with a Drupal Varnish config.
This Varnish config does the following:
+BAN and URIBAN which is used by the Drupal 8 purge module.utm_ and gclid from the URL parameter to prevent Google Analytics links from creating multiple cache objects.TL;DR: Check out the drupal8-advanced example in our examples repo, it ships with the needed modules and needed Drupal configuration.
+Note: many of these examples are on the same drupal-example-simple repo, but different branches/hashes. Be sure to get the exact branch from the examples list!
In order to fully use Varnish with Drupal 8 cache tags, you need to install the Purge and Varnish Purge modules. They ship with many submodules. We suggest installing at least the following:
+purgepurge_drushpurge_tokenspurge_uipurge_processor_cronpurge_processor_lateruntimepurge_queuer_coretagsvarnish_purgervarnish_purge_tagsGrab them all at once:
+composer require drupal/purge drupal/varnish_purge
+
+drush en purge purge_drush purge_tokens purge_ui purge_processor_cron purge_processor_lateruntime purge_queuer_coretags varnish_purger varnish_purge_tags
+Configuration > Development > Performance > Purge.Add purger.Varnish Bundled Purger (not the Varnish Purger, see the #Behind the Scenes section, for more information.).Configure.Lagoon Varnish sounds good.Configure it with:
+ +Save configuration.
That's it! If you'd like to test this locally, make sure you read the next section.
+There are a few other configurations that can be done:
+Internal Page Cache Drupal module with drush pmu page_cache. It can cause some weird double caching situations where only the Varnish cache is cleared, but not the internal cache, and changes appear very slowly to the users. Also, it uses a lot of cache storage on big sites.$config['system.performance']['cache']['page']['max_age'] in production.settings.php to 2628000. This tells Varnish to cache sites for up 1 month, which sounds like a lot, but the Drupal 8 cache tag system is so awesome that it will basically make sure that the Varnish cache is purged whenever something changes.Drupal setups on Lagoon locally have Varnish and the Drupal caches disabled as it can be rather hard to develop with all them set. This is done via the following:
+VARNISH_BYPASS=true environment variable in docker-compose.yml which tells Varnish to basically disable itself.$config['system.performance']['cache']['page']['max_age'] = 0 in development.settings.php).To test Varnish locally, change the following in docker-compose.yml:
VARNISH_BYPASS to false in the Varnish service section.LAGOON_ENVIRONMENT_TYPE to production in the x-environment section.docker-compose up -d , which restarts all services with the new environment variables.Now you should be able to test Varnish!
+ +Here is a short example assuming there is a node with the ID 1 and has the URL drupal-example.docker.amazee.io/node/1
curl -I drupal-example.docker.amazee.io/node/1 and look for these headers:X-LAGOON should include varnish which tells you that the request actually went through Varnish.Age: will be still 0 as Varnish has probably never seen this site before, and the first request will warm the varnish cache.X-Varnish-Cache will be MISS , also telling you that Varnish didn't find a previously cached version of this request.curl -I drupal-example.docker.amazee.io/node/1 again, and the headers should be:Age: will show you how many seconds ago the request has been cached. In our example it will probably something between 1-30, depending on how fast you are executing the command.X-Varnish-Cache will be HIT, telling you that Varnish successfully found a cached version of the request and returned that one to you.node/1 in Drupal.curl -I drupal-example.docker.amazee.io/node/1 , and the headers should the same as very first request:Age:0X-Varnish-Cache: MISSIf you come from other Drupal hosts or have done a Drupal 8 & Varnish tutorial before, you might have realized that there are a couple of changes in the Lagoon Drupal Varnish tutorial. Let's address them:
+Varnish Bundled Purger instead of Varnish Purger#The Varnish Purger purger sends a BAN request for each cache-tag that should be invalidated. Drupal has a lot of cache-tags, and this could lead to quite a large amount of requests sent to Varnish. Varnish Bundled Purger instead sends just one BAN request for multiple invalidations, separated nicely by pipe (|), which fits perfectly with the Varnish regular expression system of bans. This causes less requests and a smaller ban list table inside Varnish.
Purge Late runtime processor#Contradictory to the Varnish module in Drupal 7, the Drupal 8 Purge module has a slightly different approach to purging caches: It adds them to a queue which is then processed by different processors. Purge suggests using the Cron processor , which means that the Varnish cache is only purged during a cron run. This can lead to old data being cached by Varnish, as your cron is probably not configured to run every minute or so, and can result in confused editors and clients.
Instead, we suggest using the Purge Late runtime processor, which processes the queue at the end of each Drupal request. This has the advantage that if a cache-tag is added to the purge queue (because an editor edited a Drupal node, for example) the cache-tags for this node are directly purged. Together with the Varnish Bundled Purger, this means just a single additional request to Varnish at the very end of a Drupal request, which causes no noticeable processing time on the request.
Our Varnish configurations have full support for Ban Lurker. Ban Lurker helps you to maintain a clean cache and keep Varnish running smoothly. It is basically a small tool that runs through the Varnish ban list and compares them to the cached requests in the Varnish cache. Varnish bans are used to mark an object in the cache for purging. If Ban Lurker finds an item that should be "banned," it removes them from the cache and also removes the ban itself. Now any seldom-accessed objects with very long TTLs which would normally never be banned and just keep taking up cache space are removed and can be refreshed. This keeps the list of bans small and with that, less processing time for Varnish on each request. Check out the official Varnish post on Ban Lurker and some other helpful reading for more information.
Varnish doesn't cache? Or something else not working? Here a couple of ways to debug:
+ +drush p-debug-en to enable debug logging of the purge module. This should show you debugging in the Drupal log under admin/reports/dblog.Cache-Control: max-age=900, public header, where the 900 is what you configured in $config['system.performance']['cache']['page']['max_age'].VARNISH_BYPASS is not set to true (see docker-compose.yml and run docker-compose up -d varnish to make sure the environment variable is configured correctly).In order for Drupal to work with Lagoon, we need to teach Drupal about Lagoon and Lagoon about Drupal. This happens by copying specific YAML and PHP files into your Git repository.
+If you're working on a Drupal project, you can check out one of the various Drupal example projects in our examples repository. We have Drupal 8 and 9 and some variants of each depending on your needs, such as database types. Clone the repository that best suits your needs to get started!
+Here is a summary of the Lagoon- and Drupal-specific files you will find:
+.lagoon.yml - The main file that will be used by Lagoon to understand what should be deployed and many more things. This file has some sensible Drupal defaults. If you would like to edit or modify, please check the documentation for .lagoon.yml.docker-compose.yml, .dockerignore, and *.dockerfile (or Dockerfile) - These files are used to run your local Drupal development environment, they tell Docker which services to start and how to build them. They contain sensible defaults and many commented lines. We hope that it's well-commented enough to be self-describing. If you would like to find out more, see documentation for docker-compose.yml.sites/default/* - These .php and .yml files tell Drupal how to communicate with Lagoon containers both locally and in production. They also provide a straightforward system for specific overrides in development and production environments. Unlike other Drupal hosting systems, Lagoon never ever injects Drupal settings files into your Drupal. Therefore, you can edit them however you like. Like all other files, they contain sensible defaults and some commented parts.drush/aliases.drushrc.php - These files are specific to Drush and tell Drush how to talk to the Lagoon GraphQL API in order to learn about all site aliases there are.drush/drushrc.php - Some sensible defaults for Drush commands..gitignore Settings#Don't forget to make sure your .gitignore will allow you to commit the settings files.
Drupal is shipped with sites/*/settings*.php and sites/*/services*.yml in .gitignore. Remove that, as with Lagoon we don't ever have sensitive information in the Git repository.
WEBROOT in Drupal 8#Unfortunately the Drupal community has not decided on a standardized WEBROOT folder name. Some projects put Drupal within web, and others within docroot or somewhere else. The Lagoon Drupal settings files assume that your Drupal is within web, but if this is different for your Drupal, please adapt the files accordingly.
If you installed Drupal via composer, please check your composer.json and make sure that the name is NOT drupal/drupal, as this could confuse Drush and other tools of the Drupal universe, just rename it to something like myproject/drupal
docker-compose.yml#Don't forget to customize the values in lagoon-project & LAGOON_ROUTE with your site-specific name & the URL you'd like to access the site with. Here's an example:
x-environment:
+ &default-environment
+ LAGOON_PROJECT: *lagoon-project
+ # Route that should be used locally. If you are using pygmy, this route *must* end with .docker.amazee.io.
+ LAGOON_ROUTE: http://drupal-example.docker.amazee.io
+First, we need to build the defined images:
+ +This will tell docker-compose to build the Docker images for all containers that have a build: definition in the docker-compose.yml. Usually for Drupal this is the case for the cli, nginx and php images. We do this because we want to run specific build commands (like composer install) or inject specific environment variables (like WEBROOT) into the images.
Usually, building is not necessary every time you edit your Drupal code (as the code is mounted into the containers from your host), but rebuilding does not hurt. Plus, Lagoon will build the exact same Docker images during a deploy, so you can check that your build will also work during a deployment by just running docker-compose build again.
Now that the images are built, we can start the containers:
+ +This will bring up all containers. After the command is done, you can check with docker-compose ps to ensure that they are all fully up and have not crashed. If there is a problem, check the logs with docker-compose logs -f [servicename].
composer install (for Composer projects only)#In a local development environment, you probably want all dependencies downloaded and installed, so connect to the cli container and run composer install:
This might sound weird, as there was already a composer install executed during the build step, so let us explain:
docker-composer.yml mounts the whole folder into the the containers (this happens with .:/app:delegated in the volumes section). This also means that all dependencies installed during the Docker build are overwritten with the files on the host.require-dev in composer.json to exist as well, while on a production deployment they would just use unnecessary space. So we run composer install --no-dev in the Dockerfile and composer install manually.If everything went well, open the LAGOON_ROUTE defined in docker-compose.yml (for example http://drupal.docker.amazee.io) and you should be greeted by a nice Drupal error. Don't worry - that's ok right now, most important is that it tries to load a Drupal site.
If you get a 500 or similar error, make sure everything loaded properly with Composer.
+Finally it's time to install Drupal, but just before that we want to make sure everything works. We suggest using Drush for that:
+ +This should return something like:
+[drupal-example]cli-drupal:/app$ drush status
+[notice] Missing database table: key_value
+Drupal version : 8.6.1
+Site URI : http://drupal.docker.amazee.io
+Database driver : mysql
+Database hostname : mariadb
+Database port : 3306
+Database username : drupal
+Database name : drupal
+PHP binary : /usr/local/bin/php
+PHP config : /usr/local/etc/php/php.ini
+PHP OS : Linux
+Drush script : /app/vendor/drush/drush/drush
+Drush version : 9.4.0
+Drush temp : /tmp
+Drush configs : /home/.drush/drush.yml
+ /app/vendor/drush/drush/drush.yml
+Drupal root : /app/web
+Site path : sites/default
+Warning
+You may have to tell pygmy about your public key before the next step.
+If you get an error like Permission denied (publickey), check out the documentation here: pygmy - adding ssh keys
Now it is time to install Drupal (if instead you would like to import an existing SQL file, please skip to step 7, but we suggest you start with a clean Drupal installation in the beginning to be sure everything works).
+ +This should output something like:
+[drupal-example]cli-drupal:/app$ drush site-install
+You are about to DROP all tables in your 'drupal' database. Do you want to continue? (y/n): y
+Starting Drupal installation. This takes a while. Consider using the --notify global option.
+Installation complete. User name: admin User password: a7kZJekcqh
+Congratulations, you installed Drupal!
+Now you can visit the URL defined in LAGOON_ROUTE and you should see a fresh and clean installed Drupal site - Congrats!
If you already have an existing Drupal site, you probably want to import its database over to your local site.
+There are many different ways to create a database dump. If your current hosting provider has Drush installed, you can use the following:
+ +Now you have a dump.sql file that contains your whole database.
Copy this file into your Git repository and connect to the cli, and you should see the file in there:
[drupal-example]cli-drupal:/app$ ls -l dump.sql
+-rw-r--r-- 1 root root 5281 Dec 19 12:46 dump.sql
+Now you can drop the current database, and then import the dump.
+ +Verify that everything works with visiting the URL of your project. You should have a functional copy of your Drupal site!
+A Drupal site also needs the files directory. As the whole folder is mounted into the Docker containers, add the files into the correct folder (probably web/sites/default/files, sites/default/files or something similar). Remember what you've set as your WEBROOT - it may not be the same for all projects.
You are done with your local setup. The Lagoon team wishes happy Drupaling!
+ + + + + + + + +An example could be: www.example.com points to one Drupal site, while www.example.com/blog loads a blog built in another Drupal.
It would be possible to run both Drupals in a single Git repository and deploy it as a whole, but this workflow might not fit every team, and having separate Git repositories fits some situations better.
+The root application (in this example, the Drupal site for www.example.com), needs a couple of NGINX configs that will configure NGINX to be a reverse proxy to the subfolder applications:
location_prepend.conf#Create a file called location_prepend.conf in the root of your Drupal installation:
resolver 8.8.8.8 valid=30s;
+
+location ~ ^/subfolder {
+ # If $http_x_forwarded_proto is empty (If it is not set from an upstream reverseproxy).
+ # Aet it to the current scheme.
+ set_if_empty $http_x_forwarded_proto $scheme;
+
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
+ proxy_set_header X-Lagoon-Forwarded-Host $host;
+ # Will be used by downstream to know the original host.
+ proxy_set_header X-REVERSEPROXY $hostname;
+ proxy_set_header FORWARDED "";
+ # Unset FORWARDED because drupal8 gives errors if it is set.
+ proxy_set_header Proxy "";
+ # Unset Proxy because drupal8 gives errors if it is set.
+ proxy_ssl_server_name on;
+
+ # NGINX needs a variable set in order for the DNS resolution to work correctly.
+ set $subfolder_drupal_host "https://nginx-lagoonproject-${LAGOON_GIT_SAFE_BRANCH}.clustername.com:443";
+ # LAGOON_GIT_SAFE_BRANCH variable will be replaced during docker entrypoint.
+ proxy_pass $subfolder_drupal_host;
+ proxy_set_header Host $proxy_host;
+ # $proxy_host will be automatically generated by NGINX based on proxy_pass (it needs to be without scheme and port).
+
+ expires off; # make sure we honor cache headers from the proxy and not overwrite them
+Replace the following strings:
+/subfolder with the name of the subfolder you want to use. For example, /blog.nginx with the service that you want to point too in the subfolder project.lagoonproject with the Lagoon projectname of the subfolder project.Add the following to your NGINX Dockerfile (nginx.dockerfile or Dockerfile.nginx):
COPY location_prepend.conf /etc/nginx/conf.d/drupal/location_prepend.conf
+RUN fix-permissions /etc/nginx/conf.d/drupal/*
+Like the root application, we also need to teach the subfolder application (in this example, the Drupal installation for www.example.com/blog), that it is running under a subfolder. To do this, we create two files:
location_drupal_append_subfolder.conf#Create a file called location_drupal_append_subfolder.conf in the root of your subfolder Drupal installation:
# When injecting a script name that is prefixed with `subfolder`, Drupal will
+# render all URLs with `subfolder` prefixed
+fastcgi_param SCRIPT_NAME /subfolder/index.php;
+
+# If we are running via a reverse proxy, we inject the original HOST URL
+# into PHP. With this Drupal will render all URLs with the original HOST URL,
+# and not the current used HOST.
+
+# We first set the HOST to the regular host variable.
+fastcgi_param HTTP_HOST $http_host;
+# Then we overwrite it with `X-Lagoon-Forwarded-Host` if it exists.
+fastcgi_param HTTP_HOST $http_x_lagoon_forwarded_host if_not_empty;
+Replace /subfolder with the name of the subfolder you want to use. For example, /blog.
server_prepend_subfolder.conf#Create a file called server_prepend_subfolder.conf in the root of your subfolder Drupal installation:
# Check for redirects before we do the internal NGINX rewrites.
+# This is done because the internal NGINX rewrites uses `last`,
+# which instructs NGINX to not check for rewrites anymore (and
+# `if` is part of the redirect module).
+include /etc/nginx/helpers/010_redirects.conf;
+
+# This is an internal NGINX rewrite, it removes `/subfolder/`
+# from the requests so that NGINX handles the request as it would
+# have been `/` from the beginning.
+# The `last` flag is also important. It will cause NGINX not to
+# execute any more rewrites, because it would redirect forever
+# with the rewrites below.
+rewrite ^/subfolder/(.*) /$1 last;
+
+# Make sure redirects are NOT absolute, to ensure NGINX does not
+# overwrite the host of the URL - which could be something other than
+# what NGINX currently thinks it is serving.
+absolute_redirect off;
+
+# If a request just has `/subfolder` we 301 redirect to `/subfolder/`
+# (Drupal really likes a trailing slash)
+rewrite ^/subfolder /subfolder/ permanent;
+
+# Any other request we prefix 301 redirect with `/subfolder/`
+rewrite ^\/(.*) /subfolder/$1 permanent;
+Replace /subfolder with the name of the subfolder you want to use. For example, /blog.
We also need to modify the NGINX Dockerfile.
+Add the following to your NGINX Dockerfile (nginx.dockerfile or Dockerfile.nginx):
Lagoon gives developers what they dream about. It's a system that allows developers to run the exact same code in their local and production environment. The same Docker images, the same service configurations, and the same code.
+Questions? Ideas? Meet the maintainers and contributors.
+Chat with us on the Lagoon Discord: https://discord.gg/te5hHe95JE +
+We want you to understand the Lagoon infrastructure and how the services work together. Here is a schema (it's a little out of date - it doesn't include some of the more recent services we've added, or cover Kubernetes, so we're working on an update!): Lucid Chart
+As described, Lagoon is a dream come true. At amazee.io, we've been hosting Drupal for more than 8 years. This is the fourth major iteration of our hosting platform. The third iteration was built around Puppet and Ansible. Every single piece of the platform was done with configuration management. This allowed very fast setup of new servers, but at the same time was also lacking customizability for developers. We implemented some customizability, with some already with Docker in production. However, we were never completely happy with it. We realized that our existing platform wasn't enough. With the rise of decoupled Drupal, the need to run Node.js on the server side, the requests for Elasticsearch, and different Solr versions, we had to do more.
+At the same time, we've been using Docker for many years for local development. It was always an idea to use Docker for everything in production. The only problem was the connection between local development and production environments. There are other systems that allow you to run Drupal in Docker in production. But, nothing allowed you to test the exact same images and services locally and in production.
+Lagoon was born in 2017. It has since been developed into a system that runs Docker in production. Lagoon has replaced our third generation hosting platform with a cutting edge all Docker-based system.
+At amazee.io, we believe in open source. It was always troubling for us that open source code like Drupal was hosted on proprietary hosting platforms. The strength and success of a hosting company is not just their deployment systems or service configurations. It's the the people and knowledge that run the platform. The processes, skills, ability to react to unforeseen situations, and last but not least, the support they provide their clients.
+Lagoon is available under an Apache 2.0 License.
Run this command:
+lagoon add project \
+ --gitUrl <YOUR-GITHUB-REPO-URL> \
+ --openshift 1 \
+ --productionEnvironment <YOUR-PROD-ENV> \
+ --branches <THE-BRANCHES-YOU-WANT-TO-DEPLOY> \
+ --project <YOUR-PROJECT-NAME>
+--openshift is the ID of your Kubernetes cluster.“^(main|develop)$”Go to the Lagoon UI, and you should see your project listed!
+Lagoon creates a deploy key for each project. You now need to add it as a deploy key in your Git repository to allow Lagoon to download the code.
+Run the following command to get the deploy key:
+ +Copy the key and save it as a deploy key in your Git repository.
+In order for Lagoon to be able to deploy on code updates, it needs to be connected to your Git repository
+Add your Lagoon cluster's webhook endpoint to your Git repository
+<LAGOON-WEBHOOK-INGRESS>Run the following command to deploy your project:
+ +Go to the Lagoon UI and take a look at your project - you should now see the environment for this project!
+Not needed for *most* installs, but this is configured to integrate Lagoon with GitLab for user and group authentication.
+your-gitlab.com/admin/hooks pointing to: webhookhandler.lagoon.example.com and define a random secret token.Update lagoon-core-values.yml:
api:
+ additionalEnvs:
+ GITLAB_API_HOST: <<URL of GitLab example: https://your-gitlab.com>>
+ GITLAB_API_TOKEN: << Personal Access token with Access to API >>
+ GITLAB_SYSTEM_HOOK_TOKEN: << System Hook Secret Token >>
+webhook-haondler:
+ additionalEnvs:
+ GITLAB_API_HOST: <<URL of GitLab example: https://your-gitlab.com>>
+ GITLAB_API_TOKEN: << Personal Access token with Access to API >>
+ GITLAB_SYSTEM_HOOK_TOKEN: << System Hook Secret Token >>
+webhooks2tasks:
+ additionalEnvs:
+ GITLAB_API_HOST: <<URL of GitLab example: https://your-gitlab.com>>
+ GITLAB_API_TOKEN: << Personal Access token with Access to API >>
+ GITLAB_SYSTEM_HOOK_TOKEN: << System Hook Secret Token >>
+Helm update the lagoon-core helmchart.
Add Helm repository:
+ +Consider the optimal configuration of Harbor for your particular circumstances - see their docs for more recommendations:
+imageChartStorage).database.type).redis.type)Create the file harbor-values.yml inside of your config directory. The proxy-buffering annotations help with large image pushes:
expose:
+ ingress:
+ annotations:
+ kubernetes.io/tls-acme: "true"
+ nginx.ingress.kubernetes.io/proxy-buffering: "off"
+ nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
+ hosts:
+ core: harbor.lagoon.example.com
+ tls:
+ enabled: true
+ certSource: secret
+ secret:
+ secretName: harbor-harbor-ingress
+externalURL: https://harbor.lagoon.example.com
+harborAdminPassword: <your Harbor Admin Password>
+chartmuseum:
+ enabled: false
+clair:
+ enabled: false
+notary:
+ enabled: false
+trivy:
+ enabled: false
+jobservice:
+ jobLogger: stdout
+Install Harbor, checking the requirements for the currently supported Harbor versions:
+ +Visit Harbor at the URL you set in harbor.yml.
You will need to add the above Harbor credentials to the Lagoon Remote values.yml in the next step, as well as harbor-values.yml.
Now we will install Lagoon Remote into the Lagoon namespace. The RabbitMQ service is the broker.
+Create lagoon-remote-values.yml in your config directory as you did the previous two files, and update the values.
kubectl -n lagoon-core get secret lagoon-core-broker -o jsonpath="{.data.RABBITMQ_PASSWORD}" | base64 --decode
+kubectl get service lagoon-core-broker-amqp-ext \
+ -o custom-columns="NAME:.metadata.name,IP ADDRESS:.status.loadBalancer.ingress[*].ip,HOSTNAME:.status.loadBalancer.ingress[*].hostname"
+Add the Harbor configuration from the Install Harbor step.
+lagoon-build-deploy:
+ enabled: true
+ extraArgs:
+ - "--enable-harbor=true"
+ - "--harbor-url=https://harbor.lagoon.example.com"
+ - "--harbor-api=https://harbor.lagoon.example.com/api/"
+ - "--harbor-username=admin"
+ - "--harbor-password=<from harbor-harbor-core secret>"
+ rabbitMQUsername: lagoon
+ rabbitMQPassword: <from lagoon-core-broker secret>
+ rabbitMQHostname: lagoon-core-broker.lagoon-core.svc.cluster.local
+ lagoonTargetName: <name of lagoon remote, can be anything>
+ taskSSHHost: <IP of ssh service loadbalancer>
+ taskSSHPort: "22"
+ taskAPIHost: "api.lagoon.example.com"
+dbaas-operator:
+ enabled: true
+
+ mariadbProviders:
+ production:
+ environment: production
+ hostname: 172.17.0.1.nip.io
+ readReplicaHostnames:
+ - 172.17.0.1.nip.io
+ password: password
+ port: '3306'
+ user: root
+
+ development:
+ environment: development
+ hostname: 172.17.0.1.nip.io
+ readReplicaHostnames:
+ - 172.17.0.1.nip.io
+ password: password
+ port: '3306'
+ user: root
+Install Lagoon Remote:
+ +Lagoon uses the K8up backup operator: https://k8up.io. Lagoon isn’t tightly integrated with K8up, it’s more that Lagoon can create its resources in a way that K8up can automatically discover and backup.
+Lagoon has been extensively tested with K8up 1.x, but is not compatible with 2.x yet. We recommend using the 1.1.0 chart version (App version v1.2.0).
+Create new AWS User with policies:
+{
+ "Version":"2012-10-17",
+ "Statement":[
+ {
+ "Sid":"VisualEditor0",
+ "Effect":"Allow",
+ "Action":[
+ "s3:ListAllMyBuckets",
+ "s3:CreateBucket",
+ "s3:GetBucketLocation"
+ ],
+ "Resource":"*"
+ },
+ {
+ "Sid":"VisualEditor1",
+ "Effect":"Allow",
+ "Action":"s3:ListBucket",
+ "Resource":"arn:aws:s3:::baas-*"
+ },
+ {
+ "Sid":"VisualEditor2",
+ "Effect":"Allow",
+ "Action":[
+ "s3:PutObject",
+ "s3:GetObject",
+ "s3:AbortMultipartUpload",
+ "s3:DeleteObject",
+ "s3:ListMultipartUploadParts"
+ ],
+ "Resource":"arn:aws:s3:::baas-*/*"
+ }
+ ]
+}
+Create k8up-values.yml (customize for your provider):
k8up:
+ envVars:
+ - name: BACKUP_GLOBALS3ENDPOINT
+ value: 'https://s3.eu-west-1.amazonaws.com'
+ - name: BACKUP_GLOBALS3BUCKET
+ value: ''
+ - name: BACKUP_GLOBALKEEPJOBS
+ value: '1'
+ - name: BACKUP_GLOBALSTATSURL
+ value: 'https://backup.lagoon.example.com'
+ - name: BACKUP_GLOBALACCESSKEYID
+ value: ''
+ - name: BACKUP_GLOBALSECRETACCESSKEY
+ value: ''
+ - name: BACKUP_BACKOFFLIMIT
+ value: '2'
+ - name: BACKUP_GLOBALRESTORES3BUCKET
+ value: ''
+ - name: BACKUP_GLOBALRESTORES3ENDPOINT
+ value: 'https://s3.eu-west-1.amazonaws.com'
+ - name: BACKUP_GLOBALRESTORES3ACCESSKEYID
+ value: ''
+ - name: BACKUP_GLOBALRESTORES3SECRETACCESSKEY
+ value: ''
+ timezone: Europe/Zurich
+Install K8up:
+ + + +Update lagoon-core-values.yml:
Redeploy lagoon-core.
brew tap uselagoon/lagoon-clibrew install lagoonThe CLI needs to know how to communicate with Lagoon, so run the following command:
+ +Access Lagoon by authenticating with your SSH key.
+values.yml if you forget), go to Settings.You need to set the default Lagoon to your Lagoon so that it doesn’t try to use the amazee.io defaults:
+ +Now run lagoon login. Lagoon talks to SSH and authenticates against your public/private key pair, and gets a token for your username.
Verify via lagoon whoami that you are logged in.
Info
+We don’t generally recommend using the Lagoon Admin role, but you’ll need to create an admin account at first to get started. Ideally, you’ll immediately create another account to work from which is not an admin.
+Add Lagoon Charts repository to your Helm Repositories:
+ +Create a directory for the configuration files we will create, and make sure that it’s version controlled. Ensure that you reference this path in commands referencing your values.yml files.
values.yml in the directory you’ve just created. Update the endpoint URLs (change them from api.lagoon.example.com to your values).
+ Example: https://github.com/uselagoon/lagoon-charts/blob/main/charts/lagoon-core/ci/linter-values.yamlNow run helm upgrade --install command, pointing to values.yml, like so:
Lagoon Core is now installed!
+Warning
+Sometimes we run into Docker Hub pull limits. We are considering moving our images elsewhere if this continues to be a problem.
+Visit the Keycloak dashboard at the URL you defined in the values.yml for Keycloak.
adminlagoon-core-keycloak secret, key-value KEYCLOAK_ADMIN_PASSWORDRetrieve the secret like so:
+ +Click on User on top right.
+You should now be able to visit the Lagoon UI at the URL you defined in the values.yml for the UI.
lagoonadminlagoon-core-keycloak secret key-value: LAGOON-CORE-KEYCLOAKLagoon files are used to store the file output of tasks, such as backups, and can be hosted on any S3-compatible storage.
+Create new AWS User with policies:
+{
+ "Version":"2012-10-17",
+ "Statement":[
+ {
+ "Effect":"Allow",
+ "Action":[
+ "s3:ListBucket",
+ "s3:GetBucketLocation",
+ "s3:ListBucketMultipartUploads"
+ ],
+ "Resource":"arn:aws:s3:::S3_BUCKET_NAME"
+ },
+ {
+ "Effect":"Allow",
+ "Action":[
+ "s3:PutObject",
+ "s3:GetObject",
+ "s3:DeleteObject",
+ "s3:ListMultipartUploadParts",
+ "s3:AbortMultipartUpload"
+ ],
+ "Resource":"arn:aws:s3:::S3_BUCKET_NAME/*"
+ }
+ ]
+}
+Update lagoon-core-values.yml:
If you use ingress-nginx in front of lagoon-core, we suggest setting this configuration which will allow for bigger file uploads:
controller:
+config:
+ client-body-timeout: '600' # max 600 secs fileuploads
+ proxy-send-timeout: '1800' # max 30min connections - needed for websockets
+ proxy-read-timeout: '1800' # max 30min connections - needed for websockets
+ proxy-body-size: 1024m # 1GB file size
+ proxy-buffer-size: 64k # bigger buffer
+Lagoon integrates with OpenSearch to store application, container and router logs. Lagoon Logging collects the application, router and container logs from Lagoon projects, and sends them to the logs concentrator. It needs to be installed onto each lagoon-remote instance.
In addition, it should be installed in the lagoon-core cluster to collect logs from the lagoon-core service. This is configured in the LagoonLogs section.
Logging Overview: Lucid Chart
+See also: Logging.
+Read more about Lagoon logging here: https://github.com/uselagoon/lagoon-charts/tree/main/charts/lagoon-logging
+Create lagoon-logging-values.yaml:
tls:
+ caCert: |
+ << content of ca.pem from Logs-Concentrator>>
+ clientCert: |
+ << content of client.pem from Logs-Concentrator>>
+ clientKey: |
+ << content of client-key.pem from Logs-Concentrator>>
+forward:
+ username: <<Username for Lagoon Remote 1>>
+ password: <<Password for Lagoon Remote 1>>
+ host: <<ExternalIP of Logs-Concentrator Service LoadBalancer>>
+ hostName: <<Hostname in Server Cert of Logs-Concentrator>>
+ hostPort: '24224'
+ selfHostname: <<Hostname in Client Cert of Logs-Concentrator>>
+ sharedKey: <<Generated ForwardSharedKey of Logs-Concentrator>>
+ tlsVerifyHostname: false
+clusterName: <<Short Cluster Identifier>>
+logsDispatcher:
+ serviceMonitor:
+ enabled: false
+logging-operator:
+ monitoring:
+ serviceMonitor:
+ enabled: false
+lagoonLogs:
+ enabled: true
+ rabbitMQHost: lagoon-core-broker.lagoon-core.svc.cluster.local
+ rabbitMQUser: lagoon
+ rabbitMQPassword: <<RabbitMQ Lagoon Password>>
+excludeNamespaces: {}
+Install lagoon-logging:
If you'd like logs from ingress-nginx inside lagoon-logging:
ingress-nginxAdd the content of this file to ingress-nginx:
controller:
+ config:
+ log-format-upstream: >-
+ {
+ "time": "$time_iso8601",
+ "remote_addr": "$remote_addr",
+ "x-forwarded-for": "$http_x_forwarded_for",
+ "true-client-ip": "$http_true_client_ip",
+ "req_id": "$req_id",
+ "remote_user": "$remote_user",
+ "bytes_sent": $bytes_sent,
+ "request_time": $request_time,
+ "status": "$status",
+ "host": "$host",
+ "request_proto": "$server_protocol",
+ "request_uri": "$uri",
+ "request_query": "$args",
+ "request_length": $request_length,
+ "request_time": $request_time,
+ "request_method": "$request_method",
+ "http_referer": "$http_referer",
+ "http_user_agent": "$http_user_agent",
+ "namespace": "$namespace",
+ "ingress_name": "$ingress_name",
+ "service_name": "$service_name",
+ "service_port": "$service_port"
+ }
+Your logs should start flowing!
+Logs-concentrator collects the logs being sent by Lagoon clusters and augments them with additional metadata before inserting them into Elasticsearch.
+Create logs-concentrator-values.yml:
tls:
+ caCert: |
+ <<contents of ca.pem>>
+ serverCert: |
+ <<contents of server.pem
+ serverKey: |
+ <<contents of server-key.pem>>
+elasticsearchHost: elasticsearch-opendistro-es-client-service.elasticsearch.svc.cluster.local
+elasticsearchAdminPassword: <<ElasticSearch Admin Password>>
+forwardSharedKey: <<Random 32 Character Password>>
+users:
+ - username: <<Username for Lagoon Remote 1>>
+ password: <<Random Password for Lagoon Remote 1>>
+service:
+ type: LoadBalancer
+serviceMonitor:
+ enabled: false
+Install logs-concentrator:
+ +To install an OpenDistro cluster, you will need to configure TLS and secrets so that Lagoon can talk to it securely. You're going to have to create a handful of JSON files - put these in the same directory as the values files you've been creating throughout this installation process.
+Install OpenDistro Helm, according to https://opendistro.github.io/for-elasticsearch-docs/docs/install/helm/
+Generate certificates
+Note:
+CFSSL is CloudFlare's PKI/TLS swiss army knife. It is both a command line tool and an HTTP API server for signing, verifying, and bundling TLS certificates. It requires Go 1.12+ to build.
+Run the following two commands:
+ +You'll get ca-key.pem, and ca.pem. This is your CA key and self-signed certificate.
Next, we'll generate the node peering certificate. You'll need the following two files:
+{
+ "signing": {
+ "default": {
+ "expiry": "87600h"
+ },
+ "profiles": {
+ "peer": {
+ "expiry": "87600h",
+ "usages": [
+ "signing",
+ "key encipherment",
+ "server auth",
+ "client auth"
+ ]
+ },
+ "client": {
+ "expiry": "87600h",
+ "usages": [
+ "signing",
+ "key encipherment",
+ "client auth"
+ ]
+ }
+ }
+ }
+}
+Run the following two commands:
+cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer node.json | cfssljson -bare node
+rm node.csr
+You'll get node.pem and node-key.pem. This is the peer certificate that will be used by nodes in the ES cluster.
Next, we'll convert the key to the format supported by Java with the following command:
+ +Now we'll generate the admin certificate. You'll need the following file:
+ +Run the following two commands:
+cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client admin.json | cfssljson -bare admin
+rm admin.csr
+You'll get admin.pem and admin-key.pem. This is the certificate that will be used to perform admin commands on the opendistro-security plugin.
Next, we'll convert the key to the format supported by Java with the following command:
+ +Now that we have our keys and certificates, we can continue with the installation.
+Generate hashed passwords.
+elasticsearch-secrets-values.yaml needs two hashed passwords. Create them with this command (run it twice, enter a random password, store both the plaintext and hashed passwords).Create secrets:
+elasticsearch-secrets-values.yaml. See this gist as an example: https://gist.github.com/Schnitzel/43f483dfe0b23ca0dddd939b12bb4b0bInstall secrets with the following commands:
+ +You'll need to create elasticsearch-values.yaml. See this gist as an example: (fill all <\
Install Elasticsearch:
+ +Configure security inside Elasticsearch with the following:
+kubectl exec -n elasticsearch -it elasticsearch-opendistro-es-master-0 -- bash
+chmod +x /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh
+/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -nhnv -cacert /usr/share/elasticsearch/config/admin-root-ca.pem -cert /usr/share/elasticsearch/config/admin-crt.pem -key /usr/share/elasticsearch/config/admin-key.pem -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/
+Update lagoon-core-values.yaml with:
Rollout Lagoon Core:
+ +Sync all Lagoon Groups with Opendistro Elasticsearch
+ +You’ll need an app for sending and receiving GraphQL queries. We recommend GraphiQL.
+brew install --cask graphiql.We need to tell Lagoon Core about the Kubernetes cluster. The GraphQL endpoint is: https://<YOUR-API-URL>/graphql
Go to Edit HTTP Headers, and Add Header.
+AuthorizationBearer YOUR-TOKEN-HERE.lagoon.yml file. Copy the token from that file and use it for the value here.Now you’re ready to run some queries. Run the following test query to ensure everything is working correctly:
+ +This should give you the following response:
+ + +Once you get the correct response, we need to add a mutation.
+Run the following query:
+mutation addKubernetes {
+ addKubernetes(input:
+ {
+ name: "<TARGET-NAME-FROM-REMOTE-VALUES.yml>",
+ consoleUrl: "<URL-OF-K8S-CLUSTER>",
+ token: "xxxxxx”
+ routerPattern: "${environment}.${project}.lagoon.example.com"
+ }){id}
+}
+name: get from lagoon-remote-values.ymlconsoleUrl: API Endpoint of Kubernetes cluster. Get from values.ymltoken: create a token for the lagoon-build-deploy service account
Prior to Kubernetes 1.21:
+Use the lagoon-build-deploy token installed by lagoon-remote:
Info
+Authorization tokens for GraphQL are very short term so you may need to generate a new one. Run lagoon login and then cat the .lagoon.yml file to get the new token, and replace the old token in the HTTP header with the new one.
Note
+We acknowledge that this is a lot of steps, and our roadmap for the immediate future includes reducing the number of steps in this process.
+Lagoon supports Kubernetes versions 1.21 onwards. We actively test and develop against Kubernetes 1.24, also regularly testing against 1.21,1.22 and 1.25.
+The next large round of breaking changes is in Kubernetes 1.25, and we will endeavour to be across these in advance, although this will require a bump in the minimum supported version of Lagoon.
+Lagoon is currently configured only for a single ingress-nginx controller, and therefore defining an IngressClass was not necessary in the past.
In order to use the recent ingress-nginx controllers (v4 onwards, required for Kubernetes 1.22), the following configuration should be used, as per the ingress-nginx docs.
nginx-ingress should be configured as the default controller - set .controller.ingressClassResource.default: true in Helm valuesnginx-ingress should be configured to watch ingresses without IngressClass set - set .controller.watchIngressWithoutClass: true in Helm valuesThis will configure the controller to create any new ingresses with itself as the IngressClass, and also to handle any existing ingresses without an IngressClass set.
Other configurations may be possible, but have not been tested.
+Versions 2.1 and 2.2+ of Harbor are currently supported. The method of retrieving robot accounts was changed in 2.2, and the Lagoon remote-controller is able to handle these tokens. This means that Harbor has to be configured with the credentials in lagoon-build-deploy - not lagoon-core.
We recommend installing a Harbor version greater than 2.6.0 with Helm chart 1.10.0 or greater.
+Lagoon has built in configuration for the K8up backup operator. Lagoon can configure prebackup pods, schedules and retentions, and manage backups and restores for K8up. Lagoon currently only supports the 1.x versions of K8up, owing to a namespace change in v2 onwards, but we are working on a fix.
+K8up v2:
+Lagoon does not currently support K8up v2 onwards due to a namespace change here.
+We recommend installing K8up version 1.2.0 with Helm Chart 1.1.0
+Lagoon utilizes a default 'standard' StorageClass for most workloads, and the internal provisioner for most Kubernetes platforms will suffice. This should be configured to be dynamic provisioning and expandable where possible.
Lagoon also requires a StorageClass called 'bulk' to be available to support persistant pod replicas (across nodes). This StorageClass should support ReadWriteMany (RWX) access mode and should be configured to be dynamic provisioning and expandable where possible. See https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes for more information, and the production drivers list for a complete list of compatible drivers.
We have curently only included the instructions for (the now deprecated) EFS Provisioner. The production EFS CSI driver has issues with provisioning more than 120 PVCs. We are awaiting upstream possible fixes here and here - but most other providers CSI drivers should also work, as will configurations with an NFS-compatible server and provisioner.
+Lagoon uses some very involved Kubernetes and cloud-native concepts, and while full familiarity may not be necessary to install and configure Lagoon, diagnosing issues and contributing may prove difficult without a good level of familiarity.
+As an indicator, comfort with the curriculum for the Certified Kubernetes Administrator would be suggested as a minimum.
+ + + + + + + + +Download newest charts using Helm.
+ +Check with helm diff for changes (https://github.com/databus23/helm-diff).
Back up the Lagoon databases prior to any Helm actions. + We also suggest scaling the API to a single pod, to aid the database migration scripts running in the initContainers.
+Run the upgrade using Helm.
+ +(Note that as of Lagoon v2.11.0, this step is no longer required.)
+ If upgrading Lagoon Core, ensure you run the rerun_initdb.sh script to perform post upgrade migrations.
Re-scale the API pods back to their original level.
+If upgrading Lagoon Core, and you have enabled groups/user syncing for OpenSearch, you may additionally need to run the sync:opendistro-security script to update the groups in OpenSearch. This command can also be prefixed with a GROUP_REGEX=<group-to-sync to sync a single group at a time, as syncing the entire group structure may take a long time.
Check https://github.com/uselagoon/lagoon/releases for additional upgrades.
+You may want to back up the databases before upgrading Lagoon Core, the following will create backups you can use to restore from if required. You can delete them afterwards.
+kubectl --namespace lagoon-core exec -it lagoon-core-api-db-0 -- \
+ sh -c 'mysqldump --max-allowed-packet=500M --events \
+ --routines --quick --add-locks --no-autocommit \
+ --single-transaction infrastructure | gzip -9 > \
+ /var/lib/mysql/backup/$(date +%Y-%m-%d_%H%M%S).infrastructure.sql.gz'
+kubectl --namespace lagoon-core exec -it lagoon-core-keycloak-db-0 -- \
+ sh -c 'mysqldump --max-allowed-packet=500M --events \
+ --routines --quick --add-locks --no-autocommit \
+ --single-transaction keycloak | gzip -9 > \
+ /var/lib/mysql/backup/$(date +%Y-%m-%d_%H%M%S).keycloak.sql.gz'
+Have you seen the Kibana getting started video and are now ready to work with logs? We are here to help! This page will give you examples of Kibana queries you can use. This is not a Kibana 101 class, but it can help you understand some of what you can do in Kibana.
+Ready to get started? Good!
+Note
+Make sure that you have selected your tenant before starting! You can do that by on the Tenant icon on the left-hand menu. Once you have selected your tenant, click on the Discover icon again to get started.
Below you'll find examples for two common log requests:
+Discovery (#1 in screen shot below)In the search bar (#3), enter:
+openshift_project: "name of your production project"
Running the query above will give you a general look at all the traffic to your site, but what if you want to narrow in on a specific IP address? Perhaps you want to see how many times an IP has hit your site and what specific pages they were looking at. This next query should help.
+We are going to start off with the same query as above, but we are going to add a couple of things.
+client_ip and http_request.
That looks good, but what if we wanted to just show requests from a specific IP address? You can filter for the address by adding it to your search criteria.
+AND client_ip: "IP address".
Container logs will show you all stout and sterr messages for your specific container and project. We are going to show an example for getting logs from a specific container and finding specific error numbers in that container.
Want to see the logs for a specific container (php, nginx, etc)? This section will help! Let's focus on looking at NGINX logs.
+kubernetes.container_name: "nginx"
Want to see how many 500 Internal Server errors you've had in your NGINX container? You can do that by changing the search query. If you search:
+kubernetes.container_name: "nginx" AND message: "500"
That will only display 500 error messages in the NGINX container. You can search for any error message in any container that you would like!
+Kibana will also give you the option to create visualizations or graphs. We are going to create a chart to show number of hits/requests in a month using the same query we used above.
+Note
+Make sure that you select an appropriate time frame for the data in the upper right hand corner.
+Here is an example of a daily hits visualization chart:
+
Also note that you can save your visualizations (and searches)! That will make it even faster to access them in the future. And because each account has their own Kibana Tenant, no searches or visualizations are shared with another account.
+Lagoon provides access to the following logs via Kibana:
+stdout and stderr messagesudp://application-logs.lagoon.svc:5140type field contains the name of the Kubernetes namespace ($LAGOON_PROJECT-$LAGOON_ENVIRONMENT).To access the logs, please check with your Lagoon administrator to get the URL for the Kibana route (for amazee.io, this is https://logs.amazeeio.cloud/).
+Each Lagoon user account has their own login and will see the logs only for the projects to which they have access.
+Each Lagoon user account also has their own Kibana Tenant, which means no saved searches or visualizations are shared with another account.
+If you would like to know more about how to use Kibana: https://www.elastic.co/webinars/getting-started-kibana.
+ + + + + + + + +You should have a private Slack channel that was set up for you to communicate - if not, or you've forgotten how to contact us, reach out at support@amazee.io.
+If you've found a bug or security issue, please send your findings to support@amazee.io. Please DO NOT file a GitHub issue for them.
+That's great news! You can contact them via email at inquiries@amazee.io.
+We have backups available for files and databases, typically taken every 24 hours at most. These backups are stored offsite.
+We keep up to 7 daily backups and 4 weekly backups.
+If you ever need to recover or restore a backup, feel free to submit a ticket or send us a message via chat and we will be more than happy to help!
+The first thing to try is what is listed in our documentation about SSL.
+If you follow those steps, and you are still seeing an error, please submit a ticket or send us a message on chat and we can help resolve this for you.
+This was a bug that was prevalent in Drush versions 8.1.16 and 8.1.17. There error would look something like this:
+The command could not be executed successfully (returned: Array [error]
+(
+[default] => Array
+(
+[default] => Array
+(
+[driver] => mysql
+[prefix] => Array
+(
+[default] =>
+)
+, code: 0)
+Error: no database record could be found for source @main [error]
+Upgrading Drush should fix that for you. We strongly suggest that you use version 8.3 or newer. Once Drush is upgraded the command should work!
+No need to panic! This usually happens when a tenant has not been selected. To fix this, follow these steps:
+You should now be able to see your logs.
+I'm unable to SSH into any environment. I'm getting the following message: Permission denied (publickey). When I run drush sa no aliases are returned.
This typically indicates an issue with Pygmy. You can find our troubleshooting docs for Pygmy here: https://pygmy.readthedocs.io/en/master/troubleshooting/
+You will need to contact your helpful Lagoon administrator should you need to remove a route. You can use the Slack channel that was set up for you to communicate - if not, you can always reach us at support@amazee.io or on Discord.
+pygmy status, no keys are loaded#You'll need to load your SSH key into pygmy. Here's how: https://pygmy.readthedocs.io/en/master/ssh_agent
+drush sa no aliases are returned#This typically indicates an issue with Pygmy. You can find our troubleshooting docs for Pygmy here: https://pygmy.readthedocs.io/en/master/troubleshooting
+This usually means that there is no database uploaded to the project. Follow our step-by-step guide to add a database to your project.
+Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use Error: failed to start containers: amazeeio-haproxy
This is a known error! Most of the time it means that there is already something running on port 80. You can find the culprit by running the following query:
+ +That should list everything running on port 80. Kill the process running on port 80. Once port 80 is freed up, Pygmy should start up with no further errors.
+You can make that change using the Lagoon API! You can find the documentation for this change in our GraphQL documentation.
+This can be done via the Lagoon API. You can find the steps documentation for this change in our GraphQL documentation.
+Environments are fully built from scratch at each deploy, dropping the old database and files and pushing your code would result in a fresh clean build, Don’t forget to re-sync!
+It is possible to delete an environment via GraphQL. You can find the instructions in our GraphQL documentation.
+Once you've added a runtime environment variable to your production environment via GraphQL, then all you need to do a deploy in order to get your change to show up on your environment.
+For cloud hosting customers, you can SFTP to your Lagoon environment by using the following information:
+ssh.lagoon.amazeeio.cloudYour username is going to be the name of the environment you are connecting to, most commonly in the pattern PROJECTNAME-ENVIRONMENT.
You may also be interested in checking out our new Lagoon Sync tool, which you can read about here: https://github.com/uselagoon/lagoon-sync
+Authentication also happens automatically via SSH Public & Private Key Authentication.
+We can definitely help with that. Once you have your own SSL certificate, feel free to submit a ticket or send us a message via chat and we will be more than happy to help! You will need to send us the following files:
+Also, you will need to set the tls-acme option in .lagoon.yml to false.
Mounting an external volume would need to be handled completely inside of your containers, Lagoon does not provide a provision for this type of connection as part of the platform.
+A developer can handle this by installing the necessary packages into the container (via the Dockerfile), and ensuring the volume mount is connected via a pre- or post-rollout task.
+If you have a build that has been running for a long time, and want to stop it, you will need to reach out to support. Currently, builds can only be stopped by users with admin access to the cluster.
+We suggest only exposing web services (NGINX/Varnish/Node.js) in your deployed environments. Locally, you can get the ports mapped for these services by checking docker-compose ps, and then load http://localhost:<port> in your browser.
You can reach out to the team via Discord or email at uselagoon@amazee.io.
+ + + + + + + + +| Term | +Definition | +
|---|---|
| Access Mode | +Controls how a persistent volume can be accessed. | +
| Active/Standby | +Active/Standby deployments, also known as blue/green deployments, are a way to seamlessly switch over your production content. | +
| Ansible | +An open-source suite of software tools that enables infrastructure as code. | +
| AWS | +Amazon Web Services | +
| AWS Glacier | +A secure and inexpensive S3 storage for long-term backup. | +
| BitBucket | +Git hosting owned by Atlassian, which integrates with their tools. | +
| Brew | +Homebrew is a package manager for OSX. | +
| CA | +A Certificate Authority is a trusted entity that issues Secure Sockets Layer (SSL) certificates. | +
| CDN | +Content Delivery Network - distributes content via caching | +
| CI | +Continuous Integration | +
| CIDR | +Classess Inter-Domain Routing - a method of assigning IP addresses | +
| CLI | +Command Line Interface | +
| Cluster | +A unified group of servers or VMs, distributed and managed together, which serves one entity to ensure high availability, load balancing, and scalability. | +
| CMS | +Content Management System | +
| Cron job | +The cron command-line utility is a job scheduler on Unix-like operating systems. Users who set up and maintain software environments use cron to schedule jobs, also known as cron jobs, to run periodically at fixed times, dates, or intervals. | +
| Composer | +A package manager | +
| DDoS | +Distributed Denial of Service | +
| DNS | +Domain Name System | +
| Docker | +A container engine using Linux features and automating application deployment. | +
| Docker Compose | +A tool for defining and running Docker applications via YAML files. | +
| Drupal | +Open-source Content Management System | +
| Drush | +A command line shell for Drupal. | +
| EC2 | +Amazon Elastic Compute Cloud | +
| Elasticsearch | +An open-source search engine. It provides a distributed, multi-tenant-capable full-text search engine with a web interface and schema-free JSON documents. | +
| Galera | +A generic synchronous multi-master replication library for transactional databases. | +
| Git | +A free and open-source distributed version control system. | +
| Git Hash/SHA | +A generated string that identifies each commit. Uses the SHA-1 algorithm | +
| GitHub | +A proprietary version control hosting company using Git. A subsidiary of Microsoft, it offers all of the distributed version control and source code management functionality of Git as well as additional features. | +
| GitLab | +A web-based Git repository manager with CI capabilities. | +
| GraphQL | +An open-source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data. | +
| Harbor | +An open source container image registry that secures images with role-based access control, scans images for vulnerabilities, and signs images as trusted. | +
| Helm | +A package manager for Kubernetes, it helps you manage Kubernetes applications. | +
| Helm Charts | +Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. | +
| HTTP | +HyperText Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. | +
| IAM | +AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. | +
| IDE | +An integrated development environment is a software application that provides comprehensive facilities to computer programmers for software development. An IDE normally consists of at least a source code editor, build automation tools, and a debugger. | +
| Ingress controller | +An Ingress controller is a specialized load balancer for Kubernetes (and other containerized) environments. | +
| IPTables | +A command line utility for configuring Linux kernel firewall. | +
| Jenkins | +An open-source automation server. | +
| k3s | +A highly available, certified Kubernetes distribution. | +
| k3d | +k3d is a lightweight wrapper to run k3s in Docker. | +
| k8s | +Numeronym for Kubernetes (K + 8 letters + s) | +
| k8up | +K8up is a backup operator that will handle storage and app backups on a k8s/OpenShift cluster. | +
| Kibana | +An open-source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. | +
| KinD | +Kubernetes in Docker - a tool for running local Kubernetes clusters using Docker container “nodes”. Kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI. | +
| kubectl | +The Kubernetes command-line tool which allows you to run commands against Kubernetes clusters. | +
| Kubernetes | +An open-source system for automating deployment, scaling, and management of containerized applications. | +
| Lagoon | +An open-source application delivery platform for Kubernetes. | +
| Lagoonize | +Configuration changes to allow your app to run on Lagoon. | +
| Lando | +A free, open source, cross-platform, local development environment and DevOps tool built on Docker. | +
| Laravel | +A free, open-source PHP web framework, following the model–view–controller (MVC) architectural pattern and based on Symfony. | +
| Let's Encrypt | +Aa free, automated, and open certificate authority (CA). | +
| MariaDB | +A community-developed, commercially supported fork of the MySQL relational database management system, intended to remain free and open-source software under the GNU General Public License. | +
| Master node | +A single node in the cluster on which a collection of processes which manage the cluster state are running. | +
| Microservice | +The practice of breaking up an application into a series of smaller, more specialized parts, each of which communicate with one another across common interfaces such as APIs and REST interfaces like HTTP | +
| MongoDB | +MongoDB is a cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with schema. | +
| Multi-Tenant | +A single instance of software runs on a server and serves multiple tenants - a tenant is a group of users who share common access with privileges to access the software instance. The software is designed to provide each tenant a share of the resources. | +
| MVC | +Model-view-controller - an architectural pattern that separates an application into three main logical components: the model, the view, and the controller. Each of these components are built to handle specific development aspects of an application. | +
| NGINX | +NGINX is a web server which can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. | +
| Node | +Single EC2 instance (AWS virtual machine) | +
| Node.js | +An open-source, cross-platform, JavaScript runtime environment that executes JavaScript code outside of a browser. | +
| OpenSearch | +A community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data. | +
| OpenShift | +Container application platform that brings Docker and Kubernetes to the enterprise. | +
| PHP | +PHP (Personal Home Page) is a general-purpose programming language originally designed for web development. | +
| PhpStorm | +A development tool (IDE) for PHP and web projects. | +
| Pod | +A group of containers that are deployed together on the same host. The basic unit that Kubernetes works with. | +
| PostgreSQL | +A free and open-source relational database management system emphasizing extensibility and technical standards compliance. | +
| Public/Private Key | +Public-key encryption is a cryptographic system that uses two keys -- a public key known to everyone and a private or secret key known only to the recipient of the message. | +
| Puppet | +An open-source software configuration management and deployment tool. | +
| PV | +PersistentVolume - a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. | +
| PVC | +Persistent Volume Claim - a request for storage by a user. | +
| Python | +Python is an open-source, interpreted, high-level, general-purpose programming language. | +
| RabbitMQ | +An open-source message-broker software. | +
| RBAC | +Role-Based Access Control | +
| RDS | +Relational Database Service | +
| Redis | +An open source, in-memory data store used as a database, cache, streaming engine, and message broker. | +
| Restic | +An open-source backup program. | +
| ROX | +Kubernetes access mode ReadOnlyMany - the volume can be mounted as read-only by many nodes. | +
| Ruby | +An interpreted, high-level, general-purpose programming language which supports multiple programming paradigms. It was designed with an emphasis on programming productivity and simplicity. In Ruby, everything is an object, including primitive data types. | +
| RWO | +Kubernetes access mode ReadWriteOnce - the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node. | +
| RWOP | +Kubernetes access mode ReadWriteOncePod - the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+. | +
| RWX | +Kubernetes access mode ReadWriteMany - the volume can be mounted as read-write by many nodes. | +
| SHA-1 | +Secure Hash Algorithm 1, a hash function which takes an input and produces a 160-bit hash value known as a message digest – typically rendered as 40 hexadecimal digits. It was designed by the United States National Security Agency, and is a U.S. Federal Information Processing Standard. | +
| Solr | +An open-source enterprise-search platform, written in Java. | +
| SSH | +Secure Socket Shell, a network protocol that provides administrators with a secure way to access a remote computer. | +
| SSL | +Secure Socket Layer | +
| Storage Classes | +A StorageClass provides a way for Kubernetes administrators to describe the "classes" of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators | +
| Symfony | +Symfony is a PHP web application framework and a set of reusable PHP components/libraries, Drupal 8 and up are based on Symfony. | +
| TCP | +Transmission Control Protocol, a standard that defines how to establish and maintain a network conversation through which application programs can exchange data. | +
| TLS | +Transport Layer Security | +
| Trivy | +A simple and comprehensive vulnerability scanner for containers, suitable for CI. | +
| TTL | +Time to live or hop limit is a mechanism that limits the lifespan or lifetime of data in a computer or network. | +
| Uptime Robot | +Uptime monitoring service. | +
| Varnish | +A powerful, open-source HTTP engine/reverse HTTP proxy that can speed up a website by caching (or storing) a copy of a webpage the first time a user visits. | +
| VM | +Virtual Machine | +
| Webhook | +A webhook is a way for an app like GitHub, GitLab, Bitbucket, etc, to provide other applications with immediate data and act upon something, like a pull request. | +
| YAML | +Yet Another Markup Language - YAML is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. | +
[Slides]
+ + +[Slides]
+ + +[Slides]
+ + +[Slides]
+ + +Lagoon gives developers what they dream about. It's a system that allows developers to run the exact same code in their local and production environment. The same Docker images, the same service configurations, and the same code.
"},{"location":"#who-are-you","title":"Who are you?","text":"Questions? Ideas? Meet the maintainers and contributors.
Chat with us on the Lagoon Discord: https://discord.gg/te5hHe95JE
"},{"location":"#a-couple-of-things-about-lagoon","title":"A couple of things about Lagoon","text":"We want you to understand the Lagoon infrastructure and how the services work together. Here is a schema (it's a little out of date - it doesn't include some of the more recent services we've added, or cover Kubernetes, so we're working on an update!): Lucid Chart \u200c
"},{"location":"#history-of-lagoon","title":"History of Lagoon","text":"As described, Lagoon is a dream come true. At amazee.io, we've been hosting Drupal for more than 8 years. This is the fourth major iteration of our hosting platform. The third iteration was built around Puppet and Ansible. Every single piece of the platform was done with configuration management. This allowed very fast setup of new servers, but at the same time was also lacking customizability for developers. We implemented some customizability, with some already with Docker in production. However, we were never completely happy with it. We realized that our existing platform wasn't enough. With the rise of decoupled Drupal, the need to run Node.js on the server side, the requests for Elasticsearch, and different Solr versions, we had to do more. \u200c
At the same time, we've been using Docker for many years for local development. It was always an idea to use Docker for everything in production. The only problem was the connection between local development and production environments. There are other systems that allow you to run Drupal in Docker in production. But, nothing allowed you to test the exact same images and services locally and in production.
Lagoon was born in 2017. It has since been developed into a system that runs Docker in production. Lagoon has replaced our third generation hosting platform with a cutting edge all Docker-based system.
"},{"location":"#open-source","title":"Open Source","text":"At amazee.io, we believe in open source. It was always troubling for us that open source code like Drupal was hosted on proprietary hosting platforms. The strength and success of a hosting company is not just their deployment systems or service configurations. It's the the people and knowledge that run the platform. The processes, skills, ability to react to unforeseen situations, and last but not least, the support they provide their clients.
"},{"location":"#license","title":"License","text":"Lagoon is available under an Apache 2.0 License.
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
"},{"location":"code-of-conduct/#our-standards","title":"Our Standards","text":"Examples of behavior that contributes to creating a positive environment include:
Examples of unacceptable behavior by participants include:
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
"},{"location":"code-of-conduct/#scope","title":"Scope","text":"This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
"},{"location":"code-of-conduct/#enforcement","title":"Enforcement","text":"Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at uselagoon@amazee.io. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
"},{"location":"code-of-conduct/#attribution","title":"Attribution","text":"This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4.
"},{"location":"contributing/","title":"Contributing","text":"We gladly welcome any and all contributions to Lagoon!
"},{"location":"contributing/#what-kind-of-contributions-do-we-need","title":"What kind of contributions do we need?","text":"Lagoon benefits from any kind of contribution - whether it's a bugfix, new feature, documentation update, or simply some queue maintenance - we're happy that you want to help
"},{"location":"contributing/#developing-for-lagoon","title":"Developing for Lagoon","text":"There's a whole section on how to get Lagoon running on your local machine using KinD over at Developing Lagoon. This documentation is still very WIP - but there are a lot of Makefile routines to help you out.
"},{"location":"contributing/#installing-lagoon","title":"Installing Lagoon","text":"We've got another section that outlines how to install Lagoon from Helm charts at Installing Lagoon Into Existing Kubernetes Cluster - we'd love to get this process as slick as possible!
"},{"location":"contributing/#help-us-with-our-examples","title":"Help us with our examples","text":"Right now one of our biggest needs is putting together examples of Lagoon working with various content management systems, etc, other than Drupal.
If you can spin up an open source CMS or framework that we don\u2019t currently have as a Docker Compose stack, send us a PR. Look at the existing examples at https://github.com/uselagoon/lagoon-examples for tips, pointers and starter issues.
One small catch \u2013 wherever possible, we\u2019d like them to be built using our base Docker Hub images https://hub.docker.com/u/uselagoon \u2013 if we don\u2019t have a suitable image, or our images need modifying \u2013 throw us a PR (if you can) or create an issue (so someone else can) at https://github.com/uselagoon/lagoon-images.
Help us improve our existing examples, if you can - are we following best practices, is there something we\u2019re doing that doesn\u2019t make sense?
Bonus points for anyone that helps contribute to tests for any of these examples \u2013 we\u2019ve got some example tests in a couple of the projects you can use for guidance \u2013 https://github.com/amazeeio/drupal-example-simple/blob/8.x/TESTING_dockercompose.md. The testing framework we\u2019re using is Leia, from the excellent team behind Lando.
Help us to document our other examples better \u2013 we\u2019re not expecting a full manuscript, but tidy-ups, links to helpful resources and clarifying statements are all super-awesome.
If you have any questions, reach out to us on Discord!
"},{"location":"contributing/#i-found-a-security-issue","title":"I found a security issue \ud83d\udd13","text":"We take security very seriously. If you discover a security issue or think you found one, please bring it to the maintainers' attention.
Danger
Please send your findings to security@amazee.io. Please DO NOT file a GitHub issue for them.
Security reports are greatly appreciated and will receive public karma and swag! We're also working on a Bug Bounty system.
"},{"location":"contributing/#i-found-an-issue","title":"I found an issue","text":"We're always interested in fixing issues, therefore issue reports are very welcome. Please make sure to check that your issue does not already exist in the issue queue.
"},{"location":"contributing/#i-have-a-feature-request-or-idea","title":"I have a feature request or idea","text":"Cool! Create an issue and we're happy to look over it. We can't guarantee that it will be implemented. But we are always interested in hearing ideas of what we could bring to Lagoon.
Another good way is also to talk to us via Discord about your idea. Join today!
"},{"location":"contributing/#i-wrote-some-code","title":"I wrote some code","text":"Epic! Please send us a pull request for it, we will do our best to review it and merge it if possible.
"},{"location":"administering-lagoon/feature-flags/","title":"Feature flags","text":"Some Lagoon features can be controlled by setting feature flags. This is designed to assist users and administrators to roll out new platform features in a controlled manner.
"},{"location":"administering-lagoon/feature-flags/#environment-variables","title":"Environment variables","text":"The following environment variables can be set on an environment or project to toggle feature flags.
Environment Variable Name Active scope Version introduced Version removed Default Value DescriptionLAGOON_FEATURE_FLAG_ROOTLESS_WORKLOAD global 2.2.0 - disabled Set to enabled to set a non-root pod security context on the pods in this environment or project.This flag will eventually be deprecated, at which point non-root workloads will be enforced. LAGOON_FEATURE_FLAG_ISOLATION_NETWORK_POLICY global 2.2.0 - disabled Set to enabled to add a default namespace isolation network policy to each environment on deployment.This flag will eventually be deprecated, at which point the namespace isolation network policy will be enforced.NOTE: enabling and then disabling this feature will not remove any existing network policy from previous deployments. Those must be removed manually."},{"location":"administering-lagoon/feature-flags/#cluster-level-controls","title":"Cluster-level controls","text":"Feature flags may also be controlled at the cluster level. There is support for this in the lagoon-build-deploy chart. For each feature flag there are two flavours of values which can be set: default and force.
default controls the default policy for environments deployed to the cluster, but can be overridden at the project or environment level by the environment variables documented above.force also controls the policy for environments deployed to the cluster, but cannot be overridden by the environment variables documented above.Direct API interactions in Lagoon are done via GraphQL.
In order to authenticate with the API, we need a JWT (JSON Web Token) that allows us to use the GraphQL API as admin. To generate this token, open the terminal of the storage-calculator pod via your Kubernetes UI, or via kubectl and run the following command:
./create_jwt.py\nThis will return a long string which is the JWT token. Make a note of this, as we will need it to send queries.
We also need the URL of the API endpoint, which can be found under \"Ingresses\" in your Kubernetes UI or via kubectl on the command line. Make a note of this endpoint URL, which we will also need.
To compose and send GraphQL queries, we recommend GraphiQL.app, a desktop GraphQL client with features such as autocomplete. To continue with the next steps, install and start the app.
Under \"GraphQL Endpoint\", enter the API endpoint URL with /graphql on the end. Then click on \"Edit HTTP Headers\" and add a new header:
AuthorizationBearer [JWT token] (make sure that the JWT token has no spaces, as this would not work)Press ESC to close the HTTP header overlay and now we are ready to send the first GraphQL request!
Enter this in the left panel
Running a queryquery allProjects{\n allProjects {\n name\n }\n}\nAnd press the \u25b6\ufe0f button (or press CTRL+ENTER).
If all went well, your first GraphQL response should appear shortly afterwards in the right pane.
"},{"location":"administering-lagoon/graphql-queries/#creating-the-first-project","title":"Creating the first project","text":"Let's create the first project for Lagoon to deploy! For this we'll use the queries from the GraphQL query template in create-project.gql.
For each of the queries (the blocks starting with mutation {), fill in all of the empty fields marked by TODO comments and run the queries in GraphiQL.app. This will create one of each of the following two objects:
kubernetes : The Kubernetes (or Openshift) cluster to which Lagoon should deploy. Lagoon is not only capable of deploying to its own Kubernetes cluster, but also to any Kubernetes cluster anywhere in the world.project : The Lagoon project to be deployed, which is a Git repository with a .lagoon.yml configuration file committed in the root.In Lagoon, each developer authenticates via their SSH key(s). This determines their access to:
To allow access to the project, we first need to add a new group to the API:
Add group to APImutation {\n addGroup (\n input: {\n # TODO: Enter the name for your new group.\n name: \"\"\n }\n ) {\n id\n name\n }\n}\nThen we need to add a new user to the API:
Add new user to APImutation {\n addUser(\n input: {\n email: \"michael.schmid@example.com\"\n firstName: \"Michael\"\n lastName: \"Schmid\"\n comment: \"CTO\"\n }\n ) {\n # TODO: Make a note of the user ID that is returned.\n id\n }\n}\nThen we can add an SSH public key for the user to the API:
Add SSH public key for the user to APImutation {\n addSshKey(\n input: {\n # TODO: Fill in the name field.\n # This is a non-unique identifier for the SSH key.\n name: \"\"\n # TODO: Fill in the keyValue field.\n # This is the actual SSH public key (without the type at the beginning and without the comment at the end, ex. `AAAAB3NzaC1yc2EAAAADAQ...3QjzIOtdQERGZuMsi0p`).\n keyValue: \"\"\n # TODO: Fill in the keyType field.\n # Valid values are either SSH_RSA, SSH_ED25519, ECDSA_SHA2_NISTP256/384/521\n keyType: SSH_RSA\n user: {\n # TODO: Fill in the userId field.\n # This is the user ID that we noted from the addUser query.\n id:\"0\",\n email:\"michael.schmid@example.com\"\n }\n }\n ) {\n id\n }\n}\nAfter we add the key, we need to add the user to a group:
Add user to groupmutation {\n addUserToGroup (\n input: {\n user: {\n #TODO: Enter the email address of the user.\n email: \"\"\n }\n group: {\n #TODO: Enter the name of the group you want to add the user to.\n name: \"\"\n }\n #TODO: Enter the role of the user.\n role: OWNER\n\n }\n ) {\n id\n name\n }\n}\nAfter running one or more of these kinds of queries, the user will be granted access to create tokens via SSH, access containers and more.
"},{"location":"administering-lagoon/graphql-queries/#adding-notifications-to-the-project","title":"Adding notifications to the project","text":"If you want to know what is going on during a deployment, we suggest configuring notifications for your project, which provide:
As notifications can be quite different in terms of the information they need, each notification type has its own mutation.
As with users, we first add the notification:
Add notificationmutation {\n addNotificationSlack(\n input: {\n # TODO: Fill in the name field.\n # This is your own identifier for the notification.\n name: \"\"\n # TODO: Fill in the channel field.\n # This is the channel for the message to be sent to.\n channel: \"\"\n # TODO: Fill in the webhook field.\n # This is the URL of the webhook where messages should be sent, this is usually provided by the chat system to you.\n webhook: \"\"\n }\n ) {\n id\n }\n}\nAfter the notification is created, we can now assign it to our project:
Assign notification to projectmutation {\n addNotificationToProject(\n input: {\n notificationType: SLACK\n # TODO: Fill in the project field.\n # This is the project name.\n project: \"\"\n # TODO: Fill in the notification field.\n # This is the notification name.\n notificationName: \"\"\n # TODO: OPTIONAL\n # The kind notification class you're interested in defaults to DEPLOYMENT\n contentType: DEPLOYMENT/PROBLEM\n # TODO: OPTIONAL\n # Related to contentType PROBLEM, we can set the threshold for the kinds of problems\n # we'd like to be notified about\n notificationSeverityThreshold \"NONE/UNKNOWN/NEGLIGIBLE/LOW/MEDIUM/HIGH/CRITICAL\n }\n ) {\n id\n }\n}\nNow for every deployment you will receive messages in your defined channel.
"},{"location":"administering-lagoon/graphql-queries/#example-graphql-queries","title":"Example GraphQL queries","text":""},{"location":"administering-lagoon/graphql-queries/#adding-a-new-kubernetes-target","title":"Adding a new Kubernetes target","text":"Note
In Lagoon, both addKubernetes and addOpenshift can be used for both Kubernetes and OpenShift targets - either will work interchangeably.
The cluster to which Lagoon should deploy.
Add Kubernetes targetmutation {\n addKubernetes(\n input: {\n # TODO: Fill in the name field.\n # This is the unique identifier of the cluster.\n name: \"\"\n # TODO: Fill in consoleUrl field.\n # This is the URL of the Kubernetes cluster\n consoleUrl: \"\"\n # TODO: Fill in the token field.\n # This is the token of the `lagoon` service account created in this cluster (this is the same token that we also used during installation of Lagoon).\n token: \"\"\n }\n ) {\n name\n id\n }\n}\nThis query will add a group to a project. Users of that group will be able to access the project. They will be able to make changes, based on their role in that group.
Add a group to a projectmutation {\n addGroupsToProject (\n input: {\n project: {\n #TODO: Enter the name of the project.\n name: \"\"\n }\n groups: {\n #TODO: Enter the name of the group that will be added to the project.\n name: \"\"\n }\n }\n ) {\n id\n }\n}\nThis query adds a new Lagoon project to be deployed, which is a Git repository with a .lagoon.yml configuration file committed in the root.
If you omit the privateKey field, a new SSH key for the project will be generated automatically.
If you would like to reuse a key from another project. you will need to supply the key in the addProject mutation.
mutation {\n addProject(\n input: {\n # TODO: Fill in the name field.\n # This is the project name.\n name: \"\"\n # TODO: Fill in the private key field (replace newlines with '\\n').\n # This is the private key for a project, which is used to access the Git code.\n privateKey: \"\"\n # TODO: Fill in the Kubernetes field.\n # This is the ID of the Kubernetes or OpenShift to assign to the project.\n kubernetes: 0\n # TODO: Fill in the name field.\n # This is the project name.\n gitUrl: \"\"\n # TODO: Fill in the branches to be deployed.\n branches: \"\"\n # TODO: Define the production environment.\n productionEnvironment: \"\"\n }\n ) {\n name\n kubernetes {\n name\n id\n }\n gitUrl\n activeSystemsDeploy\n activeSystemsRemove\n branches\n pullrequests\n }\n}\nThis is a good query to see an overview of all projects, clusters and groups that exist within our Lagoon.
Get an overview of all projects, clusters, and groupsquery {\n allProjects {\n name\n gitUrl\n }\n allKubernetes {\n name\n id\n }\n allGroups{\n id\n name\n members {\n # This will display the users in this group.\n user {\n id\n firstName\n lastName\n }\n role\n }\n groups {\n id\n name\n }\n }\n}\nIf you want a detailed look at a single project, this query has been proven quite good:
Take a detailed look at one projectquery {\n projectByName(\n # TODO: Fill in the project name.\n name: \"\"\n ) {\n id\n branches\n gitUrl\n pullrequests\n productionEnvironment\n notifications(type: SLACK) {\n ... on NotificationSlack {\n name\n channel\n webhook\n id\n }\n }\n environments {\n name\n deployType\n environmentType\n }\n kubernetes {\n id\n }\n }\n}\nDon't remember the name of a project, but know the Git URL? Search no longer, there is a GraphQL query for that:
Query project by Git URLquery {\n projectByGitUrl(gitUrl: \"git@server.com:org/repo.git\") {\n name\n }\n}\nThe Lagoon GraphQL API can not only display objects and create objects, it also has the capability to update existing objects, using a patch object.
Update the branches to deploy within a project:
Update deploy branches.mutation {\n updateProject(\n input: { id: 109, patch: { branches: \"^(prod|stage|dev|update)$\" } }\n ) {\n id\n }\n}\nUpdate the production environment within a project:
Warning
This requires a redeploy in order for the changes to be reflected in the containers.
Update prod environment mutation {\n updateProject(\n input: { id: 109, patch: { productionEnvironment: \"main\" } }\n ) {\n id\n }\n}\nYou can also combine multiple changes at once:
Update prod environment and set deploy branches.mutation {\n updateProject(\n input: {\n id: 109\n patch: {\n productionEnvironment: \"main\"\n branches: \"^(prod|stage|dev|update)$\"\n }\n }\n ) {\n id\n }\n}\nYou can also use the Lagoon GraphQL API to delete an environment. You'll need to know the project name and the environment name in order to run the command.
Delete environment.mutation {\n deleteEnvironment(\n input: {\n # TODO: Fill in the name field.\n # This is the environment name.\n name:\"\"\n # TODO: Fill in the project field.\n # This is the project name.\n project:\"\"\n execute:true\n }\n )\n}\nWant to see what groups and users have access to a project? Want to know what their roles are? Do I have a query for you! Using the query below you can search for a project and display the groups, users, and roles that are assigned to that project.
Query groups, users, and roles assigned to projectquery search{\n projectByName(\n #TODO: Enter the name of the project.\n name: \"\"\n ) {\n id,\n branches,\n productionEnvironment,\n pullrequests,\n gitUrl,\n kubernetes {\n id\n },\n groups{\n id\n name\n groups {\n id\n name\n }\n members {\n role\n user {\n id\n email\n }\n }\n }\n }\n}\nProject metadata can be assigned using arbitrary key/value pairs. Projects can then be queried by the associated metadata; for example you may categorize projects by type of software, version number, or any other categorization you may wish to query on later.
"},{"location":"administering-lagoon/graphql-queries/#addupdate-metadata-on-a-project","title":"Add/update metadata on a project","text":"Updates to metadata expect a key/value pair. It operates as an UPSERT, meaning if a key already exists the value will be updated, otherwise inserted.
You may have any number of k/v pairs stored against a project.
Add a key/value pair to metadatamutation {\n updateProjectMetadata(\n input: { id: 1, patch: { key: \"type\", value: \"saas\" } }\n ) {\n id\n metadata\n }\n}\nQueries may be by key only (e.g return all projects where a specific key exists) or both key and value where both key and value must match.
All projects that have the version tag:
query projectsByMetadata {\n projectsByMetadata(metadata: [{key: \"version\"] ) {\n id\n name\n }\n}\nAll projects that have the version tag, specifically version 8:
query projectsByMetadata {\n projectsByMetadata(metadata: [{key: \"version\", value: \"8\"] ) {\n id\n name\n }\n}\nMetadata can be removed on a per-key basis. Other metadata key/value pairs will persist.
Remove metadatamutation {\n removeProjectMetadataByKey (\n input: { id: 1, key: \"version\" }\n ) {\n id\n metadata\n }\n}\nVersion 1.0 of Lagoon changed how you access your projects! Access to your project is handled via groups, with projects assigned to one or multiple groups. Users are added to groups with a role. Groups can also be nested within sub-groups. This change provides a lot more flexibility and the possibility to recreate real world teams within Lagoon.
"},{"location":"administering-lagoon/rbac/#roles","title":"Roles","text":"When assigning a user to a group, you need to provide a group role for that user inside this group. Each one of the 5 current existing group roles gives the user different permissions to the group and projects assigned to the group. Here are the platform-wide roles and the group roles that are currently found in Lagoon:
"},{"location":"administering-lagoon/rbac/#platform-wide-roles","title":"Platform-Wide Roles","text":""},{"location":"administering-lagoon/rbac/#platform-wide-admin","title":"Platform-Wide Admin","text":"The platform-wide admin has access to everything across all of Lagoon. That includes dangerous mutations like deleting all projects. Use very, very, very carefully.
"},{"location":"administering-lagoon/rbac/#platform-wide-owner","title":"Platform-Wide Owner","text":"The platform-wide owner has access to every Lagoon group, like the group owner role, and can be used if you need a user that needs access to everything but you don't want to assign the user to every group.
"},{"location":"administering-lagoon/rbac/#group-roles","title":"Group Roles","text":""},{"location":"administering-lagoon/rbac/#owner","title":"Owner","text":"The owner role can do everything within a group and its associated projects. They can add and manage users of a group. Be careful with this role, as it can delete projects and production environments!
"},{"location":"administering-lagoon/rbac/#maintainer","title":"Maintainer","text":"The maintainer role can do everything within a group and its associated projects except deleting the project itself or the production environment. They can add and manage users of a group.
"},{"location":"administering-lagoon/rbac/#developer","title":"Developer","text":"The developer role has SSH access only to development environments. This role cannot access, update or delete the production environment. They can run a sync task with the production environment as a source, but not as the destination. Cannot manage users of a group.
IMPORTANT
This role does not prevent the deployment of the production environment as a deployment is triggered via a Git push! You need to make sure that your Git server prevents these users from pushing into the branch defined as production environment.
"},{"location":"administering-lagoon/rbac/#reporter","title":"Reporter","text":"The reporter role has view access only. They cannot access any environments via SSH or make modifications to them. They can run cache-clear tasks. This role is mostly used for stakeholders to have access to Lagoon UI and logging.
"},{"location":"administering-lagoon/rbac/#guest","title":"Guest","text":"The guest role has the same privileges as the reporter role listed above.
Here is a table that lists the roles and the access they have:
"},{"location":"administering-lagoon/rbac/#lagoon-100-rbac-permission-matrix","title":"Lagoon 1.0.0 RBAC Permission Matrix","text":"SelfGuestDeveloperMaintainerOwnerPlatform-Wide OwnerPlatform-Wide Admin Name Resource Scope Attributes addSshKey ssh_key add userID updateSshKey ssh_key update userID deleteSshKey ssh_key delete userID getUserSshKeys ssh_key view:user userID updateUser user update userID deleteUser user delete userID Name Resource Scope Attributes getBackupsByEnvironmentId deployment view projectID getEnvironmentsByProjectId environment view projectID getEnvironmentServicesByEnvironmentId environment view projectID getEnvVarsByEnvironmentId env_var environment:view:development projectID getEnvVarsByEnvironmentId env_var environment:view:production projectID getEnvVarsByProjectId env_var project:view projectID addGroup group add getOpenshiftByProjectId openshift view projectID addProject project add getProjectByEnvironmentId project view projectID getProjectByGitUrl project view projectID getProjectByName project view projectID addRestore restore add projectID updateRestore restore update projectID taskDrushCacheClear task drushCacheClear:development projectID taskDrushCacheClear task drushCacheClear:production projectID taskDrushCron task drushCron:development projectID taskDrushCron task drushCron:production projectID getFilesByTaskId task view projectID getTasksByEnvironmentId task view projectID getTaskByRemoteId task view projectID getTaskById task view projectID addUser user add Name Resource Scope Attributes addBackup backup add projectID getBackupsByEnvironmentId backup view projectID addEnvVariable (to Environment) env_var environment:add:development projectID deleteEnvVariable (from Environment) env_var environment:delete:development projectID getEnvVarsByEnvironmentId env_var environment:viewValue:development projectID addOrUpdateEnvironment environment addOrUpdate:development projectID updateEnvironment environment update:development projectID deleteEnvironment environment delete:development projectID addDeployment environment deploy:development projectID setEnvironmentServices environment update:development projectID deployEnvironmentLatest environment deploy:development projectID deployEnvironmentBranch environment deploy:development projectID deployEnvironmentPullrequest environment deploy:development projectID deployEnvironmentPromote environment deploy:development projectID userCanSshToEnvironment environment ssh:development projectID getNotificationsByProjectId notification view projectID addTask task add:development projectID taskDrushArchiveDump task drushArchiveDump:development projectID taskDrushArchiveDump task drushArchiveDump:production projectID taskDrushSqlDump task drushSqlDump:development projectID taskDrushSqlDump task drushSqlDump:production projectID taskDrushUserLogin task drushUserLogin:destination:development environmentID taskDrushSqlSync task drushSqlSync:source:development projectID taskDrushSqlSync task drushSqlSync:source:production projectID taskDrushSqlSync task drushSqlSync:destination:development projectID taskDrushRsyncFiles task drushRsync:source:development projectID taskDrushRsyncFiles task drushRsync:source:production projectID taskDrushRsyncFiles task drushRsync:destination:development projectID deleteTask task delete projectID updateTask task update projectID uploadFilesForTask task update projectID deleteFilesForTask task delete projectID getBackupsByEnvironmentId deployment view projectID getEnvironmentsByProjectId environment view projectID getEnvironmentServicesByEnvironmentId environment view projectID getEnvVarsByEnvironmentId env_var environment:view:development projectID getEnvVarsByEnvironmentId env_var environment:view:production projectID getEnvVarsByProjectId env_var project:view projectID addGroup group add getOpenshiftByProjectId openshift view projectID addProject project add getProjectByEnvironmentId project view projectID getProjectByGitUrl project view projectID getProjectByName project view projectID addRestore restore add projectID updateRestore restore update projectID taskDrushCacheClear task drushCacheClear:development projectID taskDrushCacheClear task drushCacheClear:production projectID taskDrushCron task drushCron:development projectID taskDrushCron task drushCron:production projectID getFilesByTaskId task view projectID getTasksByEnvironmentId task view projectID getTaskByRemoteId task view projectID getTaskById task view projectID addUser user add Name Resource Scope Attributes deleteBackup backup delete projectID addEnvVariable (to Project) env_var project:add projectID addEnvVariable (to Environment) env_var environment:add:production projectID deleteEnvVariable env_var delete projectID deleteEnvVariable (from Project) env_var project:delete projectID deleteEnvVariable (from Environment) env_var environment:delete:production projectID getEnvVarsByProjectId env_var project:viewValue projectID getEnvVarsByEnvironmentId env_var environment:viewValue:production projectID addOrUpdateEnvironment environment addOrUpdate:production projectID updateEnvironment environment update:production projectID addDeployment environment deploy:production projectID deleteDeployment deployment delete projectID updateDeployment deployment update projectID setEnvironmentServices environment update:production projectID deployEnvironmentLatest environment deploy:production projectID deployEnvironmentBranch environment deploy:production projectID deployEnvironmentPullrequest environment deploy:production projectID deployEnvironmentPromote environment deploy:production projectID userCanSshToEnvironment environment ssh:production projectID updateGroup group update groupID deleteGroup group delete groupID addUserToGroup group addUser groupID removeUserFromGroup group removeUser groupID addNotificationToProject project addNotification projectID removeNotificationFromProject project removeNotification projectID updateProject project update projectID addGroupsToProject project addGroup projectID removeGroupsFromProject project removeGroup projectID addTask task add:production projectID taskDrushUserLogin task drushUserLogin:destination:production environmentID taskDrushSqlSync task drushSqlSync:destination:production projectID taskDrushRsyncFiles task drushRsync:destination:production projectID addBackup backup add projectID getBackupsByEnvironmentId backup view projectID addEnvVariable (to Environment) env_var environment:add:development projectID deleteEnvVariable (from Environment) env_var environment:delete:development projectID getEnvVarsByEnvironmentId env_var environment:viewValue:development projectID addOrUpdateEnvironment environment addOrUpdate:development projectID updateEnvironment environment update:development projectID deleteEnvironment environment delete:development projectID addDeployment environment deploy:development projectID setEnvironmentServices environment update:development projectID deployEnvironmentLatest environment deploy:development projectID deployEnvironmentBranch environment deploy:development projectID deployEnvironmentPullrequest environment deploy:development projectID deployEnvironmentPromote environment deploy:development projectID getNotificationsByProjectId notification view projectID addTask task add:development projectID taskDrushArchiveDump task drushArchiveDump:development projectID taskDrushArchiveDump task drushArchiveDump:production projectID taskDrushSqlDump task drushSqlDump:development projectID taskDrushSqlDump task drushSqlDump:production projectID taskDrushUserLogin task drushUserLogin:destination:development environmentID taskDrushSqlSync task drushSqlSync:source:development projectID taskDrushSqlSync task drushSqlSync:source:production projectID taskDrushSqlSync task drushSqlSync:destination:development projectID taskDrushRsyncFiles task drushRsync:source:development projectID taskDrushRsyncFiles task drushRsync:source:production projectID taskDrushRsyncFiles task drushRsync:destination:development projectID deleteTask task delete projectID updateTask task update projectID uploadFilesForTask task update projectID deleteFilesForTask task delete projectID getBackupsByEnvironmentId deployment view projectID getEnvironmentsByProjectId environment view projectID getEnvironmentServicesByEnvironmentId environment view projectID getEnvVarsByEnvironmentId env_var environment:view:development projectID getEnvVarsByEnvironmentId env_var environment:view:production projectID getEnvVarsByProjectId env_var project:view projectID addGroup group add getOpenshiftByProjectId openshift view projectID addProject project add getProjectByEnvironmentId project view projectID getProjectByGitUrl project view projectID getProjectByName project view projectID addRestore restore add projectID updateRestore restore update projectID taskDrushCacheClear task drushCacheClear:development projectID taskDrushCacheClear task drushCacheClear:production projectID taskDrushCron task drushCron:development projectID taskDrushCron task drushCron:production projectID getFilesByTaskId task view projectID getTasksByEnvironmentId task view projectID getTaskByRemoteId task view projectID getTaskById task view projectID addUser user add Name Resource Scope Attributes deleteEnvironment environment delete:production projectID deleteProject project delete projectID getProjectByEnvironmentId project viewPrivateKey projectID getProjectByGitUrl project viewPrivateKey projectID getProjectByName project viewPrivateKey projectID deleteBackup backup delete projectID addEnvVariable (to Project) env_var project:add projectID addEnvVariable (to Environment) env_var environment:add:production projectID deleteEnvVariable env_var delete projectID deleteEnvVariable (from Project) env_var project:delete projectID deleteEnvVariable (from Environment) env_var environment:delete:production projectID getEnvVarsByProjectId env_var project:viewValue projectID getEnvVarsByEnvironmentId env_var environment:viewValue:production projectID addOrUpdateEnvironment environment addOrUpdate:production projectID updateEnvironment environment update:production projectID addDeployment environment deploy:production projectID deleteDeployment deployment delete projectID updateDeployment deployment update projectID setEnvironmentServices environment update:production projectID deployEnvironmentLatest environment deploy:production projectID deployEnvironmentBranch environment deploy:production projectID deployEnvironmentPullrequest environment deploy:production projectID deployEnvironmentPromote environment deploy:production projectID updateGroup group update groupID deleteGroup group delete groupID addUserToGroup group addUser groupID removeUserFromGroup group removeUser groupID addNotificationToProject project addNotification projectID removeNotificationFromProject project removeNotification projectID updateProject project update projectID addGroupsToProject project addGroup projectID removeGroupsFromProject project removeGroup projectID addTask task add:production projectID taskDrushUserLogin task drushUserLogin:destination:production environmentID taskDrushSqlSync task drushSqlSync:destination:production projectID taskDrushRsyncFiles task drushRsync:destination:production projectID addBackup backup add projectID getBackupsByEnvironmentId backup view projectID addEnvVariable (to Environment) env_var environment:add:development projectID deleteEnvVariable (from Environment) env_var environment:delete:development projectID getEnvVarsByEnvironmentId env_var environment:viewValue:development projectID addOrUpdateEnvironment environment addOrUpdate:development projectID updateEnvironment environment update:development projectID deleteEnvironment environment delete:development projectID addDeployment environment deploy:development projectID setEnvironmentServices environment update:development projectID deployEnvironmentLatest environment deploy:development projectID deployEnvironmentBranch environment deploy:development projectID deployEnvironmentPullrequest environment deploy:development projectID deployEnvironmentPromote environment deploy:development projectID getNotificationsByProjectId notification view projectID addTask task add:development projectID taskDrushArchiveDump task drushArchiveDump:development projectID taskDrushArchiveDump task drushArchiveDump:production projectID taskDrushSqlDump task drushSqlDump:development projectID taskDrushSqlDump task drushSqlDump:production projectID taskDrushUserLogin task drushUserLogin:destination:development environmentID taskDrushSqlSync task drushSqlSync:source:development projectID taskDrushSqlSync task drushSqlSync:source:production projectID taskDrushSqlSync task drushSqlSync:destination:development projectID taskDrushRsyncFiles task drushRsync:source:development projectID taskDrushRsyncFiles task drushRsync:source:production projectID taskDrushRsyncFiles task drushRsync:destination:development projectID deleteTask task delete projectID updateTask task update projectID uploadFilesForTask task update projectID deleteFilesForTask task delete projectID getBackupsByEnvironmentId deployment view projectID getEnvironmentsByProjectId environment view projectID getEnvironmentServicesByEnvironmentId environment view projectID getEnvVarsByEnvironmentId env_var environment:view:development projectID getEnvVarsByEnvironmentId env_var environment:view:production projectID getEnvVarsByProjectId env_var project:view projectID addGroup group add getOpenshiftByProjectId openshift view projectID addProject project add getProjectByEnvironmentId project view projectID getProjectByGitUrl project view projectID getProjectByName project view projectID addRestore restore add projectID updateRestore restore update projectID taskDrushCacheClear task drushCacheClear:development projectID taskDrushCacheClear task drushCacheClear:production projectID taskDrushCron task drushCron:development projectID taskDrushCron task drushCron:production projectID getFilesByTaskId task view projectID getTasksByEnvironmentId task view projectID getTaskByRemoteId task view projectID getTaskById task view projectID addUser user add Name Resource Scope Attributes addOrUpdateEnvironmentStorage environment storage addNotificationSlack notification add updateNotificationSlack notification update deleteNotificationSlack notification delete addKubernetes kubernetes add updateKubernetes kubernetes update deleteKubernetes kubernetes delete deleteAllKubernetes kubernetes deleteAll getAllOpenshifts openshift viewAll getAllProjects project viewAll addSshKey ssh_key add userID updateSshKey ssh_key update userID deleteSshKey ssh_key delete userID getUserSshKeys ssh_key view:user userID updateUser user update userID deleteUser user delete userID deleteEnvironment environment delete:production projectID deleteProject project delete projectID getProjectByEnvironmentId project viewPrivateKey projectID getProjectByGitUrl project viewPrivateKey projectID getProjectByName project viewPrivateKey projectID deleteBackup backup delete projectID addEnvVariable (to Project) env_var project:add projectID addEnvVariable (to Environment) env_var environment:add:production projectID deleteEnvVariable env_var delete projectID deleteEnvVariable (from Project) env_var project:delete projectID deleteEnvVariable (from Environment) env_var environment:delete:production projectID getEnvVarsByProjectId env_var project:viewValue projectID getEnvVarsByEnvironmentId env_var environment:viewValue:production projectID addOrUpdateEnvironment environment addOrUpdate:production projectID updateEnvironment environment update:production projectID allEnvironments environment viewAll getEnvironmentStorageMonthByEnvironmentId environment storage getEnvironmentHoursMonthByEnvironmentId environment storage getEnvironmentHitsMonthByEnvironmentId environment storage addOrUpdateEnvironmentStorage environment storage addDeployment environment deploy:production projectID deleteDeployment deployment delete projectID updateDeployment deployment update projectID setEnvironmentServices environment update:production projectID deployEnvironmentLatest environment deploy:production projectID deployEnvironmentBranch environment deploy:production projectID deployEnvironmentPullrequest environment deploy:production projectID deployEnvironmentPromote environment deploy:production projectID updateGroup group update groupID deleteGroup group delete groupID addUserToGroup group addUser groupID removeUserFromGroup group removeUser groupID addNotificationToProject project addNotification projectID removeNotificationFromProject project removeNotification projectID updateProject project update projectID addGroupsToProject project addGroup projectID removeGroupsFromProject project removeGroup projectID addTask task add:production projectID taskDrushUserLogin task drushUserLogin:destination:production environmentID taskDrushSqlSync task drushSqlSync:destination:production projectID taskDrushRsyncFiles task drushRsync:destination:production projectID addBackup backup add projectID getBackupsByEnvironmentId backup view projectID addEnvVariable (to Environment) env_var environment:add:development projectID deleteEnvVariable (from Environment) env_var environment:delete:development projectID getEnvVarsByEnvironmentId env_var environment:viewValue:development projectID addOrUpdateEnvironment environment addOrUpdate:development projectID updateEnvironment environment update:development projectID deleteEnvironment environment delete:development projectID addDeployment environment deploy:development projectID setEnvironmentServices environment update:development projectID deployEnvironmentLatest environment deploy:development projectID deployEnvironmentBranch environment deploy:development projectID deployEnvironmentPullrequest environment deploy:development projectID deployEnvironmentPromote environment deploy:development projectID getNotificationsByProjectId notification view projectID addTask task add:development projectID taskDrushArchiveDump task drushArchiveDump:development projectID taskDrushArchiveDump task drushArchiveDump:production projectID taskDrushSqlDump task drushSqlDump:development projectID taskDrushSqlDump task drushSqlDump:production projectID taskDrushUserLogin task drushUserLogin:destination:development environmentID taskDrushSqlSync task drushSqlSync:source:development projectID taskDrushSqlSync task drushSqlSync:source:production projectID taskDrushSqlSync task drushSqlSync:destination:development projectID taskDrushRsyncFiles task drushRsync:source:development projectID taskDrushRsyncFiles task drushRsync:source:production projectID taskDrushRsyncFiles task drushRsync:destination:development projectID deleteTask task delete projectID updateTask task update projectID uploadFilesForTask task update projectID deleteFilesForTask task delete projectID getBackupsByEnvironmentId deployment view projectID getEnvironmentsByProjectId environment view projectID getEnvironmentServicesByEnvironmentId environment view projectID getEnvVarsByEnvironmentId env_var environment:view:development projectID getEnvVarsByEnvironmentId env_var environment:view:production projectID getEnvVarsByProjectId env_var project:view projectID addGroup group add getOpenshiftByProjectId openshift view projectID addProject project add getProjectByEnvironmentId project view projectID getProjectByGitUrl project view projectID getProjectByName project view projectID addRestore restore add projectID updateRestore restore update projectID taskDrushCacheClear task drushCacheClear:development projectID taskDrushCacheClear task drushCacheClear:production projectID taskDrushCron task drushCron:development projectID taskDrushCron task drushCron:production projectID getFilesByTaskId task view projectID getTasksByEnvironmentId task view projectID getTaskByRemoteId task view projectID getTaskById task view projectID addUser user add Name Resource Scope Attributes deleteAllBackups backup deleteAll deleteAllEnvironments environment deleteAll getEnvironmentStorageMonthByEnvironmentId environment storage getEnvironmentHoursMonthByEnvironmentId environment storage getEnvironmentHitsMonthByEnvironmentId environment storage deleteAllGroups group deleteAll deleteAllNotificationSlacks notification deleteAll removeAllNotificationsFromAllProjects notification removeAll getAllOpenshifts openshift viewAll deleteAllProjects project deleteAll deleteAllSshKeys ssh_key deleteAll removeAllSshKeysFromAllUsers ssh_key removeAll deleteAllUsers user deleteAll addOrUpdateEnvironmentStorage environment storage addNotificationSlack notification add updateNotificationSlack notification update deleteNotificationSlack notification delete addKubernetes kubernetes add updateKubernetes kubernetes update deleteKubernetes kubernetes delete deleteAllKubernetes kubernetes deleteAll getAllProjects project viewAll addSshKey ssh_key add userID updateSshKey ssh_key update userID deleteSshKey ssh_key delete userID getUserSshKeys ssh_key view:user userID updateUser user update userID deleteUser user delete userID deleteEnvironment environment delete:production projectID deleteProject project delete projectID getProjectByEnvironmentId project viewPrivateKey projectID getProjectByGitUrl project viewPrivateKey projectID getProjectByName project viewPrivateKey projectID deleteBackup backup delete projectID addEnvVariable (to Project) env_var project:add projectID addEnvVariable (to Environment) env_var environment:add:production projectID deleteEnvVariable env_var delete projectID deleteEnvVariable (from Project) env_var project:delete projectID deleteEnvVariable (from Environment) env_var environment:delete:production projectID getEnvVarsByProjectId env_var project:viewValue projectID getEnvVarsByEnvironmentId env_var environment:viewValue:production projectID addOrUpdateEnvironment environment addOrUpdate:production projectID updateEnvironment environment update:production projectID addDeployment environment deploy:production projectID deleteDeployment deployment delete projectID updateDeployment deployment update projectID setEnvironmentServices environment update:production projectID deployEnvironmentLatest environment deploy:production projectID deployEnvironmentBranch environment deploy:production projectID deployEnvironmentPullrequest environment deploy:production projectID deployEnvironmentPromote environment deploy:production projectID updateGroup group update groupID deleteGroup group delete groupID addUserToGroup group addUser groupID removeUserFromGroup group removeUser groupID addNotificationToProject project addNotification projectID removeNotificationFromProject project removeNotification projectID updateProject project update projectID addGroupsToProject project addGroup projectID removeGroupsFromProject project removeGroup projectID addTask task add:production projectID taskDrushUserLogin task drushUserLogin:destination:production environmentID taskDrushSqlSync task drushSqlSync:destination:production projectID taskDrushRsyncFiles task drushRsync:destination:production projectID addBackup backup add projectID getBackupsByEnvironmentId backup view projectID addEnvVariable (to Environment) env_var environment:add:development projectID deleteEnvVariable (from Environment) env_var environment:delete:development projectID getEnvVarsByEnvironmentId env_var environment:viewValue:development projectID addOrUpdateEnvironment environment addOrUpdate:development projectID updateEnvironment environment update:development projectID deleteEnvironment environment delete:development projectID addDeployment environment deploy:development projectID setEnvironmentServices environment update:development projectID deployEnvironmentLatest environment deploy:development projectID deployEnvironmentBranch environment deploy:development projectID deployEnvironmentPullrequest environment deploy:development projectID deployEnvironmentPromote environment deploy:development projectID getNotificationsByProjectId notification view projectID addTask task add:development projectID taskDrushArchiveDump task drushArchiveDump:development projectID taskDrushArchiveDump task drushArchiveDump:production projectID taskDrushSqlDump task drushSqlDump:development projectID taskDrushSqlDump task drushSqlDump:production projectID taskDrushUserLogin task drushUserLogin:destination:development environmentID taskDrushSqlSync task drushSqlSync:source:development projectID taskDrushSqlSync task drushSqlSync:source:production projectID taskDrushSqlSync task drushSqlSync:destination:development projectID taskDrushRsyncFiles task drushRsync:source:development projectID taskDrushRsyncFiles task drushRsync:source:production projectID taskDrushRsyncFiles task drushRsync:destination:development projectID deleteTask task delete projectID updateTask task update projectID uploadFilesForTask task update projectID deleteFilesForTask task delete projectID getBackupsByEnvironmentId deployment view projectID getEnvironmentsByProjectId environment view projectID getEnvironmentServicesByEnvironmentId environment view projectID getEnvVarsByEnvironmentId env_var environment:view:development projectID getEnvVarsByEnvironmentId env_var environment:view:production projectID getEnvVarsByProjectId env_var project:view projectID addGroup group add getOpenshiftByProjectId openshift view projectID addProject project add getProjectByEnvironmentId project view projectID getProjectByGitUrl project view projectID getProjectByName project view projectID addRestore restore add projectID updateRestore restore update projectID taskDrushCacheClear task drushCacheClear:development projectID taskDrushCacheClear task drushCacheClear:production projectID taskDrushCron task drushCron:development projectID taskDrushCron task drushCron:production projectID getFilesByTaskId task view projectID getTasksByEnvironmentId task view projectID getTaskByRemoteId task view projectID getTaskById task view projectID addUser user add"},{"location":"administering-lagoon/using-harbor/","title":"Harbor","text":"Harbor is used as the default package repository for Lagoon when deploying to Kubernetes infrastructure. Harbor provides a Docker registry and a container security scanning solution provided by Trivy.
Note
When running Lagoon locally, the configuration for Harbor is handled entirely automagically.
If you are running Lagoon locally, you can access that UI at localhost:8084. The username is admin and the password is admin.
Note
If you are hosting a site with a provider (such as amazee.io), they may not allow customer access to the Harbor UI.
Once logged in, the first screen is a list of all repositories your user has access to. Each \"repository\" in Harbor correlates to a project in Lagoon.
Within each Harbor repository, you'll see a list of container images from all environments with a single Lagoon project.
From here, you can drill down into an individual container in order to see its details, including an overview of its security scan results.
"},{"location":"administering-lagoon/using-harbor/security-scanning/","title":"Security Scanning","text":"Harbor comes with a built-in security scanning solution provided by the Trivy service. This service analyzes a specified container image for any installed packages, and collects the version numbers of those installed packages. The Trivy service then searches the National Vulnerability Database for any CVEs (common vulnerabilities and exposures) affecting those package versions. Trivy is also library aware, so it will scan any Composer files or other package library definition files and report any vulnerabilities found within those package versions. These vulnerabilities are then reported within Harbor for each individual container.
An example of a security scan in Harbor, showing applicable vulnerabilities for a scanned container:
"},{"location":"administering-lagoon/using-harbor/harbor-settings/","title":"Running Harbor Locally","text":"Lagoon supports running Harbor locally, and it is automatically used for hosting all Kubernetes-based builds (any time the project's activeSystemsDeploy value is set to lagoon_kubernetesBuildDeploy). When Harbor is ran locally, it makes use of MinIO as a storage backend, which is an AWS S3 compatible local storage solution.
Harbor is composed of multiple containers, which all require different settings in order for them to run successfully.
"},{"location":"administering-lagoon/using-harbor/harbor-settings/#environment-variables","title":"Environment Variables","text":"The following environment variables are required to be set in order for Harbor to properly start:
HARBOR_REGISTRY_STORAGE_AMAZON_BUCKETharbor-images when Lagoon is run locally or during CI testing.HARBOR_REGISTRY_STORAGE_AMAZON_REGIONus-east-1 when Lagoon is run locally or during CI testing.REGISTRY_STORAGE_S3_ACCESSKEYREGISTRY_STORAGE_S3_SECRETKEYThe following environment variables can be set if required:
HARBOR_REGISTRY_STORAGE_AMAZON_ENDPOINThttps://s3.amazonaws.com when this variable is not set.The following containers make use of configuration files:
The following containers do not require configuration files to run:
Harbor-Core requires a configuration file to start, which is located at /etc/core/app.conf within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
The configmap from which this config file is generated is stored within Lagoon in the services/harbor-core/harbor-core.yml file. Any changes made to this configmap will be persisted across container restarts.
_REDIS_URLharbor-redis:6379,100,._REDIS_URL_REGredis://harbor-redis:6379/2.ADMIRAL_URLNA.CFG_EXPIRATION5.CHART_CACHE_DRIVERredis.CLAIR_ADAPTER_URLhttp://harbor-trivy:8080.CLAIR_DBpostgres.CLAIR_DB_HOSTharbor-database.CLAIR_DB_PASSWORDtest123 when run locally or during CI testing.CLAIR_DB_PORT5432.CLAIR_DB_SSLMODEdisable.CLAIR_DB_USERNAMEpostgres.CLAIR_HEALTH_CHECK_SERVER_URLhttp://harbor-trivy:8080CLAIR_URLhttp://harbor-trivy:6060.CONFIG_PATH/etc/core/app.conf.CORE_SECRETsecret123 when Harbor is run locally or during CI testing.CORE_URLhttp://harbor-core:8080.DATABASE_TYPEpostgresql.HARBOR_ADMIN_PASSWORDadmin user.admin when run locally or during CI testing.HARBOR_NGINX_ENDPOINThttp://harbor-nginx:8080 when run locally or during CI testing.HTTP_PROXYHTTPS_PROXYJOBSERVICE_SECRETsecret123 when Harbor is run locally or during CI testing.JOBSERVICE_URLhttp://harbor-jobservice:8080.LOG_LEVELerror.NO_PROXYharbor-core,harbor-jobservice,harbor-database,harbor-trivy,harborregistry,harbor-portal,127.0.0.1,localhost,.local,.internal.PORTAL_URLhttp://harbor-portal:8080.POSTGRESQL_DATABASEregistry.POSTGRESQL_HOSTharbor-database.POSTGRESQL_MAX_IDLE_CONNS50.POSTGRESQL_MAX_OPEN_CONNS100.POSTGRESQL_PASSWORDPOSTGRESQL_PORT5432.POSTGRESQL_USERNAMEpostgres.POSTGRESQL_SSLMODEdisable.REGISTRY_HTTP_SECRETsecret123 when Harbor is run locally or during CI testing.REGISTRY_STORAGE_PROVIDER_NAMEs3.REGISTRY_URLhttp://harborregistry:5000.REGISTRYCTL_URLhttp://harborregistryctl:8080.ROBOT_TOKEN_DURATION999.SYNC_REGISTRYfalse.TOKEN_SERVICE_URLhttp://harbor-core:8080/service/token.TRIVY_ADAPTER_URLhttp://harbor-trivy:8080.WITH_CHARTMUSEUMfalse.WITH_CLAIRtrue.WITH_NOTARYfalse.WITH_TRIVYtrue.Harbor-Database requires specific environment variables to be set in order to start, which are stored within secrets as described in the services/harbor-database/harbor-core.yml file.
POSTGRES_DBpostgres.POSTGRES_PASSWORDtest123.POSTGRES_USERpostgres.Harbor-Jobservice requires a configuration file to start, which is located at /etc/jobservice/config.yml within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
The configmap from which this config file is generated is stored within Lagoon in the services/harbor-jobservice/harbor-jobservice.yml file. Any changes made to this configmap will be persisted across container restarts.
CORE_URLharbor-jobservice where harbor-core can be reached.http://harbor-core:8080.CORE_SECRETharbor-core.secret123 when Harbor is run locally or during CI testing.HTTP_PROXYHTTPS_PROXYJOBSERVICE_SECRETharbor-jobservice.secret123 when Harbor is run locally or during CI testing.LOG_LEVELerror.debug to enable very verbose logging.NO_PROXYharbor-core,harbor-jobservice,harbor-database,harbor-trivy,harborregistry,harbor-portal,127.0.0.1,localhost,.local,.internal.REGISTRY_CONTROLLER_URLharborregistryctl service.http://harborregistryctl:8080SCANNER_LOG_LEVELerror.debug to enable very verbose logging.SCANNER_STORE_REDIS_URLharbor-trivy how to connect to its Redis store.redis://harbor-redis:6379/4.Harbor-Trivy is configured via specific environment variables and does not use a config file.
"},{"location":"administering-lagoon/using-harbor/harbor-settings/harbor-trivy/#environment-variables","title":"Environment Variables","text":"SCANNER_LOG_LEVELerror.debug to enable very verbose logging.SCANNER_STORE_REDIS_URLredis://harbor-redis:6379/4.SCANNER_JOB_QUEUE_REDIS_URLredis://harbor-redis:6379/4.SCANNER_TRIVY_VULN_TYPEos,libraryHarborRegistry requires a configuration file to start, which is located at /etc/registry/config.yml within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
This config file is stored within the services/harborregistry/harborregistry.yml file and loaded into the container as /etc/registry/pre-config.yml.
A custom container entrypoint, services/harborregistry/entrypoint.sh, then transposes provided environment variables into this config file and saves the results as /etc/registry/config.yml.
CORE_SECRETharbor-core.secret123 when Harbor is run locally or during CI testing.HARBOR_NGINX_ENDPOINTharborregistry where its NGINX ingress controller, harbor-nginx, is running in order to construct proper push and pull instructions in the UI, among other things.http://harbor-nginx:8080 when run locally or during CI testing.JOBSERVICE_SECRETharbor-jobservice.secret123 when Harbor is run locally or during CI testing.REGISTRY_HTTP_SECRETharborregistry.secret123 when Harbor is run locally or during CI testing.REGISTRY_REDIS_PASSWORDharborregistryctl the password that should be used to connect to Redis.HarborRegistryCtl requires a configuration file to start, which is located at /etc/registryctl/config.yml within the container. Any changes made to this config file are temporary and will not persist once the pod is restarted.
The configmap from which this config file is generated is stored within Lagoon in the services/harborregistryctl/harborregistry.yml file. Any changes made to this configmap will be persisted across container restarts.
CORE_SECRETharbor-core.secret123 when Harbor is run locally or during CI testing.JOBSERVICE_SECRETharbor-jobservice.secret123 when Harbor is run locally or during CI testing.REGISTRY_HTTP_SECRETharborregistry.secret123 when Harbor is run locally or during CI testing.REGISTRY_REDIS_PASSWORDharborregistryctl the password that should be used to connect to Redis.Lagoon broadly classifies three levels in the application stack:
"},{"location":"applications/#languages","title":"Languages","text":"The core building blocks of any Lagoon project, these are usually provided by Lagoon-specific images.
"},{"location":"applications/#frameworks","title":"Frameworks","text":"These take those base images, and add in the necessary logic, tools and packages needed to serve a website, or drive an application.
"},{"location":"applications/#applications","title":"Applications","text":"Usually built on top of Frameworks, this is the layer that content editors or developers will interact with to shape the finished product.
When we reference any repositories for use on Lagoon, we usually refer to them in three ways:
"},{"location":"applications/#templates","title":"Templates","text":"These are fully-functional, cloneable starter repositories, maintained and updated regularly, ready to be extended and used with little customization.
"},{"location":"applications/#examples","title":"Examples","text":"These are fully functional repositories, maintained and updated regularly, but may require some effort to make work for your individual project.
"},{"location":"applications/#demos","title":"Demos","text":"These are repositories that have been built as a demonstration, and are usable for some of the concepts within, but aren't routinely maintained or updated.
For a more complete list, check out out our GitHub repository: https://www.github.com/lagoon-examples and our website https://lagoon.sh/application/
"},{"location":"applications/node/","title":"Node.js","text":""},{"location":"applications/node/#introduction","title":"Introduction","text":"Lagoon provides Node.js images that are based on the official Node Alpine images.
More information on how to adapt your project to run on Lagoon can be found in our Node.js Docker Images section.
"},{"location":"applications/options/","title":"Configuring Applications for use on Lagoon","text":""},{"location":"applications/options/#lagoonyml","title":"lagoon.yml","text":"Project- and environment-level configuration for Lagoon is provided in the .lagoon.yml file in your repository.
See lagoon-yml.md.
docker-compose.yml","text":"Service-level configuration for Lagoon in provided in the docker-compose.yml file in your repository. In particular, the lagoon.type and associated service labels are documented in the individual services.
See docker-compose-yml.md
Lagoon has the ability to provision storage for most services - the built-in Lagoon service types have a -persistent variant that can add in the necessary PVCs, volumes, etc. We have updated our examples to reflect this configuration locally.
Lagoon has configurations available for:
Lagoon also has the capability to utilize the dbaas-operator to automatically provision these databases using an underlying managed database service (i.e. RDS, Google Cloud Databases, Azure Database). This will happen automatically when these services are provisioned and configured for your cluster. If these are not available, a pod will be provisioned as a fallback.
"},{"location":"applications/options/#cache","title":"Cache","text":"Lagoon supports Redis as a cache backend. In production, some users provision a managed Redis service for their production environments to help them scale.
"},{"location":"applications/options/#search","title":"Search","text":"Lagoon supports Elasticsearch, Solr and OpenSearch as search providers. External search providers can also be configured if required.
"},{"location":"applications/options/#ingressroutes","title":"Ingress/Routes","text":"Lagoon auto-generates routes for services that have ingress requirements. Custom routes can be provided in the .lagoon.yml on a per-service basis.
Lagoon makes heavy use of environment variables, at build and runtime. Where these are used to provide critical configuration for your application (e.g. database config/credentials) - it is important that the local and Lagoon versions are named similarly.
See environment-variables.md.
"},{"location":"applications/other/","title":"Running other applications on Lagoon","text":"Even if Lagoon doesn't have a base image for your particular application, framework or language, Lagoon can still build it!
Extending on, or inheriting from the commons image, Lagoon can run almost any workload.
"},{"location":"applications/other/#hugo","title":"Hugo","text":"This brief example shows how to build a Hugo website and serve it as static files in an NGINX image. The commons image is used to add Hugo, copy the site in, and build it. The NGINX image is then used to serve the site, with the addition of a customized NGINX config.
nginx.dockerfileFROM uselagoon/commons as builder\n\nRUN apk add hugo git\nWORKDIR /app\nCOPY . /app\nRUN hugo\n\nFROM uselagoon/nginx\n\nCOPY --from=builder /app/public/ /app\nCOPY lagoon/static-files.conf /etc/nginx/conf.d/app.conf\n\nRUN fix-permissions /usr/local/openresty/nginx\nservices:\nnginx:\nbuild:\ncontext: .\ndockerfile: lagoon/nginx.Dockerfile\nlabels:\nlagoon.type: nginx\nLagoon supports a wide range of PHP-based applications, such as Drupal, Laravel, Wordpress, Magento and Symfony.
More information on how to adapt your PHP project to run on Lagoon can be found in our PHP-cli Docker Images and PHP-FPM Docker Images sections.
"},{"location":"applications/python/","title":"Python","text":""},{"location":"applications/python/#introduction","title":"Introduction","text":"Lagoon provides images for Python 3.7 and above that can be used to build web apps in a wide range of Python-based frameworks and applications.
More information on how to adapt your Python project to run on Lagoon can be found in our Python Docker Images section.
"},{"location":"applications/ruby/","title":"Ruby and Ruby on Rails","text":""},{"location":"applications/ruby/#introduction","title":"Introduction","text":"We provide images for Ruby 3.0 and above, built on the official Ruby alpine Docker images.
Below we assume that you're attempting to get a Rails app deployed on Lagoon, although most of the details described are really framework-neutral.
"},{"location":"applications/ruby/#getting-rails-running-on-lagoon","title":"Getting Rails running on Lagoon","text":""},{"location":"applications/ruby/#responding-to-requests","title":"Responding to requests","text":"The Ruby on Rails example in the Lagoon examples repository is instructive here.
In the docker-compose.yml we set up a service named ruby, which is the primary service that will be processing any dynamic requests.
If you look at the dockerfile specified for the ruby service, you'll see that we're exposing port 3000. The nginx service will direct any requests for non-static assets to the ruby service on this port (see the nginx configuration file for more details).
The Lagoon logging infrastructure is described in the docs here. Essentially, in order to make use of the infrastructure, logs need to be sent via a UDP message to udp://application-logs.lagoon.svc:5140.
In our Rails example, we're importing the logstash-logger gem, and then in our config/application.rb we're initializing it with the following:
if ENV.has_key?('LAGOON_PROJECT') && ENV.has_key?('LAGOON_ENVIRONMENT') then\nlagoon_namespace = ENV['LAGOON_PROJECT'] + \"-\" + ENV['LAGOON_ENVIRONMENT']\nLogStashLogger.configure do |config|\nconfig.customize_event do |event|\nevent[\"type\"] = lagoon_namespace\nend\nend\nconfig.logstash.host = 'application-logs.lagoon.svc'\nconfig.logstash.type = :udp\nconfig.logstash.port = 5140\nend\nThe example uses our PostgreSQL image (see the docker-compose.yml file). Configuring database access in Rails for Lagoon is very straightforward. Since Lagoon injects the database host, name, and credentials as environment variables, we can change our config/database.yml to be aware of these env vars, and consume them if they exist.
default: &default\nadapter: postgresql\nencoding: unicode\npool: <%= ENV.fetch(\"RAILS_MAX_THREADS\") { 5 } %>\nusername: <%= ENV.fetch(\"POSTGRES_USERNAME\") { \"drupal\" } %>\npassword: <%= ENV.fetch(\"POSTGRES_PASSWORD\") { \"drupal\" } %>\nhost: <%= ENV.fetch(\"POSTGRES_HOST\") { \"postgres\" } %>\ndatabase: <%= ENV.fetch(\"('POSTGRES_DATABASE'\") { \"drupal\" } %>\nThe WordPress template is configured to use Composer to install WordPress, its dependencies, and themes.
The WordPress template is based on the https://github.com/roots/bedrock boilerplate, but extended to match a standardized Lagoon deployment pattern.
"},{"location":"applications/wordpress/#composer-install","title":"Composer Install","text":"The template uses Composer to install WordPress and its themes.
"},{"location":"applications/wordpress/#database","title":"Database","text":"Lagoon can support MariaDB and PostgreSQL databases, but as support for PostgreSQL is limited in WordPress, it isn't recommended for use.
"},{"location":"applications/wordpress/#nginx-configuration","title":"NGINX configuration","text":"Lagoon doesn't have a built-in configuration for WordPress - instead, the template comes with a starting nginx.conf - please contribute any improvements you may find!
"},{"location":"applications/wordpress/#wp-cli","title":"WP-CLI","text":"The Lagoon template installs wp-cli into the cli image to manage your WordPress install.
Our official community meeting space is the Lagoon Discord.
We\u2019re starting this community as a place for all Lagoon users to collaborate, solve problems, share ideas, and contribute back to the Lagoon project. We\u2019re working to consolidate our community as it\u2019s currently spread out over Slack and various other places. We also wanted to invite all of our users and customers to join so that everyone can benefit from the community, no matter how they\u2019re using Lagoon.
Please remember that this is not to replace your current support channels - those will remain the same. This is a place to connect with other users as well as the Lagoon maintainers.
We ask that all community members review our Participation and Moderation Guidelines, as well as the Code of Conduct.
In addition to our Zoom Community Hours, we'll also be hosting Community Hours on Discord in 2023!
"},{"location":"community/moderation/","title":"Lagoon Moderation Guidelines","text":"These guidelines have been adapted from Drupal Diversity & Inclusion\u2019s Moderation Guidelines.
In Lagoon spaces, strive to promote understanding, empathy, and increase personal awareness of all people. This includes people from across the Drupal Community and the greater Technical Community, even those you may personally disagree with.
If kicked from the Discord, the kicked user can send a private message (PM) to the kicker or another Moderator, if desired, for re-admittance. If a disruptive person is engaging in what appears to be intentionally inflammatory, bullying, or harassing behavior provoking hostile responses (or acting in a hostile manner), kicking is faster and easier than trying to placate a disruptive person whose behavior is causing distress to other channel members.
The kick is not a ban. There are times when disruptive or triggering comments and statements are genuine and break the lines of communication between two parties. By speaking with a Moderator, the (potentially) disruptive person can be coached on using more sensitive, inclusive, and diverse-aware language, and on engaging in a more constructive manner.
"},{"location":"community/moderation/#tiered-responses","title":"Tiered Responses","text":"Tier One Response
User is welcomed in the channel, asked to read some scroll back, and given a link to participation guidelines.
Tier Two Response
User is gently reminded in channel to keep posts on topic, and/or of participation guidelines.
Tier Three Response
User is PM\u2019d by available Moderator to explain the problem(s) with their posts and given suggestions of what to do differently.
Tier Four Response
If behavior continues, User is kicked for no less than 24 hours from the Discord.
Intentionally disruptive individuals get kicked, not tiered. Repeated offenses will result in a ban.
"},{"location":"community/participation/","title":"Lagoon Participation Guidelines","text":"We ask that all members of our community, in any spaces, virtual or physical, adhere to our Code of Conduct.
These guidelines have been adapted from Drupal Diversity & Inclusion\u2019s Participation Guidelines.
Additional considerations for in-person Lagoon spaces
The Lagoon team reserves the right to terminate anyone\u2019s access to the Lagoon spaces.
"},{"location":"contributing-to-lagoon/api-debugging/","title":"API Debugging","text":"1 . Ensure the dev script at services/api/package.json includes the following:
node --inspect=0.0.0.0:9229\n2 . Update docker-compose.yml to map the dist folder and expose the 9229 port:
api:\nimage: ${IMAGE_REPO:-lagoon}/api\ncommand: yarn run dev\nvolumes:\n- ./services/api/src:/app/services/api/src\n- ./services/api/dist:/app/services/api/dist\ndepends_on:\n- api-db\n- local-api-data-watcher-pusher\n- keycloak\nports:\n- '3000:3000'\n- '9229:9229'\n3 . Add the following to .vscode/launch.json:
{\n// Use IntelliSense to learn about possible attributes.\n// Hover to view descriptions of existing attributes.\n// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387.\n\"version\": \"0.2.0\",\n\"configurations\": [\n{\n\"name\": \"Docker: Attach to Node\",\n\"type\": \"node\",\n\"request\": \"attach\",\n\"port\": 9229,\n\"address\": \"localhost\",\n\"outFiles\": [\"${workspaceRoot}/app/services/api/dist/**/*.js\"],\n\"localRoot\": \"${workspaceFolder}/services/api\",\n\"remoteRoot\": \"/app/services/api\",\n\"sourceMaps\": true,\n\"protocol\": \"inspector\"\n}\n]\n}\n4 . Rebuild/restart the containers:
Restart containersrm build/api && make build/api && docker-compose restart api\n5 . Restart VScode.
"},{"location":"contributing-to-lagoon/developing-lagoon/","title":"Developing Lagoon","text":"Development of Lagoon locally can now be performed on a local Kubernetes cluster, or via Docker Compose (as a fallback).
Note
The full Lagoon stack relies on a range of upstream projects which are currently incompatible with ARM-based architectures, such as the the M1/M2 Apple Silicon-based machines. For this reason, running or developing lagoon-core or lagoon-remote locally on these architectures is not currently supported. See https://github.com/uselagoon/lagoon/issues/3189 for more information.
Docker must be installed to build and run Lagoon locally.
"},{"location":"contributing-to-lagoon/developing-lagoon/#install-docker-and-docker-compose","title":"Install Docker and Docker Compose","text":"Please check the official docs for how to install Docker.
Docker Compose is included in Docker for Mac installations. For Linux installations see the directions here.
"},{"location":"contributing-to-lagoon/developing-lagoon/#configure-docker","title":"Configure Docker","text":"You will need to update your insecure registries in Docker. Read the instructions here on how to do that. We suggest adding the entire local IPv4 Private Address Spaces to avoid unnecessary reconfiguration between Kubernetes and Docker Compose. e.g. \"insecure-registries\" : [\"172.16.0.0/12\",\"192.168.0.0/16\"],
Running a Lagoon, Kubernetes, or Docker cluster on your local machine consumes a lot of resources. We recommend that you give your Docker host a minimum of 8 CPU cores and 12GB RAM.
"},{"location":"contributing-to-lagoon/developing-lagoon/#build-lagoon-locally","title":"Build Lagoon Locally","text":"Warning
Only consider building Lagoon this way if you intend to develop features or functionality for it, or want to debug internal processes. We will also be providing instruction to install Lagoon without building it (i.e. by using the published releases).
We're using make (see the Makefile) in order to build the needed Docker images, configure Kubernetes and run tests.
We have provided a number of routines in the Makefile to cover most local development scenarios. Here we will run through a complete process.
"},{"location":"contributing-to-lagoon/developing-lagoon/#build-images","title":"Build images","text":"-j8 tells make to run 8 tasks in parallel to speed the build up. Adjust as necessary.SCAN_IMAGES=false as a default to not scan the built images for vulnerabilities. If set to true, a scan.txt file will be created in the project root with the scan output.make -j8 build\nmake kind/test\nWarning
There are a lot of tests configured to run by default - please consider only testing locally the minimum that you need to ensure functionality. This can be done by specifying or removing tests from the TESTS variable in the Makefile.
This process will:
kind, kubectl, helm, jq.CHARTS_TREEISH variable in the Makefile to control which branch if needed).lagoon-test-suite-* pod, and can be accessed one test per container.Ideally, all of the tests pass and it's all done!
"},{"location":"contributing-to-lagoon/developing-lagoon/#view-the-test-progress-and-your-local-cluster","title":"View the test progress and your local cluster","text":"The test routine creates a local Kubeconfig file (called kubeconfig.kind.lagoon in the root of the project, that can be used with a Kubernetes dashboard, viewer or CLI tool to access the local cluster. We use tools like Lens, Octant, kubectl or Portainer in our workflows. Lagoon Core, Remote and Tests all build in the Lagoon namespace, and each environment creates its own namespace to run, so make sure to use the correct context when inspecting.
In order to use kubectl with the local cluster, you will need to use the correct Kubeconfig. This can be done for every command or it can be added to your preferred tool:
kubeconfig.kind.lagoonKUBECONFIG=./kubeconfig.kind.lagoon kubectl get pods -n lagoon\nThe Helm charts used to build the local Lagoon are cloned into a local folder and symlinked to lagoon-charts.kind.lagoon where you can see the configuration. We'll cover how to make easy modifications later in this documentation.
The Makefile includes a few simple routines that will make interacting with the installed Lagoon simpler:
Create local portsmake kind/port-forwards\nThis will create local ports to expose the UI (6060), API (7070) and Keycloak (8080). Note that this logs to stdout, so it should be performed in a secondary terminal/window.
make kind/get-admin-creds\nThis will retrieve the necessary credentials to interact with the Lagoon.
make kind/dev\nThis will re-push the images listed in KIND_SERVICES with the correct tag, and redeploy the lagoon-core chart. This is useful for testing small changes to Lagoon services, but does not support \"live\" development. You will need to rebuild these images locally first, e.g rm build/api && make build/api.
make kind/local-dev-patch\nThis will build the typescript services, using your locally installed Node.js (it should be >16.0). It will then:
nodemonwatching the code for changesgit clean -dfx as the dist folders are ignored by Git.make kind/local-dev-logging\nThis will create a standalone OpenDistro for Elasticsearch cluster in your local Docker, and configure Lagoon to dispatch all logs (Lagoon and project) to it, using the configuration in lagoon-logging.
Re-run tests.make kind/retest\n# OR\nmake kind/retest TESTS='[features-kubernetes]'\nThis will re-run a suite of tests (defined in the TESTS variable) against the existing cluster. It will re-push the images needed for tests (tests, local-git, and the data-watcher-pusher). You can specify tests to run by passing the TESTS variable inline.
If updating a test configuration, the tests image will need to be rebuilt and pushed, e.g rm build/tests && make build/tests && make kind/push-images IMAGES='tests' && make kind/retest TESTS='[api]'
make kind/push-images\n# OR\nmake kind/push-images IMAGES='tests local-git'\nThis will push all the images up to the image registry. Specifying IMAGES will tag and push specific images.
make kind/clean\nThis will remove the KinD Lagoon cluster from your local Docker.
"},{"location":"contributing-to-lagoon/developing-lagoon/#ansible","title":"Ansible","text":"The Lagoon test uses Ansible to run the test suite. Each range of tests for a specific function has been split into its own routine. If you are performing development work locally, select which tests to run, and update the $TESTS variable in the Makefile to reduce the concurrent tests running.
The configuration for these tests is held in three services:
tests is the Ansible test services themselves. The local testing routine runs each individual test as a separate container within a test-suite pod. These are listed below.local-git is a Git server hosted in the cluster that holds the source files for the tests. Ansible pulls and pushes to this repository throughout the testsapi-data-watcher-pusher is a set of GraphQL mutations that pre-populates local Lagoon with the necessary Kubernetes configuration, test user accounts and SSH keys, and the necessary groups and notifications. Note that this will wipe local projects and environments on each run.The individual routines relevant to Kubernetes are:
active-standby-kubernetes runs tests to check active/standby in Kubernetes.api runs tests for the API - branch/PR deployment, promotion.bitbucket, gitlab and github run tests for the specific SCM providers.drupal-php74 runs a single-pod MariaDB, MariaDB DBaaS and a Drush-specific test for a Drupal 8/9 project (drupal-php73 doesn't do the Drush test).drupal-postgres runs a single-pod PostgreSQL and a PostgreSQL DBaaS test for a Drupal 8 project.elasticsearch runs a simple NGINX proxy to an Elasticsearch single-pod.features-variables runs tests that utilize variables in Lagoon.features-kubernetes runs a range of standard Lagoon tests, specific to Kubernetes.features-kubernetes-2 runs more advanced kubernetes-specific tests - covering multi-project and subfolder configurations.nginx, node and python run basic tests against those project types.node-mongodb runs a single-pod MongoDB test and a MongoDB DBaaS test against a Node.js app.Most services are written in Node.js. As many of these services share similar Node.js code and Node.js packages, we're using a feature of Yarn, called Yarn workspaces. Yarn workspaces need a package.json in the project's root directory that defines the workspaces.
The development of the services can happen directly within Docker. Each container for each service is set up in a way that its source code is mounted into the running container (see docker-compose.yml). Node.js itself is watching the code via nodemon , and restarts the Node.js process automatically on a change.
The services not only share many Node.js packages, but also share actual custom code. This code is within node-packages/lagoon-commons. It will be automatically symlinked by Yarn workspaces. Additionally, the nodemon of the services is set up in a way that it checks for changes in node-packages and will restart the node process automatically.
Rebuild the images via:
Rebuild images make clean\n make build\nnode_modules content when I try to build / run a Node.js based image","text":"Make sure to run yarn in Lagoon's root directory, since some services have common dependencies managed by yarn workspaces.
nip.io domains","text":"ErrorError response from daemon: Get https://registry.172.18.0.2.nip.io:32080/v2/: dial tcp: lookup registry.172.18.0.2.nip.io: no such host\nThis can happen if your local resolver filters private IPs from results. You can work around this by editing /etc/resolv.conf and adding a line like nameserver 8.8.8.8 at the top to use a public resolver that doesn't filter results.
Here are some development scenarios and useful workflows for getting things done.
"},{"location":"contributing-to-lagoon/developing-lagoon/#add-tests","title":"Add tests","text":"tests/tests/features-variables.yaml and add a test case.tests image.rm build/tests\nmake -j8 build/tests\ntests image into the cluster registry.make kind/push-images IMAGES=tests\nmake kind/retest TESTS='[features-variables]'\nWe really value anything that you can offer us!
We've made building and viewing the documentation really straightforward, and the team is always ready to help out with reviews or pointers.
We use mkdocs with the excellent Material theme.
"},{"location":"contributing-to-lagoon/documentation/#viewing-and-updating-docs-locally","title":"Viewing and updating docs locally","text":"From the root of the Lagoon repository (you'll need Docker), run:
Get local docs up and running.docker run --rm -it -p 127.0.0.1:8000:8000 -v ${PWD}:/docs ghcr.io/amazeeio/mkdocs-material\nThis will start a development server on http://127.0.0.1:8000, configured to live-reload on any updates.
The customized Docker image contains all the necessary extensions.
Alternatively, to run the mkdocs package locally, you'll need to install mkdocs, and then install all of the necessary plugins.
pip3 install -r docs/requirements.txt\nmkdocs serve\nEach documentation page also has an \"edit\" pencil in the top right, that will take you to the correct page in the Git repository.
Feel free to contribute here, too - you can always use the built-in github.dev web-based editor. It's got basic Markdown previews, but none of the mkdocs loveliness.
"},{"location":"contributing-to-lagoon/documentation/#how-we-deploy-documentation","title":"How we deploy documentation","text":"We use the Deploy MkDocs GitHub Action to build all main branch pushes, and trigger a deployment of the gh-pages branch.
Lagoon has a number of moving parts, making releases quite complicated!
"},{"location":"contributing-to-lagoon/releasing/#lagoon-core-tags-and-testing","title":"Lagoon-core - tags and testing","text":"lagoon-ui and build-deploy-tool in the format core-v2.MINOR.PATCH. Note that there are no other tags or releases on build-deploy-tool, but lagoon-ui also has it's own semver releases that are based on it's features.In the chart.yaml for the lagoon-core and lagoon-test charts, update the following fields:
sample chart.yml snippets
# This is the chart version. This version number should be incremented each\n# time you make changes to the chart and its templates, including the app\n# version.\n# Versions are expected to follow Semantic Versioning (https://semver.org/)\nversion: 1.28.0\n# This is the version number of the application being deployed. This version\n# number should be incremented each time you make changes to the application.\n# Versions are not expected to follow Semantic Versioning. They should reflect\n# the version the application is using.\nappVersion: v2.14.2\n# This section is used to collect a changelog for artifacthub.io\n# It should be started afresh for each release\n# Valid supported kinds are added, changed, deprecated, removed, fixed and security\nannotations:\nartifacthub.io/changes: |\n- kind: changed\ndescription: update Lagoon appVersion to v2.14.2\nCreate a PR for this chart release, and the Github Actions suite will undertake a full suite of tests:
Usually, failures in the lint and test charts are well explained (missing/misconfigured chart settings). If a single Lagoon test failes, it may just need re-running. If multiple failures occur, they will need investigating.
Once those tests have all passed successfully, you can proceed with creating the releases:
"},{"location":"contributing-to-lagoon/releasing/#lagoon-core-releases-and-release-notes","title":"Lagoon-core - releases and release notes","text":"Lagoon remote has a release cycle separate to Lagoon Core, and as such, can be released anytime that a dependency sub-chart or service is updated.
"},{"location":"contributing-to-lagoon/tests/","title":"Tests","text":"All of our tests are written with Ansible and mostly follow this approach:
tests/files) into this Git repository.Lagoon is mostly tested in 3 different ways:
"},{"location":"contributing-to-lagoon/tests/#1-locally","title":"1. Locally","text":"During local development, the best way to test is locally. All tests are started via make. Make will download and build all the required dependencies.
make tests\nThis will run all defined tests. If you only want to run a subset of the tests, run make tests-list to see all existing tests and run them individually.
For example, make tests/node will run the Node.js Docker images tests.
In order to actually see what is happening inside the microservices, we can use make logs:
make logs\nOr only for a specific service:
Make logsmake logs service=webhook-handler\nIn order to test pull requests that are created against Lagoon, we have a fully automatic integration test running on a dedicated Jenkins instance: https://ci.lagoon.sh. It is defined inside the .Jenkinsfile, and runs automatically for every pull request that is opened.
This will build all images, start a Kubernetes cluster and run a series of tests.
The tests can be found here:
The Lagoon commons Docker image. Based on the official Alpine images.
This image has no functionality itself, but is instead a base image, intended to be extended and utilized to build other images. All the alpine-based images in Lagoon inherit components from commons.
"},{"location":"docker-images/commons/#included-tooling","title":"Included tooling","text":"docker-sleep - standardized one-hour sleepfix-permissions - automatically fixes permissions on a given directory to all group read-writewait-for - a small script to ensure that services are up and running in the correct order - based off https://github.com/eficode/wait-forentrypoint-readiness - checks to make sure that long-running entrypoints have completedentrypoints - a script to source all entrypoints under /lagoon/entrypoints/* in an alphabetical/numerical orderThe list of default entrypoints in this image is found at https://github.com/uselagoon/lagoon-images/tree/main/images/commons/lagoon/entrypoints. Subsequent downstream images will also contribute entrypoints under /lagoon that are run in the eventual image.
MariaDB is the open source successor to MySQL.
The Lagoon MariaDB image Dockerfile. Based on the official packages mariadb and mariadb-client provided by the the upstream Alpine image.
This Dockerfile is intended to be used to set up a standalone MariaDB database server.
uselagoon/mariadb-10.4uselagoon/mariadb-10.5uselagoon/mariadb-10.6uselagoon/mariadb-10.11Info
As these images are not built from the upstream MariaDB images, their support follows a different cycle - and will only receive updates as long as the underlying Alpine images receive support - see https://alpinelinux.org/releases/ for more information. In practice, most MariaDB users will only be running these containers locally - the production instances will use the Managed Cloud Databases provided by the DBaaS Operator
"},{"location":"docker-images/mariadb/#lagoon-adaptions","title":"Lagoon adaptions","text":"The default exposed port of MariaDB containers is port 3306.
To allow Lagoon to select the best way to run the MariaDB container, use lagoon.type: mariadb - this allows the DBaaS operator to provision a cloud database if available in the cluster. Use lagoon.type: mariadb-single to specifically request MariaDB in a container. Persistent storage is always provisioned for MariaDB containers at /var/lib/mysql.
This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user.readiness-probe.sh script to check when MariaDB container is ready.docker-compose.yml snippet","text":"docker-compose.yml mariadb:\nimage: uselagoon/mariadb-10.6-drupal:latest\nlabels:\n# tells Lagoon this is a MariaDB database\nlagoon.type: mariadb\nports:\n# exposes the port 3306 with a random local port, find it with `docker-compose port mariadb 3306`\n- \"3306\"\nvolumes:\n# mounts a named volume at the default path for MariaDB\n- db:/var/lib/mysql\nmysqltuner.pl - Perl script useful for database parameter tuning.mysql-backup.sh - Script for automating the daily MySQL backups on development environment.pwgen - Utility to generate random and complex passwords.my.cnf configuration file","text":"The image ships a default MariaDB configuration file, optimized to work on Lagoon. Some options are configurable via environment variables.
"},{"location":"docker-images/mariadb/#environment-variables","title":"Environment Variables","text":"Environment Variable Default Description MARIADB_DATABASE lagoon Database name created at startup. MARIADB_USER lagoon Default user created at startup. MARIADB_PASSWORD lagoon Password of default user created at startup. MARIADB_ROOT_PASSWORD Lag00n MariaDB root user's password. MARIADB_CHARSET utf8mb4 Set the server charset. MARIADB_COLLATION utf8mb4_bin Set server collation. MARIADB_MAX_ALLOWED_PACKET 64M Set themax_allowed_packet size. MARIADB_INNODB_BUFFER_POOL_SIZE 256M Set the MariaDB InnoDB buffer pool size. MARIADB_INNODB_BUFFER_POOL_INSTANCES 1 Number of InnoDB buffer pool instances. MARIADB_INNODB_LOG_FILE_SIZE 64M Size of InnoDB log file. MARIADB_LOG_SLOW (not set) Variable to control the save of slow queries. MARIADB_LOG_QUERIES (not set) Variable to control the save of ALL queries. BACKUPS_DIR /var/lib/mysql/backup Default path for databases backups. MARIADB_DATA_DIR /var/lib/mysql Path of the MariaDB data dir, be careful, changing this can occur data loss! MARIADB_COPY_DATA_DIR_SOURCE (not set) Path which the entrypoint script of mariadb will use to copy into the defined MARIADB_DATA_DIR, this can be used for prepopulating the MariaDB with a database. The scripts expects actual MariaDB data files and not a sql file! Plus it only copies data if the destination does not already have a mysql datadir in it. If the LAGOON_ENVIRONMENT_TYPE variable is set to production, performances are set accordingly by using MARIADB_INNODB_BUFFER_POOL_SIZE=1024 and MARIADB_INNODB_LOG_FILE_SIZE=256.
MongoDB is a general purpose, document-based, distributed database built for modern application developers and for the cloud era. MongoDB is a document database, which means it stores data in JSON-like documents.
4.0 Dockerfile - uselagoon/mongo-4
This Dockerfile is intended to be used to set up a standalone MongoDB database server.
"},{"location":"docker-images/mongodb/#lagoon-adaptions","title":"Lagoon adaptions","text":"This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user, and therefore also on Kubernetes or OpenShift.The Lagoon nginx image Dockerfile. Based on the official openresty/openresty images.
This Dockerfile is intended to be used as a base for any web servers within Lagoon.
"},{"location":"docker-images/nginx/#lagoon-adaptions","title":"Lagoon adaptions","text":"The default exposed port of NGINX containers is port 8080.
This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user./etc/nginx/* are parsed through envplate with a container-entrypoint.NGINX configuration (static-files.conf)","text":"Warning
By default NGINX only serves static files - this can be used for static sites that don't require a database or PHP components: for example, static site generators like Hugo, Jekyll or Gatsby.
If you need PHP, have a look at the php-fpm image and use nginx and php-fpm in tandem.
Build the content during the build process and inject it into the nginx container.
redirects-map.conf","text":"In order to create redirects, we have redirects-map.conf in place. This helps you to redirect marketing domains to sub-sites or do non-www to www redirects. If you have a lot of redirects, we suggest having redirects-map.conf stored next to your code for easier maintainability.
Note
If you only have a few redirects, there's a handy trick to create the redirects with a RUN command in your nginx.dockerfile.
Here's an example showing how to redirect www.example.com to example.com and preserve the request:
RUN echo \"~^www.example.com http://example.com\\$request_uri;\" >> /etc/nginx/redirects-map.conf\nTo get more details about the various types of redirects that can be achieved, see the documentation within the redirects-map.conf directly.
After you put the redirects-map.conf in place, you also need to include it in your nginx.dockerfile in order to get the configuration file into your build.
COPY redirects-map.conf /etc/nginx/redirects-map.conf\nBasic authentication is enabled automatically when the BASIC_AUTH_USERNAME and BASIC_AUTH_PASSWORD environment variables are set.
Warning
Automatic basic auth configuration is provided for convenience. It should not be considered a secure method of protecting your website or private data.
"},{"location":"docker-images/nginx/#environment-variables","title":"Environment Variables","text":"Some options are configurable via environment variables.
Environment Variable Default Description BASIC_AUTH restricted Set tooff to disable basic authentication. BASIC_AUTH_USERNAME (not set) Username for basic authentication. BASIC_AUTH_PASSWORD (not set) Password for basic authentication (unencrypted). FAST_HEALTH_CHECK (not set) Set to true to redirect GET requests from certain user agents (StatusCake, Pingdom, Site25x7, Uptime, nagios) to the lightweight Lagoon service healthcheck."},{"location":"docker-images/nodejs/","title":"Node.js","text":"The Lagoon Node.js Docker image. Based on the official Node Alpine images.
We ship 2 versions of Node.js images: the normal node:version image and the node:version-builder.
The builder variant of those images comes with additional tooling that is needed when you build Node.js apps (such as the build libraries, npm and Yarn). For a full list check out their Dockerfile.
uselagoon/node-12uselagoon/node-14uselagoon/node-16uselagoon/node-18uselagoon/node-20Tip
We stop updating EOL Node.js images usually with the Lagoon release that comes after the officially communicated EOL date: https://nodejs.org/en/about/releases/.
"},{"location":"docker-images/nodejs/#lagoon-adaptions","title":"Lagoon adaptions","text":"The default exposed port of Node.js containers is port 3000.
Persistent storage is configurable in Lagoon, using the lagoon.type: node-persistent. See the docs for more info
Use the following labels in your docker-compose.yml file to configure it:
lagoon.persistent = use this to define the path in the container to use as persistent storage - e.g. /app/files.lagoon.persistent.size = this to tell Lagoon how much storage to assign this path.lagoon.persistent.name = (optional) use this to tell Lagoon to use the storage defined in another named service.docker-compose.yml snippet","text":"docker-compose.yml node:\nbuild:\n# this configures a build from a Dockerfile in the root folder\ncontext: .\ndockerfile: Dockerfile\nlabels:\n# tells Lagoon this is a node service, configured with 500MB of persistent storage at /app/files\nlagoon.type: node-persistent\nlagoon.persistent: /app/files\nlagoon.persistent.size: 500Mi\nports:\n# local development only\n# this exposes the port 3000 with a random local port\n# find it with `docker-compose port node 3000`\n- \"3000\"\nvolumes:\n# local development only\n# mounts a named volume (files) at the defined path for this service to replicate production\n- files:/app/files\nOpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
uselagoon/opensearch-2Some options are configurable via environment variables.
Environment Variable Default Description OPENSEARCH_JAVA_OPTS -Xms512m -Xmx512m Sets the memory usage of the OpenSearch container. Both values need be the same value or OpenSearch will not start cleanly."},{"location":"docker-images/opensearch/#known-issues","title":"Known issues","text":"On Linux-based systems, the start of the OpenSearch container may fail due to a low vm.max_map_count setting.
opensearch_1 | ERROR: [1] bootstrap checks failed\nopensearch_1 | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]\nSolution to this issue can be found here.
"},{"location":"docker-images/php-cli/","title":"PHP-CLI","text":"The Lagoon php-cli Docker image. Based on Lagoon php-fpm image, it has all the needed command line tools for daily operations.
Containers (or pods) started from cli images are responsible for building code for Composer or Node.js based projects.
The image also contains database clis for both MariaDB and PostgreSQL.
Info
This Dockerfile is intended to be used as a base for any cli needs within Lagoon.
uselagoon/php-8.0-cliuselagoon/php-8.1-cliuselagoon/php-8.2-cliAll PHP versions use their own Dockerfiles.
"},{"location":"docker-images/php-cli/#lagoon-adaptions","title":"Lagoon adaptions","text":"This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user.COMPOSER_ALLOW_SUPERUSER=1 removes warning about use of Composer as root.80-shell-timeout.sh script checks if containers are running in a Kubernetes environment and then set a 10 minutes timeout to idle cli pods.cli containers use an SSH key injected by Lagoon or defined into SSH_PRIVATE_KEYenvironment variable.The included CLI tools are:
composer version 1.9.0 (changeable via COMPOSER_VERSION and COMPOSER_HASH_SHA256)node.js verison 17 (as of Mar 2022)npmyarnmariadb-clientpostgresql-clientBy default this image ships with the nodejs-current package (v17 as of Mar 2022). If you need another version you can remove the current version and install the one of your choice. For example, to install Node.js 16, modify your dockerfile to include:
RUN apk del nodejs-current \\\n&& apk add --no-cache nodejs=~16\nSome options are configurable via environment variables. The php-fpm environment variables also apply.
Name Default Description MARIADB_MAX_ALLOWED_PACKET 64M Controls the max allowed packet for the MySql client."},{"location":"docker-images/php-fpm/","title":"PHP-FPM","text":"The Lagoon php-fpm Docker image. Based on the official PHP Alpine images.
PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation with some additional features useful for sites of any size, especially busier sites.
FastCGI is a way of having server scripts execute time-consuming code just once instead of every time the script is loaded, reducing overhead.
Info
This Dockerfile is intended to be used as a base for any PHP needs within Lagoon. This image itself does not create a web server, rather a php-fpm fastcgi listener. You may need to adapt the php-fpm pool config.
uselagoon/php-7.3-fpmuselagoon/php-7.4-fpmuselagoon/php-8.0-fpmuselagoon/php-8.1-fpmuselagoon/php-8.2-fpmAll PHP versions use their own Dockerfiles.
Tip
We stop updating End of Life (EOL) PHP images usually with the Lagoon release that comes after the officially communicated EOL date: https://www.php.net/supported-versions.php. Previous published versions will remain available.
"},{"location":"docker-images/php-fpm/#lagoon-adaptions","title":"Lagoon adaptions","text":"This image is prepared to be used on Lagoon. There are therefore some things are already done:
fix-permissions, so this image will work with a random user./usr/local/etc/php/php.ini and /usr/local/etc/php-fpm.conf, plus all files within /usr/local/etc/php-fpm.d/ , are parsed through envplate with a container-entrypoint.PHP extensions.The included PHP config contains sensible values that will make the creation of PHP pools config easier. Here is a list of some of these. Check /usr/local/etc/php.ini, /usr/local/etc/php-fpm.conf for all of them:
max_execution_time = 900 Changeable via PHP_MAX_EXECUTION_TIME. realpath_cache_size = 256k For handling big PHP projects. memory_limit = 400M For big PHP projects (changeable via PHP_MEMORY_LIMIT). opcache.memory_consumption = 265 For big PHP projects. opcache.enable_file_override = 1 and opcache.huge_code_pages = 1 For faster PHP. display_errors = Off and display_startup_errors = Off For sensible production values (changeable via PHP_DISPLAY_ERRORS and PHP_DISPLAY_STARTUP_ERRORS). upload_max_filesize = 2048M For big file uploads. apc.shm_size = 32m and apc.enabled = 1 Changeable via PHP_APC_SHM_SIZE and PHP_APC_ENABLED. Also, php-fpm error logging happens in stderr.
\ud83d\udca1 If you don't like any of these configs, you have three possibilities:
fpm-pool config and set via php_admin_value and php_admin_flag.this documentation for Running PHP as an Apache module. This documentation refers to Apache, but it is also the case for php-fpm).Important:
1. If you want to provide your own php-fpm pool, overwrite the file /usr/local/etc/php-fpm.d/www.conf with your own config, or rename this file if you want it to have another name. If you don't do that, the provided pool will be started! 2. PHP values with the PHP_INI_SYSTEM changeable mode cannot be changed via an fpm-pool config. They need to be changed either via already provided environment variables or: 3. Provide your own php.ini or php-fpm.conf file (this is the least preferred method).
This image is shipped with an fpm-pool config (php-fpm.d/www.conf) that creates an fpm-pool and listens on port 9000. This is because we try to provide an image which already covers most needs for PHP, so you don't need to create your own. You are welcome to do so if you like, though!
Here a short description of what this file does:
dynamic and creates between 2-50 children.php-fpm pool children after 500 requests to prevent memory leaks.pong when making a fastcgi request to /ping (good for automated testing to check if the pool started).catch_workers_output = yes to see PHP errors.clear_env = no to be able to inject PHP environment variables via regular Docker environment variables.Some options are configurable via environment variables.
Environment Variable Default Description NEWRELIC_ENABLED false Enable NewRelic performance monitoring, needsNEWRELIC_LICENSE be configured. NEWRELIC_LICENSE (not set) NewRelic license to be used. Important: NEWRELIC_ENABLED needs to be set totrue in order for NewRelic to be enabled. NEWRELIC_BROWSER_MONITORING_ENABLED true This enables auto-insertion of the JavaScript fragments for NewRelic browser monitoring. Important: NEWRELIC_ENABLED needs to be set totrue in order for NewRelic to be enabled. NEWRELIC_DISTRIBUTED_TRACING_ENABLED false This enables distributed tracing. Important: NEWRELIC_ENABLED needs to be set totrue in order for NewRelic to be enabled. PHP_APC_ENABLED 1 Can be set to 0 to disable APC. PHP_APC_SHM_SIZE 32m The size of each shared memory segment given. PHP_DISPLAY_ERRORS Off Configures whether errors are printed or hidden. See php.net. PHP_DISPLAY_STARTUP_ERRORS Off Configures whether startup errors are printed or hidden. See php.net. PHP_ERROR_REPORTING Production E_ALL & ~E_DEPRECATED & ~E_STRICT Development: E_ALL & ~E_DEPRECATED & ~E_STRICT & ~E_NOTICE The desired logging level you'd like PHP to use. See php.net. PHP_FPM_PM_MAX_CHILDREN 50 The the maximum number of child processes. See php.net. PHP_FPM_PM_MAX_REQUESTS 500 The number of requests each child process should execute before re-spawning. See php.net. PHP_FPM_PM_MAX_SPARE_SERVERS 2 The desired maximum number of idle server processes. See php.net. PHP_FPM_PM_MIN_SPARE_SERVERS 2 The desired minimum number of idle server processes. See php.net. PHP_FPM_PM_PROCESS_IDLE_TIMEOUT 60s The number of seconds after which an idle process will be killed. See php.net. PHP_FPM_PM_START_SERVERS 2 The number of child processes created on startup. See php.net. PHP_MAX_EXECUTION_TIME 900 Maximum execution time of each script, in seconds. See php.net. PHP_MAX_FILE_UPLOADS 20 The maximum number of files allowed to be uploaded simultaneously. See php.net. PHP_MAX_INPUT_VARS 2000 How many input variables will be accepted. See php.net. PHP_MEMORY_LIMIT 400M Maximum amount of memory a script may consume. See php.net. XDEBUG_ENABLE (not set) Set to true to enable xdebug extension. BLACKFIRE_ENABLED (not set) Set to true to enable blackfire extension. BLACKFIRE_SERVER_ID (not set) Set to Blackfire Server ID provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true. BLACKFIRE_SERVER_TOKEN (not set) Set to Blackfire Server Token provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true. BLACKFIRE_LOG_LEVEL 3 Change the log level of the blackfire agent. Available values: log verbosity level (4: debug, 3: info, 2: warning, 1: error) See blackfire.io."},{"location":"docker-images/postgres/","title":"PostgreSQL","text":"The Lagoon PostgreSQL Docker image. Based on the official PostgreSQL Alpine images.
"},{"location":"docker-images/postgres/#supported-versions","title":"Supported versions","text":"uselagoon/postgres-11uselagoon/postgres-12uselagoon/postgres-13uselagoon/postgres-14uselagoon/postgres-15Tip
We stop updating EOL PostgreSQL images usually with the Lagoon release that comes after the officially communicated EOL date: https://www.postgresql.org/support/versioning
"},{"location":"docker-images/postgres/#lagoon-adaptions","title":"Lagoon adaptions","text":"The default exposed port of Postgres containers is port 5432.
To allow Lagoon to select the best way to run the Postgres container, use lagoon.type: postgres - this allows DBaaS operator to provision a cloud database if available in the cluster. Use lagoon.type: postgres-single to specifically request Postgres in a container. Persistent storage is always provisioned for postgres containers at /var/lib/postgresql/data.
docker-compose.yml snippet","text":"docker-compose.ymlpostgres:\nimage: uselagoon/postgres-14-drupal:latest\nlabels:\n# tells Lagoon this is a Postgres database\nlagoon.type: postgres\nports:\n# exposes the port 5432 with a random local port\n# find it with `docker-compose port postgres 5432`\n- \"5432\"\nvolumes:\n# mounts a named volume at the default path for Postgres\n- db:/var/lib/postgresql/data\nIf you have SQL statements that need to be run immediately after container startup to initialize the database, you can place those .sql files in the container's docker-entrypoint-initdb.d directory. Any .sql files contained in that directory are run automatically at startup, as part of bringing the PostgreSQL container up.
Warning
These scripts are only run if the container is started with an empty database.
"},{"location":"docker-images/python/","title":"Python","text":"The Lagoon python Docker image. Based on the official Python Alpine images.
uselagoon/python-2.7uselagoon/python-3.7uselagoon/python-3.8uselagoon/python-3.9uselagoon/python-3.10uselagoon/python-3.11Tip
We stop updating and publishing EOL Python images usually with the Lagoon release that comes after the officially communicated EOL date: https://devguide.python.org/versions/#versions. Previous published versions will remain available.
"},{"location":"docker-images/python/#lagoon-adaptions","title":"Lagoon adaptions","text":"The default exposed port of Python containers is port 8800.
Persistent storage is configurable in Lagoon, using the lagoon.type: python-persistent. See the docs for more info
Use the following labels in your docker-compose.yml file to configure it: lagoon.persistent = use this to define the path in the container to use as persistent storage - e.g. /app/files lagoon.persistent.size = this to tell Lagoon how much storage to assign this path
If you have multiple services that share the same storage, use this lagoon.persistent.name = (optional) use this to tell Lagoon to use the storage defined in another named service
docker-compose.yml snippet","text":"docker-compose.ymlpython:\nbuild:\n# this configures a build from a Dockerfile in the root folder\ncontext: .\ndockerfile: Dockerfile\nlabels:\n# tells Lagoon this is a python service, configured with 500MB of persistent storage at /app/files\nlagoon.type: python-persistent\nlagoon.persistent: /app/files\nlagoon.persistent.size: 500Mi\nports:\n# local development only\n# this exposes the port 8800 with a random local port\n# find it with `docker-compose port python 8800`\n- \"8800\"\nvolumes:\n# local development only\n# mounts a named volume (files) at the defined path for this service to replicate production\n- files:/app/files\nThe Lagoon RabbitMQ Dockerfile with management plugin installed. Based on the official rabbitmq:3-management image at docker-hub.
This Dockerfile is intended to be used to set up a standalone RabbitMQ queue broker, as well as a base image to set up a cluster with high availability queue support by default (Mirrored queues).
By default, the RabbitMQ broker is started as single node. If you want to start a cluster, you need to use the rabbitmq-cluster Docker image, based on rabbitmq image plus the rabbitmq_peer_discovery_k8s plugin.
uselagoon/rabbitmqThis image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user./etc/rabbitmq/definitions.json is parsed through envplate with a container-entrypoint.policymust exist.In the definitions.json schema file, minimal entities are defined to make the
container run: virtualhost (vhost), username , and password to access management
UI, permissions , and policies.
By default, a policy called lagoon-ha is created at startup, but it is not active because it doesn't match any queue's name pattern (see default Environment Variables).
\"policies\":[\n{\"vhost\":\"${RABBITMQ_DEFAULT_VHOST}\",\"name\":\"lagoon-ha\",\"pattern\":\"${RABBITMQ_DEFAULT_HA_PATTERN}\", \"definition\":{\"ha-mode\":\"exactly\",\"ha-params\":2,\"ha-sync-mode\":\"automatic\",\"ha-sync-batch-size\":5}}\n]\nBy default, the ha-mode is set to exactly which controls the exact number of mirroring nodes for a queue (mirrors). The number of nodes is controller by ha-params.
For further information and custom configuration, please refer to official RabbitMQ documentation.
"},{"location":"docker-images/rabbitmq/#environment-variables","title":"Environment Variables","text":"Some options are configurable via environment variables.
Environment Variable Default Description RABBITMQ_DEFAULT_USER guest Username for management UI access. RABBITMQ_DEFAULT_PASS guest Password for management UI access. RABBITMQ_DEFAULT_VHOST / RabbitMQ main virtualhost. RABBITMQ_DEFAULT_HA_PATTERN ^$ Regular expression to match for mirrored queues."},{"location":"docker-images/redis/","title":"Redis","text":"Lagoon Redis image Dockerfile, based on offical redis:alpine image.
This Dockerfile is intended to be used to set up a standalone Redis ephemeral server by default.
"},{"location":"docker-images/redis/#supported-versions","title":"Supported versions","text":"uselagoon/redis-5 or uselagoon/redis-5-persistentuselagoon/redis-6 or uselagoon/redis-6-persistentuselagoon/redis-7 or uselagoon/redis-7-persistentThere are 2 different flavors of Redis Images: Ephemeral and Persistent.
"},{"location":"docker-images/redis/#ephemeral","title":"Ephemeral","text":"The ephemeral image is intended to be used as an in-memory cache for applications and will not retain data across container restarts.
When being used as an in-memory (RAM) cache, the first thing you might want to tune if you have large caches is to adapt the MAXMEMORY variable. This variable controls the maximum amount of memory (RAM) which redis will use to store cached items.
The persistent Redis image will persist data across container restarts and can be used for queues or application data that will need persistence.
We don't typically suggest using a persistent Redis for in-memory cache scenarios as this might have unintended side-effects on your application while a Redis container is restarting and loading data from disk.
"},{"location":"docker-images/redis/#lagoon-adaptions","title":"Lagoon adaptions","text":"This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissionsso this image will work with a random user./etc/redis/* are templated using envplate via a container-entrypoint.redis.conf configuration file","text":"The image ships a default Redis configuration file, optimized to work on Lagoon.
"},{"location":"docker-images/redis/#environment-variables","title":"Environment Variables","text":"Some options are configurable via environment variables.
Environment Variable Default Description DATABASES -1 Default number of databases created at startup. LOGLEVEL notice Define the level of logs. MAXMEMORY 100mb Maximum amount of memory. MAXMEMORYPOLICY allkeys-lru The policy to use when evicting keys if Redis reaches its maximum memory usage. REDIS_PASSWORD disabled Enables authentication feature."},{"location":"docker-images/redis/#custom-configuration","title":"Custom configuration","text":"By building on the base image you can include custom configuration. See https://raw.githubusercontent.com/antirez/redis/4.0/redis.conf for full documentation of the Redis configuration file.
"},{"location":"docker-images/redis/#redis-persistent","title":"Redis-persistent","text":"Based on the Lagoon redis image, the Lagoon redis-persistent Docker image is intended for use when the Redis service must be utilized in persistent mode (ie. with a persistent volume where keys will be saved to disk).
It differs from redis only with the FLAVOR environment variable, which will use the respective Redis configuration according to the version of redis in use.
The Lagoon Redis images all come pre-loaded with the redis-cli command, which allows for querying the Redis service for information and setting config values dynamically. To use this utility, you can simply SSH into your Redis pod by using the instructions [here] (../using-lagoon-advanced/ssh.md) with redis as the pod value then run it from the terminal once you've connected.
By default, the Lagoon redis images are set to use the allkeys-lru policy. This policy will alow ANY keys stored in Redis to be evicted if/when the Redis service hits its maxmemory limit according to when the key was least recently used.
For typical installations, this is the ideal configuration, as Drupal may not set a TTL value for each key cached in Redis. If the maxmemory-policy is set to something like volatile-lru and Drupal doesn't provide these TTL tags, this would result in the Redis container filling up, being totally unable to evict ANY keys, and ceasing to accept new cache keys at all.
More information on Redis' maxmemory policies can be found in Redis' official documentation.
Proceed with Caution
Changing this setting can lead to Redis becoming completely full and cause outages as a result.
"},{"location":"docker-images/redis/#tuning-redis-maxmemory-value","title":"Tuning Redis'maxmemory value","text":"Finding the optimal amount of memory to give Redis can be quite the difficult task. Before attempting to tune your Redis cache's memory size, it is prudent to let it run normally for as long as practical, with at least a day of typical usage being the ideal minimum timeframe.
There are a few high level things you can look at when tuning these memory values:
50%, you might consider lowering the maxmemory value by 25%.50% and 75%, things are running just fine.75%, then it's worth looking at other variables to see if maxmemory needs to be increased.95% is a fairly good indicator that your redis needs a higher maxmemory setting.The following commands can be used to view information about the Redis service:
redis-cli inforedis-cli info memoryredis-cli info keyspaceredis-cli info statsIt is also possible to set values for the Redis service dynamically without a restart of the Redis service. It is important to note that these dynamically set values will not persist if the pod is restarted (which can happen as a result of a deployment, maintenance, or even just being shuffled from one node to another).
maxmemory config value dynamically to 500mb: config set maxmemory 500mbmaxmemory-policy config value dynamically to volatile-lru: config set maxmemory-policy volatile-lruThe Lagoon ruby Docker image. Based on the official Python Alpine images.
uselagoon/ruby-3.0uselagoon/ruby-3.1uselagoon/ruby-3.2Tip
We stop updating and publishing EOL Ruby images usually with the Lagoon release that comes after the officially communicated EOL date: https://www.ruby-lang.org/en/downloads/releases/. Previous versions will remain available.
"},{"location":"docker-images/ruby/#lagoon-adaptions","title":"Lagoon adaptions","text":"The default exposed port of ruby containers is port 3000.
Lagoon has no \"pre-defined\" type for Ruby services, they should be configured with the lagoon.type: generic and a port set with lagoon.port: 3000
docker-compose.yml snippet","text":"docker-compose.ymlruby:\nbuild:\n# this configures a build from a Dockerfile in the root folder\ncontext: .\ndockerfile: Dockerfile\nlabels:\n# tells Lagoon this is a generic service, configured to expose port 3000\nlagoon.type: generic\nlagoon.port: 3000\nports:\n# local development only\n# this exposes the port 3000 with a random local port\n# find it with `docker-compose port ruby 3000`\n- \"3000\"\nThe Lagoon Solr image Dockerfile. Based on the official solr:<version>-alpine images.
This Dockerfile is intended to be used to set up a standalone Solr server with an initial core mycore.
uselagoon/solr-7uselagoon/solr-8This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user.10-solr-port.sh script to fix and check Solr port.20-solr-datadir.sh script to check if Solr config is compliant for Lagoon. This sets directory paths, and configures the correct lock type.Some options are configurable via environment variables.
Environment Variable Default Description SOLR_JAVA_MEM 512M Default Java HEAP size (ie.SOLR_JAVA_MEM=\"-Xms10g -Xmx10g\"). SOLR_DATA_DIR /var/solr Path of the solr data dir. Be careful, changing this can cause data loss! SOLR_COPY_DATA_DIR_SOURCE (not set) Path which the entrypoint script of solr will use to copy into the defined SOLR_DATA_DIR, this can be used for prepopulating the Solr with a core. The scripts expects actual Solr data files! Plus it only copies data if the destination does not already have a solr core in it."},{"location":"docker-images/varnish/","title":"Varnish","text":"The Lagoon Varnish Docker images. Based on the official Varnish package
uselagoon/varnish-5uselagoon/varnish-6uselagoon/varnish-7vbox-dynamic - Dynamic backends from DNS lookups and service discovery from SRV records.vbox-bodyaccess - Varnish vmod that lets you access the request body.This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user.default.vcl configuration file","text":"The image ships a default vcl configuration file, optimized to work on Lagoon. Some options are configurable via environments variables (see Environment Variables).
Some options are configurable via environment variables.
Environment Variable Default Description VARNISH_BACKEND_HOST NGINX Default backend host. VARNISH_BACKEND_PORT 8080 Default listening Varnish port. VARNISH_SECRET lagoon_default_secret Varnish secret used to connect to management. LIBVMOD_DYNAMIC_VERSION 5.2 Default version ofvmod-dynamic module. LIBVMOD_BODYACCESS_VERSION 5.0 Default version of vmod-bodyaccess module. HTTP_RESP_HDR_LEN 8k Maximum length of any HTTP backend response header. HTTP_RESP_SIZE 32k Maximum number of bytes of HTTP backend response we will deal with. NUKE_LIMIT 150 Maximum number of objects we attempt to nuke in order to make space for an object body. CACHE_TYPE malloc Type of varnish cache. CACHE_SIZE 100M Cache size. LISTEN 8080 Default backend server port. MANAGEMENT_LISTEN 6082 Default management listening port."},{"location":"drupal/","title":"Drupal on Lagoon","text":"Lagoon was built to host Drupal sites (no, seriously, it was - at least initially!)
In this section you'll find more information on the various services that have been customised for use with Drupal.
"},{"location":"drupal/#drupal_integrations-drupal-scaffolding-package","title":"drupal_integrations Drupal scaffolding package","text":"The drupal_integrations package, available on pacakagist extends Drupal's core-composer-scaffold for use on Lagoon. It also provides additional Drush command drush la to retreive the Drush aliases for your Lagoon project.
lagoon-logs Drupal module","text":"The lagoon_logs module, availalble on drupal.org provides zero-configuration logging for Drupal on Lagoon.
Unfortunately, Drush 9 does not provide the ability to inject dynamic site aliases like Drush 8 did. We are working with the Drush team to implement this again. In the meantime, we have a workaround that allows you to use Drush 9 with Lagoon.
"},{"location":"drupal/drush-9/#basic-idea","title":"Basic Idea","text":"Drush 9 provides a new command, drush site:alias-convert , which can convert Drush 8-style site aliases over to the Drush 9 YAML site alias style. This will create a on- time export of the site aliases currently existing in Lagoon, and save them in /app/drush/sites . These are then used when running a command like drush sa.
In order to be able to use drush site:alias-convert , you need to do the following:
aliases.drushrc.php inside the drush folder to lagoon.aliases.drushrc.php.You can now convert your Drush aliases by running the following command in your project using the cli container:
docker-compose exec cli drush site:alias-convert /app/drush/sites --yes\nIt's good practice to commit the resulting YAML files into your Git repository, so that they are in place for your fellow developers.
"},{"location":"drupal/drush-9/#use-site-aliases","title":"Use Site Aliases","text":"In Drush 9, all site aliases are prefixed with a group. In our case, this is lagoon. You can show all site aliases with their prefix via:
drush sa --format=list\nand to use them:
Using Drush site aliasdrush @lagoon.main ssh\nIf a new environment in Lagoon has been created, you can run drush site:alias-convert to update the site aliases file. If running this command does not update lagoon.site.yml, try deleting lagoon.site.yml first, and then re-run drush site:alias-convert.
rsync from local to remote environments","text":"If you would like to sync files from a local environment to a remote environment, you need to pass additional parameters:
Drush rsyncdrush rsync @self:%files @lagoon.main:%files -- --omit-dir-times --no-perms --no-group --no-owner --chmod=ugo=rwX\nThis also applies to syncing one remote environment to another, if you're not using the Lagoon tasks UI to copy files between environments.
For example, if you wanted to sync the files from @lagoon.main to @lagoon.dev , and ran drush rsync @lagoon.main @lagoon.dev locally, without the extra parameters, you would probably run into a \"Cannot specify two remote aliases\" error.
To resolve this, you would first need to SSH into your destination environment drush @lagoon.dev ssh, and then execute the rsync command with parameters similar to the above:
drush rsync @lagoon.main:%files @self:%files -- --omit-dir-times --no-perms --no-group --no-owner --chmod=ugo=rwX\nThis is not necessary if you rsync from a remote to a local environment.
Also, we're working with the Drush maintainers to find a way to inject this automatically.
"},{"location":"drupal/first-deployment-of-drupal/","title":"First Deployment of Drupal","text":""},{"location":"drupal/first-deployment-of-drupal/#1-make-sure-you-are-all-set","title":"1. Make sure you are all set","text":"In order to make your first deployment a successful one, please make sure that your Drupal Project is Lagoonized and you have set up the project in Lagoon. If not, don't worry! Follow the Step-by-Step Guide which show you how this works.
"},{"location":"drupal/first-deployment-of-drupal/#2-push","title":"2. Push","text":"With Lagoon, you create a new deployment by pushing into a branch that is configured to be deployed.
If you don't have any new code to push, don't worry, you can run
Git pushgit commit --allow-empty -m \"go, go! Power Rangers!\"\ngit push\nThis will trigger a push, and the Git hosting will inform Lagoon about this push via the configured webhook.
If all is correct, you will see a notification in your configured chat system (this is configured by your friendly Lagoon administrator):
This tells you that Lagoon has just started to deploy your code. Depending on the size of the codebase and amount of containers, this will take a couple of seconds. Just relax. If you'd like to know what's happening now, check out the Build and Deploy Process of Lagoon.
You can also check your Lagoon UI to see the progress of any deployment (your Lagoon administrator has the info).
"},{"location":"drupal/first-deployment-of-drupal/#3-a-fail","title":"3. A fail","text":"Depending on the post-rollout tasks defined in .lagoon.yml , you might have run some tasks like drush updb or drush cr. These Drush tasks depend on a database existing within the environment, which obviously does not exist yet. Let's fix that! Keep reading.
With full Drush site alias support in Lagoon, you can synchronize a local database with the remote Lagoon environment.
Warning
You may have to tell pygmy about your public keys before the next step.
If you get an error like Permission denied (publickey), check out the documentation here: pygmy - adding ssh keys
First let's make sure that you can see the Drush site aliases:
Get site aliasesdrush sa\nThis should return your just deployed environment (let's assume you just pushed into develop):
[drupal-example]cli-drupal:/app$ drush sa\n@develop\n@self\ndefault\nWith this we can now synchronize the local database (which is represented in Drush via the site alias @self) with the remote one (@develop):
drush sql-sync @self @develop\nYou should see something like:
Drush sql-sync results[drupal-example]cli-drupal:/app$ drush sql-sync @self @develop\nYou will destroy data in ssh.lagoon.amazeeio.cloud/drupal and replace with data from drupal.\nDo you really want to continue? (y/n): y\nStarting to dump database on Source. [ok]\nDatabase dump saved to /home/drush-backups/drupal/20180227075813/drupal_20180227_075815.sql.gz [success]\nStarting to discover temporary files directory on Destination. [ok]\nYou will delete files in drupal-example-develop@ssh.lagoon.amazeeio.cloud:/tmp/drupal_20180227_075815.sql.gz and replace with data from /home/drush-backups/drupal/20180227075813/drupal_20180227_075815.sql.gz\nDo you really want to continue? (y/n): y\nCopying dump file from Source to Destination. [ok]\nStarting to import dump file onto Destination database.\nNow let's try another deployment, again an empty push:
Git pushgit commit --allow-empty -m \"go, go! Power Rangers!\"\ngit push\nThis time all should be green:
Click on the links in the notification, and you should see your Drupal site loaded in all its beauty! It will probably not have images yet, which we will handle in Step 6.
If it is still failing, check the logs link for more information.
"},{"location":"drupal/first-deployment-of-drupal/#5-synchronize-local-files-to-the-remote-lagoon-environment","title":"5. Synchronize local files to the remote Lagoon environment","text":"You probably guessed it: we can do it with Drush:
Drush rsyncdrush rsync @self:%files @develop:%files\nIt should show you something like:
Drush rsync results[drupal-example]cli-drupal:/app$ drush rsync @self:%files @develop:%files\nYou will delete files in drupal-example-develop@ssh.lagoon.amazeeio.cloud:/app/web/sites/default/files and replace with data from /app/web/sites/default/files/\nDo you really want to continue? (y/n): y\nIn some cases, though, it might not look correct, like here:
Drush rsync results[drupal-example]cli-drupal:/app$ drush rsync @self:%files @develop:%files\nYou will delete files in drupal-example-develop@ssh.lagoon.amazeeio.cloud:'/app/web/%files' and replace with data from '/app/web/%files'/\nDo you really want to continue? (y/n):\nThe reason for that is that the Drupal cannot resolve the path of the files directory. This most probably has to do that the Drupal is not fully configured or has a missing database. For a workaround you can use drush rsync @self:sites/default/files @develop:sites/default/files, but we suggest that you actually check your local and remote Drupal (you can test with drush status to see if the files directory is correctly configured).
As soon as Lagoon is done building and deploying it will send a second notification to the chat system, like so:
This tells you:
That's it! We hope that wasn't too hard - making devOps accessible is what we are striving for.
"},{"location":"drupal/first-deployment-of-drupal/#but-wait-how-about-other-branches-or-the-production-environment","title":"But wait, how about other branches or the production environment?","text":"That's the beauty of Lagoon: it's exactly the same: Push the branch name you defined to be your production branch and that one will be deployed.
"},{"location":"drupal/first-deployment-of-drupal/#failure-dont-worry","title":"Failure? Don't worry.","text":"Did the deployment fail? Oh no! But we're here to help:
logs link in the error notification. It will tell you where in the deployment process the failure happened.7, 8 or 9 siteUse Composer to get the latest version of the module:
Install Fastlycomposer require drupal/fastly drupal/http_cache_control drupal/purge\nYou will need to enable the following modules:
fastlyfastlypurgerhttp_cache_control (2.x)purgepurge_ui (technically optional, but this is really handy to have enabled on production)purge_processor_lateruntimepurge_processor_cronpurge_queuer_coretagspurge_drush (useful for purge via Drush, here is a list of commands)Configure the Fastly service ID and API token. You can use runtime environment variables, or you can edit the settings form found at /admin/config/services/fastly:
FASTLY_API_TOKENFASTLY_API_SERVICEA site ID is required, the module will generate one for you when you first install it. The idea behind the site ID is that it is a unique string which is appended as a cache tag on all requests. Thus, you are able to purge a single site from Fastly, even though multiple sites may flow through the same service in Fastly.
A 4 character cache tag is plenty for most sites, a 5 character cache tag is likely better for sites with millions of entities (to reduce cache tag collisions).
Set the options to what makes sense for your site. Minimum 1 hour (3600), maximum 1 week 604800). Generally something like the following will be fine:
Optionally configure the webhooks (so you can ping Slack for instance when a cache purge is sent).
"},{"location":"drupal/integrate-drupal-and-fastly/#configure-the-purge-module","title":"Configure the Purge module","text":"Visit the purge page /admin/config/development/performance/purge
Set up the following options:
"},{"location":"drupal/integrate-drupal-and-fastly/#cache-invalidation","title":"Cache Invalidation","text":"What this means is that we will be using Drupal's built in core tag queuer (add tags to the queue), the queue will be stored in the database (default), and the queue will be processed by
In order for the cron processor to run, you need to ensure that cron is running on your site. Ideally every minute. You can manually run it in your cli pod, to ensure that purge_processor_cron_cron() is being executed without errors.
[drupal8]production@cli-drupal:/app$ drush cron -v\n ...\n [notice] Starting execution of purge_processor_cron_cron(), execution of node_cron() took 21.16ms.\nThe Late runtime processor will run in hook_exit() for every page load, this can be useful to process the purges nearly as quickly as they come into the queue.
By having both, you guarantee that purges happen as soon as possible.
"},{"location":"drupal/integrate-drupal-and-fastly/#optimal-cache-header-setup","title":"Optimal Cache Header Setup","text":"Out of the box, Drupal does not have the power to set different cache lifetimes in the browser vs in Fastly. So if you do set long cache lifetimes in Drupal, often end users will not see them if their browser has cached the page. If you install the 2.x version of the HTTP Cache Control module, this will give you a lot more flexibility on what caches and for how long.
For most sites, a sensible default could be
Note
This relies on your site having accurate cache tags represented for all the content that exists on the page.
"},{"location":"drupal/integrate-drupal-and-fastly/#viewing-caching-headers-using-curl","title":"Viewing caching headers using cURL","text":"Use this function: (works in Linux and Mac OSX)
cURL functionfunction curlf() { curl -sLIXGET -H 'Fastly-Debug:1' \"$@\" | grep -iE 'X-Cache|Cache-Control|Set-Cookie|X-Varnish|X-Hits|Vary|Fastly-Debug|X-Served|surrogate-control|surrogate-key' }\n$ curlf https://www.example-site-fastly.com\ncache-control: max-age=601, public, s-maxage=2764800\nsurrogate-control: max-age=2764800, public, stale-while-revalidate=3600, stale-if-error=3600\nfastly-debug-path: (D cache-wlg10427-WLG 1612906144) (F cache-wlg10426-WLG 1612906141) (D cache-fra19179-FRA 1612906141) (F cache-fra19122-FRA 1612906141)\nfastly-debug-ttl: (H cache-wlg10427-WLG - - 3) (M cache-fra19179-FRA - - 0)\nfastly-debug-digest: 1118d9fefc8a514ca49d49cb6ece04649e1acf1663398212650bb462ba84c381\nx-served-by: cache-fra19179-FRA, cache-wlg10427-WLG\nx-cache: MISS, HIT\nx-cache-hits: 0, 1\nvary: Cookie, Accept-Encoding\nFrom the above headers we can see that:
2764800 seconds)If you ever want to remove a specific page from cache manually, there are ways to do this.
For a single page, you do not need any authentication:
Single page cURLcurl -Ssi -XPURGE -H 'Fastly-Soft-Purge:1' https://www.example.com/subpage\nFor cache tags, you need to supply your API token for authentication:
Cache tagscurl -XPOST -H \"Fastly-Key:<Fastly API Key>\" https://api.fastly.com/service/<serviceID>/purge/<surrogatekey>\nYou can always find what your site ID cache tag is by using PHP
Find site ID cache tagphp > var_dump(substr(base64_encode(md5('bananasite', true)), 0, 4));\nstring(4) \"DTRk\"\nSo you can purge your entire site from Fastly fairly easily.
"},{"location":"drupal/integrate-drupal-and-fastly/#true-client-ips","title":"True client IPs","text":"We configure Fastly to send the actual client IP back on the HTTP header True-Client-IP, you can make Drupal respect this header with the following changes in settings.php:
$settings['reverse_proxy'] = TRUE;\n$settings['reverse_proxy_header'] = 'HTTP_TRUE_CLIENT_IP';\n fastly:\n fastly:purge:all (fpall) Purge whole service.\n fastly:purge:key (fpkey) Purge cache by key.\n fastly:purge:url (fpurl) Purge cache by Url.\nsettings.php:$conf['reverse_proxy_header'] = 'HTTP_TRUE_CLIENT_IP';\nNote
This document assumes the following:
- You are using Docker.
- You are using a standard Amazee/Lagoon project with a docker-compose.yml file.
- You are on a Mac - it should work for other operating systems but folder structure and some configuration settings may be different.
"},{"location":"drupal/phpunit-and-phpstorm/#configuring-the-project","title":"Configuring the project","text":"/core/phpunit.xml.dist file to /core/phpunit.xmlEdit* /core/phpunit.xml and fill in the following variables:
mysql://drupal:drupal@mariadb:3306/drupal#db<PROJECT_URL>+Docker for MacAdd a new CLI interpreter:
... and then +<DOCKER>./docker-compose.ymlcliConnect to existing container ('docker-compose exec')<ROOT_PATH>/appAdd Remote Interpreter:
+ and select PHPUnit by Remote Interpreter<CLI_INTERPRETER><PROJECT_ROOT> -> /appUse Composer autoloader/app/vendor/autoload.php/app/web/core/phpunit.xml1. Test scope: Defined in the configuration file
2. Interpreter: <CLI_INTERPRETER>
Note
If you are not on a Mac, this may vary.
"},{"location":"drupal/phpunit-and-phpstorm/#final-checks","title":"Final checks","text":""},{"location":"drupal/phpunit-and-phpstorm/#some-final-checks-to-run-before-you-run-a-test","title":"Some final checks to run before you run a test!","text":"$ docker-compose up -dNow you have the above configuration set up it should be as straightforward as going to the test you want to run and pressing the green arrow!
Once you press this PhpStorm will use Docker to enter the CLI container, then start running PHPUnit based upon the config.
"},{"location":"drupal/step-by-step-getting-drupal-ready-to-run-on-lagoon/","title":"Step by Step: Getting Drupal ready to run on Lagoon","text":""},{"location":"drupal/step-by-step-getting-drupal-ready-to-run-on-lagoon/#1-lagoon-drupal-setting-files","title":"1. Lagoon Drupal Setting Files","text":"In order for Drupal to work with Lagoon, we need to teach Drupal about Lagoon and Lagoon about Drupal. This happens by copying specific YAML and PHP files into your Git repository.
If you're working on a Drupal project, you can check out one of the various Drupal example projects in our examples repository. We have Drupal 8 and 9 and some variants of each depending on your needs, such as database types. Clone the repository that best suits your needs to get started!
Here is a summary of the Lagoon- and Drupal-specific files you will find:
.lagoon.yml - The main file that will be used by Lagoon to understand what should be deployed and many more things. This file has some sensible Drupal defaults. If you would like to edit or modify, please check the documentation for .lagoon.yml.docker-compose.yml, .dockerignore, and *.dockerfile (or Dockerfile) - These files are used to run your local Drupal development environment, they tell Docker which services to start and how to build them. They contain sensible defaults and many commented lines. We hope that it's well-commented enough to be self-describing. If you would like to find out more, see documentation for docker-compose.yml.sites/default/* - These .php and .yml files tell Drupal how to communicate with Lagoon containers both locally and in production. They also provide a straightforward system for specific overrides in development and production environments. Unlike other Drupal hosting systems, Lagoon never ever injects Drupal settings files into your Drupal. Therefore, you can edit them however you like. Like all other files, they contain sensible defaults and some commented parts.drush/aliases.drushrc.php - These files are specific to Drush and tell Drush how to talk to the Lagoon GraphQL API in order to learn about all site aliases there are.drush/drushrc.php - Some sensible defaults for Drush commands..gitignore Settings","text":"Don't forget to make sure your .gitignore will allow you to commit the settings files.
Drupal is shipped with sites/*/settings*.php and sites/*/services*.yml in .gitignore. Remove that, as with Lagoon we don't ever have sensitive information in the Git repository.
WEBROOT in Drupal 8","text":"Unfortunately the Drupal community has not decided on a standardized WEBROOT folder name. Some projects put Drupal within web, and others within docroot or somewhere else. The Lagoon Drupal settings files assume that your Drupal is within web, but if this is different for your Drupal, please adapt the files accordingly.
If you installed Drupal via composer, please check your composer.json and make sure that the name is NOT drupal/drupal, as this could confuse Drush and other tools of the Drupal universe, just rename it to something like myproject/drupal
docker-compose.yml","text":"Don't forget to customize the values in lagoon-project & LAGOON_ROUTE with your site-specific name & the URL you'd like to access the site with. Here's an example:
x-environment:\n&default-environment\nLAGOON_PROJECT: *lagoon-project\n# Route that should be used locally. If you are using pygmy, this route *must* end with .docker.amazee.io.\nLAGOON_ROUTE: http://drupal-example.docker.amazee.io\nFirst, we need to build the defined images:
Build imagesdocker-compose build\nThis will tell docker-compose to build the Docker images for all containers that have a build: definition in the docker-compose.yml. Usually for Drupal this is the case for the cli, nginx and php images. We do this because we want to run specific build commands (like composer install) or inject specific environment variables (like WEBROOT) into the images.
Usually, building is not necessary every time you edit your Drupal code (as the code is mounted into the containers from your host), but rebuilding does not hurt. Plus, Lagoon will build the exact same Docker images during a deploy, so you can check that your build will also work during a deployment by just running docker-compose build again.
Now that the images are built, we can start the containers:
Start containersdocker-compose up -d\nThis will bring up all containers. After the command is done, you can check with docker-compose ps to ensure that they are all fully up and have not crashed. If there is a problem, check the logs with docker-compose logs -f [servicename].
composer install (for Composer projects only)","text":"In a local development environment, you probably want all dependencies downloaded and installed, so connect to the cli container and run composer install:
docker-compose exec cli bash\ncomposer install\nThis might sound weird, as there was already a composer install executed during the build step, so let us explain:
docker-composer.yml mounts the whole folder into the the containers (this happens with .:/app:delegated in the volumes section). This also means that all dependencies installed during the Docker build are overwritten with the files on the host.require-dev in composer.json to exist as well, while on a production deployment they would just use unnecessary space. So we run composer install --no-dev in the Dockerfile and composer install manually.If everything went well, open the LAGOON_ROUTE defined in docker-compose.yml (for example http://drupal.docker.amazee.io) and you should be greeted by a nice Drupal error. Don't worry - that's ok right now, most important is that it tries to load a Drupal site.
If you get a 500 or similar error, make sure everything loaded properly with Composer.
"},{"location":"drupal/step-by-step-getting-drupal-ready-to-run-on-lagoon/#6-check-status-and-install-drupal","title":"6. Check Status and Install Drupal","text":"Finally it's time to install Drupal, but just before that we want to make sure everything works. We suggest using Drush for that:
Drush statusdocker-compose exec cli bash\ndrush status\nThis should return something like:
Drush status result[drupal-example]cli-drupal:/app$ drush status\n[notice] Missing database table: key_value\nDrupal version : 8.6.1\nSite URI : http://drupal.docker.amazee.io\nDatabase driver : mysql\nDatabase hostname : mariadb\nDatabase port : 3306\nDatabase username : drupal\nDatabase name : drupal\nPHP binary : /usr/local/bin/php\nPHP config : /usr/local/etc/php/php.ini\nPHP OS : Linux\nDrush script : /app/vendor/drush/drush/drush\nDrush version : 9.4.0\nDrush temp : /tmp\nDrush configs : /home/.drush/drush.yml\n /app/vendor/drush/drush/drush.yml\nDrupal root : /app/web\nSite path : sites/default\nWarning
You may have to tell pygmy about your public key before the next step.
If you get an error like Permission denied (publickey), check out the documentation here: pygmy - adding ssh keys
Now it is time to install Drupal (if instead you would like to import an existing SQL file, please skip to step 7, but we suggest you start with a clean Drupal installation in the beginning to be sure everything works).
Install Drupaldrush site-install\nThis should output something like:
drush site-install[drupal-example]cli-drupal:/app$ drush site-install\nYou are about to DROP all tables in your 'drupal' database. Do you want to continue? (y/n): y\nStarting Drupal installation. This takes a while. Consider using the --notify global option.\nInstallation complete. User name: admin User password: a7kZJekcqh\nCongratulations, you installed Drupal!\nNow you can visit the URL defined in LAGOON_ROUTE and you should see a fresh and clean installed Drupal site - Congrats!
If you already have an existing Drupal site, you probably want to import its database over to your local site.
There are many different ways to create a database dump. If your current hosting provider has Drush installed, you can use the following:
Drush sql-dumpdrush sql-dump --result-file=dump.sql\n\nDatabase dump saved to dump.sql\nNow you have a dump.sql file that contains your whole database.
Copy this file into your Git repository and connect to the cli, and you should see the file in there:
[drupal-example]cli-drupal:/app$ ls -l dump.sql\n-rw-r--r-- 1 root root 5281 Dec 19 12:46 dump.sql\nNow you can drop the current database, and then import the dump.
Import dump.sqldrush sql-drop\n\ndrush sql-cli < dump.sql\nVerify that everything works with visiting the URL of your project. You should have a functional copy of your Drupal site!
"},{"location":"drupal/step-by-step-getting-drupal-ready-to-run-on-lagoon/#8-drupal-files-directory","title":"8. Drupal files directory","text":"A Drupal site also needs the files directory. As the whole folder is mounted into the Docker containers, add the files into the correct folder (probably web/sites/default/files, sites/default/files or something similar). Remember what you've set as your WEBROOT - it may not be the same for all projects.
You are done with your local setup. The Lagoon team wishes happy Drupaling!
"},{"location":"drupal/subfolders/","title":"Subfolders","text":"An example could be: www.example.com points to one Drupal site, while www.example.com/blog loads a blog built in another Drupal.
It would be possible to run both Drupals in a single Git repository and deploy it as a whole, but this workflow might not fit every team, and having separate Git repositories fits some situations better.
"},{"location":"drupal/subfolders/#modifications-of-root-application","title":"Modifications of root application","text":"The root application (in this example, the Drupal site for www.example.com), needs a couple of NGINX configs that will configure NGINX to be a reverse proxy to the subfolder applications:
location_prepend.conf","text":"Create a file called location_prepend.conf in the root of your Drupal installation:
resolver 8.8.8.8 valid=30s;\n\nlocation ~ ^/subfolder {\n # If $http_x_forwarded_proto is empty (If it is not set from an upstream reverseproxy).\n # Aet it to the current scheme.\n set_if_empty $http_x_forwarded_proto $scheme;\n\n proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n proxy_set_header X-Forwarded-Proto $scheme;\n proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;\n proxy_set_header X-Lagoon-Forwarded-Host $host;\n # Will be used by downstream to know the original host.\n proxy_set_header X-REVERSEPROXY $hostname;\n proxy_set_header FORWARDED \"\";\n # Unset FORWARDED because drupal8 gives errors if it is set.\n proxy_set_header Proxy \"\";\n # Unset Proxy because drupal8 gives errors if it is set.\n proxy_ssl_server_name on;\n\n # NGINX needs a variable set in order for the DNS resolution to work correctly.\n set $subfolder_drupal_host \"https://nginx-lagoonproject-${LAGOON_GIT_SAFE_BRANCH}.clustername.com:443\";\n # LAGOON_GIT_SAFE_BRANCH variable will be replaced during docker entrypoint.\n proxy_pass $subfolder_drupal_host;\n proxy_set_header Host $proxy_host;\n # $proxy_host will be automatically generated by NGINX based on proxy_pass (it needs to be without scheme and port).\n\n expires off; # make sure we honor cache headers from the proxy and not overwrite them\nReplace the following strings:
/subfolder with the name of the subfolder you want to use. For example, /blog.nginx with the service that you want to point too in the subfolder project.lagoonproject with the Lagoon projectname of the subfolder project.Add the following to your NGINX Dockerfile (nginx.dockerfile or Dockerfile.nginx):
COPY location_prepend.conf /etc/nginx/conf.d/drupal/location_prepend.conf\nRUN fix-permissions /etc/nginx/conf.d/drupal/*\nLike the root application, we also need to teach the subfolder application (in this example, the Drupal installation for www.example.com/blog), that it is running under a subfolder. To do this, we create two files:
location_drupal_append_subfolder.conf","text":"Create a file called location_drupal_append_subfolder.conf in the root of your subfolder Drupal installation:
# When injecting a script name that is prefixed with `subfolder`, Drupal will\n# render all URLs with `subfolder` prefixed\nfastcgi_param SCRIPT_NAME /subfolder/index.php;\n\n# If we are running via a reverse proxy, we inject the original HOST URL\n# into PHP. With this Drupal will render all URLs with the original HOST URL,\n# and not the current used HOST.\n\n# We first set the HOST to the regular host variable.\nfastcgi_param HTTP_HOST $http_host;\n# Then we overwrite it with `X-Lagoon-Forwarded-Host` if it exists.\nfastcgi_param HTTP_HOST $http_x_lagoon_forwarded_host if_not_empty;\nReplace /subfolder with the name of the subfolder you want to use. For example, /blog.
server_prepend_subfolder.conf","text":"Create a file called server_prepend_subfolder.conf in the root of your subfolder Drupal installation:
# Check for redirects before we do the internal NGINX rewrites.\n# This is done because the internal NGINX rewrites uses `last`,\n# which instructs NGINX to not check for rewrites anymore (and\n# `if` is part of the redirect module).\ninclude /etc/nginx/helpers/010_redirects.conf;\n\n# This is an internal NGINX rewrite, it removes `/subfolder/`\n# from the requests so that NGINX handles the request as it would\n# have been `/` from the beginning.\n# The `last` flag is also important. It will cause NGINX not to\n# execute any more rewrites, because it would redirect forever\n# with the rewrites below.\nrewrite ^/subfolder/(.*) /$1 last;\n\n# Make sure redirects are NOT absolute, to ensure NGINX does not\n# overwrite the host of the URL - which could be something other than\n# what NGINX currently thinks it is serving.\nabsolute_redirect off;\n\n# If a request just has `/subfolder` we 301 redirect to `/subfolder/`\n# (Drupal really likes a trailing slash)\nrewrite ^/subfolder /subfolder/ permanent;\n\n# Any other request we prefix 301 redirect with `/subfolder/`\nrewrite ^\\/(.*) /subfolder/$1 permanent;\nReplace /subfolder with the name of the subfolder you want to use. For example, /blog.
We also need to modify the NGINX Dockerfile.
Add the following to your NGINX Dockerfile (nginx.dockerfile or Dockerfile.nginx):
COPY location_drupal_append_subfolder.conf /etc/nginx/conf.d/drupal/location_drupal_append_subfolder.conf\nCOPY server_prepend_subfolder.conf /etc/nginx/conf.d/drupal/server_prepend_subfolder.conf\nRUN fix-permissions /etc/nginx/conf.d/drupal/*\nLearn about MariaDB with Drupal
Documentation on the MariaDB-Drupal image.
Documentation on the plain MariaDB image (the MariaDB-Drupal image is built on this).
"},{"location":"drupal/services/#redis-is-a-fast-open-source-in-memory-key-value-data-store-for-use-as-a-database-cache-message-broker-and-queue","title":"Redis is a fast, open-source, in-memory key-value data store for use as a database, cache, message broker, and queue","text":"Learn about Redis with Drupal.
Documentation on the Redis-persistent image.
"},{"location":"drupal/services/#solr-is-an-open-source-search-platform","title":"Solr is an open-source search platform","text":"Learn about Solr with Drupal.
Documentation on the Solr-Drupal image.
Documentation on the plain Solr image (the Solr-Drupal image is built on this).
"},{"location":"drupal/services/#varnish-is-a-powerful-open-source-http-engine-and-reverse-http-proxy-that-helps-to-speed-up-your-website","title":"Varnish is a powerful, open-source HTTP engine and reverse HTTP proxy that helps to speed up your website","text":"Learn about Varnish with Drupal
Documentation on the Varnish-Drupal image.
Documentation on the plain Varnish image (the Varnish-Drupal image is built on this).
"},{"location":"drupal/services/mariadb/","title":"MariaDB-Drupal","text":"The Lagoon mariadb-drupal Docker image Dockerfile is a customized mariadb image to use within Drupal projects in Lagoon. It differs from the mariadb image only for initial database setup, made by some environment variables:
MARIADB_DATABASE drupal Drupal database created at startup. MARIADB_USER drupal Default user created at startup. MARIADB_PASSWORD drupal Password of default user created at startup. If the LAGOON_ENVIRONMENT_TYPE variable is set to production, performances are set accordingly by using MARIADB_INNODB_BUFFER_POOL_SIZE=1024 and MARIADB_INNODB_LOG_FILE_SIZE=256.
During the course of development, it may be necessary to enable either query logging or slow query logging. To do so, set the environment variables MARIADB_LOG_SLOW or MARIADB_LOG_QUERIES. This can be done in docker-compose.yml.
If you would like to connect to your MySQL database inside the Docker container with an external tool like Sequel Pro, MySQL Workbench, HeidiSQL, DBeaver, plain old mysql-cli or anything else, here's how to get the IP and port info.
By default, Docker assigns a randomly published port for MySQL during each container start. This is done to prevent port collisions.
To get the published port via docker:
Run: docker port [container_name].
$ docker port drupal_example_mariadb_1\n3306/tcp -> 0.0.0.0:32797\nOr via docker-compose inside a Drupal repository:
Run: docker-compose port [service_name] [interal_port].
docker-compose port mariab 3306\n0.0.0.0:32797\nDuring development, if you are using an external database tool, it may become cumbersome to continually check and set the MySQL connection port.
To set a static port, edit your service definition in your docker-compose.yml.
mariadb:\n...\nports:\n- \"33772:3306\" # Exposes port 3306 with a 33772 on the host port. Note by doing this you are responsible for managing port collisions`.\nWarning
By setting a static port you become responsible for managing port collisions.
"},{"location":"drupal/services/mariadb/#connect-to-mysql","title":"Connect to MySQL","text":"Now you can use these details to connect to whatever database management tool you'd like.
Linux OS X IP/Host IP from containerdocker.amazee.io Port Published port from container Published port from container Username drupal drupal Password drupal drupal Database drupal drupal"},{"location":"drupal/services/nginx/","title":"NGINX-Drupal","text":"The Lagoon nginx-drupal Docker image. Optimized to work with Drupal. Based on Lagoon nginx image.
This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user.drupal.conf 's configuration file as clean and customizable as possible, we added include directives in the main sections of the file:server, location /, location @drupal and location @php.Drupal.conf customization.drupal.conf)","text":"The image includes a full NGINX working configuration for Drupal 7, 8 and 9. It includes some extra functionalities like:
humanstxt Drupal module.robotstxt Drupal module.vagrant directory for local development.Drupal.conf customization","text":"The drupal.conf file is a customized version of the nginx configuration file, optimized for Drupal. Customers have different ways of customizing it:
*.conf files.The drupal.conf file is divided into several sections. The sections we've included in our customizations are:
serverlocation /location @drupallocation @php.For each of this section, there are two includes:
*_prepend.conf*_append.confHere what the location @drupal section looks like:
location @drupal {\ninclude /etc/nginx/conf.d/drupal/location_drupal_prepend*.conf;\ninclude /etc/nginx/fastcgi.conf;\nfastcgi_param SCRIPT_NAME /index.php;\nfastcgi_param SCRIPT_FILENAME $realpath_root/index.php;\nfastcgi_pass ${NGINX_FASTCGI_PASS:-php}:9000;\ninclude /etc/nginx/conf.d/drupal/location_drupal_append*.conf;\n}\nThis configuration allows customers to create files called location_drupal_prepend.conf and location_drupal_append.conf, where they can put all the configuration they want to insert before and after the other statements.
Those files, once created, MUST exist in the nginx container, so add them to Dockerfile.nginx like so:
COPY location_drupal_prepend.conf /etc/nginx/conf.d/drupal/location_drupal_prepend.conf\nRUN fix-permissions /etc/nginx/conf.d/drupal/location_drupal_prepend.conf\nIf you're using the core Statistics module, you may run into an issue that needs a quick configuration change.
With the default NGINX configuration, the request to the tracking endpoint /core/modules/statistics/statistics.php is denied (404).
This is related to the default NGINX configuration:
drupal.conflocation ~* ^.+\\.php$ {\n try_files /dev/null @drupal;\n}\nTo fix the issue, we instead define a specific location rule and inject this as a location prepend configuration:
drupal.conf## Allow access to to the statistics endpoint.\nlocation ~* ^(/core/modules/statistics/statistics.php) {\n try_files /dev/null @php;\n}\nAnd copy this during the NGINX container build:
dockerfile.nginx# Add specific Drupal statistics module NGINX configuration.\nCOPY .lagoon/nginx/location_prepend_allow_statistics.conf /etc/nginx/conf.d/drupal/location_prepend_allow_statistics.conf\nThe Lagoon php-cli-drupal Docker image is optimized to work with Drupal. It is based on the Lagoon php-cli image, and has all the command line tools needed for the daily maintenance of a Drupal website:
drushdrupal consoledrush launcher (which will fallback to Drush 8 if there is no site installed Drush found)uselagoon/php-7.4-cli-drupaluselagoon/php-8.0-cli-drupaluselagoon/php-8.1-cli-drupalAll PHP versions use their own Dockerfiles.
"},{"location":"drupal/services/php-cli/#lagoon-adaptions","title":"Lagoon adaptions","text":"This image is prepared to be used on Lagoon. There are therefore some things already done:
fix-permissions, so this image will work with a random user.We recommend using Redis for internal caching. Add the Redis service to docker-compose.yaml.
redis:\nimage: uselagoon/redis-5\nlabels:\nlagoon.type: redis\n<< : *default-user # Uses the defined user from top.\nenvironment:\n<< : *default-environment\nAlso, to configure Redis, add the following to your settings.php.
if(getenv('LAGOON')){\n $conf['redis_client_interface'] = 'PhpRedis';\n $conf['redis_client_host'] = 'redis';\n $conf['lock_inc'] = 'sites/all/modules/contrib/redis/redis.lock.inc';\n $conf['path_inc'] = 'sites/all/modules/contrib/redis/redis.path.inc';\n $conf['cache_backends'][] = 'sites/all/modules/contrib/redis/redis.autoload.inc';\n $conf['cache_default_class'] = 'Redis_Cache';\n $conf['cache_class_cache_form'] = 'DrupalDatabaseCache';\n $conf['cache_class_cache_field'] = 'DrupalDatabaseCache';\n }\nDepending on file system structure, the module paths may need to be updated.
"},{"location":"drupal/services/redis/#drupal-8","title":"Drupal 8","text":"The Drupal 8 config is largely stock. Notably, Redis is disabled while Drupal is being installed.
settings.phpif (getenv('LAGOON')){\n $settings['redis.connection']['interface'] = 'PhpRedis';\n $settings['redis.connection']['host'] = getenv('REDIS_HOST') ?: 'redis';\n $settings['redis.connection']['port'] = getenv('REDIS_SERVICE_PORT') ?: '6379';\n $settings['cache_prefix']['default'] = getenv('LAGOON_PROJECT') . '_' . getenv('LAGOON_GIT_SAFE_BRANCH');\n // Do not set the cache during installations of Drupal.\n if (!drupal_installation_attempted() && extension_loaded('redis')) {\n $settings['cache']['default'] = 'cache.backend.redis';\n // And allows to use it without the Redis module being enabled.\n $class_loader->addPsr4('Drupal\\\\redis\\\\', 'modules/contrib/redis/src');\n $settings['bootstrap_container_definition'] = [\n 'parameters' => [],\n 'services' => [\n 'redis.factory' => [\n 'class' => 'Drupal\\redis\\ClientFactory',\n ],\n 'cache.backend.redis' => [\n 'class' => 'Drupal\\redis\\Cache\\CacheBackendFactory',\n 'arguments' => ['@redis.factory', '@cache_tags_provider.container', '@serialization.phpserialize'],\n ],\n 'cache.container' => [\n 'class' => '\\Drupal\\redis\\Cache\\PhpRedis',\n 'factory' => ['@cache.backend.redis', 'get'],\n 'arguments' => ['container'],\n ],\n 'cache_tags_provider.container' => [\n 'class' => 'Drupal\\redis\\Cache\\RedisCacheTagsChecksum',\n 'arguments' => ['@redis.factory'],\n ],\n 'serialization.phpserialize' => [\n 'class' => 'Drupal\\Component\\Serialization\\PhpSerialize',\n ],\n ],\n ];\n }\n}\nRedis can also be configured as a persistent backend.
docker-compose.ymlredis:\nimage: uselagoon/redis-5-persistent\nlabels:\nlagoon.type: redis-persistent\nenvironment:\n<< : *default-environment\nEnvironment variables are meant to store some common information about Redis.
Environment Variable Default DescriptionLOGLEVEL notice Redis loglevel DATABASES 1 Number of databases MAXMEMORY 100mb Maximum memory usage of Redis"},{"location":"drupal/services/redis/#redis-failover","title":"Redis Failover","text":"Here is a snippet to implement a Redis failover in case of the Redis container not being available (for example, during maintenance)
The following is inserted into Drupal's active settings.php file.
if (getenv('LAGOON')) {\n $contrib_path = is_dir('sites/all/modules/contrib') ? 'sites/all/modules/contrib' : 'sites/all/modules';\n $redis = DRUPAL_ROOT . '/sites/all/modules/contrib/redis';\n if (file_exists(\"$redis/redis.module\")) {\n require_once \"$redis/redis.module\";\n $conf['redis_client_host'] = getenv('REDIS_HOST') ?: 'redis';\n $conf['redis_client_port'] = getenv('REDIS_SERVICE_PORT') ?: 6379;\n $conf['cache_prefix'] = getenv('REDIS_CACHE_PREFIX') ?: getenv('LAGOON_PROJECT') . '_' . getenv('LAGOON_GIT_SAFE_BRANCH');\n try {\n // Ensure that there is a connection to redis.\n $client = Redis_Client::getClient();\n $response = $client->ping();\n if (!$response) {\n throw new \\Exception('Redis could be reached but is not responding correctly.');\n }\n $conf['redis_client_interface'] = 'PhpRedis';\n $conf['lock_inc'] = $contrib_path . '/redis/redis.lock.inc';\n $conf['path_inc'] = $contrib_path . '/redis/redis.path.inc';\n $conf['cache_backends'][] = $contrib_path . '/redis/redis.autoload.inc';\n $conf['cache_default_class'] = 'Redis_Cache';\n } catch (\\Exception $e) {\n // Redis is not available for this request we should not configure the\n // redis backend and ensure no cache is used. This will retry next\n // request.\n if (!class_exists('DrupalFakeCache')) {\n $conf['cache_backends'][] = 'includes/cache-install.inc';\n }\n $conf['cache_default_class'] = 'DrupalFakeCache';\n }\n }\n}\nFor Solr 5.5, 6.6 and 7.7, we ship the default schema files provided by the search_api_solr Drupal module. Add the Solr version you would like to use in your docker-compose.yml file, following our example.
To implement schema customizations for Solr in your project, look to how Lagoon creates our standard images.
solr section of your docker-compose.yml file, replace image: amazeeio/solr:7.7 with: build:\ncontext: .\ndockerfile: solr.dockerfile\n.lagoon/solr.solr.dockerfile.FROM amazeeio/solr:7.7\n\nCOPY .lagoon/solr /solr-conf/conf\n\nRUN precreate-core drupal /solr-conf\n\nCMD [\"solr-foreground\"]\nThe goal is to have your Solr configuration files exist at /solr-conf/conf in the image you are building.
To implement multiple cores, you will also need to ship your own Solr schema as above. The only change needed is to the CMD of the Dockerfile - repeat the pattern of precreate-core corename /solr-conf/ ; for each core you require.
FROM amazeeio/solr:7.7-drupal\n\nRUN precreate-core drupal-index1 /solr-conf && \\\nprecreate-core drupal-index2 /solr-conf && \\\nprecreate-core drupal-index3 /solr-conf\n\nCMD [\"solr-foreground\"]\nWe suggest using Drupal with a Varnish reverse proxy. Lagoon provides a varnish-drupal Docker image that has Varnish already configured with a Drupal Varnish config.
This Varnish config does the following:
BAN and URIBAN which is used by the Drupal 8 purge module.utm_ and gclid from the URL parameter to prevent Google Analytics links from creating multiple cache objects.TL;DR: Check out the drupal8-advanced example in our examples repo, it ships with the needed modules and needed Drupal configuration.
Note: many of these examples are on the same drupal-example-simple repo, but different branches/hashes. Be sure to get the exact branch from the examples list!
In order to fully use Varnish with Drupal 8 cache tags, you need to install the Purge and Varnish Purge modules. They ship with many submodules. We suggest installing at least the following:
purgepurge_drushpurge_tokenspurge_uipurge_processor_cronpurge_processor_lateruntimepurge_queuer_coretagsvarnish_purgervarnish_purge_tagsGrab them all at once:
Install Purge and Varnish Purgecomposer require drupal/purge drupal/varnish_purge\n\ndrush en purge purge_drush purge_tokens purge_ui purge_processor_cron purge_processor_lateruntime purge_queuer_coretags varnish_purger varnish_purge_tags\nConfiguration > Development > Performance > Purge.Add purger.Varnish Bundled Purger (not the Varnish Purger, see the #Behind the Scenes section, for more information.).Configure.Lagoon Varnish sounds good.Configure it with:
Configure Varnish Purge TYPE: Tag\n\n REQUEST:\n Hostname: varnish\n (or whatever your Varnish is called in docker-compose.yml)\n Port: 8080\n Path: /\n Request Method: BAN\n Scheme: http\n\n HEADERS:\n Header: Cache-Tags\n Value: [invalidations:separated_pipe]\nSave configuration.
That's it! If you'd like to test this locally, make sure you read the next section.
"},{"location":"drupal/services/varnish/#configure-drupal-for-varnish","title":"Configure Drupal for Varnish","text":"There are a few other configurations that can be done:
Internal Page Cache Drupal module with drush pmu page_cache. It can cause some weird double caching situations where only the Varnish cache is cleared, but not the internal cache, and changes appear very slowly to the users. Also, it uses a lot of cache storage on big sites.$config['system.performance']['cache']['page']['max_age'] in production.settings.php to 2628000. This tells Varnish to cache sites for up 1 month, which sounds like a lot, but the Drupal 8 cache tag system is so awesome that it will basically make sure that the Varnish cache is purged whenever something changes.Drupal setups on Lagoon locally have Varnish and the Drupal caches disabled as it can be rather hard to develop with all them set. This is done via the following:
VARNISH_BYPASS=true environment variable in docker-compose.yml which tells Varnish to basically disable itself.$config['system.performance']['cache']['page']['max_age'] = 0 in development.settings.php).To test Varnish locally, change the following in docker-compose.yml:
VARNISH_BYPASS to false in the Varnish service section.LAGOON_ENVIRONMENT_TYPE to production in the x-environment section.docker-compose up -d , which restarts all services with the new environment variables.Now you should be able to test Varnish!
Here is a short example assuming there is a node with the ID 1 and has the URL drupal-example.docker.amazee.io/node/1
curl -I drupal-example.docker.amazee.io/node/1 and look for these headers:X-LAGOON should include varnish which tells you that the request actually went through Varnish.Age: will be still 0 as Varnish has probably never seen this site before, and the first request will warm the varnish cache.X-Varnish-Cache will be MISS , also telling you that Varnish didn't find a previously cached version of this request.curl -I drupal-example.docker.amazee.io/node/1 again, and the headers should be:Age: will show you how many seconds ago the request has been cached. In our example it will probably something between 1-30, depending on how fast you are executing the command.X-Varnish-Cache will be HIT, telling you that Varnish successfully found a cached version of the request and returned that one to you.node/1 in Drupal.curl -I drupal-example.docker.amazee.io/node/1 , and the headers should the same as very first request:Age:0X-Varnish-Cache: MISSIf you come from other Drupal hosts or have done a Drupal 8 & Varnish tutorial before, you might have realized that there are a couple of changes in the Lagoon Drupal Varnish tutorial. Let's address them:
"},{"location":"drupal/services/varnish/#usage-of-varnish-bundled-purger-instead-of-varnish-purger","title":"Usage ofVarnish Bundled Purger instead of Varnish Purger","text":"The Varnish Purger purger sends a BAN request for each cache-tag that should be invalidated. Drupal has a lot of cache-tags, and this could lead to quite a large amount of requests sent to Varnish. Varnish Bundled Purger instead sends just one BAN request for multiple invalidations, separated nicely by pipe (|), which fits perfectly with the Varnish regular expression system of bans. This causes less requests and a smaller ban list table inside Varnish.
Purge Late runtime processor","text":"Contradictory to the Varnish module in Drupal 7, the Drupal 8 Purge module has a slightly different approach to purging caches: It adds them to a queue which is then processed by different processors. Purge suggests using the Cron processor , which means that the Varnish cache is only purged during a cron run. This can lead to old data being cached by Varnish, as your cron is probably not configured to run every minute or so, and can result in confused editors and clients.
Instead, we suggest using the Purge Late runtime processor, which processes the queue at the end of each Drupal request. This has the advantage that if a cache-tag is added to the purge queue (because an editor edited a Drupal node, for example) the cache-tags for this node are directly purged. Together with the Varnish Bundled Purger, this means just a single additional request to Varnish at the very end of a Drupal request, which causes no noticeable processing time on the request.
Our Varnish configurations have full support for Ban Lurker. Ban Lurker helps you to maintain a clean cache and keep Varnish running smoothly. It is basically a small tool that runs through the Varnish ban list and compares them to the cached requests in the Varnish cache. Varnish bans are used to mark an object in the cache for purging. If Ban Lurker finds an item that should be \"banned,\" it removes them from the cache and also removes the ban itself. Now any seldom-accessed objects with very long TTLs which would normally never be banned and just keep taking up cache space are removed and can be refreshed. This keeps the list of bans small and with that, less processing time for Varnish on each request. Check out the official Varnish post on Ban Lurker and some other helpful reading for more information.
Varnish doesn't cache? Or something else not working? Here a couple of ways to debug:
drush p-debug-en to enable debug logging of the purge module. This should show you debugging in the Drupal log under admin/reports/dblog.Cache-Control: max-age=900, public header, where the 900 is what you configured in $config['system.performance']['cache']['page']['max_age'].VARNISH_BYPASS is not set to true (see docker-compose.yml and run docker-compose up -d varnish to make sure the environment variable is configured correctly). lagoon add group -N groupname\nRun this command:
Add projectlagoon add project \\\n--gitUrl <YOUR-GITHUB-REPO-URL> \\\n--openshift 1 \\\n--productionEnvironment <YOUR-PROD-ENV> \\\n--branches <THE-BRANCHES-YOU-WANT-TO-DEPLOY> \\\n--project <YOUR-PROJECT-NAME>\n--openshift is the ID of your Kubernetes cluster.\u201c^(main|develop)$\u201dGo to the Lagoon UI, and you should see your project listed!
Lagoon creates a deploy key for each project. You now need to add it as a deploy key in your Git repository to allow Lagoon to download the code.
Run the following command to get the deploy key:
Get project-keylagoon get project-key --project <YOUR-PROJECT-NAME>\nCopy the key and save it as a deploy key in your Git repository.
GitHub GitLab Bitbucket
"},{"location":"installing-lagoon/add-project/#add-the-webhooks-endpoint-to-your-git-repository","title":"Add the webhooks endpoint to your Git repository","text":"In order for Lagoon to be able to deploy on code updates, it needs to be connected to your Git repository
Add your Lagoon cluster's webhook endpoint to your Git repository
<LAGOON-WEBHOOK-INGRESS>GitHub GitLab Bitbucket
"},{"location":"installing-lagoon/create-user/","title":"Create Lagoon user","text":"Add user via Lagoon CLI:
Add userlagoon add user --email user@example.com --firstName MyFirstName --lastName MyLastName\nGo to your email and click the password reset link in the email.
Run the following command to deploy your project:
Deploylagoon deploy branch -p <YOUR-PROJECT-NAME> -b <YOUR-BRANCH-NAME>\nGo to the Lagoon UI and take a look at your project - you should now see the environment for this project!
kubectl get pods --all-namespaces | grep lagoon-build\nInfo
This is only applicable to AWS installations.
Add Helm repository:
Add Helm repohelm repo add stable https://charts.helm.sh/stable\nCreate efs-provisioner-values.yml in your config directory and update the values:
efsProvisioner:\nefsFileSystemId: <efsFileSystemId>\nawsRegion: <awsRegion>\npath: /\nprovisionerName: example.com/aws-efs\nstorageClass:\nname: bulk\nisDefault: false\nreclaimPolicy: Delete\nmountOptions: []\nglobal:\ndeployEnv: prod\nInstall EFS Provisioner:
Install EFS Provisionerhelm upgrade --install --create-namespace \\\n--namespace efs-provisioner --wait \\\n-f efs-provisioner-values.yaml \\\nefs-provisioner stable/efs-provisioner\nNot needed for *most* installs, but this is configured to integrate Lagoon with GitLab for user and group authentication.
your-gitlab.com/admin/hooks pointing to: webhookhandler.lagoon.example.com and define a random secret token.Update lagoon-core-values.yml:
api:\nadditionalEnvs:\nGITLAB_API_HOST: <<URL of GitLab example: https://your-gitlab.com>>\nGITLAB_API_TOKEN: << Personal Access token with Access to API >>\nGITLAB_SYSTEM_HOOK_TOKEN: << System Hook Secret Token >>\nwebhook-haondler:\nadditionalEnvs:\nGITLAB_API_HOST: <<URL of GitLab example: https://your-gitlab.com>>\nGITLAB_API_TOKEN: << Personal Access token with Access to API >>\nGITLAB_SYSTEM_HOOK_TOKEN: << System Hook Secret Token >>\nwebhooks2tasks:\nadditionalEnvs:\nGITLAB_API_HOST: <<URL of GitLab example: https://your-gitlab.com>>\nGITLAB_API_TOKEN: << Personal Access token with Access to API >>\nGITLAB_SYSTEM_HOOK_TOKEN: << System Hook Secret Token >>\nHelm update the lagoon-core helmchart.
yarn sync:gitlab:all\nAdd Helm repository:
Add Helm repositoryhelm repo add harbor https://helm.goharbor.io\nConsider the optimal configuration of Harbor for your particular circumstances - see their docs for more recommendations:
imageChartStorage).database.type).redis.type)Create the file harbor-values.yml inside of your config directory. The proxy-buffering annotations help with large image pushes:
expose:\ningress:\nannotations:\nkubernetes.io/tls-acme: \"true\"\nnginx.ingress.kubernetes.io/proxy-buffering: \"off\"\nnginx.ingress.kubernetes.io/proxy-request-buffering: \"off\"\nhosts:\ncore: harbor.lagoon.example.com\ntls:\nenabled: true\ncertSource: secret\nsecret:\nsecretName: harbor-harbor-ingress\nexternalURL: https://harbor.lagoon.example.com\nharborAdminPassword: <your Harbor Admin Password>\nchartmuseum:\nenabled: false\nclair:\nenabled: false\nnotary:\nenabled: false\ntrivy:\nenabled: false\njobservice:\njobLogger: stdout\nInstall Harbor, checking the requirements for the currently supported Harbor versions:
Install Harborhelm upgrade --install --create-namespace \\\n--namespace harbor --wait \\\n-f harbor-values.yml \\\nharbor harbor/harbor\nVisit Harbor at the URL you set in harbor.yml.
kubectl -n harbor get secret harbor-core -o jsonpath=\"{.data.HARBOR_ADMIN_PASSWORD}\" | base64 --decode\nYou will need to add the above Harbor credentials to the Lagoon Remote values.yml in the next step, as well as harbor-values.yml.
Now we will install Lagoon Remote into the Lagoon namespace. The RabbitMQ service is the broker.
Create lagoon-remote-values.yml in your config directory as you did the previous two files, and update the values.
kubectl -n lagoon-core get secret lagoon-core-broker -o jsonpath=\"{.data.RABBITMQ_PASSWORD}\" | base64 --decode\nlagoon-core-broker.lagoon-core.svc.local\nkubectl get service lagoon-core-broker-amqp-ext \\\n-o custom-columns=\"NAME:.metadata.name,IP ADDRESS:.status.loadBalancer.ingress[*].ip,HOSTNAME:.status.loadBalancer.ingress[*].hostname\"\nkubectl -n harbor get secret harbor-harbor-core -o jsonpath=\"{.data.HARBOR_ADMIN_PASSWORD}\" | base64 --decode\nAdd the Harbor configuration from the Install Harbor step.
lagoon-remote-values.ymllagoon-build-deploy:\nenabled: true\nextraArgs:\n- \"--enable-harbor=true\"\n- \"--harbor-url=https://harbor.lagoon.example.com\"\n- \"--harbor-api=https://harbor.lagoon.example.com/api/\"\n- \"--harbor-username=admin\"\n- \"--harbor-password=<from harbor-harbor-core secret>\"\nrabbitMQUsername: lagoon\nrabbitMQPassword: <from lagoon-core-broker secret>\nrabbitMQHostname: lagoon-core-broker.lagoon-core.svc.cluster.local\nlagoonTargetName: <name of lagoon remote, can be anything>\ntaskSSHHost: <IP of ssh service loadbalancer>\ntaskSSHPort: \"22\"\ntaskAPIHost: \"api.lagoon.example.com\"\ndbaas-operator:\nenabled: true\nmariadbProviders:\nproduction:\nenvironment: production\nhostname: 172.17.0.1.nip.io\nreadReplicaHostnames:\n- 172.17.0.1.nip.io\npassword: password\nport: '3306'\nuser: root\ndevelopment:\nenvironment: development\nhostname: 172.17.0.1.nip.io\nreadReplicaHostnames:\n- 172.17.0.1.nip.io\npassword: password\nport: '3306'\nuser: root\nInstall Lagoon Remote:
Install Lagoon remotehelm upgrade --install --create-namespace \\\n--namespace lagoon \\\n-f remote-values.yaml \\\nlagoon-remote lagoon/lagoon-remote\nLagoon uses the K8up backup operator: https://k8up.io. Lagoon isn\u2019t tightly integrated with K8up, it\u2019s more that Lagoon can create its resources in a way that K8up can automatically discover and backup.
Lagoon has been extensively tested with K8up 1.x, but is not compatible with 2.x yet. We recommend using the 1.1.0 chart version (App version v1.2.0).
Create new AWS User with policies:
example K8up IAM user{\n\"Version\":\"2012-10-17\",\n\"Statement\":[\n{\n\"Sid\":\"VisualEditor0\",\n\"Effect\":\"Allow\",\n\"Action\":[\n\"s3:ListAllMyBuckets\",\n\"s3:CreateBucket\",\n\"s3:GetBucketLocation\"\n],\n\"Resource\":\"*\"\n},\n{\n\"Sid\":\"VisualEditor1\",\n\"Effect\":\"Allow\",\n\"Action\":\"s3:ListBucket\",\n\"Resource\":\"arn:aws:s3:::baas-*\"\n},\n{\n\"Sid\":\"VisualEditor2\",\n\"Effect\":\"Allow\",\n\"Action\":[\n\"s3:PutObject\",\n\"s3:GetObject\",\n\"s3:AbortMultipartUpload\",\n\"s3:DeleteObject\",\n\"s3:ListMultipartUploadParts\"\n],\n\"Resource\":\"arn:aws:s3:::baas-*/*\"\n}\n]\n}\nCreate k8up-values.yml (customize for your provider):
k8up:\nenvVars:\n- name: BACKUP_GLOBALS3ENDPOINT\nvalue: 'https://s3.eu-west-1.amazonaws.com'\n- name: BACKUP_GLOBALS3BUCKET\nvalue: ''\n- name: BACKUP_GLOBALKEEPJOBS\nvalue: '1'\n- name: BACKUP_GLOBALSTATSURL\nvalue: 'https://backup.lagoon.example.com'\n- name: BACKUP_GLOBALACCESSKEYID\nvalue: ''\n- name: BACKUP_GLOBALSECRETACCESSKEY\nvalue: ''\n- name: BACKUP_BACKOFFLIMIT\nvalue: '2'\n- name: BACKUP_GLOBALRESTORES3BUCKET\nvalue: ''\n- name: BACKUP_GLOBALRESTORES3ENDPOINT\nvalue: 'https://s3.eu-west-1.amazonaws.com'\n- name: BACKUP_GLOBALRESTORES3ACCESSKEYID\nvalue: ''\n- name: BACKUP_GLOBALRESTORES3SECRETACCESSKEY\nvalue: ''\ntimezone: Europe/Zurich\nInstall K8up:
Install K8up Step 1helm repo add appuio https://charts.appuio.ch\nkubectl apply -f https://github.com/vshn/k8up/releases/download/v1.2.0/k8up-crd.yaml\nhelm upgrade --install --create-namespace \\\n--namespace k8up \\\n-f k8up-values.yaml \\\n--version 1.1.0 \\\nk8up appuio/k8up\nUpdate lagoon-core-values.yml:
s3BAASAccessKeyID: <<Access Key ID for restore bucket>>\ns3BAASSecretAccessKey: <<Access Key Secret for restore bucket>>\nRedeploy lagoon-core.
brew tap uselagoon/lagoon-clibrew install lagoonThe CLI needs to know how to communicate with Lagoon, so run the following command:
Lagoon config lagoon config add \\\n--graphql https://YOUR-API-URL/graphql \\\n--ui https://YOUR-UI-URL \\\n--hostname YOUR.SSH.IP \\\n--lagoon YOUR-LAGOON-NAME \\\n--port 22\nAccess Lagoon by authenticating with your SSH key.
values.yml if you forget), go to Settings.You need to set the default Lagoon to your Lagoon so that it doesn\u2019t try to use the amazee.io defaults:
Lagoon config lagoon config default --lagoon <YOUR-LAGOON-NAME>\nNow run lagoon login. Lagoon talks to SSH and authenticates against your public/private key pair, and gets a token for your username.
Verify via lagoon whoami that you are logged in.
Info
We don\u2019t generally recommend using the Lagoon Admin role, but you\u2019ll need to create an admin account at first to get started. Ideally, you\u2019ll immediately create another account to work from which is not an admin.
"},{"location":"installing-lagoon/lagoon-core/","title":"Install Lagoon Core","text":""},{"location":"installing-lagoon/lagoon-core/#install-the-helm-chart","title":"Install the Helm chart","text":"Add Lagoon Charts repository to your Helm Repositories:
Add Lagoon Charts repositoryhelm repo add lagoon https://uselagoon.github.io/lagoon-charts/\nCreate a directory for the configuration files we will create, and make sure that it\u2019s version controlled. Ensure that you reference this path in commands referencing your values.yml files.
values.yml in the directory you\u2019ve just created. Update the endpoint URLs (change them from api.lagoon.example.com to your values). Example: https://github.com/uselagoon/lagoon-charts/blob/main/charts/lagoon-core/ci/linter-values.yamlNow run helm upgrade --install command, pointing to values.yml, like so:
helm upgrade --install --create-namespace --namespace lagoon-core -f values.yml lagoon-core lagoon/lagoon-core`\nLagoon Core is now installed!
Warning
Sometimes we run into Docker Hub pull limits. We are considering moving our images elsewhere if this continues to be a problem.
"},{"location":"installing-lagoon/lagoon-core/#configure-keycloak","title":"Configure Keycloak","text":"Visit the Keycloak dashboard at the URL you defined in the values.yml for Keycloak.
adminlagoon-core-keycloak secret, key-value KEYCLOAK_ADMIN_PASSWORDRetrieve the secret like so:
Retrieve secretkubectl -n lagoon-core get secret lagoon-core-keycloak -o jsonpath=\"{.data.KEYCLOAK_ADMIN_PASSWORD}\" | base64 --decode\nClick on User on top right.
You should now be able to visit the Lagoon UI at the URL you defined in the values.yml for the UI.
lagoonadminlagoon-core-keycloak secret key-value: LAGOON-CORE-KEYCLOAK kubectl -n lagoon-core get secret lagoon-core-keycloak -o jsonpath=\"{.data.KEYCLOAK_LAGOON_ADMIN_PASSWORD}\" | base64 --decode\nLagoon files are used to store the file output of tasks, such as backups, and can be hosted on any S3-compatible storage.
Create new AWS User with policies:
Example files IAM user{\n\"Version\":\"2012-10-17\",\n\"Statement\":[\n{\n\"Effect\":\"Allow\",\n\"Action\":[\n\"s3:ListBucket\",\n\"s3:GetBucketLocation\",\n\"s3:ListBucketMultipartUploads\"\n],\n\"Resource\":\"arn:aws:s3:::S3_BUCKET_NAME\"\n},\n{\n\"Effect\":\"Allow\",\n\"Action\":[\n\"s3:PutObject\",\n\"s3:GetObject\",\n\"s3:DeleteObject\",\n\"s3:ListMultipartUploadParts\",\n\"s3:AbortMultipartUpload\"\n],\n\"Resource\":\"arn:aws:s3:::S3_BUCKET_NAME/*\"\n}\n]\n}\nUpdate lagoon-core-values.yml:
s3FilesAccessKeyID: <<Access Key ID>>\ns3FilesBucket: <<Bucket Name for Lagoon Files>>\ns3FilesHost: <<S3 endpoint like \"https://s3.eu-west-1.amazonaws.com\" >>\ns3FilesSecretAccessKey: <<Access Key Secret>>\ns3FilesRegion: <<S3 Region >>\nIf you use ingress-nginx in front of lagoon-core, we suggest setting this configuration which will allow for bigger file uploads:
controller:\nconfig:\nclient-body-timeout: '600' # max 600 secs fileuploads\nproxy-send-timeout: '1800' # max 30min connections - needed for websockets\nproxy-read-timeout: '1800' # max 30min connections - needed for websockets\nproxy-body-size: 1024m # 1GB file size\nproxy-buffer-size: 64k # bigger buffer\nLagoon integrates with OpenSearch to store application, container and router logs. Lagoon Logging collects the application, router and container logs from Lagoon projects, and sends them to the logs concentrator. It needs to be installed onto each lagoon-remote instance.
In addition, it should be installed in the lagoon-core cluster to collect logs from the lagoon-core service. This is configured in the LagoonLogs section.
Logging Overview: Lucid Chart
See also: Logging.
Read more about Lagoon logging here: https://github.com/uselagoon/lagoon-charts/tree/main/charts/lagoon-logging
Create lagoon-logging-values.yaml:
tls:\ncaCert: |\n<< content of ca.pem from Logs-Concentrator>>\nclientCert: |\n<< content of client.pem from Logs-Concentrator>>\nclientKey: |\n<< content of client-key.pem from Logs-Concentrator>>\nforward:\nusername: <<Username for Lagoon Remote 1>>\npassword: <<Password for Lagoon Remote 1>>\nhost: <<ExternalIP of Logs-Concentrator Service LoadBalancer>>\nhostName: <<Hostname in Server Cert of Logs-Concentrator>>\nhostPort: '24224'\nselfHostname: <<Hostname in Client Cert of Logs-Concentrator>>\nsharedKey: <<Generated ForwardSharedKey of Logs-Concentrator>>\ntlsVerifyHostname: false\nclusterName: <<Short Cluster Identifier>>\nlogsDispatcher:\nserviceMonitor:\nenabled: false\nlogging-operator:\nmonitoring:\nserviceMonitor:\nenabled: false\nlagoonLogs:\nenabled: true\nrabbitMQHost: lagoon-core-broker.lagoon-core.svc.cluster.local\nrabbitMQUser: lagoon\nrabbitMQPassword: <<RabbitMQ Lagoon Password>>\nexcludeNamespaces: {}\nInstall lagoon-logging:
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com\n\nhelm upgrade --install --create-namespace \\\n--namespace lagoon-logging \\\n-f lagoon-logging-values.yaml \\\nlagoon-logging lagoon/lagoon-logging\nIf you'd like logs from ingress-nginx inside lagoon-logging:
ingress-nginxAdd the content of this file to ingress-nginx:
controller:\nconfig:\nlog-format-upstream: >-\n{\n\"time\": \"$time_iso8601\",\n\"remote_addr\": \"$remote_addr\",\n\"x-forwarded-for\": \"$http_x_forwarded_for\",\n\"true-client-ip\": \"$http_true_client_ip\",\n\"req_id\": \"$req_id\",\n\"remote_user\": \"$remote_user\",\n\"bytes_sent\": $bytes_sent,\n\"request_time\": $request_time,\n\"status\": \"$status\",\n\"host\": \"$host\",\n\"request_proto\": \"$server_protocol\",\n\"request_uri\": \"$uri\",\n\"request_query\": \"$args\",\n\"request_length\": $request_length,\n\"request_time\": $request_time,\n\"request_method\": \"$request_method\",\n\"http_referer\": \"$http_referer\",\n\"http_user_agent\": \"$http_user_agent\",\n\"namespace\": \"$namespace\",\n\"ingress_name\": \"$ingress_name\",\n\"service_name\": \"$service_name\",\n\"service_port\": \"$service_port\"\n}\nYour logs should start flowing!
Logs-concentrator collects the logs being sent by Lagoon clusters and augments them with additional metadata before inserting them into Elasticsearch.
Create logs-concentrator-values.yml:
tls:\ncaCert: |\n<<contents of ca.pem>>\nserverCert: |\n<<contents of server.pem\nserverKey: |\n<<contents of server-key.pem>>\nelasticsearchHost: elasticsearch-opendistro-es-client-service.elasticsearch.svc.cluster.local\nelasticsearchAdminPassword: <<ElasticSearch Admin Password>>\nforwardSharedKey: <<Random 32 Character Password>>\nusers:\n- username: <<Username for Lagoon Remote 1>>\npassword: <<Random Password for Lagoon Remote 1>>\nservice:\ntype: LoadBalancer\nserviceMonitor:\nenabled: false\nInstall logs-concentrator:
Install logs-concentratorhelm upgrade --install --create-namespace \\\n--namespace lagoon-logs-concentrator \\\n-f logs-concentrator-values.yaml \\\nlagoon-logs-concentrator lagoon/lagoon-logs-concentrator\nTo install an OpenDistro cluster, you will need to configure TLS and secrets so that Lagoon can talk to it securely. You're going to have to create a handful of JSON files - put these in the same directory as the values files you've been creating throughout this installation process.
Install OpenDistro Helm, according to https://opendistro.github.io/for-elasticsearch-docs/docs/install/helm/
"},{"location":"installing-lagoon/opendistro/#create-keys-and-certificates","title":"Create Keys and Certificates","text":"Generate certificates
Note:
CFSSL is CloudFlare's PKI/TLS swiss army knife. It is both a command line tool and an HTTP API server for signing, verifying, and bundling TLS certificates. It requires Go 1.12+ to build.
{\n\"CN\": \"ca.elasticsearch.svc.cluster.local\",\n\"hosts\": [\n\"ca.elasticsearch.svc.cluster.local\"\n],\n\"key\": {\n\"algo\": \"ecdsa\",\n\"size\": 256\n},\n\"ca\": {\n\"expiry\": \"87600h\"\n}\n}\nRun the following two commands:
Generate certificatecfssl gencert -initca ca-csr.json | cfssljson -bare ca -\nrm ca.csr\nYou'll get ca-key.pem, and ca.pem. This is your CA key and self-signed certificate.
Next, we'll generate the node peering certificate. You'll need the following two files:
ca-config.json{\n\"signing\": {\n\"default\": {\n\"expiry\": \"87600h\"\n},\n\"profiles\": {\n\"peer\": {\n\"expiry\": \"87600h\",\n\"usages\": [\n\"signing\",\n\"key encipherment\",\n\"server auth\",\n\"client auth\"\n]\n},\n\"client\": {\n\"expiry\": \"87600h\",\n\"usages\": [\n\"signing\",\n\"key encipherment\",\n\"client auth\"\n]\n}\n}\n}\n}\n{\n\"hosts\": [\n\"node.elasticsearch.svc.cluster.local\"\n],\n\"CN\": \"node.elasticsearch.svc.cluster.local\",\n\"key\": {\n\"algo\": \"ecdsa\",\n\"size\": 256\n}\n}\nRun the following two commands:
Generate certificate keyscfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer node.json | cfssljson -bare node\nrm node.csr\nYou'll get node.pem and node-key.pem. This is the peer certificate that will be used by nodes in the ES cluster.
Next, we'll convert the key to the format supported by Java with the following command:
Convert key formatopenssl pkey -in node-key.pem -out node-key.pkcs8\nNow we'll generate the admin certificate. You'll need the following file:
admin.json{\n\"CN\": \"admin.elasticsearch.svc.cluster.local\",\n\"key\": {\n\"algo\": \"ecdsa\",\n\"size\": 256\n}\n}\nRun the following two commands:
Generate admin certificate keyscfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client admin.json | cfssljson -bare admin\nrm admin.csr\nYou'll get admin.pem and admin-key.pem. This is the certificate that will be used to perform admin commands on the opendistro-security plugin.
Next, we'll convert the key to the format supported by Java with the following command:
Convert key formatopenssl pkey -in admin-key.pem -out admin-key.pkcs8\nNow that we have our keys and certificates, we can continue with the installation.
Generate hashed passwords.
elasticsearch-secrets-values.yaml needs two hashed passwords. Create them with this command (run it twice, enter a random password, store both the plaintext and hashed passwords).docker run --rm -it docker.io/amazon/opendistro-for-elasticsearch:1.12.0 sh -c \"chmod +x /usr/share/elasticsearch/plugins/opendistro_security/tools/hash.sh; /usr/share/elasticsearch/plugins/opendistro_security/tools/hash.sh\"\nCreate secrets:
elasticsearch-secrets-values.yaml. See this gist as an example: https://gist.github.com/Schnitzel/43f483dfe0b23ca0dddd939b12bb4b0bInstall secrets with the following commands:
Install secretshelm repo add incubator https://charts.helm.sh/incubator`\nhelm upgrade --namespace elasticsearch --create-namespace --install elasticsearch-secrets incubator/raw --values elasticsearch-secrets-values.yaml `\nYou'll need to create elasticsearch-values.yaml. See this gist as an example: (fill all <\\> with values) https://gist.github.com/Schnitzel/1e386654b6abf75bf4d66a544db4aa6a
Install Elasticsearch:
Install Elasticsearchhelm upgrade --namespace elasticsearch --create-namespace --install elasticsearch opendistro-es-X.Y.Z.tgz --values elasticsearch-values.yaml\nConfigure security inside Elasticsearch with the following:
Configure securitykubectl exec -n elasticsearch -it elasticsearch-opendistro-es-master-0 -- bash\nchmod +x /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh\n/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -nhnv -cacert /usr/share/elasticsearch/config/admin-root-ca.pem -cert /usr/share/elasticsearch/config/admin-crt.pem -key /usr/share/elasticsearch/config/admin-key.pem -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/\nUpdate lagoon-core-values.yaml with:
elasticsearchURL: http://elasticsearch-opendistro-es-client-service.elasticsearch.svc.cluster.local:9200\nkibanaURL: https://<<Kibana Public URL>>\nlogsDBAdminPassword: \"<<PlainText Elasticsearch Admin Password>>\"\nRollout Lagoon Core:
Rollout Lagoon Corehelm upgrade --install --create-namespace --namespace lagoon-core -f values.yaml lagoon-core lagoon/lagoon-core\nSync all Lagoon Groups with Opendistro Elasticsearch
Sync groupskubectl -n lagoon-core exec -it deploy/lagoon-core-api -- sh\nyarn run sync:opendistro-security\nYou\u2019ll need an app for sending and receiving GraphQL queries. We recommend GraphiQL.
brew install --cask graphiql.We need to tell Lagoon Core about the Kubernetes cluster. The GraphQL endpoint is: https://<YOUR-API-URL>/graphql
Go to Edit HTTP Headers, and Add Header.
AuthorizationBearer YOUR-TOKEN-HERE.lagoon.yml file. Copy the token from that file and use it for the value here.Now you\u2019re ready to run some queries. Run the following test query to ensure everything is working correctly:
Get all projectsquery allProjects {allProjects {name } }\nThis should give you the following response:
API Response {\n \"data\": {\n \"allProjects\": []\n }\n }\nRead more about GraphQL here in our documentation.
Once you get the correct response, we need to add a mutation.
Run the following query:
Add mutationmutation addKubernetes {\n addKubernetes(input:\n {\n name: \"<TARGET-NAME-FROM-REMOTE-VALUES.yml>\",\n consoleUrl: \"<URL-OF-K8S-CLUSTER>\",\n token: \"xxxxxx\u201d\n routerPattern: \"${environment}.${project}.lagoon.example.com\"\n }){id}\n}\nname: get from lagoon-remote-values.ymlconsoleUrl: API Endpoint of Kubernetes cluster. Get from values.ymltoken: create a token for the lagoon-build-deploy service account
kubectl -n lagoon create token lagoon-build-deploy --duration 3h\nPrior to Kubernetes 1.21:
Use the lagoon-build-deploy token installed by lagoon-remote:
kubectl -n lagoon describe secret \\\n$(kubectl -n lagoon get secret | grep lagoon-build-deploy | awk '{print $1}') | grep token: | awk '{print $2}'\nInfo
Authorization tokens for GraphQL are very short term so you may need to generate a new one. Run lagoon login and then cat the .lagoon.yml file to get the new token, and replace the old token in the HTTP header with the new one.
Note
We acknowledge that this is a lot of steps, and our roadmap for the immediate future includes reducing the number of steps in this process.
"},{"location":"installing-lagoon/requirements/#specific-requirements-as-of-january-2023","title":"Specific requirements (as of January 2023)","text":""},{"location":"installing-lagoon/requirements/#kubernetes","title":"Kubernetes","text":"Lagoon supports Kubernetes versions 1.21 onwards. We actively test and develop against Kubernetes 1.24, also regularly testing against 1.21,1.22 and 1.25.
The next large round of breaking changes is in Kubernetes 1.25, and we will endeavour to be across these in advance, although this will require a bump in the minimum supported version of Lagoon.
"},{"location":"installing-lagoon/requirements/#ingress-nginx","title":"ingress-nginx","text":"Lagoon is currently configured only for a single ingress-nginx controller, and therefore defining an IngressClass was not necessary in the past.
In order to use the recent ingress-nginx controllers (v4 onwards, required for Kubernetes 1.22), the following configuration should be used, as per the ingress-nginx docs.
nginx-ingress should be configured as the default controller - set .controller.ingressClassResource.default: true in Helm valuesnginx-ingress should be configured to watch ingresses without IngressClass set - set .controller.watchIngressWithoutClass: true in Helm valuesThis will configure the controller to create any new ingresses with itself as the IngressClass, and also to handle any existing ingresses without an IngressClass set.
Other configurations may be possible, but have not been tested.
"},{"location":"installing-lagoon/requirements/#harbor","title":"Harbor","text":"Versions 2.1 and 2.2+ of Harbor are currently supported. The method of retrieving robot accounts was changed in 2.2, and the Lagoon remote-controller is able to handle these tokens. This means that Harbor has to be configured with the credentials in lagoon-build-deploy - not lagoon-core.
We recommend installing a Harbor version greater than 2.6.0 with Helm chart 1.10.0 or greater.
"},{"location":"installing-lagoon/requirements/#k8up-for-backups","title":"k8up for backups","text":"Lagoon has built in configuration for the K8up backup operator. Lagoon can configure prebackup pods, schedules and retentions, and manage backups and restores for K8up. Lagoon currently only supports the 1.x versions of K8up, owing to a namespace change in v2 onwards, but we are working on a fix.
K8up v2:
Lagoon does not currently support K8up v2 onwards due to a namespace change here.
We recommend installing K8up version 1.2.0 with Helm Chart 1.1.0
"},{"location":"installing-lagoon/requirements/#storage-provisioners","title":"Storage provisioners","text":"Lagoon utilizes a default 'standard' StorageClass for most workloads, and the internal provisioner for most Kubernetes platforms will suffice. This should be configured to be dynamic provisioning and expandable where possible.
Lagoon also requires a StorageClass called 'bulk' to be available to support persistant pod replicas (across nodes). This StorageClass should support ReadWriteMany (RWX) access mode and should be configured to be dynamic provisioning and expandable where possible. See https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes for more information, and the production drivers list for a complete list of compatible drivers.
We have curently only included the instructions for (the now deprecated) EFS Provisioner. The production EFS CSI driver has issues with provisioning more than 120 PVCs. We are awaiting upstream possible fixes here and here - but most other providers CSI drivers should also work, as will configurations with an NFS-compatible server and provisioner.
"},{"location":"installing-lagoon/requirements/#how-much-kubernetes-experienceknowledge-is-required","title":"How much Kubernetes experience/knowledge is required?","text":"Lagoon uses some very involved Kubernetes and cloud-native concepts, and while full familiarity may not be necessary to install and configure Lagoon, diagnosing issues and contributing may prove difficult without a good level of familiarity.
As an indicator, comfort with the curriculum for the Certified Kubernetes Administrator would be suggested as a minimum.
"},{"location":"installing-lagoon/update-lagoon/","title":"Updating","text":"Download newest charts using Helm.
Download newest chartshelm repo update\nCheck with helm diff for changes (https://github.com/databus23/helm-diff).
helm diff upgrade --install --create-namespace --namespace lagoon-core \\\n-f values.yml lagoon-core lagoon/lagoon-core\nBack up the Lagoon databases prior to any Helm actions. We also suggest scaling the API to a single pod, to aid the database migration scripts running in the initContainers.
Run the upgrade using Helm.
Run upgradehelm upgrade --install --create-namespace --namespace lagoon-core \\\n-f values.yaml lagoon-core lagoon/lagoon-core\n(Note that as of Lagoon v2.11.0, this step is no longer required.) If upgrading Lagoon Core, ensure you run the rerun_initdb.sh script to perform post upgrade migrations.
kubectl --namespace lagoon-core exec -it lagoon-core-api-db-0 -- \\\nsh -c /rerun_initdb.sh\nRe-scale the API pods back to their original level.
If upgrading Lagoon Core, and you have enabled groups/user syncing for OpenSearch, you may additionally need to run the sync:opendistro-security script to update the groups in OpenSearch. This command can also be prefixed with a GROUP_REGEX=<group-to-sync to sync a single group at a time, as syncing the entire group structure may take a long time.
kubectl --namespace lagoon-core exec -it deploy/lagoon-core-api -- \\\nsh -c yarn sync:opendistro-security\nCheck https://github.com/uselagoon/lagoon/releases for additional upgrades.
"},{"location":"installing-lagoon/update-lagoon/#database-backups","title":"Database Backups","text":"You may want to back up the databases before upgrading Lagoon Core, the following will create backups you can use to restore from if required. You can delete them afterwards.
"},{"location":"installing-lagoon/update-lagoon/#api-db","title":"API DB","text":"Back up API DBkubectl --namespace lagoon-core exec -it lagoon-core-api-db-0 -- \\\nsh -c 'mysqldump --max-allowed-packet=500M --events \\\n --routines --quick --add-locks --no-autocommit \\\n --single-transaction infrastructure | gzip -9 > \\\n /var/lib/mysql/backup/$(date +%Y-%m-%d_%H%M%S).infrastructure.sql.gz'\nkubectl --namespace lagoon-core exec -it lagoon-core-keycloak-db-0 -- \\\nsh -c 'mysqldump --max-allowed-packet=500M --events \\\n --routines --quick --add-locks --no-autocommit \\\n --single-transaction keycloak | gzip -9 > \\\n /var/lib/mysql/backup/$(date +%Y-%m-%d_%H%M%S).keycloak.sql.gz'\nHave you seen the Kibana getting started video and are now ready to work with logs? We are here to help! This page will give you examples of Kibana queries you can use. This is not a Kibana 101 class, but it can help you understand some of what you can do in Kibana.
Ready to get started? Good!
Note
Make sure that you have selected your tenant before starting! You can do that by on the Tenant icon on the left-hand menu. Once you have selected your tenant, click on the Discover icon again to get started.
Below you'll find examples for two common log requests:
Discovery (#1 in screen shot below)In the search bar (#3), enter:
openshift_project: \"name of your production project\"
Running the query above will give you a general look at all the traffic to your site, but what if you want to narrow in on a specific IP address? Perhaps you want to see how many times an IP has hit your site and what specific pages they were looking at. This next query should help.
We are going to start off with the same query as above, but we are going to add a couple of things.
client_ip and http_request.That looks good, but what if we wanted to just show requests from a specific IP address? You can filter for the address by adding it to your search criteria.
AND client_ip: \"IP address\".Container logs will show you all stout and sterr messages for your specific container and project. We are going to show an example for getting logs from a specific container and finding specific error numbers in that container.
Want to see the logs for a specific container (php, nginx, etc)? This section will help! Let's focus on looking at NGINX logs.
kubernetes.container_name: \"nginx\"Want to see how many 500 Internal Server errors you've had in your NGINX container? You can do that by changing the search query. If you search:
kubernetes.container_name: \"nginx\" AND message: \"500\"
That will only display 500 error messages in the NGINX container. You can search for any error message in any container that you would like!
"},{"location":"logging/kibana-examples/#visualization","title":"Visualization","text":"Kibana will also give you the option to create visualizations or graphs. We are going to create a chart to show number of hits/requests in a month using the same query we used above.
Note
Make sure that you select an appropriate time frame for the data in the upper right hand corner.
Here is an example of a daily hits visualization chart:
Also note that you can save your visualizations (and searches)! That will make it even faster to access them in the future. And because each account has their own Kibana Tenant, no searches or visualizations are shared with another account.
"},{"location":"logging/kibana-examples/#troubleshooting","title":"Troubleshooting","text":""},{"location":"logging/logging/","title":"Logging","text":"Lagoon provides access to the following logs via Kibana:
stdout and stderr messagesudp://application-logs.lagoon.svc:5140type field contains the name of the Kubernetes namespace ($LAGOON_PROJECT-$LAGOON_ENVIRONMENT).To access the logs, please check with your Lagoon administrator to get the URL for the Kibana route (for amazee.io, this is https://logs.amazeeio.cloud/).
Each Lagoon user account has their own login and will see the logs only for the projects to which they have access.
Each Lagoon user account also has their own Kibana Tenant, which means no saved searches or visualizations are shared with another account.
If you would like to know more about how to use Kibana: https://www.elastic.co/webinars/getting-started-kibana.
"},{"location":"resources/faq/","title":"FAQ","text":""},{"location":"resources/faq/#how-do-i-contact-my-lagoon-administrator","title":"How do I contact my Lagoon administrator?","text":"You should have a private Slack channel that was set up for you to communicate - if not, or you've forgotten how to contact us, reach out at support@amazee.io.
"},{"location":"resources/faq/#i-found-a-bug","title":"I found a bug! \ud83d\udc1e","text":"If you've found a bug or security issue, please send your findings to support@amazee.io. Please DO NOT file a GitHub issue for them.
"},{"location":"resources/faq/#im-interested-in-amazeeios-hosting-services-with-lagoon","title":"I'm interested in amazee.io's hosting services with Lagoon","text":"That's great news! You can contact them via email at inquiries@amazee.io.
"},{"location":"resources/faq/#how-can-i-restore-a-backup","title":"How can I restore a backup?","text":"We have backups available for files and databases, typically taken every 24 hours at most. These backups are stored offsite.
We keep up to 7 daily backups and 4 weekly backups.
If you ever need to recover or restore a backup, feel free to submit a ticket or send us a message via chat and we will be more than happy to help!
"},{"location":"resources/faq/#how-can-i-download-a-database-dump","title":"How can I download a database dump?","text":""},{"location":"resources/faq/#im-getting-an-invalid-ssl-certificate-error","title":"I'm getting an invalid SSL certificate error","text":"The first thing to try is what is listed in our documentation about SSL.
If you follow those steps, and you are still seeing an error, please submit a ticket or send us a message on chat and we can help resolve this for you.
"},{"location":"resources/faq/#im-getting-an-array-error-when-running-a-drush-command","title":"I'm getting an \"Array\" error when running a Drush command","text":"This was a bug that was prevalent in Drush versions 8.1.16 and 8.1.17. There error would look something like this:
Text OnlyThe command could not be executed successfully (returned: Array [error]\n(\n[default] => Array\n(\n[default] => Array\n(\n[driver] => mysql\n[prefix] => Array\n(\n[default] =>\n)\n, code: 0)\nError: no database record could be found for source @main [error]\nUpgrading Drush should fix that for you. We strongly suggest that you use version 8.3 or newer. Once Drush is upgraded the command should work!
"},{"location":"resources/faq/#im-seeing-an-internal-server-error-when-trying-to-access-my-kibana-logs","title":"I'm seeing an Internal Server Error when trying to access my Kibana logs","text":"No need to panic! This usually happens when a tenant has not been selected. To fix this, follow these steps:
You should now be able to see your logs.
"},{"location":"resources/faq/#im-unable-to-ssh-into-any-environment","title":"I'm unable to SSH into any environment","text":"I'm unable to SSH into any environment. I'm getting the following message: Permission denied (publickey). When I run drush sa no aliases are returned.
This typically indicates an issue with Pygmy. You can find our troubleshooting docs for Pygmy here: https://pygmy.readthedocs.io/en/master/troubleshooting/
"},{"location":"resources/faq/#how-can-i-check-the-status-of-a-build","title":"How can I check the status of a build?","text":""},{"location":"resources/faq/#how-do-i-add-a-cron-job","title":"How do I add a cron job?","text":""},{"location":"resources/faq/#how-do-i-add-a-new-route","title":"How do I add a new route?","text":""},{"location":"resources/faq/#how-do-i-remove-a-route","title":"How do I remove a route?","text":"You will need to contact your helpful Lagoon administrator should you need to remove a route. You can use the Slack channel that was set up for you to communicate - if not, you can always reach us at support@amazee.io or on Discord.
"},{"location":"resources/faq/#when-i-run-pygmy-status-no-keys-are-loaded","title":"When I runpygmy status, no keys are loaded","text":"You'll need to load your SSH key into pygmy. Here's how: https://pygmy.readthedocs.io/en/master/ssh_agent
"},{"location":"resources/faq/#when-i-run-drush-sa-no-aliases-are-returned","title":"When I rundrush sa no aliases are returned","text":"This typically indicates an issue with Pygmy. You can find our troubleshooting docs for Pygmy here: https://pygmy.readthedocs.io/en/master/troubleshooting
"},{"location":"resources/faq/#my-deployments-fail-with-a-message-saying-drush-needs-a-more-functional-environment","title":"My deployments fail with a message saying: \"drush needs a more functional environment\"","text":"This usually means that there is no database uploaded to the project. Follow our step-by-step guide to add a database to your project.
"},{"location":"resources/faq/#when-i-start-pygmy-i-see-an-address-already-in-use-error","title":"When I start Pygmy I see an \"address already in use\" error?","text":"Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use Error: failed to start containers: amazeeio-haproxy
This is a known error! Most of the time it means that there is already something running on port 80. You can find the culprit by running the following query:
Text Onlynetstat -ltnp | grep -w ':80'\nThat should list everything running on port 80. Kill the process running on port 80. Once port 80 is freed up, Pygmy should start up with no further errors.
"},{"location":"resources/faq/#how-can-i-change-branchespr-environmentsproduction-on-my-project","title":"How can I change branches/PR environments/production on my project?","text":"You can make that change using the Lagoon API! You can find the documentation for this change in our GraphQL documentation.
"},{"location":"resources/faq/#how-do-i-add-a-redirect","title":"How do I add a redirect?","text":""},{"location":"resources/faq/#how-can-i-add-new-users-and-ssh-keys-to-my-projectgroup","title":"How can I add new users (and SSH keys) to my project/group?","text":"This can be done via the Lagoon API. You can find the steps documentation for this change in our GraphQL documentation.
"},{"location":"resources/faq/#can-an-environment-be-completely-deleted-to-roll-out-large-code-changes-to-my-project","title":"Can an environment be completely deleted to roll out large code changes to my project?","text":"Environments are fully built from scratch at each deploy, dropping the old database and files and pushing your code would result in a fresh clean build, Don\u2019t forget to re-sync!
It is possible to delete an environment via GraphQL. You can find the instructions in our GraphQL documentation.
"},{"location":"resources/faq/#how-do-i-get-my-new-environment-variable-to-show-up","title":"How do I get my new environment variable to show up?","text":"Once you've added a runtime environment variable to your production environment via GraphQL, then all you need to do a deploy in order to get your change to show up on your environment.
"},{"location":"resources/faq/#how-do-i-sftp-files-tofrom-my-lagoon-environment","title":"How do I SFTP files to/from my Lagoon environment?","text":"For cloud hosting customers, you can SFTP to your Lagoon environment by using the following information:
ssh.lagoon.amazeeio.cloudYour username is going to be the name of the environment you are connecting to, most commonly in the pattern PROJECTNAME-ENVIRONMENT.
You may also be interested in checking out our new Lagoon Sync tool, which you can read about here: https://github.com/uselagoon/lagoon-sync
Authentication also happens automatically via SSH Public & Private Key Authentication.
"},{"location":"resources/faq/#i-dont-want-to-use-lets-encrypt-i-have-an-ssl-certificate-i-would-like-to-install","title":"I don't want to use Let's Encrypt. I have an SSL certificate I would like to install","text":"We can definitely help with that. Once you have your own SSL certificate, feel free to submit a ticket or send us a message via chat and we will be more than happy to help! You will need to send us the following files:
Also, you will need to set the tls-acme option in .lagoon.yml to false.
Mounting an external volume would need to be handled completely inside of your containers, Lagoon does not provide a provision for this type of connection as part of the platform.
A developer can handle this by installing the necessary packages into the container (via the Dockerfile), and ensuring the volume mount is connected via a pre- or post-rollout task.
"},{"location":"resources/faq/#is-there-a-way-to-stop-a-lagoon-build","title":"Is there a way to stop a Lagoon build?","text":"If you have a build that has been running for a long time, and want to stop it, you will need to reach out to support. Currently, builds can only be stopped by users with admin access to the cluster.
"},{"location":"resources/faq/#we-installed-the-elasticsearchsolr-service-on-our-website-how-can-we-get-access-to-the-ui-port-92008983-from-a-browser","title":"We installed the Elasticsearch\\Solr service on our website. How can we get access to the UI (port 9200/8983) from a browser?","text":"We suggest only exposing web services (NGINX/Varnish/Node.js) in your deployed environments. Locally, you can get the ports mapped for these services by checking docker-compose ps, and then load http://localhost:<port> in your browser.
You can reach out to the team via Discord or email at uselagoon@amazee.io.
"},{"location":"resources/glossary/","title":"Glossary","text":"Term Definition Access Mode Controls how a persistent volume can be accessed. Active/Standby Active/Standby deployments, also known as blue/green deployments, are a way to seamlessly switch over your production content. Ansible An open-source suite of software tools that enables infrastructure as code. AWS Amazon Web Services AWS Glacier A secure and inexpensive S3 storage for long-term backup. BitBucket Git hosting owned by Atlassian, which integrates with their tools. Brew Homebrew is a package manager for OSX. CA A Certificate Authority is a trusted entity that issues Secure Sockets Layer (SSL) certificates. CDN Content Delivery Network - distributes content via caching CI Continuous Integration CIDR Classess Inter-Domain Routing - a method of assigning IP addresses CLI Command Line Interface Cluster A unified group of servers or VMs, distributed and managed together, which serves one entity to ensure high availability, load balancing, and scalability. CMS Content Management System Cron job The cron command-line utility is a job scheduler on Unix-like operating systems. Users who set up and maintain software environments use cron to schedule jobs, also known as cron jobs, to run periodically at fixed times, dates, or intervals. Composer A package manager DDoS Distributed Denial of Service DNS Domain Name System Docker A container engine using Linux features and automating application deployment. Docker Compose A tool for defining and running Docker applications via YAML files. Drupal Open-source Content Management System Drush A command line shell for Drupal. EC2 Amazon Elastic Compute Cloud Elasticsearch An open-source search engine. It provides a distributed, multi-tenant-capable full-text search engine with a web interface and schema-free JSON documents. Galera A generic synchronous multi-master replication library for transactional databases. Git A free and open-source distributed version control system. Git Hash/SHA A generated string that identifies each commit. Uses the SHA-1 algorithm GitHub A proprietary version control hosting company using Git. A subsidiary of Microsoft, it offers all of the distributed version control and source code management functionality of Git as well as additional features. GitLab A web-based Git repository manager with CI capabilities. GraphQL An open-source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data. Harbor An open source container image registry that secures images with role-based access control, scans images for vulnerabilities, and signs images as trusted. Helm A package manager for Kubernetes, it helps you manage Kubernetes applications. Helm Charts Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. HTTP HyperText Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. IAM AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. IDE An integrated development environment is a software application that provides comprehensive facilities to computer programmers for software development. An IDE normally consists of at least a source code editor, build automation tools, and a debugger. Ingress controller An Ingress controller is a specialized load balancer for Kubernetes (and other containerized) environments. IPTables A command line utility for configuring Linux kernel firewall. Jenkins An open-source automation server. k3s A highly available, certified Kubernetes distribution. k3d k3d is a lightweight wrapper to run k3s in Docker. k8s Numeronym for Kubernetes (K + 8 letters + s) k8up K8up is a backup operator that will handle storage and app backups on a k8s/OpenShift cluster. Kibana An open-source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. KinD Kubernetes in Docker - a tool for running local Kubernetes clusters using Docker container \u201cnodes\u201d. Kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI. kubectl The Kubernetes command-line tool which allows you to run commands against Kubernetes clusters. Kubernetes An open-source system for automating deployment, scaling, and management of containerized applications. Lagoon An open-source application delivery platform for Kubernetes. Lagoonize Configuration changes to allow your app to run on Lagoon. Lando A free, open source, cross-platform, local development environment and DevOps tool built on Docker. Laravel A free, open-source PHP web framework, following the model\u2013view\u2013controller (MVC) architectural pattern and based on Symfony. Let's Encrypt Aa free, automated, and open certificate authority (CA). MariaDB A community-developed, commercially supported fork of the MySQL relational database management system, intended to remain free and open-source software under the GNU General Public License. Master node A single node in the cluster on which a collection of processes which manage the cluster state are running. Microservice The practice of breaking up an application into a series of smaller, more specialized parts, each of which communicate with one another across common interfaces such as APIs and REST interfaces like HTTP MongoDB MongoDB is a cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with schema. Multi-Tenant A single instance of software runs on a server and serves multiple tenants - a tenant is a group of users who share common access with privileges to access the software instance. The software is designed to provide each tenant a share of the resources. MVC Model-view-controller - an architectural pattern that separates an application into three main logical components: the model, the view, and the controller. Each of these components are built to handle specific development aspects of an application. NGINX NGINX is a web server which can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. Node Single EC2 instance (AWS virtual machine) Node.js An open-source, cross-platform, JavaScript runtime environment that executes JavaScript code outside of a browser. OpenSearch A community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data. OpenShift Container application platform that brings Docker and Kubernetes to the enterprise. PHP PHP (Personal Home Page) is a general-purpose programming language originally designed for web development. PhpStorm A development tool (IDE) for PHP and web projects. Pod A group of containers that are deployed together on the same host. The basic unit that Kubernetes works with. PostgreSQL A free and open-source relational database management system emphasizing extensibility and technical standards compliance. Public/Private Key Public-key encryption is a cryptographic system that uses two keys -- a public key known to everyone and a private or secret key known only to the recipient of the message. Puppet An open-source software configuration management and deployment tool. PV PersistentVolume - a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. PVC Persistent Volume Claim - a request for storage by a user. Python Python is an open-source, interpreted, high-level, general-purpose programming language. RabbitMQ An open-source message-broker software. RBAC Role-Based Access Control RDS Relational Database Service Redis An open source, in-memory data store used as a database, cache, streaming engine, and message broker. Restic An open-source backup program. ROX Kubernetes access mode ReadOnlyMany - the volume can be mounted as read-only by many nodes. Ruby An interpreted, high-level, general-purpose programming language which supports multiple programming paradigms. It was designed with an emphasis on programming productivity and simplicity. In Ruby, everything is an object, including primitive data types. RWO Kubernetes access mode ReadWriteOnce - the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node. RWOP Kubernetes access mode ReadWriteOncePod - the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+. RWX Kubernetes access mode ReadWriteMany - the volume can be mounted as read-write by many nodes. SHA-1 Secure Hash Algorithm 1, a hash function which takes an input and produces a 160-bit hash value known as a message digest \u2013 typically rendered as 40 hexadecimal digits. It was designed by the United States National Security Agency, and is a U.S. Federal Information Processing Standard. Solr An open-source enterprise-search platform, written in Java. SSH Secure Socket Shell, a network protocol that provides administrators with a secure way to access a remote computer. SSL Secure Socket Layer Storage Classes A StorageClass provides a way for Kubernetes administrators to describe the \"classes\" of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators Symfony Symfony is a PHP web application framework and a set of reusable PHP components/libraries, Drupal 8 and up are based on Symfony. TCP Transmission Control Protocol, a standard that defines how to establish and maintain a network conversation through which application programs can exchange data. TLS Transport Layer Security Trivy A simple and comprehensive vulnerability scanner for containers, suitable for CI. TTL Time to live or hop limit is a mechanism that limits the lifespan or lifetime of data in a computer or network. Uptime Robot Uptime monitoring service. Varnish A powerful, open-source HTTP engine/reverse HTTP proxy that can speed up a website by caching (or storing) a copy of a webpage the first time a user visits. VM Virtual Machine Webhook A webhook is a way for an app like GitHub, GitLab, Bitbucket, etc, to provide other applications with immediate data and act upon something, like a pull request. YAML Yet Another Markup Language - YAML is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted."},{"location":"resources/tutorials-and-webinars/","title":"Tutorials, Webinars, and Videos","text":""},{"location":"resources/tutorials-and-webinars/#intro-to-lagoon-webinar","title":"Intro to Lagoon Webinar","text":"[Slides]
"},{"location":"resources/tutorials-and-webinars/#advance-lando-ing-with-lagoon","title":"Advance Lando-ing with Lagoon","text":""},{"location":"resources/tutorials-and-webinars/#webinar-lagoon-insights","title":"Webinar - Lagoon Insights","text":""},{"location":"resources/tutorials-and-webinars/#lagoon-deployment-demo","title":"Lagoon Deployment Demo","text":""},{"location":"resources/tutorials-and-webinars/#how-to-manage-multiple-drupal-sites-with-lagoon","title":"How to Manage Multiple Drupal Sites with Lagoon","text":"[Slides]
"},{"location":"resources/tutorials-and-webinars/#kubernetes-webinar-101","title":"Kubernetes Webinar 101","text":"[Slides]
"},{"location":"resources/tutorials-and-webinars/#kubernetes-webinar-102","title":"Kubernetes Webinar 102","text":"[Slides]
"},{"location":"resources/tutorials-and-webinars/#server-side-rendering-best-practices-how-we-run-decoupled-websites-with-110-million-hits-per-month","title":"Server-side Rendering Best Practices: How We Run Decoupled Websites with 110 Million Hits per Month","text":""},{"location":"resources/tutorials-and-webinars/#lagoon-opensource-docker-build-deployment-system-with-full-drupal-support","title":"Lagoon: OpenSource Docker Build & Deployment System with Full Drupal Support","text":""},{"location":"resources/tutorials-and-webinars/#how-do-i-fix-an-internal-server-error-in-kibana","title":"How do I fix an internal server error in Kibana?","text":""},{"location":"resources/tutorials-and-webinars/#how-do-i-add-a-new-route","title":"How do I add a new route?","text":""},{"location":"resources/tutorials-and-webinars/#how-do-i-check-the-status-of-a-build","title":"How do I check the status of a build?","text":""},{"location":"resources/tutorials-and-webinars/#how-do-i-add-a-redirect-in-lagoon","title":"How do I add a redirect in Lagoon?","text":""},{"location":"resources/tutorials-and-webinars/#how-do-i-download-a-database-dump","title":"How do I download a database dump?","text":""},{"location":"resources/tutorials-and-webinars/#how-do-i-add-a-cron-job","title":"How do I add a cron job?","text":""},{"location":"resources/tutorials-and-webinars/#deploying-web-applications-on-kubernetes-toby-bellwood-techweek21-talk","title":"Deploying web applications on Kubernetes - Toby Bellwood | Techweek21 Talk","text":""},{"location":"resources/tutorials-and-webinars/#dealing-with-unprecedented-scale-during-covid-19-sean-hamlin-techweek21-talk","title":"Dealing with unprecedented scale during Covid-19 - Sean Hamlin| Techweek21 Talk","text":""},{"location":"resources/tutorials-and-webinars/#silverstripe-from-local-to-live-on-lagoon-thom-toogood-techweek21-talk","title":"Silverstripe from local to live on Lagoon -Thom Toogood | Techweek21 Talk","text":""},{"location":"using-lagoon-advanced/active-standby/","title":"Active/Standby","text":""},{"location":"using-lagoon-advanced/active-standby/#configuration","title":"Configuration","text":"To change an existing project to support active/standby you'll need to configure some project settings with the Lagoon API.
productionEnviromment should be set to the branch name of the current active environment.standbyProductionEnvironment should be set to the branch name of the current environment that is in standby.mutation updateProject {\n updateProject(input:{\n id:1234\n patch:{\n productionEnvironment:\"production-brancha\"\n standbyProductionEnvironment:\"production-branchb\"\n }\n }){\n standbyProductionEnvironment\n name\n productionEnvironment\n }\n}\n.lagoon.yml - production_routes","text":"To configure a project for active/standby in the .lagoon.yml file, you'll need to configure the production_routes section with any routes you want to attach to the active environment, and any routes to the standby environment. During an active/standby switch, these routes will migrate between the two environments.
If you have two production environments, production-brancha and production-branchb, with the current active production environment as production-brancha then:
production_routes.active will direct you to production-brancha.production_routes.standby will direct you to production-branchb.During an active/standby switch, the routes will swap:
production_routes.active will direct you to production-branchb.production_routes.standby will direct you to production-brancha.production_routes:\nactive:\nroutes:\n- nginx:\n- example.com:\ntls-acme: 'false'\n- active.example.com:\ntls-acme: 'false'\nstandby:\nroutes:\n- nginx:\n- standby.example.com:\ntls-acme: 'false'\nInfo
Any routes that are under the section environments..routes will not be moved as part of active/standby. These routes will always be attached to the environment as defined. Ensure that if you do need a specific route to be migrated during an active/standby switch, that you remove them from the environments section and place them under the production_routes section specific to if it should be an active or standby route. See more about routes in .lagoon.yml.
To trigger the switching of environment routes, you can visit the standby environment in the Lagoon UI and click on the button labeled Switch Active/Standby environments. You will be prompted to confirm your action.
Once confirmed, it will take you to the tasks page where you can view the progress of the switch.
"},{"location":"using-lagoon-advanced/active-standby/#via-the-api","title":"via the API","text":"To trigger an event to switch the environments, run the following GraphQL mutation. This will tell Lagoon to begin the process.
Active Standby Switchmutation ActiveStandby {\n switchActiveStandby(\n input:{\n project:{\n name:\"drupal-example\"\n }\n }\n ){\n id\n remoteId\n }\n}\nA task is created in the current active environment tasks tab when a switch event is triggered. You can check the status of the switch here.
Using the remoteId from the switchActiveStandby mutation, we can also check the status of the task.
query getTask {\n taskByRemoteId(id: \"<remoteId>\") {\n id\n name\n created\n started\n completed\n status\n logs\n }\n}\ndrush aliases","text":"By default, projects will be created with the following aliases that will be available when active/standby is enabled on a project.
lagoon-productionlagoon-standbyThe lagoon-production alias will point to whichever site is defined as productionEnvironment, and lagoon-standby will always point to the site that is defined as standbyProductionEnvironment.
These aliases are configurable by updating the project. Be aware that changing them may require you to update any scripts that rely on them.
Update Drush Aliasesmutation updateProject {\n updateProject(input:{\n id:1234\n patch:{\n productionAlias:\"custom-lagoon-production-alias\"\n standbyAlias:\"custom-lagoon-standby-alias\"\n }\n }){\n productionAlias\n name\n standbyAlias\n }\n}\nYou need to decide which of these 2 branches are the one you want to go forward with as being the main environment and then ensure it is set as the active branch (e.g production-branchb).
.lagoon.yml file in this (now active) branch, move the routes from the production_routes.active.routes section into the environments.production-branchb section. This will mean that they are then attached to the production-branchb environment only..lagoon.yml file and re-deploy the production-branchb environment.production-brancha, you can delete it.production_routes from that branch .lagoon.yml too, just to prevent any confusion. The branch will remain as production type unless you delete and redeploy it (wiping all storage and databases, etc).production-branchb production environment, and all the other environments are development, update the project to remove the standbyProductionEnvironment from the project so that the active/standby labels on the environments go away.mutation updateProject {\n updateProject(input:{\n id:1234\n patch:{\n productionEnvironment:\"production-branchb\"\n standbyProductionEnvironment:\"\"\n }\n }){\n standbyProductionEnvironment\n name\n productionEnvironment\n }\n}\nWhen the active/standby trigger has been executed, the productionEnvironment and standbyProductionEnvironments will switch within the Lagoon API. Both environments are still classed as production environment types. We use the productionEnvironment to determine which one is labelled as active. For more information on the differences between environment types, read the documentation for environment types
query projectByName {\n projectByName(name:\"drupal-example\"){\n productionEnvironment\n standbyProductionEnvironment\n }\n}\nBefore switching environments:
Results of environment query{\n \"data\": {\n \"projectByName\": {\n \"productionEnvironment\": \"production-brancha\",\n \"standbyProductionEnvironment\": \"production-branchb\"\n }\n }\n}\nAfter switching environments:
Results of environment query{\n \"data\": {\n \"projectByName\": {\n \"productionEnvironment\": \"production-branchb\",\n \"standbyProductionEnvironment\": \"production-brancha\"\n }\n }\n}\nLagoon makes use of the k8up operator to provide backup functionality for both database data as well as containers' persistent storage volumes. This operator utilizes Restic to catalog these backups, which is typically connected to an AWS S3 bucket to provide secure, off-site storage for the generated backups.
"},{"location":"using-lagoon-advanced/backups/#production-environments","title":"Production Environments","text":""},{"location":"using-lagoon-advanced/backups/#backup-schedules","title":"Backup Schedules","text":"Backups of databases and containers' persistent storage volumes happens nightly within production environments by default.
If a different backup schedule for production backups is required, this can be specified at a project level via setting the \"Backup Schedule\" variables in the project's .lagoon.yml file.
"},{"location":"using-lagoon-advanced/backups/#backup-retention","title":"Backup Retention","text":"Production environment backups will be held according to the following schedule by default:
If a different retention period for production backups is required, this can be specified at a project level via setting the \"Backup Retention\" variables in the project's .lagoon.yml file.
"},{"location":"using-lagoon-advanced/backups/#development-environments","title":"Development Environments","text":"Backups of development environments are attempted nightly and are strictly a best effort service.
"},{"location":"using-lagoon-advanced/backups/#retrieving-backups","title":"Retrieving Backups","text":"Backups stored in Restic will be tracked within Lagoon, and can be recovered via the \"Backup\" tab for each environment in the Lagoon UI.
"},{"location":"using-lagoon-advanced/backups/#custom-backup-andor-restore-locations","title":"Custom Backup and/or Restore Locations","text":"Lagoon supports custom backup and restore locations via the use of the \"Custom Backup Settings\" and/or \"Custom Restore Settings\" variables stored in the Lagoon API for each project.
Danger
Proceed with caution: Setting these variables will override backup/restore storage locations that may be configured at a cluster level. Any misconfiguration will cause backup/restore failures.
"},{"location":"using-lagoon-advanced/base-images/","title":"Base Images","text":""},{"location":"using-lagoon-advanced/base-images/#what-is-a-base-image","title":"What is a base image?","text":"A base image is a Docker image that can be and is used by a project deployed on Lagoon. A base image provides a way to ensure that nothing is brought into the codebase/project from upstream that has not been audited. It also allows us to ensure that anything we might need on the deployed environment is available - from lower-level libraries to application-level themes and modules.
Base images save time and resources when you know what system is being deployed to - if shared packages are included in the base image, they don\u2019t have to be deployed to hundreds of sites individually.
"},{"location":"using-lagoon-advanced/base-images/#derived-images","title":"Derived images","text":"A derived image is one that extends a base image. For example, you might need to make several blog sites. You take our Drupal image, customize it to include all of the modules and themes you need for your blog sites, and deploy them all with that blog image. Templates are derived from base images.
All derived images should pull in the composer.json file (via repositories like Packagist, Satis, or GitHub) so that they are using the most recent versions of the base packages.
Further, the derived image includes a call to the script /build/pre_composer, which can be used by the base image to run scripts, updates, etc., downstream in the derived images. For instance, it should run by default when any package is updated or installed at the derived image, and the pre_composer script will then update the base image package.
Info
This document will talk about Drupal and Laravel base images as examples, as it was originally written for a client who uses those technologies in their Lagoon projects. It will be expanded to cover the contents of other base images, but none of the processes differ, no matter what the content of your base image.
Base images are managed with Composer and hosted in BitBucket, GitHub, or GitLab (whatever your team is using). Each base image has its own repository.
"},{"location":"using-lagoon-advanced/base-images/#metapackages","title":"Metapackages","text":"The metapackage is a Composer package that wraps several other components. These include, for example, the core files for Laravel or Drupal, along with any needed modules or themes. This way, you do not need to include Laravel or Drupal, etc., as a dependency in your project.
Here\u2019s an example from the composer.json in a Laravel base image:
\"require\": {\n \"amazeelabs/algm_laravel_baseimage\": \"*\"\n},\nWe only require this metapackage, which points to a GitHub repository.
"},{"location":"using-lagoon-advanced/base-images/#docker-composeyml","title":"docker-compose.yml","text":"Other pieces of your project are defined in docker-compose.yml. For example, if you have a Drupal project, you need the Drupal image, but you also need MariaDB, Solr, Redis, and Varnish. We have versions of these services optimized for Drupal, all of which are included in docker-compose.yml.
The Drupal base image contains the following contributed tools and modules, in addition to Drupal core:
The base images have provided the default values for the environment variables used by Laravel.
These are values for:
DB_CONNECTIONDB_HOSTDB_PORTDB_DATABASEDB_USERNAMEDB_PASSWORDREDIS_HOSTREDIS_PASSWORDREDIS_PORTEnsure that your config files (typically located in /config) make use of these by default.
If your project makes use of queues, you can make use of the artisan-worker service. It is a worker container, used for executing artisan queue:work. This is disabled by default - look at the comments in docker-compose.yml.
There are several parts to the process of building a base image. All of the major steps are represented in the Makefile. The Jenkinsfile contains a more stripped-down view. Taking a look at both files will give you a good understanding of what happens during this process. Most steps can be tested locally (this is important when building new versions of the base image). After you\u2019ve created and tested everything locally and pushed it up, the actual base image is built by Jenkins and pushed to Harbor.
"},{"location":"using-lagoon-advanced/base-images/#makefile-and-build-assumptions","title":"Makefile and build assumptions","text":"If you're planning on running locally, there are some minimum environment variables that need to be present to build at all.
"},{"location":"using-lagoon-advanced/base-images/#base-image-build-variables","title":"Base image build variables","text":"Variables injected into the base image build process and where to find them.
BUILD_NUMBER - This is injected by Jenkins automatically.GIT_BRANCH - This is provided by the Jenkins build process itself. Depends on the branch being built at the time (develop, main, etc.).DOCKER_REPO/DOCKER_HUB - This is defined inside the Jenkinsfile itself. It points to the Docker project and hub into which the resulting images will be pushed.DOCKER_USERNAME/DOCKER_PASSWORD - These are used to actually log into the Docker repository early in the build. These variables are stored inside of the Jenkins credentials. These are used in the Jenkinsfile itself and are not part of the Makefile. This means that if you\u2019re building base images outside of Jenkins (i.e. locally, to test, etc.) you have to run a docker login manually before running any of the make steps.In practice, this means that if you're running any of the make targets on your local machine, you'll want to ensure that these are available in the environment - even if this is just setting them when running make from the command line, as an example:
GIT_BRANCH=example_branch_name DOCKER_HUB=the_docker_hub_the_images_are_pushed_to DOCKER_REPO=your_docker_repo_here BUILD_NUMBER=<some_integer> make images_remove\nThe most important targets are the following:
images_build : Given the environment variables, this will build and tag the images for publication.images_publish : Pushes built images to a Docker repository.images_start : Will start the images for testing, etc.images_test: Runs basic tests against images.images_remove: Removes previously built images, given the build environment variables.There are several steps to the build process. Most of these are shared among the various base images. These mostly correspond to the Makefile target described above.
make images_build step is run now, which will:docker-compose build. This will produce several new Docker images from the current Git branch.make images_test target, which will differ depending on the images being tested. In most cases this is a very straightforward test to ensure that the images can be started and interacted with in some way (installing Drupal, listing files, etc.)images_publish) that will tag the images resulting from the Docker Build in Step 2 and push them to Harbor. This is described in more detail elsewhere in this guide.images_remove, which simply deletes the newly built images from the Docker host now that they are in Harbor.There are many reasons to release a new version of a base image. On Drupal or Laravel, Node.js, etc. images, it may be in order to upgrade or install a module/package for features or security. It may be about the underlying software that comes bundled in the container, such as updating the version of PHP or Node.js. It may be about updating the actual underlying images on which the base images are built.
The images that your project's base images are built on are the managed images maintained by the Lagoon team. We release updates to these underlying images on a monthly (or more fequent) basus. When these are updated, you need to build new versions of your own base images in order to incorporate the changes and upgrades bundled in the upstream images.
In this section we will demonstrate the process of updating and tagging a new release of the Drupal 8 base image. We will add a new module (ClamAV) to the base. We\u2019re demonstrating on Drupal because it has the most complex setup of the base images. The steps that are common to every base image are noted below.
"},{"location":"using-lagoon-advanced/base-images/#step-1-pull-down-the-base-image-locally","title":"Step 1 - Pull down the base image locally","text":"This is just pulling down the Git repository locally. In the case of the Drupal 8 base image. In this example, we're using Bitbucket, so we will run:
Clone Git repo.git clone ssh://git@bitbucket.biscrum.com:7999/webpro/drupal8_base_image.git\nInfo
What is demonstrated here is specific to the Drupal 8 base image. However, any changes (adding files, changing base Docker images, etc.) will be done in this step for all of the base images.
In our example, we are adding the ClamAV module to the Drupal 8 base image. This involves a few steps. The first is requiring the package so that it gets added to our composer.json file. This is done by running a composer require.
Here we run:
Install package with Composer require.composer require drupal/clamav\nWhen the Composer require process completes, the package should then appear in the composer.json file.
Here we open the composer.json file and take a look at the list of required packages, and check that the ClamAV package is listed, and see that it is there:
For any modules now added to the base image, we need to ensure that they\u2019re enabled on the template-based derived images. This is done by adding the module to the Lagoon Bundle module located at ./web/modules/lagoon/lagoon_bundle. Specifically, it requires you to add it as a dependency to the dependencies section of the lagoon_bundle.info.yml file. The Lagoon Bundle module is a utility module that exists only to help enforce dependencies across derived images.
Here we open web/modules/contrib/lagoon/lagoon_bundle/lagoon_bundle.info.yml and add clamav:clamav as a dependency:
Adding a dependency to this will ensure that whenever the Lagoon Bundle module is enabled on the derived image, its dependencies (in this case, the just-added ClamAV module) will also be enabled. This is enforced by a post-rollout script which enables lagoon_bundle on the derived images when they are rolled out.
This will depend on what you\u2019re testing. In the case of adding the ClamAV module, we want to ensure that in the base image, the module is downloaded, and that the Lagoon Bundle module enables ClamAV when it is enabled.
Here we check that the module is downloaded to /app/web/modules/contrib:
And then we check that when we enable the lagoon_bundle module, it enables clamav by running:
drush pm-enable lagoon_bundle -y\nWarning
You\u2019ll see that there is a JWT error in the container above. You can safely ignore this in the demonstration above - but, for background, you will see this error when there is no Lagoon environment for the site you\u2019re working on.
With our testing done, we can now tag and build the images.
"},{"location":"using-lagoon-advanced/base-images/#step-3-tagging-images","title":"Step 3 - Tagging images","text":"Images are versioned based on their Git tags - these should follow standard semantic versioning (semver) practices. All tags should have the structure vX.Y.Z where X, Y, and Z are integers (to be precise the X.Y.Z are themselves the semantic version - the vX.Y.Z is a tag). This is an assumption that is used to determine the image tags, so it must be adhered to.
In this example we will be tagging a new version of the Drupal 8 base image indicating that we have added ClamAV.
"},{"location":"using-lagoon-advanced/base-images/#here-we-demonstrate-how-to-tag-an-image","title":"Here we demonstrate how to tag an image","text":"We check that we have committed (but not pushed) our changes, just as you would do for any regular commit and push, using git log.
git tag.git tag -a v0.0.9 -m \u201cAdds clamAV to base.\u201dgit push --tags.git push.Danger
The tags must be pushed explicitly in their own step!
"},{"location":"using-lagoon-advanced/base-images/#how-git-tags-map-to-image-tags","title":"How Git tags map to image tags","text":"Danger
Depending on the build workflow, you will almost certainly push the changes via the develop branch before merging it into the main branch.
An important point to remember here is that the Jenkins base image build process will tag images based on the most recent commit\u2019s tag.
Images are tagged using the following rules, and images will be built for each of these that apply:
latest.development.Info
Generally you will have a trigger strategy set up here for automatic builds, but as that will differ based on your needs and setup, this explains how to build manually.
This will kick off the build process which, if successful, will push up the new images to Harbor.
If the build is not successful, you can click into the build itself and read the logs to understand where it failed.
As shown in the screenshot below from Harbor, the image we\u2019ve just built in Jenkins has been uploaded and tagged in Harbor, where it will now be scanned for any vulnerabilities. Since it was tagged as v0.0.9, an image with that tag is present, and because we built the main branch, the \u201clatest\u201d image has also been built. At this stage, the v0.0.9 and \u201clatest\u201d images are identical.
"},{"location":"using-lagoon-advanced/base-images/#acknowledgement","title":"Acknowledgement","text":"The base image structure draws heavily (and, in fact, is a fork of) Denpal. It is based on the original Drupal Composer Template, but includes everything necessary to run on Lagoon (either the local development environment or on hosted Lagoon).
"},{"location":"using-lagoon-advanced/blackfire/","title":"Blackfire","text":""},{"location":"using-lagoon-advanced/blackfire/#blackfire-variables","title":"Blackfire variables","text":"The Lagoon Base Images have support for Blackfire included in the PHP Images (see the PHP images).
In order to use Blackfire in Lagoon, these three environment variables need to be defined:
Environment Variable Default DescriptionBLACKFIRE_ENABLED (not set) Used to enable blackfire extension with setting variable to TRUE or true BLACKFIRE_SERVER_ID (not set) Set to Blackfire Server ID provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true BLACKFIRE_SERVER_TOKEN (not set) Set to Blackfire Server Token provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true"},{"location":"using-lagoon-advanced/blackfire/#local-usage-of-blackfire","title":"Local Usage of Blackfire","text":"For local usage of Blackfire with Lagoon Images, set the above environment variables for the PHP container. Here is an example for a Drupal application:
docker-compose.ymlservices:\n[[snip]]\nphp:\n[[snip]]\nenvironment:\n<< : *default-environment # loads the defined environment variables from the top\nBLACKFIRE_ENABLED: TRUE\nBLACKFIRE_SERVER_ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\nBLACKFIRE_SERVER_TOKEN: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nAfter restarting the containers, you should be able to profile via the Blackfire Browser Plugin or the Blackfire CLI.
"},{"location":"using-lagoon-advanced/blackfire/#remote-usage-of-blackfire","title":"Remote Usage of Blackfire","text":"In order to use Blackfire in deployed Lagoon environments the same enviornment variables need to be set, this time via one of the possibilities of adding environment variables to Lagoon. Important: Environment variables set in the docker-compose.yml for local development are not used by Lagoon in remote environments!
The Blackfire Agent running in the PHP containers outputs logs as normal container logs, which can be seen via docker-compose logs or via the Lagoon Logging Infrastructure for remote environments.
By default the Logs are set to Level 3 (info), via the environment variable BLACKFIRE_LOG_LEVEL the level can be increased to 4 (debug) to generate more debugging ouput.
Lagoon allows for the definition of custom tasks at environment, project, and group levels. This is presently accomplished through the GraphQL API and exposed in the UI.
"},{"location":"using-lagoon-advanced/custom-tasks/#defining-a-custom-task","title":"Defining a custom task","text":"When defining a task you need to determine a number of things.
"},{"location":"using-lagoon-advanced/custom-tasks/#which-task-do-you-want-to-run","title":"Which task do you want to run?","text":"In most cases, the custom task you will be running will be something that will be run in a shell on one of the containers in your application.
For instance, in a Node.js application, you may be interested in running a yarn audit in your node container. The command, in this case, would simply be yarn audit.
We have to define where this task will be run -- this means two things, first, which project or environment we'll be running the task in, and, second, which service.
Let's say that we'd like for our yarn audit task to be available to run in any environment in a specific project (let's say the project's ID is 42 for this example). We will therefore specify the project's ID when we create our task definition, as we will describe below.
The second question regards which environment we want to target with our task. When you set up your project, you specify several services in your docker-compose.yml. We use this service name to determine where the command is actually executed.
There are three levels of permissions to the task system corresponding to project roles. Guest, Developer, and Maintainer -- from most restrictive to least restrictive, with each role being able to invoke the tasks defined for the lower role (Developer can see Guest tasks, Maintainers can see all tasks).
"},{"location":"using-lagoon-advanced/custom-tasks/#defining-a-task","title":"Defining a task","text":"Tasks are defined by calling the addAdvancedTaskDefinition mutation. Importantly, this simply defines the task, it does not invoke it. It simply makes it avaliable to be run in an environment.
Schematically, the call looks like this
Define a new taskmutation addAdvancedTask {\n addAdvancedTaskDefinition(input:{\n name: string,\n confirmationText: string,\n type: [COMMAND|IMAGE],\n [project|environment]: int,\n description: string,\n service: string,\n command: string,\n advancedTaskDefinitionArguments: [\n {\n name: \"ENVIROMENT_VARIABLE_NAME\",\n displayName: \"Friendly Name For Variable\",\n type: [STRING | ENVIRONMENT_SOURCE_NAME | ENVIRONMENT_SOURCE_NAME_EXCLUDE_SELF]\n }\n ]\n }) {\n ... on AdvancedTaskDefinitionImage {\n id\n name\n description\n service\n image\n confirmationText\n advancedTaskDefinitionArguments {\n type\n range\n name\n displayName\n }\n ...\n }\n ... on AdvancedTaskDefinitionCommand {\n id\n name\n description\n service\n command\n advancedTaskDefinitionArguments {\n type\n range\n name\n displayName\n }\n ...\n }\n }\n}\nFields name and description are straightforward. They're simply the name and description of the task - these are used primarily in the UI.
The type field needs some explanation - for now, only platform admins are able to define IMAGE type commands - these allow for the running of specifically created task images as tasks, rather than targeting existing services. Most tasks, though, will be COMMAND types.
The [project|environment] set of fields will attach the task to either the project or environment (depending on the key you use), with the value being the id. In the case we're considering for our yarn audit we will specify we're targeting a project with an ID of 42.
We put the service we'd like to target with our task in the service field, and command is the actual command that we'd like to run.
In order to give more flexibility to the users invoking the tasks via the Lagoon UI, we support defining task arguments. These arguments are displayed as text boxes or drop downs and are required for the task to be invoked.
Here is an example of how we might set up two arguments.
Define task argumentsadvancedTaskDefinitionArguments: [\n {\n name: \"ENV_VAR_NAME_SOURCE\",\n displayName: \"Environment source\",\n type: ENVIRONMENT_SOURCE_NAME\n\n },\n {\n name: \"ENV_VAR_NAME_STRING\",\n displayName: \"Echo value\",\n type: STRING\n }\n ]\n })\nThis fragment shows both types of arguments the system currently supports. The first, ENV_VAR_NAME_SOURCE is an example of type ENVIRONMENT_SOURCE_NAME, which will present the user of the UI a dropdown of the different environments inside of a project. If we don't want to allow the task to be run on the invoking environment (say, if we want to import a database from another environment), we can restrict the environment list by using ENVIRONMENT_SOURCE_NAME_EXCLUDE_SELF. The second ENV_VAR_NAME_STRING is of type STRING and will present the user with a textbox to fill in.
The values that the user selects will be available as environment variables in the COMMAND type tasks when the task is run.
When the confirmationText field has text, it will be displayed with a confirmation modal in the UI before the user is able to run the task.
With the task now defined, the task should now show up in the tasks dropdown in the Lagoon UI.
We are also able to invoke it via the GraphQL api by using the invokeTask mutation.
mutation invokeTask {\n invokeRegisteredTask(advancedTaskDefinition: int, environment: int) {\n status\n }\n}\nNote that invokeTask will always invoke a task on a specific environment.
Let's now setup our yarn audit example.
mutation runYarnAudit {\n addAdvancedTaskDefinition(input:{\n name:\"Run yarn audit\",\n project: 42,\n type:COMMAND,\n permission:DEVELOPER,\n description: \"Runs a 'yarn audit'\",\n service:\"node\",\n command: \"yarn audit\"})\n {\n id\n }\n}\nThis, then, will define our task for our project (42). When we run this, we will get the ID of the task definition back (for argument's sake, let's say it's 9)
This task will now be available to run from the UI for anyone with the DEVELOPER or MAINTAINER role.
Danger
This is an alpha feature in Lagoon. The way DeployTarget Configurations work could change in future releases. If you decide to use this feature, you do so at your own risk.
DeployTarget configurations are a way to define how a project can deploy to multiple clusters. This feature is useful when you have two clusters, one which could be dedicated for running production workloads, and another that is used for running development workloads.
The configuration for these is not limited to just a production/development split though, so projects could perceivably target more than one specific cluster.
The basic idea of a DeployTarget configuration is that it is a way to easily define how a project can deploy across multiple clusters. It uses the existing methods of checking if a environment is valid
"},{"location":"using-lagoon-advanced/deploytarget-configs/#important-information","title":"Important Information","text":"Before going in to how to configure a project to leverage DeployTarget configurations, there are some things you need to know.
Environments now have two new fields available to them to identify which DeployTarget(Kubernetes or OpenShift) they have been created on.
kubernetesNamespacePatternkubernetesOnce an environment has been deployed to a specific DeployTarget, it will always deploy to this target, even if the DeployTarget configuration, or project configuration is modified.
By default, if no DeployTarget configurations are associated to a project, that project will continue to use the existing methods to determine which environments to deploy. These are the following fields used for this.
branchespullrequestskubernetesNamespacePatternkubernetesAs soon as any DeployTarget configurations are added to a project, then all future deployments for this project will use these configurations. What is defined in the project is ignored, and overwritten to inform users that DeployTarget configurations are in use.
DeployTarget configurations are weighted, which means that a DeployTarget configuration with a larger weight is prioritized over one with lower weight.
1. The order in which they are returned by the query is the order they are used to determine where an environment should be deployed.
Active/Standby environments can only be deployed to the same cluster, so your DeployTarget configuration must be able to deploy both those environments to the same target.
Projects that leverage the promote feature of Lagoon must be aware that DeployTarget configurations are ignored for the destination environment.
source environment is on, your DeployTarget configuration MUST be configured correctly for this source environment.source and destination environment in the same DeployTarget configuration branch regex.To configure a project to use DeployTarget configurations, the first step is to add a configuration to a project.
The following GraphQL mutation can be used, this particular example will add a DeployTarget configuration to the project with the project ID 1. It will allow only the branches that match the name main to be deployed, and pullrequests is set to false. This means no other branches will be able to deploy to this particular target, and no pull requests will be deployed to this particular target. The deployTarget is ID 1, this could be a Kubernetes cluster in a specific region, or designated for a specific type of workload (production or development).
mutation addDeployTargetConfig{\n addDeployTargetConfig(input:{\n project: 1\n branches: \"main\"\n pullrequests: \"false\"\n deployTarget: 1\n weight: 1\n }){\n id\n weight\n branches\n pullrequests\n deployTargetProjectPattern\n deployTarget{\n name\n id\n }\n project{\n name\n }\n }\n}\nInfo
deployTarget is an alias the Kubernetes or OpenShift ID in the Lagoon API
It is also possible to configure multiple DeployTarget configurations.
The following GraphQL mutation can be used, this particular example will add a DeployTarget configuration to the same project as above.
It will allow only the branches that regex match with ^feature/|^(dev|test|develop)$ to be deployed, and pullrequests is set to true so all pull requests will reach this target.
The targeted cluster in this example is ID 2, which is a completely different Kubernetes cluster to what was defined above for the main branch.
mutation addDeployTargetConfig{\n addDeployTargetConfig(input:{\n project: 1\n branches: \"^feature/|^(dev|test|develop)$\"\n pullrequests: \"true\"\n deployTarget: 2\n weight: 1\n }){\n id\n weight\n branches\n pullrequests\n deployTargetProjectPattern\n deployTarget{\n name\n id\n }\n project{\n name\n }\n }\n}\nOnce these have been added to a project, you can return all the DeployTarget configurations for a project using the following query
Get DeployTargetsquery deployTargetConfigsByProjectId{\n deployTargetConfigsByProjectId(project:1){\n id\n weight\n branches\n pullrequests\n deployTargetProjectPattern\n deployTarget{\n name\n id\n }\n project{\n name\n }\n }\n}\n# result:\n{\n \"data\": {\n \"deployTargetConfigsByProjectId\": [\n {\n \"id\": 1,\n \"weight\": 1,\n \"branches\": \"main\",\n \"pullrequests\": \"false\",\n \"deployTargetProjectPattern\": null,\n \"deployTarget\": {\n \"name\": \"production-cluster\",\n \"id\": 1\n },\n \"project\": {\n \"name\": \"my-project\"\n }\n },\n {\n \"id\": 2,\n \"weight\": 1,\n \"branches\": \"^feature/|^(dev|test|develop)$\",\n \"pullrequests\": \"true\",\n \"deployTargetProjectPattern\": null,\n \"deployTarget\": {\n \"name\": \"development-cluster\",\n \"id\": 2\n },\n \"project\": {\n \"name\": \"my-project\"\n }\n }\n ]\n }\n}\nLagoon can utilize the Aergia controller, (installed in the lagoon-remote) to automatically idle environments if they have been unused for a defined period of time. This is done in order to reduce the load on the Kubernetes clusters and improve the overall performance of production environments and development environments that are actually in use.
The environment idler has many different configuration capabilities. Here are the defaults of a standard Lagoon installation (these could be quite different in your Lagoon, check with your Lagoon administrator!)
Aergia will automatically un-idle an environment as soon as it is visited, therefore just visiting any URL of the environment will start the environment. Likewise, initiating an SSH session to the environment will also restart the services.
The un-idling will take a couple of seconds, as the Kubernetes cluster needs to start all containers again. During this time there will be waiting screen shown to the visitor that their environment is currently started.
"},{"location":"using-lagoon-advanced/environment-idling/#can-i-disable-prevent-the-idler-from-idling-my-environment","title":"Can I disable / prevent the Idler from idling my environment?","text":"Yes, there is a field autoIdle on the project (impacts all environments) and environment (if you need to target just one environment), as to whether idling is allowed to take place. A value of 1 indicates the project/environment is eligible for idling. If the project is set to 0 the environments will never be idled, even if the environment is set to 0 The default is always 1(idling is enabled).
Talk to your Lagoon administrator if you are unsure how to set these project/environment fields.
"},{"location":"using-lagoon-advanced/environment-types/","title":"Environment Types","text":"Lagoon currently differentiates between two different environment types: production and development.
When setting up your project via the Lagoon GraphQL API, you can define a productionEnvironment. On every deployment Lagoon executes, it checks if the current environment name matches what is defined in productionEnvironment. If it does, Lagoon will mark this environment as the production environment. This happens in two locations:
LAGOON_ENVIRONMENT_TYPE in every container.But that's it. Lagoon itself handles development and production environments in exactly the same way (in the end we want as few differences of the environments as possible - that's the beauty of Lagoon).
There are a couple of things that will use this information:
development environments are idled after 4 hours with no hits (don't worry, they wake up automatically). It is also possible for your Lagoon administrator to disable auto-idling on a per-environment basis, just ask!settings.php files load additional settings files for development.settings.php and production.settings.php so you can define settings and configurations different per environment type.productionEnvironment in the API or use the secret forceDeleteProductionEnvironment: true POST payload for the REST API.It is common to store API tokens or credentials for applications in environment variables.
Following best practices, those credentials are different per environment. We allow each environment to use a separate set of environment variables defined in environment variables or environment files.
As there can be environment variables defined in either the Dockerfile or during runtime (via API environment variables), we have a hierarchy of environment variables: environment variables defined in lower numbers are stronger.
ENV command)..lagoon.env.$LAGOON_GIT_BRANCH or .lagoon.env.$LAGOON_GIT_SAFE_BRANCH (if the file exists and where $LAGOON_GIT_BRANCH $LAGOON_GIT_SAFE_BRANCH are the name and safe name of the branch this Docker image has been built for), use this for overwriting variables for only specific branches..lagoon.env (if it exists), use this for overwriting variables for all branches..env..env.defaults..lagoon.env.$LAGOON_GIT_BRANCH, .lagoon.env.$LAGOON_GIT_SAFE_BRANCH, .env, and .env.defaults are all sourced by the individual containers themselves as part of running their entrypoint scripts. They are not read by Lagoon, but by the containers ENTRYPOINT scripts, which look for them in the containers working directory. If environment variables don't appear as expected, check if your container has a WORKDIR setting that points to somewhere else.
We suggest using the Lagoon API environment variable system for variables that you don't want to keep in your Git repository (like secrets or API keys), as they could be compromised by somebody having them on their local development environment or on the internet, etc.
The Lagoon API allows you to define project-wide or environment-specific variables. Additionally, they can be defined for a scope-only build-time or runtime. They are all created via the Lagoon GraphQL API. Read more on how to use the GraphQL API in our GraphQL API documentation.
"},{"location":"using-lagoon-advanced/environment-variables/#runtime-environment-variables-lagoon-api","title":"Runtime Environment Variables (Lagoon API)","text":"Runtime environment variables are automatically made available in all containers, but they are only added or updated after an environment has been re-deployed.
This defines a project wide runtime variable (available in all environments) for the project with ID 463:
mutation addRuntimeEnv {\n addEnvVariable(\n input:{\n type:PROJECT,\n typeId:463,\n scope:RUNTIME,\n name:\"MYVARIABLENAME\",\n value:\"MyVariableValue\"\n }\n ) {\n id\n }\n}\nThis defines a environment ID 546 specific runtime variable (available only in that specific environment):
mutation addRuntimeEnv {\n addEnvVariable(\n input:{\n type:ENVIRONMENT,\n typeId:546,\n scope:RUNTIME,\n name:\"MYVARIABLENAME\",\n value:\"MyVariableValue\"\n }\n ) {\n id\n }\n}\nBuild-time environment variables are only available during a build and need to be consumed in Dockerfiles via:
Using build-time environment variables
ARG MYVARIABLENAME\nARG will go after the FROM. Read the docker documentation about ARG and FROM. This defines a project-wide build-time variable (available in all environments) for the project with ID 463:
mutation addBuildtimeEnv {\n addEnvVariable(\n input:{\n type:PROJECT,\n typeId:463,\n scope:BUILD,\n name:\"MYVARIABLENAME\",\n value:\"MyVariableValue\"}\n ) {\n id\n }\n}\nThis defines an environment ID 546specific build-time variable (available only in that specific environment):
mutation addBuildtimeEnv {\n addEnvVariable(input:{type:ENVIRONMENT, typeId:546, scope:BUILD, name:\"MYVARIABLENAME\", value:\"MyVariableValue\"}) {\n id\n }\n}\nContainer registry environment variables are only available during a build and are used when attempting to log in to a private registry. They are used to store the password for the user defined in Specials \u00bb container-registries. They can be applied at the project or environment level.
This defines a project-wide container registry variable (available in all environments) for the project with ID 463:
mutation addContainerRegistryEnv {\n addEnvVariable(\n input:{\n type:PROJECT,\n typeId:463,\n scope:CONTAINER_REGISTRY,\n name:\"MY_OWN_REGISTRY_PASSWORD\",\n value:\"MySecretPassword\"})\n ) {\n id\n }\n}\nThis defines a environment ID 546 specific container registry variable (available only in that specific environment):
mutation addContainerRegistryEnv {\n addEnvVariable(\n input:{\n type:ENVIRONMENT,\n typeId:546,\n scope:CONTAINER_REGISTRY,\n name:\"MY_OWN_REGISTRY_PASSWORD\",\n value:\"MySecretPassword\"}\n ) {\n id\n }\n}\nIf you have environment variables that can safely be saved within a Git repository, we suggest adding them directly into the Git repository in an environment file. These variables will also be available within local development environments and are therefore more portable.
The syntax in the environment files is as following:
myenvironment.envMYVARIABLENAME=\"MyVariableValue\"\nMVARIABLENUMBER=4242\nDB_USER=$DB_USERNAME # Redefine DB_USER with the value of DB_USERNAME e.g. if your application expects another variable name for the Lagoon-provided variables.\n.lagoon.env.$BRANCHNAME","text":"If you want to define environment variables different per environment you can create a .lagoon.env.$BRANCHNAME e.g. for the main branch .lagoon.env.main. This helps you keep environment variables apart between environments.
.env and .env.defaults","text":".env and .env.defaults will act as the default values for environment variables if none other is defined. For example, as default environment variables for pull request environments (see Workflows).
PHP_ERROR_REPORTING","text":"This variable, if set, will define the logging level you would like PHP to use. If not supplied, it will be set dynamically based on whether this is a production or development environment.
On production environments, this value defaults to E_ALL & ~E_DEPRECATED & ~E_STRICT & ~E_NOTICE.
On development environments, this value defaults to E_ALL & ~E_DEPRECATED & ~E_STRICT.
Lagoon supports custom backup locations and credentials for any project when all four of the following variables are set as BUILD type variables. The environment variables need to be set at the project level (not per environment), and requires a Lagoon deployment after setting them (for every environment).
Please note that any use of these variables means that all environment and database backups created and managed by Lagoon will be stored using these credentials, meaning that any interruption of these credentials' may lead to failed or inaccessible backups.
Environment variable name PurposeLAGOON_BAAS_CUSTOM_BACKUP_ENDPOINT Specify the S3 compatible endpoint where any Lagoon initiated backups should be stored. An example for S3 Sydney would be: https://s3.ap-southeast-2.amazonaws.com. LAGOON_BAAS_CUSTOM_BACKUP_BUCKET Specify the bucket name where any Lagoon initiated backups should be stored.An example custom setting would be: example-restore-bucket. LAGOON_BAAS_CUSTOM_BACKUP_ACCESS_KEY Specify the access key Lagoon should use to access the custom backup bucket. An example custom setting would be: AKIAIOSFODNN7EXAMPLE. LAGOON_BAAS_CUSTOM_BACKUP_SECRET_KEY Specify the secret key Lagoon should use to access the custom backup bucket. An example custom setting would be: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. No public access is needed on the S3 bucket and can be made entirely private.
Lagoon will automatically prune the files in these S3 buckets, so no object retention policy is needed at the bucket level.
"},{"location":"using-lagoon-advanced/environment-variables/#custom-restore-location","title":"Custom Restore Location","text":"Lagoon supports custom restore locations and credentials for any project when all four of the following variables are set as BUILD type environment variables. The environment variables need to be set at the project level (not per environment), and requires a Lagoon deployment after setting them (for every environment).
Please note that any use of these variables means that all environment and database snapshots restored by Lagoon will be stored using these credentials. This means that any interruption of these credentials' access may lead to failed or inaccessible restored files.
Environment variable name PurposeLAGOON_BAAS_CUSTOM_RESTORE_ENDPOINT Specify the S3 compatible endpoint where any Lagoon initiated restores should be stored. An example for S3 Sydney would be: https://s3.ap-southeast-2.amazonaws.com. LAGOON_BAAS_CUSTOM_RESTORE_BUCKET Specify the bucket name where any Lagoon initiated restores should be stored.An example custom setting would be: example-restore-bucket. LAGOON_BAAS_CUSTOM_RESTORE_ACCESS_KEY Specify the access key Lagoon should use to access the custom restore bucket. An example custom setting would be: AKIAIOSFODNN7EXAMPLE. LAGOON_BAAS_CUSTOM_RESTORE_SECRET_KEY Specify the secret key Lagoon should use to access the custom restore bucket. An example custom setting would be: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. The S3 bucket must have public access enabled, as Lagoon will create presigned URLs for the objects inside the bucket as needed.
An example AWS IAM policy that you can create to allow access to just the S3 bucket example-restore-bucket is:
{\n\"Version\": \"2012-10-17\",\n\"Statement\": [\n{\n\"Effect\": \"Allow\",\n\"Action\": [\n\"s3:GetBucketLocation\",\n\"s3:ListBucket\"\n],\n\"Resource\": [\n\"arn:aws:s3:::example-restore-bucket\"\n]\n},\n{\n\"Effect\": \"Allow\",\n\"Action\": [\n\"s3:PutObject\",\n\"s3:GetObject\",\n\"s3:GetObjectVersion\",\n\"s3:GetBucketLocation\",\n\"s3:PutObjectAcl\"\n],\n\"Resource\": [\n\"arn:aws:s3:::example-restore-bucket/*\"\n]\n}\n]\n}\nFor increased security and reduced storage costs you can opt into removing restored backups after a set lifetime (e.g. 7 days). Lagoon caters for this scenario gracefully and will re-create any restored snapshots as needed.
"},{"location":"using-lagoon-advanced/graphql/","title":"GraphQL","text":""},{"location":"using-lagoon-advanced/graphql/#connect-to-graphql-api","title":"Connect to GraphQL API","text":"API interactions in Lagoon are done via GraphQL. In order to authenticate to the API, you need a JWT (JSON Web Token), which will authenticate you against the API via your SSH public key.
To generate this token, use the remote shell via the token command:
ssh -p [PORT] -t lagoon@[HOST] token\nExample for amazee.io:
Get amazee.io tokenssh -p 32222 -t lagoon@ssh.lagoon.amazeeio.cloud token\nThis will return a long string, which is the JWT token.
We also need the URL of the API endpoint. Ask your Lagoon administrator for this.
On amazee.io this is https://api.lagoon.amazeeio.cloud/graphql.
Now we need a GraphQL client! Technically this is just HTTP, but we suggest GraphiQL. It has a nice UI that allows you to write GraphQL requests with autocomplete. Download, install and start it. [GraphiQL App]
Enter the API endpoint URL. Then click on \"Edit HTTP Headers\" and add a new Header:
AuthorizationBearer [jwt token] (make sure that the JWT token has no spaces, that won't work)Close the HTTP Header overlay (press ESC) and now you are ready to make your first GraphQL Request!
Enter this on the left window:
Get all projectsquery whatIsThere {\n allProjects {\n id\n gitUrl\n name\n branches\n pullrequests\n productionEnvironment\n environments {\n name\n environmentType\n }\n }\n}\nAnd press the \u25b6\ufe0f button (or press CTRL+ENTER).
If all went well, you should see your first GraphQL response.
"},{"location":"using-lagoon-advanced/graphql/#mutations","title":"Mutations","text":"The Lagoon GraphQL API can not only display objects and create objects, but it also has the capability to update existing objects. All of Lagoon's GraphQL uses best practices.
Mutation queries in GraphQL modify the data in the data store, and return a value. They can be used to insert, update, and delete data. Mutations are defined as a part of the schema.
Update the branches to deploy within a project:
Update deploy branchesmutation editProjectBranches {\n updateProject(input:{id:109, patch:{branches:\"^(prod|stage|dev|update)$\"}}) {\n id\n }\n}\nUpdate the production environment within a project:
Warning
This requires a redeploy in order for all changes to be reflected in the containers.
Update production environmentmutation editProjectProductionEnvironment {\n updateProject(input:{id:109, patch:{productionEnvironment:\"prod\"}}) {\n id\n }\n}\nYou can also combine multiple changes into a single query:
Multiple changesmutation editProjectProductionEnvironmentAndBranches {\n updateProject(input:{id:109, patch:{productionEnvironment:\"prod\", branches:\"^(prod|stage|dev|update)$\"}}) {\n id\n }\n}\nNode.js has integrated web server capabilities. Plus, with Express, these can be extended even more.
Unfortunately, Node.js does not handle shutting itself down very nicely out of the box. This causes many issues with containerized systems. The biggest issue is that when a Node.js container is told to shut down, it will immediately kill all active connections, and does not allow them to stop gracefully.
This part explains how you can teach Node.js to behave like a real web server: finishing active requests and then gracefully shutting down.
As an example we use a no-frills Node.js server with Express:
app.jsconst express = require('express');\nconst app = express();\n// Adds a 5 second delay for all requests.\napp.use((req, res, next) => setTimeout(next, 5000));\napp.get('/', function (req, res) {\nres.send(\"Hello World\");\n})\nconst server = app.listen(3000, function () {\nconsole.log('Example app listening on port 3000!');\n})\nThis will just show \"Hello World\" in when the web server is visited at localhost:3000. Note the 5 second delay in the response in order to simulate a request that takes some computing time.
If we run the above example and stop the Node.js process while the request is handled (within the 5 seconds), we will see that the Node.js server immediately kills the connection, and our browser will show an error.
To explain to our Node.js server that it should wait for all the requests to be finished before actually stopping itself, we add the following code:
Graceful Shutdownconst startGracefulShutdown = () => {\nconsole.log('Starting shutdown of express...');\nserver.close(function () {\nconsole.log('Express shut down.');\n});\n}\nprocess.on('SIGTERM', startGracefulShutdown);\nprocess.on('SIGINT', startGracefulShutdown);\nThis basically calls server.close(), which will instruct the Node.js HTTP server to:
It will do this on SIGINT (when you press CTRL + C) or on SIGTERM (the standard signal for a process to terminate).
With this small addition, our Node.js will wait until all requests are finished, and then stop itself.
If we were not running Node.js in a containerized environment, we would probably want to include some additional code that actually kills the Node.js server after a couple of seconds, as it is technically possible that some requests are either taking very long or are never stopped. Because it is running in a containerized system, if the container is not stopped, Docker and Kubernetes will run a SIGKILL after a couple of seconds (usually 30) which cannot be handled by the process itself, so this is not a concern for us.
If we only implemented Part A, we would have a good experience. In the real world, many Node.js systems are built with Yarn or NPM, which provide not only package management systems to Node.js, but also script management.
With these script functionalities, we simplify the start of our application. We can see many package.json files that look like:
{\n\"name\": \"node\",\n\"version\": \"1.0.0\",\n\"main\": \"index.js\",\n\"license\": \"MIT\",\n\"dependencies\": {\n\"express\": \"^4.15.3\"\n},\n\"scripts\": {\n\"start\": \"node index.js\"\n}\n}\nand with the defined scripts section we can run our application just with:
yarn start\nor
Start applicationnpm start\nThis is nice and makes the life of developers easier. So we also end up using the same within Dockerfiles:
.dockerfileCMD [\"yarn\", \"start\"]\nUnfortunately there is a big problem with this:
If yarn or npm get a SIGINT or SIGTERM signal, they correctly forward the signal to spawned child process (in this case node index.js). However, it does not wait for the child processes to stop. Instead, yarn/npm immediately stop themselves. This signals to Docker/Kubernetes that the container is finished and Docker/Kubernetes will kill all children processes immediately. There are issues open for Yarn and NPM but unfortunately they are not solved yet.
The solution for the problem is to not use Yarn or NPM to start your application and instead use node directly:
CMD [\"node\", \"index.js\"]\nThis allows Node.js to properly terminate and Docker/Kubernetes will wait for Node.js to be finished.
"},{"location":"using-lagoon-advanced/private-repositories/","title":"Private Repositories","text":"ARG LAGOON_SSH_PRIVATE_KEY to your dockerfile (before the step of the build process that needs the SSH key).RUN /lagoon/entrypoints/05-ssh-key.sh to your dockerfile (before the step of the build process that needs the SSH key).RUN /lagoon/entrypoints/05-ssh-key.sh && composer install && rm /home/.ssh/key\nWhen a Lagoon project is created, by default an associated SSH \"project key\" is generated and the private key made available inside the CLI pods of the project. A service account default-user@project is also created and given MAINTAINER access to the project. The SSH \"project key\" is attached to that default-user@project.
The result of this is that from inside the CLI pod of any environment it is possible to SSH to any other environment within the same project. This access is used for running tasks from the command line such as synchronizing databases between environments (e.g. drush sql-sync).
There is more information on the MAINTAINER role available in the RBAC documentation.
It is possible to specify an SSH private key when creating a project, but this is not recommended as it has security implications.
"},{"location":"using-lagoon-advanced/service-types/","title":"Service Types","text":"The below lists all service types that can be defined via lagoon.type within a docker-compose.yml file.
Warning
Once a lagoon.type is defined and the environment is deployed, changing it to a different type is not supported and could result in a broken environment.
basic","text":"Basic container, good to use for most applications that don't have an existing template. No persistent storage. The port can be changed using a label. If an autogenerated route is not required (e.g. for an internal-facing service, set lagoon.autogeneratedroute: false in the docker-compose.yml)
3000 3000 Yes No lagoon.service.port, lagoon.autogeneratedroute"},{"location":"using-lagoon-advanced/service-types/#basic-persistent","title":"basic-persistent","text":"Like basic. Will also generate persistent storage, defines mount location via lagoon.persistent.
3000 3000 Yes Yes lagoon.service.port, lagoon.autogeneratedroute, lagoon.persistent, lagoon.persistent.name, lagoon.persistent.size, lagoon.persistent.class"},{"location":"using-lagoon-advanced/service-types/#cli","title":"cli","text":"Use for any kind of CLI container (like PHP, Node.js, etc). Automatically gets the customer SSH private key that is mounted in /var/run/secrets/lagoon/sshkey/ssh-privatekey.
cli-persistent","text":"Like cli, expects lagoon.persistent.name to be given the name of a service that has persistent storage, which will be mounted under defined lagoon.persistent label. Does NOT generate its own persistent storage, only used to mount another service's persistent storage.
lagoon.persistent.name, lagoon.persistent"},{"location":"using-lagoon-advanced/service-types/#elasticsearch","title":"elasticsearch","text":"Elasticsearch container, will auto-generate persistent storage under /usr/share/elasticsearch/data.
localhost:9200/_cluster/health?local=true 9200 No Yes lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#kibana","title":"kibana","text":"Kibana container.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter TCP connection on5601 5601 Yes No -"},{"location":"using-lagoon-advanced/service-types/#logstash","title":"logstash","text":"Logstash container.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter TCP connection on9600 9600 No No -"},{"location":"using-lagoon-advanced/service-types/#mariadb","title":"mariadb","text":"A meta-service which will tell Lagoon to automatically decide between mariadb-single and mariadb-dbaas.
mariadb-single","text":"MariaDB container. Creates cron job for backups running every 24h executing /lagoon/mysql-backup.sh 127.0.0.1.
3306 3306 No Yes lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#mariadb-dbaas","title":"mariadb-dbaas","text":"Uses a shared MariaDB server via the DBaaS Operator.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter Not Needed3306 No - -"},{"location":"using-lagoon-advanced/service-types/#mongo","title":"mongo","text":"A meta-service which will tell Lagoon to automatically decide between mongo-single and mongo-dbaas.
mongo-single","text":"MongoDB container, will generate persistent storage of min 1GB mounted at /data/db.
27017 27017 No Yes lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#mongo-dbaas","title":"mongo-dbaas","text":"Uses a shared MongoDB server via the DBaaS Operator.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter Not Needed27017 No - -"},{"location":"using-lagoon-advanced/service-types/#nginx","title":"nginx","text":"NGINX container. No persistent storage.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameterlocalhost:50000/nginx_status 8080 Yes No lagoon.autogeneratedroute"},{"location":"using-lagoon-advanced/service-types/#nginx-php","title":"nginx-php","text":"Like nginx, but additionally a php container.
localhost:50000/nginx_status, PHP: /usr/sbin/check_fcgi 8080 Yes No lagoon.autogeneratedroute"},{"location":"using-lagoon-advanced/service-types/#nginx-php-persistent","title":"nginx-php-persistent","text":"Like nginx-php. Will generate persistent storage, defines mount location via lagoon.persistent.
localhost:50000/nginx_status, PHP: /usr/sbin/check_fcgi http on 8080 Yes Yes lagoon.autogeneratedroute, lagoon.persistent, lagoon.persistent.name, lagoon.persistent.size, lagoon.persistent.class"},{"location":"using-lagoon-advanced/service-types/#node","title":"node","text":"Node.js container. No persistent storage.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter TCP connection on3000 3000 Yes No lagoon.autogeneratedroute"},{"location":"using-lagoon-advanced/service-types/#node-persistent","title":"node-persistent","text":"Like node. Will generate persistent storage, defines mount location via lagoon.persistent.
3000 3000 Yes Yes lagoon.autogeneratedroute, lagoon.persistent, lagoon.persistent.name, lagoon.persistent.size, lagoon.persistent.class"},{"location":"using-lagoon-advanced/service-types/#none","title":"none","text":"Instructs Lagoon to completely ignore this service.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter - - - - -"},{"location":"using-lagoon-advanced/service-types/#opensearch","title":"opensearch","text":"OpenSearch container, will auto-generate persistent storage under /usr/share/opensearch/data.
localhost:9200/_cluster/health?local=true 9200 No Yes lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#postgres","title":"postgres","text":"A meta-service which will tell Lagoon to automatically decide between postgres-single and postgres-dbaas.
postgres-single","text":"Postgres container. Creates cron job for backups running every 24h executing /lagoon/postgres-backup.sh localhost.
5432 5432 No Yes lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#postgres-dbaas","title":"postgres-dbaas","text":"Uses a shared PostgreSQL server via the DBaaS Operator.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter Not Needed5432 No - -"},{"location":"using-lagoon-advanced/service-types/#python","title":"python","text":"Python container. No persistent storage.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter HTTP connection on8800 8800 Yes No lagoon.autogeneratedroute"},{"location":"using-lagoon-advanced/service-types/#python-persistent","title":"python-persistent","text":"Python container. With persistent storage.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter HTTP connection on8800 8800 Yes Yes lagoon.autogeneratedroute"},{"location":"using-lagoon-advanced/service-types/#redis","title":"redis","text":"Redis container.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter TCP connection on6379 6379 No No -"},{"location":"using-lagoon-advanced/service-types/#redis-persistent","title":"redis-persistent","text":"Redis container with auto-generated persistent storage mounted under /data.
6379 6379 No Yes lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#solr","title":"solr","text":"Solr container with auto-generated persistent storage mounted under /var/solr.
8983 8983 No Yes lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#varnish","title":"varnish","text":"Varnish container.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter HTTP requestlocalhost:8080/varnish_status 8080 Yes No lagoon.autogeneratedroute"},{"location":"using-lagoon-advanced/service-types/#varnish-persistent","title":"varnish-persistent","text":"Varnish container with auto-generated persistent storage mounted under /var/cache/varnish.
localhost:8080/varnish_status 8080 Yes Yes lagoon.autogeneratedroute, lagoon.persistent.size"},{"location":"using-lagoon-advanced/service-types/#worker","title":"worker","text":"Use for any kind of worker container (like queue workers, etc.) where there is no exposed service port.
Healthcheck Exposed Ports Auto Generated Routes Storage Additional customization parameter - No No No -"},{"location":"using-lagoon-advanced/service-types/#worker-persistent","title":"worker-persistent","text":"Like worker, expects lagoon.persistent.name to be given the name of a service that has persistent storage, which will be mounted under defined lagoon.persistent label. Does NOT generate its own persistent storage, only used to mount another service's persistent storage.
lagoon.persistent.name, lagoon.persistent"},{"location":"using-lagoon-advanced/setting-up-xdebug-with-lagoon/","title":"Setting up Xdebug with Lagoon","text":""},{"location":"using-lagoon-advanced/setting-up-xdebug-with-lagoon/#enable-xdebug-extension-in-the-containers","title":"Enable Xdebug extension in the containers","text":"The Lagoon base images are pre-configured with Xdebug but, for performance reasons, the extension is not loaded by default. To enable the extension, the XDEBUG_ENABLE environment variable must be set to true:
docker-compose.yml file, the environment variable already exists. Uncomment these lines.The default Xdebug configuration requires a \"trigger\" to activate the extension and start a session. You can view the complete documentation for activating the debugger but the most straightforward instructions are below.
"},{"location":"using-lagoon-advanced/setting-up-xdebug-with-lagoon/#cli","title":"CLI","text":"The php-cli image is configured to always activate Xdebug when it\u2019s enabled, so there is nothing else that needs to be done. Running any PHP script will start a debugging session.
Install a browser extension to set/unset an activation cookie.
Make sure the activation cookie is set for the website you want to start debugging.
"},{"location":"using-lagoon-advanced/setting-up-xdebug-with-lagoon/#configure-phpstorm","title":"Configure PHPStorm","text":"launch.json for PHP.Add correct path mappings. For a typical Drupal site, an example would be:
launch.json\"pathMappings\": {\n\"/app\": \"${workspaceFolder}\",\n},\nIn the Run tab of Visual Studio Code, click the green arrow next to \u201cListen for Xdebug\u201d
host.docker.internal or your IP address xdebug.client_port 9003 XDEBUG_LOG set to anything to enable logging. Logs will be saved to /tmp/xdebug.log. If you are using the lagoon-examples then you can uncomment some existing lines.XDEBUG_SESSION cookie is set.nc -l 9003 command line tool to open the Xdebug port. If everything is configured in PHP correctly, you should get a Xdebug init response when you load a webpage or run a Drush command.xdebug.client_host has been set correctly. For local debugging with Docker for Mac, this value should be host.docker.internal. For remote debugging this value should be your IP address. If this value was not correctly determined, you can override it by setting the DOCKERHOST environment variable.lando ssh. You won\u2019t be able to debug things by running lando drush or lando php.Verify that Docker for Mac networking is not broken. On your host machine, run nc -l 9003, then in a new terminal window, run:
docker-compose run cli nc -zv host.docker.internal 9003\nYou should see a message like: host.docker.internal (192.168.65.2:9003) open.
Ensure the host host.docker.internal can be reached. If docker has been installed manually (and not through Docker Desktop), this host will not resolve. You can force this to resolve with an additional snippet in your docker-compose.yml file (instructions taken from this blog post):
services:\ncli:\nextra_hosts:\nhost.docker.internal: host-gateway\nphp:\nextra_hosts:\nhost.docker.internal: host-gateway\nIf you're running older images you may still be using Xdebug version 2. All the information on this page still applies, but some of the configuration names and values have changes:
v3 v2 xdebug.mode xdebug.remote_enabled On xdebug.client_host xdebug.remote_hosthost.docker.internal or your IP address xdebug.client_port xdebug.remote_port 9000"},{"location":"using-lagoon-advanced/simplesaml/","title":"SimpleSAML","text":""},{"location":"using-lagoon-advanced/simplesaml/#simplesamlphp","title":"SimpleSAMLphp","text":"This is an example of how to add SimpleSAMLphp to your project and then modify configuration to serve it via NGINX.
"},{"location":"using-lagoon-advanced/simplesaml/#requirements","title":"Requirements","text":"Add SimpleSAMLphp to your project:
Add SimpleSAMLphp to your project via Composercomposer req simplesamlphp/simplesamlphp\nCopy authsources.php and config.php from vendor/simplesamlphp/simplesamlphp/config-templates to somewhere outside vendor directory, such as conf/simplesamlphp. You also need saml20-idp-remote.php from vendor/simplesamlphp/simplesamlphp/metadata-templates.
In config.php set following values for Lagoon:
Base URL path where SimpleSAMLphp is accessed:
config.php 'baseurlpath' => 'https://YOUR_DOMAIN.TLD/simplesaml/',\nStore sessions to database:
config.php 'store.type' => 'sql',\n 'store.sql.dsn' => vsprintf('mysql:host=%s;port=%s;dbname=%s', [\n getenv('MARIADB_HOST'),\n getenv('MARIADB_PORT'),\n getenv('MARIADB_DATABASE'),\n ]),\nAlter other settings to your liking:
technicalcontact and timezone.Add authsources (IdPs) to authsources.php, see example:
'default-sp' => [\n 'saml:SP',\n // The entity ID of this SP.\n 'entityID' => 'https://YOUR_DOMAIN.TLD',\n // The entity ID of the IdP this should SP should contact.\n // Can be NULL/unset, in which case the user will be shown a list of available IdPs.\n 'idp' => 'https://YOUR_IDP_DOMAIN.TLD',\n // The URL to the discovery service.\n // Can be NULL/unset, in which case a builtin discovery service will be used.\n 'discoURL' => null,\n 'NameIDFormat' => 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient',\n 'certificate' => '/app/conf/simplesamlphp/certs/saml.crt',\n 'privatekey' => '/app/conf/simplesamlphp/certs/saml.pem',\n 'redirect.sign' => TRUE,\n 'redirect.validate' => TRUE,\n 'authproc' => [\n 50 => [\n 'class' => 'core:AttributeCopy',\n 'urn:oid:1.3.6.1.4.1.5923.1.1.1.6' => 'eduPersonPrincipalName',\n ],\n 51 => [\n 'class' => 'core:AttributeCopy',\n 'urn:oid:2.5.4.42' => 'givenName',\n ],\n 52 => [\n 'class' => 'core:AttributeCopy',\n 'urn:oid:2.5.4.4' => 'sn',\n ],\n 53 => [\n 'class' => 'core:AttributeCopy',\n 'urn:oid:0.9.2342.19200300.100.1.3' => 'mail',\n ],\n ],\n ],\nAdd IdP metadata to saml20-idp-remote.php, see example:
<?php\n/**\n * SAML 2.0 remote IdP metadata for SimpleSAMLphp.\n *\n * Remember to remove the IdPs you don't use from this file.\n *\n * See: https://simplesamlphp.org/docs/stable/simplesamlphp-reference-idp-remote\n */\n/**\n * Some IdP.\n */\n$metadata['https://YOUR_IDP_DOMAIN.TLD'] = [\n'entityid' => 'https://YOUR_IDP_DOMAIN.TLD',\n'name' => [\n'en' => 'Some IdP',\n],\n'description' => 'Some IdP',\n...\n];\nIn your build process, copy configuration files to SimpleSAMLphp:
vendor/simplesamlphp/simplesamlphp/config/authsources.phpvendor/simplesamlphp/simplesamlphp/config/config.phpvendor/simplesamlphp/simplesamlphp/metadata/saml20-idp-remote.phpCreate file lagoon/nginx/location_prepend_simplesamlphp.conf:
location ^~ /simplesaml {\nalias /app/vendor/simplesamlphp/simplesamlphp/www;\nlocation ~ ^(?<prefix>/simplesaml)(?<phpfile>.+?\\.php)(?<pathinfo>/.*)?$ {\ninclude fastcgi_params;\nfastcgi_pass ${NGINX_FASTCGI_PASS:-php}:9000;\nfastcgi_param SCRIPT_FILENAME $document_root$phpfile;\n# Must be prepended with the baseurlpath\nfastcgi_param SCRIPT_NAME /simplesaml$phpfile;\nfastcgi_param PATH_INFO $pathinfo if_not_empty;\n}\n}\nThis will route /simplesaml URLs to SimpleSAMLphp in vendor.
Modify nginx.dockerfile and add location_prepend_simplesamlphp.conf to the image:
ARG CLI_IMAGE\nFROM ${CLI_IMAGE} as cli\n\nFROM amazeeio/nginx-drupal\n\nCOPY --from=cli /app /app\n\nCOPY lagoon/nginx/location_prepend_simplesamlphp.conf /etc/nginx/conf.d/drupal/location_prepend_simplesamlphp.conf\nRUN fix-permissions /etc/nginx/conf.d/drupal/location_prepend_simplesamlphp.conf\n\n# Define where the Drupal Root is located\nENV WEBROOT=public\nLagoon allows you to connect to your running containers via SSH. The containers themselves don't actually have an SSH server installed, but instead you connect via SSH to Lagoon, which then itself creates a remote shell connection via the Kubernetes API for you.
"},{"location":"using-lagoon-advanced/ssh/#ensure-you-are-set-up-for-ssh-access","title":"Ensure you are set up for SSH access","text":""},{"location":"using-lagoon-advanced/ssh/#generating-an-ssh-key","title":"Generating an SSH Key","text":"It is recommended to generate a separate SSH key for each device as opposed to sharing the same key between multiple computers. Instructions for generating an SSH key on various systems can be found below:
"},{"location":"using-lagoon-advanced/ssh/#osx-mac","title":"OSX (Mac)","text":"Mac
"},{"location":"using-lagoon-advanced/ssh/#linux-ubuntu","title":"Linux (Ubuntu)","text":"Linux
"},{"location":"using-lagoon-advanced/ssh/#windows","title":"Windows","text":"Windows
"},{"location":"using-lagoon-advanced/ssh/#ssh-agent","title":"SSH Agent","text":""},{"location":"using-lagoon-advanced/ssh/#osx-mac_1","title":"OSX (Mac)","text":"OSX does not have its SSH agent configured to load configured SSH keys at startup, which can cause some headaches. You can find a handy guide to configuring this capability here: https://www.backarapper.com/add-ssh-keys-to-ssh-agent-on-startup-in-macos/
"},{"location":"using-lagoon-advanced/ssh/#linux","title":"Linux","text":"Linux distributions vary in how they use the ssh-agent . You can find a general guide here: https://www.ssh.com/academy/ssh/agent
SSH key support in Windows has improved markedly as of recently, and is now supported natively. A handy guide to configuring the Windows 10 SSH agent can be found here: https://richardballard.co.uk/ssh-keys-on-windows-10/
"},{"location":"using-lagoon-advanced/ssh/#uploading-ssh-keys","title":"Uploading SSH Keys","text":""},{"location":"using-lagoon-advanced/ssh/#via-the-ui","title":"Via the UI","text":"You can upload your SSH key(s) through the UI. Log in as you normally would.
In the upper right hand corner, click on Settings:
You will then see a page where you can upload your SSH key(s), and it will show any uploaded keys. Paste your key into the text box, give it a name, and click \"Add.\" That's it! Add additional keys as needed.
"},{"location":"using-lagoon-advanced/ssh/#via-command-line","title":"Via Command Line","text":"A general example of using the Lagoon API via GraphQL to add an SSH key to a user can be found here
"},{"location":"using-lagoon-advanced/ssh/#ssh-into-a-pod","title":"SSH into a pod","text":""},{"location":"using-lagoon-advanced/ssh/#connection","title":"Connection","text":"Connecting is straightforward and follows the following pattern:
SSHssh -p [PORT] -t [PROJECT-ENVIRONMENT-NAME]@[HOST]\nPORT - The remote shell SSH endpoint port (for amazee.io: 32222).HOST - The remote shell SSH endpoint host (for amazee.io ssh.lagoon.amazeeio.cloud).PROJECT-ENVIRONMENT-NAME - The environment you want to connect to. This is most commonly in the pattern PROJECTNAME-ENVIRONMENT.As an example:
SSH examplessh -p 32222 -t drupal-example-main@ssh.lagoon.amazeeio.cloud\nThis will connect you to the project drupal-example on the environment main.
By default, the remote shell will try to connect you to the container defined with the type cli. If you would like to connect to another pod/service you can define it via:
ssh -p [PORT] -t [PROJECT-ENVIRONMENT-NAME]@[HOST] service=[SERVICE-NAME]\nIf your pod/service contains multiple containers, Lagoon will connect you to the first defined container. You can also define the specific container to connect to via:
Define containerssh -p [PORT] -t [PROJECT-ENVIRONMENT-NAME]@[HOST] service=[SERVICE-NAME] container=[CONTAINER-NAME]\nFor example, to connect to the php container within the nginx pod:
ssh -p 32222 -t drupal-example-main@ssh.lagoon.amazeeio.cloud service=nginx container=php\nThe common case of copying a file into your cli pod can be acheived with the usual SSH-compatible tools.
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 32222 [local_path] [project_name]-[environment_name]@ssh.lagoon.amazeeio.cloud:[remote_path]\nrsync --rsh='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 32222' [local_path] [project_name]-[environment_name]@ssh.lagoon.amazeeio.cloud:[remote_path]\nssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 32222 [project_name]-[environment_name]@ssh.lagoon.amazee.io tar -zcf - [remote_path] | tar -zxf - -C /tmp/\nIn the rare case that you need to specify a non-CLI service you can specify the service=... and/or container=... arguments in the copy command.
Piping tar through the ssh connection is the simplest method, and can be used to copy a file or directory using the usual tar flags:
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 32222 [project_name]-[environment_name]@ssh.lagoon.amazee.io service=solr tar -zcf - [remote_path] | tar -zxf - -C /tmp/\nYou can also use rsync with a wrapper script to reorder the arguments to ssh in the manner required by Lagoon's SSH service:
#!/usr/bin/env sh\nsvc=$1 user=$3 host=$4\nshift 4\nexec ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 32222 -l \"$user\" \"$host\" \"$svc\" \"$@\"\nPut that in an executable shell script rsh.sh and specify the service=... in the rsync command:
rsync --rsh=\"/path/to/rsh.sh service=cli\" /tmp/foo [project_name]-[environment_name]@ssh.lagoon.amazeeio.cloud:/tmp/foo\nThe script could also be adjusted to also handle a container=... argument.
In order to automatically trigger new deployments using Azure Pipelines follow these instructions:
id_rsa_lagoon. For more information about secure files have a look at the Azure Documentation Site.azure-pipelines.yml:pool:\nvmImage: 'ubuntu-latest'\nstages:\n# .. other stages\n- stage: Deploy\ncondition: and(succeeded(), in(variables['Build.SourceBranch'], 'refs/heads/staging', 'refs/heads/develop'))\njobs:\n- job: DeployLagoon\nsteps:\n- task: DownloadSecureFile@1\nname: lagoonSshKey\ndisplayName: 'Download Lagoon SSH key'\ninputs:\nsecureFile: id_rsa_lagoon\n- script: |\ncurl -L \"https://github.com/amazeeio/lagoon-cli/releases/download/0.9.2/lagoon-cli-0.9.2-linux-amd64\" -o ./lagoon\nchmod +x ./lagoon\ndisplayName: 'Download lagoon-cli'\n- script: ./lagoon login -i $(lagoonSshKey.secureFilePath)\ndisplayName: 'Log into Lagoon'\n- script: ./lagoon deploy branch -e $(Build.SourceBranchName) -p my-awesome-project -b $(Build.SourceBranchName) --force\ndisplayName: 'Trigger deployment using lagoon-cli'\nThis will trigger a new deployment whenever changes are made on the develop or staging branch. Adjust these values accordingly so they fit your deployment strategy and configuration.
There may be a case where you want to push without a deployment. Make sure your commit message contains \"[skip deploy]\" or \"[deploy skip]\" and Lagoon will not trigger a deployment from that commit.
Lagoon tries to support any development workflow possible. It specifically does not enforce any workflows onto teams, so that each development team can define how they would like to develop and deploy their code.
"},{"location":"using-lagoon-advanced/workflows/#fixed-branches","title":"Fixed Branches","text":"The most straightforward workflows are deployment-based on some fixed branches:
You define which branches (like develop, staging and main, which would be ^(develop|staging|main)$ as regular expressions) that Lagoon should deploy and it will do so. Done!
If you would like to test a new feature, merge them into a branch that you have set up locally and push, and Lagoon will deploy the feature and you can test. When all is good, merge the branch into your production branch and push.
"},{"location":"using-lagoon-advanced/workflows/#feature-branches","title":"Feature Branches","text":"A bit more advanced are feature branches. Since Lagoon supports the ability to define the branches you would like to deploy via regular expressions, you can also extend the above regular expression to this: ^feature\\/|^(staging|main)$. This will instruct Lagoon to deploy all branches that start with feature/, plus the branches called staging and main. Our development workflow could be as following:
main called feature/myfeature and push feature/myfeature.feature/myfeature as a new environment, where you can test your feature independently of any other features.feature/myfeature into the main branch and it will deploy to your production environment.If you like, you can also merge feature/myfeature and any other feature branches into staging first, in order to test the functionality of multiple features together. After you have tested the features together on staging, you can merge the features into main.
This workflow needs a high level of branch pruning and cleanliness in your Git repository. Since each feature branch will create its own Lagoon environment, you can have very quickly generate a LOT of environments, which all of them will use resources. Be sure to merge or delete unused branches.
Because of this, it could make sense to think about a pull request based workflow.
"},{"location":"using-lagoon-advanced/workflows/#pull-requests","title":"Pull requests","text":"Even more advanced are workflows via pull requests. Such workflows need the support of a Git hosting which supports pull requests (also called merge requests). The idea of pull request-based workflows lies behind that idea that you can test a feature together with a target branch, without actually needing to merge yet, as Lagoon will do the merging for you during the build.
In our example we would configure Lagoon to deploy the branches ^(staging|main)$ and the pull requests to .* (to deploy all pull requests). Now our workflow would be:
main called feature/myfeature and push feature/myfeature (no deployment will happen now because we have only specific staging and main as our branches to be deployed).feature/myfeature into main.feature/myfeature branch on top of the main branch and deploy that resulting code for you.feature/myfeature branch just as if it had been merged into main, so all changes that have happened in main since you created the feature/myfeature branch from it will be there, and you don't need to worry that you might have an older version of the main branch.main. This will now trigger a deployment of main.Some teams might opt to create the pull request against a shared staging branch and then merge the staging branch into the main branch via another pull request. This depends on the kind of Git workflow you're using.
Additionally, in Lagoon you can define that only pull requests with a specific text in the title are deployed. [BUILD] defined as regular expression will only deploy pull requests that have a title like [BUILD] My Pull Request, while a pull request with that title My other Pull Request is not automatically deployed. This helps to keep the amount of environments small and allows for pull requests that don't need an environment yet.
Automatic pull request environments are a fantastic thing. But it would also be handy to have the database synced from another environment when those environments are created. Lagoon can handle that!
The following example will sync the staging database on the first rollout of the pull request environment:
.lagoon.ymltasks:\npost-rollout:\n- run:\nname: IF no Drupal installed & Pullrequest = Sync database from staging\ncommand: |\nif [[ -n ${LAGOON_PR_BASE_BRANCH} ]] && tables=$(drush sqlq 'show tables;') && [ -z \"$tables\" ]; then\ndrush -y sql-sync @staging default\nfi\nservice: cli\nshell: bash\nAnother way of deploying your code into an environment is the promotion workflow.
The idea behind the promotion workflow comes from this (as an example):
If you merge the branch staging into the main branch, and if there are no changes to main , so main and staging have the exact same code in Git, it could still technically be possible that the resulting Docker images are slightly different. This is because it's possible that between the last staging deployment and the current main deployment, some upstream Docker images may have changed, or dependencies loaded from the various package managers may have changed. This is a very small chance, but it's there.
For this situation, Lagoon understands the concept of promoting Lagoon images from one environment to another. This basically means that it will take the already built and deployed Docker images from one environment, and will use those exact same Docker images for another environment.
In our example, we want to promote the Docker images from the main environment to the production environment:
main. Make sure that the environment has deployed successfully.production in your Git repository. This could lead to weird confusions (like people pushing into this branch, etc).curl request: curl -X POST \\\nhttps://rest.lagoon.amazeeio.cloud/promote \\\n-H 'Content-Type: application/json' \\\n-d '{\n \"projectName\":\"myproject\",\n \"sourceEnvironmentName\": \"main\",\n \"branchName\": \"production\"\n }'\nThis tells Lagoon that you want to promote from the source main to the destination production (yes, it really uses branchName as destination, which is a bit unfortunate, but it will be fixed soon).
Lagoon will now do the following:
main in order to load the .lagoon.yml and docker-compose.yml files (Lagoon still needs these in order to fully work).docker-compose.yml , but with LAGOON_GIT_BRANCH=production as environment variable.main environment and use them (instead of building Images or tagging them from upstream).You will receive the same notifications of success or failures like any other deployment.
"},{"location":"using-lagoon-the-basics/","title":"Overview","text":""},{"location":"using-lagoon-the-basics/#requirements","title":"Requirements","text":""},{"location":"using-lagoon-the-basics/#docker","title":"Docker","text":"To run a Lagoon Project, your system must meet the requirements to run Docker. We suggest installing the latest version of Docker for your workstation. You can download Docker here. We also suggest allowing Docker at least 4 CPUs and 4 GB RAM.
"},{"location":"using-lagoon-the-basics/#local-development-environments","title":"Local Development Environments","text":"TL;DR: install and start pygmy:
brew tap pygmystack/pygmy; # (1)\nbrew install pygmy;\npygmy up\nPygmy is a container stack for local development, developed collaboratively with the Lagoon team.
Learn more about Lagoon, pygmy, and Local Development Environments
"},{"location":"using-lagoon-the-basics/#step-by-step-guides","title":"Step by Step Guides","text":".lagoon.yml","text":"This is the main file that will be used by Lagoon to understand what should be deployed, as well as many other things. See documentation for .lagoon.yml.
docker-compose.yml","text":"This file is used by Docker Compose to start your local development environment. Lagoon also uses it to understand which of the services should be deployed, which type, and how to build them. This happens via labels. See documentation for docker-compose.yml.
Some Docker images and containers need additional customizations from the provided images. This usually has two reasons:
All images are pushed to https://hub.docker.com/u/uselagoon. We suggest always using the latest tag (like uselagoon/nginx:latest) as they are kept up to date in terms of features and security.
If you choose to use a specific Lagoon version of an image like uselagoon/nginx:20.10.0 or uselagoon/node-10:20.10.0 it is your own responsibility to upgrade the version of the images as soon as a new Lagoon version is released!
This document describes what actually happens during a Lagoon build and deployment. It is heavily simplified from what actually happens, but it will help you to understand what is happening under the hood every time that Lagoon deploys new code for you.
Watch the video below for a walk-through of the deployment process.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#1-set-up-openshift-projectkubernetes-namespace-for-environment","title":"1. Set up OpenShift Project/Kubernetes Namespace for Environment","text":"First, Lagoon checks if the OpenShift project/Kubernetes namespace for the given environment exists and is correctly set up. It will make sure that we have the needed service accounts, create secrets, and will configure environment variables into a ConfigMap lagoon-env which is filled with information like the environment type and name, the Lagoon project name, and so on.
Next, Lagoon will check out your code from Git. It needs that to be able to read the .lagoon.yml, docker-compose.yml and any .env files, but also to build the Docker images.
Note that Lagoon will only process these actions if the branch/PR matches the branch regex set in Lagoon. Based on how the deployment has been triggered, different things will happen:
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#branch-webhook-push","title":"Branch Webhook Push","text":"If the deployment is triggered automatically via a Git webhook and is for a single branch, Lagoon will check out the Git SHA which is included in the webhook payload. This will trigger a deployment for every Git SHA pushed.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#branch-rest-trigger","title":"Branch REST trigger","text":"If you trigger a branch deployment manually via the REST API (via the UI, or GraphQL) and do NOT define a SHA in the POST payload, Lagoon will just check out the latest commit in that branch and deploy it.
If the deployment is a pull request (PR) deployment, Lagoon will load the base and the HEAD branch and SHAs for the pull request and will:
HEAD branch (the branch that the PR originates from) on top of the base branch.If the merge fails, Lagoon will also stop and inform you about this.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#3-build-image","title":"3. Build Image","text":"For each service defined in the docker-compose.yml Lagoon will check if images need to be built or not. If they need to be built, this will happen now. The order of building is based on the order they are configured in docker-compose.yml , and some build arguments are injected:
LAGOON_GIT_SHALAGOON_GIT_BRANCHLAGOON_PROJECTLAGOON_BUILD_TYPE (either pullrequest, branch or promote)LAGOON_SSH_PRIVATE_KEY - The SSH private key that is used to clone the source repository. Use RUN /lagoon/entrypoints/05-ssh-key.sh to convert the build argument into an actual key at /home/.ssh/key which will be used by SSH and Git automatically. For safety, remove the key again via RUN rm /home/.ssh/key.LAGOON_GIT_SOURCE_REPOSITORY - The full Git URL of the source repository.Also, if this is a pull request build:
LAGOON_PR_HEAD_BRANCHLAGOON_PR_HEAD_SHALAGOON_PR_BASE_BRANCHLAGOON_PR_BASE_SHALAGOON_PR_TITLEAdditionally, for each already built image, its name is also injected. If your docker-compose.yml is configured to first build the cli image and then the nginx image, the name of the nginx image is injected as NGINX_IMAGE.
Next, Lagoon will configure Kubernetes or OpenShift with all services and routes that are defined from the service types, plus possible additional custom routes that you have defined in .lagoon.yml.
In this step it will expose all defined routes in the LAGOON_ROUTES as comma separated URLs. It will also define one route as the \"main\" route, in this order:
.lagoon.yml.docker-compose.yml.The \"main\" route is injected via the LAGOON_ROUTE environment variable.
Now it is time to push the previously built Docker images into the internal Docker image registry.
For services that didn't specify a Dockerfile to be built in docker-compose.yml and only gave an image, they are also tagged and will cause the internal Docker image registry to know about the images, so that they can be used in containers.
Lagoon will now create persistent storage (PVC) for each service that needs and requested persistent storage.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#7-cron-jobs","title":"7. Cron jobs","text":"For each service that requests a cron job (like MariaDB), plus for each custom cron job defined in .lagoon.yml, Lagoon will now generate the cron job environment variables which are later injected into the Deployment.
Now Lagoon will check the .lagoon.yml file for defined tasks in pre-rollout and will run them one by one in the defined services. Note that these tasks are executed on the pods currently running (so cannot utilize features or scripts that only exist in the latest commit) and therefore they are also not run on first deployments.
If any of them fail, Lagoon will immediately stop and notify you, and the rollout will not proceed.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#9-deploymentconfigs-statefulsets-daemonsets","title":"9. DeploymentConfigs, Statefulsets, Daemonsets","text":"This is probably the most important step. Based on the defined service type, Lagoon will create the Deployment, Statefulset or Daemonsets for the service. (Note that Deployments are analogous to DeploymentConfigs in OpenShift)
It will include all previously gathered information like the cron jobs, the location of persistent storage, the pushed images and so on.
Creation of these objects will also automatically cause Kubernetes or OpenShift to trigger new deployments of the pods if necessary, like when an environment variable has changed or an image has changed. But if there is no change, there will be no deployment! This means if you only update the PHP code in your application, the Varnish, Solr, MariaDB, Redis and any other service that is defined but does not include your code will not be deployed. This makes everything much much faster.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#10-wait-for-all-rollouts-to-be-done","title":"10. Wait for all rollouts to be done","text":"Now Lagoon waits! It waits for all of the just-triggered deployments of the new pods to be finished, as well as for their health checks to be successful.
If any of the deployments or health checks fail, the deployment will be stopped here, and you will be informed via the defined notification systems (like Slack) that the deployment has failed.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#11-run-defined-post-rollout-tasks","title":"11. Run defined post-rollout tasks","text":"Now Lagoon will check the .lagoon.yml file for defined tasks in post-rollout and will run them one by one in the defined services.
If any of them fail, Lagoon will immediately stop and notify you.
"},{"location":"using-lagoon-the-basics/build-and-deploy-process/#12-success","title":"12. Success","text":"If all went well and nothing threw any errors, Lagoon will mark this build as successful and inform you via defined notifications. \u2705
"},{"location":"using-lagoon-the-basics/configure-webhooks/","title":"Configure Webhooks","text":"Your Lagoon administrator will also give you the route to the webhook-handler. You will add this to your repository as an outgoing webhook, and choose which events to send to Lagoon. Typically, you will send all push and pull request events. In Lagoon it is possible to add a regular expression to determine which branches and pull requests actually result in a deploy, and your Lagoon administrator can set that up for you. For example, all branches that start with feature- could be deployed to Lagoon.
If you are an amazee.io customer, the route to the webhook-handler is: https://hooks.lagoon.amazeeio.cloud.
Danger
Managing the following settings will require you to have a high level of access to these repositories, which will be controlled by your organization. If you cannot access these settings, please contact your systems administrator or the appropriate person within your organization.
"},{"location":"using-lagoon-the-basics/configure-webhooks/#github","title":"GitHub","text":"Add webhook in your GitHub repository. Payload URL is the route to the webhook-handler of your Lagoon instance, provided by your Lagoon administrator.Content type to application/json. Let me select individual events.\"Push and Pull request events, and then filter further in the Lagoon configuration of your project. Active.Add webhook to save your configuration.URL is the route to the webhook-handler of your Lagoon instance, provided by your Lagoon administrator.Trigger events which will send a notification to Lagoon. We suggest that you send Push events and Merge request events, and then filter further in the Lagoon configuration of your project. Add webhookto save your configuration.Title is for your reference.URL is the route to the webhook-handler of your Lagoon instance, provided by your Lagoon administrator.Choose from a full list of triggers and select the following:
5. Click Save to save the webhook configurations for Bitbucket.
The docker-compose.yml file is used by Lagoon to:
Docker Compose (the tool) is very strict in validating the content of the YAML file, so we can only do configuration within labels of a service definition.
Warning
Lagoon only reads the labels, service names, image names and build definitions from a docker-compose.yml file. Definitions like: ports, environment variables, volumes, networks, links, users, etc. are IGNORED.
This is intentional, as the docker-compose file is there to define your local environment configuration. Lagoon learns from the lagoon.type the type of service you are deploying and from that knows about ports, networks and any additional configuration that this service might need.
Here a straightforward example of a docker-compose.yml file for Drupal:
version: '2.3'\nx-lagoon-project:\n# Lagoon project name (leave `&lagoon-project` when you edit this)\n&lagoon-project drupal-example\nx-volumes:\n&default-volumes\n# Define all volumes you would like to have real-time mounted into the docker containers\nvolumes:\n- .:/app:delegated\nx-environment:\n&default-environment\nLAGOON_PROJECT: *lagoon-project\n# Route that should be used locally, if you are using pygmy, this route *must* end with .docker.amazee.io\nLAGOON_ROUTE: http://drupal-example.docker.amazee.io\n# Uncomment if you want to have the system behave as it will in production\n#LAGOON_ENVIRONMENT_TYPE: production\n# Uncomment to enable Xdebug and then restart via `docker-compose up -d`\n#XDEBUG_ENABLE: \"true\"\nx-user:\n&default-user\n# The default user under which the containers should run. Change this if you are on linux and run with another user than ID `1000`\nuser: '1000'\nservices:\nnginx:\nbuild:\ncontext: .\ndockerfile: nginx.dockerfile\nlabels:\nlagoon.type: nginx-php-persistent # (1)\nlagoon.persistent: /app/web/sites/default/files/\nphp:\nbuild:\ncontext: .\ndockerfile: php.dockerfile\nlabels:\nlagoon.type: nginx-php-persistent # (2)\nlagoon.name: nginx\nlagoon.persistent: /app/web/sites/default/files/\nmariadb:\nimage: amazeeio/mariadb-drupal\nlabels:\nlagoon.type: mariadb\nx-lagoon-project:
This is the machine name of your project, define it here. We\u2019ll use \u201cdrupal-example.\u201d
x-volumes:
This tells Lagoon what to mount into the container. Your web application lives in /app, but you can add or change this if needed.
x-environment:
.docker.amazee.io.LAGOON_ENVIRONMENT_TYPE: production.DEBUG_ENABLE: \"true\".x-user:
You are unlikely to need to change this, unless you are on Linux and would like to run with a user other than 1000.
services","text":"This defines all the services you want to deploy. Unfortunately, Docker Compose calls them services, even though they are actually containers. Going forward we'll be calling them services, and throughout this documentation.
The name of the service (nginx, php, and mariadb in the example above) is used by Lagoon as the name of the Kubernetes pod (yet another term - again, we'll be calling them services) that is generated, plus also any additional Kubernetes objects that are created based on the defined lagoon.type, which could be things like services, routes, persistent storage, etc.
Please note that service names adhere to the RFC 1035 DNS label standard. Service names must:
Warning
Once you have set the name of a service, do NOT rename it. This will cause all kind of havoc in your containers and break things.
"},{"location":"using-lagoon-the-basics/docker-compose-yml/#docker-images","title":"Docker Images","text":""},{"location":"using-lagoon-the-basics/docker-compose-yml/#build","title":"build","text":"If you want Lagoon to build a Dockerfile for your service during every deployment, you can define it here:
build
contextdocker build command.dockerfile:Warning
Lagoon does NOT support the short version of build: <Dockerfile> and will fail if it finds such a definition.
image","text":"If you don't need to build a Dockerfile and just want to use an existing Dockerfile, define it via image.
Lagoon needs to know what type of service you are deploying in order to configure the correct Kubernetes or OpenShift objects.
This is done via the lagoon.type label. There are many different types to choose from. Check Service Types to see all of them and their additional configuration possibilities.
If you'd like Lagoon to ignore a service completely - for example, you need a container only during local development - give it the type none.
Some containers need persistent storage. Lagoon allows for each container to have a maximum of one persistent storage volume attached to the container. You can configure the container to request its own persistent storage volume (which can then be mounted by other container), or you can tell the container to mount the persistent storage created by another container.
In many cases, Lagoon knows where that persistent storage needs to go. For example, for a MariaDB container, Lagoon knows that the persistent storage should be put into /var/lib/mysql , and puts it there automatically without any extra configuration to define that. For some situations, though, Lagoon needs your help to know where to put the persistent storage:
lagoon.persistent - The absolute path where the persistent storage should be mounted (the above example uses /app/web/sites/default/files/ which is where Drupal expects its persistent storage).lagoon.persistent.name - Tells Lagoon to not create a new persistent storage for that service, but instead mounts the persistent storage of another defined service into this service.lagoon.persistent.size - The size of persistent storage you require (Lagoon usually gives you minimum 5G of persistent storage, if you need more, define it here).lagoon.persistent.class - By default Lagoon automatically assigns the right storage class for your service (like SSDs for MySQL, bulk storage for Nginx, etc.). If you need to overwrite this, you can do so here. This is highly dependent on the underlying Kubernetes/OpenShift that Lagoon runs on. Ask your Lagoon administrator about this.The docker-compose.yml file also supports per-service enabling and disabling of autogenerated routes
lagoon.autogeneratedroute: false label will stop a route from being automatically created for that service. It can be applied to all services with autogenerated routes, but is mostly useful for the basic and basic-persistent service types when used to create an additional internal-facing service for a database service or similar. The inverse is also true - it will enable an auto-generated route for a service when the .lagoon.yml file disables them.Kubernetes and OpenShift don't deploy plain containers. Instead, they deploy pods, with each one or more containers. Usually Lagoon creates a single pod with a container inside for each defined docker-compose service. For some cases, we need to put two containers inside a single pod, as these containers are so dependent on each other that they should always stay together. An example for such a situation is the PHP and NGINX containers that both contain PHP code of a web application like Drupal.
For these cases, it is possible to tell Lagoon which services should stay together, which is done in the following way (remember that we are calling containers services because of docker-compose:
lagoon.type that expects two services (in the example this is nginx-php-persistent defined on the nginx and php services).lagoon.name of the second one with the first one. (in the example this is done with defining lagoon.name: nginx).This will cause Lagoon to realize that the nginx and php containers are combined in a pod that will be called nginx.
Warning
Once you have set the lagooon.name of a service, do NOT rename it. This will cause all kind of havoc in your containers and break things.
Lagoon still needs to understand which of the two services is the actual individual service type (nginx and php in this case). It does this by searching for service names with the same name that are given by the type, so nginx-php-persistent expects one service with the name nginx and one with php in the docker-compose.yml. If for any reason you want to use different names for the services, or you need for than one pod with the type nginx-php-persistent there is an additional label lagoon.deployment.servicetype which can be used to define the actual service type.
An example:
docker-compose.ymlnginx:\nbuild:\ncontext: .\ndockerfile: nginx.dockerfile\nlabels:\nlagoon.type: nginx-php-persistent\nlagoon.persistent: /app/web/sites/default/files/\nlagoon.name: nginx # If this isn't present, Lagoon will use the container name, which in this case is nginx.\nlagoon.deployment.servicetype: nginx\nphp:\nbuild:\ncontext: .\ndockerfile: php.dockerfile\nlabels:\nlagoon.type: nginx-php-persistent\nlagoon.persistent: /app/web/sites/default/files/\nlagoon.name: nginx # We want this service to be part of the NGINX pod in Lagoon.\nlagoon.deployment.servicetype: php\nIn the example above, the services are named nginx and php (but you can call them whatever you want). The lagoon.name tells Lagoon which services go together - all of the services with the same name go together.
In order for Lagoon to realize which one is the nginx and which one is the php service, we define it via lagoon.deployment.servicetype: nginx and lagoon.deployment.servicetype: php.
Lagoon uses Helm for templating on Kubernetes. To do this, a series of Charts are included with the build-deploy-tool image.
By default, Lagoon expects that services from custom templates are rolled out via a DeploymentConfig object within Kubernetes or Openshift. It monitors the rollout based on this object. In some cases, the services that are defined via custom deployment need a different way of monitoring. This can be defined via lagoon.rollout:
deploymentconfig - This is the default. Expects a DeploymentConfig object in the template for the service.statefulset - Expects a Statefulset object in the template for the service.daemonset - Expects a Daemonset object in the template for the service.false - Will not monitor any rollouts, and will just be happy if the template applies and does not throw any errors.You can also overwrite the rollout for just one specific environment. This is done in .lagoon.yml.
BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.
With the release of Lagoon v2.11.0, Lagoon now provides support for BuildKit-based docker-compose builds. To enable BuildKit for your Project or Environment, add DOCKER_BUILDKIT=1 as a build-time variable.
Bug
Note that while using BuildKit locally, you may experience some known issues.
Failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: This message means that your build has tried to access a Docker image that hasn't been built yet. As BuildKit builds in parallel, if you have a Docker image that inherits another one (as we do in Drupal with the CLI). You can use the target field inside the build to reconfigure as a multi-stage buildvolumes_from in Docker Compose v2 - this service (that provides SSH access into locally running containers) has been deprecated by Docker Compose. The section can be removed from your docker-compose.yml file if you don't require SSH access from inside your local environment, or can be worked around on a project-by-project basis - see https://github.com/pygmystack/pygmy/issues/333#issuecomment-1274091375 for more information.Note
If you are deploying a Drupal Project, skip this and read the Drupal-specific first deployment documentation.
"},{"location":"using-lagoon-the-basics/first-deployment/#1-make-sure-you-are-ready","title":"1. Make sure you are ready","text":"In order to make your first deployment a successful one, please make sure that your project is Lagoonized and that you have set up the project in Lagoon. If not, or you're not sure, or that doesn't sound familiar, don't worry, go back and follow the Step-by-Step Guides which show you how this works, and then come back and deploy!
"},{"location":"using-lagoon-the-basics/first-deployment/#2-push","title":"2. Push","text":"With Lagoon, you create a new deployment by pushing into a branch that is configured to be deployed.
If you don't have any new code to push, don't worry! Run:
Git pushgit commit --allow-empty -m \"go, go! Power Rangers!\"\ngit push\nThis will trigger a push, and your Git hosting will inform Lagoon about this push via the configured webhook.
If all is correct, you should see a notification in your configured chat system (this has been configured by your friendly Lagoon administrator):
This informs you that Lagoon has just started to deploy your code. Depending on the size of the code and amount of containers, this will take a couple of seconds. Just relax. If you want to know what's happening now, check out the Build and Deploy Process of Lagoon.
You can also check your Lagoon UI to see the progress of any deployment (your Lagoon administrator has the info).
"},{"location":"using-lagoon-the-basics/first-deployment/#3-its-done","title":"3. It's done","text":"As soon as Lagoon is done building and deploying it will send a second notification to the chat system, here an example:
It tells you:
You can also quickly tell what kind of notification it is by the emoji at the beginning - whether it's just info that the build has started, a success, or fail.
That's it! We hope that wasn't too hard - making devOps accessible is what we are striving for!
"},{"location":"using-lagoon-the-basics/first-deployment/#but-wait-how-about-other-branches-or-the-production-environment","title":"But wait, how about other branches or the production environment?","text":"That's the beauty of Lagoon: it's exactly the same! Just push the name of the branch and that one will be deployed.
"},{"location":"using-lagoon-the-basics/first-deployment/#failure-dont-worry","title":"Failure? Don't worry","text":"Did the deployment fail? Oh no! But we're here to help:
Logs link in the error notification, it will tell you where in the deployment process the failure happened.Congratulations, you're this close to going live with your website on Lagoon! In order to make this as seamless as possible, we've got this final checklist for you. It leads you through the last few things you should check before taking your site live.
"},{"location":"using-lagoon-the-basics/going-live/#check-your-lagoonyml","title":"Check your.lagoon.yml","text":""},{"location":"using-lagoon-the-basics/going-live/#routes-ssl","title":"Routes / SSL","text":"Check to be sure that all routes have been set up in your .lagoon.yml. Be aware that if you don't point the domains towards Lagoon, you should disable Let's Encrypt (LE) certificate creation, as it will lead to issues. Domains not pointing towards Lagoon will be disabled after a while in order to not exceed the Let's Encrypt quotas.
If you use Certificate Authority (CA) signed certificates, you can set tls-acme to false , but leave the insecure flag set to Allow or Redirect. In the case of CA certificates, let your Lagoon administrator know the routes and the SSL certificate that needs to be put in place.
environments:\nmain:\nroutes:\n- nginx:\n- example.com:\ntls-acme: 'false'\ninsecure: Allow\n- www.example.com:\ntls-acme: 'false'\ninsecure: Allow\nAs soon as the DNS entries point towards your Lagoon installation, you can switch the flags: tls-acme to true and insecure to Redirect
environments:\nmain:\nroutes:\n- nginx:\n- example.com:\ntls-acme: 'true'\ninsecure: Redirect\n- www.example.com:\ntls-acme: 'true'\ninsecure: Redirect\nNote
As checking every page of your website might be a bit a tedious job, you can make use of mixed-content-scan. This will crawl the entire site and give you back pages that include assets from a non-HTTPS site.
"},{"location":"using-lagoon-the-basics/going-live/#redirects","title":"Redirects","text":"If you need non-www to www redirects, make sure you have them set up in the redirects-map.conf - see Documentation.
Check if your cron jobs have been set up for your production environment - see .lagoon.yml.
To make it as smooth as possible for you to get your site pointing to our servers, we have dedicated load-balancer DNS records. Those technical DNS resource records are used for getting your site linked to the amazee.io infrastructure and serve no other purpose. If you are in doubt of the CNAME record, ask your Lagoon administrator about the exact CNAME you need to set up.
Example on amazee.io : <region-identifier>.amazee.io
Before you switch over your domain to Lagoon, make sure you lower the Time-to-Live (TTL) before you go live. This will ensure that the switch from the old to the new servers will go quickly. We usually advise a TTL of 300-600 seconds prior to the DNS switch. More information about TTL.
"},{"location":"using-lagoon-the-basics/going-live/#recommended-settings-for-fastly-cname-record","title":"Recommended settings for Fastly (CNAME record):","text":"The recommended method of pointing your domain's DNS records at Lagoon is via a CNAME record as shown below:
CNAME: cdn.amazee.io
If your DNS provider does not support the use of CNAME records, you can use these A records instead. Please ensure you set up individual records for each IP listed below:
A: 151.101.2.191A: 151.101.66.191A: 151.101.130.191A: 151.101.194.191Note
We do not suggest configuring any static IP addresses in your DNS zones. The Lagoon load balancer infrastructure may change over time which can have impact on your site availability if you configure a static IP address.
"},{"location":"using-lagoon-the-basics/going-live/#root-domains","title":"Root Domains","text":"Configuring the root domain (e.g. example.com) can be a bit tricky because the DNS specification does not allow the root domain to point to a CNAME entry. Depending on your DNS provider, the record name is different:
If your DNS provider needs an IP address for the root domain, get in touch with your Lagoon administrator to give you the load balancer IP addresses.
"},{"location":"using-lagoon-the-basics/going-live/#production-environment","title":"Production environment","text":"Lagoon understands the concept of development and production environments. Development environments automatically send noindex and nofollow headers in order to prohibit indexing by search engines.
X-Robots-Tag: noindex, nofollow
During project setup, the production environment should already be defined. If that's omitted, your environment will run in development mode. You can check if the environment is set as production environment in the Lagoon user interface. If the production environment is not set, let your Lagoon administrator know, and they will configure the system accordingly.
"},{"location":"using-lagoon-the-basics/lagoon-yml/","title":".lagoon.yml","text":"The .lagoon.yml file is the central file to set up your project. It contains configuration in order to do the following:
The .lagoon.yml file must be placed at the root of your Git repository.
docker-compose-yaml","text":"Tells the build script which Docker Compose YAML file should be used, in order to learn which services and containers should be deployed. This defaults to docker-compose.yml, but could be used for a specific Lagoon Docker Compose YAML file if needed.
environment_variables.git_sha","text":"This setting allows you to enable injecting the deployed Git SHA into your project as an environment variable. By default this is disabled. Setting the value to true sets the SHA as the environment variable LAGOON_GIT_SHA.
Routes are used to direct traffic to services. Each service in an environnment can have routes, in which the domain names are defined manually or automatically. The top level routes section applies to all routes in all environments.
routes.autogenerate","text":"This allows you to configure automatically created routes. Manual routes are defined per environment.
enabled: Set to false to disable autogenerated routes. Default is true.allowPullrequests: Set to true to override enabled: false for pull requests..lagoon.ymlroutes:\nautogenerate:\nenabled: false\nallowPullrequests: true\ninsecure: Configures HTTP connections. Default is Allow.Allow: Route will respond to HTTP and HTTPS.Redirect: Route will redirect any HTTP request to HTTPS.prefixes: Configure prefixes for the autogenerated routes of each environment. This is useful for things like language prefix domains, or a multi-domain site using the Drupal domain module.
routes:\nautogenerate:\nprefixes:\n- www\n- de\n- fr\n- it\nThere are different type of tasks you can define, and they differ in when exactly they are executed in a build flow:
"},{"location":"using-lagoon-the-basics/lagoon-yml/#pre-rollout-tasks-pre_rolloutirun","title":"Pre-Rollout Tasks -pre_rollout.[i].run","text":"Here you can specify tasks which will run against your project after all images have been successfully built, but before:
This feature enables you to, for example, create a database dump before updating your application. This can make it easier to roll back in case of a problem with the deploy.
Info
The pre-rollout tasks run in the existing pods before they are updated, which means:
post_rollout.[i].run","text":"Here you can specify tasks which need to run against your project, after:
Common uses for post-rollout tasks include running drush updb, drush cim, or clearing various caches.
namecommand/app. Keep this in mind if you need to cd into a specific location to run your task.servicecontainernginx-php), you will need to specify which container in the pod to connect to (e.g. the php container within the nginx pod).shellsh is used, but if the container also has other shells (like bash, you can define it here). This is useful if you want to run some small if/else bash scripts within the post-rollouts. See the example below to learn how to write a script with multiple lines.whenNote: If you would like to temporarily disable pre/post-rollout tasks during a deployment, you can set either of the following environment variables in the API at the project or environment level (see how on Environment Variables).
LAGOON_PREROLLOUT_DISABLED=trueLAGOON_POSTROLLOUT_DISABLED=trueHere are some useful examples of post-rollout tasks that you may want to use or adapt for your projects.
Run only if Drupal not installed:
.lagoon.yml- run:\n name: IF no Drupal installed\n command: | # (1)\nif tables=$(drush sqlq \"show tables like 'node';\") && [ -z \"$tables\" ]; then\n#### whatever you like\nfi\nservice: cli\n shell: bash\nDifferent tasks based on branch name:
.lagoon.yml- run:\nname: Different tasks based on branch name\ncommand: |\n### Runs if current branch is not 'production'\nservice: cli\nwhen: LAGOON_GIT_BRANCH != \"production\"\nRun shell script:
.lagoon.yml- run:\nname: Run Script\ncommand: './scripts/script.sh'\nservice: cli\nTarget specific container in pod:
.lagoon.yml- run:\nname: show php env variables\ncommand: env\nservice: nginx\ncontainer: php\nDrupal & Drush 9: Sync database & files from master environment:
.lagoon.yml- run:\n name: Sync DB and Files from master if we are not on master\n command: |\n# Only if we don't have a database yet\nif tables=$(drush sqlq 'show tables;') && [ -z \"$tables\" ]; then\ndrush sql-sync @lagoon.master @self # (1)\ndrush rsync @lagoon.master:%files @self:%files -- --omit-dir-times --no-perms --no-group --no-owner --chmod=ugo=rwX\n fi\nservice: cli\n when: LAGOON_ENVIRONMENT_TYPE != \"production\"\nbackup-retention.production.monthly","text":"Specify the number of monthly backups Lagoon should retain for your project's production environment(s).
The global default is 1 if this value is not specified.
backup-retention.production.weekly","text":"Specify the number of weekly backups Lagoon should retain for your project's production environment(s).
The global default is 6 if this value is not specified.
backup-retention.production.daily","text":"Specify the number of daily backups Lagoon should retain for your project's production environment(s).
The global default is 7 if this value is not specified.
backup-retention.production.hourly","text":"Specify the number of hourly backups Lagoon should retain for your project's production environment(s).
The global default is 0 if this value is not specified.
backup-schedule.production","text":"Specify the backup schedule for this project. Accepts cron-compatible syntax with the notable exception that the Minute block must be the letter M. Any other value in the Minute block will cause the Lagoon build to fail. This allows Lagoon to randomly choose a specific minute for these backups to happen, while users can specify the remainder of the schedule down to the hour.
The global default is M H(22-2) * * * if this value is not specified. Take note that these backups will use the cluster's local timezone.
Environment names match your deployed branches or pull requests. This allows for each environment to have a different config. In our example it will apply to the main and staging environment.
environments.[name].routes","text":"Manual routes are domain names that are configured per environment to direct traffic to a service. Since all environments get automatically created routes by default, it is typical that manual routes are only setup for the production environment, using the main domain of the project's website like www.example.com.
Tip
Since Lagoon has no control over the manual routes, you'll need to ensure the DNS records are configured properly at your DNS provider. You can likely set a CNAME record to point to the automatic route.
The first element after the environment is the target service, nginx in our example. This is how we identify which service incoming requests will be sent to.
The simplest route is example.com, as seen in our example .lagoon.yml - you can see it has no additional configuration. This will assume that you want a Let's Encrypt certificate for your route and no redirect from HTTPS to HTTP.
In the \"www.example.com\" example below, we see three more options (also notice the : at the end of the route and that the route is wrapped in \", that's important!):
- \"www.example.com\":\ntls-acme: true\ninsecure: Redirect\nhstsEnabled: true\ntls-acme","text":"tls-acme: Configures automatic TLS certificate generation via Let's Encrypt. Default is true, set to false to disable automatic certificates.insecure: Configures HTTP connections. Default is Allow.Allow: Route will respond to HTTP and HTTPS.Redirect: Route will redirect any HTTP request to HTTPS.hstsEnabled: Adds the Strict-Transport-Security header. Default is false.hstsMaxAge: Configures the max-age directive. Default is 31536000 (1 year).hstsPreload: Sets the preload directive. Default is false.hstsIncludeSubdomains: Sets the includeSubDomains directive. Default is false.Info
If you plan to switch from a SSL certificate signed by a Certificate Authority (CA) to a Let's Encrypt certificate, it's best to get in touch with your Lagoon administrator to oversee the transition. There are known issues during the transition. The workaround would be manually removing the CA certificate and then triggering the Let's Encrypt process.
"},{"location":"using-lagoon-the-basics/lagoon-yml/#monitoring-a-specific-path","title":"Monitoring a specific path","text":"When UptimeRobot is configured for your cluster (Kubernetes or OpenShift), Lagoon will inject annotations to each route/ingress for use by the stakater/IngressControllerMonitor. The default action is to monitor the homepage of the route. If you have a specific route to be monitored, this can be overridden by adding a monitoring-path to your route specification. A common use is to set up a path for monitoring which bypasses caching to give a more real-time monitoring of your site.
- \"www.example.com\":\nmonitoring-path: \"/bypass-cache\"\nWarning
Route/Ingress annotations are only supported by projects that deploy into clusters that run nginx-ingress controllers! Check with your Lagoon administrator if this is supported.
annotations can be a YAML map of annotations supported by the nginx-ingress controller. This is specifically useful for easy redirects and other configurations.Some annotations are disallowed or partially restricted in Lagoon. The table below describes these rules.
If your .lagoon.yml contains one of these annotations it will cause a build failure.
nginx.ingress.kubernetes.io/auth-snippet Disallowed nginx.ingress.kubernetes.io/configuration-snippet Restricted to rewrite, add_header, set_real_ip, and more_set_headers directives. nginx.ingress.kubernetes.io/modsecurity-snippet Disallowed nginx.ingress.kubernetes.io/server-snippet Restricted to rewrite, add_header, set_real_ip, and more_set_headers directives. nginx.ingress.kubernetes.io/stream-snippet Disallowed nginx.ingress.kubernetes.io/use-regex Disallowed"},{"location":"using-lagoon-the-basics/lagoon-yml/#ingress-annotations-redirects","title":"Ingress annotations redirects","text":"In this example any requests to example.ch will be redirected to https://www.example.ch while keeping folders or query parameters intact (example.com/folder?query -> https://www.example.ch/folder?query).
- \"example.ch\":\nannotations:\nnginx.ingress.kubernetes.io/permanent-redirect: https://www.example.ch$request_uri\n- www.example.ch\nYou can of course also redirect to any other URL not hosted on Lagoon, this will direct requests to example.de to https://www.google.com
- \"example.de\":\nannotations:\nnginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com\nWarning
Kubernetes will only process a single nginx.ingress.kubernetes.io/server-snippet annotation. Please ensure that if you use this annotation on a non-production environment route that you also include the add_header X-Robots-Tag \"noindex, nofollow\"; annotation as part of your server-snippet. This is needed to stop robots from crawling development environments as the default server-snippet set to prevent this in development environments in the ingress templates will get overwritten with any server-snippets set in .lagoon.yml.
Some configurations involve a reverse proxy (like a CDN) in front of the Kubernetes clusters. In these configurations, the IP of the reverse proxy will appear as the REMOTE_ADDR HTTP_X_REAL_IP HTTP_X_FORWARDED_FOR headers field in your applications. The original IP of the requester can be found in the HTTP_X_ORIGINAL_FORWARDED_FOR header.
If you want the original IP to appear in the REMOTE_ADDR HTTP_X_REAL_IP HTTP_X_FORWARDED_FOR headers, you need to tell the ingress which reverse proxy IPs you want to trust:
- \"example.ch\":\nannotations:\nnginx.ingress.kubernetes.io/server-snippet: |\nset_real_ip_from 1.2.3.4/32;\nThis example would trust the CIDR 1.2.3.4/32 (the IP 1.2.3.4 in this case). Therefore if there is a request sent to the Kubernetes cluster from the IP 1.2.3.4 the X-Forwarded-For Header is analyzed and its contents injected into REMOTE_ADDR HTTP_X_REAL_IP HTTP_X_FORWARDED_FOR headers.
Environments.[name].types","text":"The Lagoon build process checks the lagoon.type label from the docker-compose.yml file in order to learn what type of service should be deployed (read more about them in the documentation of docker-compose.yml).
Sometimes you might want to override the type just for a single environment, and not for all of them. For example, if you want a standalone MariaDB database (instead of letting the Service Broker/operator provision a shared one) for your non-production environment called develop:
service-name: service-type
service-name is the name of the service from docker-compose.yml you would like to override.service-type the type of the service you would like to use in your override.Example for setting up MariaDB_Galera:
.lagoon.ymlenvironments:\ndevelop:\ntypes:\nmariadb: mariadb-single\nenvironments.[name].templates","text":"The Lagoon build process checks the lagoon.template label from the docker-compose.yml file in order to check if the service needs a custom template file (read more about them in the documentation of docker-compose.yml).
Sometimes you might want to override the template just for a single environment, and not for all of them:
service-name: template-file
service-name is the name of the service from docker-compose.yml you would like to override.template-file is the path and name of the template to use for this service in this environment.environments:\nmain:\ntemplates:\nmariadb: mariadb.main.deployment.yml\nenvironments.[name].rollouts","text":"The Lagoon build process checks the lagoon.rollout label from the docker-compose.yml file in order to check if the service needs a special rollout type (read more about them in the documentation of docker-compose.yml)
Sometimes you might want to override the rollout type just for a single environment, especially if you also overwrote the template type for the environment:
service-name: rollout-type
service-name is the name of the service from docker-compose.yml you would like to override.rollout-type is the type of rollout. See documentation of docker-compose.yml) for possible values.environments:\nmain:\nrollouts:\nmariadb: statefulset\nenvironments.[name].autogenerateRoutes","text":"This allows for any environments to get autogenerated routes when route autogeneration is disabled.
.lagoon.ymlroutes:\nautogenerate:\nenabled: false\nenvironments:\ndevelop:\nautogenerateRoutes: true\nenvironments.[name].cronjobs","text":"Cron jobs must be defined explicitly for each environment, since it is typically not desirable to run the same ones for all environments. Depending on the defined schedule, cron jobs may run as a Kubernetes native CronJob or as an in-pod cron job via the crontab of the defined service.
cronjobs:\n- name: Hourly Drupal Cron\nschedule: \"M * * * *\" # Once per hour, at a random minute.\ncommand: drush cron\nservice: cli\n- name: Nightly Drupal Cron\nschedule: \"M 0 * * *\" # Once per day, at a random minute from 00:00 to 00:59.\ncommand: drush cron\nservice: cli\nname: Any name that will identify the purpose and distinguish it from other cron jobs.schedule: The schedule for executing the cron job. Lagoon uses an extended version of the crontab format. If you're not sure about the syntax, use a crontab generator.
M for the minute, and your cron job will run once per hour at a random minute (the same minute each hour), or M/15 to run it every 15 mins, but with a random offset from the hour (like 6,21,36,51). It is a good idea to spread out your cron jobs using this feature, rather than have them all fire off on minute 0.H for the hour, and your cron job will run once per day at a random hour (the same hour every day), or H(2-4) to run it once per day within the hours of 2-4.Timezones:
command: The command to execute. This executes in the WORKDIR of the service. For Lagoon images, this is /app.Warning
Cronjobs may run in-pod, via crontab, which doesn't support multiline commands. If you need a complex or multiline cron command, you must put it in a script that can be used as the command. Consider whether a pre- or post-rollout task would work.
service: Which service of your project to run the command in. For most projects, this should be the cli service.In Lagoon, the same Git repository can be added to multiple projects, creating what is called a polysite. This allows you to run the same codebase, but allow for different, isolated, databases and persistent files. In .lagoon.yml , we currently only support specifying custom routes for a polysite project. The key difference from a standard project is that the environments becomes the second-level element, and the project name the top level.
To utilize this, you will need to:
poly-project1 and poly-project2 below)poly-project1:\nenvironments:\nmain:\nroutes:\n- nginx:\n- project1.com\npoly-project2:\nenvironments:\nmain:\nroutes:\n- nginx:\n- project2.com\napi","text":"Info If you run directly on amazee.io hosted Lagoon you will not need this key set.
With the key api you can define another URL that should be used by the Lagoon CLI and drush to connect to the Lagoon GraphQL API. This needs to be a full URL with a scheme, like: http://localhost:3000 This usually does not need to be changed, but there might be situations where your Lagoon administrator tells you to do so.
ssh","text":"Info If you run directly on amazee.io hosted Lagoon you will not need this key set.
With the key ssh you can define another SSH endpoint that should be used by the Lagoon CLI and drush to connect to the Lagoon remote shell service. This needs to be a hostname and a port separated by a colon, like: localhost:2020 This usually does not need to be changed, but there might be situations where your Lagoon administrator tells you to do so.
container-registries","text":"The container-registries block allows you to define your own private container registries to pull custom or private images. To use a private container registry, you will need a username, password, and optionally the url for your registry. If you don't specify a url in your YAML, it will default to using Docker Hub.
There are 2 ways to define the password used for your registry user.
Create an environment variable in the Lagoon API with the type container_registry:
lagoon add variable -p <project_name> -N <registry_password_variable_name> -V <password_goes_here> -S container_registryThe name of the variable you create can then be set as the password:
.lagoon.ymlcontainer-registries:\nmy-custom-registry:\nusername: myownregistryuser\npassword: <registry_password_variable_name>\nurl: my.own.registry.com\nYou can also define the password directly in the .lagoon.yml file in plain text:
container-registries:\ndocker-hub:\nusername: dockerhubuser\npassword: MySecretPassword\nTo consume a custom or private container registry image, you need to update the service inside your docker-compose.yml file to use a build context instead of defining an image:
services:\nmariadb:\nbuild:\ncontext: .\ndockerfile: Dockerfile.mariadb\nOnce the docker-compose.yml file has been updated to use a build, you need to create the Dockerfile.<service> and then set your private image as the FROM <repo>/<name>:<tag>
FROM dockerhubuser/my-private-database:tag\n.lagoon.yml","text":"This is an example .lagoon.yml which showcases all possible settings. You will need to adapt it to your project.
docker-compose-yaml: docker-compose.yml\nenvironment_variables:\ngit_sha: 'true'\ntasks:\npre-rollout:\n- run:\nname: drush sql-dump\ncommand: mkdir -p /app/web/sites/default/files/private/ && drush sql-dump --ordered-dump --gzip --result-file=/app/web/sites/default/files/private/pre-deploy-dump.sql.gz\nservice: cli\npost-rollout:\n- run:\nname: drush cim\ncommand: drush -y cim\nservice: cli\nshell: bash\n- run:\nname: drush cr\ncommand: drush -y cr\nservice: cli\nroutes:\nautogenerate:\ninsecure: Redirect\nenvironments:\nmain:\nroutes:\n- nginx:\n- example.com\n- example.net\n- \"www.example.com\":\ntls-acme: true\ninsecure: Redirect\nhstsEnabled: true\n- \"example.ch\":\nannotations:\nnginx.ingress.kubernetes.io/permanent-redirect: https://www.example.ch$request_uri\n- www.example.ch\ntypes:\nmariadb: mariadb\ntemplates:\nmariadb: mariadb.main.deployment.yml\nrollouts:\nmariadb: statefulset\ncronjobs:\n- name: drush cron\nschedule: \"M * * * *\" # This will run the cron once per hour.\ncommand: drush cron\nservice: cli\nstaging:\ncronjobs:\n- name: drush cron\nschedule: \"M * * * *\" # This will run the cron once per hour.\ncommand: drush cron\nservice: cli\nfeature/feature-branch:\ncronjobs:\n- name: drush cron\nschedule: \"H * * * *\" # This will run the cron once per hour.\ncommand: drush cron\nservice: cli\nThese settings have been deprecated and should be removed from use in your .lagoon.yml.
routes.autogenerate.insecureThe None option is equivalent to Redirect.
environments.[name].monitoring_urlsenvironments.[name].routes.[service].[route].hstsenvironments.[name].routes.[service].[route].insecureThe None option is equivalent to Redirect.
Even though Lagoon has only a hard dependency on Docker and Docker Compose (which is mostly shipped with Docker) there are some things which are nice for local development that are not included in Docker:
You do not need to install Lagoon locally to use it locally! That sounds confusing but follow the documentation. Lagoon is the system that deploys your local development environment to your production environment, it's not the environment itself.
"},{"location":"using-lagoon-the-basics/local-development-environments/#pygmy-or-lando-the-choice-is-yours","title":"pygmy or Lando - the choice is yours","text":"Lagoon has traditionally worked best with pygmy , which is the amazee.io flavored system of the above tools and works out of the box with Lagoon. It lives at https://github.com/pygmystack/pygmy
pygmy is written in Golang, so to install it, run:
brew tap pygmystack/pygmy && brew install pygmy\nFor detailed usage or installation info on pygmy, see its documentation.
As announced in our blog post, Lagoon is now also compatible with Lando! For more information, please see the documentation at https://docs.lando.dev/config/lagoon.html to get yourself up and running.
Lando's workflow for Lagoon will be familiar to users of Lando, and will also be the easiest way for Lagoon newcomers to get up and running. Pygmy presents a closer integration with Docker, which will lend itself better to more complex scenarios and use cases but will also require a deeper understanding.
We have previously evaluated adding support for other systems like Docksal and Docker4Drupal, and while we may add support for these in the future, our current focus is on supporting using Lando and pygmy. If you do have Lagoon running with one of these (or other) tools, we would love for you to submit a PR on GitHub!
"},{"location":"using-lagoon-the-basics/setup-project/","title":"Set Up a New Project","text":"Note
We are working hard on getting our CLI and GraphQL API set up to allow everyone using Lagoon to setup and configure their projects themselves. Right now, it needs more testing before we can release those features, so hold tight!
Until then, the setup of a new project involves talking to your Lagoon administrator, which is ok, as they are much friendlier than APIs. \ud83d\ude0a
Please have the following information ready for your Lagoon administrator:
--) are not allowed within a project namegit@example.com:test/test.git).We suggest deploying specific important branches (like develop and main) and pull requests. But that's all up to you! (see Workflows for some more information)
This means that the .lagoon.yml and docker-compose.yml files are available in your Git repository and configured accordingly.
If this is not the case, check out the list of Step-by-Step Guides on how to do so before proceeding.
"},{"location":"using-lagoon-the-basics/setup-project/#2-provide-access-to-your-code","title":"2. Provide access to your code","text":"In order to deploy your code, Lagoon needs access to it. By design and for security, Lagoon only needs read access to your Git repository.
Your Lagoon administrator will tell you the SSH public key or the Git account to give read access to.
"},{"location":"using-lagoon-the-basics/setup-project/#3-configure-webhooks","title":"3. Configure Webhooks","text":"Lagoon needs to be informed about a couple of events that are happening to your Git repository. Currently these are pushes and pull requests, but we may add more in the future.
As Lagoon supports many different Git hosts, we have split off those instructions into this documentation: Configure Webhooks.
"},{"location":"using-lagoon-the-basics/setup-project/#4-next-first-deployment","title":"4. Next: First deployment","text":"Congratulations, you are now ready to run your first deployment.
"}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml new file mode 100644 index 0000000000..7c0eb9ed29 --- /dev/null +++ b/sitemap.xml @@ -0,0 +1,558 @@ + +To change an existing project to support active/standby you'll need to configure some project settings with the Lagoon API.
+productionEnviromment should be set to the branch name of the current active environment.standbyProductionEnvironment should be set to the branch name of the current environment that is in standby.mutation updateProject {
+ updateProject(input:{
+ id:1234
+ patch:{
+ productionEnvironment:"production-brancha"
+ standbyProductionEnvironment:"production-branchb"
+ }
+ }){
+ standbyProductionEnvironment
+ name
+ productionEnvironment
+ }
+}
+.lagoon.yml - production_routes#To configure a project for active/standby in the .lagoon.yml file, you'll need to configure the production_routes section with any routes you want to attach to the active environment, and any routes to the standby environment. During an active/standby switch, these routes will migrate between the two environments.
If you have two production environments, production-brancha and production-branchb, with the current active production environment as production-brancha then:
production_routes.active will direct you to production-brancha.production_routes.standby will direct you to production-branchb.During an active/standby switch, the routes will swap:
+production_routes.active will direct you to production-branchb.production_routes.standby will direct you to production-brancha.production_routes:
+ active:
+ routes:
+ - nginx:
+ - example.com:
+ tls-acme: 'false'
+ - active.example.com:
+ tls-acme: 'false'
+ standby:
+ routes:
+ - nginx:
+ - standby.example.com:
+ tls-acme: 'false'
+Info
+Any routes that are under the section environments..routes will not be moved as part of active/standby. These routes will always be attached to the environment as defined. Ensure that if you do need a specific route to be migrated during an active/standby switch, that you remove them from the environments section and place them under the production_routes section specific to if it should be an active or standby route. See more about routes in .lagoon.yml.
To trigger the switching of environment routes, you can visit the standby environment in the Lagoon UI and click on the button labeled Switch Active/Standby environments. You will be prompted to confirm your action.
Once confirmed, it will take you to the tasks page where you can view the progress of the switch.
+To trigger an event to switch the environments, run the following GraphQL mutation. This will tell Lagoon to begin the process.
+mutation ActiveStandby {
+ switchActiveStandby(
+ input:{
+ project:{
+ name:"drupal-example"
+ }
+ }
+ ){
+ id
+ remoteId
+ }
+}
+A task is created in the current active environment tasks tab when a switch event is triggered. You can check the status of the switch here.
Using the remoteId from the switchActiveStandby mutation, we can also check the status of the task.
query getTask {
+ taskByRemoteId(id: "<remoteId>") {
+ id
+ name
+ created
+ started
+ completed
+ status
+ logs
+ }
+}
+drush aliases#By default, projects will be created with the following aliases that will be available when active/standby is enabled on a project.
+lagoon-productionlagoon-standbyThe lagoon-production alias will point to whichever site is defined as productionEnvironment, and lagoon-standby will always point to the site that is defined as standbyProductionEnvironment.
These aliases are configurable by updating the project. Be aware that changing them may require you to update any scripts that rely on them.
+mutation updateProject {
+ updateProject(input:{
+ id:1234
+ patch:{
+ productionAlias:"custom-lagoon-production-alias"
+ standbyAlias:"custom-lagoon-standby-alias"
+ }
+ }){
+ productionAlias
+ name
+ standbyAlias
+ }
+}
+You need to decide which of these 2 branches are the one you want to go forward with as being the main environment and then ensure it is set as the active branch (e.g production-branchb).
.lagoon.yml file in this (now active) branch, move the routes from the production_routes.active.routes section into the environments.production-branchb section. This will mean that they are then attached to the production-branchb environment only..lagoon.yml file and re-deploy the production-branchb environment.production-brancha, you can delete it.production_routes from that branch .lagoon.yml too, just to prevent any confusion. The branch will remain as production type unless you delete and redeploy it (wiping all storage and databases, etc).production-branchb production environment, and all the other environments are development, update the project to remove the standbyProductionEnvironment from the project so that the active/standby labels on the environments go away.mutation updateProject {
+ updateProject(input:{
+ id:1234
+ patch:{
+ productionEnvironment:"production-branchb"
+ standbyProductionEnvironment:""
+ }
+ }){
+ standbyProductionEnvironment
+ name
+ productionEnvironment
+ }
+}
+When the active/standby trigger has been executed, the productionEnvironment and standbyProductionEnvironments will switch within the Lagoon API. Both environments are still classed as production environment types. We use the productionEnvironment to determine which one is labelled as active. For more information on the differences between environment types, read the documentation for environment types
query projectByName {
+ projectByName(name:"drupal-example"){
+ productionEnvironment
+ standbyProductionEnvironment
+ }
+}
+Before switching environments:
+{
+ "data": {
+ "projectByName": {
+ "productionEnvironment": "production-brancha",
+ "standbyProductionEnvironment": "production-branchb"
+ }
+ }
+}
+After switching environments:
+ + + + + + + + + +Lagoon makes use of the k8up operator to provide backup functionality for both database data as well as containers' persistent storage volumes. This operator utilizes Restic to catalog these backups, which is typically connected to an AWS S3 bucket to provide secure, off-site storage for the generated backups.
+Backups of databases and containers' persistent storage volumes happens nightly within production environments by default.
+If a different backup schedule for production backups is required, this can be specified at a project level via setting the "Backup Schedule" variables in the project's .lagoon.yml file.
+Production environment backups will be held according to the following schedule by default:
+If a different retention period for production backups is required, this can be specified at a project level via setting the "Backup Retention" variables in the project's .lagoon.yml file.
+Backups of development environments are attempted nightly and are strictly a best effort service.
+Backups stored in Restic will be tracked within Lagoon, and can be recovered via the "Backup" tab for each environment in the Lagoon UI.
+Lagoon supports custom backup and restore locations via the use of the "Custom Backup Settings" and/or "Custom Restore Settings" variables stored in the Lagoon API for each project.
+Danger
+Proceed with caution: Setting these variables will override backup/restore storage locations that may be configured at a cluster level. Any misconfiguration will cause backup/restore failures.
+A base image is a Docker image that can be and is used by a project deployed on Lagoon. A base image provides a way to ensure that nothing is brought into the codebase/project from upstream that has not been audited. It also allows us to ensure that anything we might need on the deployed environment is available - from lower-level libraries to application-level themes and modules.
+Base images save time and resources when you know what system is being deployed to - if shared packages are included in the base image, they don’t have to be deployed to hundreds of sites individually.
+A derived image is one that extends a base image. For example, you might need to make several blog sites. You take our Drupal image, customize it to include all of the modules and themes you need for your blog sites, and deploy them all with that blog image. Templates are derived from base images.
+All derived images should pull in the composer.json file (via repositories like Packagist, Satis, or GitHub) so that they are using the most recent versions of the base packages.
Further, the derived image includes a call to the script /build/pre_composer, which can be used by the base image to run scripts, updates, etc., downstream in the derived images. For instance, it should run by default when any package is updated or installed at the derived image, and the pre_composer script will then update the base image package.
Info
+This document will talk about Drupal and Laravel base images as examples, as it was originally written for a client who uses those technologies in their Lagoon projects. It will be expanded to cover the contents of other base images, but none of the processes differ, no matter what the content of your base image.
+Base images are managed with Composer and hosted in BitBucket, GitHub, or GitLab (whatever your team is using). Each base image has its own repository.
+The metapackage is a Composer package that wraps several other components. These include, for example, the core files for Laravel or Drupal, along with any needed modules or themes. This way, you do not need to include Laravel or Drupal, etc., as a dependency in your project.
+Here’s an example from the composer.json in a Laravel base image:
We only require this metapackage, which points to a GitHub repository.
+docker-compose.yml#Other pieces of your project are defined in docker-compose.yml. For example, if you have a Drupal project, you need the Drupal image, but you also need MariaDB, Solr, Redis, and Varnish. We have versions of these services optimized for Drupal, all of which are included in docker-compose.yml.
The Drupal base image contains the following contributed tools and modules, in addition to Drupal core:
+The base images have provided the default values for the environment variables used by Laravel.
+These are values for:
+DB_CONNECTIONDB_HOSTDB_PORTDB_DATABASEDB_USERNAMEDB_PASSWORDREDIS_HOSTREDIS_PASSWORDREDIS_PORTEnsure that your config files (typically located in /config) make use of these by default.
If your project makes use of queues, you can make use of the artisan-worker service. It is a worker container, used for executing artisan queue:work. This is disabled by default - look at the comments in docker-compose.yml.
There are several parts to the process of building a base image. All of the major steps are represented in the Makefile. The Jenkinsfile contains a more stripped-down view. Taking a look at both files will give you a good understanding of what happens during this process. Most steps can be tested locally (this is important when building new versions of the base image). After you’ve created and tested everything locally and pushed it up, the actual base image is built by Jenkins and pushed to Harbor.
+If you're planning on running locally, there are some minimum environment variables that need to be present to build at all.
+Variables injected into the base image build process and where to find them.
+BUILD_NUMBER - This is injected by Jenkins automatically.GIT_BRANCH - This is provided by the Jenkins build process itself. Depends on the branch being built at the time (develop, main, etc.).DOCKER_REPO/DOCKER_HUB - This is defined inside the Jenkinsfile itself. It points to the Docker project and hub into which the resulting images will be pushed.DOCKER_USERNAME/DOCKER_PASSWORD - These are used to actually log into the Docker repository early in the build. These variables are stored inside of the Jenkins credentials. These are used in the Jenkinsfile itself and are not part of the Makefile. This means that if you’re building base images outside of Jenkins (i.e. locally, to test, etc.) you have to run a docker login manually before running any of the make steps.In practice, this means that if you're running any of the make targets on your local machine, you'll want to ensure that these are available in the environment - even if this is just setting them when running make from the command line, as an example:
GIT_BRANCH=example_branch_name DOCKER_HUB=the_docker_hub_the_images_are_pushed_to DOCKER_REPO=your_docker_repo_here BUILD_NUMBER=<some_integer> make images_remove
+The most important targets are the following:
+images_build : Given the environment variables, this will build and tag the images for publication.images_publish : Pushes built images to a Docker repository.images_start : Will start the images for testing, etc.images_test: Runs basic tests against images.images_remove: Removes previously built images, given the build environment variables.There are several steps to the build process. Most of these are shared among the various base images. These mostly correspond to the Makefile target described above.
+make images_build step is run now, which will:docker-compose build. This will produce several new Docker images from the current Git branch.make images_test target, which will differ depending on the images being tested. In most cases this is a very straightforward test to ensure that the images can be started and interacted with in some way (installing Drupal, listing files, etc.)images_publish) that will tag the images resulting from the Docker Build in Step 2 and push them to Harbor. This is described in more detail elsewhere in this guide.images_remove, which simply deletes the newly built images from the Docker host now that they are in Harbor.There are many reasons to release a new version of a base image. On Drupal or Laravel, Node.js, etc. images, it may be in order to upgrade or install a module/package for features or security. It may be about the underlying software that comes bundled in the container, such as updating the version of PHP or Node.js. It may be about updating the actual underlying images on which the base images are built.
+The images that your project's base images are built on are the managed images maintained by the Lagoon team. We release updates to these underlying images on a monthly (or more fequent) basus. When these are updated, you need to build new versions of your own base images in order to incorporate the changes and upgrades bundled in the upstream images.
+In this section we will demonstrate the process of updating and tagging a new release of the Drupal 8 base image. We will add a new module (ClamAV) to the base. We’re demonstrating on Drupal because it has the most complex setup of the base images. The steps that are common to every base image are noted below.
+This is just pulling down the Git repository locally. In the case of the Drupal 8 base image. In this example, we're using Bitbucket, so we will run:
+ +
Info
+What is demonstrated here is specific to the Drupal 8 base image. However, any changes (adding files, changing base Docker images, etc.) will be done in this step for all of the base images.
+In our example, we are adding the ClamAV module to the Drupal 8 base image. This involves a few steps. The first is requiring the package so that it gets added to our composer.json file. This is done by running a composer require.
Here we run:
+ +
When the Composer require process completes, the package should then appear in the composer.json file.
Here we open the composer.json file and take a look at the list of required packages, and check that the ClamAV package is listed, and see that it is there:
For any modules now added to the base image, we need to ensure that they’re enabled on the template-based derived images. This is done by adding the module to the Lagoon Bundle module located at ./web/modules/lagoon/lagoon_bundle. Specifically, it requires you to add it as a dependency to the dependencies section of the lagoon_bundle.info.yml file. The Lagoon Bundle module is a utility module that exists only to help enforce dependencies across derived images.
Here we open web/modules/contrib/lagoon/lagoon_bundle/lagoon_bundle.info.yml and add clamav:clamav as a dependency:
Adding a dependency to this will ensure that whenever the Lagoon Bundle module is enabled on the derived image, its dependencies (in this case, the just-added ClamAV module) will also be enabled. This is enforced by a post-rollout script which enables lagoon_bundle on the derived images when they are rolled out.
This will depend on what you’re testing. In the case of adding the ClamAV module, we want to ensure that in the base image, the module is downloaded, and that the Lagoon Bundle module enables ClamAV when it is enabled.
+Here we check that the module is downloaded to /app/web/modules/contrib:
And then we check that when we enable the lagoon_bundle module, it enables clamav by running:
Warning
+You’ll see that there is a JWT error in the container above. You can safely ignore this in the demonstration above - but, for background, you will see this error when there is no Lagoon environment for the site you’re working on.
+With our testing done, we can now tag and build the images.
+Images are versioned based on their Git tags - these should follow standard semantic versioning (semver) practices. All tags should have the structure vX.Y.Z where X, Y, and Z are integers (to be precise the X.Y.Z are themselves the semantic version - the vX.Y.Z is a tag). This is an assumption that is used to determine the image tags, so it must be adhered to.
+In this example we will be tagging a new version of the Drupal 8 base image indicating that we have added ClamAV.
+We check that we have committed (but not pushed) our changes, just as you would do for any regular commit and push, using git log.
git tag.git tag -a v0.0.9 -m “Adds clamAV to base.”git push --tags.git push.Danger
+The tags must be pushed explicitly in their own step!
+
Danger
+Depending on the build workflow, you will almost certainly push the changes via the develop branch before merging it into the main branch.
+An important point to remember here is that the Jenkins base image build process will tag images based on the most recent commit’s tag.
+Images are tagged using the following rules, and images will be built for each of these that apply:
+latest.development.Info
+Generally you will have a trigger strategy set up here for automatic builds, but as that will differ based on your needs and setup, this explains how to build manually.
+
This will kick off the build process which, if successful, will push up the new images to Harbor.
+If the build is not successful, you can click into the build itself and read the logs to understand where it failed.
+As shown in the screenshot below from Harbor, the image we’ve just built in Jenkins has been uploaded and tagged in Harbor, where it will now be scanned for any vulnerabilities. Since it was tagged as v0.0.9, an image with that tag is present, and because we built the main branch, the “latest” image has also been built. At this stage, the v0.0.9 and “latest” images are identical.
+
The base image structure draws heavily (and, in fact, is a fork of) Denpal. It is based on the original Drupal Composer Template, but includes everything necessary to run on Lagoon (either the local development environment or on hosted Lagoon).
+ + + + + + + + +The Lagoon Base Images have support for Blackfire included in the PHP Images (see the PHP images).
+In order to use Blackfire in Lagoon, these three environment variables need to be defined:
+| Environment Variable | +Default | +Description | +
|---|---|---|
BLACKFIRE_ENABLED |
+(not set) | +Used to enable blackfire extension with setting variable to TRUE or true |
+
BLACKFIRE_SERVER_ID |
+(not set) | +Set to Blackfire Server ID provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true |
+
BLACKFIRE_SERVER_TOKEN |
+(not set) | +Set to Blackfire Server Token provided by Blackfire.io. Needs BLACKFIRE_ENABLED set to true |
+
For local usage of Blackfire with Lagoon Images, set the above environment variables for the PHP container. Here is an example for a Drupal application:
+services:
+
+[[snip]]
+
+ php:
+ [[snip]]
+
+ environment:
+ << : *default-environment # loads the defined environment variables from the top
+ BLACKFIRE_ENABLED: TRUE
+ BLACKFIRE_SERVER_ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+ BLACKFIRE_SERVER_TOKEN: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+After restarting the containers, you should be able to profile via the Blackfire Browser Plugin or the Blackfire CLI.
+In order to use Blackfire in deployed Lagoon environments the same enviornment variables need to be set, this time via one of the possibilities of adding environment variables to Lagoon. Important: Environment variables set in the docker-compose.yml for local development are not used by Lagoon in remote environments!
The Blackfire Agent running in the PHP containers outputs logs as normal container logs, which can be seen via docker-compose logs or via the Lagoon Logging Infrastructure for remote environments.
By default the Logs are set to Level 3 (info), via the environment variable BLACKFIRE_LOG_LEVEL the level can be increased to 4 (debug) to generate more debugging ouput.
Lagoon allows for the definition of custom tasks at environment, project, and group levels. This is presently accomplished through the GraphQL API and exposed in the UI.
+When defining a task you need to determine a number of things.
+In most cases, the custom task you will be running will be something that will be run in a shell on one of the containers in your application.
+For instance, in a Node.js application, you may be interested in running a yarn audit in your node container. The command, in this case, would simply be yarn audit.
We have to define where this task will be run -- this means two things, first, which project or environment we'll be running the task in, and, second, which service.
+Let's say that we'd like for our yarn audit task to be available to run in any environment in a specific project (let's say the project's ID is 42 for this example). We will therefore specify the project's ID when we create our task definition, as we will describe below.
The second question regards which environment we want to target with our task. When you set up your project, you specify several services in your docker-compose.yml. We use this service name to determine where the command is actually executed.
There are three levels of permissions to the task system corresponding to project roles. Guest, Developer, and Maintainer -- from most restrictive to least restrictive, with each role being able to invoke the tasks defined for the lower role (Developer can see Guest tasks, Maintainers can see all tasks).
+Tasks are defined by calling the addAdvancedTaskDefinition mutation. Importantly, this simply defines the task, it does not invoke it. It simply makes it avaliable to be run in an environment.
Schematically, the call looks like this
+mutation addAdvancedTask {
+ addAdvancedTaskDefinition(input:{
+ name: string,
+ confirmationText: string,
+ type: [COMMAND|IMAGE],
+ [project|environment]: int,
+ description: string,
+ service: string,
+ command: string,
+ advancedTaskDefinitionArguments: [
+ {
+ name: "ENVIROMENT_VARIABLE_NAME",
+ displayName: "Friendly Name For Variable",
+ type: [STRING | ENVIRONMENT_SOURCE_NAME | ENVIRONMENT_SOURCE_NAME_EXCLUDE_SELF]
+ }
+ ]
+ }) {
+ ... on AdvancedTaskDefinitionImage {
+ id
+ name
+ description
+ service
+ image
+ confirmationText
+ advancedTaskDefinitionArguments {
+ type
+ range
+ name
+ displayName
+ }
+ ...
+ }
+ ... on AdvancedTaskDefinitionCommand {
+ id
+ name
+ description
+ service
+ command
+ advancedTaskDefinitionArguments {
+ type
+ range
+ name
+ displayName
+ }
+ ...
+ }
+ }
+}
+Fields name and description are straightforward. They're simply the name and description of the task - these are used primarily in the UI.
The type field needs some explanation - for now, only platform admins are able to define IMAGE type commands - these allow for the running of specifically created task images as tasks, rather than targeting existing services. Most tasks, though, will be COMMAND types.
The [project|environment] set of fields will attach the task to either the project or environment (depending on the key you use), with the value being the id. In the case we're considering for our yarn audit we will specify we're targeting a project with an ID of 42.
We put the service we'd like to target with our task in the service field, and command is the actual command that we'd like to run.
In order to give more flexibility to the users invoking the tasks via the Lagoon UI, we support defining task arguments. These arguments are displayed as text boxes or drop downs and are required for the task to be invoked.
+Here is an example of how we might set up two arguments.
+advancedTaskDefinitionArguments: [
+ {
+ name: "ENV_VAR_NAME_SOURCE",
+ displayName: "Environment source",
+ type: ENVIRONMENT_SOURCE_NAME
+
+ },
+ {
+ name: "ENV_VAR_NAME_STRING",
+ displayName: "Echo value",
+ type: STRING
+ }
+ ]
+ })
+This fragment shows both types of arguments the system currently supports.
+The first, ENV_VAR_NAME_SOURCE is an example of type ENVIRONMENT_SOURCE_NAME, which will present the user of the UI a dropdown of the different environments inside of a project. If we don't want to allow the task to be run on the invoking environment (say, if we want to import a database from another environment), we can restrict the environment list by using ENVIRONMENT_SOURCE_NAME_EXCLUDE_SELF.
+The second ENV_VAR_NAME_STRING is of type STRING and will present the user with a textbox to fill in.
The values that the user selects will be available as environment variables in the COMMAND type tasks when the task is run.
When the confirmationText field has text, it will be displayed with a confirmation modal in the UI before the user is able to run the task.
With the task now defined, the task should now show up in the tasks dropdown in the Lagoon UI.
+We are also able to invoke it via the GraphQL api by using the invokeTask mutation.
mutation invokeTask {
+ invokeRegisteredTask(advancedTaskDefinition: int, environment: int) {
+ status
+ }
+}
+Note that invokeTask will always invoke a task on a specific environment.
Let's now setup our yarn audit example.
mutation runYarnAudit {
+ addAdvancedTaskDefinition(input:{
+ name:"Run yarn audit",
+ project: 42,
+ type:COMMAND,
+ permission:DEVELOPER,
+ description: "Runs a 'yarn audit'",
+ service:"node",
+ command: "yarn audit"})
+ {
+ id
+ }
+}
+This, then, will define our task for our project (42). When we run this, we will get the ID of the task definition back (for argument's sake, let's say it's 9)
This task will now be available to run from the UI for anyone with the DEVELOPER or MAINTAINER role.
Danger
+This is an alpha feature in Lagoon. +The way DeployTarget Configurations work could change in future releases. +If you decide to use this feature, you do so at your own risk.
+DeployTarget configurations are a way to define how a project can deploy to multiple clusters. This feature is useful when you have two clusters, one which could be dedicated for running production workloads, and another that is used for running development workloads.
+The configuration for these is not limited to just a production/development split though, so projects could perceivably target more than one specific cluster.
+The basic idea of a DeployTarget configuration is that it is a way to easily define how a project can deploy across multiple clusters. It uses the existing methods of checking if a environment is valid
+Before going in to how to configure a project to leverage DeployTarget configurations, there are some things you need to know.
+Environments now have two new fields available to them to identify which DeployTarget(Kubernetes or OpenShift) they have been created on.
+kubernetesNamespacePatternkubernetesOnce an environment has been deployed to a specific DeployTarget, it will always deploy to this target, even if the DeployTarget configuration, or project configuration is modified.
+By default, if no DeployTarget configurations are associated to a project, that project will continue to use the existing methods to determine which environments to deploy. These are the following fields used for this.
+branchespullrequestskubernetesNamespacePatternkubernetesAs soon as any DeployTarget configurations are added to a project, then all future deployments for this project will use these configurations. What is defined in the project is ignored, and overwritten to inform users that DeployTarget configurations are in use.
+DeployTarget configurations are weighted, which means that a DeployTarget configuration with a larger weight is prioritized over one with lower weight.
+1. The order in which they are returned by the query is the order they are used to determine where an environment should be deployed.
+Active/Standby environments can only be deployed to the same cluster, so your DeployTarget configuration must be able to deploy both those environments to the same target.
+Projects that leverage the promote feature of Lagoon must be aware that DeployTarget configurations are ignored for the destination environment.
source environment is on, your DeployTarget configuration MUST be configured correctly for this source environment.source and destination environment in the same DeployTarget configuration branch regex.To configure a project to use DeployTarget configurations, the first step is to add a configuration to a project.
+The following GraphQL mutation can be used, this particular example will add a DeployTarget configuration to the project with the project ID 1.
+It will allow only the branches that match the name main to be deployed, and pullrequests is set to false.
+This means no other branches will be able to deploy to this particular target, and no pull requests will be deployed to this particular target.
+The deployTarget is ID 1, this could be a Kubernetes cluster in a specific region, or designated for a specific type of workload (production or development).
mutation addDeployTargetConfig{
+ addDeployTargetConfig(input:{
+ project: 1
+ branches: "main"
+ pullrequests: "false"
+ deployTarget: 1
+ weight: 1
+ }){
+ id
+ weight
+ branches
+ pullrequests
+ deployTargetProjectPattern
+ deployTarget{
+ name
+ id
+ }
+ project{
+ name
+ }
+ }
+}
+Info
+deployTarget is an alias the Kubernetes or OpenShift ID in the Lagoon API
It is also possible to configure multiple DeployTarget configurations.
+The following GraphQL mutation can be used, this particular example will add a DeployTarget configuration to the same project as above.
+It will allow only the branches that regex match with ^feature/|^(dev|test|develop)$ to be deployed, and pullrequests is set to true so all pull requests will reach this target.
The targeted cluster in this example is ID 2, which is a completely different Kubernetes cluster to what was defined above for the main branch.
mutation addDeployTargetConfig{
+ addDeployTargetConfig(input:{
+ project: 1
+ branches: "^feature/|^(dev|test|develop)$"
+ pullrequests: "true"
+ deployTarget: 2
+ weight: 1
+ }){
+ id
+ weight
+ branches
+ pullrequests
+ deployTargetProjectPattern
+ deployTarget{
+ name
+ id
+ }
+ project{
+ name
+ }
+ }
+}
+Once these have been added to a project, you can return all the DeployTarget configurations for a project using the following query
+query deployTargetConfigsByProjectId{
+ deployTargetConfigsByProjectId(project:1){
+ id
+ weight
+ branches
+ pullrequests
+ deployTargetProjectPattern
+ deployTarget{
+ name
+ id
+ }
+ project{
+ name
+ }
+ }
+}
+# result:
+{
+ "data": {
+ "deployTargetConfigsByProjectId": [
+ {
+ "id": 1,
+ "weight": 1,
+ "branches": "main",
+ "pullrequests": "false",
+ "deployTargetProjectPattern": null,
+ "deployTarget": {
+ "name": "production-cluster",
+ "id": 1
+ },
+ "project": {
+ "name": "my-project"
+ }
+ },
+ {
+ "id": 2,
+ "weight": 1,
+ "branches": "^feature/|^(dev|test|develop)$",
+ "pullrequests": "true",
+ "deployTargetProjectPattern": null,
+ "deployTarget": {
+ "name": "development-cluster",
+ "id": 2
+ },
+ "project": {
+ "name": "my-project"
+ }
+ }
+ ]
+ }
+}
+Lagoon can utilize the Aergia controller, (installed in the lagoon-remote) to automatically idle environments if they have been unused for a defined period of time. This is done in order to reduce the load on the Kubernetes clusters and improve the overall performance of production environments and development environments that are actually in use.
The environment idler has many different configuration capabilities. Here are the defaults of a standard Lagoon installation (these could be quite different in your Lagoon, check with your Lagoon administrator!)
+Aergia will automatically un-idle an environment as soon as it is visited, therefore just visiting any URL of the environment will start the environment. Likewise, initiating an SSH session to the environment will also restart the services.
+The un-idling will take a couple of seconds, as the Kubernetes cluster needs to start all containers again. During this time there will be waiting screen shown to the visitor that their environment is currently started.
+Yes, there is a field autoIdle on the project (impacts all environments) and environment (if you need to target just one environment), as to whether idling is allowed to take place. A value of 1 indicates the project/environment is eligible for idling. If the project is set to 0 the environments will never be idled, even if the environment is set to 0
+The default is always 1(idling is enabled).
Talk to your Lagoon administrator if you are unsure how to set these project/environment fields.
+ + + + + + + + +Lagoon currently differentiates between two different environment types: production and development.
When setting up your project via the Lagoon GraphQL API, you can define a productionEnvironment. On every deployment Lagoon executes, it checks if the current environment name matches what is defined in productionEnvironment. If it does, Lagoon will mark this environment as the production environment. This happens in two locations:
LAGOON_ENVIRONMENT_TYPE in every container.But that's it. Lagoon itself handles development and production environments in exactly the same way (in the end we want as few differences of the environments as possible - that's the beauty of Lagoon).
There are a couple of things that will use this information:
+development environments are idled after 4 hours with no hits (don't worry, they wake up automatically). It is also possible for your Lagoon administrator to disable auto-idling on a per-environment basis, just ask!settings.php files load additional settings files for development.settings.php and production.settings.php so you can define settings and configurations different per environment type.productionEnvironment in the API or use the secret forceDeleteProductionEnvironment: true POST payload for the REST API.It is common to store API tokens or credentials for applications in environment variables.
+Following best practices, those credentials are different per environment. We allow each environment to use a separate set of environment variables defined in environment variables or environment files.
+As there can be environment variables defined in either the Dockerfile or during runtime (via API environment variables), we have a hierarchy of environment variables: environment variables defined in lower numbers are stronger.
+ENV command)..lagoon.env.$LAGOON_GIT_BRANCH or .lagoon.env.$LAGOON_GIT_SAFE_BRANCH (if the file exists and where $LAGOON_GIT_BRANCH $LAGOON_GIT_SAFE_BRANCH are the name and safe name of the branch this Docker image has been built for), use this for overwriting variables for only specific branches..lagoon.env (if it exists), use this for overwriting variables for all branches..env..env.defaults..lagoon.env.$LAGOON_GIT_BRANCH, .lagoon.env.$LAGOON_GIT_SAFE_BRANCH, .env, and .env.defaults are all sourced by the individual containers themselves as part of running their entrypoint scripts. They are not read by Lagoon, but by the containers ENTRYPOINT scripts, which look for them in the containers working directory. If environment variables don't appear as expected, check if your container has a WORKDIR setting that points to somewhere else.
We suggest using the Lagoon API environment variable system for variables that you don't want to keep in your Git repository (like secrets or API keys), as they could be compromised by somebody having them on their local development environment or on the internet, etc.
+The Lagoon API allows you to define project-wide or environment-specific variables. Additionally, they can be defined for a scope-only build-time or runtime. They are all created via the Lagoon GraphQL API. Read more on how to use the GraphQL API in our GraphQL API documentation.
+Runtime environment variables are automatically made available in all containers, but they are only added or updated after an environment has been re-deployed.
+This defines a project wide runtime variable (available in all environments) for the project with ID 463:
mutation addRuntimeEnv {
+ addEnvVariable(
+ input:{
+ type:PROJECT,
+ typeId:463,
+ scope:RUNTIME,
+ name:"MYVARIABLENAME",
+ value:"MyVariableValue"
+ }
+ ) {
+ id
+ }
+}
+This defines a environment ID 546 specific runtime variable (available only in that specific environment):
mutation addRuntimeEnv {
+ addEnvVariable(
+ input:{
+ type:ENVIRONMENT,
+ typeId:546,
+ scope:RUNTIME,
+ name:"MYVARIABLENAME",
+ value:"MyVariableValue"
+ }
+ ) {
+ id
+ }
+}
+Build-time environment variables are only available during a build and need to be consumed in Dockerfiles via:
+ +Typically theARG will go after the FROM. Read the docker documentation about ARG and FROM.
+This defines a project-wide build-time variable (available in all environments) for the project with ID 463:
mutation addBuildtimeEnv {
+ addEnvVariable(
+ input:{
+ type:PROJECT,
+ typeId:463,
+ scope:BUILD,
+ name:"MYVARIABLENAME",
+ value:"MyVariableValue"}
+ ) {
+ id
+ }
+}
+This defines an environment ID 546specific build-time variable (available only in that specific environment):
mutation addBuildtimeEnv {
+ addEnvVariable(input:{type:ENVIRONMENT, typeId:546, scope:BUILD, name:"MYVARIABLENAME", value:"MyVariableValue"}) {
+ id
+ }
+}
+Container registry environment variables are only available during a build and are used when attempting to log in to a private registry. They are used to store the password for the user defined in Specials » container-registries. They can be applied at the project or environment level.
This defines a project-wide container registry variable (available in all environments) for the project with ID 463:
mutation addContainerRegistryEnv {
+ addEnvVariable(
+ input:{
+ type:PROJECT,
+ typeId:463,
+ scope:CONTAINER_REGISTRY,
+ name:"MY_OWN_REGISTRY_PASSWORD",
+ value:"MySecretPassword"})
+ ) {
+ id
+ }
+}
+This defines a environment ID 546 specific container registry variable (available only in that specific environment):
mutation addContainerRegistryEnv {
+ addEnvVariable(
+ input:{
+ type:ENVIRONMENT,
+ typeId:546,
+ scope:CONTAINER_REGISTRY,
+ name:"MY_OWN_REGISTRY_PASSWORD",
+ value:"MySecretPassword"}
+ ) {
+ id
+ }
+}
+If you have environment variables that can safely be saved within a Git repository, we suggest adding them directly into the Git repository in an environment file. These variables will also be available within local development environments and are therefore more portable.
+The syntax in the environment files is as following:
+MYVARIABLENAME="MyVariableValue"
+MVARIABLENUMBER=4242
+DB_USER=$DB_USERNAME # Redefine DB_USER with the value of DB_USERNAME e.g. if your application expects another variable name for the Lagoon-provided variables.
+.lagoon.env.$BRANCHNAME#If you want to define environment variables different per environment you can create a .lagoon.env.$BRANCHNAME e.g. for the main branch .lagoon.env.main. This helps you keep environment variables apart between environments.
.env and .env.defaults#.env and .env.defaults will act as the default values for environment variables if none other is defined. For example, as default environment variables for pull request environments (see Workflows).
PHP_ERROR_REPORTING#This variable, if set, will define the logging level you would like PHP to use. If not supplied, it will be set dynamically based on whether this is a production or development environment.
+On production environments, this value defaults to E_ALL & ~E_DEPRECATED & ~E_STRICT & ~E_NOTICE.
On development environments, this value defaults to E_ALL & ~E_DEPRECATED & ~E_STRICT.
Lagoon supports custom backup locations and credentials for any project when all four of the following variables are set as BUILD type variables. The environment variables need to be set at the project level (not per environment), and requires a Lagoon deployment after setting them (for every environment).
Please note that any use of these variables means that all environment and database backups created and managed by Lagoon will be stored using these credentials, meaning that any interruption of these credentials' may lead to failed or inaccessible backups.
+| Environment variable name | +Purpose | +
|---|---|
LAGOON_BAAS_CUSTOM_BACKUP_ENDPOINT |
+Specify the S3 compatible endpoint where any Lagoon initiated backups should be stored. An example for S3 Sydney would be: https://s3.ap-southeast-2.amazonaws.com. |
+
LAGOON_BAAS_CUSTOM_BACKUP_BUCKET |
+Specify the bucket name where any Lagoon initiated backups should be stored.An example custom setting would be: example-restore-bucket. |
+
LAGOON_BAAS_CUSTOM_BACKUP_ACCESS_KEY |
+Specify the access key Lagoon should use to access the custom backup bucket. An example custom setting would be: AKIAIOSFODNN7EXAMPLE. |
+
LAGOON_BAAS_CUSTOM_BACKUP_SECRET_KEY |
+Specify the secret key Lagoon should use to access the custom backup bucket. An example custom setting would be: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. |
+
No public access is needed on the S3 bucket and can be made entirely private.
+Lagoon will automatically prune the files in these S3 buckets, so no object retention policy is needed at the bucket level.
+Lagoon supports custom restore locations and credentials for any project when all four of the following variables are set as BUILD type environment variables. The environment variables need to be set at the project level (not per environment), and requires a Lagoon deployment after setting them (for every environment).
Please note that any use of these variables means that all environment and database snapshots restored by Lagoon will be stored using these credentials. This means that any interruption of these credentials' access may lead to failed or inaccessible restored files.
+| Environment variable name | +Purpose | +
|---|---|
LAGOON_BAAS_CUSTOM_RESTORE_ENDPOINT |
+Specify the S3 compatible endpoint where any Lagoon initiated restores should be stored. An example for S3 Sydney would be: https://s3.ap-southeast-2.amazonaws.com. |
+
LAGOON_BAAS_CUSTOM_RESTORE_BUCKET |
+Specify the bucket name where any Lagoon initiated restores should be stored.An example custom setting would be: example-restore-bucket. |
+
LAGOON_BAAS_CUSTOM_RESTORE_ACCESS_KEY |
+Specify the access key Lagoon should use to access the custom restore bucket. An example custom setting would be: AKIAIOSFODNN7EXAMPLE. |
+
LAGOON_BAAS_CUSTOM_RESTORE_SECRET_KEY |
+Specify the secret key Lagoon should use to access the custom restore bucket. An example custom setting would be: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. |
+
The S3 bucket must have public access enabled, as Lagoon will create presigned URLs for the objects inside the bucket as needed.
+An example AWS IAM policy that you can create to allow access to just the S3 bucket example-restore-bucket is:
{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "s3:GetBucketLocation",
+ "s3:ListBucket"
+ ],
+ "Resource": [
+ "arn:aws:s3:::example-restore-bucket"
+ ]
+ },
+ {
+ "Effect": "Allow",
+ "Action": [
+ "s3:PutObject",
+ "s3:GetObject",
+ "s3:GetObjectVersion",
+ "s3:GetBucketLocation",
+ "s3:PutObjectAcl"
+ ],
+ "Resource": [
+ "arn:aws:s3:::example-restore-bucket/*"
+ ]
+ }
+ ]
+}
+For increased security and reduced storage costs you can opt into removing restored backups after a set lifetime (e.g. 7 days). Lagoon caters for this scenario gracefully and will re-create any restored snapshots as needed.
+ + + + + + + + +API interactions in Lagoon are done via GraphQL. In order to authenticate to the API, you need a JWT (JSON Web Token), which will authenticate you against the API via your SSH public key.
+To generate this token, use the remote shell via the token command:
Example for amazee.io:
+ +This will return a long string, which is the JWT token.
+We also need the URL of the API endpoint. Ask your Lagoon administrator for this.
+ +On amazee.io this is https://api.lagoon.amazeeio.cloud/graphql.
Now we need a GraphQL client! Technically this is just HTTP, but we suggest GraphiQL. It has a nice UI that allows you to write GraphQL requests with autocomplete. Download, install and start it. [GraphiQL App]
+Enter the API endpoint URL. Then click on "Edit HTTP Headers" and add a new Header:
+AuthorizationBearer [jwt token] (make sure that the JWT token has no spaces, that won't work)
Close the HTTP Header overlay (press ESC) and now you are ready to make your first GraphQL Request!
+Enter this on the left window:
+query whatIsThere {
+ allProjects {
+ id
+ gitUrl
+ name
+ branches
+ pullrequests
+ productionEnvironment
+ environments {
+ name
+ environmentType
+ }
+ }
+}
+And press the ▶️ button (or press CTRL+ENTER).
+
If all went well, you should see your first GraphQL response.
+The Lagoon GraphQL API can not only display objects and create objects, but it also has the capability to update existing objects. All of Lagoon's GraphQL uses best practices.
+Mutation queries in GraphQL modify the data in the data store, and return a value. They can be used to insert, update, and delete data. Mutations are defined as a part of the schema.
+Update the branches to deploy within a project:
+mutation editProjectBranches {
+ updateProject(input:{id:109, patch:{branches:"^(prod|stage|dev|update)$"}}) {
+ id
+ }
+}
+Update the production environment within a project:
+Warning
+This requires a redeploy in order for all changes to be reflected in the containers.
+mutation editProjectProductionEnvironment {
+ updateProject(input:{id:109, patch:{productionEnvironment:"prod"}}) {
+ id
+ }
+}
+You can also combine multiple changes into a single query:
+ + + + + + + + + +Node.js has integrated web server capabilities. Plus, with Express, these can be extended even more.
+Unfortunately, Node.js does not handle shutting itself down very nicely out of the box. This causes many issues with containerized systems. The biggest issue is that when a Node.js container is told to shut down, it will immediately kill all active connections, and does not allow them to stop gracefully.
+This part explains how you can teach Node.js to behave like a real web server: finishing active requests and then gracefully shutting down.
+As an example we use a no-frills Node.js server with Express:
+const express = require('express');
+const app = express();
+
+// Adds a 5 second delay for all requests.
+app.use((req, res, next) => setTimeout(next, 5000));
+
+app.get('/', function (req, res) {
+ res.send("Hello World");
+})
+
+const server = app.listen(3000, function () {
+ console.log('Example app listening on port 3000!');
+})
+This will just show "Hello World" in when the web server is visited at localhost:3000. Note the 5 second delay in the response in order to simulate a request that takes some computing time.
If we run the above example and stop the Node.js process while the request is handled (within the 5 seconds), we will see that the Node.js server immediately kills the connection, and our browser will show an error.
+To explain to our Node.js server that it should wait for all the requests to be finished before actually stopping itself, we add the following code:
+const startGracefulShutdown = () => {
+ console.log('Starting shutdown of express...');
+ server.close(function () {
+ console.log('Express shut down.');
+ });
+}
+
+process.on('SIGTERM', startGracefulShutdown);
+process.on('SIGINT', startGracefulShutdown);
+This basically calls server.close(), which will instruct the Node.js HTTP server to:
It will do this on SIGINT (when you press CTRL + C) or on SIGTERM (the standard signal for a process to terminate).
With this small addition, our Node.js will wait until all requests are finished, and then stop itself.
+If we were not running Node.js in a containerized environment, we would probably want to include some additional code that actually kills the Node.js server after a couple of seconds, as it is technically possible that some requests are either taking very long or are never stopped. Because it is running in a containerized system, if the container is not stopped, Docker and Kubernetes will run a SIGKILL after a couple of seconds (usually 30) which cannot be handled by the process itself, so this is not a concern for us.
If we only implemented Part A, we would have a good experience. In the real world, many Node.js systems are built with Yarn or NPM, which provide not only package management systems to Node.js, but also script management.
+With these script functionalities, we simplify the start of our application. We can see many package.json files that look like:
{
+ "name": "node",
+ "version": "1.0.0",
+ "main": "index.js",
+ "license": "MIT",
+ "dependencies": {
+ "express": "^4.15.3"
+ },
+ "scripts": {
+ "start": "node index.js"
+ }
+}
+and with the defined scripts section we can run our application just with:
or
+ +This is nice and makes the life of developers easier. So we also end up using the same within Dockerfiles:
+ +Unfortunately there is a big problem with this:
+If yarn or npm get a SIGINT or SIGTERM signal, they correctly forward the signal to spawned child process (in this case node index.js). However, it does not wait for the child processes to stop. Instead, yarn/npm immediately stop themselves. This signals to Docker/Kubernetes that the container is finished and Docker/Kubernetes will kill all children processes immediately. There are issues open for Yarn and NPM but unfortunately they are not solved yet.
The solution for the problem is to not use Yarn or NPM to start your application and instead use node directly:
This allows Node.js to properly terminate and Docker/Kubernetes will wait for Node.js to be finished.
+ + + + + + + + +ARG LAGOON_SSH_PRIVATE_KEY to your dockerfile (before the step of the build process that needs the SSH key).RUN /lagoon/entrypoints/05-ssh-key.sh to your dockerfile (before the step of the build process that needs the SSH key).When a Lagoon project is created, by default an associated SSH "project key" is generated and the private key made available inside the CLI pods of the project. A service account default-user@project is also created and given MAINTAINER access to the project. The SSH "project key" is attached to that default-user@project.
The result of this is that from inside the CLI pod of any environment it is possible to SSH to any other environment within the same project. This access is used for running tasks from the command line such as synchronizing databases between environments (e.g. drush sql-sync).
There is more information on the MAINTAINER role available in the RBAC documentation.
It is possible to specify an SSH private key when creating a project, but this is not recommended as it has security implications.
+ + + + + + + + +The below lists all service types that can be defined via lagoon.type within a docker-compose.yml file.
Warning
+Once a lagoon.type is defined and the environment is deployed, changing it to a different type is not supported and could result in a broken environment.
basic#Basic container, good to use for most applications that don't have an existing template. No persistent storage. The port can be changed using a label. If an autogenerated route is not required (e.g. for an internal-facing service, set lagoon.autogeneratedroute: false in the docker-compose.yml)
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 3000 |
+3000 |
+Yes | +No | +lagoon.service.port, lagoon.autogeneratedroute |
+
basic-persistent#Like basic. Will also generate persistent storage, defines mount location via lagoon.persistent.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 3000 |
+3000 |
+Yes | +Yes | +lagoon.service.port, lagoon.autogeneratedroute, lagoon.persistent, lagoon.persistent.name, lagoon.persistent.size, lagoon.persistent.class |
+
cli#Use for any kind of CLI container (like PHP, Node.js, etc). Automatically gets the customer SSH private key that is mounted in /var/run/secrets/lagoon/sshkey/ssh-privatekey.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +No | +No | +No | +- | +
cli-persistent#Like cli, expects lagoon.persistent.name to be given the name of a service that has persistent storage, which will be mounted under defined lagoon.persistent label. Does NOT generate its own persistent storage, only used to mount another service's persistent storage.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +No | +No | +Yes | +lagoon.persistent.name, lagoon.persistent |
+
elasticsearch#Elasticsearch container, will auto-generate persistent storage under /usr/share/elasticsearch/data.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
HTTP on localhost:9200/_cluster/health?local=true |
+9200 | +No | +Yes | +lagoon.persistent.size |
+
kibana#Kibana container.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 5601 |
+5601 |
+Yes | +No | +- | +
logstash#Logstash container.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 9600 |
+9600 |
+No | +No | +- | +
mariadb#A meta-service which will tell Lagoon to automatically decide between mariadb-single and mariadb-dbaas.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +- | +- | +- | +- | +
mariadb-single#MariaDB container. Creates cron job for backups running every 24h executing /lagoon/mysql-backup.sh 127.0.0.1.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 3306 |
+3306 |
+No | +Yes | +lagoon.persistent.size |
+
mariadb-dbaas#Uses a shared MariaDB server via the DBaaS Operator.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| Not Needed | +3306 |
+No | +- | +- | +
mongo#A meta-service which will tell Lagoon to automatically decide between mongo-single and mongo-dbaas.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +- | +- | +- | +- | +
mongo-single#MongoDB container, will generate persistent storage of min 1GB mounted at /data/db.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 27017 |
+27017 |
+No | +Yes | +lagoon.persistent.size |
+
mongo-dbaas#Uses a shared MongoDB server via the DBaaS Operator.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| Not Needed | +27017 |
+No | +- | +- | +
nginx#NGINX container. No persistent storage.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
localhost:50000/nginx_status |
+8080 |
+Yes | +No | +lagoon.autogeneratedroute |
+
nginx-php#Like nginx, but additionally a php container.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
NGINX: localhost:50000/nginx_status, PHP: /usr/sbin/check_fcgi |
+8080 |
+Yes | +No | +lagoon.autogeneratedroute |
+
nginx-php-persistent#Like nginx-php. Will generate persistent storage, defines mount location via lagoon.persistent.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
NGINX: localhost:50000/nginx_status, PHP: /usr/sbin/check_fcgi |
+http on 8080 |
+Yes | +Yes | +lagoon.autogeneratedroute, lagoon.persistent, lagoon.persistent.name, lagoon.persistent.size, lagoon.persistent.class |
+
node#Node.js container. No persistent storage.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 3000 |
+3000 |
+Yes | +No | +lagoon.autogeneratedroute |
+
node-persistent#Like node. Will generate persistent storage, defines mount location via lagoon.persistent.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 3000 |
+3000 |
+Yes | +Yes | +lagoon.autogeneratedroute, lagoon.persistent, lagoon.persistent.name, lagoon.persistent.size, lagoon.persistent.class |
+
none#Instructs Lagoon to completely ignore this service.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +- | +- | +- | +- | +
opensearch#OpenSearch container, will auto-generate persistent storage under /usr/share/opensearch/data.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
HTTP on localhost:9200/_cluster/health?local=true |
+9200 | +No | +Yes | +lagoon.persistent.size |
+
postgres#A meta-service which will tell Lagoon to automatically decide between postgres-single and postgres-dbaas.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +- | +- | +- | +- | +
postgres-single#Postgres container. Creates cron job for backups running every 24h executing /lagoon/postgres-backup.sh localhost.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 5432 |
+5432 |
+No | +Yes | +lagoon.persistent.size |
+
postgres-dbaas#Uses a shared PostgreSQL server via the DBaaS Operator.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| Not Needed | +5432 |
+No | +- | +- | +
python#Python container. No persistent storage.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
HTTP connection on 8800 |
+8800 |
+Yes | +No | +lagoon.autogeneratedroute |
+
python-persistent#Python container. With persistent storage.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
HTTP connection on 8800 |
+8800 |
+Yes | +Yes | +lagoon.autogeneratedroute |
+
redis#Redis container.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 6379 |
+6379 |
+No | +No | +- | +
redis-persistent#Redis container with auto-generated persistent storage mounted under /data.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 6379 |
+6379 |
+No | +Yes | +lagoon.persistent.size |
+
solr#Solr container with auto-generated persistent storage mounted under /var/solr.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
TCP connection on 8983 |
+8983 |
+No | +Yes | +lagoon.persistent.size |
+
varnish#Varnish container.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
HTTP request localhost:8080/varnish_status |
+8080 |
+Yes | +No | +lagoon.autogeneratedroute |
+
varnish-persistent#Varnish container with auto-generated persistent storage mounted under /var/cache/varnish.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
HTTP request localhost:8080/varnish_status |
+8080 |
+Yes | +Yes | +lagoon.autogeneratedroute, lagoon.persistent.size |
+
worker#Use for any kind of worker container (like queue workers, etc.) where there is no exposed service port.
+| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +No | +No | +No | +- | +
worker-persistent#Like worker, expects lagoon.persistent.name to be given the name of a service that has persistent storage, which will be mounted under defined lagoon.persistent label. Does NOT generate its own persistent storage, only used to mount another service's persistent storage.
| Healthcheck | +Exposed Ports | +Auto Generated Routes | +Storage | +Additional customization parameter | +
|---|---|---|---|---|
| - | +No | +No | +Yes | +lagoon.persistent.name, lagoon.persistent |
+
The Lagoon base images are pre-configured with Xdebug but, for performance
+reasons, the extension is not loaded by default. To enable the extension, the
+XDEBUG_ENABLE environment variable must be set to true:
docker-compose.yml
+ file, the environment variable already exists. Uncomment these lines.The default Xdebug configuration requires a "trigger" to activate the extension +and start a session. You can view the complete documentation +for activating the debugger but the most straightforward instructions are below.
+The php-cli image is configured to always activate Xdebug when it’s enabled,
+so there is nothing else that needs to be done. Running any PHP script will
+start a debugging session.
Install a browser extension +to set/unset an activation cookie.
+Make sure the activation cookie is set for the website you want to start +debugging.
+launch.json for PHP.Add correct path mappings. For a typical Drupal site, an example would be:
+ +In the Run tab of Visual Studio Code, click the green arrow next to + “Listen for Xdebug”
+
| Directive | +Local Value | +
|---|---|
| xdebug.mode | +debug | +
| xdebug.client_host | +host.docker.internal or your IP address |
+
| xdebug.client_port | +9003 | +
XDEBUG_LOG set to anything to enable logging.
+ Logs will be saved to /tmp/xdebug.log. If you are using the lagoon-examples
+ then you can uncomment some existing lines.XDEBUG_SESSION cookie is set.nc -l 9003 command line tool to open the
+ Xdebug port. If everything is configured in PHP correctly, you should get a
+ Xdebug init response when you load a webpage or run a Drush command.xdebug.client_host has been set correctly. For local
+ debugging with Docker for Mac, this value should be host.docker.internal.
+ For remote debugging this value should be your IP address. If this value was
+ not correctly determined, you can override it by setting the DOCKERHOST
+ environment variable.lando ssh. You won’t be able to debug
+ things by running lando drush or lando php.Verify that Docker for Mac networking is not broken. On your host machine, run
+ nc -l 9003, then in a new terminal window, run:
You should see a message like:
+host.docker.internal (192.168.65.2:9003) open.
Ensure the host host.docker.internal can be reached. If docker has been
+ installed manually (and not through Docker Desktop), this host will not
+ resolve. You can force this to resolve with an additional snippet in your
+ docker-compose.yml file (instructions taken from this blog post):
If you're running older images you may still be using Xdebug version 2. All the +information on this page still applies, but some of the configuration names and +values have changes:
+| v3 | +v2 | ++ |
|---|---|---|
| xdebug.mode | +xdebug.remote_enabled | +On | +
| xdebug.client_host | +xdebug.remote_host | +host.docker.internal or your IP address |
+
| xdebug.client_port | +xdebug.remote_port | +9000 | +
This is an example of how to add SimpleSAMLphp to your project and then modify configuration to serve it via NGINX.
+Add SimpleSAMLphp to your project:
+ +Copy authsources.php and config.php from vendor/simplesamlphp/simplesamlphp/config-templates to somewhere outside vendor directory, such as conf/simplesamlphp. You also need saml20-idp-remote.php from vendor/simplesamlphp/simplesamlphp/metadata-templates.
In config.php set following values for Lagoon:
Base URL path where SimpleSAMLphp is accessed:
+ +Store sessions to database:
+ 'store.type' => 'sql',
+
+ 'store.sql.dsn' => vsprintf('mysql:host=%s;port=%s;dbname=%s', [
+ getenv('MARIADB_HOST'),
+ getenv('MARIADB_PORT'),
+ getenv('MARIADB_DATABASE'),
+ ]),
+Alter other settings to your liking:
+technicalcontact and timezone.Add authsources (IdPs) to authsources.php, see example:
'default-sp' => [
+ 'saml:SP',
+
+ // The entity ID of this SP.
+ 'entityID' => 'https://YOUR_DOMAIN.TLD',
+
+ // The entity ID of the IdP this should SP should contact.
+ // Can be NULL/unset, in which case the user will be shown a list of available IdPs.
+ 'idp' => 'https://YOUR_IDP_DOMAIN.TLD',
+
+ // The URL to the discovery service.
+ // Can be NULL/unset, in which case a builtin discovery service will be used.
+ 'discoURL' => null,
+
+ 'NameIDFormat' => 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient',
+
+ 'certificate' => '/app/conf/simplesamlphp/certs/saml.crt',
+ 'privatekey' => '/app/conf/simplesamlphp/certs/saml.pem',
+ 'redirect.sign' => TRUE,
+ 'redirect.validate' => TRUE,
+
+ 'authproc' => [
+ 50 => [
+ 'class' => 'core:AttributeCopy',
+ 'urn:oid:1.3.6.1.4.1.5923.1.1.1.6' => 'eduPersonPrincipalName',
+ ],
+ 51 => [
+ 'class' => 'core:AttributeCopy',
+ 'urn:oid:2.5.4.42' => 'givenName',
+ ],
+ 52 => [
+ 'class' => 'core:AttributeCopy',
+ 'urn:oid:2.5.4.4' => 'sn',
+ ],
+ 53 => [
+ 'class' => 'core:AttributeCopy',
+ 'urn:oid:0.9.2342.19200300.100.1.3' => 'mail',
+ ],
+ ],
+ ],
+Add IdP metadata to saml20-idp-remote.php, see example:
<?php
+/**
+ * SAML 2.0 remote IdP metadata for SimpleSAMLphp.
+ *
+ * Remember to remove the IdPs you don't use from this file.
+ *
+ * See: https://simplesamlphp.org/docs/stable/simplesamlphp-reference-idp-remote
+ */
+
+/**
+ * Some IdP.
+ */
+$metadata['https://YOUR_IDP_DOMAIN.TLD'] = [
+ 'entityid' => 'https://YOUR_IDP_DOMAIN.TLD',
+ 'name' => [
+ 'en' => 'Some IdP',
+ ],
+ 'description' => 'Some IdP',
+
+ ...
+
+];
+In your build process, copy configuration files to SimpleSAMLphp:
+vendor/simplesamlphp/simplesamlphp/config/authsources.phpvendor/simplesamlphp/simplesamlphp/config/config.phpvendor/simplesamlphp/simplesamlphp/metadata/saml20-idp-remote.phpCreate file lagoon/nginx/location_prepend_simplesamlphp.conf:
location ^~ /simplesaml {
+ alias /app/vendor/simplesamlphp/simplesamlphp/www;
+
+ location ~ ^(?<prefix>/simplesaml)(?<phpfile>.+?\.php)(?<pathinfo>/.*)?$ {
+ include fastcgi_params;
+ fastcgi_pass ${NGINX_FASTCGI_PASS:-php}:9000;
+ fastcgi_param SCRIPT_FILENAME $document_root$phpfile;
+ # Must be prepended with the baseurlpath
+ fastcgi_param SCRIPT_NAME /simplesaml$phpfile;
+ fastcgi_param PATH_INFO $pathinfo if_not_empty;
+ }
+}
+This will route /simplesaml URLs to SimpleSAMLphp in vendor.
Modify nginx.dockerfile and add location_prepend_simplesamlphp.conf to the image:
ARG CLI_IMAGE
+FROM ${CLI_IMAGE} as cli
+
+FROM amazeeio/nginx-drupal
+
+COPY --from=cli /app /app
+
+COPY lagoon/nginx/location_prepend_simplesamlphp.conf /etc/nginx/conf.d/drupal/location_prepend_simplesamlphp.conf
+RUN fix-permissions /etc/nginx/conf.d/drupal/location_prepend_simplesamlphp.conf
+
+# Define where the Drupal Root is located
+ENV WEBROOT=public
+Lagoon allows you to connect to your running containers via SSH. The containers themselves don't actually have an SSH server installed, but instead you connect via SSH to Lagoon, which then itself creates a remote shell connection via the Kubernetes API for you.
+It is recommended to generate a separate SSH key for each device as opposed to sharing the same key between multiple computers. Instructions for generating an SSH key on various systems can be found below:
+OSX does not have its SSH agent configured to load configured SSH keys at startup, which can cause some headaches. You can find a handy guide to configuring this capability here: https://www.backarapper.com/add-ssh-keys-to-ssh-agent-on-startup-in-macos/
+Linux distributions vary in how they use the ssh-agent . You can find a general guide here: https://www.ssh.com/academy/ssh/agent
SSH key support in Windows has improved markedly as of recently, and is now supported natively. A handy guide to configuring the Windows 10 SSH agent can be found here: https://richardballard.co.uk/ssh-keys-on-windows-10/
+You can upload your SSH key(s) through the UI. Log in as you normally would.
+In the upper right hand corner, click on Settings:
+
You will then see a page where you can upload your SSH key(s), and it will show any uploaded keys. Paste your key into the text box, give it a name, and click "Add." That's it! Add additional keys as needed.
+
A general example of using the Lagoon API via GraphQL to add an SSH key to a user can be found here
+Connecting is straightforward and follows the following pattern:
+ +PORT - The remote shell SSH endpoint port (for amazee.io: 32222).HOST - The remote shell SSH endpoint host (for amazee.io ssh.lagoon.amazeeio.cloud).PROJECT-ENVIRONMENT-NAME - The environment you want to connect to. This is most commonly in the pattern PROJECTNAME-ENVIRONMENT.As an example:
+ +This will connect you to the project drupal-example on the environment main.
By default, the remote shell will try to connect you to the container defined with the type cli. If you would like to connect to another pod/service you can define it via:
If your pod/service contains multiple containers, Lagoon will connect you to the first defined container. You can also define the specific container to connect to via:
+ssh -p [PORT] -t [PROJECT-ENVIRONMENT-NAME]@[HOST] service=[SERVICE-NAME] container=[CONTAINER-NAME]
+For example, to connect to the php container within the nginx pod:
ssh -p 32222 -t drupal-example-main@ssh.lagoon.amazeeio.cloud service=nginx container=php
+The common case of copying a file into your cli pod can be acheived with the usual SSH-compatible tools.
scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 32222 [local_path] [project_name]-[environment_name]@ssh.lagoon.amazeeio.cloud:[remote_path]
+rsync --rsh='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 32222' [local_path] [project_name]-[environment_name]@ssh.lagoon.amazeeio.cloud:[remote_path]
+ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 32222 [project_name]-[environment_name]@ssh.lagoon.amazee.io tar -zcf - [remote_path] | tar -zxf - -C /tmp/
+In the rare case that you need to specify a non-CLI service you can specify the service=... and/or container=... arguments in the copy command.
Piping tar through the ssh connection is the simplest method, and can be used to copy a file or directory using the usual tar flags:
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -P 32222 [project_name]-[environment_name]@ssh.lagoon.amazee.io service=solr tar -zcf - [remote_path] | tar -zxf - -C /tmp/
+You can also use rsync with a wrapper script to reorder the arguments to ssh in the manner required by Lagoon's SSH service:
#!/usr/bin/env sh
+svc=$1 user=$3 host=$4
+shift 4
+exec ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 32222 -l "$user" "$host" "$svc" "$@"
+Put that in an executable shell script rsh.sh and specify the service=... in the rsync command:
rsync --rsh="/path/to/rsh.sh service=cli" /tmp/foo [project_name]-[environment_name]@ssh.lagoon.amazeeio.cloud:/tmp/foo
+The script could also be adjusted to also handle a container=... argument.
In order to automatically trigger new deployments using Azure Pipelines follow these instructions:
+id_rsa_lagoon. For more information about secure files have a look at the Azure Documentation Site.azure-pipelines.yml:pool:
+ vmImage: 'ubuntu-latest'
+
+stages:
+ # .. other stages
+ - stage: Deploy
+ condition: and(succeeded(), in(variables['Build.SourceBranch'], 'refs/heads/staging', 'refs/heads/develop'))
+ jobs:
+ - job: DeployLagoon
+ steps:
+ - task: DownloadSecureFile@1
+ name: lagoonSshKey
+ displayName: 'Download Lagoon SSH key'
+ inputs:
+ secureFile: id_rsa_lagoon
+ - script: |
+ curl -L "https://github.com/amazeeio/lagoon-cli/releases/download/0.9.2/lagoon-cli-0.9.2-linux-amd64" -o ./lagoon
+ chmod +x ./lagoon
+ displayName: 'Download lagoon-cli'
+ - script: ./lagoon login -i $(lagoonSshKey.secureFilePath)
+ displayName: 'Log into Lagoon'
+ - script: ./lagoon deploy branch -e $(Build.SourceBranchName) -p my-awesome-project -b $(Build.SourceBranchName) --force
+ displayName: 'Trigger deployment using lagoon-cli'
+This will trigger a new deployment whenever changes are made on the develop or staging branch. Adjust these values accordingly so they fit your deployment strategy and configuration.
There may be a case where you want to push without a deployment. Make sure your commit message contains "[skip deploy]" or "[deploy skip]" and Lagoon will not trigger a deployment from that commit.
Lagoon tries to support any development workflow possible. It specifically does not enforce any workflows onto teams, so that each development team can define how they would like to develop and deploy their code.
+The most straightforward workflows are deployment-based on some fixed branches:
+You define which branches (like develop, staging and main, which would be ^(develop|staging|main)$ as regular expressions) that Lagoon should deploy and it will do so. Done!
If you would like to test a new feature, merge them into a branch that you have set up locally and push, and Lagoon will deploy the feature and you can test. When all is good, merge the branch into your production branch and push.
+A bit more advanced are feature branches. Since Lagoon supports the ability to define the branches you would like to deploy via regular expressions, you can also extend the above regular expression to this: ^feature\/|^(staging|main)$. This will instruct Lagoon to deploy all branches that start with feature/, plus the branches called staging and main. Our development workflow could be as following:
main called feature/myfeature and push feature/myfeature.feature/myfeature as a new environment, where you can test your feature independently of any other features.feature/myfeature into the main branch and it will deploy to your production environment.If you like, you can also merge feature/myfeature and any other feature branches into staging first, in order to test the functionality of multiple features together. After you have tested the features together on staging, you can merge the features into main.
This workflow needs a high level of branch pruning and cleanliness in your Git repository. Since each feature branch will create its own Lagoon environment, you can have very quickly generate a LOT of environments, which all of them will use resources. Be sure to merge or delete unused branches.
+Because of this, it could make sense to think about a pull request based workflow.
+Even more advanced are workflows via pull requests. Such workflows need the support of a Git hosting which supports pull requests (also called merge requests). The idea of pull request-based workflows lies behind that idea that you can test a feature together with a target branch, without actually needing to merge yet, as Lagoon will do the merging for you during the build.
+In our example we would configure Lagoon to deploy the branches ^(staging|main)$ and the pull requests to .* (to deploy all pull requests). Now our workflow would be:
main called feature/myfeature and push feature/myfeature (no deployment will happen now because we have only specific staging and main as our branches to be deployed).feature/myfeature into main.feature/myfeature branch on top of the main branch and deploy that resulting code for you.feature/myfeature branch just as if it had been merged into main, so all changes that have happened in main since you created the feature/myfeature branch from it will be there, and you don't need to worry that you might have an older version of the main branch.main. This will now trigger a deployment of main.Some teams might opt to create the pull request against a shared staging branch and then merge the staging branch into the main branch via another pull request. This depends on the kind of Git workflow you're using.
Additionally, in Lagoon you can define that only pull requests with a specific text in the title are deployed. [BUILD] defined as regular expression will only deploy pull requests that have a title like [BUILD] My Pull Request, while a pull request with that title My other Pull Request is not automatically deployed. This helps to keep the amount of environments small and allows for pull requests that don't need an environment yet.
Automatic pull request environments are a fantastic thing. But it would also be handy to have the database synced from another environment when those environments are created. Lagoon can handle that!
+The following example will sync the staging database on the first rollout of the pull request environment:
+tasks:
+ post-rollout:
+ - run:
+ name: IF no Drupal installed & Pullrequest = Sync database from staging
+ command: |
+ if [[ -n ${LAGOON_PR_BASE_BRANCH} ]] && tables=$(drush sqlq 'show tables;') && [ -z "$tables" ]; then
+ drush -y sql-sync @staging default
+ fi
+ service: cli
+ shell: bash
+Another way of deploying your code into an environment is the promotion workflow.
+The idea behind the promotion workflow comes from this (as an example):
+If you merge the branch staging into the main branch, and if there are no changes to main , so main and staging have the exact same code in Git, it could still technically be possible that the resulting Docker images are slightly different. This is because it's possible that between the last staging deployment and the current main deployment, some upstream Docker images may have changed, or dependencies loaded from the various package managers may have changed. This is a very small chance, but it's there.
For this situation, Lagoon understands the concept of promoting Lagoon images from one environment to another. This basically means that it will take the already built and deployed Docker images from one environment, and will use those exact same Docker images for another environment.
+In our example, we want to promote the Docker images from the main environment to the production environment:
main. Make sure that the environment has deployed successfully.production in your Git repository. This could lead to weird confusions (like people pushing into this branch, etc).curl request: curl -X POST \
+ https://rest.lagoon.amazeeio.cloud/promote \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "projectName":"myproject",
+ "sourceEnvironmentName": "main",
+ "branchName": "production"
+ }'
+This tells Lagoon that you want to promote from the source main to the destination production (yes, it really uses branchName as destination, which is a bit unfortunate, but it will be fixed soon).
Lagoon will now do the following:
+main in order to load the .lagoon.yml and docker-compose.yml files (Lagoon still needs these in order to fully work).docker-compose.yml , but with LAGOON_GIT_BRANCH=production as environment variable.main environment and use them (instead of building Images or tagging them from upstream).You will receive the same notifications of success or failures like any other deployment.
+ + + + + + + + +This document describes what actually happens during a Lagoon build and deployment. It is heavily simplified from what actually happens, but it will help you to understand what is happening under the hood every time that Lagoon deploys new code for you.
+Watch the video below for a walk-through of the deployment process.
+ + +First, Lagoon checks if the OpenShift project/Kubernetes namespace for the given environment exists and is correctly set up. It will make sure that we have the needed service accounts, create secrets, and will configure environment variables into a ConfigMap lagoon-env which is filled with information like the environment type and name, the Lagoon project name, and so on.
Next, Lagoon will check out your code from Git. It needs that to be able to read the .lagoon.yml, docker-compose.yml and any .env files, but also to build the Docker images.
Note that Lagoon will only process these actions if the branch/PR matches the branch regex set in Lagoon. Based on how the deployment has been triggered, different things will happen:
+If the deployment is triggered automatically via a Git webhook and is for a single branch, Lagoon will check out the Git SHA which is included in the webhook payload. This will trigger a deployment for every Git SHA pushed.
+If you trigger a branch deployment manually via the REST API (via the UI, or GraphQL) and do NOT define a SHA in the POST payload, Lagoon will just check out the latest commit in that branch and deploy it.
If the deployment is a pull request (PR) deployment, Lagoon will load the base and the HEAD branch and SHAs for the pull request and will:
+HEAD branch (the branch that the PR originates from) on top of the base branch.If the merge fails, Lagoon will also stop and inform you about this.
+For each service defined in the docker-compose.yml Lagoon will check if images need to be built or not. If they need to be built, this will happen now. The order of building is based on the order they are configured in docker-compose.yml , and some build arguments are injected:
LAGOON_GIT_SHALAGOON_GIT_BRANCHLAGOON_PROJECTLAGOON_BUILD_TYPE (either pullrequest, branch or promote)LAGOON_SSH_PRIVATE_KEY - The SSH private key that is used to clone the source repository. Use RUN /lagoon/entrypoints/05-ssh-key.sh to convert the build argument into an actual key at /home/.ssh/key which will be used by SSH and Git automatically. For safety, remove the key again via RUN rm /home/.ssh/key.LAGOON_GIT_SOURCE_REPOSITORY - The full Git URL of the source repository.Also, if this is a pull request build:
+LAGOON_PR_HEAD_BRANCHLAGOON_PR_HEAD_SHALAGOON_PR_BASE_BRANCHLAGOON_PR_BASE_SHALAGOON_PR_TITLEAdditionally, for each already built image, its name is also injected. If your docker-compose.yml is configured to first build the cli image and then the nginx image, the name of the nginx image is injected as NGINX_IMAGE.
Next, Lagoon will configure Kubernetes or OpenShift with all services and routes that are defined from the service types, plus possible additional custom routes that you have defined in .lagoon.yml.
In this step it will expose all defined routes in the LAGOON_ROUTES as comma separated URLs. It will also define one route as the "main" route, in this order:
.lagoon.yml.docker-compose.yml.The "main" route is injected via the LAGOON_ROUTE environment variable.
Now it is time to push the previously built Docker images into the internal Docker image registry.
+For services that didn't specify a Dockerfile to be built in docker-compose.yml and only gave an image, they are also tagged and will cause the internal Docker image registry to know about the images, so that they can be used in containers.
Lagoon will now create persistent storage (PVC) for each service that needs and requested persistent storage.
+For each service that requests a cron job (like MariaDB), plus for each custom cron job defined in .lagoon.yml, Lagoon will now generate the cron job environment variables which are later injected into the Deployment.
Now Lagoon will check the .lagoon.yml file for defined tasks in pre-rollout and will run them one by one in the defined services. Note that these tasks are executed on the pods currently running (so cannot utilize features or scripts that only exist in the latest commit) and therefore they are also not run on first deployments.
If any of them fail, Lagoon will immediately stop and notify you, and the rollout will not proceed.
+This is probably the most important step. Based on the defined service type, Lagoon will create the Deployment, Statefulset or Daemonsets for the service. (Note that Deployments are analogous to DeploymentConfigs in OpenShift)
+It will include all previously gathered information like the cron jobs, the location of persistent storage, the pushed images and so on.
+Creation of these objects will also automatically cause Kubernetes or OpenShift to trigger new deployments of the pods if necessary, like when an environment variable has changed or an image has changed. But if there is no change, there will be no deployment! This means if you only update the PHP code in your application, the Varnish, Solr, MariaDB, Redis and any other service that is defined but does not include your code will not be deployed. This makes everything much much faster.
+Now Lagoon waits! It waits for all of the just-triggered deployments of the new pods to be finished, as well as for their health checks to be successful.
+If any of the deployments or health checks fail, the deployment will be stopped here, and you will be informed via the defined notification systems (like Slack) that the deployment has failed.
+Now Lagoon will check the .lagoon.yml file for defined tasks in post-rollout and will run them one by one in the defined services.
If any of them fail, Lagoon will immediately stop and notify you.
+If all went well and nothing threw any errors, Lagoon will mark this build as successful and inform you via defined notifications. ✅
+ + + + + + + + +Your Lagoon administrator will also give you the route to the webhook-handler. You will add this to your repository as an outgoing webhook, and choose which events to send to Lagoon. Typically, you will send all push and pull request events. In Lagoon it is possible to add a regular expression to determine which branches and pull requests actually result in a deploy, and your Lagoon administrator can set that up for you. For example, all branches that start with feature- could be deployed to Lagoon.
If you are an amazee.io customer, the route to the webhook-handler is: https://hooks.lagoon.amazeeio.cloud.
+
Danger
+Managing the following settings will require you to have a high level of access to these repositories, which will be controlled by your organization. If you cannot access these settings, please contact your systems administrator or the appropriate person within your organization.
+Add webhook in your GitHub repository.
+ 
Payload URL is the route to the webhook-handler of your Lagoon instance, provided by your Lagoon administrator.Content type to application/json.
+ 
Let me select individual events."Push and Pull request events, and then filter further in the Lagoon configuration of your project.
+ 
Active.Add webhook to save your configuration.
URL is the route to the webhook-handler of your Lagoon instance, provided by your Lagoon administrator.Trigger events which will send a notification to Lagoon. We suggest that you send Push events and Merge request events, and then filter further in the Lagoon configuration of your project.
+ 
Add webhookto save your configuration.Title is for your reference.URL is the route to the webhook-handler of your Lagoon instance, provided by your Lagoon administrator.Choose from a full list of triggers and select the following:
+5. Click Save to save the webhook configurations for Bitbucket.
The docker-compose.yml file is used by Lagoon to:
Docker Compose (the tool) is very strict in validating the content of the YAML file, so we can only do configuration within labels of a service definition.
Warning
+Lagoon only reads the labels, service names, image names and build definitions from a docker-compose.yml file. Definitions like: ports, environment variables, volumes, networks, links, users, etc. are IGNORED.
This is intentional, as the docker-compose file is there to define your local environment configuration. Lagoon learns from the lagoon.type the type of service you are deploying and from that knows about ports, networks and any additional configuration that this service might need.
Here a straightforward example of a docker-compose.yml file for Drupal:
version: '2.3'
+
+x-lagoon-project:
+ # Lagoon project name (leave `&lagoon-project` when you edit this)
+ &lagoon-project drupal-example
+
+x-volumes:
+ &default-volumes
+ # Define all volumes you would like to have real-time mounted into the docker containers
+ volumes:
+ - .:/app:delegated
+
+x-environment:
+ &default-environment
+ LAGOON_PROJECT: *lagoon-project
+ # Route that should be used locally, if you are using pygmy, this route *must* end with .docker.amazee.io
+ LAGOON_ROUTE: http://drupal-example.docker.amazee.io
+ # Uncomment if you want to have the system behave as it will in production
+ #LAGOON_ENVIRONMENT_TYPE: production
+ # Uncomment to enable Xdebug and then restart via `docker-compose up -d`
+ #XDEBUG_ENABLE: "true"
+
+x-user:
+ &default-user
+ # The default user under which the containers should run. Change this if you are on linux and run with another user than ID `1000`
+ user: '1000'
+
+services:
+
+ nginx:
+ build:
+ context: .
+ dockerfile: nginx.dockerfile
+ labels:
+ lagoon.type: nginx-php-persistent # (1)
+ lagoon.persistent: /app/web/sites/default/files/
+
+ php:
+ build:
+ context: .
+ dockerfile: php.dockerfile
+ labels:
+ lagoon.type: nginx-php-persistent # (2)
+ lagoon.name: nginx
+ lagoon.persistent: /app/web/sites/default/files/
+
+ mariadb:
+ image: amazeeio/mariadb-drupal
+ labels:
+ lagoon.type: mariadb
+x-lagoon-project:
This is the machine name of your project, define it here. We’ll use “drupal-example.”
+x-volumes:
This tells Lagoon what to mount into the container. Your web application lives in /app, but you can add or change this if needed.
x-environment:
.docker.amazee.io.LAGOON_ENVIRONMENT_TYPE: production.DEBUG_ENABLE: "true".x-user:
You are unlikely to need to change this, unless you are on Linux and would like to run with a user other than 1000.
services#This defines all the services you want to deploy. Unfortunately, Docker Compose calls them services, even though they are actually containers. Going forward we'll be calling them services, and throughout this documentation.
+The name of the service (nginx, php, and mariadb in the example above) is used by Lagoon as the name of the Kubernetes pod (yet another term - again, we'll be calling them services) that is generated, plus also any additional Kubernetes objects that are created based on the defined lagoon.type, which could be things like services, routes, persistent storage, etc.
Please note that service names adhere to the RFC 1035 DNS label standard. Service names must:
+Warning
+Once you have set the name of a service, do NOT rename it. This will cause all kind of havoc in your containers and break things.
+build#If you want Lagoon to build a Dockerfile for your service during every deployment, you can define it here:
+build
contextdocker build command.dockerfile:Warning
+Lagoon does NOT support the short version of build: <Dockerfile> and will fail if it finds such a definition.
image#If you don't need to build a Dockerfile and just want to use an existing Dockerfile, define it via image.
Lagoon needs to know what type of service you are deploying in order to configure the correct Kubernetes or OpenShift objects.
+This is done via the lagoon.type label. There are many different types to choose from. Check Service Types to see all of them and their additional configuration possibilities.
If you'd like Lagoon to ignore a service completely - for example, you need a container only during local development - give it the type none.
Some containers need persistent storage. Lagoon allows for each container to have a maximum of one persistent storage volume attached to the container. You can configure the container to request its own persistent storage volume (which can then be mounted by other container), or you can tell the container to mount the persistent storage created by another container.
+In many cases, Lagoon knows where that persistent storage needs to go. For example, for a MariaDB container, Lagoon knows that the persistent storage should be put into /var/lib/mysql , and puts it there automatically without any extra configuration to define that. For some situations, though, Lagoon needs your help to know where to put the persistent storage:
lagoon.persistent - The absolute path where the persistent storage should be mounted (the above example uses /app/web/sites/default/files/ which is where Drupal expects its persistent storage).lagoon.persistent.name - Tells Lagoon to not create a new persistent storage for that service, but instead mounts the persistent storage of another defined service into this service.lagoon.persistent.size - The size of persistent storage you require (Lagoon usually gives you minimum 5G of persistent storage, if you need more, define it here).lagoon.persistent.class - By default Lagoon automatically assigns the right storage class for your service (like SSDs for MySQL, bulk storage for Nginx, etc.). If you need to overwrite this, you can do so here. This is highly dependent on the underlying Kubernetes/OpenShift that Lagoon runs on. Ask your Lagoon administrator about this.The docker-compose.yml file also supports per-service enabling and disabling of autogenerated routes
+lagoon.autogeneratedroute: false label will stop a route from being automatically created for that service. It can be applied to all services with autogenerated routes, but is mostly useful for the basic and basic-persistent service types when used to create an additional internal-facing service for a database service or similar. The inverse is also true - it will enable an auto-generated route for a service when the .lagoon.yml file disables them.Kubernetes and OpenShift don't deploy plain containers. Instead, they deploy pods, with each one or more containers. Usually Lagoon creates a single pod with a container inside for each defined docker-compose service. For some cases, we need to put two containers inside a single pod, as these containers are so dependent on each other that they should always stay together. An example for such a situation is the PHP and NGINX containers that both contain PHP code of a web application like Drupal.
For these cases, it is possible to tell Lagoon which services should stay together, which is done in the following way (remember that we are calling containers services because of docker-compose:
lagoon.type that expects two services (in the example this is nginx-php-persistent defined on the nginx and php services).lagoon.name of the second one with the first one. (in the example this is done with defining lagoon.name: nginx).This will cause Lagoon to realize that the nginx and php containers are combined in a pod that will be called nginx.
Warning
+Once you have set the lagooon.name of a service, do NOT rename it. This will cause all kind of havoc in your containers and break things.
Lagoon still needs to understand which of the two services is the actual individual service type (nginx and php in this case). It does this by searching for service names with the same name that are given by the type, so nginx-php-persistent expects one service with the name nginx and one with php in the docker-compose.yml. If for any reason you want to use different names for the services, or you need for than one pod with the type nginx-php-persistent there is an additional label lagoon.deployment.servicetype which can be used to define the actual service type.
An example:
+nginx:
+ build:
+ context: .
+ dockerfile: nginx.dockerfile
+ labels:
+ lagoon.type: nginx-php-persistent
+ lagoon.persistent: /app/web/sites/default/files/
+ lagoon.name: nginx # If this isn't present, Lagoon will use the container name, which in this case is nginx.
+ lagoon.deployment.servicetype: nginx
+php:
+ build:
+ context: .
+ dockerfile: php.dockerfile
+ labels:
+ lagoon.type: nginx-php-persistent
+ lagoon.persistent: /app/web/sites/default/files/
+ lagoon.name: nginx # We want this service to be part of the NGINX pod in Lagoon.
+ lagoon.deployment.servicetype: php
+In the example above, the services are named nginx and php (but you can call them whatever you want). The lagoon.name tells Lagoon which services go together - all of the services with the same name go together.
In order for Lagoon to realize which one is the nginx and which one is the php service, we define it via lagoon.deployment.servicetype: nginx and lagoon.deployment.servicetype: php.
Lagoon uses Helm for templating on Kubernetes. To do this, a series of Charts are included with the build-deploy-tool image.
By default, Lagoon expects that services from custom templates are rolled out via a DeploymentConfig object within Kubernetes or Openshift. It monitors the rollout based on this object. In some cases, the services that are defined via custom deployment need a different way of monitoring. This can be defined via lagoon.rollout:
deploymentconfig - This is the default. Expects a DeploymentConfig object in the template for the service.statefulset - Expects a Statefulset object in the template for the service.daemonset - Expects a Daemonset object in the template for the service.false - Will not monitor any rollouts, and will just be happy if the template applies and does not throw any errors.You can also overwrite the rollout for just one specific environment. This is done in .lagoon.yml.
BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.
+With the release of Lagoon v2.11.0, Lagoon now provides support for BuildKit-based docker-compose builds. To enable BuildKit for your Project or Environment, add DOCKER_BUILDKIT=1 as a build-time variable.
Bug
+Note that while using BuildKit locally, you may experience some known issues.
+Failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: This message means that your build has tried to access a Docker image that hasn't been built yet. As BuildKit builds in parallel, if you have a Docker image that inherits another one (as we do in Drupal with the CLI). You can use the target field inside the build to reconfigure as a multi-stage buildvolumes_from in Docker Compose v2 - this service (that provides SSH access into locally running containers) has been deprecated by Docker Compose. The section can be removed from your docker-compose.yml file if you don't require SSH access from inside your local environment, or can be worked around on a project-by-project basis - see https://github.com/pygmystack/pygmy/issues/333#issuecomment-1274091375 for more information.
Note
+If you are deploying a Drupal Project, skip this and read the Drupal-specific first deployment documentation.
+In order to make your first deployment a successful one, please make sure that your project is Lagoonized and that you have set up the project in Lagoon. If not, or you're not sure, or that doesn't sound familiar, don't worry, go back and follow the Step-by-Step Guides which show you how this works, and then come back and deploy!
+With Lagoon, you create a new deployment by pushing into a branch that is configured to be deployed.
+If you don't have any new code to push, don't worry! Run:
+ +This will trigger a push, and your Git hosting will inform Lagoon about this push via the configured webhook.
+If all is correct, you should see a notification in your configured chat system (this has been configured by your friendly Lagoon administrator):
+
This informs you that Lagoon has just started to deploy your code. Depending on the size of the code and amount of containers, this will take a couple of seconds. Just relax. If you want to know what's happening now, check out the Build and Deploy Process of Lagoon.
+You can also check your Lagoon UI to see the progress of any deployment (your Lagoon administrator has the info).
+As soon as Lagoon is done building and deploying it will send a second notification to the chat system, here an example:
+
It tells you:
+You can also quickly tell what kind of notification it is by the emoji at the beginning - whether it's just info that the build has started, a success, or fail.
+That's it! We hope that wasn't too hard - making devOps accessible is what we are striving for!
+That's the beauty of Lagoon: it's exactly the same! Just push the name of the branch and that one will be deployed.
+Did the deployment fail? Oh no! But we're here to help:
+Logs link in the error notification, it will tell you where in the deployment process the failure happened.Congratulations, you're this close to going live with your website on Lagoon! In order to make this as seamless as possible, we've got this final checklist for you. It leads you through the last few things you should check before taking your site live.
+.lagoon.yml#Check to be sure that all routes have been set up in your .lagoon.yml. Be aware that if you don't point the domains towards Lagoon, you should disable Let's Encrypt (LE) certificate creation, as it will lead to issues. Domains not pointing towards Lagoon will be disabled after a while in order to not exceed the Let's Encrypt quotas.
If you use Certificate Authority (CA) signed certificates, you can set tls-acme to false , but leave the insecure flag set to Allow or Redirect. In the case of CA certificates, let your Lagoon administrator know the routes and the SSL certificate that needs to be put in place.
environments:
+ main:
+ routes:
+ - nginx:
+ - example.com:
+ tls-acme: 'false'
+ insecure: Allow
+ - www.example.com:
+ tls-acme: 'false'
+ insecure: Allow
+As soon as the DNS entries point towards your Lagoon installation, you can switch the flags: tls-acme to true and insecure to Redirect
environments:
+ main:
+ routes:
+ - nginx:
+ - example.com:
+ tls-acme: 'true'
+ insecure: Redirect
+ - www.example.com:
+ tls-acme: 'true'
+ insecure: Redirect
+Note
+As checking every page of your website might be a bit a tedious job, you can make use of mixed-content-scan. This will crawl the entire site and give you back pages that include assets from a non-HTTPS site.
+If you need non-www to www redirects, make sure you have them set up in the redirects-map.conf - see Documentation.
Check if your cron jobs have been set up for your production environment - see .lagoon.yml.
To make it as smooth as possible for you to get your site pointing to our servers, we have dedicated load-balancer DNS records. Those technical DNS resource records are used for getting your site linked to the amazee.io infrastructure and serve no other purpose. If you are in doubt of the CNAME record, ask your Lagoon administrator about the exact CNAME you need to set up.
+Example on amazee.io : <region-identifier>.amazee.io
Before you switch over your domain to Lagoon, make sure you lower the Time-to-Live (TTL) before you go live. This will ensure that the switch from the old to the new servers will go quickly. We usually advise a TTL of 300-600 seconds prior to the DNS switch. More information about TTL.
+The recommended method of pointing your domain's DNS records at Lagoon is via a CNAME record as shown below:
+ +CNAME: cdn.amazee.io
If your DNS provider does not support the use of CNAME records, you can use these A records instead. Please ensure you set up individual records for each IP listed below:
+A: 151.101.2.191A: 151.101.66.191A: 151.101.130.191A: 151.101.194.191Note
+We do not suggest configuring any static IP addresses in your DNS zones. The Lagoon load balancer infrastructure may change over time which can have impact on your site availability if you configure a static IP address.
+Configuring the root domain (e.g. example.com) can be a bit tricky because the DNS specification does not allow the root domain to point to a CNAME entry. Depending on your DNS provider, the record name is different:
+If your DNS provider needs an IP address for the root domain, get in touch with your Lagoon administrator to give you the load balancer IP addresses.
+Lagoon understands the concept of development and production environments. Development environments automatically send noindex and nofollow headers in order to prohibit indexing by search engines.
X-Robots-Tag: noindex, nofollow
During project setup, the production environment should already be defined. If that's omitted, your environment will run in development mode. You can check if the environment is set as production environment in the Lagoon user interface. If the production environment is not set, let your Lagoon administrator know, and they will configure the system accordingly.
+
To run a Lagoon Project, your system must meet the requirements to run Docker. We suggest installing the latest version of Docker for your workstation. You can download Docker here. We also suggest allowing Docker at least 4 CPUs and 4 GB RAM.
+TL;DR: install and start pygmy:
Pygmy is a container stack for local development, developed collaboratively with the Lagoon team.
+Learn more about Lagoon, pygmy, and Local Development Environments
+.lagoon.yml#This is the main file that will be used by Lagoon to understand what should be deployed, as well as many other things. See documentation for .lagoon.yml.
docker-compose.yml#This file is used by Docker Compose to start your local development environment. Lagoon also uses it to understand which of the services should be deployed, which type, and how to build them. This happens via labels. See documentation for docker-compose.yml.
Some Docker images and containers need additional customizations from the provided images. This usually has two reasons:
+| Type | +Versions | +Dockerfile | +
|---|---|---|
| MariaDB | +10.4, 10.5, 10.6, 10.11 | +mariadb/Dockerfile | +
| PostgreSQL | +11, 12, 13, 14, 15 | +postgres/Dockerfile | +
| MongoDB | +4 | +mongo/Dockerfile | +
| NGINX | +openresty/1.21 | +nginx/Dockerfile | +
| Node.js | +16, 18, 20 | +node/Dockerfile | +
| PHP FPM | +8.0, 8.1, 8.2 | +php/fpm/Dockerfile | +
| PHP CLI | +8.0, 8.1, 8.2 | +php/cli/Dockerfile | +
| Python | +3.7, 3.8, 3.9, 3.10, 3.11 | +python/Dockerfile | +
| Redis | +5, 6, 7 | +redis/Dockerfile | +
| Solr | +7, 8 | +solr/Dockerfile | +
| Varnish | +5, 6, 7 | +varnish/Dockerfile | +
| Opensearch | +2 | +opensearch/Dockerfiles | +
| RabbitMQ | +3.10 | +rabbitmq/Dockerfile | +
| Ruby | +3.0, 3.1, 3.2 | +ruby/Dockerfile | +
All images are pushed to https://hub.docker.com/u/uselagoon. We suggest always using the latest tag (like uselagoon/nginx:latest) as they are kept up to date in terms of features and security.
If you choose to use a specific Lagoon version of an image like uselagoon/nginx:20.10.0 or uselagoon/node-10:20.10.0 it is your own responsibility to upgrade the version of the images as soon as a new Lagoon version is released!
The .lagoon.yml file is the central file to set up your project. It contains configuration in order to do the following:
The .lagoon.yml file must be placed at the root of your Git repository.
docker-compose-yaml#Tells the build script which Docker Compose YAML file should be used, in order to learn which services and containers should be deployed. This defaults to docker-compose.yml, but could be used for a specific Lagoon Docker Compose YAML file if needed.
environment_variables.git_sha#This setting allows you to enable injecting the deployed Git SHA into your project as an environment variable. By default this is disabled. Setting the value to true sets the SHA as the environment variable LAGOON_GIT_SHA.
Routes are used to direct traffic to services. Each service in an environnment
+can have routes, in which the domain names are defined manually or
+automatically. The top level routes section applies to all routes in all
+environments.
routes.autogenerate#This allows you to configure automatically created routes. Manual +routes are defined per environment.
+enabled: Set to false to disable autogenerated routes. Default is true.allowPullrequests: Set to true to override enabled: false for pull
+ requests.
+insecure: Configures HTTP connections. Default is Allow.Allow: Route will respond to HTTP and HTTPS.Redirect: Route will redirect any HTTP request to HTTPS.prefixes: Configure prefixes for the autogenerated routes of each
+ environment. This is useful for things like language prefix domains, or a
+ multi-domain site using the Drupal domain module.
There are different type of tasks you can define, and they differ in when exactly they are executed in a build flow:
+pre_rollout.[i].run#Here you can specify tasks which will run against your project after all images have been successfully built, but before:
+This feature enables you to, for example, create a database dump before updating your application. +This can make it easier to roll back in case of a problem with the deploy.
+Info
+The pre-rollout tasks run in the existing pods before they are updated, which means:
+post_rollout.[i].run#Here you can specify tasks which need to run against your project, after:
+Common uses for post-rollout tasks include running drush updb, drush cim, or clearing various caches.
namecommand/app. Keep this in mind if you need to cd into a specific location to run your task.servicecontainernginx-php), you will need to specify which container in the pod to connect to (e.g. the php container within the nginx pod).shellsh is used, but if the container also has other shells (like bash, you can define it here). This is useful if you want to run some small if/else bash scripts within the post-rollouts. See the example below to learn how to write a script with multiple lines.whenNote: If you would like to temporarily disable pre/post-rollout tasks during a deployment, you can set either of the following environment variables in the API at the project or environment level (see how on Environment Variables).
+LAGOON_PREROLLOUT_DISABLED=trueLAGOON_POSTROLLOUT_DISABLED=trueHere are some useful examples of post-rollout tasks that you may want to use or adapt for your projects.
+Run only if Drupal not installed:
+- run:
+ name: IF no Drupal installed
+ command: | # (1)
+ if tables=$(drush sqlq "show tables like 'node';") && [ -z "$tables" ]; then
+ #### whatever you like
+ fi
+ service: cli
+ shell: bash
+Different tasks based on branch name:
+- run:
+ name: Different tasks based on branch name
+ command: |
+ ### Runs if current branch is not 'production'
+ service: cli
+ when: LAGOON_GIT_BRANCH != "production"
+Run shell script:
+ +Target specific container in pod:
+ +Drupal & Drush 9: Sync database & files from master environment:
+- run:
+ name: Sync DB and Files from master if we are not on master
+ command: |
+ # Only if we don't have a database yet
+ if tables=$(drush sqlq 'show tables;') && [ -z "$tables" ]; then
+ drush sql-sync @lagoon.master @self # (1)
+ drush rsync @lagoon.master:%files @self:%files -- --omit-dir-times --no-perms --no-group --no-owner --chmod=ugo=rwX
+ fi
+ service: cli
+ when: LAGOON_ENVIRONMENT_TYPE != "production"
+backup-retention.production.monthly#Specify the number of monthly backups Lagoon should retain for your project's production environment(s).
+The global default is 1 if this value is not specified.
backup-retention.production.weekly#Specify the number of weekly backups Lagoon should retain for your project's production environment(s).
+The global default is 6 if this value is not specified.
backup-retention.production.daily#Specify the number of daily backups Lagoon should retain for your project's production environment(s).
+The global default is 7 if this value is not specified.
backup-retention.production.hourly#Specify the number of hourly backups Lagoon should retain for your project's production environment(s).
+The global default is 0 if this value is not specified.
backup-schedule.production#Specify the backup schedule for this project. Accepts cron-compatible syntax with the notable exception that the Minute block must be the letter M. Any other value in the Minute block will cause the Lagoon build to fail. This allows Lagoon to randomly choose a specific minute for these backups to happen, while users can specify the remainder of the schedule down to the hour.
The global default is M H(22-2) * * * if this value is not specified. Take note that these backups will use the cluster's local timezone.
Environment names match your deployed branches or pull requests. This allows for each environment to have a different config. In our example it will apply to the main and staging environment.
environments.[name].routes#Manual routes are domain names that are configured per environment to direct
+traffic to a service. Since all environments get automatically created
+routes by default, it is typical that manual routes are
+only setup for the production environment, using the main domain of the
+project's website like www.example.com.
Tip
+Since Lagoon has no control over the manual routes, you'll need to ensure
+the DNS records are configured properly at your DNS provider. You can likely
+set a CNAME record to point to the automatic route.
The first element after the environment is the target service, nginx in our
+example. This is how we identify which service incoming requests will be sent
+to.
The simplest route is example.com, as seen in our example
+.lagoon.yml - you can see it has no additional
+configuration. This will assume that you want a Let's Encrypt certificate for
+your route and no redirect from HTTPS to HTTP.
In the "www.example.com" example below, we see three more options (also
+notice the : at the end of the route and that the route is wrapped in ",
+that's important!):
tls-acme#tls-acme: Configures automatic TLS certificate generation via Let's Encrypt.
+ Default is true, set to false to disable automatic certificates.insecure: Configures HTTP connections. Default is Allow.Allow: Route will respond to HTTP and HTTPS.Redirect: Route will redirect any HTTP request to HTTPS.hstsEnabled: Adds the Strict-Transport-Security header. Default is
+ false.hstsMaxAge: Configures the max-age directive. Default is 31536000 (1
+ year).hstsPreload: Sets the preload directive. Default is false.hstsIncludeSubdomains: Sets the includeSubDomains directive. Default is
+ false.Info
+If you plan to switch from a SSL certificate signed by a Certificate +Authority (CA) to a Let's Encrypt certificate, it's best to get in touch +with your Lagoon administrator to oversee the transition. There are known +issues during the +transition. The workaround would be manually removing the CA certificate and +then triggering the Let's Encrypt process.
+When UptimeRobot is configured for your cluster (Kubernetes or OpenShift), Lagoon will inject annotations to each route/ingress for use by the stakater/IngressControllerMonitor. The default action is to monitor the homepage of the route. If you have a specific route to be monitored, this can be overridden by adding a monitoring-path to your route specification. A common use is to set up a path for monitoring which bypasses caching to give a more real-time monitoring of your site.
Warning
+Route/Ingress annotations are only supported by projects that deploy into clusters that run nginx-ingress controllers! Check with your Lagoon administrator if this is supported.
+annotations can be a YAML map of annotations supported by the nginx-ingress controller. This is specifically useful for easy redirects and other configurations.Some annotations are disallowed or partially restricted in Lagoon. +The table below describes these rules.
+If your .lagoon.yml contains one of these annotations it will cause a build failure.
| Annotation | +Notes | +
|---|---|
nginx.ingress.kubernetes.io/auth-snippet |
+Disallowed | +
nginx.ingress.kubernetes.io/configuration-snippet |
+Restricted to rewrite, add_header, set_real_ip, and more_set_headers directives. |
+
nginx.ingress.kubernetes.io/modsecurity-snippet |
+Disallowed | +
nginx.ingress.kubernetes.io/server-snippet |
+Restricted to rewrite, add_header, set_real_ip, and more_set_headers directives. |
+
nginx.ingress.kubernetes.io/stream-snippet |
+Disallowed | +
nginx.ingress.kubernetes.io/use-regex |
+Disallowed | +
In this example any requests to example.ch will be redirected to https://www.example.ch while keeping folders or query parameters intact (example.com/folder?query -> https://www.example.ch/folder?query).
- "example.ch":
+ annotations:
+ nginx.ingress.kubernetes.io/permanent-redirect: https://www.example.ch$request_uri
+- www.example.ch
+You can of course also redirect to any other URL not hosted on Lagoon, this will direct requests to example.de to https://www.google.com
- "example.de":
+ annotations:
+ nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com
+Warning
+Kubernetes will only process a single nginx.ingress.kubernetes.io/server-snippet annotation. Please ensure that if you use this annotation on a non-production environment route that you also include the add_header X-Robots-Tag "noindex, nofollow"; annotation as part of your server-snippet. This is needed to stop robots from crawling development environments as the default server-snippet set to prevent this in development environments in the ingress templates will get overwritten with any server-snippets set in .lagoon.yml.
Some configurations involve a reverse proxy (like a CDN) in front of the Kubernetes clusters. In these configurations, the IP of the reverse proxy will appear as the REMOTE_ADDR HTTP_X_REAL_IP HTTP_X_FORWARDED_FOR headers field in your applications. The original IP of the requester can be found in the HTTP_X_ORIGINAL_FORWARDED_FOR header.
If you want the original IP to appear in the REMOTE_ADDR HTTP_X_REAL_IP HTTP_X_FORWARDED_FOR headers, you need to tell the ingress which reverse proxy IPs you want to trust:
- "example.ch":
+ annotations:
+ nginx.ingress.kubernetes.io/server-snippet: |
+ set_real_ip_from 1.2.3.4/32;
+This example would trust the CIDR 1.2.3.4/32 (the IP 1.2.3.4 in this case). Therefore if there is a request sent to the Kubernetes cluster from the IP 1.2.3.4 the X-Forwarded-For Header is analyzed and its contents injected into REMOTE_ADDR HTTP_X_REAL_IP HTTP_X_FORWARDED_FOR headers.
Environments.[name].types#The Lagoon build process checks the lagoon.type label from the docker-compose.yml file in order to learn what type of service should be deployed (read more about them in the documentation of docker-compose.yml).
Sometimes you might want to override the type just for a single environment, and not for all of them. For example, if you want a standalone MariaDB database (instead of letting the Service Broker/operator provision a shared one) for your non-production environment called develop:
service-name: service-type
service-name is the name of the service from docker-compose.yml you would like to override.service-type the type of the service you would like to use in your override.Example for setting up MariaDB_Galera:
+ +environments.[name].templates#The Lagoon build process checks the lagoon.template label from the docker-compose.yml file in order to check if the service needs a custom template file (read more about them in the documentation of docker-compose.yml).
Sometimes you might want to override the template just for a single environment, and not for all of them:
+service-name: template-file
service-name is the name of the service from docker-compose.yml you would like to override.template-file is the path and name of the template to use for this service in this environment.environments.[name].rollouts#The Lagoon build process checks the lagoon.rollout label from the docker-compose.yml file in order to check if the service needs a special rollout type (read more about them in the documentation of docker-compose.yml)
Sometimes you might want to override the rollout type just for a single environment, especially if you also overwrote the template type for the environment:
+service-name: rollout-type
service-name is the name of the service from docker-compose.yml you would like to override.rollout-type is the type of rollout. See documentation of docker-compose.yml) for possible values.environments.[name].autogenerateRoutes#This allows for any environments to get autogenerated routes when route autogeneration is disabled.
+routes:
+ autogenerate:
+ enabled: false
+environments:
+ develop:
+ autogenerateRoutes: true
+environments.[name].cronjobs#Cron jobs must be defined explicitly for each environment, since it is typically
+not desirable to run the same ones for all environments. Depending on the
+defined schedule, cron jobs may run as a Kubernetes native CronJob or as an
+in-pod cron job via the crontab of the defined service.
cronjobs:
+ - name: Hourly Drupal Cron
+ schedule: "M * * * *" # Once per hour, at a random minute.
+ command: drush cron
+ service: cli
+ - name: Nightly Drupal Cron
+ schedule: "M 0 * * *" # Once per day, at a random minute from 00:00 to 00:59.
+ command: drush cron
+ service: cli
+name: Any name that will identify the purpose and distinguish it from other
+ cron jobs.schedule: The schedule for executing the cron job. Lagoon uses an extended
+ version of the crontab format. If you're not sure about the syntax, use a
+ crontab generator.
M for the minute, and your cron job will run once per
+ hour at a random minute (the same minute each hour), or M/15 to run it
+ every 15 mins, but with a random offset from the hour (like
+ 6,21,36,51). It is a good idea to spread out your cron jobs using this
+ feature, rather than have them all fire off on minute 0.H for the hour, and your cron job will run once per day
+ at a random hour (the same hour every day), or H(2-4) to run it once
+ per day within the hours of 2-4.Timezones:
+command: The command to execute. This executes in the WORKDIR of the
+ service. For Lagoon images, this is /app.Warning
+Cronjobs may run in-pod, via crontab, which doesn't support multiline + commands. If + you need a complex or multiline cron command, you must put it in a + script that can be used as the command. Consider whether a pre- or post-rollout task would work.
+service: Which service of your project to run the command in. For most
+ projects, this should be the cli service.In Lagoon, the same Git repository can be added to multiple projects, creating what is called a polysite. This allows you to run the same codebase, but allow for different, isolated, databases and persistent files. In .lagoon.yml , we currently only support specifying custom routes for a polysite project. The key difference from a standard project is that the environments becomes the second-level element, and the project name the top level.
To utilize this, you will need to:
+poly-project1 and poly-project2 below)poly-project1:
+ environments:
+ main:
+ routes:
+ - nginx:
+ - project1.com
+poly-project2:
+ environments:
+ main:
+ routes:
+ - nginx:
+ - project2.com
+api#If you run directly on amazee.io hosted Lagoon you will not need this key set.
+With the key api you can define another URL that should be used by the Lagoon CLI and drush to connect to the Lagoon GraphQL API. This needs to be a full URL with a scheme, like: http://localhost:3000 This usually does not need to be changed, but there might be situations where your Lagoon administrator tells you to do so.
ssh#If you run directly on amazee.io hosted Lagoon you will not need this key set.
+With the key ssh you can define another SSH endpoint that should be used by the Lagoon CLI and drush to connect to the Lagoon remote shell service. This needs to be a hostname and a port separated by a colon, like: localhost:2020 This usually does not need to be changed, but there might be situations where your Lagoon administrator tells you to do so.
container-registries#The container-registries block allows you to define your own private container registries to pull custom or private images. To use a private container registry, you will need a username, password, and optionally the url for your registry. If you don't specify a url in your YAML, it will default to using Docker Hub.
There are 2 ways to define the password used for your registry user.
+Create an environment variable in the Lagoon API with the type container_registry:
lagoon add variable -p <project_name> -N <registry_password_variable_name> -V <password_goes_here> -S container_registryThe name of the variable you create can then be set as the password:
+container-registries:
+ my-custom-registry:
+ username: myownregistryuser
+ password: <registry_password_variable_name>
+ url: my.own.registry.com
+You can also define the password directly in the .lagoon.yml file in plain text:
container-registries:
+ docker-hub:
+ username: dockerhubuser
+ password: MySecretPassword
+To consume a custom or private container registry image, you need to update the service inside your docker-compose.yml file to use a build context instead of defining an image:
Once the docker-compose.yml file has been updated to use a build, you need to create the Dockerfile.<service> and then set your private image as the FROM <repo>/<name>:<tag>
.lagoon.yml#This is an example .lagoon.yml which showcases all possible settings. You will need to adapt it to your project.
These settings have been deprecated and should be removed from use in your .lagoon.yml.
routes.autogenerate.insecureThe None option is equivalent to Redirect.
environments.[name].monitoring_urlsenvironments.[name].routes.[service].[route].hstsenvironments.[name].routes.[service].[route].insecureThe None option is equivalent to Redirect.
Even though Lagoon has only a hard dependency on Docker and Docker Compose (which is mostly shipped with Docker) there are some things which are nice for local development that are not included in Docker:
+You do not need to install Lagoon locally to use it locally! That sounds confusing but follow the documentation. Lagoon is the system that deploys your local development environment to your production environment, it's not the environment itself.
+Lagoon has traditionally worked best with pygmy , which is the amazee.io flavored system of the above tools and works out of the box with Lagoon. It lives at https://github.com/pygmystack/pygmy
pygmy is written in Golang, so to install it, run:
For detailed usage or installation info on pygmy, see its documentation.
+As announced in our blog post, Lagoon is now also compatible with Lando! For more information, please see the documentation at https://docs.lando.dev/config/lagoon.html to get yourself up and running.
+Lando's workflow for Lagoon will be familiar to users of Lando, and will also be the easiest way for Lagoon newcomers to get up and running. Pygmy presents a closer integration with Docker, which will lend itself better to more complex scenarios and use cases but will also require a deeper understanding.
+We have previously evaluated adding support for other systems like Docksal and Docker4Drupal, and while we may add support for these in the future, our current focus is on supporting using Lando and pygmy. If you do have Lagoon running with one of these (or other) tools, we would love for you to submit a PR on GitHub!
+ + + + + + + + +Note
+We are working hard on getting our CLI and GraphQL API set up to allow everyone using Lagoon to setup and configure their projects themselves. Right now, it needs more testing before we can release those features, so hold tight!
+Until then, the setup of a new project involves talking to your Lagoon administrator, which is ok, as they are much friendlier than APIs. 😊
+Please have the following information ready for your Lagoon administrator:
+--) are not allowed within a project namegit@example.com:test/test.git).We suggest deploying specific important branches (like develop and main) and pull requests. But that's all up to you! (see Workflows for some more information)
This means that the .lagoon.yml and docker-compose.yml files are available in your Git repository and configured accordingly.
If this is not the case, check out the list of Step-by-Step Guides on how to do so before proceeding.
+In order to deploy your code, Lagoon needs access to it. By design and for security, Lagoon only needs read access to your Git repository.
+Your Lagoon administrator will tell you the SSH public key or the Git account to give read access to.
+Lagoon needs to be informed about a couple of events that are happening to your Git repository. Currently these are pushes and pull requests, but we may add more in the future.
+As Lagoon supports many different Git hosts, we have split off those instructions into this documentation: Configure Webhooks.
+Congratulations, you are now ready to run your first deployment.
+ + + + + + + + +