Skip to content

Commit

Permalink
Merge pull request #156 from buildkite/hosted_agents_api
Browse files Browse the repository at this point in the history
Creating a hosted queue via GraphQL and REST
  • Loading branch information
matthewborden authored Dec 19, 2024
2 parents f5ada00 + c68dcb0 commit 7857c25
Show file tree
Hide file tree
Showing 15 changed files with 525 additions and 62 deletions.
2 changes: 2 additions & 0 deletions data/nav.yml
Original file line number Diff line number Diff line change
Expand Up @@ -725,6 +725,8 @@
path: apis/graphql/cookbooks/clusters
- name: GitHub rate limits
path: apis/graphql/cookbooks/github-rate-limits
- name: Hosted agents
path: apis/graphql/cookbooks/hosted-agents
- name: Jobs
path: apis/graphql/cookbooks/jobs
- name: Pipelines
Expand Down
2 changes: 1 addition & 1 deletion pages/apis/descriptions/_common_create_cluster_fields.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@
- `color` (optional) provides the background color for this emoji and uses hex code syntax (for example, `#FFE0F1`).

> 📘 A default queue is not automatically created
> Unlike creating a new cluster through the [Buildkite interface](#create-a-cluster-using-the-buildkite-interface), a default queue is not automatically created using this API call. To create a new/default queue for any new cluster created through an API call, you need to manually [create a new queue](/docs/pipelines/clusters/manage-queues#create-a-queue).
> Unlike creating a new cluster through the [Buildkite interface](#create-a-cluster-using-the-buildkite-interface), a default queue is not automatically created using this API call. To create a new/default queue for any new cluster created through an API call, you need to manually [create a new queue](/docs/pipelines/clusters/manage-queues#create-a-self-hosted-queue).
98 changes: 98 additions & 0 deletions pages/apis/graphql/cookbooks/hosted_agents.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Hosted agents

A collection of common tasks with [Hosted agents](/docs/pipelines/hosted-agents) using the GraphQL API.

## Create a Buildkite hosted queue

```graphql
mutation {
clusterQueueCreate(
input: {
organizationId: "organization-id"
clusterId: "cluster-id"
key: "hosted_linux_small"
description: "Small AMD64 Linux agents hosted by Buildkite."
hostedAgents: {
instanceShape: LINUX_AMD64_2X4
}
}
) {
clusterQueue {
id
uuid
key
description
dispatchPaused
hosted
hostedAgents {
instanceShape {
name
size
vcpu
memory
}
}
createdBy {
id
uuid
name
email
avatar {
url
}
}
}
}
}
```

Creates a small Buildkite hosted queue using AMD64-based Linux Buildkite hosted agents. The `instanceShape` value is referenced from the [InstanceShape](/docs/apis/graphql/schemas/enum/hostedagentinstanceshapename) enum, and represents the combination of machine type, architecture, CPU and Memory available to each job running on a hosted queue. The `LINUX_AMD64_2X4` value is a Linux AMD64 2 vCPU and 4 GB memory instance.

Learn more about the instance shapes available for [Linux](#instance-shape-values-for-linux) and [Mac](#instance-shape-values-for-mac) Buildkite hosted agents.

## Change the instance shape of a Buildkite hosted queue's agents

```graphql
mutation {
clusterQueueUpdate(
input: {
organizationId: "organization-id"
id: "cluster-queue-id"
hostedAgents: {
instanceShape: LINUX_AMD64_4X16
}
}
) {
clusterQueue {
id
hostedAgents {
instanceShape {
name
size
vcpu
memory
}
}
}
}
}
```

To increase the size of the AMD64-based Linux agent instances for a Buildkite hosted queue, update the `instanceShape` value to a one of a greater size, such as `LINUX_AMD64_4X8`, which is a 4 vCPU and 8 GB memory. This allows you to scale the resources available to each job running on this Buildkite hosted queue.

Learn more about the instance shapes available for [Linux](#instance-shape-values-for-linux) and [Mac](#instance-shape-values-for-mac) Buildkite hosted agents.

> 📘
> It is only possible to change the _size_ of the current instance shape assigned to this queue. It is not possible to change the current instance shape's machine type (from macOS to Linux, or vice versa), or for a Linux machine, its architecture (from AMD64 to ARM64, or vice versa).
## Instance shape values for Linux

Specify the appropriate **Instance shape** for the `instanceShape` value in your GraphQL API mutation.

<%= render_markdown partial: 'shared/hosted_agents/hosted_agents_instance_shape_table_linux' %>

## Instance shape values for Mac

Specify the appropriate **Instance shape** for the `instanceShape` value in your GraphQL API mutation.

<%= render_markdown partial: 'shared/hosted_agents/hosted_agents_instance_shape_table_mac' %>
1 change: 1 addition & 0 deletions pages/apis/graphql/graphql_cookbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ There are recipes for a range of different topics, including:
- [Builds](/docs/apis/graphql/cookbooks/builds)
- [Clusters](/docs/apis/graphql/cookbooks/clusters)
- [GitHub rate limits](/docs/apis/graphql/cookbooks/github-rate-limits)
- [Hosted agents](/docs/apis/graphql/cookbooks/hosted-agents)
- [Jobs](/docs/apis/graphql/cookbooks/jobs)
- [Pipelines](/docs/apis/graphql/cookbooks/pipelines)
- [Pipeline templates](/docs/apis/graphql/cookbooks/pipeline-templates)
Expand Down
111 changes: 110 additions & 1 deletion pages/apis/rest_api/clusters.md
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ Required scope: `read_clusters`

Success response: `200 OK`

### Create a queue
### Create a self-hosted queue

```bash
curl -H "Authorization: Bearer $TOKEN" \
Expand Down Expand Up @@ -392,11 +392,91 @@ Required [request body properties](/docs/api#request-body-properties):
</tbody>
</table>

Required scope: `write_clusters`

Success response: `201 Created`

Error responses:

<table class="responsive-table">
<tbody>
<tr><th><code>422 Unprocessable Entity</code></th><td><code>{ "message": "Validation failed: Reason for failure" }</code></td></tr>
</tbody>
</table>

### Create a Buildkite hosted queue

```bash
curl -H "Authorization: Bearer $TOKEN" \
-X POST "https://api.buildkite.com/v2/organizations/{org.slug}/clusters/{cluster.id}/queues" \
-H "Content-Type: application/json" \
-d '{ "key": "default", "description": "Queue of hosted Buildkite agents", "hostedAgents": { "instanceShape": "LINUX_AMD64_2X4" } }'
```

```json
{
"id": "01885682-55a7-44f5-84f3-0402fb452e66",
"graphql_id": "Q2x1c3Rlci0tLTQyZjFhN2RhLTgxMmQtNDQzMC05M2Q4LTFjYzdjMzNhNmJjZg==",
"key": "default",
"description": "Queue of hosted Buildkite agents",
"url": "http://api.buildkite.com/v2/organizations/test/clusters/42f1a7da-812d-4430-93d8-1cc7c33a6bcf/queues/01885682-55a7-44f5-84f3-0402fb452e66",
"web_url": "http://buildkite.com/organizations/test/clusters/42f1a7da-812d-4430-93d8-1cc7c33a6bcf/queues/01885682-55a7-44f5-84f3-0402fb452e66",
"cluster_url": "http://api.buildkite.com/v2/organizations/test/clusters/42f1a7da-812d-4430-93d8-1cc7c33a6bcf",
"dispatch_paused": false,
"dispatch_paused_by": null,
"dispatch_paused_at": null,
"dispatch_paused_note": null,
"created_at": "2023-05-03T04:17:55.867Z",
"created_by": {
"id": "0187dfd4-92cf-4b01-907b-1146c8525dde",
"graphql_id": "VXNlci0tLTAxODdkZmQ0LTkyY2YtNGIwMS05MDdiLTExNDZjODUyNWRkZQ==",
"name": "Sam Kim",
"email": "[email protected]",
"avatar_url": "https://www.gravatar.com/avatar/example",
"created_at": "2023-05-03T04:17:43.118Z"
},
"hosted": true,
"hosted_agents": {
"instance_shape": {
"machine_type": "linux",
"architecture": "amd64",
"cpu": 2,
"memory": 4,
"name": "LINUX_AMD64_2X4"
}
}
}
```

Required [request body properties](/docs/api#request-body-properties):

<table class="responsive-table">
<tbody>
<tr><th><code>key</code></th><td>Key for the queue.<br><em>Example:</em> <code>"default"</code>
</tbody>
</table>

Optional [request body properties](/docs/api#request-body-properties):

<table class="responsive-table">
<tbody>
<tr><th><code>description</code></th><td>Description for the queue.<br><em>Example:</em> <code>"The default queue for this cluster"</code>
<tr>
<th><code>hostedAgents</code></th>
<td>
Configures this queue to use <a href="/docs/pipelines/hosted-agents">Buildkite hosted agents</a>, along with its <em>instance shape</em>. This makes the queue a <a href="/docs/pipelines/clusters/manage-queues#create-a-buildkite-hosted-queue">Buildkite hosted queue</a>.
<br>
<em>Example:</em>
<br/>
<code>
{ "instanceShape": "LINUX_AMD64_2X4" }
</code>
<br/>
<code>instanceShape</code> (required when <code>hostedAgents</code> is specified): Describes the machine type, architecture, CPU, and RAM to provision for Buildkite hosted agent instances running jobs in this queue.
<br/>
Learn more about the instance shapes available for <a href="#queues-instance-shape-values-for-linux">Linux</a> and <a href="#queues-instance-shape-values-for-mac">Mac</a> hosted agents.
</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -451,6 +531,23 @@ curl -H "Authorization: Bearer $TOKEN" \
<table class="responsive-table">
<tbody>
<tr><th><code>description</code></th><td>Description for the queue.<br><em>Example:</em> <code>"The default queue for this cluster"</code>
<tr>
<th><code>hostedAgents</code></th>
<td>
Configures this queue to use <a href="/docs/pipelines/hosted-agents">Buildkite hosted agents</a>, along with its <em>instance shape</em>. This makes the queue a <a href="/docs/pipelines/clusters/manage-queues#create-a-buildkite-hosted-queue">Buildkite hosted queue</a>.
<br>
<em>Example:</em>
<br/>
<code>
{ "instanceShape": "LINUX_AMD64_2X4" }
</code>
<br/>
<code>instanceShape</code> (required when <code>hostedAgents</code> is specified): Describes the machine type, architecture, CPU, and RAM to provision for Buildkite hosted agent instances running jobs in this queue.
<br/>
It is only possible to change the <em>size</em> of the current instance shape assigned to this queue. It is not possible to change the current instance shape's machine type (from macOS to Linux, or vice versa), or for a Linux machine, its architecture (from AMD64 to ARM64, or vice versa).<br/>
Learn more about the instance shapes available for <a href="#queues-instance-shape-values-for-linux">Linux</a> and <a href="#queues-instance-shape-values-for-mac">Mac</a> Buildkite hosted agents.
</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -599,6 +696,18 @@ Error responses:
</tbody>
</table>

### Instance shape values for Linux

Specify the appropriate **Instance shape** for the `instanceShape` value in your REST API call.

<%= render_markdown partial: 'shared/hosted_agents/hosted_agents_instance_shape_table_linux' %>

### Instance shape values for Mac

Specify the appropriate **Instance shape** for the `instanceShape` value in your REST API call.

<%= render_markdown partial: 'shared/hosted_agents/hosted_agents_instance_shape_table_mac' %>

## Agent tokens

An agent token is used to [connect agents to a cluster](/docs/pipelines/clusters/manage-clusters#connect-agents-to-a-cluster).
Expand Down
2 changes: 1 addition & 1 deletion pages/pipelines/clusters.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Learn more about working with clusters in [Manage clusters](/docs/pipelines/clus
The most common queue attributes are based on infrastructure set-ups, such as:

- Architecture (x86, arm64, Apple silicon, etc.)
- Size of agents (small, medium, large)
- Size of agents (small, medium, large, extra large)
- Type of machine (macOS, Linux, Windows, GPU, etc.)

Therefore, an example queue would be `small_mac_silicon`.
Expand Down
4 changes: 2 additions & 2 deletions pages/pipelines/clusters/manage_clusters.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,9 +238,9 @@ where:

## Manage maintainers on a cluster

Buildkite administrators or users with the [_change organization_ permission](/docs/platform/team-management/permissions) can create clusters.
A user who is a [_Buildkite organization administrator_](/docs/platform/team-management/permissions#manage-teams-and-permissions-organization-level-permissions) can create clusters. Learn more about user permissions in Buildkite from [User and team permissions](/docs/platform/team-management/permissions).

As one of these types of users, you can add and manage other users or teams in your Buildkite organization as _maintainers_ of a cluster in the organization. A cluster maintainer can:
As a Buildkite organization administrator, you can add and manage other users or teams in your Buildkite organization as _maintainers_ of a cluster in the organization. A cluster maintainer can:

- Update or delete the cluster.
- Manage [agent tokens](/docs/agent/v3/tokens) associated with the cluster.
Expand Down
Loading

0 comments on commit 7857c25

Please sign in to comment.