diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 0e4a5703f9..f5aad7d831 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -10,7 +10,7 @@ jobs: - uses: actions/checkout@v2 - uses: actions/setup-go@v2 with: - go-version: '^1.13.1' + go-version: '1.15.x' - uses: actions/setup-python@v2 with: python-version: '3.x' diff --git a/rfcs/text/0012-orchestrator-field-set.md b/rfcs/text/0012-orchestrator-field-set.md index 207af5c763..5b624a9faa 100644 --- a/rfcs/text/0012-orchestrator-field-set.md +++ b/rfcs/text/0012-orchestrator-field-set.md @@ -1,6 +1,6 @@ # 0012: Orchestrator field set creation -- Stage: **0 (strawperson)** +- Stage: **1 (draft)** - Date: **2021-01-11** There is currently no ECS field set for container orchestration engines. There is an example of an ECS @@ -19,9 +19,79 @@ add a new schema rather than change existing material. ## Fields - +The proposed change adds six fields, as described below: + +``` +--- +- name: orchestrator + title: Orchestrator + group: 2 + short: Fields relevant to container orchestrators. + description: > + Fields that describe the resources which container orchestrators manage or + act upon. + type: group + fields: + - name: cluster + level: extended + type: object + description: > + Orchestrator cluster details. + + - name: cluster.name + level: extended + type: keyword + description: > + Name of the cluster. + + - name: cluster.url + level: extended + type: keyword + description: > + URL of the cluster. + + - name: type + level: extended + type: keyword + example: kubernetes + description: > + Orchestrator cluster type (e.g. kubernetes, nomad or cloudfoundry). + + - name: organization + level: extended + type: keyword + example: elastic + description: > + Organization affected by the event (for multi-tenant orchestrator setups). + + - name: namespace + level: extended + type: keyword + example: kube-system + description: > + Namespace in which the action is taking place. + + - name: resource.name + level: extended + type: keyword + example: test-pod-cdcws + description: > + Name of the resource being acted upon. + + - name: resource.type + level: extended + type: keyword + example: service + description: > + Type of resource being acted upon. + + - name: api_version + level: extended + example: v1beta1 + type: keyword + description: > + API version being used to carry out the action +``` +The `orchestrator` field set will be used to capture typical concepts employed +by container orchestrators to manage resources. The key intent of this is to create +a consistent method by which audit logs from container orchestrators can +be compared. For example, this would allow the creation of open source detection +rulesets for suspicious Kubernetes events based on audit logs, which can be easily +transferred from one cluster to another without depending on the specifics of +parsing implementations. + +This might also have use in performance and monitoring tooling which exists around +container orchestrators, allowing for the definition of shareable dashboards and +alert definitions. ## Source data - +Examples of source data include: + +- [Kubernetes audit logs][1] +- [Kubernetes node logs][2] +- [HashiCorp Nomad audit logs][3] +- [Falco alert logs][4] + +### Kubernetes audit log + +```json +{ + "_index": "filebeat-7.7.0-2020.12.31-000001", + "_type": "_doc", + "_id": "KbmPuXYBaTdcl42uyGfl", + "_version": 1, + "_score": null, + "_source": { + "@timestamp": "2020-12-31T16:09:35.735Z", + "log": { + "offset": 7248566, + "file": { + "path": "/tmp/host-logs/kube-apiserver-audit.log" + } + }, + "cloud.provider": "gcp", + "event.action": "create", + "orchestrator.cluster": { + "name": "test-dev", + }, + "orchestrator.type": "kubernetes", + "orchestrator.subresource": "attach", + "orchestrator.resource.type": "pod", + "orchestrator.namespace": "default", + "orchestrator.resource.name": "test", + "orchestrator.api_version": "v1", + "user.name": "system:serviceaccount:test" + } +} +``` + +### Hashicorp Nomad audit log + +```json +{ + "created_at": "2020-03-24T13:09:35.704224536-04:00", + "event_type": "audit", + "orchestrator.api_version": "v1", + "orchestrator.namespace": "default", + "orchestrator.resource.type": "nodes", + "orchestrator.type": "nomad", + "payload": { + "id": "8b826146-b264-af15-6526-29cb905145aa", + "stage": "OperationComplete", + "type": "audit", + "timestamp": "2020-03-24T13:09:35.703865005-04:00", + "version": 1, + "auth": { + "accessor_id": "a162f017-bcf7-900c-e22a-a2a8cbbcef53", + "name": "Bootstrap Token", + "global": true, + "create_time": "2020-03-24T17:08:35.086591881Z" + }, + "request": { + "id": "02f0ac35-c7e8-0871-5a58-ee9dbc0a70ea", + "event.action": "GET", + "request_meta": { + "remote_address": "127.0.0.1:33648", + "user_agent": "Go-http-client/1.1" + }, + "node_meta": { + "ip": "127.0.0.1:4646" + } + }, + "response": { + "status_code": 200 + } + } +} +``` +### Kubernetes-specific logic + +The key concern here is the dominance of one particular container orchestration +system - Kubernetes - over the rest of the ecosystem. Other orchestrators include +options like HashiCorp Nomad, Docker Swarm, and Apache Mesos, but it is unclear to +what extent the alternatives share the same logical primitives as Kubernetes. An +attempt has been made to ensure that the proposed field set is as generic and flexible +as possible, however it would be useful to consider in some detail whether the +preference is to keep the field set as generic as possible, or large enough to +cover all the logical primitives of popular orchestrators. Input from contributors +who have experience with the various alternative orchestration providers would be +particularly valuable. * Stage 0: https://github.com/elastic/ecs/pull/1209 +* Stage 1: https://github.com/elastic/ecs/pull/1230