Skip to content

Commit

Permalink
add images
Browse files Browse the repository at this point in the history
Signed-off-by: Tao Yu <[email protected]>
  • Loading branch information
yyuuttaaoo committed Aug 19, 2023
1 parent 6c6995e commit 9a65d93
Show file tree
Hide file tree
Showing 10 changed files with 15 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
title: 管理日志采集sidecar容器最佳实践
---

import logoImageUrl from '/img/docs/best-practices/log-sidecar-logo.png';

<p align="center">
<img src="https://cdn.nlark.com/yuque/0/2023/png/22577445/1692329262461-b5d46880-1893-46e6-9168-b578199135bb.png#averageHue=%23f8f2ef&clientId=uf01ce5f9-459e-4&from=paste&height=220&id=u26fd26d7&originHeight=440&originWidth=980&originalType=binary&ratio=2&rotation=0&showTitle=false&size=244483&status=done&style=none&taskId=u94f77b71-7dfc-4160-bc81-a654b7def58&title=&width=490" alt="logo" width="450"/>
<img src={logoImageUrl} alt="logo" width="450"/>
</p>

## 摘要
Expand All @@ -14,8 +16,10 @@ title: 管理日志采集sidecar容器最佳实践

可观测系统是IT系统的眼睛,在K8S的 [官方文档](https://kubernetes.io/docs/concepts/cluster-administration/logging/)中介绍了多种的可观测数据采集形式,总结起来主要有下述3种:原生方式、DaemonSet方式和Sidecar方式。 三种方式都有利有弊,没有哪种方式能够完美的解决100%问题的,所以要根据场景来贴合。其中Sidecar方式为每个Pod单独部署采集agent,相对资源占用较多,但稳定性、灵活性以及多租户隔离性较强,建议在Job类任务采集场景或作为PaaS平台为多个业务方服务的集群使用该方式。

import archImageUrl from '/img/docs/best-practices/log-sidecar-arch.png';

<p align="center">
<img src="https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328115374-c77f5928-73c5-4029-a9e4-087de748f76c.png#averageHue=%23eeeeee&clientId=u644a9709-16c0-4&height=199&id=ByKKE&originHeight=436&originWidth=988&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=u93765ae5-996a-4bbd-9c55-b90b489cbf2&title=&width=451" alt="arch" width="450"/>
<img src={archImageUrl} alt="arch" width="450"/>
</p>

- 稳定性:Sidecar采集方式利用K8s同一个Pod内的容器可以共享存储和网络的特性,无容器发现过程。同时只要采集容器没有退出,共享存储卷上的文件就不会被删除,因此无论是业务容器的生命周期非常短或是日志采集出现延时的情况都可以保证采集数据的完整性。
Expand Down Expand Up @@ -161,8 +165,10 @@ flushers:
关联到机器组
import agentGroupImageUrl from '/img/docs/best-practices/log-sidecar-agent-group.png';
<p align="center">
<img src="https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328107464-45e9a018-c695-4235-bbdc-635783bada67.png#averageHue=%23f2f3f1&clientId=u644a9709-16c0-4&id=hKVm7&originHeight=656&originWidth=1776&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=ua68b8681-34b5-48d6-a0db-5f1e732b7d6&title=" alt="agent-group" width="650"/>
<img src={agentGroupImageUrl} alt="agent-group" width="650"/>
</p>
定义iLogtail SidecarSet配置,如下:
Expand Down Expand Up @@ -264,26 +270,26 @@ spec:
将Nginx Mock Job Apply到K8s集群后,发现创建的Pod都被注入了 iLogtail Sidecar 容器,说明kruise.io/inject-ilogtail: "true"起到了效果,如下:
![injection](https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328101339-80117741-b56e-4335-8868-007ea75989ec.png#averageHue=%23f5f5f5&clientId=u644a9709-16c0-4&id=fmvTf&originHeight=396&originWidth=2236&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=ue59e8880-93a5-4968-9186-9a4463f34e2&title=)
![injection](/img/docs/best-practices/log-sidecar-injection.png)
若没有此项配置,将只有nginx container而没有ilogtail container。
通过查阅K8s事件可知,ilogtail container优先于nginx container创建,说明KRUISE_CONTAINER_PRIORITY起到了效果,如下:
![priority](https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328098442-517bfcf4-1d33-4754-a5df-8c85e46b11f3.png#averageHue=%23f6f6f6&clientId=u644a9709-16c0-4&id=QsODa&originHeight=466&originWidth=2392&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=ud86988aa-23e2-4777-aec1-673922aecbc&title=)
![priority](/img/docs/best-practices/log-sidecar-priority.png)
若没有此项配置,ilogtail container可能晚于nginx container启动,导致采集数据不完整。
通过查阅K8s事件还可知,ilogtail container在nginx container完成后退出,说明KRUISE_TERMINATE_SIDECAR_WHEN_JOB_EXIT起到了效果,如下:
![exit](https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328096048-e281bd3b-71c6-4e91-b65c-2897d73f2b3a.png#averageHue=%23f4f4f4&clientId=u644a9709-16c0-4&id=pHMVZ&originHeight=362&originWidth=2350&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=u4e854dfb-0172-495c-bf6c-38647ca5e05&title=)
![exit](/img/docs/best-practices/log-sidecar-exit.png)
若没有此项配置,Pod将一直处于Running状态,因为Sidecar不会随业务容器退出,除非手动Delete Job。
以上两项Sidecar容器启动和退出的正确顺序保证了iLogtail采集数据的完整性。查询采集到Kafka的数据,从点位计算可知,数据采集完整,共100条(92-40+106-58=100),如下:
![kafka0](https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328092903-e60a6259-2cbb-4401-a506-74c1a09077c2.png#averageHue=%23fcc456&clientId=u644a9709-16c0-4&id=tc2GP&originHeight=672&originWidth=2568&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=u298fa8a1-2d3c-4c47-988e-1f59a0b3aa3&title=)
![kafka1](https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328089947-047e0856-5a59-4a1b-ad91-7ec6749d693c.png#averageHue=%23fcc456&clientId=u644a9709-16c0-4&id=MuyIy&originHeight=718&originWidth=2574&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=uaaf60971-29f6-452e-bdd3-85c1d83d615&title=)
![kafka0](/img/docs/best-practices/log-sidecar-kafka0.png)
![kafka1](/img/docs/best-practices/log-sidecar-kafka1.png)
### 独立升级iLogtail Sidecar容器(edge -> latest)[](https://openkruise.io/zh/docs/best-practices/log-container-sidecarset#%E7%8B%AC%E7%AB%8B%E5%8D%87%E7%BA%A7filebeat-sidecar%E5%AE%B9%E5%99%A8version-7162---7163)
Expand All @@ -295,7 +301,7 @@ spec:
通过K8s事件可以发现Pod并没有重建,ilogtail容器由于声明发生了变化进行了重建,而同时nginx容器并没有受影响中断。如下:
![inplace-upgrade](https://cdn.nlark.com/yuque/0/2023/png/22577445/1692328081736-02a2db39-9273-4286-a2ab-ce119fddcd2f.png#averageHue=%23f5f5f5&clientId=u644a9709-16c0-4&id=ZjyEA&originHeight=752&originWidth=2336&originalType=binary&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=uf7e565a4-9505-4a97-995e-ce29428ec76&title=)
![inplace-upgrade](/img/docs/best-practices/log-sidecar-inplace-upgrade.png)
该特性依赖Kruise原地升级的能力实现,详情参考文档:[Kruise原地升级](https://openkruise.io/zh/docs/next/core-concepts/inplace-update/)。 不过独立升级sidecar容器也存在一定的风险性,如果sidecar容器升级过程中失败,则将导致Pod Not Ready,进而影响业务,因此SidecarSet本身提供了非常丰富的灰度发布能力来尽量规避该风险, 详情参考文档:[Kruise SidecarSet](https://openkruise.io/zh/docs/next/user-manuals/sidecarset#sidecar%E6%9B%B4%E6%96%B0%E7%AD%96%E7%95%A5),如下:
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 9a65d93

Please sign in to comment.