# bcs-monitor 本地开发10分钟启动教程(helm) / 文档

bcs-services/bcs-monitor/docs/helm_deploy.md

@@ -0,0 +1,192 @@
+
+
+# bcs-monitor 本地开发10分钟启动教程(helm) / 文档
+
+
+
+## 机器环境依赖
+
+```
+go               1.17
+Docker           20.10.17
+k8s(minikube)    1.23.5
+git              2.38.1      
+helm             3.9.4
+```
+
+
+
+## 操作步骤
+
+### 1.下载源代码
+
+git clone https://github.com/Tencent/bk-bcs.git
+
+### 2.切换目录，进入到bcs-monitor目录
+
+```
+[root@control-plane helm]# cd /data/workspace/src/deploy/bcs-monitor/bcs-monitor/
+[root@control-plane bcs-monitor]# ll
+total 372
+drwxr-xr-x.  2 root root     25 Feb 13 11:48 bin
+-rw-r--r--.  1 root ryaoxinwangoot      0 Feb 13 11:37 CHANGELOG.md
+drwxr-xr-x.  3 root root     25 Feb 13 11:37 cmd
+-rw-r--r--.  1 root root    148 Feb 13 11:37 Dockerfile
+drwxr-xr-x.  2 root root     41 Feb 13 11:37 docs
+drwxr-xr-x.  2 root root     78 Feb 13 11:37 etc
+-rw-r--r--.  1 root root  11602 Feb 13 11:37 go.mod
+-rw-r--r--.  1 root root 342884 Feb 13 11:37 go.sum
+-rw-r--r--.  1 root root   1088 Feb 13 11:37 LICENSE
+-rw-r--r--.  1 root root   2141 Feb 13 11:37 Makefile
+drwxr-xr-x. 13 root root    163 Feb 13 11:37 pkg
+-rw-r--r--.  1 root root     16 Feb 13 11:37 README.md
+-rw-r--r--.  1 root root   1141 Feb 13 11:37 run-test.sh
+drwxr-xr-x.  4 root root     54 Feb 13 11:37 test
+-rw-r--r--.  1 root root      5 Feb 13 11:37 VERSION
+
+```
+
+### 3.构建docker镜像
+
+```
+注意： dockerfile中复制bin目录下的可行性文件，所以需要先make build，后make docker
+make build
+make docker 
+```
+
+### 4.修改values.yaml，按照自己的配置执行helm chat install 安装bcs-monitor
+
+```
+helm install --generate-name  --debug  ./bcs-monitor   -n bcs-monitor
+```
+
+## 要点
+
+### 1.关于storegw
+
+#### config.storegw.type 与config.storegw.config.url
+
+storegw 对接 prometheus，所以确定自己value.yaml中声明configmap的内容中config.storegw.type值为PROMETHEUS,并在config.storegw.config.url设置
+
+prometheus的访问url
+
+```
+
+config
+  storegw:
+    #- type: BCS_SYSTEM
+    - type: PROMETHEUS
+      config: 
+      	url: "http://192.168.37.133:30090"
+
+```
+
+```
+[root@control-plane bcs-monitor]# k get svc -n monitoring
+NAME                                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
+alertmanager-operated                          ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   5h1m
+prometheus-operated                            ClusterIP   None             <none>        9090/TCP                     5h1m
+prometheus-operator-grafana                    ClusterIP   10.107.72.98     <none>        80/TCP                       5h1m
+prometheus-operator-kube-p-alertmanager        ClusterIP   10.96.18.143     <none>        9093/TCP                     5h1m
+prometheus-operator-kube-p-operator            ClusterIP   10.98.193.108    <none>        443/TCP                      5h1m
+prometheus-operator-kube-p-prometheus          NodePort    10.107.100.32    <none>        9090:30090/TCP               5h1m
+prometheus-operator-kube-state-metrics         ClusterIP   10.111.36.218    <none>        8080/TCP                     5h1m
+prometheus-operator-prometheus-node-exporter   ClusterIP   10.102.102.230   <none>        9100/TCP                     5h1m
+
+```
+<font color='red'> 
+如上prometheus servie nodeport为30090
+</font>
+
+
+
+
+
+### 2.关于权限
+
+value.yaml中默认dev: false不开启权限校验,该字段控制deployment-xxx.yaml中关于权限校验的configmap挂载
+
+```
+credentials:
+  enabled: false
+```
+
+Error: Invalid Key: Key must be PEM encoded PKCS1 or PKCS8 private key如果容器中出现这个错误，修改配置文件中关于configmap的配置，确保最后挂载到容器中的配置文件不包含以下三种权限认证的配置
+
+```
+config:
+  auth_conf:
+    host: ""
+    is_gateway: ""
+    ssm_host: ""
+  bkapigw_conf:
+    host: ""
+    jwt_public_key: ""
+
+  bcs_conf:
+    verify: false
+
+
+
+```
+
+同时需要进入本地开发模式
+
+values.global.env.BCS_MONITOR_USERNAME:  "test_dev"   ##（不为空的字符串）
+
+values.config.run_env: dev
+
+## 补充
+
+### 1.prometheus安装
+
+storegw对接 prometheus，所以需要提供prometheus
+
+
+
+安装方式1.单机版
+
+github搜索prometheus release包自行解压安装
+
+
+
+安装方式2.prometheus-operator（推荐）
+
+```
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+kubectl create ns monitoring
+helm install prometheus-operator prometheus-community/kube-prometheus-stack --prometheus.service.type=NodePort --namespace monitoring 
+```
+
+
+
+有个别镜像需要翻墙不好下载，如下
+
+```
+docker pull paymanfu/ingress-nginx_kube-webhook-certgen:v1.3.0
+docker tag  paymanfu/ingress-nginx_kube-webhook-certgen:v1.3.0  registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0
+
+
+
+docker pull k8scopy/kube-state-metrics:v2.7.0
+docker tag  k8scopy/kube-state-metrics:v2.7.0 registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0
+```
+
+
+
+
+
+
+
+## 部署效果图
+
+![image-20230216135432844](E:\goworkspace\src\bk-bcs\bcs-services\bcs-monitor\docs\helm_deploy.assets\image-20230216135432844.png)
+
+
+
+![image-20230216135517887](E:\goworkspace\src\bk-bcs\bcs-services\bcs-monitor\docs\helm_deploy.assets\image-20230216135517887.png)
+
+![image-20230216135455203](E:\goworkspace\src\bk-bcs\bcs-services\bcs-monitor\docs\helm_deploy.assets\image-20230216135455203.png)
+
+![image-20230216135556366](E:\goworkspace\src\bk-bcs\bcs-services\bcs-monitor\docs\helm_deploy.assets\image-20230216135556366.png)

install/helm/bcs-monitor/Chart.yaml

@@ -4,10 +4,8 @@ description: A Helm chart for module bcs-monitor of BlueKing Container Service(B
 type: application
 version: 1.28.0-alpha.75
 appVersion: v1.28.0-alpha.75
-dependencies:
-  - name: common
-    repository: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
-    version: 1.4.2
-  - name: bcs-common
-    repository: file://../bcs-common
-    version: x.x.x
+##可选 如果环境缺少prometheus可选
+#dependencies:
+#  - name: kube-prometheus-stack
+#    repository: https://prometheus-community.github.io/helm-charts
+#    version: 45.0.0

install/helm/bcs-monitor/templates/_helpers.tpl

@@ -83,13 +83,8 @@ Create the name of the service account to use
   value: {{ .Values.global.storage.redis.password }}
 - name: BCS_APIGW_TOKEN
   value: {{ .Values.global.env.BK_BCS_gatewayToken}}
-- name: BCS_APIGW_PUBLIC_KEY
-  valueFrom:
-    secretKeyRef:
-      name: bcs-jwt
-      key: public.key
-- name: bcsEtcdHost
-  value: "{{ include "bcs-common.etcd.host" ( dict "localStorage" .Values.storage "globalStorage" .Values.global.storage "namespace" .Release.Namespace ) }}"
+- name: BCS_MONITOR_USERNAME
+  value: {{ .Values.global.env.BCS_MONITOR_USERNAME}}
 - name: BKIAM_GATEWAY_SERVER
   value: {{ .Values.global.bkIAM.gateWayHost}}
 {{- end }}

install/helm/bcs-monitor/templates/configmap.yaml

@@ -1,16 +1,17 @@
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: bcs-monitor
+  name: bcs-monitor-config
   labels:
     app: bcs-monitor
     chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
     release: "{{ .Release.Name }}"
 data:
-  config.yaml: |
+  bcs-monitor.yml: |
     {{- toYaml .Values.config | nindent 4 }}
 
 ---
+{{- if eq .Values.credentials.enabled true }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -22,8 +23,9 @@ metadata:
 data:
   credentials.yaml: |-
     {{- toYaml .Values.config_credentials | nindent 4 }}
-
+{{- end }}
 ---
+{{- if eq .Values.credentials.enabled true }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -35,3 +37,4 @@ metadata:
 data:
   credentials.yaml: |-
     {{- toYaml .Values.config_credentials_manager | nindent 4 }}
+{{- end }}

install/helm/bcs-monitor/templates/deployment-api.yaml

@@ -32,20 +32,19 @@ spec:
       securityContext:
         {{- toYaml .Values.podSecurityContext | nindent 8 }}
       containers:
-        - name: {{ .Chart.Name }}
+        - name: bcs-monitor-api
           securityContext:
             {{- toYaml .Values.securityContext | nindent 12 }}
-          image: {{ include "common.images.image" ( dict "imageRoot" .Values.image "global" .Values.global) }}
-          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          image: bcs-monitor:latest
+          imagePullPolicy: Never
           env:
             {{- include "bcs-monitor.envs" . | trim | nindent 12 }}
+          command: ["./bcs-monitor"]
           args:
-            - api
+            - "api"
             - --bind-address=$(POD_IP)
-            - --http-port=10902
-            - --config=/etc/bcs-monitor/config.yaml
-            - --credential-config=/etc/bcs-monitor-cred/credentials.yaml
-            - --credential-config=/etc/bcs-monitor-cred-manager/credentials.yaml
+            - "--http-port=10902"
+            - "--config=./config/bcs-monitor.yml"
           ports:
             - name: http
               containerPort: 10902
@@ -59,39 +58,29 @@ spec:
               path: /-/ready
               port: http
           volumeMounts:
-          - name: config
-            mountPath: /etc/bcs-monitor
-          - name: config-credentials
-            mountPath: /etc/bcs-monitor-cred
-          - name: config-credentials-manager
-            mountPath: /etc/bcs-monitor-cred-manager
-          - name: etcd-certs
-            mountPath: /data/bcs/cert/etcd
-          - name: bcs-certs
-            mountPath: /data/bcs/cert/bcs
+            - mountPath: "/config"
+              name: config-volume
+            {{- if eq .Values.credentials.enabled true }}
+            - name: config-credentials
+              mountPath: /etc/bcs-monitor-cred
+            - name: config-credentials-manager
+              mountPath: /etc/bcs-monitor-cred-manager
+            - name: etcd-certs
+              mountPath: /data/bcs/cert/etcd
+            - name: bcs-certs
+              mountPath: /data/bcs/cert/bcs
+            {{- end }}
           resources:
             {{- toYaml .Values.resources | nindent 12 }}
-
       volumes:
-      - name: config
-        configMap:
-          name: bcs-monitor
-      - name: config-credentials
-        configMap:
-          name: bcs-monitor-credentials
-      - name: config-credentials-manager
-        configMap:
-          name: bcs-monitor-credentials-manager
-      {{- include "bcs-common.volumes.bcsServices" . | indent 6 }}
-      {{- with .Values.nodeSelector }}
-      nodeSelector:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-      {{- with .Values.affinity }}
-      affinity:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-      {{- with .Values.tolerations }}
-      tolerations:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
+        - name: config-volume
+          configMap:
+            name: bcs-monitor-config     #定义的bcs-monitor.yaml
+        {{- if eq .Values.credentials.enabled true }}
+        - name: config-credentials
+          configMap:
+            name: bcs-monitor-credentials
+        - name: config-credentials-manager
+          configMap:
+            name: bcs-monitor-credentials-manager
+        {{- end }}