Error when launching GraphScope on kubernetes cluster: (409)

[hi-jiangsong]

各位大侠，小弟初接触graphscope，按照文档执行，遇到许多坎坷，前面的还可以看看脚本解决，当前卡在创建session时失败，错误日志记录如下：

2021-03-12 14:47:54,555 [INFO][session:379]: Initializing graphscope session with parameters: {'num_workers': 2, 'k8s_namespace': None, 'k8s_service_type': 'NodePort', 'k8s_gs_image': 'swr.cxxxxxx/graphscope:0.2.0', 'k8s_etcd_image': 'quay.io/coreos/etcd:v3.4.13', 'k8s_image_pull_policy': 'IfNotPresent', 'k8s_image_pull_secrets': [], 'k8s_gie_graph_manager_image': 'swr.xxxxxxxxx/maxgraph_standalone_manager:0.2.0', 'k8s_zookeeper_image': 'zookeeper:3.4.14', 'k8s_coordinator_cpu': 1.0, 'k8s_coordinator_mem': '2Gi', 'k8s_etcd_cpu': 0.5, 'k8s_etcd_mem': '128Mi', 'k8s_zookeeper_cpu': 0.5, 'k8s_zookeeper_mem': '256Mi', 'k8s_gie_graph_manager_cpu': 1.0, 'k8s_gie_graph_manager_mem': '2Gi', 'k8s_vineyard_cpu': 0.5, 'k8s_vineyard_mem': '512Mi', 'k8s_vineyard_shared_mem': '2Gi', 'k8s_engine_cpu': 0.5, 'k8s_engine_mem': '2Gi', 'k8s_volumes': {}, 'k8s_waiting_for_delete': False, 'timeout_seconds': 600, 'addr': None, 'enable_k8s': True, 'k8s_minikube_vm_driver': False, 'k8s_client_config': {}}
2021-03-12 14:47:54,633 [INFO][cluster:405]: Launching coordinator...
2021-03-12 14:47:57,679 [INFO][utils:144]: coordinator-yptbuy-799ffbcc67-h6rs4: 0/1 nodes are available: 1 Insufficient memory.
2021-03-12 14:51:35,740 [INFO][cluster:424]: Launching GIE graph manager ...
2021-03-12 14:51:36,765 [ERROR][cluster:766]: Error when launching GraphScope on kubernetes cluster: (409)
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '30b0625a-3974-4d56-8315-9af9626395e6', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9a115f2b-4f76-40cc-8fa2-bc429675c962', 'Date': 'Fri, 12 Mar 2021 06:51:35 GMT', 'Content-Length': '246'})

我使用的是单节点的物理机部署，其中日志提示内存不足，我因此调整了参数，也看了机器free -g如下，很疑惑，为啥还是不足
total used free shared buff/cache available
Mem: 31 1 28 0 1 28
Swap: 0 0 0
最后面那个error应该是失败的直接原因，不是很明白这个ERROR是什么原因？

如下是我创建session的代码：

import os
import graphscope

graphscope.set_option(show_log=True)
graphscope.set_option(log_level="DEBUG")
graphscope.set_option(k8s_gs_image="swr.cn-xxxxxxxxxx/graphscope:0.2.0")
graphscope.set_option(k8s_gie_graph_manager_image="swr.xxxxxxx/maxgraph_standalone_manager:0.2.0")

graphscope.set_option(k8s_coordinator_mem="2Gi")
graphscope.set_option(k8s_gie_graph_manager_mem="2Gi")
graphscope.set_option(k8s_vineyard_shared_mem="2Gi")
graphscope.set_option(k8s_engine_mem="2Gi")

sess = graphscope.session()

还请大侠不吝赐教，多谢！

[lidongze0629]

Hi, hi-jiangsong. Thank you for posting this question in the discussions and welcome to the GraphScope discussions.

I think you should use kubectl describe node [NODE_NAME] rather than free -g to view the memory of kubernetes node.

$ kubectl get node

NAME                      STATUS   ROLES    AGE   VERSION
izj6cjaz33icbq1wohq67pz   Ready    master   95d   v1.19.2

$ kubectl describe node izj6cjaz33icbq1wohq67pz
.... some other info
Capacity:
  cpu:                24
  ephemeral-storage:  309505004Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             47784232Ki
  pods:               110
....

So the cpu and memory of this node is 24 cores and about 48G(47784232Ki)

[hi-jiangsong]

多谢！ lidongze0629
我按照你说的查看了下，内存缺失已经满了，而且有很多Non-terminated Pods未释放，我猜测是因为前面失败直接terminated导致的。我用kind delete后重新create cluster，现在已经可以完成session创建了，并看到了这个日志：
2021-03-12 15:48:40,048 [INFO][rpc:111]: GraphScope coordinator service connected.

非常感谢！