Dogfooding by SSJ

[ds-ssj]

Task 1: 在本地试用 GraphScope

• 安装 GraphScope
• 载入 ogbn_mag 内置图
• Project ogbn_mag到一个简单图，选择点标签为 paper, 边标签为 cites, 属性都为空。
• 在简单图上运行一个 Kcore 算法, 并打印结果为 pandas.DataFrame
• 在 ogbn_mag图上启动交互引擎 （Interactive）
• 通过 gremlin 计算图有多少点，边
• 对于 ogbn_mag图，使用 learning 定义并执行一个 GCN 训练过程，可参考 CI 或文档中定义的过程
• 退出 GraphScope。

Task 2: 在本地开发调试 GraphScope

• 通过 vineyardd直接启动 vineyard

• 从源代码编译 Analytical Engine，找到编译出的binary grape_engine， 通过 grape_engine直接运行 analytical_engine，然后退出 grape_engine 进程

  • 需要安装 protobuf, java (jni), OpenMP

    brew install protobuf
    brew install llvm
    
    export JAVA_HOME=`/usr/libexec/java_home`
    export CXX=/opt/homebrew/opt/llvm/bin/clang++
    
  • 编译方式1

    cmake ../analytical_engine
    make -j
    
  • 编译方式2

     ./gs install-deps dev --cn
     ./gs make analytical
    

• 为 GAE 添加一条日志 ( analytical_engine/core/server/analytical_server.cc)，并重新编译，启动并找到新添加的日志

• 从源代码编译 Interactive Engine，找到编译出的 frontend-x-SNAPSHOT.jar和 gaia_executor,直接运行这个 jar 包和这个 gaia_executor，使 frontend 能够连接到 executor。

  • 分别在/opt/graphscope/lib/和/opt/graphscope/bin下
  • 参考 GIE: Dev and Test

• 了解 Frontend 和 Executor 的代码位置和组织结构，为 GIE 的 Frontend (interactive_engine/frontend/src/main/java/com/alibaba/graphscope/frontend/Frontend.java) 和 executor (interactive_engine/executor/assembly/v6d/src/bin/gaia_executor.rs) 各添加一条日志，并重新编译，启动并找到新添加的日志

• 从源代码编译 Learning Engine, 找到 libgraphlearn_shared.so

• 使用 setup.py 安装 coordinator 和 client。 注意因依赖关系， client 安装要在 coordinator 之前

• 继续安装 GAE，GIE，和 GLE 到预设安装目录。（上一步只有编译。相当于 make 和 make install 的区别）

• 找到 Client， Coordinator， GAE，GIE 和 GLE 的安装位置目录

• 若一切正常，这时你能通过 python 拉起 graphscope。启动python，import graphscope 并载入一个内置 modern_graph 图，在图上启动图交互引擎，发送一个查所有点的语句。

• 退出 graphscope
  

Task 3: 运行本地测试

• 找到 CI 定义的位置和文件
• 找到单元测试的位置，参考在 CI 中是如何运行这些单元测试的步骤，并在本地运行CI上的本地测试 （如果没有，可开 Issue 加入运行）

  # example
  brew install act
  act -j build-gae
  
• GAE 的单元测试
• GIE 的单元测试
• GLE 的单元测试
• 运行一组 Python 中的单元测试 test_min.py 。（Python 中的单元测试更像是集成测试，因为往往由多个引擎参与）
• 在 python/graphscope/tests/minitest下新增一个test 文件，在里面添加一个涉及多个引擎的测试，其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。
  

Task 4: 在 Kubernetes 上使用 GraphScope （Python SDK）

• 通过客户端在 K8S 环境下安装 Graphscope，建立一个 Session，通过 kubectl 查看拉起来的 Pod 和 Service （可通过 minikube/kind 在单机搭建 K8S 环境）
• 通过载图 API 载一个图 ogbn_mag
• 在图上运行一个 LPA 算法，并打印结果为 pandas.DataFrame
• 在图上启动交互引擎 （Interactive）
• 通过 gremlin 计算图有多少点，边
• 使用 learning 定义并执行一个 GCN 训练过程
• 关闭 Session

Task 5: 在 Kubernetes 上使用 GraphScope （Helm）

• 通过 Helm 安装 GraphScope，并载图，运行算法，Gremlin 和 Learning

Task 6: 独立部署

• 独立部署 GAE，并载图 modern_graph，运行 LPA 算法
• 独立部署 GIE，发送 gremlin 查询点边数量
  

Task 7: 打包镜像与了解 Kubernetes 上运行的基本概念

• 构建 GAE 的镜像
• 构建 GIE 的镜像 （包括 Frontend 和 Executor）
• 构建 GLE 的镜像
• 构建 Coordinator 的镜像
• 为 GAE 添加一条日志 ( analytical_engine/core/server/analytical_server.cc) 并重新构建镜像，在不使用 docker push的情况下，使用新的镜像启动并在日志中可以找到新添加的日志 （使用 registry 和 tag 来选择使用哪个镜像）
• 为 GIE 的 Frontend (interactive_engine/frontend/src/main/java/com/alibaba/graphscope/frontend/Frontend.java) 和 executor (interactive_engine/executor/assembly/v6d/src/bin/gaia_executor.rs) 各添加一条日志 并重新构建 Frontend 和 Executor 镜像，在不使用 docker push的情况下，使用新的镜像启动并可以找到新添加的日志
• 为 Coordinator 添加一个异常 (coordinator/gscoordinator/coordinator.py)，并重新构建 Coordinator 镜像。启动并找到异常的报错。（日志在控制台中或者在 Coordinator Pod 的日志中）
• 清除所有 GraphScope 的资源，包括镜像，容器。
  

Task 8: 运行 K8S 上的测试

• 找到在 Kubernetes 上运行的测试，找到在 CI 中是如何运行这些单元测试的
• 参考 CI 的步骤，运行 test_demo_scripts.py 中的 test_demo_distribute 测试
• 尝试添加一个在 Kubernetes 上运行的测试，并运行。其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。

---

Advance 部分

Task 9 : 打包为 Wheel 包

• 从源码构建 Python graphscope-client Wheels
• 从源码构建 Python graphscope wheels
• 安装构建出的 Wheel

System Info

$ uname -a
Darwin B-KYVP27MJ-2147.local 21.6.0 Darwin Kernel Version 21.6.0: Thu Mar  9 20:12:21 PST 2023; root:xnu-8020.240.18.700.8~1/RELEASE_ARM64_T6000 arm64

$ python3 --version
Python 3.11.2

[ds-ssj]

Error:

[ds-ssj]

Solution:

pip3 install gremlinpython==3.6.3rc1

[ds-ssj]

Graph Loading:

[ds-ssj]

Thanks for your help. This bug was fixed by your solution. Now I have the same problem as @yecol when using projection.

Loading edge labeled writes: 100%|██████████| 10/10 [00:00<00:00, 255.51it/s]
I20230423 14:21:54.438951 979747 etcd_meta_service.cc:159] etcd lock use 4492 microseconds
I20230423 14:21:54.438979 979766 etcd_meta_service.cc:42] etcd watch use 194499 microseconds, event size = 1
I20230423 14:21:54.448297 979748 etcd_meta_service.cc:235] etcd (last) txn use 454 microseconds
I20230423 14:21:54.448415 979766 etcd_meta_service.cc:42] etcd watch use 9414 microseconds, event size = 49
I20230423 14:21:54.452672 979766 etcd_meta_service.cc:42] etcd watch use 4172 microseconds, event size = 1
I20230423 14:21:54.482782 979766 etcd_meta_service.cc:42] etcd watch use 30099 microseconds, event size = 1
I20230423 14:21:54.482797 979748 etcd_meta_service.cc:159] etcd lock use 4599 microseconds
I20230423 14:21:54.491819 979766 etcd_meta_service.cc:42] etcd watch use 9006 microseconds, event size = 2
I20230423 14:21:54.491828 979747 etcd_meta_service.cc:235] etcd (last) txn use 418 microseconds
I20230423 14:21:54.495616 979766 etcd_meta_service.cc:42] etcd watch use 3779 microseconds, event size = 1
I20230423 14:21:54.525315 979855 /tmp/vineyard-20230313-17412-7mpvto/v6d-0.13.3/modules/graph/vertex_map/arrow_vertex_map_impl.h:73] ArrowVertexMap<int64_t, uint64_t>
	memory: 16.519348 MB
	o2g size: 246310, load factor: 0.441839
	o2g memory: 14.640076 MB
I20230423 14:21:54.577769 979748 etcd_meta_service.cc:159] etcd lock use 4383 microseconds
I20230423 14:21:54.577769 979766 etcd_meta_service.cc:42] etcd watch use 82140 microseconds, event size = 1
I20230423 14:21:54.582770 979747 etcd_meta_service.cc:235] etcd (last) txn use 4939 microseconds
I20230423 14:21:54.582849 979766 etcd_meta_service.cc:42] etcd watch use 5041 microseconds, event size = 29
I20230423 14:21:54.586604 979766 etcd_meta_service.cc:42] etcd watch use 3729 microseconds, event size = 1
I20230423 14:21:54.591121 979855 /Users/siyuan/tmp/graphscope/analytical_engine/core/grape_instance.cc:201] Unloading Graph graph_GXi5xr84
I20230423 14:21:54.592691 979855 /opt/graphscope/include/graphscope/core/object/gs_object.h:66] Object graph_GXi5xr84[LabeledFragmentWrapper] is destructed.
I20230423 14:21:54.654697 979748 etcd_meta_service.cc:159] etcd lock use 16666 microseconds
I20230423 14:21:54.654713 979766 etcd_meta_service.cc:42] etcd watch use 68097 microseconds, event size = 1
I20230423 14:21:54.655112 979766 etcd_meta_service.cc:42] etcd watch use 386 microseconds, event size = 7
I20230423 14:21:54.655118 979747 etcd_meta_service.cc:235] etcd (last) txn use 378 microseconds
I20230423 14:21:54.659689 979766 etcd_meta_service.cc:42] etcd watch use 4555 microseconds, event size = 1
I20230423 14:21:54.663640 979855 /Users/siyuan/tmp/graphscope/analytical_engine/core/grape_instance.cc:201] Unloading Graph graph_N2qmWqBl
I20230423 14:21:54.663698 979855 /opt/graphscope/include/graphscope/core/object/gs_object.h:66] Object graph_N2qmWqBl[LabeledFragmentWrapper] is destructed.

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/ssj/./gs.py:9 in <module>                                                                 │
│                                                                                                  │
│    6 graphscope.set_option(show_log=True)                                                       │
│    7 graphscope.set_option(log_level='Debug')                                                   │
│    8 g = load_ogbn_mag()                                                                         │
│ ❱  9 g = g.project(vertices={"paper"}, edges={"cites"})                                          │
│   10                                                                                             │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/graphscope/framework/graph.py:1126 in project         │
│                                                                                                  │
│   1123 │   ):                                                                                    │
│   1124 │   │   if not self.loaded():                                                             │
│   1125 │   │   │   raise RuntimeError("The graph is not loaded")                                 │
│ ❱ 1126 │   │   return self._session._wrapper(self._graph_node.project(vertices, edges))          │
│   1127 │                                                                                         │
│   1128 │   def _attach_interactive_instance(self, instance):                                     │
│   1129 │   │   """Store the instance when a new interactive instance is started.                 │
│                                                                                                  │
│ /opt/homebrew/lib/python3.11/site-packages/graphscope/framework/graph.py:691 in project          │
│                                                                                                  │
│    688 │   │   """                                                                               │
│    689 │   │   check_argument(self.graph_type == graph_def_pb2.ARROW_PROPERTY)                   │
│    690 │   │   op = dag_utils.project_arrow_property_graph(                                      │
│ ❱  691 │   │   │   self, json.dumps(vertices), json.dumps(edges)                                 │
│    692 │   │   )                                                                                 │
│    693 │   │   # construct dag node                                                              │
│    694 │   │   graph_dag_node = GraphDAGNode(                                                    │
│                                                                                                  │
│ /opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3. │
│ 11/json/__init__.py:231 in dumps                                                                 │
│                                                                                                  │
│   228 │   │   check_circular and allow_nan and                                                   │
│   229 │   │   cls is None and indent is None and separators is None and                          │
│   230 │   │   default is None and not sort_keys and not kw):                                     │
│ ❱ 231 │   │   return _default_encoder.encode(obj)                                                │
│   232 │   if cls is None:                                                                        │
│   233 │   │   cls = JSONEncoder                                                                  │
│   234 │   return cls(                                                                            │
│                                                                                                  │
│ /opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3. │
│ 11/json/encoder.py:200 in encode                                                                 │
│                                                                                                  │
│   197 │   │   # This doesn't pass the iterator directly to ''.join() because the                 │
│   198 │   │   # exceptions aren't as detailed.  The list call should be roughly                  │
│   199 │   │   # equivalent to the PySequence_Fast that ''.join() would do.                       │
│ ❱ 200 │   │   chunks = self.iterencode(o, _one_shot=True)                                        │
│   201 │   │   if not isinstance(chunks, (list, tuple)):                                          │
│   202 │   │   │   chunks = list(chunks)                                                          │
│   203 │   │   return ''.join(chunks)                                                             │
│                                                                                                  │
│ /opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3. │
│ 11/json/encoder.py:258 in iterencode                                                             │
│                                                                                                  │
│   255 │   │   │   │   markers, self.default, _encoder, self.indent, floatstr,                    │
│   256 │   │   │   │   self.key_separator, self.item_separator, self.sort_keys,                   │
│   257 │   │   │   │   self.skipkeys, _one_shot)                                                  │
│ ❱ 258 │   │   return _iterencode(o, 0)                                                           │
│   259                                                                                            │
│   260 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,                      │
│   261 │   │   _key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot,                 │
│                                                                                                  │
│ /opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3. │
│ 11/json/encoder.py:180 in default                                                                │
│                                                                                                  │
│   177 │   │   │   │   return JSONEncoder.default(self, o)                                        │
│   178 │   │                                                                                      │
│   179 │   │   """                                                                                │
│ ❱ 180 │   │   raise TypeError(f'Object of type {o.__class__.__name__} '                          │
│   181 │   │   │   │   │   │   f'is not JSON serializable')                                       │
│   182 │                                                                                          │
│   183 │   def encode(self, o):                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Object of type set is not JSON serializable

[siyuan0322]

It's caused by bad grammer. This line g.project(vertices={"paper"}, edges={"cites"}) should be
simple_g = sub_graph.project(vertices={"paper": []}, edges={"cites": []})
As the {} is a dict should consist of a key and a value. And the key means the label, the values means a list of properties to project, where [] means no property.

[ds-ssj]

Oops. 👍

[sighingnow]

The MPI issue has been fixed by #2650

[sighingnow]

The .project() error message is improved in #2667.

[ds-ssj]

Use ./gs install-deps dev --cn:

Building and installing cppkafka-0.4.0.
/tmp/gs-local-deps ~/Documents/project/graphscope
curl: (23) Failure writing output to destination

Build analytical_engine:

clang: error: unsupported option '-fopenmp'

[sighingnow]

@siyuan0322

[siyuan0322]

curl: (23) Failure writing output to destination

Please confirm why curl reported that error? It's the network throttle or permission to write to /tmp? If it's the network, retry may fix that. If the permission, you could assign another dependencies location by ./gs install-deps dev -l ~/gs-deps

./gs install-deps -h to see all available options.

clang: error: unsupported option '-fopenmp'

You're using mac? Please use toolchains provided by homebrew.

export CXX=/opt/homebrew/opt/llvm/bin/clang++

[ds-ssj]

When I execute make interactive , an error appears:

Is it also related to the Python version?

[siyuan0322]

Weird, why it choose to build grpc from source. Is your pip version upgraded to latest?

[ds-ssj]

pip version is: pip 23.1.2 from /opt/homebrew/lib/python3.11/site-packages/pip (python 3.11)

[siyuan0322]

Could you find the full output and send it to me in dingding

[ds-ssj]

In Task3, when I execute this command in the gae.yml

make analytical-java BUILD_TEST=ON NUMPROC=12

It reports:

I try to update the version of maven-shade-plugin to 3.4.1, but it still reports: