Dogfooding by LiSu #2700

LiSu · 2023-05-14T14:50:21Z

LiSu
May 14, 2023
Collaborator

System Info

$ uname -a
Linux j63c09243.sqa.eu95 4.9.151-015.ali3000.alios7.x86_64 #1 SMP Tue Mar 12 19:10:26 CST 2019 x86_64 x86_64 x86_64 GNU/Linux

$ python3 --version
Python 3.8.16

Task 1: 在本地试用 GraphScope

安装 GraphScope
载入 ogbn_mag 内置图
Project ogbn_mag到一个简单图，选择点标签为 paper, 边标签为 cites, 属性都为空。
在简单图上运行一个 Kcore 算法, 并打印结果为 pandas.DataFrame
在 ogbn_mag图上启动交互引擎（Interactive）
通过 gremlin 计算图有多少点，边
对于 ogbn_mag图，使用 learning 定义并执行一个 GCN 训练过程，可参考 CI 或文档中定义的过程
退出 GraphScope。

Task 2: 在本地开发调试 GraphScope

Task 3: 运行本地测试

找到 CI 定义的位置和文件
找到单元测试的位置，参考在 CI 中是如何运行这些单元测试的步骤，并在本地运行CI上的本地测试（如果没有，可开 Issue 加入运行）
GAE 的单元测试
GIE 的单元测试
GLE 的单元测试
运行一组 Python 中的单元测试 test_min.py 。（Python 中的单元测试更像是集成测试，因为往往由多个引擎参与）
在 python/graphscope/tests/minitest下新增一个test 文件，在里面添加一个涉及多个引擎的测试，其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。

Task 4: 在 Kubernetes 上使用 GraphScope （Python SDK）

通过客户端在 K8S 环境下安装 Graphscope，建立一个 Session，通过 kubectl 查看拉起来的 Pod 和 Service （可通过 minikube/kind 在单机搭建 K8S 环境）
通过载图 API 载一个图 ogbn_mag
在图上运行一个 LPA 算法，并打印结果为 pandas.DataFrame
在图上启动交互引擎（Interactive）
通过 gremlin 计算图有多少点，边
使用 learning 定义并执行一个 GCN 训练过程
关闭 Session

Task 5: 在 Kubernetes 上使用 GraphScope （Helm）

通过 Helm 安装 GraphScope，并载图，运行算法，Gremlin 和 Learning

Task 6: 独立部署

独立部署 GAE，并载图 modern_graph，运行 LPA 算法
独立部署 GIE，发送 gremlin 查询点边数量

Task 7: 打包镜像与了解 Kubernetes 上运行的基本概念

构建 GAE 的镜像
构建 GIE 的镜像（包括 Frontend 和 Executor）
构建 GLE 的镜像
构建 Coordinator 的镜像
为 GAE 添加一条日志 ( analytical_engine/core/server/analytical_server.cc) 并重新构建镜像，在不使用 docker push的情况下，使用新的镜像启动并在日志中可以找到新添加的日志（使用 registry 和 tag 来选择使用哪个镜像）
为 GIE 的 Frontend (interactive_engine/frontend/src/main/java/com/alibaba/graphscope/frontend/Frontend.java) 和 executor (interactive_engine/executor/assembly/v6d/src/bin/gaia_executor.rs) 各添加一条日志并重新构建 Frontend 和 Executor 镜像，在不使用 docker push的情况下，使用新的镜像启动并可以找到新添加的日志
为 Coordinator 添加一个异常 (coordinator/gscoordinator/coordinator.py)，并重新构建 Coordinator 镜像。启动并找到异常的报错。（日志在控制台中或者在 Coordinator Pod 的日志中）
清除所有 GraphScope 的资源，包括镜像，容器。

Task 8: 运行 K8S 上的测试

找到在 Kubernetes 上运行的测试，找到在 CI 中是如何运行这些单元测试的
参考 CI 的步骤，运行 test_demo_scripts.py 中的 test_demo_distribute 测试
尝试添加一个在 Kubernetes 上运行的测试，并运行。其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。

Task 9 : 打包为 Wheel 包

从源码构建 Python graphscope-client Wheels
从源码构建 Python graphscope wheels
安装构建出的 Wheel

LiSu · 2023-05-14T15:55:40Z

LiSu
May 14, 2023
Collaborator Author

CentOS7上Python版本过低，需手动安装Python3
安装GraphScope时遇到“ModuleNotFoundError: No module named '_ctypes'”, 重装Python后解决，具体报错信息：

Collecting hdfs3
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7a/6b/1a5c0324767ba60043d0e58e1eeaa28dfdd12ad5fef7f0cd3518d4466e12/hdfs3-0.3.1.tar.gz (44 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.5/44.5 kB 1.1 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [62 lines of output]
running dist_info
creating /tmp/pip-modern-metadata-dq2alklf/hdfs3.egg-info
writing /tmp/pip-modern-metadata-dq2alklf/hdfs3.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-modern-metadata-dq2alklf/hdfs3.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-modern-metadata-dq2alklf/hdfs3.egg-info/top_level.txt
writing manifest file '/tmp/pip-modern-metadata-dq2alklf/hdfs3.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-modern-metadata-dq2alklf/hdfs3.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE.txt'
writing manifest file '/tmp/pip-modern-metadata-dq2alklf/hdfs3.egg-info/SOURCES.txt'
creating '/tmp/pip-modern-metadata-dq2alklf/hdfs3-0.3.1.dist-info'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 380, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
super(_BuildMetaLegacyBackend,
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in run_setup
exec(code, locals())
File "", line 7, in
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/init.py", line 107, in setup
return distutils.core.setup(**attrs)
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1244, in run_command
super().run_command(command)
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/command/dist_info.py", line 104, in run
bdist_wheel = self.get_finalized_command('bdist_wheel')
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 304, in get_finalized_command
cmd_obj = self.distribution.get_command_obj(command, create)
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 860, in get_command_obj
klass = self.get_command_class(command)
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 989, in get_command_class
self.cmdclass[command] = cmdclass = ep.load()
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/init.py", line 208, in load
module = import_module(match.group('module'))
File "/usr/local/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 843, in exec_module
File "", line 219, in _call_with_frames_removed
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 28, in
from .macosx_libfile import calculate_macosx_platform_tag
File "/tmp/pip-build-env-ibttqz8a/overlay/lib/python3.8/site-packages/wheel/macosx_libfile.py", line 43, in
import ctypes
File "/usr/local/lib/python3.8/ctypes/init.py", line 7, in
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
─> See above for output.

0 replies

LiSu · 2023-05-14T16:14:18Z

LiSu
May 14, 2023
Collaborator Author

拉起interactive引擎失败，安装Java后成功。
pip3 install tensorflow==2.8 后GLE训练成功

0 replies

LiSu · 2023-06-05T09:12:08Z

LiSu
Jun 5, 2023
Collaborator Author

镜像graphscope/graphscope-dev:latest的vineyard版本为0.14.5，执行./gs make analytical报错：

CMake Error at CMakeLists.txt:264 (find_package):
Could not find a configuration file for package "vineyard" that is
compatible with requested version "0.14.8".
The following configuration files were considered but not accepted:
/opt/vineyard/lib/cmake/vineyard/vineyard-config.cmake, version: 0.14.5
/opt/graphscope/lib/cmake/vineyard/vineyard-config.cmake, version: 0.14.5

重新安装vineyard 0.14.8无效，本地build安装后fix

2 replies

LiSu Jun 5, 2023
Collaborator Author

@siyuan0322

LiSu Jun 6, 2023
Collaborator Author

graphscope/graphscope-dev的vineyard版本已更新为0.14.8

LiSu · 2023-06-05T11:49:17Z

LiSu
Jun 5, 2023
Collaborator Author

在vineyard 0.15.0下make interactive失败，提示undefined reference to vineyard classes, 回退到0.14.5成功

@sighingnow

1 reply

LiSu Jun 6, 2023
Collaborator Author

重新pull graphscope/graphscope-dev:latest之后（vineyard 0.14.8）成功

LiSu · 2023-06-06T03:08:57Z

LiSu
Jun 6, 2023
Collaborator Author

文档中没找到使用 setup.py 安装 coordinator 和 client的具体说明，e.g.，client的setup.py在graphscope/python目录下

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dogfooding by LiSu #2700

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Dogfooding by LiSu #2700

LiSu May 14, 2023 Collaborator

System Info

Task 1: 在本地试用 GraphScope

Task 2: 在本地开发调试 GraphScope

Task 3: 运行本地测试

Task 4: 在 Kubernetes 上使用 GraphScope （Python SDK）

Task 5: 在 Kubernetes 上使用 GraphScope （Helm）

Task 6: 独立部署

Task 7: 打包镜像与了解 Kubernetes 上运行的基本概念

Task 8: 运行 K8S 上的测试

Task 9 : 打包为 Wheel 包

Replies: 5 comments · 3 replies

LiSu May 14, 2023 Collaborator Author

LiSu May 14, 2023 Collaborator Author

LiSu Jun 5, 2023 Collaborator Author

LiSu Jun 5, 2023 Collaborator Author

LiSu Jun 6, 2023 Collaborator Author

LiSu Jun 5, 2023 Collaborator Author

LiSu Jun 6, 2023 Collaborator Author

LiSu Jun 6, 2023 Collaborator Author

LiSu
May 14, 2023
Collaborator

Replies: 5 comments 3 replies

LiSu
May 14, 2023
Collaborator Author

LiSu
May 14, 2023
Collaborator Author

LiSu
Jun 5, 2023
Collaborator Author

LiSu Jun 5, 2023
Collaborator Author

LiSu Jun 6, 2023
Collaborator Author

LiSu
Jun 5, 2023
Collaborator Author

LiSu Jun 6, 2023
Collaborator Author

LiSu
Jun 6, 2023
Collaborator Author