Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: update celeborn.md #314

Merged
merged 1 commit into from
Feb 26, 2024
Merged

doc: update celeborn.md #314

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions byzer-lang/zh-cn/velox/celeborn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
这个是我们新的架构图，byzer-notebook对接多个byzer-engine,然后byzer-engine对接RSS服务，同时RSS配置有Velox Engine。

下图是我的系统版本, JDK使用1.8:

![img.png](images/system.png)

废话太多了，我们直接开始。首先，我们先看下Gluten的仓库，看下Gluten怎么结合Celeborn,这里是个天坑，目前Gluten只能支持指定版本的Celeborn，不然可能因为Api异常报出一些Warning。
Expand Down Expand Up @@ -153,15 +154,12 @@ mvn clean package -Passembly -Pdisable-java8-doclint -Plocal -Pshade -DskipTests

### 设置Gluten和Celeborn

```
```

### 启动byzer-lang

- 把gluten-velox-bundle-spark3.3_2.12-ubuntu_22.04-1.1.0-SNAPSHOT.jar从gluten/package/目录下拷贝到byzer-lang/spark/下

- 把celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar从celeborn/spark/目录下拷贝到byzer-lang/spark/下

### 启动byzer-lang

PS: 请自行更改byzer的目录
```shell
nohup java -Xms10g -Xmx10g -cp /home/jayce/bigdata/byzer-lang-all-in-one-linux-amd64-3.3.0-2.4.0-SNAPSHOT/main/byzer-lang-3.3.0-2.12-2.4.0-SNAPSHOT.jar:/home/jayce/bigdata/byzer-lang-all-in-one-linux-amd64-3.3.0-2.4.0-SNAPSHOT/spark/*:/home/jayce/bigdata/byzer-lang-all-in-one-linux-amd64-3.3.0-2.4.0-SNAPSHOT/libs/*:/home/jayce/bigdata/byzer-lang-all-in-one-linux-amd64-3.3.0-2.4.0-SNAPSHOT/plugin/* \
Expand Down
Loading