Skip to content

Commit

Permalink
Merge pull request #18 from yunify/daemon_process
Browse files Browse the repository at this point in the history
Add Daemon process
  • Loading branch information
martinyunify authored Aug 25, 2017
2 parents 5095464 + c6422d0 commit 999f318
Show file tree
Hide file tree
Showing 3,090 changed files with 1,703,104 additions and 54,116 deletions.
The diff you're trying to view is too large. We only load the first 3000 changed files.
5 changes: 3 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,8 @@ print-% :
# perform go build on project
go-build :
go build -o bin/hostnic $(GO_BUILD_FLAGS) $(GIT_REPOSITORY)/cmd/hostnic/
go build -o bin/nicclearner $(GO_BUILD_FLAGS) $(GIT_REPOSITORY)/cmd/niccleaner/
go build -o bin/daemon $(GO_BUILD_FLAGS) $(GIT_REPOSITORY)/cmd/daemon/


.PHONY : default all go-build

.PHONY : default all go-build
137 changes: 93 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,57 +4,106 @@ English|[中文](README_zh.md)

**hostnic-cni** is a [Container Network Interface](https://github.com/containernetworking/cni) plugin. This plugin will create a new nic by IaaS api and attach to host, then move the nic to container network namespace. Support IaaS :[QingCloud](http://qingcloud.com).



### Usage

1. Download CNI package from [CNI release page](https://github.com/containernetworking/cni/releases) and extract to /opt/cni/bin/.
2. Download hostnic from [release page](https://github.com/yunify/hostnic-cni/releases) , and put to /opt/cni/bin/
3. Add cni config

```bash

cat >/etc/cni/net.d/10-hostnic.conf <<EOF
{
"cniVersion": "0.3.0",
"name": "hostnic",
"type": "hostnic",
"provider": "qingcloud",
"args": {
"providerConfigFile":"/etc/qingcloud/client.yaml",
"vxNets":["vxnet-xxxxx","vxnet-xxxx"]
},
"ipam":{
"routes":[{"dst":"kubernetes service cidr","gw":"hostip or 0.0.0.0"}]
},
"isGateway": true
}
EOF

cat >/etc/cni/net.d/99-loopback.conf <<EOF
{
"cniVersion": "0.2.0",
"type": "loopback"
}
EOF
```
3. Add cloud provider config

```bash
cat >/etc/qingcloud/client.yaml <<EOF
qy_access_key_id: "Your access key id"
qy_secret_access_key: "Your secret access key"
# your instance zone
zone: "pek3a"
EOF
```
1. Download hostnic from [release page](https://github.com/yunify/hostnic-cni/releases) , and put hostnic to /opt/cni/bin/
1. Add cloud provider config

```bash
cat >/etc/qingcloud/client.yaml <<EOF
qy_access_key_id: "Your access key id"
qy_secret_access_key: "Your secret access key"
# your instance zone
zone: "pek3a"
EOF
```
1. Launch daemon process
Daemon process is used as a nic manager which allocates and destroys nics in the background. It serves requests from hostcni and maintain nic info and nic cache pool.
it accepts a few params. As listed below.
```bash
[root@i-zwa7jztl bin]# ./daemon start -h
hostnic-cni is a Container Network Interface plugin.
This plugin will create a new nic by IaaS api and attach to host,
then move the nic to container network namespace
Usage:
daemon start [flags]
Flags:
--CleanUpCacheOnExit Delete cached nic on exit
--PoolSize int The size of nic pool (default 3)
--QyAccessFilePath string Path of QingCloud Access file (default "/etc/qingcloud/client.yaml")
-h, --help help for start
--vxnets stringSlice ids of vxnet
Global Flags:
--bindAddr string port of daemon process(e.g. socket port 127.0.0.1:31080 [fe80::1%lo0]:80 ) (default ":31080")
--config string config file (default is $HOME/.daemon.yaml)
--loglevel string daemon process log level(debug,info,warn,error) (default "info")
--manageAddr string addr of daemon monitor(e.g. socket port 127.0.0.1:31080 [fe80::1%lo0]:80 ) (default ":31081")
```
e.g.
```bash
./bin/daemon start --bindAddr :31080 --vxnets vxnet-xxxxxxx,vxnet-xxxxxxx --PoolSize 3 --loglevel debug
INFO[0000] Collect existing nic as gateway cadidate
DEBU[0000] Found nic 52:54:03:41:e9:16 on host
DEBU[0000] Found nic 52:54:20:82:68:5c on host
DEBU[0000] Found nic 52:54:0b:48:04:52 on host
INFO[0000] Found following nic as gateway
INFO[0000] vxnet: vxnet-oca1g0z gateway: 192.168.4.253
INFO[0000] vxnet: vxnet-oilq879 gateway: 192.168.3.251
INFO[0000] vxnet: vxnet-2n6g6gx gateway: 192.168.0.3
DEBU[0002] start to wait until channel is not full.
DEBU[0002] put 52:54:27:6b:17:65 into channel
DEBU[0007] start to wait until channel is not full.
DEBU[0007] put 52:54:57:83:d0:ab into channel
DEBU[0011] start to wait until channel is not full.
DEBU[0011] put 52:54:d6:86:46:d6 into channel
DEBU[0015] start to wait until channel is not full.
```
The daemon process would fill nic pool with pre-allocated nics and wait until new request comes
1. Add cni config
```bash
cat >/etc/cni/net.d/10-hostnic.conf <<EOF
{
"cniVersion": "0.3.1",
"name": "hostnic",
"type": "hostnic",
"bindaddr":"localhost:31080",
"ipam":{
"routes":[{"dst":"kubernetes service cidr","gw":"hostip or 0.0.0.0"}]
},
"isGateway": true
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<EOF
{
"cniVersion": "0.2.0",
"type": "loopback"
}
EOF
```
### CNI config Description
* **provider** IaaS provider, current only support qingcloud
* **providerConfigFile** IaaS provider api config
* **vxNets** nic vxnet, support multi, all vxnet should in same vpc.
* **ipam** add custom routing rules for nic, (optional)
* **bindaddr** server addr where daemon listens to
### Special notes for Kubernetes users
Hostnic may not work as expected when it is used with Kubernetes framework due to the constrains in the design of kubernetes. However, we've provided a work around to help users setup kubernetes cluster.
When a new service is defined in kubernetes cluster, it will get a cluster ip. And kube-proxy will maintain a port mapping tables on host machine to redirect service request to corresponding pod. And all of the network payload will be routed to host machine before it is sent to router and the service request will be handled correctly. In this way, kubernetes helps user achieve high availability of service. However, when the pod is attached to network directly(this is what hostnic did), Service ip is not recognied by router and service requests will not be processed.
Expand Down
133 changes: 92 additions & 41 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,50 +7,101 @@
### 使用说明

1.[CNI release 页面](https://github.com/containernetworking/cni/releases) 下载 CNI 包,解压到 /opt/cni/bin 下。
2.[release 页面](https://github.com/yunify/hostnic-cni/releases) 下载 hostnic 放置到 /opt/cni/bin/ 路径下。
3. 增加 cni 的配置

```bash

cat >/etc/cni/net.d/10-hostnic.conf <<EOF
{
"cniVersion": "0.3.0",
"name": "hostnic",
"type": "hostnic",
"provider": "qingcloud",
"args": {
"providerConfigFile":"/etc/qingcloud/client.yaml",
"vxNets":["vxnet-xxxxx","vxnet-xxxx"]
},
"ipam":{
1.[release 页面](https://github.com/yunify/hostnic-cni/releases) 下载 hostnic 放置到 /opt/cni/bin/ 路径下。
1. 增加 IaaS 的 sdk 配置文件

```bash
cat >/etc/qingcloud/client.yaml <<EOF
qy_access_key_id: "Your access key id"
qy_secret_access_key: "Your secret access key"
# your instance zone
zone: "pek3a"
EOF
```
1. 启动后台进程
后台进程主要负责后台异步的进行网卡的申请销毁。给hostnic程序提供服务。它监听本地端口,维护网卡信息,并管理缓存网卡池
启动后台进程需要如下参数
```bash
[root@i-zwa7jztl bin]# ./daemon start -h
hostnic-cni is a Container Network Interface plugin.
This plugin will create a new nic by IaaS api and attach to host,
then move the nic to container network namespace
Usage:
daemon start [flags]
Flags:
--CleanUpCacheOnExit Delete cached nic on exit
--PoolSize int The size of nic pool (default 3)
--QyAccessFilePath string Path of QingCloud Access file (default "/etc/qingcloud/client.yaml")
-h, --help help for start
--vxnets stringSlice ids of vxnet
Global Flags:
--bindAddr string port of daemon process(e.g. socket port 127.0.0.1:31080 [fe80::1%lo0]:80 ) (default ":31080")
--config string config file (default is $HOME/.daemon.yaml)
--loglevel string daemon process log level(debug,info,warn,error) (default "info")
--manageAddr string addr of daemon monitor(e.g. socket port 127.0.0.1:31080 [fe80::1%lo0]:80 ) (default ":31081")
```
例如
```bash
./bin/daemon start --bindAddr :31080 --vxnets vxnet-xxxxxxx,vxnet-xxxxxxx --PoolSize 3 --loglevel debug
INFO[0000] Collect existing nic as gateway cadidate
DEBU[0000] Found nic 52:54:03:41:e9:16 on host
DEBU[0000] Found nic 52:54:20:82:68:5c on host
DEBU[0000] Found nic 52:54:0b:48:04:52 on host
INFO[0000] Found following nic as gateway
INFO[0000] vxnet: vxnet-oca1g0z gateway: 192.168.4.253
INFO[0000] vxnet: vxnet-oilq879 gateway: 192.168.3.251
INFO[0000] vxnet: vxnet-2n6g6gx gateway: 192.168.0.3
DEBU[0002] start to wait until channel is not full.
DEBU[0002] put 52:54:27:6b:17:65 into channel
DEBU[0007] start to wait until channel is not full.
DEBU[0007] put 52:54:57:83:d0:ab into channel
DEBU[0011] start to wait until channel is not full.
DEBU[0011] put 52:54:d6:86:46:d6 into channel
DEBU[0015] start to wait until channel is not full.
```
后台进程会将缓冲池充满,并等待请求到来。
1. 增加 cni 的配置
```bash
cat >/etc/cni/net.d/10-hostnic.conf <<EOF
{
"cniVersion": "0.3.1",
"name": "hostnic",
"type": "hostnic",
"bindaddr":"localhost:31080",
"ipam":{
"routes":[{"dst":"kubernetes service cidr","gw":"hostip or 0.0.0.0"}]
 },
"isGateway": true
}
EOF

cat >/etc/cni/net.d/99-loopback.conf <<EOF
{
"cniVersion": "0.2.0",
"type": "loopback"
}
EOF
```
3. 增加 IaaS 的 sdk 配置文件

```bash
cat >/etc/qingcloud/client.yaml <<EOF
qy_access_key_id: "Your access key id"
qy_secret_access_key: "Your secret access key"
# your instance zone
zone: "pek3a"
EOF
```
},
"isGateway": true
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<EOF
{
"cniVersion": "0.2.0",
"type": "loopback"
}
EOF
```
### 配置说明
* **provider** IaaS 的提供方,当前只支持 qingcloud,未来会支持更多。
* **providerConfigFile** IaaS 提供方的配置文件
* **vxNets** nic 所在的私有网络,数组格式,支持多个,多个私有网络必须在同一个 vpc 下。
* **ipam** 给nic设置路由条目。(可选)
### kubernetes用户的说明
kubernetes管理集群时会给每一个服务分配一个集群ip地址。kube-proxy会负责服务负载均衡,由于默认的方案所有pod的网络请求通过主机的网卡转发后才会进入路由器。但给pod单独分配网卡后,流量不再经过kube-proxy,所有调用服务的请求会失败。这里我们提供设置路由的功能来解决这个问题。pod分配网卡后,用户可以指定将集群ip网段的请求转发给虚拟机,交由kube-proxy去处理。由于网关必须满足一定条件,用户可以指定网关为0.0.0.0,插件会自动寻找可用网管,如果都不满足条件,会自动分配一个。
61 changes: 61 additions & 0 deletions cmd/daemon/cmd/clean.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
//
// =========================================================================
// Copyright (C) 2017 by Yunify, Inc...
// -------------------------------------------------------------------------
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this work except in compliance with the License.
// You may obtain a copy of the License in the LICENSE file, or at:
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// =========================================================================
//

package cmd

import (
"github.com/spf13/cobra"
"github.com/spf13/viper"
"io/ioutil"
"net/http"
log "github.com/sirupsen/logrus"
)

// cleanCmd represents the clean command
var cleanCmd = &cobra.Command{
Use: "clean",
Short: "Clean up nic cache",
Long: `hostnic-cni is a Container Network Interface plugin.
This plugin will create a new nic by IaaS api and attach to host,
then move the nic to container network namespace`,
Run: func(cmd *cobra.Command, args []string) {
resp, err := http.Get("http://"+viper.GetString("manageAddr")+"/clearcache")
if err != nil {
log.Error(err)
return
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
log.Info(string(body))
},
}

func init() {
RootCmd.AddCommand(cleanCmd)

// Here you will define your flags and configuration settings.

// Cobra supports Persistent Flags which will work for this command
// and all subcommands, e.g.:
// cleanCmd.PersistentFlags().String("foo", "", "A help for foo")

// Cobra supports local flags which will only run when this command
// is called directly, e.g.:
// cleanCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}
Loading

0 comments on commit 999f318

Please sign in to comment.