Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Verify OVS bridge is created by the plugin at docker run for the first container #15

Open
nerdalert opened this issue Jul 19, 2015 · 2 comments
Open

Verify OVS bridge is created by the plugin at docker run for the first container #15

nerdalert opened this issue Jul 19, 2015 · 2 comments

Comments

@nerdalert
Copy link
Contributor

Hey @Runseb can you double check your OVS bridge isn't created by default. I think I saw a readme update that has it being created manually. There is a br create method in ovs_bridge.go func (driver *driver) setupBridge() that should create it. If you could double check that was or wasn't working that would be awesome. Below are before and afters and output.

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:0c:29:58:1c:01 brd ff:ff:ff:ff:ff:ff
    inet 172.16.86.150/24 brd 172.16.86.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe58:1c01/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:0c:29:58:1c:0b brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.250/24 brd 192.168.1.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe58:1c0b/64 scope link
       valid_lft forever preferred_lft forever
16: ovs-system@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
    link/ether 0e:88:46:2b:39:e8 brd ff:ff:ff:ff:ff:ff

OVS config

$ ovs-vsctl show

45045b76-ec16-4828-bafb-07b138f6a565
    Manager "ptcp:6640"
    ovs_version: "2.3.0"

Start the plugin with defaults, ovsbr-docker0 with an ip of 172.18.40.1/24 (containers land in that network)

./docker-ovs-plugin  -d

or from source

git clone https://github.com/gopher-net/docker-ovs-plugin.git
cd docker-ovs-plugin/plugin/
# get dependancies
go get ./...
go run main.go -d

Start Docker (note, docker0 still gets created. I am not sure how to disable that yet. @dave-tucker might know. Since we are passing what we want as the default bridge, it would make sense to not create another default bridge that is unused or worse overlapping. Whatever the name passed to the default-network should be what is setup in OVS.

docker -d -D --default-network=ovs:ovsbr-docker0

Start a container

docker run -i -t --rm busybox

After a container starts, the network create is called and the default bridge for the plugin (ovsbr-docker0) is created and assigned an L3 ip addr. (Note: we need to add a masquerade option there to enable a NAT.)

$ ifconfig
docker0   Link encap:Ethernet  HWaddr b2:d1:59:e3:94:d1
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::b0d1:59ff:fee3:94d1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:58:1c:01
          inet addr:172.16.86.150  Bcast:172.16.86.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe58:1c01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10526 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5933 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1099310 (1.0 MiB)  TX bytes:939698 (917.6 KiB)
          Interrupt:19 Base address:0x2000

eth1      Link encap:Ethernet  HWaddr 00:0c:29:58:1c:0b
          inet addr:192.168.1.250  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe58:1c0b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1286098 errors:66 dropped:186 overruns:0 frame:0
          TX packets:8144 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:765846208 (730.3 MiB)  TX bytes:560943 (547.7 KiB)
          Interrupt:19 Base address:0x2080

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1751 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1751 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:913292 (891.8 KiB)  TX bytes:913292 (891.8 KiB)

ovs-veth0-c3caa Link encap:Ethernet  HWaddr 9e:8e:dc:75:42:7d
          inet6 addr: fe80::9c8e:dcff:fe75:427d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:508 (508.0 B)  TX bytes:836 (836.0 B)

ovsbr-docker0 Link encap:Ethernet  HWaddr d6:6a:3c:03:92:44
          inet addr:172.18.40.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::d46a:3cff:fe03:9244/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:508 (508.0 B)  TX bytes:508 (508.0 B)

# OVS

# ovs-vsctl show
45045b76-ec16-4828-bafb-07b138f6a565
    Manager "ptcp:6640"
        is_connected: true
    Bridge "ovsbr-docker0"
        Port "ovsbr-docker0"
            Interface "ovsbr-docker0"
                type: internal
        Port "ovs-veth0-c3caa"
            Interface "ovs-veth0-c3caa"
    ovs_version: "2.3.0"
@sebgoa
Copy link
Contributor

sebgoa commented Jul 19, 2015

@nerdalert yeah I will check but I remember having to create it by hand. And indeed docker0 bridge gets created anyway. I tried passing docker run -d -b=ovsbr-docker0 but that did not work…

I will test again and send the error message.

@nerdalert
Copy link
Contributor Author

Thanks @Runseb , I think I recreated the issue on vivid with 4.1.1. Its seems like some delay in the netlink bridge getting created after the OVS bridge is created. I am going to stick a retry for the time being on the netlink.GetLinkByName to give OVS and netlink enough time to synchronize if that makes sense to you all. Still curious what changed. Pasting the logs in case anything jumps out for you.

uname -a
Linux ub-1504 4.1.1-040101-generic #201507030635 SMP Fri Jul 3 10:38:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Error:

Jul 19 17:25:27 ub-1504 kernel: [ 3526.631825] openvswitch: netlink: Key type 62 is out of range max 22
Jul 19 17:25:27 ub-1504 systemd-udevd[4488]: conflicting device node '/dev/mapper/docker-8:1-536425-9a3611b06a46825c41fd8ef73a21d84d9dbad827416bb3f473c973668fd1977a' found, link to '/dev/dm-3' will not be created
Jul 19 17:25:27 ub-1504 systemd-udevd[4488]: Could not generate persistent MAC address for ovsbr: No such file or directory
Jul 19 17:25:27 ub-1504 kernel: [ 3526.634692] device ovsbr entered promiscuous mode
Jul 19 17:25:27 ub-1504 systemd[1]: message repeated 3 times: [ Device dev-disk-by\x2duuid-34986662\x2da6cf\x2d4cba\x2dbf15\x2d700ec2bf7b65.device appeared twice with different sysfs paths /sys/devices/virtual/block/loop0 and /sys/devices/virtual/block/dm-3]
Jul 19 17:25:27 ub-1504 systemd[1]: Started ifup for ovsbr.
Jul 19 17:25:27 ub-1504 systemd[1]: Starting ifup for ovsbr...
Jul 19 17:25:27 ub-1504 sh[4542]: Unknown interface ovsbr

Adding a retry:

    iface, err := netlink.LinkByName(name)
    if err != nil {
        log.Debugf("error retrieving the new OVS bridge link [ %s ], retrying in 1 second..", bridgeName)
        time.Sleep(1 * time.Second)
        iface, err = netlink.LinkByName(name)
        if err != nil {
            log.Errorf("Error retrieving the new OVS bridge from netlink: %s", err)
            return err
        }
    }

nerdalert added a commit that referenced this issue Jul 20, 2015
@nerdalert
Merge pull request #17 from nerdalert/brent/bridge-create
fad1989 
issue #15 add a retry if netlink and ovs are not in sync after br add
nerdalert added a commit that referenced this issue Jul 20, 2015
@nerdalert
Merge pull request #18 from nerdalert/brent/binary-update
82f4932 
updated the go binary for testing in issue #15
@nerdalert nerdalert mentioned this issue Jul 31, 2015
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nerdalert @sebgoa