Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

patched OVS with ports other than vale created ports #266

Closed
Linerd opened this issue Jan 2, 2017 · 18 comments
Closed

patched OVS with ports other than vale created ports #266

Linerd opened this issue Jan 2, 2017 · 18 comments

Comments

@Linerd
Copy link

Linerd commented Jan 2, 2017

My patched OVS works great with ports created with vale-ctl, but when I tried to attach other ports to the OVS bridge(the physical interfaces or the interfaces within VM), it failed. I got the "failed to add port XX, invalid argument" error.

I am trying to figure out if this is my problem or the inner limitation of the current patch. When I switch back to the original OVS I can attach whatever interface I want.

PS. just try to make sure I understand correctly, it is impossible to run OVS on two vale switches on the same host right? Coz every bridge I create will automatically be attached to vale0.

@vmaffione
Copy link
Collaborator

Just to understand, are you using the patch at "utils/switch-modules/ovs-datapath/0001-datapath-Add-support-for-netmap-VALE.patch" ?

@Linerd
Copy link
Author

Linerd commented Jan 2, 2017

@vmaffione yes, actually it is slightly modified by @micchie to support OVS 2.6.0, but I assume it shouldn't be that different

@vmaffione
Copy link
Collaborator

Hi,
Attaching a linux interface to OVS works only if the interface has native netmap support.
This means that VALE ports (e.g. interfaces created with vale-ctl -n) and ixgbe,e1000,veth,etc. (with patched drivers) can be attached to an OVS bridge with "ovs-vsctl add-port".
This limitation is due to a confilct: both OVS and the emulated (generic) adapter use the same functionality to intercept packets, and so they cannot work together.

The patch for OVS 2.6.1 has been published here https://github.com/luigirizzo/netmap/blob/master/utils/switch-modules/ovs-2.6.1/0001-datapath-netmap-VALE-support.patch .

For the second question: yes, it is not possible to use two VALE switches with OVS on the same host, and it is a limitation of the current patch. This is actually more a problem of OVS, since afaik in the openvswitch kernel module there is no abstraction for the "bridge" object. This means that all the OVS bridges and ports you create share the same "datapath" object in kernelspace. As a consequence, there is no an easy way to tell (in kernel space) what OVS bridge a port is part of.
Actually, I was trying to circumvent this problem (see https://github.com/vmaffione/ovs/tree/netmap-2.6.1-multi), but it may be not convenient.

@Linerd
Copy link
Author

Linerd commented Jan 8, 2017

@vmaffione when OVS is built with the new patch, the generated openvswitch.ko file in ovs/datapath/linux/ does not depend on netmap. Is it this expected? The old patch files used to build openvswitch.ko that depends on netmap

@Linerd
Copy link
Author

Linerd commented Jan 8, 2017

@vmaffione the build process says "netmap_attach, netmap_detach, netmap_bdg_ctl undefined", thus the built ovs module does not depend on netmap. When inserted, invalid parameter error is reported.

@vmaffione
Copy link
Collaborator

The patch fo 2.6.1 works fine to me. Are you using 2.4.0 or 2.6.1?
Of course openvswitch.ko must depend on netmap.ko

@Linerd
Copy link
Author

Linerd commented Jan 9, 2017 via email

@vmaffione
Copy link
Collaborator

I'm using linux 4.8.13 on archlinux. But this should not be important. Have you used the --with-netmap switch to point at the netmap sources? Are you sure you have applied the patch to the 2.6.1 tar.gz from the ovs website, or the v2.6.1 tag on the ovs github repo?

@Linerd
Copy link
Author

Linerd commented Jan 9, 2017 via email

@vmaffione
Copy link
Collaborator

Have you installed netmap first and then installed openvswitch modules with

´´´´

make modules_install

´´´´

Yours may be a depmod problem. You can also try to run depmod -a manually.

@Linerd
Copy link
Author

Linerd commented Jan 9, 2017 via email

@vmaffione
Copy link
Collaborator

No. Attach and detach functions are defined in netmap_kern.h, which is not exported for now. So if you get an undefined reference error it means that either your --with-netmap path is wrong or that you are compiling netmap and ovs against two different kernel build infrastructures (you typically wantto.use se /lib/modules/´uname -r´/build for noth)

@vmaffione vmaffione reopened this Jan 9, 2017
@Linerd
Copy link
Author

Linerd commented Jan 10, 2017

@vmaffione I tried again, this time I make sure the same kernel build is used, but still can't generate the correct ovs module. The patch @micchie gave me months ago works, but his patch needs some fix for the latest netmap build. (I will email you his patch just for reference, maybe you could find his patch useful).

@vmaffione
Copy link
Collaborator

vmaffione commented Jan 10, 2017 via email

@Linerd
Copy link
Author

Linerd commented Jan 10, 2017

@vmaffione now it works!

I see that this patched version of OVS still can't work with mininet(it is stuck at starting switches stage), but I believe mininet is using veth pairs to connect hosts. Is there anything I can do to make mininet work? I have already loaded the netmap version of veth module.

@vmaffione
Copy link
Collaborator

For sure you need to fetch the latest code to get a bug-fix for veth that just went in.
The support for veth has been recently reworked (Dec 22nd 2016), so that it does not suffer anymore from previous limitations (e.g. both ends of a veth pair needed to be in netmap mode), and race conditions (crashes were possible). Now veth in native netmap mode work exactly like netmap pipes.

You can attach veth interfaces to vale switches in general, this works ok. It works ok also in conjuction with OVS patched with netmap, you can use veth interfaces similarly to how you use persistent VALE ports.

I've no idea why mininet does not work, it can simply be that the OVS patch is very experimental and incomplete. One of the problems is that broadcast does not work properly if you create more OVS bridges. A single datapath (in this case a VALE switch) is used to implement all the bridges, so the bridges are not isolated.

@Linerd
Copy link
Author

Linerd commented Jan 11, 2017

@vmaffione

I built the latest netmap and loaded both netmap.ko and veth.ko, and I created a simple mininet topology with the command mn, and the process is stuck at creating the switches, dmesg gives me the following errors. If I don't load veth.ko, the topology can be created, but h1 cannot ping h2.

[12429.730416] (null): netmap queues/slots: TX 1/1024, RX 1/1024
[12429.730637] (null): netmap queues/slots: TX 1/1024, RX 1/1024
[12429.731912] IPv6: ADDRCONF(NETDEV_UP): h1-eth0: link is not ready
[12429.734490] IPv6: ADDRCONF(NETDEV_CHANGE): h1-eth0: link becomes ready
[12429.737168] (null): netmap queues/slots: TX 1/1024, RX 1/1024
[12429.737453] (null): netmap queues/slots: TX 1/1024, RX 1/1024
[12429.740032] IPv6: ADDRCONF(NETDEV_UP): h2-eth0: link is not ready
[12429.743699] IPv6: ADDRCONF(NETDEV_CHANGE): h2-eth0: link becomes ready
[12429.756165] device ovs-system entered promiscuous mode
[12429.756900] BUG: unable to handle kernel NULL pointer dereference at 000000000000001e
[12429.756914] IP: [] ovs_vport_add+0x117/0x170 [openvswitch]
[12429.756927] PGD 20cefe067 PUD 20ceff067 PMD 0
[12429.756932] Oops: 0000 [#1] SMP
[12429.756936] Modules linked in: veth(OE) openvswitch(OE) netmap(OE) rfcomm pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat xt_addrtype iptable_filter ip_tables xt_conntrack x_tables br_netfilter bridge stp llc aufs bnep intel_rapl snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel btusb kvm irqbypass crct10dif_pclmul crc32_pclmul btrtl btbcm hp_wmi gpio_ich btintel sparse_keymap snd_hda_codec_realtek snd_hda_codec_generic bluetooth snd_hda_intel input_leds ghash_clmulni_intel aesni_intel aes_x86_64 snd_hda_codec snd_hda_core snd_hwdep snd_pcm lrw gf128mul glue_helper snd_seq_midi snd_seq_midi_event snd_rawmidi ablk_helper snd_seq cryptd snd_seq_device snd_timer snd serio_raw soundcore shpchp
[12429.757024] 8250_fintek lpc_ich ie31200_edac tpm_infineon edac_core mei_me mei mac_hid nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 gre nf_conntrack_ipv6 nf_nat_ipv6 libcrc32c nf_defrag_ipv6 nf_nat nf_conntrack parport_pc ppdev lp parport autofs4 hid_generic hid_logitech_hidpp hid_logitech_dj usbhid hid uas usb_storage nouveau mxm_wmi video i2c_algo_bit ttm psmouse ahci drm_kms_helper libahci syscopyarea sysfillrect sysimgblt fb_sys_fops e1000e drm ptp pps_core pata_acpi wmi fjes [last unloaded: openvswitch]
[12429.757084] CPU: 2 PID: 6770 Comm: ovs-vswitchd Tainted: G OE 4.4.0-59-generic #80-Ubuntu
[12429.757089] Hardware name: Hewlett-Packard HP Z210 Workstation/1588h, BIOS J51 v01.19 08/12/2011
[12429.757093] task: ffff880210ce7000 ti: ffff880200d2c000 task.ti: ffff880200d2c000
[12429.757097] RIP: 0010:[] [] ovs_vport_add+0x117/0x170 [openvswitch]
[12429.757107] RSP: 0018:ffff880200d2fa68 EFLAGS: 00010203
[12429.757110] RAX: 0000000000000016 RBX: ffffffffc08a0700 RCX: 0000000000000000
[12429.757114] RDX: 000000000002178c RSI: ffff88022dd1a0c0 RDI: ffff880225001900
[12429.757117] RBP: ffff880200d2fa78 R08: 000000000001a0c0 R09: ffffffffc0887f7e
[12429.757121] R10: ffffea00088a5ec0 R11: ffff880144a0b000 R12: 0000000000000016
[12429.757125] R13: ffff880220ad5980 R14: ffff8802085cd800 R15: 0000000000000001
[12429.757129] FS: 00007fd100122980(0000) GS:ffff88022dd00000(0000) knlGS:0000000000000000
[12429.757133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12429.757137] CR2: 000000000000001e CR3: 0000000144a01000 CR4: 00000000000406e0
[12429.757140] Stack:
[12429.757142] ffff880200d2faa8 ffff880221c0a300 ffff880200d2fa90 ffffffffc087b822
[12429.757149] ffff880200d2fb50 ffff880200d2fb08 ffffffffc087ccf2 ffff8800c902f814
[12429.757155] ffff8800c902f824 ffffffff00000001 0000000000000000 ffff880035357f00
[12429.757161] Call Trace:
[12429.757168] [] new_vport+0x12/0x50 [openvswitch]
[12429.757175] [] ovs_vport_cmd_new+0x142/0x280 [openvswitch]
[12429.757181] [] genl_family_rcv_msg+0x1e4/0x3e0
[12429.757187] [] ? __alloc_skb+0x5b/0x1f0
[12429.757191] [] ? genl_family_rcv_msg+0x3e0/0x3e0
[12429.757195] [] genl_rcv_msg+0x76/0xb0
[12429.757199] [] netlink_rcv_skb+0xa4/0xc0
[12429.757206] [] genl_rcv+0x28/0x40
[12429.757209] [] netlink_unicast+0x12f/0x1b0
[12429.757213] [] netlink_sendmsg+0x4a1/0x5f0
[12429.757219] [] ? aa_sock_msg_perm+0x61/0x150
[12429.757224] [] sock_sendmsg+0x38/0x50
[12429.757229] [] ___sys_sendmsg+0x281/0x290
[12429.757234] [] ? rtnl_unlock+0xe/0x10
[12429.757238] [] ? dev_ioctl+0x1ae/0x580
[12429.757243] [] ? sock_do_ioctl+0x42/0x50
[12429.757247] [] ? __fget_light+0x25/0x60
[12429.757251] [] __sys_sendmsg+0x51/0x90
[12429.757255] [] SyS_sendmsg+0x12/0x20
[12429.757260] [] entry_SYSCALL_64_fastpath+0x16/0x71
[12429.757264] Code: 83 48 8b 7b 38 e8 6a df 87 c0 84 c0 75 0c 48 c7 c0 9f ff ff ff 5b 41 5c 5d c3 4c 89 e7 ff 53 08 48 3d 00 f0 ff ff 49 89 c4 77 3c <48> 8b 40 08 49 8b 34 24 48 8b 78 60 e8 58 fb ff ff 48 8b 10 49
[12429.757320] RIP [] ovs_vport_add+0x117/0x170 [openvswitch]
[12429.757327] RSP
[12429.757330] CR2: 000000000000001e
[12429.762939] ---[ end trace 2325cc85140cc23b ]---

@vmaffione
Copy link
Collaborator

Yes, the OVS patch is experimental, you need to debug it.

Some pieces of advice:

  • Try to remove && vale_prefix(ovs_dp_name(parms->dp)) in datapath/vport.c::ovs_vport_add() to see if the crash goes away
  • Add print statements before and after the DEV_NETMAP block in ovs_vport_add() (e.g. D("before"); and D("after");), to check that the crash happens in that block.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants