route lookup sometimes returns incorrect result

[semihalf-berestovskyy-andriy]

Note: the issue was created automatically with bugzilla2github tool

Bugzilla Bug ID: 104

Date: 2016-10-15 11:37:38 +0200
From: [email protected]
To: [email protected]

Last updated: 2016-10-15 11:58:17 +0200

Bugzilla Comment ID: 192

Date: 2016-10-15 11:37:38 +0200
From: [email protected]

Route lookup sometimes returns an incorrect next hop. The problem
was noticed with GRE tunnels, but it looks like it may be a more
generic problem in OFP 2.0.0.

I tried GRE tunnels with this setup through cli:

ifconfig fp0 11.0.0.1/24
ifconfig fp1 12.0.0.1/24
route add 20.0.0.0/24 gw 12.0.0.2 dev fp1
ifconfig tunnel gre gre1 local 12.0.0.1 remote 20.0.0.1 peer 200.0.0.1 100.0.0.1

Traffic to 200.0.0.1 was forwarded by the fast path, all looked fine
(I did not check the actual GRE pakets sent out, though).

After adding the following second GRE tunnel, forwarding to the first
tunnel stopped working.

ifconfig tunnel gre gre2 local 12.0.0.1 remote 20.0.0.2 peer 200.0.1.1 100.0.1.1

After some investigation it seems that forwarding lookup for packets
destined to 200.0.0.1 returns the second tunnel as the next hop,
not the first one as it should.

With GDB I can see that routes in ofp_rt_mtrie_lookup.c look ok in here
(rules[7] and rules[8] correspond to the two GRE tunnels):

(gdb) p shm->rules[0]
$46 = {used = 1 '\001', masklen = 32 ' ', vrf = 0, addr = 16777227, data = {{
flags = 2097152, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {
flags = 0, gw = 0, port = 0, vlan = 0}}}
(gdb) p shm->rules[1]
$47 = {used = 1 '\001', masklen = 24 '\030', vrf = 0, addr = 11, data = {{
flags = 1, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0, port = 0,
vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}}}
(gdb) p shm->rules[2]
$48 = {used = 1 '\001', masklen = 8 '\b', vrf = 0, addr = 11, data = {{
flags = 2, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0, port = 0,
vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}}}
(gdb) p shm->rules[3]
$49 = {used = 1 '\001', masklen = 32 ' ', vrf = 0, addr = 16777228, data = {{
flags = 2097152, gw = 0, port = 1, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {
flags = 0, gw = 0, port = 0, vlan = 0}}}
(gdb) p shm->rules[4]
$50 = {used = 1 '\001', masklen = 24 '\030', vrf = 0, addr = 12, data = {{
flags = 1, gw = 0, port = 1, vlan = 0}, {flags = 0, gw = 0, port = 0,
vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}}}
(gdb) p shm->rules[5]
$51 = {used = 1 '\001', masklen = 8 '\b', vrf = 0, addr = 12, data = {{
flags = 2, gw = 0, port = 1, vlan = 0}, {flags = 0, gw = 0, port = 0,
vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}}}
(gdb) p shm->rules[6]
$52 = {used = 1 '\001', masklen = 24 '\030', vrf = 0, addr = 20, data = {{
flags = 2, gw = 33554444, port = 1, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {
flags = 0, gw = 0, port = 0, vlan = 0}}}
(gdb) p shm->rules[7]
$53 = {used = 1 '\001', masklen = 32 ' ', vrf = 0, addr = 16777416, data = {{
flags = 2, gw = 0, port = 10, vlan = 1}, {flags = 0, gw = 0, port = 0,
vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}}}
(gdb) p shm->rules[8]
$54 = {used = 1 '\001', masklen = 32 ' ', vrf = 0, addr = 16842952, data = {{
flags = 2, gw = 0, port = 10, vlan = 2}, {flags = 0, gw = 0, port = 0,
vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}}}
(gdb) p shm->rules[9]
$55 = {used = 0 '\000', masklen = 0 '\000', vrf = 0, addr = 0, data = {{
flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0, port = 0,
vlan = 0}, {flags = 0, gw = 0, port = 0, vlan = 0}, {flags = 0, gw = 0,
port = 0, vlan = 0}}}

However, ofp_rtl_search() for 200.0.0.1 returns the second tunnel
(vlan = 2) as the next hop. Here we are inside the lookup
(addr=16777416 i.e. 200.0.0.1):

(gdb) bt
#0 ofp_rtl_search (tree=, addr_be=)
at ofp_rt_mtrie_lookup.c:428
#1 0x00000000004136e2 in ofp_get_next_hop (vrf=,
addr=16777416, flags=flags@entry=0x7f0c361f9a1c) at ofp_route.c:449
#2 0x0000000000407182 in ofp_ipv4_processing (pkt=0x7f0c231e7ac0)
at ofp_pkt_processing.c:329
#3 0x0000000000407ee2 in ofp_packet_input (pkt=0x7f0c231e7ac0,
in_queue=in_queue@entry=0x0, pkt_func=0x407c80 <ofp_eth_vlan_processing>)
at ofp_pkt_processing.c:1419
#4 0x000000000040596b in pkt_io_recv (arg=) at app_main.c:77
#5 0x00007f0c65322b6f in odp_run_start_routine (arg=0x7ffca4e7ff48)
at linux.c:40
#6 0x00007f0c64ab560a in start_thread () from /lib64/libpthread.so.0
#7 0x00007f0c647efbbd in clone () from /lib64/libc.so.6

But the search returns the wrong next hop:

(gdb) fin
Run till exit from #0 ofp_rtl_search (tree=,
addr_be=) at ofp_rt_mtrie_lookup.c:428
ofp_get_next_hop (vrf=, addr=16777416,
flags=flags@entry=0x7f0c361f9a1c) at ofp_route.c:456
456 }
Value returned is $56 = (struct ofp_nh_entry *) 0x7f0c39410044

(gdb) p *(struct ofp_nh_entry *) 0x7f0c39410044
$57 = {flags = 2, gw = 0, port = 10, vlan = 2}