UDP port forwarding corrupts DNS packets

[simondeziel]

When using OVN port forward as configured through LXD, DNS traffic is corrupted. Here's the corruption as observed over UDPv4:

# 10.207.239.2 is the IP assigned to u1
+ lxc network forward port add ovn-virtual-network 192.0.2.1 udp 53 10.207.239.2
+ dig a @192.0.2.1 u1.lxd
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> a @192.0.2.1 u1.lxd
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53495
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; WARNING: Message has 22 extra bytes at end

;; QUESTION SECTION:
;u1.lxd.				IN	A

;; ANSWER SECTION:
.			0	CLASS1232 OPT	10 8 tonCduVVIdo=

;; Query time: 4 msec
;; SERVER: 192.0.2.1#53(192.0.2.1) (UDP)
;; WHEN: Wed Sep 20 00:43:16 UTC 2023
;; MSG SIZE  rcvd: 69

And through UDPv6 forwards:

# fd42:e6c:5d68:b832:216:3eff:feeb:52b1 is the IP assigned to u1
+ lxc network forward port add ovn-virtual-network 2001:db8:1:2::1 udp 53 fd42:e6c:5d68:b832:216:3eff:feeb:52b1
+ dig aaaa @2001:db8:1:2::1 u1.lxd
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> aaaa @2001:db8:1:2::1 u1.lxd
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22090
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; WARNING: Message has 34 extra bytes at end

;; QUESTION SECTION:
;u1.lxd.				IN	AAAA

;; ANSWER SECTION:
.			0	CLASS1232 OPT	10 8 Tj9/xCfbVaY=

;; Query time: 0 msec
;; SERVER: 2001:db8:1:2::1#53(2001:db8:1:2::1) (UDP)
;; WHEN: Wed Sep 20 00:43:16 UTC 2023
;; MSG SIZE  rcvd: 81

In both cases, the extra bytes is always 22 for UDPv4 34 for UDPv6. Our environment uses OVN 23.06.1 which isn't the latest but I couldn't find any relevant commit in recent history.

I couldn't easily extract reproducing steps using just OVN commands but will happily try to if a dev could guide me into extracting more debug info. Thanks!

[dceara]

I think the potential fix for #228 (comment) will also address the issue reported here.

[simondeziel]

I'll try and get that tested soon, thanks!

[dceara]

@simondeziel Thanks! I also posted the formal patch on the dev mailing list:
https://patchwork.ozlabs.org/project/ovn/patch/[email protected]/

[simondeziel]

@dceara I've yet to test your patch but with OVN v23.09.1, I'm no longer observing garbled DNS replies. I'll let you know how it goes with your patch.

[dceara]

@simondeziel I wonder if that's not because of 4b10571 (which is in v23.09.1) but that introduces other issues, e.g. problems with EDNS: #228