openshift Slow performance on pod to pod communication over vxlan in openshift-s

admin

Slow performance on pod to pod communication over vxlan in openshift-sdn
7 V0 W% m; I; x2 v

环境
# p* }* \+ ^' g5 v5 {- VRed Hat OpenShift Container Platform
Red Hat CoreOS
kernel-4.18.0-193.41.1.el8_2.x86_64
iptables-nft
问题
TCP and UDP iperf3 test between pods on different nodes over openshift-sdn is very slow. Even if the nodes are on the same Hypervisor or when an iperf between the node IPs is fast.
5 N$ U' V1 a" x: i9 [& CRaw
[  5]   0.00-1.00   sec  5.31 MBytes  44.5 Mbits/sec   18    620 KBytes
[  5]   1.00-2.00   sec  4.12 MBytes  34.5 Mbits/sec    0    625 KBytes
[  5]   2.00-3.00   sec  4.02 MBytes  33.7 Mbits/sec    0    628 KBytes
[  5]   3.00-4.00   sec  4.13 MBytes  34.6 Mbits/sec    0    640 KBytes
# u4 v, H5 U$ w[  5]   4.00-5.00   sec  4.15 MBytes  34.8 Mbits/sec    0    665 KBytes
[  5]   5.00-6.00   sec  3.95 MBytes  33.1 Mbits/sec    7    673 KBytes
. b1 M8 t: h8 s( h; e6 C[  5]   6.00-7.00   sec  4.03 MBytes  33.8 Mbits/sec    3    675 KBytes
# f4 {0 J' `% a6 dThe same iperf3 tests after adding the iptables rules in the resolution section where performance is boosted from MB/s to GB/s:
! T; A! d( R0 J0 U8 P4 T0 [Raw
/ H# x1 k+ p9 O1 s% u- ~[  5] 490.00-491.00 sec   382 MBytes  3.20 Gbits/sec    4   1.02 MBytes
/ [! i& ^  G. @[  5] 491.00-492.00 sec   403 MBytes  3.38 Gbits/sec    4    957 KBytes
5 G; P! m: Z* ]( }[  5] 492.00-493.00 sec   404 MBytes  3.39 Gbits/sec   12    869 KBytes
3 d5 u3 b/ N9 s( A( Y[  5] 493.00-494.00 sec   398 MBytes  3.34 Gbits/sec    0   1.10 MBytes
[  5] 494.00-495.00 sec   384 MBytes  3.23 Gbits/sec   11   1.02 MBytes
Conntrack shows several UNREPLIED entries at the vxlan port
Raw
$ cat /proc/net/nf_conntrack | egrep udp | egrep dport=4789 | egrep UNREPLIED | wc -l
232
$ cat /proc/net/nf_conntrack | egrep udp | egrep dport=4789 | wc -l
232
决议
! X; @, H2 H9 {) zThis issue for IPv4 is resolved in releases:

4.9.0
' m! \8 P' R& I- V4.8.10
$ F* f* Z: h$ F  I. t  A4.7.30
# r9 g7 {* V9 @4 c. b  V' ]6 G. X: u4.6.45
Workaround
! `. e1 O0 ^5 ^) d- T* nApply these these iptables rules on the affected nodes:

Raw
/ m2 l$ K: {# A/ T! o9 g# iptables -t raw -A OUTPUT -p udp --dport 4789 -j NOTRACK
# iptables -t raw -A PREROUTING -p udp --dport 4789 -j NOTRACK
根源
Unlike other protocols like DNS, VXLAN doesn't have conversations where one client sends a packet from ${IP1}:${PORT1} to ${IP2}:${PORT2} and expects an answer from the server coming from ${IP2}:${PORT2} to ${IP1}:${PORT1}. Instead, whenever some host wants to communicate with the other host, it will always send packets from ${IP1}:${RANDOM_PORT} to ${IP2}:${VXLAN_PORT:-4789} and if the other hosts sends a packet to the first host, then it would send a packet from ${IP2}:${RANDOM_PORT} to ${IP1}:${VXLAN_PORT:-4789}. What will never happen is that some packet gets replied to the random port used as client port of another packet, so doing connection tracking in VXLAN is not required and doesn't make sense.

However, although not required, doing such connection tracking can have negative side effects in the performance on some scenario, specially if the number of iptables rules of the cluster is high due to the cluster having a very big number of services.

Each vxlan packet would be unnecessarily traversing the iptables rules which can cause delays. As this is a sequential operation and a check needs to be done for each rule it slows UDP packets down considerably. The vxlan makes the following calls:

Raw
vxlan_xmit_one()->udp_tunnel_xmit_skb()->iptunnel_xmit()->ip_local_out()
On the egress side the ip_local_out() routine will call into the netfilter routines as will incoming vxlan packets on the ingress side. With the iptables rules as per the resolution section in place the vxlan packets will not traverse the NAT rules as nf_conntrack is required to do that which mitigates the delay and improves bandwidth.

诊断步骤
Check the number of iptables rules on the nodes:

Raw
# iptables-save | wc -l
153900