VPC VR: remote access VPN clients get no reply from LB rules or the router's public addresses (reply leaves by the public NIC's gateway) #14547

problem

A remote access VPN (L2TP/IPsec) client on a VPC can reach tier VMs and port forwards, but gets no answer from
anything the VPC router itself terminates on a public address: an LB rule (haproxy on the router), or a ping to one of
the router's public IPs. The request reaches the router. The reply goes out of the public NIC to that network's gateway
instead of into the client's ppp interface.

Example, with CloudStack's default client range (local_ip 10.1.2.1, clients 10.1.2.2-10.1.2.8), source NAT
203.0.113.2/28 on eth1, a second public network 198.51.100.128/25 on eth2 with an LB rule on 198.51.100.131, and a
tier 192.0.2.0/24 on eth3:

$ ip rule show
0:      from all lookup local
220:    from all lookup 220
32759:  from 198.51.100.128/25 lookup Table_eth2
32760:  from all fwmark 0x67 lookup Table_eth3
32761:  from 203.0.113.0/28 lookup Table_eth1
32762:  from all fwmark 0x66 lookup Table_eth2
32763:  from all fwmark 0x65 lookup Table_eth1
32766:  from all lookup main

$ ip route show table Table_eth2
default via 198.51.100.129 dev eth2 proto static
throw 192.0.2.0/24 proto static
throw 198.51.100.128/25 proto static

main holds the client's host route (10.1.2.2 dev ppp0), but the haproxy SYN-ACK from 198.51.100.131 to 10.1.2.2
matches rule 32759 first, and Table_eth2 has no entry for the client range, so it takes the default route out of
eth2. The same happens to an ICMP echo reply from 198.51.100.130, and from 203.0.113.2 through Table_eth1.

A port forward on the same address works, because the reply leaves the backend VM with its tier address as source
(reverse NAT happens after routing), matches no from rule and is routed by main. Pings to local_ip and to the
tier gateway work for the same reason.

Root cause, line numbers from the 4.22 branch (56306ab), same code in main:

• systemvm/debian/opt/cloud/bin/cs/CsAddress.py:848-861 (post_config_change, VPC branch): for each public NIC, a
  default route in Table_<dev>, a throw for each tier network (CsRoute.add_network_route, cs/CsRoute.py:63-74),
  a throw for the NIC's own network, and ip rule add from <public network> table Table_<dev>
  (cs/CsRule.py:33-35). Nothing is added for the remote access VPN client range.
• systemvm/debian/opt/cloud/bin/configure.py:1292-1329 (remoteaccessvpn_iptables) only adds iptables rules; the
  VPN never touches routing.

Impact: on any VPC with a remote access VPN, clients can't use LB rules or reach the router's public addresses through
the tunnel. For CKS clusters this means a Service of type LoadBalancer can't be reached over the VPN, while the API
server through its port forward can. This is not a regression; 4.20 builds the same tables. Isolated networks build
similar per-NIC tables (CsAddress.py:834-846) and are probably affected too; not tested.

versions

Seen on 4.23.0.0 with KVM, VPC with redundant routers, advanced networking. The code is the same on the 4.22 branch
and main.

The steps to reproduce the bug

1. Create a VPC with a tier and a VM in it, and enable remote access VPN on the source NAT IP. Add a VPN user.
2. Acquire a public IP, and add an LB rule on port 80 to the VM (an ACL rule allowing port 80 to the tier).
3. Connect an L2TP/IPsec client.
4. From the client, curl http://<LB address>/ times out and ping <router public IP> gets no reply. A port forward
   from the same public IP to the VM works.
5. On the router, tcpdump -ni any host <client address> shows the replies leaving the public NIC towards its gateway.

What to do about it?

Add a throw for each client network to every per-NIC table, the same way tier networks are thrown, so a reply to a
VPN client continues to main, where the ppp host routes are. Reconcile them on every config pass, since a NIC change
can flush and rebuild a table, and remove them when the VPN is removed. Because those routes would also send replies
to a spoofed client address into that client's tunnel, drop packets sourced from the client range that don't arrive
on ppp+. PR to follow.