Netronome网卡Firewall设计

本文Netronome Firewall主要是验证Netronome智能网卡P4功能，以Firewall功能特性实现简单的ACL防火墙功能，整体上分为数据平面和控制平面。

netty物联网系统架构（Netronome智能网卡实现ACLFirewall）(1)

如上图所示，数据平面使用P4设计了基于v1model架构的pipeline，控制平面基于ONOS控制器，由于Netronome提供nfp-sdk6-rte对P4runtime支持较差(有人验证Ternary matching和Packet-Out都不支持，具体支持个人也没验证过，不过nfp-sdk版本多年不更新...)，这里基于nfp-sdk6-rte中的rtecli命令行在ONOS中包了个南向协议，并且加了个驱动来纳管Netronome智能网卡，实现了基本的设备连接、pipeline配置和流表增删功能。

这里驱动没有按ONOS规范设计，仅简化为了满足ONOS纳管设备要求，其实按驱动行为规范实现，我这里图简便简化处理了，也有人在智能网卡侧跑一个代理将Openflow或P4runtime转成rtecli来管理，实现复杂了点，我没采用该方案。

Firewall测试直接用netns隔离vf，并配置ip，实现L2互通，然后配置ACL来验证，更复杂的组网场景理论上结合智能网卡的真实物理口(p0&p4)和丰富的vf接口和pipeline中fwd表项或对pipeline重新设计也能得到应用。

数据面设计

数据面基于v1model架构，设计包含两个表的P4 Pipeline: t_acl -> t_fwd，先执行t_acl表判断流量是否满足acl规则，再执行t_fwd表进行转发(目前仅实现端口转发和二层转发)，具体局部table设计如下:

direct_counter(CounterType.packets_and_bytes) t_acl_counter; action allow() { t_acl_counter.count(); } action deny() { mark_to_drop(); t_acl_counter.count(); } action nop() { t_acl_counter.count(); } // acl table table t_acl { key = { standard_metadata.ingress_port : ternary; hdr.ethernet.src_addr : ternary; hdr.ethernet.dst_addr : ternary; hdr.ethernet.ether_type : ternary; hdr.ipv4.src_addr : ternary; hdr.ipv4.dst_addr : ternary; hdr.ipv4.protocol : ternary; hdr.tcp.src_port : ternary; hdr.tcp.dst_port : ternary; hdr.udp.src_port : ternary; hdr.udp.dst_port : ternary; } actions = { allow; deny; nop; } size = 1024; default_action = nop(); counters = t_acl_counter; }

direct_counter(CounterType.packets_and_bytes) t_fwd_counter; action fwd(port_t port) { standard_metadata.egress_spec = port; t_fwd_counter.count(); } table t_fwd { key = { standard_metadata.ingress_port : ternary; hdr.ethernet.dst_addr : ternary; } actions = { fwd; } default_action = fwd(0); counters = t_fwd_counter; size = 1024; }

控制面设计

控制面基于ONOS，实现基本的设备纳管，Pipeline配置和流表增删功能，主要包括：rtecli南向协议、rtecli驱动和实现Firewall的app，如下几个模块的实现描述.

rtecli南向协议通过将rtecli命令行包装了一下，提供基本的design load(pipeline配置)、设备连接、流表增删等功能.

rtecli驱动基于onos中驱动行为机制，编写了rtecli驱动使用的xml并实现基本的驱动行为，实现基本的Handshaker.

Firewall onos app为标准的ONOS app，onos版本基于2.4.0，提供onos cli命令对ACL防火墙策略和端口/L2转发配置提供命令行(后面可能会增加北向接口和UI)，Firewall服务提供核心的ACL防火墙流表和端口/L2转发流表实现(通过rtecli控制智能网卡，将acl策略和fwd表下发/删除流表).

nfp-sdk6-rte启动

# mod reload sudo modprobe -r -v nfp && sudo modprobe nfp nfp_pf_netdev=0 nfp_dev_cpp=1 # start nfp-sdk6-rte sudo systemctl start nfp-sdk6-rte # check nfp-sdk6-rte status sudo systemctl status nfp-sdk6-rte

通过ip a已经可以看到默认创建的4个vf接口，可启动nfp-sdk6-rte设置NUM_VFS环境变量来设置启动vf数目

10: vf0_0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether d6:61:9c:56:1c:92 brd ff:ff:ff:ff:ff:ff 11: vf0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether fe:a2:d2:49:f6:cd brd ff:ff:ff:ff:ff:ff 12: vf0_2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 26:1d:8e:b3:36:13 brd ff:ff:ff:ff:ff:ff 13: vf0_3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 6a:ec:a4:ba:13:f4 brd ff:ff:ff:ff:ff:ff

onos & firewall app启动

# install nfp sdk, 由于onos南向通过rtecli来和智能网卡通信，所以也要安装nfp-sdk sudo dpkg -i nfp-sdk_6.1.0.1-preview-3243-2_amd64.deb # onos 启动 make onos-run # firewall app编译 make app-build # firewall app加载 make app-reload

netcfg配置

make netcfg-firewall

firewall-app/src/main/resources/nfp_nics.json配置文件：

{ "apps": { "cn.pcl.firewall": { "nfp_nics": [ { "dp_id": "rtecli:nfp-nic1", "driver": "rtecli", "rte_host": "172.16.30.15", "rte_port": "20206", "nffw_path": "path of firewall.nffw", "design_path": "path of pif_design.json", "p4cfg_path": "" } ] } } }

vf接口netns隔离

sudo ip netns add ns0 sudo ip netns add ns1 sudo ip netns add ns2 sudo ip netns add ns3 sudo ip link set vf0_0 netns ns0 sudo ip link set vf0_1 netns ns1 sudo ip link set vf0_2 netns ns2 sudo ip link set vf0_3 netns ns3

配置ip及checksum

# 配置ip sudo ip netns exec ns0 /bin/bash ifconfig vf0_0 192.168.10.5 netmask 255.255.255.0 up # checksum off ethtool -K vf0_0 tx off ethtool -K vf0_0 rx off

# 配置ip sudo ip netns exec ns1 /bin/bash ifconfig vf0_0 192.168.10.10 netmask 255.255.255.0 up # checksum off ethtool -K vf0_1 tx off ethtool -K vf0_1 rx off

测试L2互通

# login onos cli make onos-cli # apply fwd flow rules fwd -d=00:15:4d:00:00:00 add rtecli:nfp-nic1 fwd 768 fwd -d=00:15:4d:00:00:01 add rtecli:nfp-nic1 fwd 769 # show fwd flow rules fwd show rtecli:nfp-nic1 09:54:46 =================================================================== | ID | MATCH | ACTION | =================================================================== | ID | IngressPort | DstMac | Action | Output | ------------------------------------------------------------------- | FWD-1000 | | 00:15:4d:00:00:00 | fwd | 768 | | FWD-1001 | | 00:15:4d:00:00:01 | fwd | 769 | -------------------------------------------------------------------

sudo ip netns exec ns0 /bin/bash ping 192.168.10.10

PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data. 64 bytes from 192.168.10.10: icmp_seq=1 ttl=64 time=0.273 ms 64 bytes from 192.168.10.10: icmp_seq=2 ttl=64 time=0.220 ms 64 bytes from 192.168.10.10: icmp_seq=3 ttl=64 time=0.215 ms 64 bytes from 192.168.10.10: icmp_seq=4 ttl=64 time=0.212 ms 64 bytes from 192.168.10.10: icmp_seq=5 ttl=64 time=0.209 ms 64 bytes from 192.168.10.10: icmp_seq=6 ttl=64 time=0.213 ms

sudo ip netns exec ns1 /bin/bash ping 192.168.10.5

PING 192.168.10.5 (192.168.10.5) 56(84) bytes of data. 64 bytes from 192.168.10.5: icmp_seq=1 ttl=64 time=0.252 ms 64 bytes from 192.168.10.5: icmp_seq=2 ttl=64 time=0.218 ms 64 bytes from 192.168.10.5: icmp_seq=3 ttl=64 time=0.212 ms 64 bytes from 192.168.10.5: icmp_seq=4 ttl=64 time=0.213 ms 64 bytes from 192.168.10.5: icmp_seq=5 ttl=64 time=0.213 ms 64 bytes from 192.168.10.5: icmp_seq=6 ttl=64 time=0.208 ms 64 bytes from 192.168.10.5: icmp_seq=7 ttl=64 time=0.212 ms

sudo ip netns exec ns1 /bin/bash iperf -s -i 1

sudo ip netns exec ns0 /bin/bash iperf -c 192.168.10.10 -i 1

------------------------------------------------------------ Client connecting to 192.168.10.10, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.10.5 port 35252 connected with 192.168.10.10 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 958 MBytes 8.03 Gbits/sec [ 3] 1.0- 2.0 sec 972 MBytes 8.15 Gbits/sec [ 3] 2.0- 3.0 sec 934 MBytes 7.83 Gbits/sec [ 3] 3.0- 4.0 sec 938 MBytes 7.87 Gbits/sec [ 3] 4.0- 5.0 sec 926 MBytes 7.77 Gbits/sec [ 3] 5.0- 6.0 sec 927 MBytes 7.78 Gbits/sec [ 3] 6.0- 7.0 sec 918 MBytes 7.70 Gbits/sec [ 3] 7.0- 8.0 sec 921 MBytes 7.73 Gbits/sec [ 3] 8.0- 9.0 sec 913 MBytes 7.66 Gbits/sec [ 3] 0.0-10.0 sec 9.12 GBytes 7.83 Gbits/sec

------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.10.10 port 5001 connected with 192.168.10.5 port 35252 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 954 MBytes 8.01 Gbits/sec [ 4] 1.0- 2.0 sec 973 MBytes 8.16 Gbits/sec [ 4] 2.0- 3.0 sec 933 MBytes 7.83 Gbits/sec [ 4] 3.0- 4.0 sec 939 MBytes 7.88 Gbits/sec [ 4] 4.0- 5.0 sec 927 MBytes 7.78 Gbits/sec [ 4] 5.0- 6.0 sec 925 MBytes 7.76 Gbits/sec [ 4] 6.0- 7.0 sec 918 MBytes 7.70 Gbits/sec [ 4] 7.0- 8.0 sec 921 MBytes 7.73 Gbits/sec [ 4] 8.0- 9.0 sec 913 MBytes 7.66 Gbits/sec [ 4] 9.0-10.0 sec 927 MBytes 7.78 Gbits/sec [ 4] 0.0-10.0 sec 9.12 GBytes 7.83 Gbits/sec

sudo ip netns exec ns1 /bin/bash iperf -s -i 1 -u

sudo ip netns exec ns0 /bin/bash iperf -c 192.168.10.10 -i 1 -u

------------------------------------------------------------ Client connecting to 192.168.10.10, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.10.5 port 35162 connected with 192.168.10.10 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 129 KBytes 1.06 Mbits/sec [ 3] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 2.0- 3.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 3.0- 4.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 4.0- 5.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 5.0- 6.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 6.0- 7.0 sec 129 KBytes 1.06 Mbits/sec [ 3] 7.0- 8.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 8.0- 9.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 9.0-10.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec [ 3] Sent 893 datagrams [ 3] Server Report: [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.001 ms 0/ 893 (0%)

------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.10.10 port 5001 connected with 192.168.10.5 port 35162 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 128 KBytes 1.05 Mbits/sec 0.003 ms 0/ 89 (0%) [ 3] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec 0.002 ms 0/ 89 (0%) [ 3] 2.0- 3.0 sec 128 KBytes 1.05 Mbits/sec 0.002 ms 0/ 89 (0%) [ 3] 3.0- 4.0 sec 128 KBytes 1.05 Mbits/sec 0.001 ms 0/ 89 (0%) [ 3] 4.0- 5.0 sec 128 KBytes 1.05 Mbits/sec 0.001 ms 0/ 89 (0%) [ 3] 5.0- 6.0 sec 129 KBytes 1.06 Mbits/sec 0.001 ms 0/ 90 (0%) [ 3] 6.0- 7.0 sec 128 KBytes 1.05 Mbits/sec 0.002 ms 0/ 89 (0%) [ 3] 7.0- 8.0 sec 128 KBytes 1.05 Mbits/sec 0.001 ms 0/ 89 (0%) [ 3] 8.0- 9.0 sec 128 KBytes 1.05 Mbits/sec 0.002 ms 0/ 89 (0%) [ 3] 9.0-10.0 sec 128 KBytes 1.05 Mbits/sec 0.001 ms 0/ 89 (0%) [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.001 ms 0/ 893 (0%)

ACL防火墙场景1

# login onos cli make onos-cli # apply acl flow rules acl -t=ipv4 --ipDst=192.168.10.10 add rtecli:nfp-nic1 deny # show acl flow rules acl show rtecli:nfp-nic1 ======================================================================================================================================================== | ID | MATCH | ACTION | ======================================================================================================================================================== | ID | IngressPort | SrcMac | DstMac | EthType | Protocol | SrcIp | DstIp | SrcPort | DstPort | Action | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ACL-1000 | | | | ipv4 | | | 192.168.10.10 | | | deny | --------------------------------------------------------------------------------------------------------------------------------------------------------

ping 192.168.10.10 PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data. ^C --- 192.168.10.10 ping statistics --- 11 packets transmitted, 0 received, 100% packet loss, time 10218ms

ping 192.168.10.5 PING 192.168.10.5 (192.168.10.5) 56(84) bytes of data. ^C --- 192.168.10.5 ping statistics --- 9 packets transmitted, 0 received, 100% packet loss, time 8186ms

sudo ip netns exec ns1 /bin/bash iperf -s -i 1

sudo ip netns exec ns0 /bin/bash iperf -c 192.168.10.10 -i 1

# iperf client connect failed: Connection timed out # iperf server 看不到client端连接日志

sudo ip netns exec ns1 /bin/bash iperf -s -i 1 -u

sudo ip netns exec ns0 /bin/bash iperf -c 192.168.10.10 -i 1 -u

# iperf client 正常发包(udp无连接) # iperf server 看不到client端连接日志

场景2

# login onos cli make onos-cli acl -t=ipv4 --protocol=icmp --ipDst=192.168.10.10 add rtecli:nfp-nic1 deny acl show rtecli:nfp-nic1 ======================================================================================================================================================== | ID | MATCH | ACTION | ======================================================================================================================================================== | ID | IngressPort | SrcMac | DstMac | EthType | Protocol | SrcIp | DstIp | SrcPort | DstPort | Action | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ACL-1000 | | | | ipv4 | icmp | | 192.168.10.10 | | | deny | --------------------------------------------------------------------------------------------------------------------------------------------------------

ping 192.168.10.10 PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data. ^C --- 192.168.10.10 ping statistics --- 11 packets transmitted, 0 received, 100% packet loss, time 10235ms

ping 192.168.10.5 PING 192.168.10.5 (192.168.10.5) 56(84) bytes of data. ^C --- 192.168.10.5 ping statistics --- 9 packets transmitted, 0 received, 100% packet loss, time 8186ms

iperf -c 192.168.10.10 -i 1 ------------------------------------------------------------ Client connecting to 192.168.10.10, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.10.5 port 35256 connected with 192.168.10.10 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 883 MBytes 7.41 Gbits/sec [ 3] 1.0- 2.0 sec 917 MBytes 7.69 Gbits/sec [ 3] 2.0- 3.0 sec 918 MBytes 7.70 Gbits/sec [ 3] 3.0- 4.0 sec 915 MBytes 7.67 Gbits/sec [ 3] 4.0- 5.0 sec 921 MBytes 7.72 Gbits/sec [ 3] 5.0- 6.0 sec 910 MBytes 7.63 Gbits/sec [ 3] 6.0- 7.0 sec 926 MBytes 7.77 Gbits/sec [ 3] 7.0- 8.0 sec 919 MBytes 7.71 Gbits/sec [ 3] 8.0- 9.0 sec 920 MBytes 7.72 Gbits/sec [ 3] 9.0-10.0 sec 906 MBytes 7.60 Gbits/sec [ 3] 0.0-10.0 sec 8.92 GBytes 7.66 Gbits/sec

iperf -s -i 1 ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.10.10 port 5001 connected with 192.168.10.5 port 35256 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 881 MBytes 7.39 Gbits/sec [ 4] 1.0- 2.0 sec 916 MBytes 7.68 Gbits/sec [ 4] 2.0- 3.0 sec 918 MBytes 7.70 Gbits/sec [ 4] 3.0- 4.0 sec 916 MBytes 7.68 Gbits/sec [ 4] 4.0- 5.0 sec 919 MBytes 7.71 Gbits/sec [ 4] 5.0- 6.0 sec 911 MBytes 7.64 Gbits/sec [ 4] 6.0- 7.0 sec 925 MBytes 7.76 Gbits/sec [ 4] 7.0- 8.0 sec 920 MBytes 7.72 Gbits/sec [ 4] 8.0- 9.0 sec 921 MBytes 7.72 Gbits/sec [ 4] 9.0-10.0 sec 906 MBytes 7.60 Gbits/sec [ 4] 0.0-10.0 sec 8.92 GBytes 7.66 Gbits/sec

场景3

# login onos cli make onos-cli acl -t=ipv4 --protocol=tcp --ipDst=192.168.10.10 --tpDst=5001 add rtecli:nfp-nic1 deny acl show rtecli:nfp-nic1 09:32:35 ======================================================================================================================================================== | ID | MATCH | ACTION | ======================================================================================================================================================== | ID | IngressPort | SrcMac | DstMac | EthType | Protocol | SrcIp | DstIp | SrcPort | DstPort | Action | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ACL-1000 | | | | ipv4 | icmp | | 192.168.10.10 | | | deny | | ACL-1001 | | | | ipv4 | tcp | | 192.168.10.10 | | 5001 | deny | --------------------------------------------------------------------------------------------------------------------------------------------------------

iperf -c 192.168.10.10 -i 1 connect failed: Connection timed out

,