Commit | Line | Data |
---|---|---|
1da177e4 LT |
1 | |
2 | ||
d012827e | 3 | HOWTO for the linux packet generator |
1da177e4 LT |
4 | ------------------------------------ |
5 | ||
4e081e0c BH |
6 | Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel |
7 | or as a module. A module is preferred; modprobe pktgen if needed. Once | |
ca5b542c BH |
8 | running, pktgen creates a thread for each CPU with affinity to that CPU. |
9 | Monitoring and controlling is done via /proc. It is easiest to select a | |
10 | suitable sample script and configure that. | |
1da177e4 LT |
11 | |
12 | On a dual CPU: | |
13 | ||
14 | ps aux | grep pkt | |
15 | root 129 0.3 0.0 0 0 ? SW 2003 523:20 [pktgen/0] | |
16 | root 130 0.3 0.0 0 0 ? SW 2003 509:50 [pktgen/1] | |
17 | ||
18 | ||
2fe0ae78 | 19 | For monitoring and control pktgen creates: |
1da177e4 LT |
20 | /proc/net/pktgen/pgctrl |
21 | /proc/net/pktgen/kpktgend_X | |
22 | /proc/net/pktgen/ethX | |
23 | ||
24 | ||
9ceb87fc JDB |
25 | Tuning NIC for max performance |
26 | ============================== | |
27 | ||
ca5b542c | 28 | The default NIC settings are (likely) not tuned for pktgen's artificial |
9ceb87fc JDB |
29 | overload type of benchmarking, as this could hurt the normal use-case. |
30 | ||
31 | Specifically increasing the TX ring buffer in the NIC: | |
32 | # ethtool -G ethX tx 1024 | |
33 | ||
34 | A larger TX ring can improve pktgen's performance, while it can hurt | |
35 | in the general case, 1) because the TX ring buffer might get larger | |
ca5b542c | 36 | than the CPU's L1/L2 cache, 2) because it allows more queueing in the |
9ceb87fc JDB |
37 | NIC HW layer (which is bad for bufferbloat). |
38 | ||
ca5b542c | 39 | One should hesitate to conclude that packets/descriptors in the HW |
9ceb87fc | 40 | TX ring cause delay. Drivers usually delay cleaning up the |
ca5b542c BH |
41 | ring-buffers for various performance reasons, and packets stalling |
42 | the TX ring might just be waiting for cleanup. | |
9ceb87fc | 43 | |
ca5b542c BH |
44 | This cleanup issue is specifically the case for the driver ixgbe |
45 | (Intel 82599 chip). This driver (ixgbe) combines TX+RX ring cleanups, | |
9ceb87fc JDB |
46 | and the cleanup interval is affected by the ethtool --coalesce setting |
47 | of parameter "rx-usecs". | |
48 | ||
ca5b542c | 49 | For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6): |
9ceb87fc JDB |
50 | # ethtool -C ethX rx-usecs 30 |
51 | ||
52 | ||
1da177e4 LT |
53 | Viewing threads |
54 | =============== | |
d012827e JDB |
55 | /proc/net/pktgen/kpktgend_0 |
56 | Running: | |
57 | Stopped: eth1 | |
58 | Result: OK: add_device=eth1 | |
1da177e4 | 59 | |
ca5b542c BH |
60 | Most important are the devices assigned to the thread. Note that a |
61 | device can only belong to one thread. | |
1da177e4 LT |
62 | |
63 | ||
64 | Viewing devices | |
65 | =============== | |
66 | ||
ca5b542c BH |
67 | The Params section holds configured information. The Current section |
68 | holds running statistics. The Result is printed after a run or after | |
69 | interruption. Example: | |
1da177e4 | 70 | |
d012827e | 71 | /proc/net/pktgen/eth1 |
1da177e4 LT |
72 | |
73 | Params: count 10000000 min_pkt_size: 60 max_pkt_size: 60 | |
74 | frags: 0 delay: 0 clone_skb: 1000000 ifname: eth1 | |
75 | flows: 0 flowlen: 0 | |
76 | dst_min: 10.10.11.2 dst_max: | |
77 | src_min: src_max: | |
78 | src_mac: 00:00:00:00:00:00 dst_mac: 00:04:23:AC:FD:82 | |
79 | udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9 | |
80 | src_mac_count: 0 dst_mac_count: 0 | |
81 | Flags: | |
82 | Current: | |
83 | pkts-sofar: 10000000 errors: 39664 | |
84 | started: 1103053986245187us stopped: 1103053999346329us idle: 880401us | |
85 | seq_num: 10000011 cur_dst_mac_offset: 0 cur_src_mac_offset: 0 | |
86 | cur_saddr: 0x10a0a0a cur_daddr: 0x20b0a0a | |
87 | cur_udp_dst: 9 cur_udp_src: 9 | |
88 | flows: 0 | |
89 | Result: OK: 13101142(c12220741+d880401) usec, 10000000 (60byte,0frags) | |
90 | 763292pps 390Mb/sec (390805504bps) errors: 39664 | |
91 | ||
5d3f083d ML |
92 | Configuring threads and devices |
93 | ================================ | |
7c95a9d9 BH |
94 | This is done via the /proc interface, and most easily done via pgset |
95 | as defined in the sample scripts. | |
1da177e4 LT |
96 | |
97 | Examples: | |
98 | ||
99 | pgset "clone_skb 1" sets the number of copies of the same packet | |
100 | pgset "clone_skb 0" use single SKB for all transmits | |
38b2cf29 AS |
101 | pgset "burst 8" uses xmit_more API to queue 8 copies of the same |
102 | packet and update HW tx queue tail pointer once. | |
103 | "burst 1" is the default | |
1da177e4 LT |
104 | pgset "pkt_size 9014" sets packet size to 9014 |
105 | pgset "frags 5" packet will consist of 5 fragments | |
106 | pgset "count 200000" sets number of packets to send, set to zero | |
d0f19d82 | 107 | for continuous sends until explicitly stopped. |
1da177e4 LT |
108 | |
109 | pgset "delay 5000" adds delay to hard_start_xmit(). nanoseconds | |
110 | ||
111 | pgset "dst 10.0.0.1" sets IP destination address | |
112 | (BEWARE! This generator is very aggressive!) | |
113 | ||
114 | pgset "dst_min 10.0.0.1" Same as dst | |
115 | pgset "dst_max 10.0.0.254" Set the maximum destination IP. | |
116 | pgset "src_min 10.0.0.1" Set the minimum (or only) source IP. | |
117 | pgset "src_max 10.0.0.254" Set the maximum source IP. | |
118 | pgset "dst6 fec0::1" IPV6 destination address | |
119 | pgset "src6 fec0::2" IPV6 source address | |
120 | pgset "dstmac 00:00:00:00:00:00" sets MAC destination address | |
121 | pgset "srcmac 00:00:00:00:00:00" sets MAC source address | |
122 | ||
896a7cf8 ED |
123 | pgset "queue_map_min 0" Sets the min value of tx queue interval |
124 | pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices | |
125 | To select queue 1 of a given device, | |
126 | use queue_map_min=1 and queue_map_max=1 | |
127 | ||
d012827e | 128 | pgset "src_mac_count 1" Sets the number of MACs we'll range through. |
1da177e4 LT |
129 | The 'minimum' MAC is what you set with srcmac. |
130 | ||
131 | pgset "dst_mac_count 1" Sets the number of MACs we'll range through. | |
132 | The 'minimum' MAC is what you set with dstmac. | |
133 | ||
134 | pgset "flag [name]" Set a flag to determine behaviour. Current flags | |
72f8e06f MK |
135 | are: IPSRC_RND # IP source is random (between min/max) |
136 | IPDST_RND # IP destination is random | |
137 | UDPSRC_RND, UDPDST_RND, | |
138 | MACSRC_RND, MACDST_RND | |
139 | TXSIZE_RND, IPV6, | |
f0e82fd0 | 140 | MPLS_RND, VID_RND, SVID_RND |
72f8e06f | 141 | FLOW_SEQ, |
896a7cf8 ED |
142 | QUEUE_MAP_RND # queue map random |
143 | QUEUE_MAP_CPU # queue map mirrors smp_processor_id() | |
72f8e06f MK |
144 | UDPCSUM, |
145 | IPSEC # IPsec encapsulation (needs CONFIG_XFRM) | |
146 | NODE_ALLOC # node specific memory allocation | |
f1f00d8f | 147 | NO_TIMESTAMP # disable timestamping |
896a7cf8 | 148 | |
e5f79d11 | 149 | pgset spi SPI_VALUE Set specific SA used to transform packet. |
1da177e4 LT |
150 | |
151 | pgset "udp_src_min 9" set UDP source port min, If < udp_src_max, then | |
152 | cycle through the port range. | |
153 | ||
154 | pgset "udp_src_max 9" set UDP source port max. | |
155 | pgset "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then | |
156 | cycle through the port range. | |
157 | pgset "udp_dst_max 9" set UDP destination port max. | |
158 | ||
ca6549af SW |
159 | pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example |
160 | outer label=16,middle label=32, | |
161 | inner label=0 (IPv4 NULL)) Note that | |
162 | there must be no spaces between the | |
163 | arguments. Leading zeros are required. | |
164 | Do not set the bottom of stack bit, | |
fa00e7e1 | 165 | that's done automatically. If you do |
ca6549af SW |
166 | set the bottom of stack bit, that |
167 | indicates that you want to randomly | |
168 | generate that address and the flag | |
169 | MPLS_RND will be turned on. You | |
170 | can have any mix of random and fixed | |
171 | labels in the label stack. | |
172 | ||
173 | pgset "mpls 0" turn off mpls (or any invalid argument works too!) | |
174 | ||
f0e82fd0 FF |
175 | pgset "vlan_id 77" set VLAN ID 0-4095 |
176 | pgset "vlan_p 3" set priority bit 0-7 (default 0) | |
177 | pgset "vlan_cfi 0" set canonical format identifier 0-1 (default 0) | |
178 | ||
179 | pgset "svlan_id 22" set SVLAN ID 0-4095 | |
180 | pgset "svlan_p 3" set priority bit 0-7 (default 0) | |
181 | pgset "svlan_cfi 0" set canonical format identifier 0-1 (default 0) | |
182 | ||
183 | pgset "vlan_id 9999" > 4095 remove vlan and svlan tags | |
184 | pgset "svlan 9999" > 4095 remove svlan tag | |
185 | ||
186 | ||
187 | pgset "tos XX" set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00) | |
188 | pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00) | |
189 | ||
1da177e4 LT |
190 | pgset stop aborts injection. Also, ^C aborts generator. |
191 | ||
43d28b65 DT |
192 | pgset "rate 300M" set rate to 300 Mb/s |
193 | pgset "ratep 1000000" set rate to 1Mpps | |
1da177e4 | 194 | |
62f64aed AS |
195 | pgset "xmit_mode netif_receive" RX inject into stack netif_receive_skb() |
196 | Works with "burst" but not with "clone_skb". | |
197 | Default xmit_mode is "start_xmit". | |
198 | ||
7c95a9d9 BH |
199 | Sample scripts |
200 | ============== | |
1da177e4 | 201 | |
7c95a9d9 BH |
202 | A collection of small tutorial scripts for pktgen is in the |
203 | samples/pktgen directory: | |
1da177e4 LT |
204 | |
205 | pktgen.conf-1-1 # 1 CPU 1 dev | |
206 | pktgen.conf-1-2 # 1 CPU 2 dev | |
207 | pktgen.conf-2-1 # 2 CPU's 1 dev | |
208 | pktgen.conf-2-2 # 2 CPU's 2 dev | |
209 | pktgen.conf-1-1-rdos # 1 CPU 1 dev w. route DoS | |
210 | pktgen.conf-1-1-ip6 # 1 CPU 1 dev ipv6 | |
211 | pktgen.conf-1-1-ip6-rdos # 1 CPU 1 dev ipv6 w. route DoS | |
212 | pktgen.conf-1-1-flows # 1 CPU 1 dev multiple flows. | |
213 | ||
ca5b542c BH |
214 | Run in shell: ./pktgen.conf-X-Y |
215 | This does all the setup including sending. | |
1da177e4 LT |
216 | |
217 | ||
218 | Interrupt affinity | |
219 | =================== | |
ca5b542c BH |
220 | Note that when adding devices to a specific CPU it is a good idea to |
221 | also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound | |
222 | to the same CPU. This reduces cache bouncing when freeing skbs. | |
1da177e4 | 223 | |
e5f79d11 FD |
224 | Enable IPsec |
225 | ============ | |
ca5b542c BH |
226 | Default IPsec transformation with ESP encapsulation plus transport mode |
227 | can be enabled by simply setting: | |
e5f79d11 FD |
228 | |
229 | pgset "flag IPSEC" | |
230 | pgset "flows 1" | |
231 | ||
232 | To avoid breaking existing testbed scripts for using AH type and tunnel mode, | |
ca5b542c | 233 | you can use "pgset spi SPI_VALUE" to specify which transformation mode |
e5f79d11 FD |
234 | to employ. |
235 | ||
1da177e4 LT |
236 | |
237 | Current commands and configuration options | |
238 | ========================================== | |
239 | ||
240 | ** Pgcontrol commands: | |
241 | ||
242 | start | |
243 | stop | |
244 | ||
245 | ** Thread commands: | |
246 | ||
247 | add_device | |
248 | rem_device_all | |
1da177e4 LT |
249 | |
250 | ||
251 | ** Device commands: | |
252 | ||
253 | count | |
254 | clone_skb | |
255 | debug | |
256 | ||
257 | frags | |
258 | delay | |
259 | ||
260 | src_mac_count | |
261 | dst_mac_count | |
262 | ||
d012827e | 263 | pkt_size |
1da177e4 LT |
264 | min_pkt_size |
265 | max_pkt_size | |
266 | ||
ca6549af SW |
267 | mpls |
268 | ||
1da177e4 LT |
269 | udp_src_min |
270 | udp_src_max | |
271 | ||
272 | udp_dst_min | |
273 | udp_dst_max | |
274 | ||
275 | flag | |
276 | IPSRC_RND | |
1da177e4 LT |
277 | IPDST_RND |
278 | UDPSRC_RND | |
279 | UDPDST_RND | |
280 | MACSRC_RND | |
281 | MACDST_RND | |
72f8e06f MK |
282 | TXSIZE_RND |
283 | IPV6 | |
284 | MPLS_RND | |
285 | VID_RND | |
286 | SVID_RND | |
287 | FLOW_SEQ | |
288 | QUEUE_MAP_RND | |
289 | QUEUE_MAP_CPU | |
290 | UDPCSUM | |
e5f79d11 | 291 | IPSEC |
72f8e06f | 292 | NODE_ALLOC |
f1f00d8f | 293 | NO_TIMESTAMP |
1da177e4 LT |
294 | |
295 | dst_min | |
296 | dst_max | |
297 | ||
298 | src_min | |
299 | src_max | |
300 | ||
301 | dst_mac | |
302 | src_mac | |
303 | ||
304 | clear_counters | |
305 | ||
306 | dst6 | |
307 | src6 | |
308 | ||
309 | flows | |
310 | flowlen | |
311 | ||
43d28b65 DT |
312 | rate |
313 | ratep | |
314 | ||
62f64aed AS |
315 | xmit_mode <start_xmit|netif_receive> |
316 | ||
317 | ||
1da177e4 LT |
318 | References: |
319 | ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/ | |
320 | ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/ | |
321 | ||
322 | Paper from Linux-Kongress in Erlangen 2004. | |
323 | ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf | |
324 | ||
325 | Thanks to: | |
326 | Grant Grundler for testing on IA-64 and parisc, Harald Welte, Lennert Buytenhek | |
327 | Stephen Hemminger, Andi Kleen, Dave Miller and many others. | |
328 | ||
329 | ||
ca6549af | 330 | Good luck with the linux net-development. |