net: vrf: Address comments from last documentation update
[deliverable/linux.git] / Documentation / networking / vrf.txt
1 Virtual Routing and Forwarding (VRF)
2 ====================================
3 The VRF device combined with ip rules provides the ability to create virtual
4 routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
5 Linux network stack. One use case is the multi-tenancy problem where each
6 tenant has their own unique routing tables and in the very least need
7 different default gateways.
8
9 Processes can be "VRF aware" by binding a socket to the VRF device. Packets
10 through the socket then use the routing table associated with the VRF
11 device. An important feature of the VRF device implementation is that it
12 impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
13 (ie., they do not need to be run in each VRF). The design also allows
14 the use of higher priority ip rules (Policy Based Routing, PBR) to take
15 precedence over the VRF device rules directing specific traffic as desired.
16
17 In addition, VRF devices allow VRFs to be nested within namespaces. For
18 example network namespaces provide separation of network interfaces at the
19 device layer, VLANs on the interfaces within a namespace provide L2 separation
20 and then VRF devices provide L3 separation.
21
22 Design
23 ------
24 A VRF device is created with an associated route table. Network interfaces
25 are then enslaved to a VRF device:
26
27 +-----------------------------+
28 | vrf-blue | ===> route table 10
29 +-----------------------------+
30 | | |
31 +------+ +------+ +-------------+
32 | eth1 | | eth2 | ... | bond1 |
33 +------+ +------+ +-------------+
34 | |
35 +------+ +------+
36 | eth8 | | eth9 |
37 +------+ +------+
38
39 Packets received on an enslaved device and are switched to the VRF device
40 in the IPv4 and IPv6 processing stacks giving the impression that packets
41 flow through the VRF device. Similarly on egress routing rules are used to
42 send packets to the VRF device driver before getting sent out the actual
43 interface. This allows tcpdump on a VRF device to capture all packets into
44 and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be
45 applied using the VRF device to specify rules that apply to the VRF domain
46 as a whole.
47
48 [1] Packets in the forwarded state do not flow through the device, so those
49 packets are not seen by tcpdump. Will revisit this limitation in a
50 future release.
51
52 [2] Iptables on ingress supports PREROUTING with skb->dev set to the real
53 ingress device and both INPUT and PREROUTING rules with skb->dev set to
54 the VRF device. For egress POSTROUTING and OUTPUT rules can be written
55 using either the VRF device or real egress device.
56
57 Setup
58 -----
59 1. VRF device is created with an association to a FIB table.
60 e.g, ip link add vrf-blue type vrf table 10
61 ip link set dev vrf-blue up
62
63 2. An l3mdev FIB rule directs lookups to the table associated with the device.
64 A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
65 l3mdev rule for IPv4 and IPv6 when the first device is created with a
66 default preference of 1000. Users may delete the rule if desired and add
67 with a different priority or install per-VRF rules.
68
69 Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:
70 ip ru add oif vrf-blue table 10
71 ip ru add iif vrf-blue table 10
72
73 3. Set the default route for the table (and hence default route for the VRF).
74 ip route add table 10 unreachable default
75
76 4. Enslave L3 interfaces to a VRF device.
77 ip link set dev eth1 master vrf-blue
78
79 Local and connected routes for enslaved devices are automatically moved to
80 the table associated with VRF device. Any additional routes depending on
81 the enslaved device are dropped and will need to be reinserted to the VRF
82 FIB table following the enslavement.
83
84 The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
85 addresses as VRF enslavement changes.
86 sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
87
88 5. Additional VRF routes are added to associated table.
89 ip route add table 10 ...
90
91
92 Applications
93 ------------
94 Applications that are to work within a VRF need to bind their socket to the
95 VRF device:
96
97 setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
98
99 or to specify the output device using cmsg and IP_PKTINFO.
100
101 TCP services running in the default VRF context (ie., not bound to any VRF
102 device) can work across all VRF domains by enabling the tcp_l3mdev_accept
103 sysctl option:
104 sysctl -w net.ipv4.tcp_l3mdev_accept=1
105
106 netfilter rules on the VRF device can be used to limit access to services
107 running in the default VRF context as well.
108
109 The default VRF does not have limited scope with respect to port bindings.
110 That is, if a process does a wildcard bind to a port in the default VRF it
111 owns the port across all VRF domains within the network namespace.
112
113 ################################################################################
114
115 Using iproute2 for VRFs
116 =======================
117 iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
118 section lists both commands where appropriate -- with the vrf keyword and the
119 older form without it.
120
121 1. Create a VRF
122
123 To instantiate a VRF device and associate it with a table:
124 $ ip link add dev NAME type vrf table ID
125
126 As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
127 covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
128 device create.
129
130 2. List VRFs
131
132 To list VRFs that have been created:
133 $ ip [-d] link show type vrf
134 NOTE: The -d option is needed to show the table id
135
136 For example:
137 $ ip -d link show type vrf
138 11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
139 link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
140 vrf table 1 addrgenmode eui64
141 12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
142 link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
143 vrf table 10 addrgenmode eui64
144 13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
145 link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
146 vrf table 66 addrgenmode eui64
147 14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
148 link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
149 vrf table 81 addrgenmode eui64
150
151
152 Or in brief output:
153
154 $ ip -br link show type vrf
155 mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
156 red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
157 blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
158 green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
159
160
161 3. Assign a Network Interface to a VRF
162
163 Network interfaces are assigned to a VRF by enslaving the netdevice to a
164 VRF device:
165 $ ip link set dev NAME master NAME
166
167 On enslavement connected and local routes are automatically moved to the
168 table associated with the VRF device.
169
170 For example:
171 $ ip link set dev eth0 master mgmt
172
173
174 4. Show Devices Assigned to a VRF
175
176 To show devices that have been assigned to a specific VRF add the master
177 option to the ip command:
178 $ ip link show vrf NAME
179 $ ip link show master NAME
180
181 For example:
182 $ ip link show vrf red
183 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
184 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
185 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
186 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
187 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
188 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
189
190
191 Or using the brief output:
192 $ ip -br link show vrf red
193 eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
194 eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
195 eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST>
196
197
198 5. Show Neighbor Entries for a VRF
199
200 To list neighbor entries associated with devices enslaved to a VRF device
201 add the master option to the ip command:
202 $ ip [-6] neigh show vrf NAME
203 $ ip [-6] neigh show master NAME
204
205 For example:
206 $ ip neigh show vrf red
207 10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
208 10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
209
210 $ ip -6 neigh show vrf red
211 2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
212
213
214 6. Show Addresses for a VRF
215
216 To show addresses for interfaces associated with a VRF add the master
217 option to the ip command:
218 $ ip addr show vrf NAME
219 $ ip addr show master NAME
220
221 For example:
222 $ ip addr show vrf red
223 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
224 link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
225 inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
226 valid_lft forever preferred_lft forever
227 inet6 2002:1::2/120 scope global
228 valid_lft forever preferred_lft forever
229 inet6 fe80::ff:fe00:202/64 scope link
230 valid_lft forever preferred_lft forever
231 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
232 link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
233 inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
234 valid_lft forever preferred_lft forever
235 inet6 2002:2::2/120 scope global
236 valid_lft forever preferred_lft forever
237 inet6 fe80::ff:fe00:203/64 scope link
238 valid_lft forever preferred_lft forever
239 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
240 link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
241
242 Or in brief format:
243 $ ip -br addr show vrf red
244 eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
245 eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
246 eth5 DOWN
247
248
249 7. Show Routes for a VRF
250
251 To show routes for a VRF use the ip command to display the table associated
252 with the VRF device:
253 $ ip [-6] route show vrf NAME
254 $ ip [-6] route show table ID
255
256 For example:
257 $ ip route show vrf red
258 prohibit default
259 broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2
260 10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2
261 local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2
262 broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2
263 broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2
264 10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2
265 local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2
266 broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2
267
268 $ ip -6 route show vrf red
269 local 2002:1:: dev lo proto none metric 0 pref medium
270 local 2002:1::2 dev lo proto none metric 0 pref medium
271 2002:1::/120 dev eth1 proto kernel metric 256 pref medium
272 local 2002:2:: dev lo proto none metric 0 pref medium
273 local 2002:2::2 dev lo proto none metric 0 pref medium
274 2002:2::/120 dev eth2 proto kernel metric 256 pref medium
275 local fe80:: dev lo proto none metric 0 pref medium
276 local fe80:: dev lo proto none metric 0 pref medium
277 local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium
278 local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium
279 fe80::/64 dev eth1 proto kernel metric 256 pref medium
280 fe80::/64 dev eth2 proto kernel metric 256 pref medium
281 ff00::/8 dev red metric 256 pref medium
282 ff00::/8 dev eth1 metric 256 pref medium
283 ff00::/8 dev eth2 metric 256 pref medium
284
285
286 8. Route Lookup for a VRF
287
288 A test route lookup can be done for a VRF:
289 $ ip [-6] route get vrf NAME ADDRESS
290 $ ip [-6] route get oif NAME ADDRESS
291
292 For example:
293 $ ip route get 10.2.1.40 vrf red
294 10.2.1.40 dev eth1 table red src 10.2.1.2
295 cache
296
297 $ ip -6 route get 2002:1::32 vrf red
298 2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium
299
300
301 9. Removing Network Interface from a VRF
302
303 Network interfaces are removed from a VRF by breaking the enslavement to
304 the VRF device:
305 $ ip link set dev NAME nomaster
306
307 Connected routes are moved back to the default table and local entries are
308 moved to the local table.
309
310 For example:
311 $ ip link set dev eth0 nomaster
312
313 --------------------------------------------------------------------------------
314
315 Commands used in this example:
316
317 cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
318 1 mgmt
319 10 red
320 66 blue
321 81 green
322 EOF
323
324 function vrf_create
325 {
326 VRF=$1
327 TBID=$2
328
329 # create VRF device
330 ip link add ${VRF} type vrf table ${TBID}
331
332 if [ "${VRF}" != "mgmt" ]; then
333 ip route add table ${TBID} unreachable default
334 fi
335 ip link set dev ${VRF} up
336 }
337
338 vrf_create mgmt 1
339 ip link set dev eth0 master mgmt
340
341 vrf_create red 10
342 ip link set dev eth1 master red
343 ip link set dev eth2 master red
344 ip link set dev eth5 master red
345
346 vrf_create blue 66
347 ip link set dev eth3 master blue
348
349 vrf_create green 81
350 ip link set dev eth4 master green
351
352
353 Interface addresses from /etc/network/interfaces:
354 auto eth0
355 iface eth0 inet static
356 address 10.0.0.2
357 netmask 255.255.255.0
358 gateway 10.0.0.254
359
360 iface eth0 inet6 static
361 address 2000:1::2
362 netmask 120
363
364 auto eth1
365 iface eth1 inet static
366 address 10.2.1.2
367 netmask 255.255.255.0
368
369 iface eth1 inet6 static
370 address 2002:1::2
371 netmask 120
372
373 auto eth2
374 iface eth2 inet static
375 address 10.2.2.2
376 netmask 255.255.255.0
377
378 iface eth2 inet6 static
379 address 2002:2::2
380 netmask 120
381
382 auto eth3
383 iface eth3 inet static
384 address 10.2.3.2
385 netmask 255.255.255.0
386
387 iface eth3 inet6 static
388 address 2002:3::2
389 netmask 120
390
391 auto eth4
392 iface eth4 inet static
393 address 10.2.4.2
394 netmask 255.255.255.0
395
396 iface eth4 inet6 static
397 address 2002:4::2
398 netmask 120
This page took 0.04691 seconds and 5 git commands to generate.