Commit | Line | Data |
---|---|---|
c9eda125 SY |
1 | Copyright (C) 2015 Freescale Semiconductor Inc. |
2 | ||
3 | DPAA2 (Data Path Acceleration Architecture Gen2) | |
4 | ------------------------------------------------ | |
5 | ||
6 | This document provides an overview of the Freescale DPAA2 architecture | |
7 | and how it is integrated into the Linux kernel. | |
8 | ||
9 | Contents summary | |
10 | -DPAA2 overview | |
11 | -Overview of DPAA2 objects | |
12 | -DPAA2 Linux driver architecture overview | |
13 | -bus driver | |
7dc08183 | 14 | -DPRC driver |
c9eda125 | 15 | -allocator |
7dc08183 | 16 | -DPIO driver |
c9eda125 | 17 | -Ethernet |
7dc08183 | 18 | -MAC |
c9eda125 SY |
19 | |
20 | DPAA2 Overview | |
21 | -------------- | |
22 | ||
23 | DPAA2 is a hardware architecture designed for high-speeed network | |
24 | packet processing. DPAA2 consists of sophisticated mechanisms for | |
25 | processing Ethernet packets, queue management, buffer management, | |
26 | autonomous L2 switching, virtual Ethernet bridging, and accelerator | |
27 | (e.g. crypto) sharing. | |
28 | ||
29 | A DPAA2 hardware component called the Management Complex (or MC) manages the | |
30 | DPAA2 hardware resources. The MC provides an object-based abstraction for | |
31 | software drivers to use the DPAA2 hardware. | |
32 | ||
33 | The MC uses DPAA2 hardware resources such as queues, buffer pools, and | |
34 | network ports to create functional objects/devices such as network | |
35 | interfaces, an L2 switch, or accelerator instances. | |
36 | ||
37 | The MC provides memory-mapped I/O command interfaces (MC portals) | |
38 | which DPAA2 software drivers use to operate on DPAA2 objects: | |
39 | ||
7dc08183 SY |
40 | The diagram below shows an overview of the DPAA2 resource management |
41 | architecture: | |
42 | ||
c9eda125 SY |
43 | +--------------------------------------+ |
44 | | OS | | |
45 | | DPAA2 drivers | | |
46 | | | | | |
47 | +-----------------------------|--------+ | |
48 | | | |
49 | | (create,discover,connect | |
50 | | config,use,destroy) | |
51 | | | |
52 | DPAA2 | | |
53 | +------------------------| mc portal |-+ | |
54 | | | | | |
55 | | +- - - - - - - - - - - - -V- - -+ | | |
56 | | | | | | |
57 | | | Management Complex (MC) | | | |
58 | | | | | | |
59 | | +- - - - - - - - - - - - - - - -+ | | |
60 | | | | |
61 | | Hardware Hardware | | |
62 | | Resources Objects | | |
63 | | --------- ------- | | |
64 | | -queues -DPRC | | |
65 | | -buffer pools -DPMCP | | |
66 | | -Eth MACs/ports -DPIO | | |
67 | | -network interface -DPNI | | |
68 | | profiles -DPMAC | | |
69 | | -queue portals -DPBP | | |
70 | | -MC portals ... | | |
71 | | ... | | |
72 | | | | |
73 | +--------------------------------------+ | |
74 | ||
75 | The MC mediates operations such as create, discover, | |
76 | connect, configuration, and destroy. Fast-path operations | |
77 | on data, such as packet transmit/receive, are not mediated by | |
78 | the MC and are done directly using memory mapped regions in | |
79 | DPIO objects. | |
80 | ||
81 | Overview of DPAA2 Objects | |
82 | ------------------------- | |
7dc08183 SY |
83 | The section provides a brief overview of some key DPAA2 objects. |
84 | A simple scenario is described illustrating the objects involved | |
85 | in creating a network interfaces. | |
c9eda125 SY |
86 | |
87 | -DPRC (Datapath Resource Container) | |
88 | ||
7dc08183 | 89 | A DPRC is a container object that holds all the other |
c9eda125 SY |
90 | types of DPAA2 objects. In the example diagram below there |
91 | are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) | |
92 | in the container. | |
93 | ||
94 | +---------------------------------------------------------+ | |
95 | | DPRC | | |
96 | | | | |
97 | | +-------+ +-------+ +-------+ +-------+ +-------+ | | |
98 | | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | | |
99 | | +-------+ +-------+ +-------+ +---+---+ +---+---+ | | |
100 | | | DPMCP | | DPIO | | | |
101 | | +-------+ +-------+ | | |
102 | | | DPMCP | | | |
103 | | +-------+ | | |
104 | | | | |
105 | +---------------------------------------------------------+ | |
106 | ||
7dc08183 SY |
107 | From the point of view of an OS, a DPRC behaves similar to a plug and |
108 | play bus, like PCI. DPRC commands can be used to enumerate the contents | |
109 | of the DPRC, discover the hardware objects present (including mappable | |
110 | regions and interrupts). | |
c9eda125 | 111 | |
7dc08183 | 112 | DPRC.1 (bus) |
c9eda125 SY |
113 | | |
114 | +--+--------+-------+-------+-------+ | |
115 | | | | | | | |
7dc08183 SY |
116 | DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1 |
117 | DPMCP.2 DPIO.2 | |
118 | DPMCP.3 | |
c9eda125 SY |
119 | |
120 | Hardware objects can be created and destroyed dynamically, providing | |
121 | the ability to hot plug/unplug objects in and out of the DPRC. | |
122 | ||
7dc08183 | 123 | A DPRC has a mappable MMIO region (an MC portal) that can be used |
c9eda125 SY |
124 | to send MC commands. It has an interrupt for status events (like |
125 | hotplug). | |
126 | ||
127 | All objects in a container share the same hardware "isolation context". | |
128 | This means that with respect to an IOMMU the isolation granularity | |
129 | is at the DPRC (container) level, not at the individual object | |
130 | level. | |
131 | ||
132 | DPRCs can be defined statically and populated with objects | |
133 | via a config file passed to the MC when firmware starts | |
134 | it. There is also a Linux user space tool called "restool" | |
135 | that can be used to create/destroy containers and objects | |
136 | dynamically. | |
137 | ||
138 | -DPAA2 Objects for an Ethernet Network Interface | |
139 | ||
140 | A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX | |
141 | queuing mechanisms, configuration mechanisms, buffer management, | |
142 | physical ports, and interrupts. DPAA2 uses a more granular approach | |
7dc08183 SY |
143 | utilizing multiple hardware objects. Each object provides specialized |
144 | functions. Groups of these objects are used by software to provide | |
145 | Ethernet network interface functionality. This approach provides | |
146 | efficient use of finite hardware resources, flexibility, and | |
147 | performance advantages. | |
c9eda125 SY |
148 | |
149 | The diagram below shows the objects needed for a simple | |
150 | network interface configuration on a system with 2 CPUs. | |
151 | ||
152 | +---+---+ +---+---+ | |
153 | CPU0 CPU1 | |
154 | +---+---+ +---+---+ | |
155 | | | | |
156 | +---+---+ +---+---+ | |
157 | DPIO DPIO | |
158 | +---+---+ +---+---+ | |
159 | \ / | |
160 | \ / | |
161 | \ / | |
162 | +---+---+ | |
163 | DPNI --- DPBP,DPMCP | |
164 | +---+---+ | |
165 | | | |
166 | | | |
167 | +---+---+ | |
168 | DPMAC | |
169 | +---+---+ | |
170 | | | |
171 | port/PHY | |
172 | ||
173 | Below the objects are described. For each object a brief description | |
174 | is provided along with a summary of the kinds of operations the object | |
7dc08183 SY |
175 | supports and a summary of key resources of the object (MMIO regions |
176 | and IRQs). | |
c9eda125 SY |
177 | |
178 | -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a | |
179 | hardware device that connects to an Ethernet PHY and allows | |
180 | physical transmission and reception of Ethernet frames. | |
7dc08183 SY |
181 | -MMIO regions: none |
182 | -IRQs: DPNI link change | |
c9eda125 | 183 | -commands: set link up/down, link config, get stats, |
7dc08183 | 184 | IRQ config, enable, reset |
c9eda125 SY |
185 | |
186 | -DPNI (Datapath Network Interface): contains TX/RX queues, | |
7dc08183 SY |
187 | network interface configuration, and RX buffer pool configuration |
188 | mechanisms. The TX/RX queues are in memory and are identified by | |
189 | queue number. | |
190 | -MMIO regions: none | |
191 | -IRQs: link state | |
c9eda125 | 192 | -commands: port config, offload config, queue config, |
7dc08183 | 193 | parse/classify config, IRQ config, enable, reset |
c9eda125 SY |
194 | |
195 | -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue | |
7dc08183 SY |
196 | packets and do hardware buffer pool management operations. The DPAA2 |
197 | architecture separates the mechanism to access queues (the DPIO object) | |
198 | from the queues themselves. The DPIO provides an MMIO interface to | |
199 | enqueue/dequeue packets. To enqueue something a descriptor is written | |
200 | to the DPIO MMIO region, which includes the target queue number. | |
201 | There will typically be one DPIO assigned to each CPU. This allows all | |
202 | CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are | |
203 | expected to be shared by different DPAA2 drivers. | |
204 | -MMIO regions: queue operations, buffer management | |
205 | -IRQs: data availability, congestion notification, buffer | |
c9eda125 | 206 | pool depletion |
7dc08183 | 207 | -commands: IRQ config, enable, reset |
c9eda125 SY |
208 | |
209 | -DPBP (Datapath Buffer Pool): represents a hardware buffer | |
210 | pool. | |
7dc08183 SY |
211 | -MMIO regions: none |
212 | -IRQs: none | |
c9eda125 SY |
213 | -commands: enable, reset |
214 | ||
215 | -DPMCP (Datapath MC Portal): provides an MC command portal. | |
216 | Used by drivers to send commands to the MC to manage | |
217 | objects. | |
7dc08183 SY |
218 | -MMIO regions: MC command portal |
219 | -IRQs: command completion | |
220 | -commands: IRQ config, enable, reset | |
c9eda125 SY |
221 | |
222 | Object Connections | |
223 | ------------------ | |
224 | Some objects have explicit relationships that must | |
225 | be configured: | |
226 | ||
227 | -DPNI <--> DPMAC | |
228 | -DPNI <--> DPNI | |
229 | -DPNI <--> L2-switch-port | |
230 | A DPNI must be connected to something such as a DPMAC, | |
231 | another DPNI, or L2 switch port. The DPNI connection | |
232 | is made via a DPRC command. | |
233 | ||
234 | +-------+ +-------+ | |
235 | | DPNI | | DPMAC | | |
236 | +---+---+ +---+---+ | |
237 | | | | |
238 | +==========+ | |
239 | ||
240 | -DPNI <--> DPBP | |
241 | A network interface requires a 'buffer pool' (DPBP | |
242 | object) which provides a list of pointers to memory | |
243 | where received Ethernet data is to be copied. The | |
244 | Ethernet driver configures the DPBPs associated with | |
245 | the network interface. | |
246 | ||
247 | Interrupts | |
248 | ---------- | |
249 | All interrupts generated by DPAA2 objects are message | |
250 | interrupts. At the hardware level message interrupts | |
251 | generated by devices will normally have 3 components-- | |
252 | 1) a non-spoofable 'device-id' expressed on the hardware | |
253 | bus, 2) an address, 3) a data value. | |
254 | ||
255 | In the case of DPAA2 devices/objects, all objects in the | |
256 | same container/DPRC share the same 'device-id'. | |
257 | For ARM-based SoC this is the same as the stream ID. | |
258 | ||
259 | ||
260 | DPAA2 Linux Driver Overview | |
261 | --------------------------- | |
262 | ||
263 | This section provides an overview of the Linux kernel drivers for | |
264 | DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" | |
265 | drivers and 2) functional object drivers (such as Ethernet). | |
266 | ||
267 | As described previously, a DPRC is a container that holds the other | |
268 | types of DPAA2 objects. It is functionally similar to a plug-and-play | |
269 | bus controller. | |
270 | ||
271 | Each object in the DPRC is a Linux "device" and is bound to a driver. | |
272 | The diagram below shows the Linux drivers involved in a networking | |
273 | scenario and the objects bound to each driver. A brief description | |
274 | of each driver follows. | |
275 | ||
276 | +------------+ | |
277 | | OS Network | | |
278 | | Stack | | |
279 | +------------+ +------------+ | |
280 | | Allocator |. . . . . . . | Ethernet | | |
7dc08183 | 281 | |(DPMCP,DPBP)| | (DPNI) | |
c9eda125 SY |
282 | +-.----------+ +---+---+----+ |
283 | . . ^ | | |
284 | . . <data avail, | |<enqueue, | |
285 | . . tx confirm> | | dequeue> | |
286 | +-------------+ . | | | |
287 | | DPRC driver | . +---+---V----+ +---------+ | |
7dc08183 SY |
288 | | (DPRC) | . . . . . .| DPIO driver| | MAC | |
289 | +----------+--+ | (DPIO) | | (DPMAC) | | |
c9eda125 SY |
290 | | +------+-----+ +-----+---+ |
291 | |<dev add/remove> | | | |
292 | | | | | |
293 | +----+--------------+ | +--+---+ | |
7dc08183 | 294 | | MC-bus driver | | | PHY | |
c9eda125 | 295 | | | | |driver| |
7dc08183 | 296 | | /soc/fsl-mc | | +--+---+ |
c9eda125 SY |
297 | +-------------------+ | | |
298 | | | | |
299 | ================================ HARDWARE =========|=================|====== | |
300 | DPIO | | |
301 | | | | |
302 | DPNI---DPBP | | |
303 | | | | |
304 | DPMAC | | |
305 | | | | |
306 | PHY ---------------+ | |
307 | ===================================================|======================== | |
308 | ||
309 | A brief description of each driver is provided below. | |
310 | ||
7dc08183 | 311 | MC-bus driver |
c9eda125 | 312 | ------------- |
7dc08183 SY |
313 | The MC-bus driver is a platform driver and is probed from a |
314 | node in the device tree (compatible "fsl,qoriq-mc") passed in by boot | |
315 | firmware. It is responsible for bootstrapping the DPAA2 kernel | |
316 | infrastructure. | |
c9eda125 SY |
317 | Key functions include: |
318 | -registering a new bus type named "fsl-mc" with the kernel, | |
319 | and implementing bus call-backs (e.g. match/uevent/dev_groups) | |
7dc08183 | 320 | -implementing APIs for DPAA2 driver registration and for device |
c9eda125 | 321 | add/remove |
7dc08183 SY |
322 | -creates an MSI IRQ domain |
323 | -doing a 'device add' to expose the 'root' DPRC, in turn triggering | |
324 | a bind of the root DPRC to the DPRC driver | |
c9eda125 SY |
325 | |
326 | DPRC driver | |
327 | ----------- | |
7dc08183 | 328 | The DPRC driver is bound to DPRC objects and does runtime management |
c9eda125 | 329 | of a bus instance. It performs the initial bus scan of the DPRC |
7dc08183 SY |
330 | and handles interrupts for container events such as hot plug by |
331 | re-scanning the DPRC. | |
c9eda125 SY |
332 | |
333 | Allocator | |
334 | ---------- | |
335 | Certain objects such as DPMCP and DPBP are generic and fungible, | |
336 | and are intended to be used by other drivers. For example, | |
337 | the DPAA2 Ethernet driver needs: | |
338 | -DPMCPs to send MC commands, to configure network interfaces | |
339 | -DPBPs for network buffer pools | |
340 | ||
341 | The allocator driver registers for these allocatable object types | |
342 | and those objects are bound to the allocator when the bus is probed. | |
343 | The allocator maintains a pool of objects that are available for | |
344 | allocation by other DPAA2 drivers. | |
345 | ||
346 | DPIO driver | |
347 | ----------- | |
348 | The DPIO driver is bound to DPIO objects and provides services that allow | |
7dc08183 SY |
349 | other drivers such as the Ethernet driver to enqueue and dequeue data for |
350 | their respective objects. | |
c9eda125 SY |
351 | Key services include: |
352 | -data availability notifications | |
353 | -hardware queuing operations (enqueue and dequeue of data) | |
354 | -hardware buffer pool management | |
355 | ||
7dc08183 SY |
356 | To transmit a packet the Ethernet driver puts data on a queue and |
357 | invokes a DPIO API. For receive, the Ethernet driver registers | |
358 | a data availability notification callback. To dequeue a packet | |
359 | a DPIO API is used. | |
360 | ||
c9eda125 | 361 | There is typically one DPIO object per physical CPU for optimum |
7dc08183 | 362 | performance, allowing different CPUs to simultaneously enqueue |
c9eda125 SY |
363 | and dequeue data. |
364 | ||
365 | The DPIO driver operates on behalf of all DPAA2 drivers | |
366 | active in the kernel-- Ethernet, crypto, compression, | |
367 | etc. | |
368 | ||
369 | Ethernet | |
370 | -------- | |
371 | The Ethernet driver is bound to a DPNI and implements the kernel | |
372 | interfaces needed to connect the DPAA2 network interface to | |
373 | the network stack. | |
374 | ||
375 | Each DPNI corresponds to a Linux network interface. | |
376 | ||
377 | MAC driver | |
378 | ---------- | |
379 | An Ethernet PHY is an off-chip, board specific component and is managed | |
380 | by the appropriate PHY driver via an mdio bus. The MAC driver | |
381 | plays a role of being a proxy between the PHY driver and the | |
382 | MC. It does this proxy via the MC commands to a DPMAC object. | |
7dc08183 SY |
383 | If the PHY driver signals a link change, the MAC driver notifies |
384 | the MC via a DPMAC command. If a network interface is brought | |
385 | up or down, the MC notifies the DPMAC driver via an interrupt and | |
386 | the driver can take appropriate action. |