Commit | Line | Data |
---|---|---|
17926a79 DH |
1 | ====================== |
2 | RxRPC NETWORK PROTOCOL | |
3 | ====================== | |
4 | ||
5 | The RxRPC protocol driver provides a reliable two-phase transport on top of UDP | |
6 | that can be used to perform RxRPC remote operations. This is done over sockets | |
7 | of AF_RXRPC family, using sendmsg() and recvmsg() with control data to send and | |
8 | receive data, aborts and errors. | |
9 | ||
10 | Contents of this document: | |
11 | ||
12 | (*) Overview. | |
13 | ||
14 | (*) RxRPC protocol summary. | |
15 | ||
16 | (*) AF_RXRPC driver model. | |
17 | ||
18 | (*) Control messages. | |
19 | ||
20 | (*) Socket options. | |
21 | ||
22 | (*) Security. | |
23 | ||
24 | (*) Example client usage. | |
25 | ||
26 | (*) Example server usage. | |
27 | ||
651350d1 DH |
28 | (*) AF_RXRPC kernel interface. |
29 | ||
5873c083 DH |
30 | (*) Configurable parameters. |
31 | ||
17926a79 DH |
32 | |
33 | ======== | |
34 | OVERVIEW | |
35 | ======== | |
36 | ||
37 | RxRPC is a two-layer protocol. There is a session layer which provides | |
38 | reliable virtual connections using UDP over IPv4 (or IPv6) as the transport | |
39 | layer, but implements a real network protocol; and there's the presentation | |
40 | layer which renders structured data to binary blobs and back again using XDR | |
41 | (as does SunRPC): | |
42 | ||
43 | +-------------+ | |
44 | | Application | | |
45 | +-------------+ | |
46 | | XDR | Presentation | |
47 | +-------------+ | |
48 | | RxRPC | Session | |
49 | +-------------+ | |
50 | | UDP | Transport | |
51 | +-------------+ | |
52 | ||
53 | ||
54 | AF_RXRPC provides: | |
55 | ||
56 | (1) Part of an RxRPC facility for both kernel and userspace applications by | |
57 | making the session part of it a Linux network protocol (AF_RXRPC). | |
58 | ||
59 | (2) A two-phase protocol. The client transmits a blob (the request) and then | |
60 | receives a blob (the reply), and the server receives the request and then | |
61 | transmits the reply. | |
62 | ||
63 | (3) Retention of the reusable bits of the transport system set up for one call | |
64 | to speed up subsequent calls. | |
65 | ||
66 | (4) A secure protocol, using the Linux kernel's key retention facility to | |
67 | manage security on the client end. The server end must of necessity be | |
68 | more active in security negotiations. | |
69 | ||
70 | AF_RXRPC does not provide XDR marshalling/presentation facilities. That is | |
71 | left to the application. AF_RXRPC only deals in blobs. Even the operation ID | |
72 | is just the first four bytes of the request blob, and as such is beyond the | |
73 | kernel's interest. | |
74 | ||
75 | ||
76 | Sockets of AF_RXRPC family are: | |
77 | ||
78 | (1) created as type SOCK_DGRAM; | |
79 | ||
80 | (2) provided with a protocol of the type of underlying transport they're going | |
81 | to use - currently only PF_INET is supported. | |
82 | ||
83 | ||
84 | The Andrew File System (AFS) is an example of an application that uses this and | |
85 | that has both kernel (filesystem) and userspace (utility) components. | |
86 | ||
87 | ||
88 | ====================== | |
89 | RXRPC PROTOCOL SUMMARY | |
90 | ====================== | |
91 | ||
92 | An overview of the RxRPC protocol: | |
93 | ||
94 | (*) RxRPC sits on top of another networking protocol (UDP is the only option | |
95 | currently), and uses this to provide network transport. UDP ports, for | |
96 | example, provide transport endpoints. | |
97 | ||
98 | (*) RxRPC supports multiple virtual "connections" from any given transport | |
99 | endpoint, thus allowing the endpoints to be shared, even to the same | |
100 | remote endpoint. | |
101 | ||
102 | (*) Each connection goes to a particular "service". A connection may not go | |
103 | to multiple services. A service may be considered the RxRPC equivalent of | |
104 | a port number. AF_RXRPC permits multiple services to share an endpoint. | |
105 | ||
106 | (*) Client-originating packets are marked, thus a transport endpoint can be | |
107 | shared between client and server connections (connections have a | |
108 | direction). | |
109 | ||
110 | (*) Up to a billion connections may be supported concurrently between one | |
111 | local transport endpoint and one service on one remote endpoint. An RxRPC | |
112 | connection is described by seven numbers: | |
113 | ||
114 | Local address } | |
115 | Local port } Transport (UDP) address | |
116 | Remote address } | |
117 | Remote port } | |
118 | Direction | |
119 | Connection ID | |
120 | Service ID | |
121 | ||
122 | (*) Each RxRPC operation is a "call". A connection may make up to four | |
123 | billion calls, but only up to four calls may be in progress on a | |
124 | connection at any one time. | |
125 | ||
126 | (*) Calls are two-phase and asymmetric: the client sends its request data, | |
127 | which the service receives; then the service transmits the reply data | |
128 | which the client receives. | |
129 | ||
130 | (*) The data blobs are of indefinite size, the end of a phase is marked with a | |
131 | flag in the packet. The number of packets of data making up one blob may | |
132 | not exceed 4 billion, however, as this would cause the sequence number to | |
133 | wrap. | |
134 | ||
135 | (*) The first four bytes of the request data are the service operation ID. | |
136 | ||
137 | (*) Security is negotiated on a per-connection basis. The connection is | |
138 | initiated by the first data packet on it arriving. If security is | |
139 | requested, the server then issues a "challenge" and then the client | |
140 | replies with a "response". If the response is successful, the security is | |
141 | set for the lifetime of that connection, and all subsequent calls made | |
142 | upon it use that same security. In the event that the server lets a | |
143 | connection lapse before the client, the security will be renegotiated if | |
144 | the client uses the connection again. | |
145 | ||
146 | (*) Calls use ACK packets to handle reliability. Data packets are also | |
147 | explicitly sequenced per call. | |
148 | ||
c17cb8b5 | 149 | (*) There are two types of positive acknowledgment: hard-ACKs and soft-ACKs. |
17926a79 DH |
150 | A hard-ACK indicates to the far side that all the data received to a point |
151 | has been received and processed; a soft-ACK indicates that the data has | |
152 | been received but may yet be discarded and re-requested. The sender may | |
153 | not discard any transmittable packets until they've been hard-ACK'd. | |
154 | ||
155 | (*) Reception of a reply data packet implicitly hard-ACK's all the data | |
156 | packets that make up the request. | |
157 | ||
158 | (*) An call is complete when the request has been sent, the reply has been | |
159 | received and the final hard-ACK on the last packet of the reply has | |
160 | reached the server. | |
161 | ||
162 | (*) An call may be aborted by either end at any time up to its completion. | |
163 | ||
164 | ||
165 | ===================== | |
166 | AF_RXRPC DRIVER MODEL | |
167 | ===================== | |
168 | ||
169 | About the AF_RXRPC driver: | |
170 | ||
171 | (*) The AF_RXRPC protocol transparently uses internal sockets of the transport | |
172 | protocol to represent transport endpoints. | |
173 | ||
174 | (*) AF_RXRPC sockets map onto RxRPC connection bundles. Actual RxRPC | |
175 | connections are handled transparently. One client socket may be used to | |
176 | make multiple simultaneous calls to the same service. One server socket | |
177 | may handle calls from many clients. | |
178 | ||
179 | (*) Additional parallel client connections will be initiated to support extra | |
180 | concurrent calls, up to a tunable limit. | |
181 | ||
182 | (*) Each connection is retained for a certain amount of time [tunable] after | |
183 | the last call currently using it has completed in case a new call is made | |
184 | that could reuse it. | |
185 | ||
186 | (*) Each internal UDP socket is retained [tunable] for a certain amount of | |
187 | time [tunable] after the last connection using it discarded, in case a new | |
188 | connection is made that could use it. | |
189 | ||
190 | (*) A client-side connection is only shared between calls if they have have | |
191 | the same key struct describing their security (and assuming the calls | |
192 | would otherwise share the connection). Non-secured calls would also be | |
193 | able to share connections with each other. | |
194 | ||
195 | (*) A server-side connection is shared if the client says it is. | |
196 | ||
197 | (*) ACK'ing is handled by the protocol driver automatically, including ping | |
198 | replying. | |
199 | ||
200 | (*) SO_KEEPALIVE automatically pings the other side to keep the connection | |
201 | alive [TODO]. | |
202 | ||
203 | (*) If an ICMP error is received, all calls affected by that error will be | |
204 | aborted with an appropriate network error passed through recvmsg(). | |
205 | ||
206 | ||
207 | Interaction with the user of the RxRPC socket: | |
208 | ||
209 | (*) A socket is made into a server socket by binding an address with a | |
210 | non-zero service ID. | |
211 | ||
212 | (*) In the client, sending a request is achieved with one or more sendmsgs, | |
213 | followed by the reply being received with one or more recvmsgs. | |
214 | ||
215 | (*) The first sendmsg for a request to be sent from a client contains a tag to | |
216 | be used in all other sendmsgs or recvmsgs associated with that call. The | |
217 | tag is carried in the control data. | |
218 | ||
219 | (*) connect() is used to supply a default destination address for a client | |
220 | socket. This may be overridden by supplying an alternate address to the | |
221 | first sendmsg() of a call (struct msghdr::msg_name). | |
222 | ||
223 | (*) If connect() is called on an unbound client, a random local port will | |
224 | bound before the operation takes place. | |
225 | ||
226 | (*) A server socket may also be used to make client calls. To do this, the | |
227 | first sendmsg() of the call must specify the target address. The server's | |
228 | transport endpoint is used to send the packets. | |
229 | ||
230 | (*) Once the application has received the last message associated with a call, | |
231 | the tag is guaranteed not to be seen again, and so it can be used to pin | |
232 | client resources. A new call can then be initiated with the same tag | |
233 | without fear of interference. | |
234 | ||
235 | (*) In the server, a request is received with one or more recvmsgs, then the | |
236 | the reply is transmitted with one or more sendmsgs, and then the final ACK | |
237 | is received with a last recvmsg. | |
238 | ||
239 | (*) When sending data for a call, sendmsg is given MSG_MORE if there's more | |
240 | data to come on that call. | |
241 | ||
242 | (*) When receiving data for a call, recvmsg flags MSG_MORE if there's more | |
243 | data to come for that call. | |
244 | ||
245 | (*) When receiving data or messages for a call, MSG_EOR is flagged by recvmsg | |
246 | to indicate the terminal message for that call. | |
247 | ||
248 | (*) A call may be aborted by adding an abort control message to the control | |
249 | data. Issuing an abort terminates the kernel's use of that call's tag. | |
250 | Any messages waiting in the receive queue for that call will be discarded. | |
251 | ||
252 | (*) Aborts, busy notifications and challenge packets are delivered by recvmsg, | |
253 | and control data messages will be set to indicate the context. Receiving | |
254 | an abort or a busy message terminates the kernel's use of that call's tag. | |
255 | ||
256 | (*) The control data part of the msghdr struct is used for a number of things: | |
257 | ||
258 | (*) The tag of the intended or affected call. | |
259 | ||
260 | (*) Sending or receiving errors, aborts and busy notifications. | |
261 | ||
262 | (*) Notifications of incoming calls. | |
263 | ||
264 | (*) Sending debug requests and receiving debug replies [TODO]. | |
265 | ||
266 | (*) When the kernel has received and set up an incoming call, it sends a | |
267 | message to server application to let it know there's a new call awaiting | |
268 | its acceptance [recvmsg reports a special control message]. The server | |
269 | application then uses sendmsg to assign a tag to the new call. Once that | |
270 | is done, the first part of the request data will be delivered by recvmsg. | |
271 | ||
272 | (*) The server application has to provide the server socket with a keyring of | |
273 | secret keys corresponding to the security types it permits. When a secure | |
274 | connection is being set up, the kernel looks up the appropriate secret key | |
275 | in the keyring and then sends a challenge packet to the client and | |
276 | receives a response packet. The kernel then checks the authorisation of | |
277 | the packet and either aborts the connection or sets up the security. | |
278 | ||
279 | (*) The name of the key a client will use to secure its communications is | |
280 | nominated by a socket option. | |
281 | ||
282 | ||
283 | Notes on recvmsg: | |
284 | ||
285 | (*) If there's a sequence of data messages belonging to a particular call on | |
286 | the receive queue, then recvmsg will keep working through them until: | |
287 | ||
288 | (a) it meets the end of that call's received data, | |
289 | ||
290 | (b) it meets a non-data message, | |
291 | ||
292 | (c) it meets a message belonging to a different call, or | |
293 | ||
294 | (d) it fills the user buffer. | |
295 | ||
296 | If recvmsg is called in blocking mode, it will keep sleeping, awaiting the | |
297 | reception of further data, until one of the above four conditions is met. | |
298 | ||
299 | (2) MSG_PEEK operates similarly, but will return immediately if it has put any | |
300 | data in the buffer rather than sleeping until it can fill the buffer. | |
301 | ||
302 | (3) If a data message is only partially consumed in filling a user buffer, | |
303 | then the remainder of that message will be left on the front of the queue | |
304 | for the next taker. MSG_TRUNC will never be flagged. | |
305 | ||
306 | (4) If there is more data to be had on a call (it hasn't copied the last byte | |
307 | of the last data message in that phase yet), then MSG_MORE will be | |
308 | flagged. | |
309 | ||
310 | ||
311 | ================ | |
312 | CONTROL MESSAGES | |
313 | ================ | |
314 | ||
315 | AF_RXRPC makes use of control messages in sendmsg() and recvmsg() to multiplex | |
316 | calls, to invoke certain actions and to report certain conditions. These are: | |
317 | ||
318 | MESSAGE ID SRT DATA MEANING | |
319 | ======================= === =========== =============================== | |
320 | RXRPC_USER_CALL_ID sr- User ID App's call specifier | |
321 | RXRPC_ABORT srt Abort code Abort code to issue/received | |
322 | RXRPC_ACK -rt n/a Final ACK received | |
323 | RXRPC_NET_ERROR -rt error num Network error on call | |
324 | RXRPC_BUSY -rt n/a Call rejected (server busy) | |
325 | RXRPC_LOCAL_ERROR -rt error num Local error encountered | |
326 | RXRPC_NEW_CALL -r- n/a New call received | |
327 | RXRPC_ACCEPT s-- n/a Accept new call | |
328 | ||
329 | (SRT = usable in Sendmsg / delivered by Recvmsg / Terminal message) | |
330 | ||
331 | (*) RXRPC_USER_CALL_ID | |
332 | ||
333 | This is used to indicate the application's call ID. It's an unsigned long | |
334 | that the app specifies in the client by attaching it to the first data | |
335 | message or in the server by passing it in association with an RXRPC_ACCEPT | |
336 | message. recvmsg() passes it in conjunction with all messages except | |
337 | those of the RXRPC_NEW_CALL message. | |
338 | ||
339 | (*) RXRPC_ABORT | |
340 | ||
341 | This is can be used by an application to abort a call by passing it to | |
342 | sendmsg, or it can be delivered by recvmsg to indicate a remote abort was | |
343 | received. Either way, it must be associated with an RXRPC_USER_CALL_ID to | |
344 | specify the call affected. If an abort is being sent, then error EBADSLT | |
345 | will be returned if there is no call with that user ID. | |
346 | ||
347 | (*) RXRPC_ACK | |
348 | ||
349 | This is delivered to a server application to indicate that the final ACK | |
350 | of a call was received from the client. It will be associated with an | |
351 | RXRPC_USER_CALL_ID to indicate the call that's now complete. | |
352 | ||
353 | (*) RXRPC_NET_ERROR | |
354 | ||
355 | This is delivered to an application to indicate that an ICMP error message | |
356 | was encountered in the process of trying to talk to the peer. An | |
357 | errno-class integer value will be included in the control message data | |
358 | indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call | |
359 | affected. | |
360 | ||
361 | (*) RXRPC_BUSY | |
362 | ||
363 | This is delivered to a client application to indicate that a call was | |
364 | rejected by the server due to the server being busy. It will be | |
365 | associated with an RXRPC_USER_CALL_ID to indicate the rejected call. | |
366 | ||
367 | (*) RXRPC_LOCAL_ERROR | |
368 | ||
369 | This is delivered to an application to indicate that a local error was | |
370 | encountered and that a call has been aborted because of it. An | |
371 | errno-class integer value will be included in the control message data | |
372 | indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call | |
373 | affected. | |
374 | ||
375 | (*) RXRPC_NEW_CALL | |
376 | ||
377 | This is delivered to indicate to a server application that a new call has | |
378 | arrived and is awaiting acceptance. No user ID is associated with this, | |
379 | as a user ID must subsequently be assigned by doing an RXRPC_ACCEPT. | |
380 | ||
381 | (*) RXRPC_ACCEPT | |
382 | ||
383 | This is used by a server application to attempt to accept a call and | |
384 | assign it a user ID. It should be associated with an RXRPC_USER_CALL_ID | |
385 | to indicate the user ID to be assigned. If there is no call to be | |
386 | accepted (it may have timed out, been aborted, etc.), then sendmsg will | |
387 | return error ENODATA. If the user ID is already in use by another call, | |
388 | then error EBADSLT will be returned. | |
389 | ||
390 | ||
391 | ============== | |
392 | SOCKET OPTIONS | |
393 | ============== | |
394 | ||
395 | AF_RXRPC sockets support a few socket options at the SOL_RXRPC level: | |
396 | ||
397 | (*) RXRPC_SECURITY_KEY | |
398 | ||
399 | This is used to specify the description of the key to be used. The key is | |
400 | extracted from the calling process's keyrings with request_key() and | |
401 | should be of "rxrpc" type. | |
402 | ||
403 | The optval pointer points to the description string, and optlen indicates | |
404 | how long the string is, without the NUL terminator. | |
405 | ||
406 | (*) RXRPC_SECURITY_KEYRING | |
407 | ||
408 | Similar to above but specifies a keyring of server secret keys to use (key | |
409 | type "keyring"). See the "Security" section. | |
410 | ||
411 | (*) RXRPC_EXCLUSIVE_CONNECTION | |
412 | ||
413 | This is used to request that new connections should be used for each call | |
414 | made subsequently on this socket. optval should be NULL and optlen 0. | |
415 | ||
416 | (*) RXRPC_MIN_SECURITY_LEVEL | |
417 | ||
418 | This is used to specify the minimum security level required for calls on | |
419 | this socket. optval must point to an int containing one of the following | |
420 | values: | |
421 | ||
422 | (a) RXRPC_SECURITY_PLAIN | |
423 | ||
424 | Encrypted checksum only. | |
425 | ||
426 | (b) RXRPC_SECURITY_AUTH | |
427 | ||
428 | Encrypted checksum plus packet padded and first eight bytes of packet | |
429 | encrypted - which includes the actual packet length. | |
430 | ||
431 | (c) RXRPC_SECURITY_ENCRYPTED | |
432 | ||
433 | Encrypted checksum plus entire packet padded and encrypted, including | |
434 | actual packet length. | |
435 | ||
436 | ||
437 | ======== | |
438 | SECURITY | |
439 | ======== | |
440 | ||
441 | Currently, only the kerberos 4 equivalent protocol has been implemented | |
442 | (security index 2 - rxkad). This requires the rxkad module to be loaded and, | |
443 | on the client, tickets of the appropriate type to be obtained from the AFS | |
444 | kaserver or the kerberos server and installed as "rxrpc" type keys. This is | |
445 | normally done using the klog program. An example simple klog program can be | |
446 | found at: | |
447 | ||
448 | http://people.redhat.com/~dhowells/rxrpc/klog.c | |
449 | ||
450 | The payload provided to add_key() on the client should be of the following | |
451 | form: | |
452 | ||
453 | struct rxrpc_key_sec2_v1 { | |
454 | uint16_t security_index; /* 2 */ | |
455 | uint16_t ticket_length; /* length of ticket[] */ | |
456 | uint32_t expiry; /* time at which expires */ | |
457 | uint8_t kvno; /* key version number */ | |
458 | uint8_t __pad[3]; | |
459 | uint8_t session_key[8]; /* DES session key */ | |
460 | uint8_t ticket[0]; /* the encrypted ticket */ | |
461 | }; | |
462 | ||
463 | Where the ticket blob is just appended to the above structure. | |
464 | ||
465 | ||
466 | For the server, keys of type "rxrpc_s" must be made available to the server. | |
467 | They have a description of "<serviceID>:<securityIndex>" (eg: "52:2" for an | |
468 | rxkad key for the AFS VL service). When such a key is created, it should be | |
469 | given the server's secret key as the instantiation data (see the example | |
470 | below). | |
471 | ||
472 | add_key("rxrpc_s", "52:2", secret_key, 8, keyring); | |
473 | ||
474 | A keyring is passed to the server socket by naming it in a sockopt. The server | |
475 | socket then looks the server secret keys up in this keyring when secure | |
476 | incoming connections are made. This can be seen in an example program that can | |
477 | be found at: | |
478 | ||
479 | http://people.redhat.com/~dhowells/rxrpc/listen.c | |
480 | ||
481 | ||
482 | ==================== | |
483 | EXAMPLE CLIENT USAGE | |
484 | ==================== | |
485 | ||
486 | A client would issue an operation by: | |
487 | ||
488 | (1) An RxRPC socket is set up by: | |
489 | ||
490 | client = socket(AF_RXRPC, SOCK_DGRAM, PF_INET); | |
491 | ||
492 | Where the third parameter indicates the protocol family of the transport | |
493 | socket used - usually IPv4 but it can also be IPv6 [TODO]. | |
494 | ||
495 | (2) A local address can optionally be bound: | |
496 | ||
497 | struct sockaddr_rxrpc srx = { | |
498 | .srx_family = AF_RXRPC, | |
499 | .srx_service = 0, /* we're a client */ | |
500 | .transport_type = SOCK_DGRAM, /* type of transport socket */ | |
501 | .transport.sin_family = AF_INET, | |
502 | .transport.sin_port = htons(7000), /* AFS callback */ | |
503 | .transport.sin_address = 0, /* all local interfaces */ | |
504 | }; | |
505 | bind(client, &srx, sizeof(srx)); | |
506 | ||
507 | This specifies the local UDP port to be used. If not given, a random | |
508 | non-privileged port will be used. A UDP port may be shared between | |
509 | several unrelated RxRPC sockets. Security is handled on a basis of | |
510 | per-RxRPC virtual connection. | |
511 | ||
512 | (3) The security is set: | |
513 | ||
514 | const char *key = "AFS:cambridge.redhat.com"; | |
515 | setsockopt(client, SOL_RXRPC, RXRPC_SECURITY_KEY, key, strlen(key)); | |
516 | ||
517 | This issues a request_key() to get the key representing the security | |
518 | context. The minimum security level can be set: | |
519 | ||
520 | unsigned int sec = RXRPC_SECURITY_ENCRYPTED; | |
521 | setsockopt(client, SOL_RXRPC, RXRPC_MIN_SECURITY_LEVEL, | |
522 | &sec, sizeof(sec)); | |
523 | ||
524 | (4) The server to be contacted can then be specified (alternatively this can | |
525 | be done through sendmsg): | |
526 | ||
527 | struct sockaddr_rxrpc srx = { | |
528 | .srx_family = AF_RXRPC, | |
529 | .srx_service = VL_SERVICE_ID, | |
530 | .transport_type = SOCK_DGRAM, /* type of transport socket */ | |
531 | .transport.sin_family = AF_INET, | |
532 | .transport.sin_port = htons(7005), /* AFS volume manager */ | |
533 | .transport.sin_address = ..., | |
534 | }; | |
535 | connect(client, &srx, sizeof(srx)); | |
536 | ||
537 | (5) The request data should then be posted to the server socket using a series | |
538 | of sendmsg() calls, each with the following control message attached: | |
539 | ||
540 | RXRPC_USER_CALL_ID - specifies the user ID for this call | |
541 | ||
542 | MSG_MORE should be set in msghdr::msg_flags on all but the last part of | |
543 | the request. Multiple requests may be made simultaneously. | |
544 | ||
025dfdaf | 545 | If a call is intended to go to a destination other than the default |
17926a79 DH |
546 | specified through connect(), then msghdr::msg_name should be set on the |
547 | first request message of that call. | |
548 | ||
549 | (6) The reply data will then be posted to the server socket for recvmsg() to | |
550 | pick up. MSG_MORE will be flagged by recvmsg() if there's more reply data | |
551 | for a particular call to be read. MSG_EOR will be set on the terminal | |
552 | read for a call. | |
553 | ||
554 | All data will be delivered with the following control message attached: | |
555 | ||
556 | RXRPC_USER_CALL_ID - specifies the user ID for this call | |
557 | ||
558 | If an abort or error occurred, this will be returned in the control data | |
559 | buffer instead, and MSG_EOR will be flagged to indicate the end of that | |
560 | call. | |
561 | ||
562 | ||
563 | ==================== | |
564 | EXAMPLE SERVER USAGE | |
565 | ==================== | |
566 | ||
567 | A server would be set up to accept operations in the following manner: | |
568 | ||
569 | (1) An RxRPC socket is created by: | |
570 | ||
571 | server = socket(AF_RXRPC, SOCK_DGRAM, PF_INET); | |
572 | ||
573 | Where the third parameter indicates the address type of the transport | |
574 | socket used - usually IPv4. | |
575 | ||
576 | (2) Security is set up if desired by giving the socket a keyring with server | |
577 | secret keys in it: | |
578 | ||
579 | keyring = add_key("keyring", "AFSkeys", NULL, 0, | |
580 | KEY_SPEC_PROCESS_KEYRING); | |
581 | ||
582 | const char secret_key[8] = { | |
583 | 0xa7, 0x83, 0x8a, 0xcb, 0xc7, 0x83, 0xec, 0x94 }; | |
584 | add_key("rxrpc_s", "52:2", secret_key, 8, keyring); | |
585 | ||
586 | setsockopt(server, SOL_RXRPC, RXRPC_SECURITY_KEYRING, "AFSkeys", 7); | |
587 | ||
588 | The keyring can be manipulated after it has been given to the socket. This | |
589 | permits the server to add more keys, replace keys, etc. whilst it is live. | |
590 | ||
591 | (2) A local address must then be bound: | |
592 | ||
593 | struct sockaddr_rxrpc srx = { | |
594 | .srx_family = AF_RXRPC, | |
595 | .srx_service = VL_SERVICE_ID, /* RxRPC service ID */ | |
596 | .transport_type = SOCK_DGRAM, /* type of transport socket */ | |
597 | .transport.sin_family = AF_INET, | |
598 | .transport.sin_port = htons(7000), /* AFS callback */ | |
599 | .transport.sin_address = 0, /* all local interfaces */ | |
600 | }; | |
601 | bind(server, &srx, sizeof(srx)); | |
602 | ||
603 | (3) The server is then set to listen out for incoming calls: | |
604 | ||
605 | listen(server, 100); | |
606 | ||
607 | (4) The kernel notifies the server of pending incoming connections by sending | |
608 | it a message for each. This is received with recvmsg() on the server | |
609 | socket. It has no data, and has a single dataless control message | |
610 | attached: | |
611 | ||
612 | RXRPC_NEW_CALL | |
613 | ||
614 | The address that can be passed back by recvmsg() at this point should be | |
615 | ignored since the call for which the message was posted may have gone by | |
616 | the time it is accepted - in which case the first call still on the queue | |
617 | will be accepted. | |
618 | ||
619 | (5) The server then accepts the new call by issuing a sendmsg() with two | |
620 | pieces of control data and no actual data: | |
621 | ||
622 | RXRPC_ACCEPT - indicate connection acceptance | |
623 | RXRPC_USER_CALL_ID - specify user ID for this call | |
624 | ||
625 | (6) The first request data packet will then be posted to the server socket for | |
626 | recvmsg() to pick up. At that point, the RxRPC address for the call can | |
627 | be read from the address fields in the msghdr struct. | |
628 | ||
629 | Subsequent request data will be posted to the server socket for recvmsg() | |
630 | to collect as it arrives. All but the last piece of the request data will | |
631 | be delivered with MSG_MORE flagged. | |
632 | ||
633 | All data will be delivered with the following control message attached: | |
634 | ||
635 | RXRPC_USER_CALL_ID - specifies the user ID for this call | |
636 | ||
637 | (8) The reply data should then be posted to the server socket using a series | |
638 | of sendmsg() calls, each with the following control messages attached: | |
639 | ||
640 | RXRPC_USER_CALL_ID - specifies the user ID for this call | |
641 | ||
642 | MSG_MORE should be set in msghdr::msg_flags on all but the last message | |
643 | for a particular call. | |
644 | ||
645 | (9) The final ACK from the client will be posted for retrieval by recvmsg() | |
646 | when it is received. It will take the form of a dataless message with two | |
647 | control messages attached: | |
648 | ||
649 | RXRPC_USER_CALL_ID - specifies the user ID for this call | |
650 | RXRPC_ACK - indicates final ACK (no data) | |
651 | ||
652 | MSG_EOR will be flagged to indicate that this is the final message for | |
653 | this call. | |
654 | ||
655 | (10) Up to the point the final packet of reply data is sent, the call can be | |
656 | aborted by calling sendmsg() with a dataless message with the following | |
657 | control messages attached: | |
658 | ||
659 | RXRPC_USER_CALL_ID - specifies the user ID for this call | |
660 | RXRPC_ABORT - indicates abort code (4 byte data) | |
661 | ||
662 | Any packets waiting in the socket's receive queue will be discarded if | |
663 | this is issued. | |
664 | ||
665 | Note that all the communications for a particular service take place through | |
666 | the one server socket, using control messages on sendmsg() and recvmsg() to | |
667 | determine the call affected. | |
651350d1 DH |
668 | |
669 | ||
670 | ========================= | |
671 | AF_RXRPC KERNEL INTERFACE | |
672 | ========================= | |
673 | ||
674 | The AF_RXRPC module also provides an interface for use by in-kernel utilities | |
675 | such as the AFS filesystem. This permits such a utility to: | |
676 | ||
677 | (1) Use different keys directly on individual client calls on one socket | |
678 | rather than having to open a whole slew of sockets, one for each key it | |
679 | might want to use. | |
680 | ||
681 | (2) Avoid having RxRPC call request_key() at the point of issue of a call or | |
682 | opening of a socket. Instead the utility is responsible for requesting a | |
683 | key at the appropriate point. AFS, for instance, would do this during VFS | |
684 | operations such as open() or unlink(). The key is then handed through | |
685 | when the call is initiated. | |
686 | ||
687 | (3) Request the use of something other than GFP_KERNEL to allocate memory. | |
688 | ||
689 | (4) Avoid the overhead of using the recvmsg() call. RxRPC messages can be | |
690 | intercepted before they get put into the socket Rx queue and the socket | |
691 | buffers manipulated directly. | |
692 | ||
693 | To use the RxRPC facility, a kernel utility must still open an AF_RXRPC socket, | |
01dd2fbf | 694 | bind an address as appropriate and listen if it's to be a server socket, but |
651350d1 DH |
695 | then it passes this to the kernel interface functions. |
696 | ||
697 | The kernel interface functions are as follows: | |
698 | ||
699 | (*) Begin a new client call. | |
700 | ||
701 | struct rxrpc_call * | |
702 | rxrpc_kernel_begin_call(struct socket *sock, | |
703 | struct sockaddr_rxrpc *srx, | |
704 | struct key *key, | |
705 | unsigned long user_call_ID, | |
706 | gfp_t gfp); | |
707 | ||
708 | This allocates the infrastructure to make a new RxRPC call and assigns | |
709 | call and connection numbers. The call will be made on the UDP port that | |
710 | the socket is bound to. The call will go to the destination address of a | |
711 | connected client socket unless an alternative is supplied (srx is | |
712 | non-NULL). | |
713 | ||
714 | If a key is supplied then this will be used to secure the call instead of | |
715 | the key bound to the socket with the RXRPC_SECURITY_KEY sockopt. Calls | |
716 | secured in this way will still share connections if at all possible. | |
717 | ||
718 | The user_call_ID is equivalent to that supplied to sendmsg() in the | |
719 | control data buffer. It is entirely feasible to use this to point to a | |
720 | kernel data structure. | |
721 | ||
722 | If this function is successful, an opaque reference to the RxRPC call is | |
723 | returned. The caller now holds a reference on this and it must be | |
724 | properly ended. | |
725 | ||
726 | (*) End a client call. | |
727 | ||
728 | void rxrpc_kernel_end_call(struct rxrpc_call *call); | |
729 | ||
730 | This is used to end a previously begun call. The user_call_ID is expunged | |
731 | from AF_RXRPC's knowledge and will not be seen again in association with | |
732 | the specified call. | |
733 | ||
734 | (*) Send data through a call. | |
735 | ||
736 | int rxrpc_kernel_send_data(struct rxrpc_call *call, struct msghdr *msg, | |
737 | size_t len); | |
738 | ||
739 | This is used to supply either the request part of a client call or the | |
740 | reply part of a server call. msg.msg_iovlen and msg.msg_iov specify the | |
741 | data buffers to be used. msg_iov may not be NULL and must point | |
742 | exclusively to in-kernel virtual addresses. msg.msg_flags may be given | |
743 | MSG_MORE if there will be subsequent data sends for this call. | |
744 | ||
745 | The msg must not specify a destination address, control data or any flags | |
746 | other than MSG_MORE. len is the total amount of data to transmit. | |
747 | ||
748 | (*) Abort a call. | |
749 | ||
750 | void rxrpc_kernel_abort_call(struct rxrpc_call *call, u32 abort_code); | |
751 | ||
752 | This is used to abort a call if it's still in an abortable state. The | |
753 | abort code specified will be placed in the ABORT message sent. | |
754 | ||
755 | (*) Intercept received RxRPC messages. | |
756 | ||
757 | typedef void (*rxrpc_interceptor_t)(struct sock *sk, | |
758 | unsigned long user_call_ID, | |
759 | struct sk_buff *skb); | |
760 | ||
761 | void | |
762 | rxrpc_kernel_intercept_rx_messages(struct socket *sock, | |
763 | rxrpc_interceptor_t interceptor); | |
764 | ||
765 | This installs an interceptor function on the specified AF_RXRPC socket. | |
766 | All messages that would otherwise wind up in the socket's Rx queue are | |
767 | then diverted to this function. Note that care must be taken to process | |
768 | the messages in the right order to maintain DATA message sequentiality. | |
769 | ||
770 | The interceptor function itself is provided with the address of the socket | |
771 | and handling the incoming message, the ID assigned by the kernel utility | |
772 | to the call and the socket buffer containing the message. | |
773 | ||
774 | The skb->mark field indicates the type of message: | |
775 | ||
776 | MARK MEANING | |
777 | =============================== ======================================= | |
778 | RXRPC_SKB_MARK_DATA Data message | |
779 | RXRPC_SKB_MARK_FINAL_ACK Final ACK received for an incoming call | |
780 | RXRPC_SKB_MARK_BUSY Client call rejected as server busy | |
781 | RXRPC_SKB_MARK_REMOTE_ABORT Call aborted by peer | |
782 | RXRPC_SKB_MARK_NET_ERROR Network error detected | |
783 | RXRPC_SKB_MARK_LOCAL_ERROR Local error encountered | |
784 | RXRPC_SKB_MARK_NEW_CALL New incoming call awaiting acceptance | |
785 | ||
786 | The remote abort message can be probed with rxrpc_kernel_get_abort_code(). | |
787 | The two error messages can be probed with rxrpc_kernel_get_error_number(). | |
788 | A new call can be accepted with rxrpc_kernel_accept_call(). | |
789 | ||
790 | Data messages can have their contents extracted with the usual bunch of | |
791 | socket buffer manipulation functions. A data message can be determined to | |
792 | be the last one in a sequence with rxrpc_kernel_is_data_last(). When a | |
793 | data message has been used up, rxrpc_kernel_data_delivered() should be | |
794 | called on it.. | |
795 | ||
796 | Non-data messages should be handled to rxrpc_kernel_free_skb() to dispose | |
797 | of. It is possible to get extra refs on all types of message for later | |
798 | freeing, but this may pin the state of a call until the message is finally | |
799 | freed. | |
800 | ||
801 | (*) Accept an incoming call. | |
802 | ||
803 | struct rxrpc_call * | |
804 | rxrpc_kernel_accept_call(struct socket *sock, | |
805 | unsigned long user_call_ID); | |
806 | ||
807 | This is used to accept an incoming call and to assign it a call ID. This | |
808 | function is similar to rxrpc_kernel_begin_call() and calls accepted must | |
809 | be ended in the same way. | |
810 | ||
811 | If this function is successful, an opaque reference to the RxRPC call is | |
812 | returned. The caller now holds a reference on this and it must be | |
813 | properly ended. | |
814 | ||
815 | (*) Reject an incoming call. | |
816 | ||
817 | int rxrpc_kernel_reject_call(struct socket *sock); | |
818 | ||
819 | This is used to reject the first incoming call on the socket's queue with | |
820 | a BUSY message. -ENODATA is returned if there were no incoming calls. | |
821 | Other errors may be returned if the call had been aborted (-ECONNABORTED) | |
822 | or had timed out (-ETIME). | |
823 | ||
824 | (*) Record the delivery of a data message and free it. | |
825 | ||
826 | void rxrpc_kernel_data_delivered(struct sk_buff *skb); | |
827 | ||
828 | This is used to record a data message as having been delivered and to | |
829 | update the ACK state for the call. The socket buffer will be freed. | |
830 | ||
831 | (*) Free a message. | |
832 | ||
833 | void rxrpc_kernel_free_skb(struct sk_buff *skb); | |
834 | ||
835 | This is used to free a non-DATA socket buffer intercepted from an AF_RXRPC | |
836 | socket. | |
837 | ||
838 | (*) Determine if a data message is the last one on a call. | |
839 | ||
840 | bool rxrpc_kernel_is_data_last(struct sk_buff *skb); | |
841 | ||
842 | This is used to determine if a socket buffer holds the last data message | |
843 | to be received for a call (true will be returned if it does, false | |
844 | if not). | |
845 | ||
846 | The data message will be part of the reply on a client call and the | |
847 | request on an incoming call. In the latter case there will be more | |
848 | messages, but in the former case there will not. | |
849 | ||
850 | (*) Get the abort code from an abort message. | |
851 | ||
852 | u32 rxrpc_kernel_get_abort_code(struct sk_buff *skb); | |
853 | ||
854 | This is used to extract the abort code from a remote abort message. | |
855 | ||
856 | (*) Get the error number from a local or network error message. | |
857 | ||
858 | int rxrpc_kernel_get_error_number(struct sk_buff *skb); | |
859 | ||
860 | This is used to extract the error number from a message indicating either | |
861 | a local error occurred or a network error occurred. | |
76181c13 DH |
862 | |
863 | (*) Allocate a null key for doing anonymous security. | |
864 | ||
865 | struct key *rxrpc_get_null_key(const char *keyname); | |
866 | ||
867 | This is used to allocate a null RxRPC key that can be used to indicate | |
868 | anonymous security for a particular domain. | |
5873c083 DH |
869 | |
870 | ||
871 | ======================= | |
872 | CONFIGURABLE PARAMETERS | |
873 | ======================= | |
874 | ||
875 | The RxRPC protocol driver has a number of configurable parameters that can be | |
876 | adjusted through sysctls in /proc/net/rxrpc/: | |
877 | ||
878 | (*) req_ack_delay | |
879 | ||
880 | The amount of time in milliseconds after receiving a packet with the | |
881 | request-ack flag set before we honour the flag and actually send the | |
882 | requested ack. | |
883 | ||
884 | Usually the other side won't stop sending packets until the advertised | |
885 | reception window is full (to a maximum of 255 packets), so delaying the | |
886 | ACK permits several packets to be ACK'd in one go. | |
887 | ||
888 | (*) soft_ack_delay | |
889 | ||
890 | The amount of time in milliseconds after receiving a new packet before we | |
891 | generate a soft-ACK to tell the sender that it doesn't need to resend. | |
892 | ||
893 | (*) idle_ack_delay | |
894 | ||
895 | The amount of time in milliseconds after all the packets currently in the | |
896 | received queue have been consumed before we generate a hard-ACK to tell | |
897 | the sender it can free its buffers, assuming no other reason occurs that | |
898 | we would send an ACK. | |
899 | ||
900 | (*) resend_timeout | |
901 | ||
902 | The amount of time in milliseconds after transmitting a packet before we | |
903 | transmit it again, assuming no ACK is received from the receiver telling | |
904 | us they got it. | |
905 | ||
906 | (*) max_call_lifetime | |
907 | ||
908 | The maximum amount of time in seconds that a call may be in progress | |
909 | before we preemptively kill it. | |
910 | ||
911 | (*) dead_call_expiry | |
912 | ||
913 | The amount of time in seconds before we remove a dead call from the call | |
914 | list. Dead calls are kept around for a little while for the purpose of | |
915 | repeating ACK and ABORT packets. | |
916 | ||
917 | (*) connection_expiry | |
918 | ||
919 | The amount of time in seconds after a connection was last used before we | |
920 | remove it from the connection list. Whilst a connection is in existence, | |
921 | it serves as a placeholder for negotiated security; when it is deleted, | |
922 | the security must be renegotiated. | |
923 | ||
924 | (*) transport_expiry | |
925 | ||
926 | The amount of time in seconds after a transport was last used before we | |
927 | remove it from the transport list. Whilst a transport is in existence, it | |
928 | serves to anchor the peer data and keeps the connection ID counter. | |
817913d8 DH |
929 | |
930 | (*) rxrpc_rx_window_size | |
931 | ||
932 | The size of the receive window in packets. This is the maximum number of | |
933 | unconsumed received packets we're willing to hold in memory for any | |
934 | particular call. | |
935 | ||
936 | (*) rxrpc_rx_mtu | |
937 | ||
938 | The maximum packet MTU size that we're willing to receive in bytes. This | |
939 | indicates to the peer whether we're willing to accept jumbo packets. | |
940 | ||
941 | (*) rxrpc_rx_jumbo_max | |
942 | ||
943 | The maximum number of packets that we're willing to accept in a jumbo | |
944 | packet. Non-terminal packets in a jumbo packet must contain a four byte | |
945 | header plus exactly 1412 bytes of data. The terminal packet must contain | |
946 | a four byte header plus any amount of data. In any event, a jumbo packet | |
947 | may not exceed rxrpc_rx_mtu in size. |