Add macro support
[normand.git] / README.adoc
CommitLineData
bb2f9e9c
PP
1// Show ToC at a specific location for a GitHub rendering
2ifdef::env-github[]
3:toc: macro
4endif::env-github[]
5
6ifndef::env-github[]
71aaa3f7 7:toc: left
bb2f9e9c
PP
8endif::env-github[]
9
10// This is to mimic what GitHub does so that anchors work in an offline
11// rendering too.
12:idprefix:
13:idseparator: -
71aaa3f7 14
bb2f9e9c 15// Other attributes
71aaa3f7
PP
16:py3: Python{nbsp}3
17
bb2f9e9c
PP
18= Normand
19Philippe Proulx
20
df0f8552
PP
21image::normand-logo.png[]
22
71aaa3f7
PP
23[.normal]
24image:https://img.shields.io/pypi/v/normand.svg?label=Latest%20version[link="https://pypi.python.org/pypi/normand"]
25
26[.lead]
27_**Normand**_ is a text-to-binary processor with its own language.
28
29This package offers both a portable {py3} module and a command-line
30tool.
31
320644e2 32WARNING: This version of Normand is 0.11, meaning both the Normand
71aaa3f7
PP
33language and the module/CLI interface aren't stable.
34
bb2f9e9c
PP
35ifdef::env-github[]
36// ToC location for a GitHub rendering
37toc::[]
38endif::env-github[]
39
71aaa3f7
PP
40== Introduction
41
42The purpose of Normand is to consume human-readable text representing
43bytes and to produce the corresponding binary data.
44
45.Simple bytes input.
46====
47Consider the following Normand input:
48
49----
504f 55 32 bb $167 fe %10100111 a9 $-32
51----
52
53The generated nine bytes are:
54
55----
564f 55 32 bb a7 fe a7 a9 e0
57----
58====
59
60As you can see in the last example, the fundamental unit of the Normand
61language is the _byte_. The order in which you list bytes will be the
62order of the generated data.
63
64The Normand language is more than simple lists of bytes, though. Its
65main features are:
66
67Comments, including a bunch of insignificant symbols which may improve readability::
68+
69Input:
70+
71----
72ff bb %1101:0010 # This is a comment
7378 29 af $192 # This too # 99 $-80
74fe80::6257:18ff:fea3:4229
7560:57:18:a3:42:29
7610839636-5d65-4a68-8e6a-21608ddf7258
77----
78+
79Output:
80+
81----
82ff bb d2 78 29 af c0 99 b0 fe 80 62 57 18 ff fe
83a3 42 29 60 57 18 a3 42 29 10 83 96 36 5d 65 4a
8468 8e 6a 21 60 8d df 72 58
85----
86
87Hexadecimal, decimal, and binary byte constants::
88+
89Input:
90+
91----
92aa bb $247 $-89 %0011_0010 %11.01= 10/10
93----
94+
95Output:
96+
97----
98aa bb f7 a7 32 da
99----
100
101UTF-8, UTF-16, and UTF-32 literal strings::
102+
103Input:
104+
105----
106"hello world!" 00
107u16le"stress\nverdict 🤣"
108----
109+
110Output:
111+
112----
11368 65 6c 6c 6f 20 77 6f 72 6c 64 21 00 73 00 74 ┆ hello world!•s•t
11400 72 00 65 00 73 00 73 00 0a 00 76 00 65 00 72 ┆ •r•e•s•s•••v•e•r
11500 64 00 69 00 63 00 74 00 20 00 3e d8 23 dd ┆ •d•i•c•t• •>•#•
116----
117
118Labels: special variables holding the offset where they're defined::
119+
120----
121<beg> b2 52 e3 bc 91 05
122$100 $50 <chair> 33 9f fe
12325 e9 89 8a <end>
124----
125
126Variables::
127+
128----
1295e 65 {tower = 47} c6 7f f2 c4
13044 {hurl = tower - 14} b5 {tower = hurl} 26 2d
131----
132+
133The value of a variable assignment is the evaluation of a valid {py3}
134expression which may include label and variable names.
135
269f6eb3 136Fixed-length number with a given length (8{nbsp}bits to 64{nbsp}bits) and byte order::
71aaa3f7
PP
137+
138Input:
139+
140----
141{strength = 4}
142{be} 67 <lbl> 44 $178 {(end - lbl) * 8 + strength : 16} $99 <end>
143{le} {-1993 : 32}
269f6eb3 144{-3.141593 : 64}
71aaa3f7
PP
145----
146+
147Output:
148+
149----
269f6eb3
PP
15067 44 b2 00 2c 63 37 f8 ff ff 7f bd c2 82 fb 21
15109 c0
71aaa3f7
PP
152----
153+
269f6eb3 154The encoded number is the evaluation of a valid {py3} expression which
05f81895
PP
155may include label and variable names.
156
157https://en.wikipedia.org/wiki/LEB128[LEB128] integer::
158+
159Input:
160+
161----
162aa bb cc {-1993 : sleb128} <meow> dd ee ff
163{meow * 199 : uleb128}
164----
165+
166Output:
167+
168----
169aa bb cc b7 70 dd ee ff e3 07
170----
171+
172The encoded integer is the evaluation of a valid {py3} expression which
71aaa3f7
PP
173may include label and variable names.
174
27d52a19
PP
175Conditional::
176+
177Input:
178+
179----
180aa bb cc
181
182(
183 "foo"
184
185 !if {ICITTE > 10}
186 "bar"
187 !end
188) * 4
189----
190+
191Output:
192+
193----
194aa bb cc 66 6f 6f 66 6f 6f 66 6f 6f 62 61 72 66 ┆ •••foofoofoobarf
1956f 6f 62 61 72 ┆ oobar
196----
197
71aaa3f7
PP
198Repetition::
199+
200Input:
201+
202----
2adf4336 203aa bb * 5 cc <zoom> "yeah\0" * {zoom * 3}
e57a18e1
PP
204
205!repeat 3
206 ff ee "juice"
207!end
71aaa3f7
PP
208----
209+
210Output:
211+
212----
2adf4336
PP
213aa bb bb bb bb bb cc 79 65 61 68 00 79 65 61 68 ┆ •••••••yeah•yeah
21400 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ •yeah•yeah•yeah•
21579 65 61 68 00 79 65 61 68 00 79 65 61 68 00 79 ┆ yeah•yeah•yeah•y
21665 61 68 00 79 65 61 68 00 79 65 61 68 00 79 65 ┆ eah•yeah•yeah•ye
21761 68 00 79 65 61 68 00 79 65 61 68 00 79 65 61 ┆ ah•yeah•yeah•yea
21868 00 79 65 61 68 00 79 65 61 68 00 79 65 61 68 ┆ h•yeah•yeah•yeah
71aaa3f7 21900 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ •yeah•yeah•yeah•
e57a18e1
PP
220ff ee 6a 75 69 63 65 ff ee 6a 75 69 63 65 ff ee ┆ ••juice••juice••
2216a 75 69 63 65 ┆ juice
71aaa3f7
PP
222----
223
676f6189
PP
224Alignment::
225+
226Input:
227+
228----
229{be}
230
231 {199:32}
232@64 {43:64}
233@16 {-123:16}
234@32~255 {5584:32}
235----
236+
237Output:
238+
239----
24000 00 00 c7 00 00 00 00 00 00 00 00 00 00 00 2b
241ff 85 ff ff 00 00 15 d0
242----
71aaa3f7
PP
243
244Multilevel grouping::
245+
246Input:
247+
248----
249ff ((aa bb "zoom" cc) * 5) * 3 $-34 * 4
250----
251+
252Output:
253+
254----
255ff aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa ┆ •••zoom•••zoom••
256bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a ┆ •zoom•••zoom•••z
2576f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f ┆ oom•••zoom•••zoo
2586d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc ┆ m•••zoom•••zoom•
259aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb ┆ ••zoom•••zoom•••
2607a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f ┆ zoom•••zoom•••zo
2616f 6d cc aa bb 7a 6f 6f 6d cc de de de de ┆ om•••zoom•••••
262----
263
320644e2
PP
264Macros::
265+
266Input:
267+
268----
269!macro hello(world)
270 "hello"
271 !if world " world" !end
272!end
273
274!repeat 17
275 ff ff ff ff
276 m:hello({ICITTE > 15 and ICITTE < 60})
277!end
278----
279+
280Output:
281+
282----
283ff ff ff ff 68 65 6c 6c 6f ff ff ff ff 68 65 6c ┆ ••••hello••••hel
2846c 6f ff ff ff ff 68 65 6c 6c 6f 20 77 6f 72 6c ┆ lo••••hello worl
28564 ff ff ff ff 68 65 6c 6c 6f 20 77 6f 72 6c 64 ┆ d••••hello world
286ff ff ff ff 68 65 6c 6c 6f 20 77 6f 72 6c 64 ff ┆ ••••hello world•
287ff ff ff 68 65 6c 6c 6f ff ff ff ff 68 65 6c 6c ┆ •••hello••••hell
2886f ff ff ff ff 68 65 6c 6c 6f ff ff ff ff 68 65 ┆ o••••hello••••he
2896c 6c 6f ff ff ff ff 68 65 6c 6c 6f ff ff ff ff ┆ llo••••hello••••
29068 65 6c 6c 6f ff ff ff ff 68 65 6c 6c 6f ff ff ┆ hello••••hello••
291ff ff 68 65 6c 6c 6f ff ff ff ff 68 65 6c 6c 6f ┆ ••hello••••hello
292ff ff ff ff 68 65 6c 6c 6f ff ff ff ff 68 65 6c ┆ ••••hello••••hel
2936c 6f ff ff ff ff 68 65 6c 6c 6f ┆ lo••••hello
294----
295
71aaa3f7
PP
296Precise error reporting::
297+
298----
299/tmp/meow.normand:10:24 - Expecting a bit (`0` or `1`).
300----
301+
302----
303/tmp/meow.normand:32:6 - Unexpected character `k`.
304----
305+
306----
320644e2 307/tmp/meow.normand:24:19 - Illegal (unknown or unreachable) variable/label name `meow` in expression `(meow - 45) // 8`; the legal names are {`ICITTE`, `mix`, `zoom`}.
71aaa3f7
PP
308----
309+
310----
320644e2 311/tmp/meow.normand:18:9 - Value 315 is outside the 8-bit range when evaluating expression `end - ICITTE`.
71aaa3f7
PP
312----
313
314You can use Normand to track data source files in your favorite VCS
315instead of raw binary files. The binary files that Normand generates can
316be used to test file format decoding, including malformatted data, for
317example, as well as for education.
318
319See <<learn-normand>> to explore all the Normand features.
320
321== Install Normand
322
323Normand requires Python ≥ 3.4.
324
325To install Normand:
326
327----
328$ python3 -m pip install --user normand
329----
330
331See
332https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-to-the-user-site[Installing to the User Site]
333to learn more about a user site installation.
334
335[NOTE]
336====
337Normand has a single module file, `normand.py`, which you can copy as is
af3cf417 338to your project to use it (both the <<python3-api,`normand.parse()`>>
71aaa3f7
PP
339function and the <<command-line-tool,command-line tool>>).
340
341`normand.py` has _no external dependencies_, but if you're using
342Python{nbsp}3.4, you'll need a local copy of the standard `typing`
343module.
344====
345
346== Learn Normand
347
348A Normand text input is a sequence of items which represent a sequence
349of raw bytes.
350
351[[state]] During the processing of items to data, Normand relies on a
352current state:
353
354[%header%autowidth]
355|===
af3cf417 356|State variable |Description |Initial value: <<python3-api,{py3} API>> |Initial value: <<command-line-tool,CLI>>
71aaa3f7
PP
357
358|[[cur-offset]] Current offset
359|
05f81895 360The current offset has an effect on the value of <<label,labels>> and of
269f6eb3 361the special `ICITTE` name in <<fixed-length-number,fixed-length
27d52a19
PP
362number>>, <<leb-128-integer,LEB128 integer>>,
363<<variable-assignment,variable assignment>>,
364<<conditional-block,conditional block>>, <<repetition-block,repetition
320644e2
PP
365block>>, <<macro-expansion,macro expansion>>, and
366<<post-item-repetition,post-item repetition>> expression evaluation.
71aaa3f7
PP
367
368Each generated byte increments the current offset.
369
370A <<current-offset-setting,current offset setting>> may change the
676f6189
PP
371current offset without generating data.
372
373An <<current-offset-alignment,current offset alignment>> generates
374padding bytes to make the current offset satisfy a given alignment.
71aaa3f7
PP
375|`init_offset` parameter of the `parse()` function.
376|`--offset` option.
377
378|[[cur-bo]] Current byte order
379|
05f81895 380The current byte order has an effect on the encoding of
269f6eb3 381<<fixed-length-number,fixed-length numbers>>.
71aaa3f7
PP
382
383A <<current-byte-order-setting,current byte order setting>> may change
384the current byte order.
385|`init_byte_order` parameter of the `parse()` function.
386|`--byte-order` option.
387
388|<<label,Labels>>
389|Mapping of label names to integral values.
390|`init_labels` parameter of the `parse()` function.
391|One or more `--label` options.
392
393|<<variable-assignment,Variables>>
27d52a19 394|Mapping of variable names to integral or floating point number values.
71aaa3f7
PP
395|`init_variables` parameter of the `parse()` function.
396|One or more `--var` options.
397|===
398
399The available items are:
400
401* A <<byte-constant,constant integer>> representing a single byte.
402
403* A <<literal-string,literal string>> representing a sequence of bytes
404 encoding UTF-8, UTF-16, or UTF-32 data.
405
406* A <<current-byte-order-setting,current byte order setting>> (big or
407 little endian).
408
269f6eb3
PP
409* A <<fixed-length-number,fixed-length number>> (integer or
410 floating point) using the <<cur-bo,current byte order>> and of which
411 the value is the result of a {py3} expression.
05f81895
PP
412
413* An <<leb128-integer,LEB128 integer>> of which the value is the result
414 of a {py3} expression.
71aaa3f7
PP
415
416* A <<current-offset-setting,current offset setting>>.
417
676f6189
PP
418* A <<current-offset-alignment,current offset alignment>>.
419
71aaa3f7
PP
420* A <<label,label>>, that is, a named constant holding the current
421 offset.
422+
423This is similar to an assembly label.
424
425* A <<variable-assignment,variable assignment>> associating a name to
426 the integral result of an evaluated {py3} expression.
427
428* A <<group,group>>, that is, a scoped sequence of items.
429
27d52a19
PP
430* A <<conditional-block,conditional block>>.
431
e57a18e1
PP
432* A <<repetition-block,repetition block>>.
433
320644e2
PP
434* A <<macro-definition-block,macro definition block>>.
435
436* A <<macro-expansion,macro expansion>>.
437
e57a18e1
PP
438Moreover, you can repeat many items above a constant or variable number
439of times with the ``pass:[*]`` operator _after_ the item to repeat. This
440is called a <<post-item-repetition,post-item repetition>>.
71aaa3f7
PP
441
442A Normand comment may exist:
443
444* Between items, possibly within a group.
445* Between the nibbles of a constant hexadecimal byte.
446* Between the bits of a constant binary byte.
e57a18e1
PP
447* Between the last item and the ``pass:[*]`` character of a post-item
448 repetition, and between that ``pass:[*]`` character and the following
449 number or expression.
261c5ecf
PP
450* Between the ``!repeat``/``!r`` block opening and the following
451 constant integer, name, or expression of a repetition block.
452* Between the ``!if`` block opening and the following name or expression
453 of a conditional block.
71aaa3f7
PP
454
455A comment is anything between two ``pass:[#]`` characters on the same
456line, or from ``pass:[#]`` until the end of the line. Whitespaces and
457the following symbol characters are also considered comments where a
458comment may exist:
459
460----
e57a18e1 461/ \ ? & : ; . , + [ ] _ = | -
71aaa3f7
PP
462----
463
464The latter serve to improve readability so that you may write, for
465example, a MAC address or a UUID as is.
466
467You can test the examples of this section with the `normand`
468<<command-line-tool,command-line tool>> as such:
469
470----
471$ normand file | hexdump -C
472----
473
474where `file` is the name of a file containing the Normand input.
475
476=== Byte constant
477
478A _byte constant_ represents a single byte.
479
480A byte constant is:
481
482Hexadecimal form::
483 Two consecutive hexits.
484
485Decimal form::
486 A decimal number after the `$` prefix.
487
488Binary form::
489 Eight bits after the `%` prefix.
490
491====
492Input:
493
494----
495ab cd [3d 8F] CC
496----
497
498Output:
499
500----
501ab cd 3d 8f cc
502----
503====
504
505====
506Input:
507
508----
509$192 %1100/0011 $ -77
510----
511
512Output:
513
514----
515c0 c3 b3
516----
517====
518
519====
520Input:
521
522----
52358f64689-6316-4d55-8a1a-04cada366172
524fe80::6257:18ff:fea3:4229
525----
526
527Output:
528
529----
53058 f6 46 89 63 16 4d 55 8a 1a 04 ca da 36 61 72 ┆ X•F•c•MU•••••6ar
531fe 80 62 57 18 ff fe a3 42 29 ┆ ••bW••••B)
532----
533====
534
535====
536Input:
537
538----
539%01110011 %01100001 %01101100 %01110101 %01110100
540----
541
542Output:
543
544----
54573 61 6c 75 74 ┆ salut
546----
547====
548
549=== Literal string
550
551A _literal string_ represents the UTF-8-, UTF-16-, or UTF-32-encoded
552bytes of a string.
553
554The string to encode isn't implicitly null-terminated: use `\0` at the
555end of the string to add a null character.
556
557A literal string is:
558
559. **Optional**: one of the following encodings instead of UTF-8:
560+
561--
562[horizontal]
563`u16be`:: UTF-16BE.
564`u16le`:: UTF-16LE.
565`u32be`:: UTF-32BE.
566`u32le`:: UTF-32LE.
567--
568
569. The ``pass:["]`` prefix.
570
571. A sequence of zero or more characters, possibly containing escape
572 sequences.
573+
574An escape sequence is the ``\`` character followed by one of:
575+
576--
577[horizontal]
578`0`:: Null (U+0000)
579`a`:: Alert (U+0007)
580`b`:: Backspace (U+0008)
581`e`:: Escape (U+001B)
582`f`:: Form feed (U+000C)
583`n`:: End of line (U+000A)
584`r`:: Carriage return (U+000D)
585`t`:: Character tabulation (U+0009)
586`v`:: Line tabulation (U+000B)
587``\``:: Reverse solidus (U+005C)
588``pass:["]``:: Quotation mark (U+0022)
589--
590
591. The ``pass:["]`` suffix.
592
593====
594Input:
595
596----
597"coucou tout le monde!"
598----
599
600Output:
601
602----
60363 6f 75 63 6f 75 20 74 6f 75 74 20 6c 65 20 6d ┆ coucou tout le m
6046f 6e 64 65 21 ┆ onde!
605----
606====
607
608====
609Input:
610
611----
612u16le"I am not young enough to know everything."
613----
614
615Output:
616
617----
61849 00 20 00 61 00 6d 00 20 00 6e 00 6f 00 74 00 ┆ I• •a•m• •n•o•t•
61920 00 79 00 6f 00 75 00 6e 00 67 00 20 00 65 00 ┆ •y•o•u•n•g• •e•
6206e 00 6f 00 75 00 67 00 68 00 20 00 74 00 6f 00 ┆ n•o•u•g•h• •t•o•
62120 00 6b 00 6e 00 6f 00 77 00 20 00 65 00 76 00 ┆ •k•n•o•w• •e•v•
62265 00 72 00 79 00 74 00 68 00 69 00 6e 00 67 00 ┆ e•r•y•t•h•i•n•g•
6232e 00 ┆ .•
624----
625====
626
627====
628Input:
629
630----
631u32be "\"illusion is the first\nof all pleasures\" 🦉"
632----
633
634Output:
635
636----
63700 00 00 22 00 00 00 69 00 00 00 6c 00 00 00 6c ┆ •••"•••i•••l•••l
63800 00 00 75 00 00 00 73 00 00 00 69 00 00 00 6f ┆ •••u•••s•••i•••o
63900 00 00 6e 00 00 00 20 00 00 00 69 00 00 00 73 ┆ •••n••• •••i•••s
64000 00 00 20 00 00 00 74 00 00 00 68 00 00 00 65 ┆ ••• •••t•••h•••e
64100 00 00 20 00 00 00 66 00 00 00 69 00 00 00 72 ┆ ••• •••f•••i•••r
64200 00 00 73 00 00 00 74 00 00 00 0a 00 00 00 6f ┆ •••s•••t•••••••o
64300 00 00 66 00 00 00 20 00 00 00 61 00 00 00 6c ┆ •••f••• •••a•••l
64400 00 00 6c 00 00 00 20 00 00 00 70 00 00 00 6c ┆ •••l••• •••p•••l
64500 00 00 65 00 00 00 61 00 00 00 73 00 00 00 75 ┆ •••e•••a•••s•••u
64600 00 00 72 00 00 00 65 00 00 00 73 00 00 00 22 ┆ •••r•••e•••s•••"
64700 00 00 20 00 01 f9 89 ┆ ••• ••••
648----
649====
650
651=== Current byte order setting
652
653This special item sets the <<cur-bo,_current byte order_>>.
654
655The two accepted forms are:
656
657[horizontal]
658``pass:[{be}]``:: Set the current byte order to big endian.
659``pass:[{le}]``:: Set the current byte order to little endian.
660
269f6eb3 661=== Fixed-length number
71aaa3f7 662
269f6eb3
PP
663A _fixed-length number_ represents a fixed number of bytes encoding
664either:
665
666* An unsigned or signed integer (two's complement).
667+
668The available lengths are 8, 16, 24, 32, 40, 48, 56, and 64.
669
670* A floating point number
671 ([IEEE{nbsp}754-2008[https://standards.ieee.org/standard/754-2008.html]).
672+
673The available length are 32 (_binary32_) and 64 (_binary64_).
71aaa3f7 674
269f6eb3
PP
675The value is the result of evaluating a {py3} expression using the
676<<cur-bo,current byte order>>.
677
678A fixed-length number is:
71aaa3f7
PP
679
680. The ``pass:[{]`` prefix.
681
682. A valid {py3} expression.
05f81895 683+
269f6eb3 684For a fixed-length number at some source location{nbsp}__**L**__, this
05f81895
PP
685expression may contain the name of any accessible <<label,label>> (not
686within a nested group), including the name of a label defined
687after{nbsp}__**L**__, as well as the name of any
688<<variable-assignment,variable>> known at{nbsp}__**L**__.
689+
269f6eb3
PP
690The value of the special name `ICITTE` (`int` type) in this expression
691is the <<cur-offset,current offset>> (before encoding the number).
71aaa3f7
PP
692
693. The `:` character.
694
269f6eb3
PP
695. An encoding length in bits amongst:
696+
697--
27d52a19 698The expression evaluates to an `int` or `bool` value::
269f6eb3 699 `8`, `16`, `24`, `32`, `40`, `48`, `56`, and `64`.
27d52a19
PP
700+
701NOTE: Normand automatically converts a `bool` value to `int`.
269f6eb3
PP
702
703The expression evaluates to a `float` value::
704 `32` and `64`.
705--
71aaa3f7
PP
706
707. The `}` suffix.
708
709====
710Input:
711
712----
713{le} {345:16}
714{be} {-0xabcd:32}
715----
716
717Output:
718
719----
72059 01 ff ff 54 33
721----
722====
723
724====
725Input:
726
727----
728{be}
729
730# String length in bits
731{8 * (str_end - str_beg) : 16}
732
733# String
734<str_beg>
735 "hello world!"
736<str_end>
737----
738
739Output:
740
741----
74200 60 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 ┆ •`hello world!
743----
744====
745
746====
747Input:
748
749----
750{20 - ICITTE : 8} * 10
751----
752
753Output:
754
755----
75614 13 12 11 10 0f 0e 0d 0c 0b
757----
758====
759
269f6eb3
PP
760====
761Input:
762
763----
764{le}
765{2 * 0.0529 : 32}
766----
767
768Output:
769
770----
771ac ad d8 3d
772----
773====
774
05f81895
PP
775=== LEB128 integer
776
777An _LEB128 integer_ represents a variable number of bytes encoding an
778unsigned or signed integer which is the result of evaluating a {py3}
779expression following the https://en.wikipedia.org/wiki/LEB128[LEB128]
780format.
781
782An LEB128 integer is:
783
784. The ``pass:[{]`` prefix.
785
27d52a19
PP
786. A valid {py3} expression of which the evaluation result type
787 is `int` or `bool` (automatically converted to `int`).
05f81895
PP
788+
789For an LEB128 integer at some source location{nbsp}__**L**__, this
790expression may contain:
791+
792--
793* The name of any <<label,label>> defined before{nbsp}__**L**__.
320644e2
PP
794* The name of any <<variable-assignment,variable>> known
795 at{nbsp}__**L**__.
05f81895
PP
796--
797+
269f6eb3
PP
798The value of the special name `ICITTE` (`int` type) in this expression
799is the <<cur-offset,current offset>> (before encoding the integer).
05f81895
PP
800
801. The `:` character.
802
803. One of:
804+
805--
806[horizontal]
807`uleb128`:: Use the unsigned LEB128 format.
808`sleb128`:: Use the signed LEB128 format.
809--
810
811. The `}` suffix.
812
813====
814Input:
815
816----
817{624485 : uleb128}
818----
819
820Output:
821
822----
823e5 8e 26
824----
825====
826
827====
828Input:
829
830----
831aa bb cc dd
832<meow>
833ee ff
834{-981238311 + (meow * -23) : sleb128}
835"hello"
836----
837
c2b79cf6
PP
838Output:
839
05f81895
PP
840----
841aa bb cc dd ee ff fd fa 8d ac 7c 68 65 6c 6c 6f ┆ ••••••••••|hello
842----
843====
844
71aaa3f7
PP
845=== Current offset setting
846
847This special item sets the <<cur-offset,_current offset_>>.
848
849A current offset setting is:
850
851. The `<` prefix.
852
853. A positive integer (hexadecimal starting with `0x` or `0X` accepted)
854 which is the new current offset.
855
856. The `>` suffix.
857
858====
859Input:
860
861----
862 {ICITTE : 8} * 8
863<0x61> {ICITTE : 8} * 8
864----
865
866Output:
867
868----
86900 01 02 03 04 05 06 07 61 62 63 64 65 66 67 68 ┆ ••••••••abcdefgh
870----
871====
872
873====
874Input:
875
876----
877aa bb cc dd <meow> ee ff
878<12> 11 22 33 <mix> 44 55
879{meow : 8} {mix : 8}
880----
881
882Output:
883
884----
885aa bb cc dd ee ff 11 22 33 44 55 04 0f ┆ •••••••"3DU••
886----
887====
888
676f6189
PP
889=== Current offset alignment
890
00deb9fa 891A _current offset alignment_ represents zero or more padding bytes to
676f6189
PP
892make the <<cur-offset,current offset>> meet a given
893https://en.wikipedia.org/wiki/Data_structure_alignment[alignment] value.
894
895More specifically, for an alignment value of{nbsp}__**N**__{nbsp}bits,
896a current offset alignment represents the required padding bytes until
897the current offset is a multiple of __**N**__{nbsp}/{nbsp}8.
898
899A current offset alignment is:
900
901. The `@` prefix.
902
903. A positive integer (hexadecimal starting with `0x` or `0X` accepted)
904 which is the alignment value in _bits_.
905+
906This value must be greater than zero and a multiple of{nbsp}8.
907
908. **Optional**:
909+
910--
911. The ``pass:[~]`` prefix.
912. A positive integer (hexadecimal starting with `0x` or `0X` accepted)
913 which is the value of the byte to use as padding to align the
914 <<cur-offset,current offset>>.
915--
916+
917Without this section, the padding byte value is zero.
918
919====
920Input:
921
922----
92311 22 (@32 aa bb cc) * 3
924----
925
926Output:
927
928----
92911 22 00 00 aa bb cc 00 aa bb cc 00 aa bb cc
930----
931====
932
933====
934Input:
935
936----
937{le}
93877 88
939@32~0xcc {-893.5:32}
940@128~0x55 "meow"
941----
942
943Output:
944
945----
94677 88 cc cc 00 60 5f c4 55 55 55 55 55 55 55 55 ┆ w••••`_•UUUUUUUU
9476d 65 6f 77 ┆ meow
948----
949====
950
951====
952Input:
953
954----
955aa bb cc <29> @64~255 "zoom"
956----
957
958Output:
959
960----
961aa bb cc ff ff ff 7a 6f 6f 6d ┆ ••••••zoom
962----
963====
964
71aaa3f7
PP
965=== Label
966
967A _label_ associates a name to the <<cur-offset,current offset>>.
968
969All the labels of a whole Normand input must have unique names.
970
05f81895 971A label must not share the name of a <<variable-assignment,variable>>
71aaa3f7
PP
972name.
973
71aaa3f7
PP
974A label is:
975
976. The `<` prefix.
977
27d52a19 978. A valid {py3} name which is not `ICITTE`.
71aaa3f7
PP
979
980. The `>` suffix.
981
982=== Variable assignment
983
984A _variable assignment_ associates a name to the integral result of an
985evaluated {py3} expression.
986
05f81895 987A variable assignment is:
71aaa3f7
PP
988
989. The ``pass:[{]`` prefix.
990
27d52a19 991. A valid {py3} name which is not `ICITTE`.
71aaa3f7
PP
992
993. The `=` character.
994
27d52a19
PP
995. A valid {py3} expression of which the evaluation result type
996 is `int`, `float`, or `bool` (automatically converted to `int`).
05f81895
PP
997+
998For a variable assignment at some source location{nbsp}__**L**__, this
320644e2
PP
999expression may contain:
1000+
1001--
1002* The name of any <<label,label>> defined before{nbsp}__**L**__
1003 which isn't within a nested group.
1004* The name of any <<variable-assignment,variable>> known
1005 at{nbsp}__**L**__.
1006--
05f81895 1007+
269f6eb3
PP
1008The value of the special name `ICITTE` (`int` type) in this expression
1009is the <<cur-offset,current offset>>.
71aaa3f7
PP
1010
1011. The `}` suffix.
1012
1013====
1014Input:
1015
1016----
1017{mix = 101} {le}
1018{meow = 42} 11 22 {meow:8} 33 {meow = ICITTE + 17}
1019"yooo" {meow + mix : 16}
1020----
1021
1022Output:
1023
1024----
102511 22 2a 33 79 6f 6f 6f 7a 00 ┆ •"*3yoooz•
1026----
1027====
1028
1029=== Group
1030
1031A _group_ is a scoped sequence of items.
1032
1033The <<label,labels>> within a group aren't visible outside of it.
1034
e57a18e1
PP
1035The main purpose of a group is to <<post-item-repetition,repeat>> more
1036than a single item and to isolate labels.
71aaa3f7
PP
1037
1038A group is:
1039
261c5ecf 1040. The `(`, `!group`, or `!g` opening.
71aaa3f7
PP
1041
1042. Zero or more items.
1043
261c5ecf
PP
1044. Depending on the group opening:
1045+
1046--
1047`(`::
1048 The `)` closing.
1049
1050`!group`::
1051`!g`::
1052 The `!end` closing.
1053--
71aaa3f7
PP
1054
1055====
1056Input:
1057
1058----
1059((aa bb cc) dd () ee) "leclerc"
1060----
1061
1062Output:
1063
1064----
1065aa bb cc dd ee 6c 65 63 6c 65 72 63 ┆ •••••leclerc
1066----
1067====
1068
1069====
1070Input:
1071
1072----
261c5ecf
PP
1073!group
1074 (aa bb cc) * 3 dd ee
1075!end * 5
71aaa3f7
PP
1076----
1077
1078Output:
1079
1080----
1081aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa bb
1082cc aa bb cc dd ee aa bb cc aa bb cc aa bb cc dd
1083ee aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa
1084bb cc aa bb cc dd ee
1085----
1086====
1087
1088====
1089Input:
1090
1091----
1092{be}
1093(
1094 <str_beg> u16le"sébastien diaz" <str_end>
1095 {ICITTE - str_beg : 8}
1096 {(end - str_beg) * 5 : 24}
1097) * 3
1098<end>
1099----
1100
1101Output:
1102
1103----
110473 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
11056e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 e0 ┆ n• •d•i•a•z•••••
110673 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
11076e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 40 ┆ n• •d•i•a•z••••@
110873 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
11096e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 00 a0 ┆ n• •d•i•a•z•••••
1110----
1111====
1112
27d52a19
PP
1113=== Conditional block
1114
1115A _conditional block_ represents either the bytes of one or more items
1116if some expression is true, or no bytes at all if it's false.
1117
1118A conditional block is:
1119
261c5ecf 1120. The `!if` opening.
27d52a19
PP
1121
1122. One of:
1123
1124** The ``pass:[{]`` prefix, a valid {py3} expression of which the
1125 evaluation result type is `int` or `bool` (automatically converted to
1126 `int`), and the ``pass:[}]`` suffix.
1127+
320644e2
PP
1128For a conditional block at some source location{nbsp}__**L**__, this
1129expression may contain:
27d52a19
PP
1130+
1131--
1132* The name of any <<label,label>> defined before{nbsp}__**L**__
1133 which isn't within a nested group.
1134* The name of any <<variable-assignment,variable>> known
320644e2 1135 at{nbsp}__**L**__.
27d52a19
PP
1136--
1137+
1138The value of the special name `ICITTE` (`int` type) in this expression
1139is the <<cur-offset,current offset>> (before handling the contained
1140items).
1141
1142** A valid {py3} name.
1143+
1144For the name `__NAME__`, this is equivalent to the
1145`pass:[{]__NAME__pass:[}]` form above.
1146
1147. Zero or more items.
1148
261c5ecf 1149. The `!end` closing.
27d52a19
PP
1150
1151====
1152Input:
1153
1154----
1155{at = 1}
1156{rep_count = 9}
1157
1158!repeat rep_count
1159 "meow "
1160
1161 !if {ICITTE > 25}
1162 "mix"
1163
1164 !if {at < rep_count} 20 !end
1165 !end
1166
1167 {at = at + 1}
1168!end
1169----
1170
1171Output:
1172
1173----
11746d 65 6f 77 20 6d 65 6f 77 20 6d 65 6f 77 20 6d ┆ meow meow meow m
117565 6f 77 20 6d 65 6f 77 20 6d 65 6f 77 20 6d 69 ┆ eow meow meow mi
117678 20 6d 65 6f 77 20 6d 69 78 20 6d 65 6f 77 20 ┆ x meow mix meow
11776d 69 78 20 6d 65 6f 77 20 6d 69 78 ┆ mix meow mix
1178----
1179====
1180
1181====
1182Input:
1183
1184----
1185<str_beg>
1186u16le"meow mix!"
1187<str_end>
1188
1189!if {str_end - str_beg > 10}
1190 " BIG"
1191!end
1192----
1193
1194Output:
1195
1196----
11976d 00 65 00 6f 00 77 00 20 00 6d 00 69 00 78 00 ┆ m•e•o•w• •m•i•x•
119821 00 20 42 49 47 ┆ !• BIG
1199----
1200====
1201
e57a18e1 1202=== Repetition block
71aaa3f7 1203
e57a18e1
PP
1204A _repetition block_ represents the bytes of one or more items repeated
1205a given number of times.
676f6189 1206
e57a18e1 1207A repetition block is:
71aaa3f7 1208
261c5ecf 1209. The `!repeat` or `!r` opening.
71aaa3f7 1210
2adf4336
PP
1211. One of:
1212
1213** A positive integer (hexadecimal starting with `0x` or `0X` accepted)
1214 which is the number of times to repeat the previous item.
1215
27d52a19
PP
1216** The ``pass:[{]`` prefix, a valid {py3} expression of which the
1217 evaluation result type is `int` or `bool` (automatically converted to
1218 `int`), and the ``pass:[}]`` suffix.
05f81895 1219+
320644e2
PP
1220For a repetition block at some source location{nbsp}__**L**__, this
1221expression may contain:
05f81895
PP
1222+
1223--
27d52a19
PP
1224* The name of any <<label,label>> defined before{nbsp}__**L**__
1225 which isn't within a nested group.
05f81895 1226* The name of any <<variable-assignment,variable>> known
320644e2 1227 at{nbsp}__**L**__.
05f81895
PP
1228--
1229+
e57a18e1
PP
1230The value of the special name `ICITTE` (`int` type) in this expression
1231is the <<cur-offset,current offset>> (before handling the items to
1232repeat).
1233
1234** A valid {py3} name.
1235+
1236For the name `__NAME__`, this is equivalent to the
1237`pass:[{]__NAME__pass:[}]` form above.
1238
1239. Zero or more items.
1240
261c5ecf 1241. The `!end` closing.
e57a18e1
PP
1242
1243You may also use a <<post-item-repetition,post-item repetition>> after
1244some items. The form ``!repeat{nbsp}__X__{nbsp}__ITEMS__{nbsp}!end``
1245is equivalent to ``(__ITEMS__){nbsp}pass:[*]{nbsp}__X__``.
71aaa3f7
PP
1246
1247====
1248Input:
1249
1250----
e57a18e1
PP
1251!repeat 0x100
1252 {end - ICITTE - 1 : 8}
1253!end
1254
1255<end>
71aaa3f7
PP
1256----
1257
1258Output:
1259
1260----
1261ff fe fd fc fb fa f9 f8 f7 f6 f5 f4 f3 f2 f1 f0 ┆ ••••••••••••••••
1262ef ee ed ec eb ea e9 e8 e7 e6 e5 e4 e3 e2 e1 e0 ┆ ••••••••••••••••
1263df de dd dc db da d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 ┆ ••••••••••••••••
1264cf ce cd cc cb ca c9 c8 c7 c6 c5 c4 c3 c2 c1 c0 ┆ ••••••••••••••••
1265bf be bd bc bb ba b9 b8 b7 b6 b5 b4 b3 b2 b1 b0 ┆ ••••••••••••••••
1266af ae ad ac ab aa a9 a8 a7 a6 a5 a4 a3 a2 a1 a0 ┆ ••••••••••••••••
12679f 9e 9d 9c 9b 9a 99 98 97 96 95 94 93 92 91 90 ┆ ••••••••••••••••
12688f 8e 8d 8c 8b 8a 89 88 87 86 85 84 83 82 81 80 ┆ ••••••••••••••••
12697f 7e 7d 7c 7b 7a 79 78 77 76 75 74 73 72 71 70 ┆ •~}|{zyxwvutsrqp
12706f 6e 6d 6c 6b 6a 69 68 67 66 65 64 63 62 61 60 ┆ onmlkjihgfedcba`
12715f 5e 5d 5c 5b 5a 59 58 57 56 55 54 53 52 51 50 ┆ _^]\[ZYXWVUTSRQP
12724f 4e 4d 4c 4b 4a 49 48 47 46 45 44 43 42 41 40 ┆ ONMLKJIHGFEDCBA@
12733f 3e 3d 3c 3b 3a 39 38 37 36 35 34 33 32 31 30 ┆ ?>=<;:9876543210
12742f 2e 2d 2c 2b 2a 29 28 27 26 25 24 23 22 21 20 ┆ /.-,+*)('&%$#"!
12751f 1e 1d 1c 1b 1a 19 18 17 16 15 14 13 12 11 10 ┆ ••••••••••••••••
12760f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 00 ┆ ••••••••••••••••
1277----
1278====
1279
2adf4336
PP
1280====
1281Input:
1282
1283----
1284{times = 1}
e57a18e1 1285
2adf4336 1286aa bb cc dd
e57a18e1
PP
1287
1288!repeat 3
2adf4336 1289 <here>
e57a18e1
PP
1290
1291 !repeat {here + 1}
1292 ee ff
1293 !end
1294
1295 11 22 !repeat times 33 !end
1296
2adf4336 1297 {times = times + 1}
e57a18e1
PP
1298!end
1299
2adf4336
PP
1300"coucou!"
1301----
1302
1303Output:
1304
1305----
1306aa bb cc dd ee ff ee ff ee ff ee ff ee ff 11 22 ┆ •••••••••••••••"
130733 ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ 3•••••••••••••••
1308ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1309ff ee ff ee ff 11 22 33 33 ee ff ee ff ee ff ee ┆ ••••••"33•••••••
1310ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1311ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1312ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1313ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1314ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1315ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1316ff ee ff ee ff ee ff ee ff ee ff ee ff 11 22 33 ┆ ••••••••••••••"3
131733 33 63 6f 75 63 6f 75 21 ┆ 33coucou!
1318----
1319====
1320
320644e2
PP
1321=== Macro definition block
1322
1323A _macro definition block_ associates a name and parameter names to
1324a group of items.
1325
1326A macro definition block doesn't lead to generated bytes itself: a
1327<<macro-expansion,macro expansion>> does so.
1328
1329A macro definition may only exist at the root level, that is, not within
1330a <<group,group>>, a <<repetition-block,repetition block>>, a
1331<<conditional-block,conditional block>>, or another
1332<<macro-definition-block,macro definition block>>.
1333
1334All macro definitions must have unique names.
1335
1336A macro definition is:
1337
1338. The `!macro` or `!m` opening.
1339
1340. A valid {py3} name (the macro name).
1341
1342. The `(` parameter name list prefix.
1343
1344. A comma-separated list of zero or more unique parameter names,
1345 each one being a valid {py3} name.
1346
1347. The `)` parameter name list suffix.
1348
1349. Zero or more items except, recursively, a macro definition block.
1350
1351. The `!end` closing.
1352
1353====
1354----
1355!macro bake()
1356 {le} {ICITTE * 8 : 16}
1357 u16le"predict explode"
1358!end
1359----
1360====
1361
1362====
1363----
1364!macro nail(rep, with_extra, val)
1365 {iter = 1}
1366
1367 !repeat rep
1368 {val + iter : uleb128}
1369 {0xdeadbeef : 32}
1370 {iter = iter + 1}
1371 !end
1372
1373 !if with_extra
1374 "meow mix\0"
1375 !end
1376!end
1377----
1378====
1379
1380=== Macro expansion
1381
1382A _macro expansion_ expands the items of a defined
1383<<macro-definition-block,macro>>.
1384
1385The macro to expand must be defined _before_ the expansion.
1386
1387The <<state,state>> before handling the first item of the chosen macro
1388is:
1389
1390<<cur-offset,Current offset>>::
1391 Unchanged.
1392
1393<<cur-bo,Current byte order>>::
1394 Unchanged.
1395
1396Variables::
1397 The only available variables initially are the macro parameters.
1398
1399Labels::
1400 None.
1401
1402The state after having handled the last item of the chosen macro is:
1403
1404Current offset::
1405 The one before handling the first item of the macro plus the size
1406 of the generated data of the macro expansion.
1407+
1408IMPORTANT: This means <<current-offset-setting,current offset setting>>
1409items within the expanded macro don't impact the final current offset.
1410
1411Current byte order::
1412 The one before handling the first item of the macro.
1413
1414Variables::
1415 The ones before handling the first item of the macro.
1416
1417Labels::
1418 The ones before handling the first item of the macro.
1419
1420A macro expansion is:
1421
1422. The `m:` prefix.
1423
1424. A valid {py3} name (the name of the macro to expand).
1425
1426. The `(` parameter value list prefix.
1427
1428. A comma-separated list of zero or more unique parameter values.
1429+
1430The number of parameter values must match the number of parameter
1431names of the definition of the chosen macro.
1432+
1433A parameter value is one of:
1434+
1435--
1436* A positive integer (hexadecimal starting with `0x` or `0X` accepted).
1437
1438* The ``pass:[{]`` prefix, a valid {py3} expression of which the
1439 evaluation result type is `int` or `bool` (automatically converted to
1440 `int`), and the ``pass:[}]`` suffix.
1441+
1442For a macro expansion at some source location{nbsp}__**L**__, this
1443expression may contain:
1444
1445** The name of any <<label,label>> defined before{nbsp}__**L**__
1446 which isn't within a nested group.
1447** The name of any <<variable-assignment,variable>> known
1448 at{nbsp}__**L**__.
1449
1450+
1451The value of the special name `ICITTE` (`int` type) in this expression
1452is the <<cur-offset,current offset>> (before handling the items of the
1453chosen macro).
1454
1455* A valid {py3} name.
1456+
1457For the name `__NAME__`, this is equivalent to the
1458`pass:[{]__NAME__pass:[}]` form above.
1459--
1460
1461. The `)` parameter value list suffix.
1462
1463====
1464Input:
1465
1466----
1467!macro bake()
1468 {le} {ICITTE * 8 : 16}
1469 u16le"predict explode"
1470!end
1471
1472"hello [" m:bake() "] world"
1473
1474m:bake() * 5
1475----
1476
1477Output:
1478
1479----
148068 65 6c 6c 6f 20 5b 38 00 70 00 72 00 65 00 64 ┆ hello [8•p•r•e•d
148100 69 00 63 00 74 00 20 00 65 00 78 00 70 00 6c ┆ •i•c•t• •e•x•p•l
148200 6f 00 64 00 65 00 5d 20 77 6f 72 6c 64 70 01 ┆ •o•d•e•] worldp•
148370 00 72 00 65 00 64 00 69 00 63 00 74 00 20 00 ┆ p•r•e•d•i•c•t• •
148465 00 78 00 70 00 6c 00 6f 00 64 00 65 00 70 02 ┆ e•x•p•l•o•d•e•p•
148570 00 72 00 65 00 64 00 69 00 63 00 74 00 20 00 ┆ p•r•e•d•i•c•t• •
148665 00 78 00 70 00 6c 00 6f 00 64 00 65 00 70 03 ┆ e•x•p•l•o•d•e•p•
148770 00 72 00 65 00 64 00 69 00 63 00 74 00 20 00 ┆ p•r•e•d•i•c•t• •
148865 00 78 00 70 00 6c 00 6f 00 64 00 65 00 70 04 ┆ e•x•p•l•o•d•e•p•
148970 00 72 00 65 00 64 00 69 00 63 00 74 00 20 00 ┆ p•r•e•d•i•c•t• •
149065 00 78 00 70 00 6c 00 6f 00 64 00 65 00 70 05 ┆ e•x•p•l•o•d•e•p•
149170 00 72 00 65 00 64 00 69 00 63 00 74 00 20 00 ┆ p•r•e•d•i•c•t• •
149265 00 78 00 70 00 6c 00 6f 00 64 00 65 00 ┆ e•x•p•l•o•d•e•
1493----
1494====
1495
1496====
1497Input:
1498
1499----
1500!macro A(val, is_be)
1501 {le}
1502
1503 !if is_be
1504 {be}
1505 !end
1506
1507 {val : 16}
1508!end
1509
1510!macro B(rep, is_be)
1511 {iter = 1}
1512
1513 !repeat rep
1514 m:A({iter * 3}, is_be)
1515 {iter = iter + 1}
1516 !end
1517!end
1518
1519m:B(5, 1)
1520m:B(3, 0)
1521----
1522
1523Output:
1524
1525----
152600 03 00 06 00 09 00 0c 00 0f 03 00 06 00 09 00
1527----
1528====
1529
e57a18e1
PP
1530=== Post-item repetition
1531
1532A _post-item repetition_ represents the bytes of an item repeated a
1533given number of times.
1534
1535A post-item repetition is:
1536
27d52a19 1537. One of those items:
e57a18e1 1538
27d52a19
PP
1539** A <<byte-constant,byte constant>>.
1540** A <<literal-string,literal string>>.
1541** A <<fixed-length-number,fixed-length number>>.
1542** An <<leb128-integer,LEB128 integer>>.
320644e2 1543** A <<macro-expansion,macro-expansion>>.
27d52a19 1544** A <<group,group>>.
e57a18e1
PP
1545
1546. The ``pass:[*]`` character.
1547
1548. One of:
1549
1550** A positive integer (hexadecimal starting with `0x` or `0X` accepted)
1551 which is the number of times to repeat the previous item.
1552
27d52a19
PP
1553** The ``pass:[{]`` prefix, a valid {py3} expression of which the
1554 evaluation result type is `int` or `bool` (automatically converted to
1555 `int`), and the ``pass:[}]`` suffix.
e57a18e1 1556+
320644e2
PP
1557For a post-item repetition at some source location{nbsp}__**L**__, this
1558expression may contain:
e57a18e1
PP
1559+
1560--
27d52a19
PP
1561* The name of any <<label,label>> defined before{nbsp}__**L**__
1562 which isn't within a nested group and
1563 which isn't part of the repeated item.
e57a18e1
PP
1564* The name of any <<variable-assignment,variable>> known
1565 at{nbsp}__**L**__, which isn't part of its repeated item, and which
320644e2 1566 doesn't.
e57a18e1
PP
1567--
1568+
1569The value of the special name `ICITTE` (`int` type) in this expression
1570is the <<cur-offset,current offset>> (before handling the items to
1571repeat).
1572
1573** A valid {py3} name.
1574+
1575For the name `__NAME__`, this is equivalent to the
1576`pass:[{]__NAME__pass:[}]` form above.
1577
1578You may also use a <<repetition-block,repetition block>>. The form
1579``__ITEM__{nbsp}pass:[*]{nbsp}__X__`` is equivalent to
1580``!repeat{nbsp}__X__{nbsp}__ITEM__{nbsp}!end``.
1581
1582====
1583Input:
1584
1585----
1586{end - ICITTE - 1 : 8} * 0x100 <end>
1587----
1588
1589Output:
1590
1591----
1592ff fe fd fc fb fa f9 f8 f7 f6 f5 f4 f3 f2 f1 f0 ┆ ••••••••••••••••
1593ef ee ed ec eb ea e9 e8 e7 e6 e5 e4 e3 e2 e1 e0 ┆ ••••••••••••••••
1594df de dd dc db da d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 ┆ ••••••••••••••••
1595cf ce cd cc cb ca c9 c8 c7 c6 c5 c4 c3 c2 c1 c0 ┆ ••••••••••••••••
1596bf be bd bc bb ba b9 b8 b7 b6 b5 b4 b3 b2 b1 b0 ┆ ••••••••••••••••
1597af ae ad ac ab aa a9 a8 a7 a6 a5 a4 a3 a2 a1 a0 ┆ ••••••••••••••••
15989f 9e 9d 9c 9b 9a 99 98 97 96 95 94 93 92 91 90 ┆ ••••••••••••••••
15998f 8e 8d 8c 8b 8a 89 88 87 86 85 84 83 82 81 80 ┆ ••••••••••••••••
16007f 7e 7d 7c 7b 7a 79 78 77 76 75 74 73 72 71 70 ┆ •~}|{zyxwvutsrqp
16016f 6e 6d 6c 6b 6a 69 68 67 66 65 64 63 62 61 60 ┆ onmlkjihgfedcba`
16025f 5e 5d 5c 5b 5a 59 58 57 56 55 54 53 52 51 50 ┆ _^]\[ZYXWVUTSRQP
16034f 4e 4d 4c 4b 4a 49 48 47 46 45 44 43 42 41 40 ┆ ONMLKJIHGFEDCBA@
16043f 3e 3d 3c 3b 3a 39 38 37 36 35 34 33 32 31 30 ┆ ?>=<;:9876543210
16052f 2e 2d 2c 2b 2a 29 28 27 26 25 24 23 22 21 20 ┆ /.-,+*)('&%$#"!
16061f 1e 1d 1c 1b 1a 19 18 17 16 15 14 13 12 11 10 ┆ ••••••••••••••••
16070f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 00 ┆ ••••••••••••••••
1608----
1609====
1610
1611====
1612Input:
1613
1614----
1615{times = 1}
1616aa bb cc dd
1617(
1618 <here>
1619 (ee ff) * {here + 1}
1620 11 22 33 * {times}
1621 {times = times + 1}
1622) * 3
1623"coucou!"
1624----
1625
1626Output:
1627
1628----
1629aa bb cc dd ee ff ee ff ee ff ee ff ee ff 11 22 ┆ •••••••••••••••"
163033 ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ 3•••••••••••••••
1631ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1632ff ee ff ee ff 11 22 33 33 ee ff ee ff ee ff ee ┆ ••••••"33•••••••
1633ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1634ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1635ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1636ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1637ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1638ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1639ff ee ff ee ff ee ff ee ff ee ff ee ff 11 22 33 ┆ ••••••••••••••"3
164033 33 63 6f 75 63 6f 75 21 ┆ 33coucou!
1641----
1642====
1643
71aaa3f7
PP
1644== Command-line tool
1645
1646If you <<install-normand,installed>> the `normand` package, then you
1647can use the `normand` command-line tool:
1648
1649----
1650$ normand <<< '"ma gang de malades"' | hexdump -C
1651----
1652
1653----
165400000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
165500000010 65 73 |es|
1656----
1657
1658If you copy the `normand.py` module to your own project, then you can
1659run the module itself:
1660
1661----
1662$ python3 -m normand <<< '"ma gang de malades"' | hexdump -C
1663----
1664
1665----
166600000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
166700000010 65 73 |es|
1668----
1669
1670Without a path argument, the `normand` tool reads from the standard
1671input.
1672
1673The `normand` tool prints the generated binary data to the standard
1674output.
1675
1676Various options control the initial <<state,state>> of the processor:
1677use the `--help` option to learn more.
1678
1679== {py3} API
1680
e57a18e1 1681The whole `normand` package/module public API is:
71aaa3f7
PP
1682
1683[source,python]
1684----
e57a18e1 1685# Byte order.
71aaa3f7
PP
1686class ByteOrder(enum.Enum):
1687 # Big endian.
1688 BE = ...
1689
1690 # Little endian.
1691 LE = ...
1692
1693
e57a18e1
PP
1694# Text location.
1695class TextLocation:
71aaa3f7
PP
1696 # Line number.
1697 @property
1698 def line_no(self) -> int:
1699 ...
1700
1701 # Column number.
1702 @property
1703 def col_no(self) -> int:
1704 ...
1705
1706
e57a18e1 1707# Parsing error.
71aaa3f7
PP
1708class ParseError(RuntimeError):
1709 # Source text location.
1710 @property
e57a18e1 1711 def text_loc(self) -> TextLocation:
71aaa3f7
PP
1712 ...
1713
1714
e57a18e1
PP
1715# Variables dictionary type (for type hints).
1716VariablesT = typing.Dict[str, typing.Union[int, float]]
1717
1718
1719# Labels dictionary type (for type hints).
1720LabelsT = typing.Dict[str, int]
1b8aa84a
PP
1721
1722
e57a18e1 1723# Parsing result.
71aaa3f7
PP
1724class ParseResult:
1725 # Generated data.
1726 @property
1727 def data(self) -> bytearray:
1728 ...
1729
1730 # Updated variable values.
1731 @property
1b8aa84a 1732 def variables(self) -> SymbolsT:
71aaa3f7
PP
1733 ...
1734
1735 # Updated main group label values.
1736 @property
1b8aa84a 1737 def labels(self) -> SymbolsT:
71aaa3f7
PP
1738 ...
1739
1740 # Final offset.
1741 @property
1742 def offset(self) -> int:
1743 ...
1744
1745 # Final byte order.
1746 @property
1b8aa84a 1747 def byte_order(self) -> typing.Optional[ByteOrder]:
71aaa3f7
PP
1748 ...
1749
1b8aa84a 1750
e57a18e1
PP
1751# Parses the `normand` input using the initial state defined by
1752# `init_variables`, `init_labels`, `init_offset`, and `init_byte_order`,
1753# and returns the corresponding parsing result.
71aaa3f7 1754def parse(normand: str,
1b8aa84a
PP
1755 init_variables: typing.Optional[SymbolsT] = None,
1756 init_labels: typing.Optional[SymbolsT] = None,
71aaa3f7
PP
1757 init_offset: int = 0,
1758 init_byte_order: typing.Optional[ByteOrder] = None) -> ParseResult:
1759 ...
1760----
1761
1762The `normand` parameter is the actual <<learn-normand,Normand input>>
1763while the other parameters control the initial <<state,state>>.
1764
1765The `parse()` function raises a `ParseError` instance should it fail to
1766parse the `normand` string for any reason.
bf8f3b38
PP
1767
1768== Development
1769
1770Normand is a https://python-poetry.org/[Poetry] project.
1771
1772To develop it, install it through Poetry and enter the virtual
1773environment:
1774
1775----
1776$ poetry install
1777$ poetry shell
1778$ normand <<< '"lol" * 10 0a'
1779----
1780
1781`normand.py` is processed by:
1782
1783* https://microsoft.github.io/pyright/[Pyright]
1784* https://github.com/psf/black[Black]
1785* https://pycqa.github.io/isort/[isort]
1786
1787=== Testing
1788
1789Use https://docs.pytest.org/[pytest] to test Normand once the package is
1790part of your virtual environment, for example:
1791
1792----
1793$ poetry install
1794$ poetry run pip3 install pytest
1795$ poetry run pytest
1796----
1797
1798The `pytest` project is currently not a development dependency in
1799`pyproject.toml` due to backward compatibiliy issues with
1800Python{nbsp}3.4.
1801
1802In the `tests` directory, each `*.nt` file is a test. The file name
1803prefix indicates what it's meant to test:
1804
1805`pass-`::
1806 Everything above the `---` line is the valid Normand input
1807 to test.
1808+
1809Everything below the `---` line is the expected data
1810(whitespace-separated hexadecimal bytes).
1811
1812`fail-`::
1813 Everything above the `---` line is the invalid Normand input
1814 to test.
1815+
1816Everything below the `---` line is the expected error message having
1817this form:
1818+
1819----
1820LINE:COL - MESSAGE
1821----
1822
1823=== Contributing
1824
1825Normand uses https://review.lttng.org/admin/repos/normand,general[Gerrit]
1826for code review.
1827
1828To report a bug, https://github.com/efficios/normand/issues/new[create a
1829GitHub issue].
This page took 0.091944 seconds and 4 git commands to generate.