1 // Show ToC at a specific location for a GitHub rendering
10 // This is to mimic what GitHub does so that anchors work in an offline
21 image::normand-logo.png[]
24 image:https://img.shields.io/pypi/v/normand.svg?label=Latest%20version[link="https://pypi.python.org/pypi/normand"]
27 _**Normand**_ is a text-to-binary processor with its own language.
29 This package offers both a portable {py3} module and a command-line
32 WARNING: This version of Normand is 0.10, meaning both the Normand
33 language and the module/CLI interface aren't stable.
36 // ToC location for a GitHub rendering
42 The purpose of Normand is to consume human-readable text representing
43 bytes and to produce the corresponding binary data.
47 Consider the following Normand input:
50 4f 55 32 bb $167 fe %10100111 a9 $-32
53 The generated nine bytes are:
56 4f 55 32 bb a7 fe a7 a9 e0
60 As you can see in the last example, the fundamental unit of the Normand
61 language is the _byte_. The order in which you list bytes will be the
62 order of the generated data.
64 The Normand language is more than simple lists of bytes, though. Its
67 Comments, including a bunch of insignificant symbols which may improve readability::
72 ff bb %1101:0010 # This is a comment
73 78 29 af $192 # This too # 99 $-80
74 fe80::6257:18ff:fea3:4229
76 10839636-5d65-4a68-8e6a-21608ddf7258
82 ff bb d2 78 29 af c0 99 b0 fe 80 62 57 18 ff fe
83 a3 42 29 60 57 18 a3 42 29 10 83 96 36 5d 65 4a
84 68 8e 6a 21 60 8d df 72 58
87 Hexadecimal, decimal, and binary byte constants::
92 aa bb $247 $-89 %0011_0010 %11.01= 10/10
101 UTF-8, UTF-16, and UTF-32 literal strings::
107 u16le"stress\nverdict 🤣"
113 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 00 73 00 74 ┆ hello world!•s•t
114 00 72 00 65 00 73 00 73 00 0a 00 76 00 65 00 72 ┆ •r•e•s•s•••v•e•r
115 00 64 00 69 00 63 00 74 00 20 00 3e d8 23 dd ┆ •d•i•c•t• •>•#•
118 Labels: special variables holding the offset where they're defined::
121 <beg> b2 52 e3 bc 91 05
122 $100 $50 <chair> 33 9f fe
129 5e 65 {tower = 47} c6 7f f2 c4
130 44 {hurl = tower - 14} b5 {tower = hurl} 26 2d
133 The value of a variable assignment is the evaluation of a valid {py3}
134 expression which may include label and variable names.
136 Fixed-length number with a given length (8{nbsp}bits to 64{nbsp}bits) and byte order::
142 {be} 67 <lbl> 44 $178 {(end - lbl) * 8 + strength : 16} $99 <end>
150 67 44 b2 00 2c 63 37 f8 ff ff 7f bd c2 82 fb 21
154 The encoded number is the evaluation of a valid {py3} expression which
155 may include label and variable names.
157 https://en.wikipedia.org/wiki/LEB128[LEB128] integer::
162 aa bb cc {-1993 : sleb128} <meow> dd ee ff
163 {meow * 199 : uleb128}
169 aa bb cc b7 70 dd ee ff e3 07
172 The encoded integer is the evaluation of a valid {py3} expression which
173 may include label and variable names.
194 aa bb cc 66 6f 6f 66 6f 6f 66 6f 6f 62 61 72 66 ┆ •••foofoofoobarf
195 6f 6f 62 61 72 ┆ oobar
203 aa bb * 5 cc <zoom> "yeah\0" * {zoom * 3}
213 aa bb bb bb bb bb cc 79 65 61 68 00 79 65 61 68 ┆ •••••••yeah•yeah
214 00 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ •yeah•yeah•yeah•
215 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 79 ┆ yeah•yeah•yeah•y
216 65 61 68 00 79 65 61 68 00 79 65 61 68 00 79 65 ┆ eah•yeah•yeah•ye
217 61 68 00 79 65 61 68 00 79 65 61 68 00 79 65 61 ┆ ah•yeah•yeah•yea
218 68 00 79 65 61 68 00 79 65 61 68 00 79 65 61 68 ┆ h•yeah•yeah•yeah
219 00 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ •yeah•yeah•yeah•
220 ff ee 6a 75 69 63 65 ff ee 6a 75 69 63 65 ff ee ┆ ••juice••juice••
221 6a 75 69 63 65 ┆ juice
240 00 00 00 c7 00 00 00 00 00 00 00 00 00 00 00 2b
241 ff 85 ff ff 00 00 15 d0
244 Multilevel grouping::
249 ff ((aa bb "zoom" cc) * 5) * 3 $-34 * 4
255 ff aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa ┆ •••zoom•••zoom••
256 bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a ┆ •zoom•••zoom•••z
257 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f ┆ oom•••zoom•••zoo
258 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc ┆ m•••zoom•••zoom•
259 aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb ┆ ••zoom•••zoom•••
260 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f ┆ zoom•••zoom•••zo
261 6f 6d cc aa bb 7a 6f 6f 6d cc de de de de ┆ om•••zoom•••••
264 Precise error reporting::
267 /tmp/meow.normand:10:24 - Expecting a bit (`0` or `1`).
271 /tmp/meow.normand:32:6 - Unexpected character `k`.
275 /tmp/meow.normand:24:19 - Illegal (unknown or unreachable) variable/label name `meow` in expression `(meow - 45) // 8`; the legal names are {`mix`, `zoom`}.
279 /tmp/meow.normand:18:9 - Value 315 is outside the 8-bit range when evaluating expression `end - ICITTE` at byte offset 45.
282 You can use Normand to track data source files in your favorite VCS
283 instead of raw binary files. The binary files that Normand generates can
284 be used to test file format decoding, including malformatted data, for
285 example, as well as for education.
287 See <<learn-normand>> to explore all the Normand features.
291 Normand requires Python ≥ 3.4.
296 $ python3 -m pip install --user normand
300 https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-to-the-user-site[Installing to the User Site]
301 to learn more about a user site installation.
305 Normand has a single module file, `normand.py`, which you can copy as is
306 to your project to use it (both the <<python3-api,`normand.parse()`>>
307 function and the <<command-line-tool,command-line tool>>).
309 `normand.py` has _no external dependencies_, but if you're using
310 Python{nbsp}3.4, you'll need a local copy of the standard `typing`
316 A Normand text input is a sequence of items which represent a sequence
319 [[state]] During the processing of items to data, Normand relies on a
324 |State variable |Description |Initial value: <<python3-api,{py3} API>> |Initial value: <<command-line-tool,CLI>>
326 |[[cur-offset]] Current offset
328 The current offset has an effect on the value of <<label,labels>> and of
329 the special `ICITTE` name in <<fixed-length-number,fixed-length
330 number>>, <<leb-128-integer,LEB128 integer>>,
331 <<variable-assignment,variable assignment>>,
332 <<conditional-block,conditional block>>, <<repetition-block,repetition
333 block>>, and <<post-item-repetition,post-item repetition>> expression
336 Each generated byte increments the current offset.
338 A <<current-offset-setting,current offset setting>> may change the
339 current offset without generating data.
341 An <<current-offset-alignment,current offset alignment>> generates
342 padding bytes to make the current offset satisfy a given alignment.
343 |`init_offset` parameter of the `parse()` function.
346 |[[cur-bo]] Current byte order
348 The current byte order has an effect on the encoding of
349 <<fixed-length-number,fixed-length numbers>>.
351 A <<current-byte-order-setting,current byte order setting>> may change
352 the current byte order.
353 |`init_byte_order` parameter of the `parse()` function.
354 |`--byte-order` option.
357 |Mapping of label names to integral values.
358 |`init_labels` parameter of the `parse()` function.
359 |One or more `--label` options.
361 |<<variable-assignment,Variables>>
362 |Mapping of variable names to integral or floating point number values.
363 |`init_variables` parameter of the `parse()` function.
364 |One or more `--var` options.
367 The available items are:
369 * A <<byte-constant,constant integer>> representing a single byte.
371 * A <<literal-string,literal string>> representing a sequence of bytes
372 encoding UTF-8, UTF-16, or UTF-32 data.
374 * A <<current-byte-order-setting,current byte order setting>> (big or
377 * A <<fixed-length-number,fixed-length number>> (integer or
378 floating point) using the <<cur-bo,current byte order>> and of which
379 the value is the result of a {py3} expression.
381 * An <<leb128-integer,LEB128 integer>> of which the value is the result
382 of a {py3} expression.
384 * A <<current-offset-setting,current offset setting>>.
386 * A <<current-offset-alignment,current offset alignment>>.
388 * A <<label,label>>, that is, a named constant holding the current
391 This is similar to an assembly label.
393 * A <<variable-assignment,variable assignment>> associating a name to
394 the integral result of an evaluated {py3} expression.
396 * A <<group,group>>, that is, a scoped sequence of items.
398 * A <<conditional-block,conditional block>>.
400 * A <<repetition-block,repetition block>>.
402 Moreover, you can repeat many items above a constant or variable number
403 of times with the ``pass:[*]`` operator _after_ the item to repeat. This
404 is called a <<post-item-repetition,post-item repetition>>.
406 A Normand comment may exist:
408 * Between items, possibly within a group.
409 * Between the nibbles of a constant hexadecimal byte.
410 * Between the bits of a constant binary byte.
411 * Between the last item and the ``pass:[*]`` character of a post-item
412 repetition, and between that ``pass:[*]`` character and the following
413 number or expression.
414 * Between the ``!repeat``/``!r`` block opening and the following
415 constant integer, name, or expression of a repetition block.
416 * Between the ``!if`` block opening and the following name or expression
417 of a conditional block.
419 A comment is anything between two ``pass:[#]`` characters on the same
420 line, or from ``pass:[#]`` until the end of the line. Whitespaces and
421 the following symbol characters are also considered comments where a
425 / \ ? & : ; . , + [ ] _ = | -
428 The latter serve to improve readability so that you may write, for
429 example, a MAC address or a UUID as is.
431 You can test the examples of this section with the `normand`
432 <<command-line-tool,command-line tool>> as such:
435 $ normand file | hexdump -C
438 where `file` is the name of a file containing the Normand input.
442 A _byte constant_ represents a single byte.
447 Two consecutive hexits.
450 A decimal number after the `$` prefix.
453 Eight bits after the `%` prefix.
473 $192 %1100/0011 $ -77
487 58f64689-6316-4d55-8a1a-04cada366172
488 fe80::6257:18ff:fea3:4229
494 58 f6 46 89 63 16 4d 55 8a 1a 04 ca da 36 61 72 ┆ X•F•c•MU•••••6ar
495 fe 80 62 57 18 ff fe a3 42 29 ┆ ••bW••••B)
503 %01110011 %01100001 %01101100 %01110101 %01110100
509 73 61 6c 75 74 ┆ salut
515 A _literal string_ represents the UTF-8-, UTF-16-, or UTF-32-encoded
518 The string to encode isn't implicitly null-terminated: use `\0` at the
519 end of the string to add a null character.
523 . **Optional**: one of the following encodings instead of UTF-8:
533 . The ``pass:["]`` prefix.
535 . A sequence of zero or more characters, possibly containing escape
538 An escape sequence is the ``\`` character followed by one of:
544 `b`:: Backspace (U+0008)
545 `e`:: Escape (U+001B)
546 `f`:: Form feed (U+000C)
547 `n`:: End of line (U+000A)
548 `r`:: Carriage return (U+000D)
549 `t`:: Character tabulation (U+0009)
550 `v`:: Line tabulation (U+000B)
551 ``\``:: Reverse solidus (U+005C)
552 ``pass:["]``:: Quotation mark (U+0022)
555 . The ``pass:["]`` suffix.
561 "coucou tout le monde!"
567 63 6f 75 63 6f 75 20 74 6f 75 74 20 6c 65 20 6d ┆ coucou tout le m
568 6f 6e 64 65 21 ┆ onde!
576 u16le"I am not young enough to know everything."
582 49 00 20 00 61 00 6d 00 20 00 6e 00 6f 00 74 00 ┆ I• •a•m• •n•o•t•
583 20 00 79 00 6f 00 75 00 6e 00 67 00 20 00 65 00 ┆ •y•o•u•n•g• •e•
584 6e 00 6f 00 75 00 67 00 68 00 20 00 74 00 6f 00 ┆ n•o•u•g•h• •t•o•
585 20 00 6b 00 6e 00 6f 00 77 00 20 00 65 00 76 00 ┆ •k•n•o•w• •e•v•
586 65 00 72 00 79 00 74 00 68 00 69 00 6e 00 67 00 ┆ e•r•y•t•h•i•n•g•
595 u32be "\"illusion is the first\nof all pleasures\" 🦉"
601 00 00 00 22 00 00 00 69 00 00 00 6c 00 00 00 6c ┆ •••"•••i•••l•••l
602 00 00 00 75 00 00 00 73 00 00 00 69 00 00 00 6f ┆ •••u•••s•••i•••o
603 00 00 00 6e 00 00 00 20 00 00 00 69 00 00 00 73 ┆ •••n••• •••i•••s
604 00 00 00 20 00 00 00 74 00 00 00 68 00 00 00 65 ┆ ••• •••t•••h•••e
605 00 00 00 20 00 00 00 66 00 00 00 69 00 00 00 72 ┆ ••• •••f•••i•••r
606 00 00 00 73 00 00 00 74 00 00 00 0a 00 00 00 6f ┆ •••s•••t•••••••o
607 00 00 00 66 00 00 00 20 00 00 00 61 00 00 00 6c ┆ •••f••• •••a•••l
608 00 00 00 6c 00 00 00 20 00 00 00 70 00 00 00 6c ┆ •••l••• •••p•••l
609 00 00 00 65 00 00 00 61 00 00 00 73 00 00 00 75 ┆ •••e•••a•••s•••u
610 00 00 00 72 00 00 00 65 00 00 00 73 00 00 00 22 ┆ •••r•••e•••s•••"
611 00 00 00 20 00 01 f9 89 ┆ ••• ••••
615 === Current byte order setting
617 This special item sets the <<cur-bo,_current byte order_>>.
619 The two accepted forms are:
622 ``pass:[{be}]``:: Set the current byte order to big endian.
623 ``pass:[{le}]``:: Set the current byte order to little endian.
625 === Fixed-length number
627 A _fixed-length number_ represents a fixed number of bytes encoding
630 * An unsigned or signed integer (two's complement).
632 The available lengths are 8, 16, 24, 32, 40, 48, 56, and 64.
634 * A floating point number
635 ([IEEE{nbsp}754-2008[https://standards.ieee.org/standard/754-2008.html]).
637 The available length are 32 (_binary32_) and 64 (_binary64_).
639 The value is the result of evaluating a {py3} expression using the
640 <<cur-bo,current byte order>>.
642 A fixed-length number is:
644 . The ``pass:[{]`` prefix.
646 . A valid {py3} expression.
648 For a fixed-length number at some source location{nbsp}__**L**__, this
649 expression may contain the name of any accessible <<label,label>> (not
650 within a nested group), including the name of a label defined
651 after{nbsp}__**L**__, as well as the name of any
652 <<variable-assignment,variable>> known at{nbsp}__**L**__.
654 The value of the special name `ICITTE` (`int` type) in this expression
655 is the <<cur-offset,current offset>> (before encoding the number).
659 . An encoding length in bits amongst:
662 The expression evaluates to an `int` or `bool` value::
663 `8`, `16`, `24`, `32`, `40`, `48`, `56`, and `64`.
665 NOTE: Normand automatically converts a `bool` value to `int`.
667 The expression evaluates to a `float` value::
694 # String length in bits
695 {8 * (str_end - str_beg) : 16}
706 00 60 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 ┆ •`hello world!
714 {20 - ICITTE : 8} * 10
720 14 13 12 11 10 0f 0e 0d 0c 0b
741 An _LEB128 integer_ represents a variable number of bytes encoding an
742 unsigned or signed integer which is the result of evaluating a {py3}
743 expression following the https://en.wikipedia.org/wiki/LEB128[LEB128]
746 An LEB128 integer is:
748 . The ``pass:[{]`` prefix.
750 . A valid {py3} expression of which the evaluation result type
751 is `int` or `bool` (automatically converted to `int`).
753 For an LEB128 integer at some source location{nbsp}__**L**__, this
754 expression may contain:
757 * The name of any <<label,label>> defined before{nbsp}__**L**__.
758 * The name of any <<variable-assignment,variable>> known at{nbsp}__**L**__
759 which doesn't, directly or indirectly, refer to a label
760 defined after{nbsp}__**L**__.
763 The value of the special name `ICITTE` (`int` type) in this expression
764 is the <<cur-offset,current offset>> (before encoding the integer).
772 `uleb128`:: Use the unsigned LEB128 format.
773 `sleb128`:: Use the signed LEB128 format.
799 {-981238311 + (meow * -23) : sleb128}
806 aa bb cc dd ee ff fd fa 8d ac 7c 68 65 6c 6c 6f ┆ ••••••••••|hello
810 === Current offset setting
812 This special item sets the <<cur-offset,_current offset_>>.
814 A current offset setting is:
818 . A positive integer (hexadecimal starting with `0x` or `0X` accepted)
819 which is the new current offset.
828 <0x61> {ICITTE : 8} * 8
834 00 01 02 03 04 05 06 07 61 62 63 64 65 66 67 68 ┆ ••••••••abcdefgh
842 aa bb cc dd <meow> ee ff
843 <12> 11 22 33 <mix> 44 55
850 aa bb cc dd ee ff 11 22 33 44 55 04 0f ┆ •••••••"3DU••
854 === Current offset alignment
856 A _current offset alignment_ represents zero or more padding bytes to
857 make the <<cur-offset,current offset>> meet a given
858 https://en.wikipedia.org/wiki/Data_structure_alignment[alignment] value.
860 More specifically, for an alignment value of{nbsp}__**N**__{nbsp}bits,
861 a current offset alignment represents the required padding bytes until
862 the current offset is a multiple of __**N**__{nbsp}/{nbsp}8.
864 A current offset alignment is:
868 . A positive integer (hexadecimal starting with `0x` or `0X` accepted)
869 which is the alignment value in _bits_.
871 This value must be greater than zero and a multiple of{nbsp}8.
876 . The ``pass:[~]`` prefix.
877 . A positive integer (hexadecimal starting with `0x` or `0X` accepted)
878 which is the value of the byte to use as padding to align the
879 <<cur-offset,current offset>>.
882 Without this section, the padding byte value is zero.
888 11 22 (@32 aa bb cc) * 3
894 11 22 00 00 aa bb cc 00 aa bb cc 00 aa bb cc
911 77 88 cc cc 00 60 5f c4 55 55 55 55 55 55 55 55 ┆ w••••`_•UUUUUUUU
920 aa bb cc <29> @64~255 "zoom"
926 aa bb cc ff ff ff 7a 6f 6f 6d ┆ ••••••zoom
932 A _label_ associates a name to the <<cur-offset,current offset>>.
934 All the labels of a whole Normand input must have unique names.
936 A label must not share the name of a <<variable-assignment,variable>>
943 . A valid {py3} name which is not `ICITTE`.
947 === Variable assignment
949 A _variable assignment_ associates a name to the integral result of an
950 evaluated {py3} expression.
952 A variable assignment is:
954 . The ``pass:[{]`` prefix.
956 . A valid {py3} name which is not `ICITTE`.
960 . A valid {py3} expression of which the evaluation result type
961 is `int`, `float`, or `bool` (automatically converted to `int`).
963 For a variable assignment at some source location{nbsp}__**L**__, this
964 expression may contain the name of any accessible <<label,label>> (not
965 within a nested group), including the name of a label defined
966 after{nbsp}__**L**__, as well as the name of any
967 <<variable-assignment,variable>> known at{nbsp}__**L**__.
969 The value of the special name `ICITTE` (`int` type) in this expression
970 is the <<cur-offset,current offset>>.
979 {meow = 42} 11 22 {meow:8} 33 {meow = ICITTE + 17}
980 "yooo" {meow + mix : 16}
986 11 22 2a 33 79 6f 6f 6f 7a 00 ┆ •"*3yoooz•
992 A _group_ is a scoped sequence of items.
994 The <<label,labels>> within a group aren't visible outside of it.
996 The main purpose of a group is to <<post-item-repetition,repeat>> more
997 than a single item and to isolate labels.
1001 . The `(`, `!group`, or `!g` opening.
1003 . Zero or more items.
1005 . Depending on the group opening:
1020 ((aa bb cc) dd () ee) "leclerc"
1026 aa bb cc dd ee 6c 65 63 6c 65 72 63 ┆ •••••leclerc
1035 (aa bb cc) * 3 dd ee
1042 aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa bb
1043 cc aa bb cc dd ee aa bb cc aa bb cc aa bb cc dd
1044 ee aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa
1045 bb cc aa bb cc dd ee
1055 <str_beg> u16le"sébastien diaz" <str_end>
1056 {ICITTE - str_beg : 8}
1057 {(end - str_beg) * 5 : 24}
1065 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
1066 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 e0 ┆ n• •d•i•a•z•••••
1067 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
1068 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 40 ┆ n• •d•i•a•z••••@
1069 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
1070 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 00 a0 ┆ n• •d•i•a•z•••••
1074 === Conditional block
1076 A _conditional block_ represents either the bytes of one or more items
1077 if some expression is true, or no bytes at all if it's false.
1079 A conditional block is:
1081 . The `!if` opening.
1085 ** The ``pass:[{]`` prefix, a valid {py3} expression of which the
1086 evaluation result type is `int` or `bool` (automatically converted to
1087 `int`), and the ``pass:[}]`` suffix.
1089 For a repetition at some source location{nbsp}__**L**__, this expression
1093 * The name of any <<label,label>> defined before{nbsp}__**L**__
1094 which isn't within a nested group.
1095 * The name of any <<variable-assignment,variable>> known
1096 at{nbsp}__**L**__ which doesn't, directly or indirectly, refer to a
1097 label defined after{nbsp}__**L**__.
1100 The value of the special name `ICITTE` (`int` type) in this expression
1101 is the <<cur-offset,current offset>> (before handling the contained
1104 ** A valid {py3} name.
1106 For the name `__NAME__`, this is equivalent to the
1107 `pass:[{]__NAME__pass:[}]` form above.
1109 . Zero or more items.
1111 . The `!end` closing.
1126 !if {at < rep_count} 20 !end
1136 6d 65 6f 77 20 6d 65 6f 77 20 6d 65 6f 77 20 6d ┆ meow meow meow m
1137 65 6f 77 20 6d 65 6f 77 20 6d 65 6f 77 20 6d 69 ┆ eow meow meow mi
1138 78 20 6d 65 6f 77 20 6d 69 78 20 6d 65 6f 77 20 ┆ x meow mix meow
1139 6d 69 78 20 6d 65 6f 77 20 6d 69 78 ┆ mix meow mix
1151 !if {str_end - str_beg > 10}
1159 6d 00 65 00 6f 00 77 00 20 00 6d 00 69 00 78 00 ┆ m•e•o•w• •m•i•x•
1160 21 00 20 42 49 47 ┆ !• BIG
1164 === Repetition block
1166 A _repetition block_ represents the bytes of one or more items repeated
1167 a given number of times.
1169 A repetition block is:
1171 . The `!repeat` or `!r` opening.
1175 ** A positive integer (hexadecimal starting with `0x` or `0X` accepted)
1176 which is the number of times to repeat the previous item.
1178 ** The ``pass:[{]`` prefix, a valid {py3} expression of which the
1179 evaluation result type is `int` or `bool` (automatically converted to
1180 `int`), and the ``pass:[}]`` suffix.
1182 For a repetition at some source location{nbsp}__**L**__, this expression
1186 * The name of any <<label,label>> defined before{nbsp}__**L**__
1187 which isn't within a nested group.
1188 * The name of any <<variable-assignment,variable>> known
1189 at{nbsp}__**L**__ which doesn't, directly or indirectly, refer to a
1190 label defined after{nbsp}__**L**__.
1193 The value of the special name `ICITTE` (`int` type) in this expression
1194 is the <<cur-offset,current offset>> (before handling the items to
1197 ** A valid {py3} name.
1199 For the name `__NAME__`, this is equivalent to the
1200 `pass:[{]__NAME__pass:[}]` form above.
1202 . Zero or more items.
1204 . The `!end` closing.
1206 You may also use a <<post-item-repetition,post-item repetition>> after
1207 some items. The form ``!repeat{nbsp}__X__{nbsp}__ITEMS__{nbsp}!end``
1208 is equivalent to ``(__ITEMS__){nbsp}pass:[*]{nbsp}__X__``.
1215 {end - ICITTE - 1 : 8}
1224 ff fe fd fc fb fa f9 f8 f7 f6 f5 f4 f3 f2 f1 f0 ┆ ••••••••••••••••
1225 ef ee ed ec eb ea e9 e8 e7 e6 e5 e4 e3 e2 e1 e0 ┆ ••••••••••••••••
1226 df de dd dc db da d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 ┆ ••••••••••••••••
1227 cf ce cd cc cb ca c9 c8 c7 c6 c5 c4 c3 c2 c1 c0 ┆ ••••••••••••••••
1228 bf be bd bc bb ba b9 b8 b7 b6 b5 b4 b3 b2 b1 b0 ┆ ••••••••••••••••
1229 af ae ad ac ab aa a9 a8 a7 a6 a5 a4 a3 a2 a1 a0 ┆ ••••••••••••••••
1230 9f 9e 9d 9c 9b 9a 99 98 97 96 95 94 93 92 91 90 ┆ ••••••••••••••••
1231 8f 8e 8d 8c 8b 8a 89 88 87 86 85 84 83 82 81 80 ┆ ••••••••••••••••
1232 7f 7e 7d 7c 7b 7a 79 78 77 76 75 74 73 72 71 70 ┆ •~}|{zyxwvutsrqp
1233 6f 6e 6d 6c 6b 6a 69 68 67 66 65 64 63 62 61 60 ┆ onmlkjihgfedcba`
1234 5f 5e 5d 5c 5b 5a 59 58 57 56 55 54 53 52 51 50 ┆ _^]\[ZYXWVUTSRQP
1235 4f 4e 4d 4c 4b 4a 49 48 47 46 45 44 43 42 41 40 ┆ ONMLKJIHGFEDCBA@
1236 3f 3e 3d 3c 3b 3a 39 38 37 36 35 34 33 32 31 30 ┆ ?>=<;:9876543210
1237 2f 2e 2d 2c 2b 2a 29 28 27 26 25 24 23 22 21 20 ┆ /.-,+*)('&%$#"!
1238 1f 1e 1d 1c 1b 1a 19 18 17 16 15 14 13 12 11 10 ┆ ••••••••••••••••
1239 0f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 00 ┆ ••••••••••••••••
1258 11 22 !repeat times 33 !end
1269 aa bb cc dd ee ff ee ff ee ff ee ff ee ff 11 22 ┆ •••••••••••••••"
1270 33 ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ 3•••••••••••••••
1271 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1272 ff ee ff ee ff 11 22 33 33 ee ff ee ff ee ff ee ┆ ••••••"33•••••••
1273 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1274 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1275 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1276 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1277 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1278 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1279 ff ee ff ee ff ee ff ee ff ee ff ee ff 11 22 33 ┆ ••••••••••••••"3
1280 33 33 63 6f 75 63 6f 75 21 ┆ 33coucou!
1284 === Post-item repetition
1286 A _post-item repetition_ represents the bytes of an item repeated a
1287 given number of times.
1289 A post-item repetition is:
1291 . One of those items:
1293 ** A <<byte-constant,byte constant>>.
1294 ** A <<literal-string,literal string>>.
1295 ** A <<fixed-length-number,fixed-length number>>.
1296 ** An <<leb128-integer,LEB128 integer>>.
1297 ** A <<group,group>>.
1299 . The ``pass:[*]`` character.
1303 ** A positive integer (hexadecimal starting with `0x` or `0X` accepted)
1304 which is the number of times to repeat the previous item.
1306 ** The ``pass:[{]`` prefix, a valid {py3} expression of which the
1307 evaluation result type is `int` or `bool` (automatically converted to
1308 `int`), and the ``pass:[}]`` suffix.
1310 For a repetition at some source location{nbsp}__**L**__, this expression
1314 * The name of any <<label,label>> defined before{nbsp}__**L**__
1315 which isn't within a nested group and
1316 which isn't part of the repeated item.
1317 * The name of any <<variable-assignment,variable>> known
1318 at{nbsp}__**L**__, which isn't part of its repeated item, and which
1319 doesn't, directly or indirectly, refer to a label defined
1320 after{nbsp}__**L**__.
1323 The value of the special name `ICITTE` (`int` type) in this expression
1324 is the <<cur-offset,current offset>> (before handling the items to
1327 ** A valid {py3} name.
1329 For the name `__NAME__`, this is equivalent to the
1330 `pass:[{]__NAME__pass:[}]` form above.
1332 You may also use a <<repetition-block,repetition block>>. The form
1333 ``__ITEM__{nbsp}pass:[*]{nbsp}__X__`` is equivalent to
1334 ``!repeat{nbsp}__X__{nbsp}__ITEM__{nbsp}!end``.
1340 {end - ICITTE - 1 : 8} * 0x100 <end>
1346 ff fe fd fc fb fa f9 f8 f7 f6 f5 f4 f3 f2 f1 f0 ┆ ••••••••••••••••
1347 ef ee ed ec eb ea e9 e8 e7 e6 e5 e4 e3 e2 e1 e0 ┆ ••••••••••••••••
1348 df de dd dc db da d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 ┆ ••••••••••••••••
1349 cf ce cd cc cb ca c9 c8 c7 c6 c5 c4 c3 c2 c1 c0 ┆ ••••••••••••••••
1350 bf be bd bc bb ba b9 b8 b7 b6 b5 b4 b3 b2 b1 b0 ┆ ••••••••••••••••
1351 af ae ad ac ab aa a9 a8 a7 a6 a5 a4 a3 a2 a1 a0 ┆ ••••••••••••••••
1352 9f 9e 9d 9c 9b 9a 99 98 97 96 95 94 93 92 91 90 ┆ ••••••••••••••••
1353 8f 8e 8d 8c 8b 8a 89 88 87 86 85 84 83 82 81 80 ┆ ••••••••••••••••
1354 7f 7e 7d 7c 7b 7a 79 78 77 76 75 74 73 72 71 70 ┆ •~}|{zyxwvutsrqp
1355 6f 6e 6d 6c 6b 6a 69 68 67 66 65 64 63 62 61 60 ┆ onmlkjihgfedcba`
1356 5f 5e 5d 5c 5b 5a 59 58 57 56 55 54 53 52 51 50 ┆ _^]\[ZYXWVUTSRQP
1357 4f 4e 4d 4c 4b 4a 49 48 47 46 45 44 43 42 41 40 ┆ ONMLKJIHGFEDCBA@
1358 3f 3e 3d 3c 3b 3a 39 38 37 36 35 34 33 32 31 30 ┆ ?>=<;:9876543210
1359 2f 2e 2d 2c 2b 2a 29 28 27 26 25 24 23 22 21 20 ┆ /.-,+*)('&%$#"!
1360 1f 1e 1d 1c 1b 1a 19 18 17 16 15 14 13 12 11 10 ┆ ••••••••••••••••
1361 0f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 00 ┆ ••••••••••••••••
1373 (ee ff) * {here + 1}
1383 aa bb cc dd ee ff ee ff ee ff ee ff ee ff 11 22 ┆ •••••••••••••••"
1384 33 ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ 3•••••••••••••••
1385 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1386 ff ee ff ee ff 11 22 33 33 ee ff ee ff ee ff ee ┆ ••••••"33•••••••
1387 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1388 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1389 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1390 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1391 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1392 ff ee ff ee ff ee ff ee ff ee ff ee ff ee ff ee ┆ ••••••••••••••••
1393 ff ee ff ee ff ee ff ee ff ee ff ee ff 11 22 33 ┆ ••••••••••••••"3
1394 33 33 63 6f 75 63 6f 75 21 ┆ 33coucou!
1398 == Command-line tool
1400 If you <<install-normand,installed>> the `normand` package, then you
1401 can use the `normand` command-line tool:
1404 $ normand <<< '"ma gang de malades"' | hexdump -C
1408 00000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
1412 If you copy the `normand.py` module to your own project, then you can
1413 run the module itself:
1416 $ python3 -m normand <<< '"ma gang de malades"' | hexdump -C
1420 00000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
1424 Without a path argument, the `normand` tool reads from the standard
1427 The `normand` tool prints the generated binary data to the standard
1430 Various options control the initial <<state,state>> of the processor:
1431 use the `--help` option to learn more.
1435 The whole `normand` package/module public API is:
1440 class ByteOrder(enum.Enum):
1452 def line_no(self) -> int:
1457 def col_no(self) -> int:
1462 class ParseError(RuntimeError):
1463 # Source text location.
1465 def text_loc(self) -> TextLocation:
1469 # Variables dictionary type (for type hints).
1470 VariablesT = typing.Dict[str, typing.Union[int, float]]
1473 # Labels dictionary type (for type hints).
1474 LabelsT = typing.Dict[str, int]
1481 def data(self) -> bytearray:
1484 # Updated variable values.
1486 def variables(self) -> SymbolsT:
1489 # Updated main group label values.
1491 def labels(self) -> SymbolsT:
1496 def offset(self) -> int:
1501 def byte_order(self) -> typing.Optional[ByteOrder]:
1505 # Parses the `normand` input using the initial state defined by
1506 # `init_variables`, `init_labels`, `init_offset`, and `init_byte_order`,
1507 # and returns the corresponding parsing result.
1508 def parse(normand: str,
1509 init_variables: typing.Optional[SymbolsT] = None,
1510 init_labels: typing.Optional[SymbolsT] = None,
1511 init_offset: int = 0,
1512 init_byte_order: typing.Optional[ByteOrder] = None) -> ParseResult:
1516 The `normand` parameter is the actual <<learn-normand,Normand input>>
1517 while the other parameters control the initial <<state,state>>.
1519 The `parse()` function raises a `ParseError` instance should it fail to
1520 parse the `normand` string for any reason.
1524 Normand is a https://python-poetry.org/[Poetry] project.
1526 To develop it, install it through Poetry and enter the virtual
1532 $ normand <<< '"lol" * 10 0a'
1535 `normand.py` is processed by:
1537 * https://microsoft.github.io/pyright/[Pyright]
1538 * https://github.com/psf/black[Black]
1539 * https://pycqa.github.io/isort/[isort]
1543 Use https://docs.pytest.org/[pytest] to test Normand once the package is
1544 part of your virtual environment, for example:
1548 $ poetry run pip3 install pytest
1552 The `pytest` project is currently not a development dependency in
1553 `pyproject.toml` due to backward compatibiliy issues with
1556 In the `tests` directory, each `*.nt` file is a test. The file name
1557 prefix indicates what it's meant to test:
1560 Everything above the `---` line is the valid Normand input
1563 Everything below the `---` line is the expected data
1564 (whitespace-separated hexadecimal bytes).
1567 Everything above the `---` line is the invalid Normand input
1570 Everything below the `---` line is the expected error message having
1579 Normand uses https://review.lttng.org/admin/repos/normand,general[Gerrit]
1582 To report a bug, https://github.com/efficios/normand/issues/new[create a