8 image:https://img.shields.io/pypi/v/normand.svg?label=Latest%20version[link="https://pypi.python.org/pypi/normand"]
11 _**Normand**_ is a text-to-binary processor with its own language.
13 This package offers both a portable {py3} module and a command-line
16 WARNING: This version of Normand is 0.1, meaning both the Normand
17 language and the module/CLI interface aren't stable.
21 The purpose of Normand is to consume human-readable text representing
22 bytes and to produce the corresponding binary data.
26 Consider the following Normand input:
29 4f 55 32 bb $167 fe %10100111 a9 $-32
32 The generated nine bytes are:
35 4f 55 32 bb a7 fe a7 a9 e0
39 As you can see in the last example, the fundamental unit of the Normand
40 language is the _byte_. The order in which you list bytes will be the
41 order of the generated data.
43 The Normand language is more than simple lists of bytes, though. Its
46 Comments, including a bunch of insignificant symbols which may improve readability::
51 ff bb %1101:0010 # This is a comment
52 78 29 af $192 # This too # 99 $-80
53 fe80::6257:18ff:fea3:4229
55 10839636-5d65-4a68-8e6a-21608ddf7258
61 ff bb d2 78 29 af c0 99 b0 fe 80 62 57 18 ff fe
62 a3 42 29 60 57 18 a3 42 29 10 83 96 36 5d 65 4a
63 68 8e 6a 21 60 8d df 72 58
66 Hexadecimal, decimal, and binary byte constants::
71 aa bb $247 $-89 %0011_0010 %11.01= 10/10
80 UTF-8, UTF-16, and UTF-32 literal strings::
86 u16le"stress\nverdict 🤣"
92 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 00 73 00 74 ┆ hello world!•s•t
93 00 72 00 65 00 73 00 73 00 0a 00 76 00 65 00 72 ┆ •r•e•s•s•••v•e•r
94 00 64 00 69 00 63 00 74 00 20 00 3e d8 23 dd ┆ •d•i•c•t• •>•#•
97 Labels: special variables holding the offset where they're defined::
100 <beg> b2 52 e3 bc 91 05
101 $100 $50 <chair> 33 9f fe
108 5e 65 {tower = 47} c6 7f f2 c4
109 44 {hurl = tower - 14} b5 {tower = hurl} 26 2d
112 The value of a variable assignment is the evaluation of a valid {py3}
113 expression which may include label and variable names.
115 Value encoding with a specific length (8{nbsp}bits to 64{nbsp}bits) and byte order::
121 {be} 67 <lbl> 44 $178 {(end - lbl) * 8 + strength : 16} $99 <end>
128 67 44 b2 00 2c 63 37 f8 ff ff
131 The encoded value is the evaluation of a valid {py3} expression which
132 may include label and variable names.
139 aa bb * 5 cc "yeah\0" * 8
145 aa bb bb bb bb bb cc 79 65 61 68 00 79 65 61 68 ┆ •••••••yeah.yeah
146 00 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ •yeah•yeah•yeah•
147 79 65 61 68 00 79 65 61 68 00 79 65 61 68 00 ┆ yeah•yeah•yeah•
151 Multilevel grouping::
156 ff ((aa bb "zoom" cc) * 5) * 3 $-34 * 4
162 ff aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa ┆ •••zoom•••zoom••
163 bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a ┆ •zoom•••zoom•••z
164 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f ┆ oom•••zoom•••zoo
165 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc ┆ m•••zoom•••zoom•
166 aa bb 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb ┆ ••zoom•••zoom•••
167 7a 6f 6f 6d cc aa bb 7a 6f 6f 6d cc aa bb 7a 6f ┆ zoom•••zoom•••zo
168 6f 6d cc aa bb 7a 6f 6f 6d cc de de de de ┆ om•••zoom•••••
171 Precise error reporting::
174 /tmp/meow.normand:10:24 - Expecting a bit (`0` or `1`).
178 /tmp/meow.normand:32:6 - Unexpected character `k`.
182 /tmp/meow.normand:24:19 - Unknown variable/label name `meow` in expression `(meow - 45) // 8`.
186 /tmp/meow.normand:18:9 - Value 315 is outside the 8-bit range when evaluating expression `end - ICITTE` at byte offset 45.
189 You can use Normand to track data source files in your favorite VCS
190 instead of raw binary files. The binary files that Normand generates can
191 be used to test file format decoding, including malformatted data, for
192 example, as well as for education.
194 See <<learn-normand>> to explore all the Normand features.
198 Normand requires Python ≥ 3.4.
203 $ python3 -m pip install --user normand
207 https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-to-the-user-site[Installing to the User Site]
208 to learn more about a user site installation.
212 Normand has a single module file, `normand.py`, which you can copy as is
213 to your project to use it (both the <<python-3-api,`normand.parse()`>>
214 function and the <<command-line-tool,command-line tool>>).
216 `normand.py` has _no external dependencies_, but if you're using
217 Python{nbsp}3.4, you'll need a local copy of the standard `typing`
223 A Normand text input is a sequence of items which represent a sequence
226 [[state]] During the processing of items to data, Normand relies on a
231 |State variable |Description |Initial value: <<python-3-api,{py3} API>> |Initial value: <<command-line-tool,CLI>>
233 |[[cur-offset]] Current offset
235 The current offset has an effect on the value of
236 <<label,labels>> and of the special `ICITTE` name in <<value,value>> and
237 <<variable-assignment,variable assignment>> expression evaluation.
239 Each generated byte increments the current offset.
241 A <<current-offset-setting,current offset setting>> may change the
243 |`init_offset` parameter of the `parse()` function.
246 |[[cur-bo]] Current byte order
248 The current byte order has an effect on the encoding of <<value,values>>.
250 A <<current-byte-order-setting,current byte order setting>> may change
251 the current byte order.
252 |`init_byte_order` parameter of the `parse()` function.
253 |`--byte-order` option.
256 |Mapping of label names to integral values.
257 |`init_labels` parameter of the `parse()` function.
258 |One or more `--label` options.
260 |<<variable-assignment,Variables>>
261 |Mapping of variable names to integral values.
262 |`init_variables` parameter of the `parse()` function.
263 |One or more `--var` options.
266 The available items are:
268 * A <<byte-constant,constant integer>> representing a single byte.
270 * A <<literal-string,literal string>> representing a sequence of bytes
271 encoding UTF-8, UTF-16, or UTF-32 data.
273 * A <<current-byte-order-setting,current byte order setting>> (big or
276 * A <<value,{py3} expression to be evaluated>> as an unsigned or signed
277 integer to be encoded on one or more bytes using the current byte
280 * A <<current-offset-setting,current offset setting>>.
282 * A <<label,label>>, that is, a named constant holding the current
285 This is similar to an assembly label.
287 * A <<variable-assignment,variable assignment>> associating a name to
288 the integral result of an evaluated {py3} expression.
290 * A <<group,group>>, that is, a scoped sequence of items.
292 Moreover, you can <<repetition,repeat>> any item above, except an offset
293 or a label, a given number of times. This is called a repetition.
295 A Normand comment may exist:
297 * Between items, possibly within a group.
298 * Between the nibbles of a constant hexadecimal byte.
299 * Between the bits of a constant binary byte.
300 * Between the last item and the ``pass:[*]`` character of a repetition,
301 and between that ``pass:[*]`` character and the following number.
303 A comment is anything between two ``pass:[#]`` characters on the same
304 line, or from ``pass:[#]`` until the end of the line. Whitespaces and
305 the following symbol characters are also considered comments where a
309 ! @ / \ ? & : ; . , + [ ] _ = | -
312 The latter serve to improve readability so that you may write, for
313 example, a MAC address or a UUID as is.
315 You can test the examples of this section with the `normand`
316 <<command-line-tool,command-line tool>> as such:
319 $ normand file | hexdump -C
322 where `file` is the name of a file containing the Normand input.
326 A _byte constant_ represents a single byte.
331 Two consecutive hexits.
334 A decimal number after the `$` prefix.
337 Eight bits after the `%` prefix.
357 $192 %1100/0011 $ -77
371 58f64689-6316-4d55-8a1a-04cada366172
372 fe80::6257:18ff:fea3:4229
378 58 f6 46 89 63 16 4d 55 8a 1a 04 ca da 36 61 72 ┆ X•F•c•MU•••••6ar
379 fe 80 62 57 18 ff fe a3 42 29 ┆ ••bW••••B)
387 %01110011 %01100001 %01101100 %01110101 %01110100
393 73 61 6c 75 74 ┆ salut
399 A _literal string_ represents the UTF-8-, UTF-16-, or UTF-32-encoded
402 The string to encode isn't implicitly null-terminated: use `\0` at the
403 end of the string to add a null character.
407 . **Optional**: one of the following encodings instead of UTF-8:
417 . The ``pass:["]`` prefix.
419 . A sequence of zero or more characters, possibly containing escape
422 An escape sequence is the ``\`` character followed by one of:
428 `b`:: Backspace (U+0008)
429 `e`:: Escape (U+001B)
430 `f`:: Form feed (U+000C)
431 `n`:: End of line (U+000A)
432 `r`:: Carriage return (U+000D)
433 `t`:: Character tabulation (U+0009)
434 `v`:: Line tabulation (U+000B)
435 ``\``:: Reverse solidus (U+005C)
436 ``pass:["]``:: Quotation mark (U+0022)
439 . The ``pass:["]`` suffix.
445 "coucou tout le monde!"
451 63 6f 75 63 6f 75 20 74 6f 75 74 20 6c 65 20 6d ┆ coucou tout le m
452 6f 6e 64 65 21 ┆ onde!
460 u16le"I am not young enough to know everything."
466 49 00 20 00 61 00 6d 00 20 00 6e 00 6f 00 74 00 ┆ I• •a•m• •n•o•t•
467 20 00 79 00 6f 00 75 00 6e 00 67 00 20 00 65 00 ┆ •y•o•u•n•g• •e•
468 6e 00 6f 00 75 00 67 00 68 00 20 00 74 00 6f 00 ┆ n•o•u•g•h• •t•o•
469 20 00 6b 00 6e 00 6f 00 77 00 20 00 65 00 76 00 ┆ •k•n•o•w• •e•v•
470 65 00 72 00 79 00 74 00 68 00 69 00 6e 00 67 00 ┆ e•r•y•t•h•i•n•g•
479 u32be "\"illusion is the first\nof all pleasures\" 🦉"
485 00 00 00 22 00 00 00 69 00 00 00 6c 00 00 00 6c ┆ •••"•••i•••l•••l
486 00 00 00 75 00 00 00 73 00 00 00 69 00 00 00 6f ┆ •••u•••s•••i•••o
487 00 00 00 6e 00 00 00 20 00 00 00 69 00 00 00 73 ┆ •••n••• •••i•••s
488 00 00 00 20 00 00 00 74 00 00 00 68 00 00 00 65 ┆ ••• •••t•••h•••e
489 00 00 00 20 00 00 00 66 00 00 00 69 00 00 00 72 ┆ ••• •••f•••i•••r
490 00 00 00 73 00 00 00 74 00 00 00 0a 00 00 00 6f ┆ •••s•••t•••••••o
491 00 00 00 66 00 00 00 20 00 00 00 61 00 00 00 6c ┆ •••f••• •••a•••l
492 00 00 00 6c 00 00 00 20 00 00 00 70 00 00 00 6c ┆ •••l••• •••p•••l
493 00 00 00 65 00 00 00 61 00 00 00 73 00 00 00 75 ┆ •••e•••a•••s•••u
494 00 00 00 72 00 00 00 65 00 00 00 73 00 00 00 22 ┆ •••r•••e•••s•••"
495 00 00 00 20 00 01 f9 89 ┆ ••• ••••
499 === Current byte order setting
501 This special item sets the <<cur-bo,_current byte order_>>.
503 The two accepted forms are:
506 ``pass:[{be}]``:: Set the current byte order to big endian.
507 ``pass:[{le}]``:: Set the current byte order to little endian.
511 A _value_ represents a fixed number of bytes encoding an unsigned or
512 signed integer which is the result of evaluating a {py3} expression
513 using the <<cur-bo,current byte order>>.
515 For a value at some source location{nbsp}__**L**__, its {py3} expression
516 may contain the name of any accessible <<label,label>>, including the
517 name of a label defined after{nbsp}__**L**__, as well as the name of any
518 <<variable-assignment,variable>> known at{nbsp}__**L**__.
520 An accessible label is either:
522 * Outside of the current <<group,group>>.
523 * Within the same immediate group (not within a nested group).
525 In the {py3} expression of a value, the value of the special name
526 `ICITTE` is the <<cur-offset,current offset>> (before encoding the
531 . The ``pass:[{]`` prefix.
533 . A valid {py3} expression.
537 . An encoding length in bits amongst `8`, `16`, `24`, `32`, `40`,
538 `48`, `56`, and `64`.
563 # String length in bits
564 {8 * (str_end - str_beg) : 16}
575 00 60 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 ┆ •`hello world!
583 {20 - ICITTE : 8} * 10
589 14 13 12 11 10 0f 0e 0d 0c 0b
593 === Current offset setting
595 This special item sets the <<cur-offset,_current offset_>>.
597 A current offset setting is:
601 . A positive integer (hexadecimal starting with `0x` or `0X` accepted)
602 which is the new current offset.
611 <0x61> {ICITTE : 8} * 8
617 00 01 02 03 04 05 06 07 61 62 63 64 65 66 67 68 ┆ ••••••••abcdefgh
625 aa bb cc dd <meow> ee ff
626 <12> 11 22 33 <mix> 44 55
633 aa bb cc dd ee ff 11 22 33 44 55 04 0f ┆ •••••••"3DU••
639 A _label_ associates a name to the <<cur-offset,current offset>>.
641 All the labels of a whole Normand input must have unique names.
643 A label may not share the name of a <<variable-assignment,variable>>
646 A label name may not be `ICITTE` (see <<value>> and
647 <<variable-assignment>> to learn more).
653 . A valid {py3} name which is not `ICITTE`.
657 === Variable assignment
659 A _variable assignment_ associates a name to the integral result of an
660 evaluated {py3} expression.
662 For a variable assignment at some source location{nbsp}__**L**__, its
663 {py3} expression may contain the name of any accessible <<label,label>>,
664 including the name of a label defined after{nbsp}__**L**__, as well as
665 the name of any variable known at{nbsp}__**L**__.
667 An accessible label is either:
669 * Outside of the current <<group,group>>.
670 * Within the same immediate group (not within a nested group).
672 A variable name may not be `ICITTE` (see <<value>> and
673 <<variable-assignment>> to learn more).
675 In the {py3} expression of a variable assignment, the special name
676 `ICITTE` is the <<cur-offset,current offset>>.
680 . The ``pass:[{]`` prefix.
682 . A valid {py3} name which is not `ICITTE`.
686 . A valid {py3} expression.
695 {meow = 42} 11 22 {meow:8} 33 {meow = ICITTE + 17}
696 "yooo" {meow + mix : 16}
702 11 22 2a 33 79 6f 6f 6f 7a 00 ┆ •"*3yoooz•
708 A _group_ is a scoped sequence of items.
710 The <<label,labels>> within a group aren't visible outside of it.
712 The main purpose of a group is to <<repetition,repeat>> more than a
719 . Zero or more items.
727 ((aa bb cc) dd () ee) "leclerc"
733 aa bb cc dd ee 6c 65 63 6c 65 72 63 ┆ •••••leclerc
741 ((aa bb cc) * 3 dd ee) * 5
747 aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa bb
748 cc aa bb cc dd ee aa bb cc aa bb cc aa bb cc dd
749 ee aa bb cc aa bb cc aa bb cc dd ee aa bb cc aa
760 <str_beg> u16le"sébastien diaz" <str_end>
761 {ICITTE - str_beg : 8}
762 {(end - str_beg) * 5 : 24}
770 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
771 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 e0 ┆ n• •d•i•a•z•••••
772 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
773 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 01 40 ┆ n• •d•i•a•z••••@
774 73 00 e9 00 62 00 61 00 73 00 74 00 69 00 65 00 ┆ s•••b•a•s•t•i•e•
775 6e 00 20 00 64 00 69 00 61 00 7a 00 1c 00 00 a0 ┆ n• •d•i•a•z•••••
781 A _repetition_ represents the bytes of an item repeated a given number
788 . The ``pass:[*]`` character.
790 . A positive integer (hexadecimal starting with `0x` or `0X` accepted)
791 which is the number of times to repeat the previous item.
797 {end - ICITTE - 1 : 8} * 0x100 <end>
803 ff fe fd fc fb fa f9 f8 f7 f6 f5 f4 f3 f2 f1 f0 ┆ ••••••••••••••••
804 ef ee ed ec eb ea e9 e8 e7 e6 e5 e4 e3 e2 e1 e0 ┆ ••••••••••••••••
805 df de dd dc db da d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 ┆ ••••••••••••••••
806 cf ce cd cc cb ca c9 c8 c7 c6 c5 c4 c3 c2 c1 c0 ┆ ••••••••••••••••
807 bf be bd bc bb ba b9 b8 b7 b6 b5 b4 b3 b2 b1 b0 ┆ ••••••••••••••••
808 af ae ad ac ab aa a9 a8 a7 a6 a5 a4 a3 a2 a1 a0 ┆ ••••••••••••••••
809 9f 9e 9d 9c 9b 9a 99 98 97 96 95 94 93 92 91 90 ┆ ••••••••••••••••
810 8f 8e 8d 8c 8b 8a 89 88 87 86 85 84 83 82 81 80 ┆ ••••••••••••••••
811 7f 7e 7d 7c 7b 7a 79 78 77 76 75 74 73 72 71 70 ┆ •~}|{zyxwvutsrqp
812 6f 6e 6d 6c 6b 6a 69 68 67 66 65 64 63 62 61 60 ┆ onmlkjihgfedcba`
813 5f 5e 5d 5c 5b 5a 59 58 57 56 55 54 53 52 51 50 ┆ _^]\[ZYXWVUTSRQP
814 4f 4e 4d 4c 4b 4a 49 48 47 46 45 44 43 42 41 40 ┆ ONMLKJIHGFEDCBA@
815 3f 3e 3d 3c 3b 3a 39 38 37 36 35 34 33 32 31 30 ┆ ?>=<;:9876543210
816 2f 2e 2d 2c 2b 2a 29 28 27 26 25 24 23 22 21 20 ┆ /.-,+*)('&%$#"!
817 1f 1e 1d 1c 1b 1a 19 18 17 16 15 14 13 12 11 10 ┆ ••••••••••••••••
818 0f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 00 ┆ ••••••••••••••••
824 If you <<install-normand,installed>> the `normand` package, then you
825 can use the `normand` command-line tool:
828 $ normand <<< '"ma gang de malades"' | hexdump -C
832 00000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
836 If you copy the `normand.py` module to your own project, then you can
837 run the module itself:
840 $ python3 -m normand <<< '"ma gang de malades"' | hexdump -C
844 00000000 6d 61 20 67 61 6e 67 20 64 65 20 6d 61 6c 61 64 |ma gang de malad|
848 Without a path argument, the `normand` tool reads from the standard
851 The `normand` tool prints the generated binary data to the standard
854 Various options control the initial <<state,state>> of the processor:
855 use the `--help` option to learn more.
859 The whole `normand` package/module API is:
863 class ByteOrder(enum.Enum):
871 VarsT = typing.Dict[str, int]
877 def line_no(self) -> int:
882 def col_no(self) -> int:
886 class ParseError(RuntimeError):
887 # Source text location.
889 def text_loc(self) -> TextLoc:
896 def data(self) -> bytearray:
899 # Updated variable values.
901 def variables(self) -> VarsT:
904 # Updated main group label values.
906 def labels(self) -> VarsT:
911 def offset(self) -> int:
916 def byte_order(self) -> typing.Optional[int]:
919 def parse(normand: str,
920 init_variables: typing.Optional[VarsT] = None,
921 init_labels: typing.Optional[VarsT] = None,
922 init_offset: int = 0,
923 init_byte_order: typing.Optional[ByteOrder] = None) -> ParseResult:
927 The `normand` parameter is the actual <<learn-normand,Normand input>>
928 while the other parameters control the initial <<state,state>>.
930 The `parse()` function raises a `ParseError` instance should it fail to
931 parse the `normand` string for any reason.