+// SPDX-FileCopyrightText: 2023 Philippe Proulx <eeppeliteloop@gmail.com>
+// SPDX-License-Identifier: CC-BY-SA-4.0
+
// Show ToC at a specific location for a GitHub rendering
ifdef::env-github[]
:toc: macro
This package offers both a portable {py3} module and a command-line
tool.
-WARNING: This version of Normand is 0.20, meaning both the Normand
+WARNING: This version of Normand is 0.23, meaning both the Normand
language and the module/CLI interface aren't stable.
ifdef::env-github[]
+
----
{strength = 4}
-{be} 67 <lbl> 44 $178 {(end - lbl) * 8 + strength : 16} $99 <end>
-{le} {-1993 : 32}
-{-3.141593 : 64}
+!be 67 <lbl> 44 $178 [(end - lbl) * 8 + strength : 16] $99 <end>
+!le [-1993 : 32]
+[-3.141593 : 64be]
----
+
Output:
+
----
-67 44 b2 00 2c 63 37 f8 ff ff 7f bd c2 82 fb 21
-09 c0
+67 44 b2 00 2c 63 37 f8 ff ff c0 09 21 fb 82 c2
+bd 7f
----
+
The encoded number is the evaluation of a valid {py3} expression which
Input:
+
----
-aa bb cc {-1993 : sleb128} <meow> dd ee ff
-{meow * 199 : uleb128}
+aa bb cc [-1993 : sleb128] <meow> dd ee ff
+[meow * 199 : uleb128]
----
+
Output:
Input:
+
----
-{be}
+!be
- {199:32}
-@64 {43:64}
-@16 {-123:16}
-@32~255 {5584:32}
+ [199:32]
+@64 [43:64]
+@16 [-123:16]
+@32~255 [5584:32]
----
+
Output:
Input:
+
----
-{le}
-{0xdeadbeef:32}
-{-1993:16}
-{9:16}
+!le
+[0xdeadbeef:32]
+[-1993:16]
+[9:16]
+0x40
-{ICITTE:8}
+[ICITTE:8]
"meow mix"
+200~FFh
-{ICITTE:8}
+[ICITTE:8]
----
+
Output:
ff ff ff ff ff ff ff ff c8 ┆ •••••••••
----
+Transformation::
++
+Input:
++
+----
+"end of file @ " [end:8]
+
+!transform gzip
+ "this part will be gzipped"
+!end
+
+<end>
+----
++
+Output:
++
+----
+65 6e 64 20 6f 66 20 66 69 6c 65 20 40 20 3c 1f ┆ end of file @ <•
+8b 08 00 7b 7b 26 65 02 ff 2b c9 c8 2c 56 28 48 ┆ •••{{&e••+••,V(H
+2c 2a 51 28 cf cc c9 51 48 4a 55 48 af ca 2c 28 ┆ ,*Q(•••QHJUH••,(
+48 4d 01 00 d4 cc 5b 8a 19 00 00 00 ┆ HM••••[•••••
+----
+
Multilevel grouping::
+
Input:
function and the <<command-line-tool,command-line tool>>).
`normand.py` has _no external dependencies_, but if you're using
-Python{nbsp}3.4, you'll need a local copy of the standard `typing`
-module.
+Python{nbsp}3.4 or Python{nbsp}3.5, you'll need a local copy of the
+standard `typing` module.
====
== Design goals
Compare the following Normand input and some hypothetical XML
equivalent, for example:
+
-.Actual normand input.
+.Actual Normand input.
----
ff dd 01 ab $192 $-128 %1101:0011
-{end:8}
+[end:8]
{iter = 1}
!if {not something}
# five times because xyz
!repeat 5
- "hello world " {iter:8}
+ "hello world " [iter:8]
{iter = iter + 1}
!end
!end
|
The current offset has an effect on the value of <<label,labels>> and of
the special `ICITTE` name in <<fixed-length-number,fixed-length
-number>>, <<leb-128-integer,LEB128 integer>>, <<string,string>>,
+number>>, <<leb128-integer,LEB128 integer>>, <<string,string>>,
<<filling,filling>>, <<variable-assignment,variable assignment>>,
<<conditional-block,conditional block>>, <<repetition-block,repetition
block>>, <<macro-expansion,macro expansion>>, and
|[[cur-bo]] Current byte order
|
-The current byte order has an effect on the encoding of
+The current byte order can have an effect on the encoding of
<<fixed-length-number,fixed-length numbers>>.
A <<current-byte-order-setting,current byte order setting>> may change
little endian).
* A <<fixed-length-number,fixed-length number>> (integer or
- floating point) using the <<cur-bo,current byte order>> and of which
- the value is the result of a {py3} expression.
+ floating point), possibly using the <<cur-bo,current byte order>>, and
+ of which the value is the result of a {py3} expression.
* An <<leb128-integer,LEB128 integer>> of which the value is the result
of a {py3} expression.
* A <<repetition-block,repetition block>>.
+* A <<transformation-block,transformation block>>.
+
* A <<macro-definition-block,macro definition block>>.
* A <<macro-expansion,macro expansion>>.
nibbles and binary bits of <<byte-constant,byte constants>>:
----
-/ \ ? & : ; . , [ ] _ = | -
+& , - . / : ; = ? \ _ |
----
The latter serve to improve readability so that you may write, for
* The `0b` or `0B` prefix followed with one or more bits (`0` or `1`).
* One or more bits followed with the `b` or `B` suffix.
+In general, anything between `pass:[{]` and `}` is a {py3} expression.
+
You can test the examples of this section with the `normand`
<<command-line-tool,command-line tool>> as such:
Input:
----
-ab cd [3d 8F] CC
+ab cd (3d 8F) CC
----
Output:
The two accepted forms are:
[horizontal]
-``pass:[{be}]``:: Set the current byte order to big endian.
-``pass:[{le}]``:: Set the current byte order to little endian.
+`!be`:: Set the current byte order to big endian.
+`!le`:: Set the current byte order to little endian.
=== Fixed-length number
* A floating point number
(https://standards.ieee.org/standard/754-2008.html[IEEE{nbsp}754-2008]).
+
-The available length are 32 (_binary32_) and 64 (_binary64_).
+The available lengths are 32 (_binary32_) and 64 (_binary64_).
-The value is the result of evaluating a {py3} expression using the
-<<cur-bo,current byte order>>.
+The value is the result of evaluating a {py3} expression.
+
+The byte order to use to encode the value is either directly specified
+or is the <<cur-bo,current byte order>>.
A fixed-length number is:
-. The ``pass:[{]`` prefix.
+. The `[` prefix.
. A valid {py3} expression.
+
For a fixed-length number at some source location{nbsp}__**L**__, this
expression may contain the name of any accessible <<label,label>> (not
within a nested group), including the name of a label defined
-after{nbsp}__**L**__, as well as the name of any
-<<variable-assignment,variable>> known at{nbsp}__**L**__.
+after{nbsp}__**L**__ (except within a
+<<transformation-block,transformation block>>), as well as the name of
+any <<variable-assignment,variable>> known at{nbsp}__**L**__.
+
The value of the special name `ICITTE` (`int` type) in this expression
is the <<cur-offset,current offset>> (before encoding the number).
`32` and `64`.
--
-. The `}` suffix.
+. **Optional**: a suffix of the previous encoding length, without
+ any whitespace, amongst:
++
+--
+[horizontal]
+`be`:: Encode in big endian.
+`le`:: Encode in little endian.
+--
++
+Without this suffix, the encoding byte order is the <<cur-bo,current
+byte order>> which must be defined if the encoding length is greater
+than eight.
+
+. The `]` suffix.
====
Input:
----
-{le} {345:16}
-{be} {-0xabcd:32}
+[345:16le]
+[-0xabcd:32be]
----
Output:
Input:
----
-{be}
+!be
# String length in bits
-{8 * (str_end - str_beg) : 16}
+[8 * (str_end - str_beg) : 16]
# String
<str_beg>
Input:
----
-{20 - ICITTE : 8} * 10
+[20 - ICITTE : 8] * 10
----
Output:
Input:
----
-{le}
-{2 * 0.0529 : 32}
+[2 * 0.0529 : 32le]
----
Output:
An LEB128 integer is:
-. The ``pass:[{]`` prefix.
+. The `[` prefix.
. A valid {py3} expression of which the evaluation result type
is `int` or `bool` (automatically converted to `int`).
`sleb128`:: Use the signed LEB128 format.
--
-. The `}` suffix.
+. The `]` suffix.
====
Input:
----
-{624485 : uleb128}
+[624485 : uleb128]
----
Output:
aa bb cc dd
<meow>
ee ff
-{-981238311 + (meow * -23) : sleb128}
+[-981238311 + (meow * -23) : sleb128]
"hello"
----
Encoding suffix form:: {empty}
+
-. The ``pass:[{]`` prefix.
+. The `[` prefix.
. A valid {py3} expression of which the evaluation result type
is `bool`, `int`, `float`, or `str` (the first three automatically
ISO/IEC 8859-16.
--
-. The `}` suffix.
+. The `]` suffix.
====
Input:
{iter = 1}
!repeat 10
- {iter : s:u8} " "
+ u8{iter} " "
{iter = iter + 1}
!end
----
----
{meow = 'salut jérémie'}
-{meow.upper() : s:latin1}
+[meow.upper() : s:latin1]
----
Output:
Input:
----
- {ICITTE : 8} * 8
-<0x61> {ICITTE : 8} * 8
+ [ICITTE : 8] * 8
+<0x61> [ICITTE : 8] * 8
----
Output:
----
aa bb cc dd <meow> ee ff
<12> 11 22 33 <mix> 44 55
-{meow : 8} {mix : 8}
+[meow : 8] [mix : 8]
----
Output:
Input:
----
-{le}
+!le
77 88
-@32~0xcc {-893.5:32}
+@32~0xcc [-893.5:32]
@128~0x55 "meow"
----
** The ``pass:[{]`` prefix, a valid {py3} expression of which the
evaluation result type is `int` or `bool` (automatically converted to
- `int`), and the ``pass:[}]`` suffix.
+ `int`), and the `}` suffix.
+
For a filling at some source location{nbsp}__**L**__, this expression
may contain:
** A valid {py3} name.
+
For the name `__NAME__`, this is equivalent to the
-`pass:[{]__NAME__pass:[}]` form above.
+`pass:[{]__NAME__}` form above.
+
This value must be greater than or equal to the current offset where
----
!macro part(iter, fill)
- <0> "particular security " {ord('0') + iter : 8} +fill~0x80
+ <0> "particular security " [ord('0') + iter : 8] +fill~0x80
!end
{iter = 1}
Input:
----
-{mix = 101} {le}
-{meow = 42} 11 22 {meow:8} 33 {meow = ICITTE + 17}
-"yooo" {meow + mix : 16}
+{mix = 101} !le
+{meow = 42} 11 22 [meow:8] 33 {meow = ICITTE + 17}
+"yooo" [meow + mix : 16]
----
Output:
. The `(`, `!group`, or `!g` opening.
-. Zero or more items.
+. Zero or more items except, recursively, a macro definition block.
. Depending on the group opening:
+
Input:
----
-{be}
+!be
(
<str_beg> u16le"sébastien diaz" <str_end>
- {ICITTE - str_beg : 8}
- {(end - str_beg) * 5 : 24}
+ [ICITTE - str_beg : 8]
+ [(end - str_beg) * 5 : 24]
) * 3
<end>
----
** The ``pass:[{]`` prefix, a valid {py3} expression of which the
evaluation result type is `int` or `bool` (automatically converted to
- `int`), and the ``pass:[}]`` suffix.
+ `int`), and the `}` suffix.
+
For a conditional block at some source location{nbsp}__**L**__, this
expression may contain:
** A valid {py3} name.
+
For the name `__NAME__`, this is equivalent to the
-`pass:[{]__NAME__pass:[}]` form above.
+`pass:[{]__NAME__}` form above.
-. Zero or more items to be handled when the condition is true.
+. Zero or more items to be handled when the condition is true
+ except, recursively, a macro definition block.
. **Optional**:
.. The `!else` opening.
-.. Zero or more items to be handled when the condition is false.
+.. Zero or more items to be handled when the condition is false
+ except, recursively, a macro definition block
. The `!end` closing.
** The ``pass:[{]`` prefix, a valid {py3} expression of which the
evaluation result type is `int` or `bool` (automatically converted to
- `int`), and the ``pass:[}]`` suffix.
+ `int`), and the `}` suffix.
+
For a repetition block at some source location{nbsp}__**L**__, this
expression may contain:
** A valid {py3} name.
+
For the name `__NAME__`, this is equivalent to the
-`pass:[{]__NAME__pass:[}]` form above.
+`pass:[{]__NAME__}` form above.
-. Zero or more items.
+. Zero or more items except, recursively, a macro definition block.
. The `!end` closing.
----
!repeat 0o400
- {end - ICITTE - 1 : 8}
+ [end - ICITTE - 1 : 8]
!end
<end>
----
====
+=== Transformation block
+
+A _transformation block_ represents the bytes of one or more items
+transformed into other bytes by a function.
+
+As of this version, Normand only offers a predetermined set of
+transformation functions.
+
+An encoded block is:
+
+. The `!transform` or `!t` opening.
+
+. A transformation function name amongst:
++
+--
+[horizontal]
+`base64`::
+`b64`::
+ Standard https://datatracker.ietf.org/doc/html/rfc4648.html#section-4[Base64].
+
+`base64u`::
+`b64u`::
+ URL-safe Base64, using `-` instead of `pass:[+]` and `_` instead of
+ `/`.
+
+`base32`::
+`b32`::
+ Standard https://datatracker.ietf.org/doc/html/rfc4648.html#section-6[Base32].
+
+`base16`::
+`b16`::
+ Standard https://datatracker.ietf.org/doc/html/rfc4648.html#section-8[Base16].
+
+`ascii85`::
+`a85`::
+ https://en.wikipedia.org/wiki/Ascii85[Ascii85] without padding.
+
+`ascii85p`::
+`a85p`::
+ Ascii85 with padding.
+
+`base85`::
+`b85`::
+ https://en.wikipedia.org/wiki/Ascii85[Base85] (like Git-style binary
+ diffs) without padding.
+
+`base85p`::
+`b85p`::
+ Base85 with padding.
+
+`quopri`::
+`qp`::
+ MIME
+ https://datatracker.ietf.org/doc/html/rfc2045#section-6.7[quoted-printable]
+ without quoted whitespaces.
+
+`quoprit`::
+`qpt`::
+ MIME quoted-printable with quoted whitespaces.
+
+`gzip`::
+`gz`::
+ https://en.wikipedia.org/wiki/Gzip[gzip].
+
+`bzip2`::
+`bz2`::
+ https://en.wikipedia.org/wiki/Bzip2[bzip2].
+--
+
+. Zero or more items except, recursively, a macro definition block.
++
+Any {py3} expression within any of those items may not refer to a future
+<<label,label>>.
++
+The value of the special name `ICITTE` in any {py3} expression within
+any of those items is the <<cur-offset,current offset>> _before_ Normand
+applies the transformation function. Therefore, labels defined within
+those items also have the current offset value _before_ Normand applies
+the transformation function.
+
+. The `!end` closing.
+
+The <<cur-offset,current offset>> after having handled the last item of
+a transformation block is the value of the current offset before
+handling the first item plus the size of the generated (transformed)
+bytes. In other words, <<current-offset-setting,current offset
+settings>> within the items of the block have no impact outside said
+block.
+
+====
+Input:
+
+----
+aa bb cc dd
+
+"size of compressed section: " [end - start : 8]
+
+<start>
+
+!transform bzip2
+ "this will be compressed!"
+ 89*100 00*5000
+!end
+
+<end>
+
+"yes!"
+----
+
+Output:
+
+----
+aa bb cc dd 73 69 7a 65 20 6f 66 20 63 6f 6d 70 ┆ ••••size of comp
+72 65 73 73 65 64 20 73 65 63 74 69 6f 6e 3a 20 ┆ ressed section:
+52 42 5a 68 39 31 41 59 26 53 59 68 e1 8c fc 00 ┆ RBZh91AY&SYh••••
+00 33 d1 e0 c0 00 60 00 5e 66 dc 80 00 20 00 80 ┆ •3••••`•^f••• ••
+00 08 20 00 31 40 d3 43 23 26 20 ca 87 a9 a1 e8 ┆ •• •1@•C#& •••••
+18 29 44 80 9c 80 49 bf cc b3 e8 45 ed e2 76 ad ┆ •)D•••I••••E••v•
+0f 12 8b 8a d6 cd 40 04 7e 2e e4 8a 70 a1 20 d1 ┆ ••••••@•~.••p• •
+c3 19 f8 79 65 73 21 ┆ •••yes!
+----
+====
+
+====
+Input:
+
+----
+88*16
+
+!t a85
+ "I am determined to be cheerful and happy in whatever situation "
+ "I may find myself. For I have learned that the greater part of "
+ "our misery or unhappiness is determined not by our circumstance "
+ "but by our disposition."
+!end
+
+@128~99h
+
+!t qp <beg> [ICITTE - beg : 8] * 50 !end
+----
+
+Output:
+
+----
+88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 ┆ ••••••••••••••••
+38 4b 5f 47 59 2b 43 6f 26 2a 41 54 44 58 25 44 ┆ 8K_GY+Co&*ATDX%D
+49 6d 3f 24 46 44 69 3a 32 41 4b 59 4a 72 41 53 ┆ Im?$FDi:2AKYJrAS
+23 6d 6f 46 5f 69 31 2f 44 49 61 6c 27 40 3b 70 ┆ #moF_i1/DIal'@;p
+31 32 2b 44 47 5e 39 47 41 28 45 2c 41 54 68 58 ┆ 12+DG^9GA(E,AThX
+2a 2b 45 4d 37 3d 46 5e 5d 42 2b 44 66 2d 5b 68 ┆ *+EM7=F^]B+Df-[h
+2b 44 6b 50 34 2b 44 2c 3e 2a 41 30 3e 60 37 46 ┆ +DkP4+D,>*A0>`7F
+28 4b 30 22 2f 67 2a 57 25 45 5a 64 70 72 42 4f ┆ (K0"/g*W%EZdprBO
+51 27 71 2b 44 62 55 74 45 63 2c 48 21 2b 45 56 ┆ Q'q+DbUtEc,H!+EV
+3a 2a 46 3c 47 5b 3d 41 4b 59 57 2b 41 52 54 5b ┆ :*F<G[=AKYW+ART[
+6c 45 5a 66 3d 30 45 63 60 46 42 41 66 75 23 37 ┆ lEZf=0Ec`FBAfu#7
+45 5a 66 34 35 46 28 4b 42 3b 2b 45 29 39 43 46 ┆ EZf45F(KB;+E)9CF
+60 28 6c 24 45 2c 5d 4e 2f 41 54 4d 6f 38 42 6c ┆ `(l$E,]N/ATMo8Bl
+62 44 2d 41 54 56 4c 28 44 2f 21 6d 21 41 30 3e ┆ bD-ATVL(D/!m!A0>
+63 2e 46 3c 47 25 3c 2b 45 29 43 43 2b 43 66 2c ┆ c.F<G%<+E)CC+Cf,
+2b 40 73 29 58 30 46 43 42 26 73 41 4b 59 48 29 ┆ +@s)X0FCB&sAKYH)
+46 3c 47 25 3c 2b 45 29 43 43 2b 43 6f 32 2d 45 ┆ F<G%<+E)CC+Co2-E
+2c 54 66 33 46 44 35 5a 32 2f 63 99 99 99 99 99 ┆ ,Tf3FD5Z2/c•••••
+3d 30 30 3d 30 31 3d 30 32 3d 30 33 3d 30 34 3d ┆ =00=01=02=03=04=
+30 35 3d 30 36 3d 30 37 3d 30 38 3d 30 39 0a 3d ┆ 05=06=07=08=09•=
+30 42 3d 30 43 0d 3d 30 45 3d 30 46 3d 31 30 3d ┆ 0B=0C•=0E=0F=10=
+31 31 3d 31 32 3d 31 33 3d 31 34 3d 31 35 3d 31 ┆ 11=12=13=14=15=1
+36 3d 31 37 3d 31 38 3d 31 39 3d 31 41 3d 31 42 ┆ 6=17=18=19=1A=1B
+3d 31 43 3d 31 44 3d 31 45 3d 31 46 20 21 22 23 ┆ =1C=1D=1E=1F !"#
+24 25 26 27 28 29 2a 2b 2c 2d 3d 0a 2e 2f 30 31 ┆ $%&'()*+,-=•./01
+----
+====
+
=== Macro definition block
A _macro definition block_ associates a name and parameter names to
====
----
!macro bake()
- {le} {ICITTE * 8 : 16}
+ !le [ICITTE * 8 : 16]
u16le"predict explode"
!end
----
{iter = 1}
!repeat rep
- {val + iter : uleb128}
- {0xdeadbeef : 32}
+ [val + iter : uleb128]
+ [0xdeadbeef : 32]
{iter = iter + 1}
!end
* The ``pass:[{]`` prefix, a valid {py3} expression of which the
evaluation result type is `int` or `bool` (automatically converted to
- `int`), and the ``pass:[}]`` suffix.
+ `int`), and the `}` suffix.
+
For a macro expansion at some source location{nbsp}__**L**__, this
expression may contain:
----
!macro bake()
- {le} {ICITTE * 8 : 16}
+ !le [ICITTE * 8 : 16]
u16le"predict explode"
!end
----
!macro A(val, is_be)
- {le}
+ !le
!if is_be
- {be}
+ !be
!end
- {val : 16}
+ [val : 16]
!end
!macro B(rep, is_be)
Input:
----
-!macro flt32be(val) {be} {val : 32} !end
+!macro flt32be(val) !be [val : 32] !end
"CHEETOS"
m:flt32be(-42.17)
** An <<leb128-integer,LEB128 integer>>.
** A <<string,string>>.
** A <<macro-expansion,macro-expansion>>.
+** A <<transformation-block,transformation block>>.
** A <<group,group>>.
. The ``pass:[*]`` character.
** The ``pass:[{]`` prefix, a valid {py3} expression of which the
evaluation result type is `int` or `bool` (automatically converted to
- `int`), and the ``pass:[}]`` suffix.
+ `int`), and the `}` suffix.
+
For a post-item repetition at some source location{nbsp}__**L**__, this
expression may contain:
Input:
----
-{end - ICITTE - 1 : 8} * 0x100 <end>
+[end - ICITTE - 1 : 8] * 0x100 <end>
----
Output:
* https://github.com/psf/black[Black]
* https://pycqa.github.io/isort/[isort]
+Licensing and copyright follows the
+https://reuse.software/tutorial/[REUSE] specification and is checked
+with the https://github.com/fsfe/reuse-tool[reuse tool].
+
=== Testing
Use https://docs.pytest.org/[pytest] to test Normand once the package is
projects / normand.git / blobdiff