GNU poke Manual ¶

1.1 Motivation ¶

The main purpose of GNU poke is to manipulate structured binary data in terms of abstractions provided by the user. The Poke type definitions can be seen as a sort of declarative specifications for decoding and encoding procedures. The user specifies the structure of the data to be manipulated, and poke uses that information to automagically decode and encode the data. Under this perspective, struct types correspond to sequences of instructions, array types to repetitions or loops, union types to alternatives or conditionals, and so on.

• Decode-Compute-Encode
• Describe-Compute

void double_number (int fd, off_t offset, int endian)
{
   int number, i;
   unsigned char b[4];

   /* Decode.  */
   lseek (fd, offset, SEEK_SET);
   for (i = 0; i < 4; ++i)
      read (fd, &b[i], 1);

   if (endian == BIG)
     number = b[0] << 24 | b[1] << 16 | b[2] << 8 | b[3];
   else
     number = b[3] << 24 | b[2] << 16 | b[1] << 8 | b[0];

   /* Compute.  */
   number = number * 2;

   /* Encode.  */
   if (endian == BIG)
   {
     b[0] = (number >> 24) & 0xff;
     b[1] = (number >> 16) & 0xff;
     b[2] = (number >> 8) & 0xff;
     b[3] = number & 0xff;
   }
   else
   {
     b[3] = (number >> 24) & 0xff;
     b[2] = (number >> 16) & 0xff;
     b[1] = (number >> 8) & 0xff;
     b[0] = number & 0xff;
   }

   lseek (fd, offset, SEEK_SET);
   for (i = 0; i < 4; ++i)
      write (fd, &b[i], 1);
}

type Packet =
  struct
  {
    uint<16> magic == 0xef;
    uint<32> size;
    byte[size] data @ 8#B;
  };

1.2 Nomenclature ¶

GNU poke is a new program and it introduces many new concepts. It is a good idea to clarify how we call things in the poke community. Unless everyone uses the same nomenclature to refer to pokish thingies, it is gonna get very confusing very soon!

First of all we have poke, the program. Since “poke” is a very common English word, when the context is not clear we either use the full denomination GNU poke, or quote the word using some other notation.

Then we have Poke, with upper case P, which is the name of the domain-specific programming language implemented by poke, the program.

This distinction is important. For example, when people talk about “poke programmers” they refer to the group of people hacking GNU poke. When they talk about “Poke programmers” they refer to the people who write programs using the Poke programming language.

Finally, a pickle is a Poke source file containing definitions of types, variables, functions, etc, that conceptually apply to some definite domain. For example, elf.pk is a pickle that provides facilities to poke ELF object files. Pickles are not necessarily related to file formats: a set of functions to work with bit patterns, for example, could be implemented in a pickle named bitpatterns.pk.

We hope this helps to clarify things.

1.3 Invoking poke ¶

Synopsis:

poke [option...] [file]

The following options are available.

‘-l’

‘--load=file’

Load the given file as a Poke program. Any number of ‘-l’ options can be specified, and they are loaded in the given order.

‘-L file’

Load the given file as a Poke program and exit. The rest of the command-line is not processed by poke, and is available to the Poke script in the argv variable. This is commonly used along with a shebang (see Scripts) to implement Poke scripts.

Commanding poke from the command line:

‘-c’

‘--command=cmd’

Execute the given command. Any number of ‘-c’ options can be specified, and they are executed in the given order.

‘-s’

‘--source=file’

Load file as a command file. Any number of ‘-s’ options may be specified, and they are loaded in the given order. See Command Files.

Styling text output:

‘--color=how’ ¶

Whether to use styled output, and how. Valid options for how are ‘yes’, ‘no’, ‘auto’, ‘html’ and ‘test’.

‘--style-dark’

Use the default style that works better for dark backgrounds. This is the default.

‘--style-bright’

Use the default style that works better for bright backgrounds.

‘--style=file’

Use file as the CSS to use for styling poke, instead of the default style.

Other options:

-q

--no-init-file

Do not load the ~/.pokerc init file.

--no-auto-map

Do not load map-files automatically when poke opens IO spaces.

--no-hserver

Do not run the terminal hyperlinks server.

--no-stdtypes

Do not define standard Poke types (such as int, char, etc) in the Poke incremental compiler.

--quiet

Be as terse as possible.

--help

Print a help message and exit.

--version

Show version and exit.

The following environment variables, if set, are used by poke:

POKE_LOAD_PATH

List of file paths separated by the colon characters (:) which is prepended to the load_path when poke starts.

1.4 Commanding poke ¶

GNU poke is primarily an interactive editor that works in the command line. However, it is also possible to use it in a non-interactive way. This chapter documents both possibilities.

• The REPL
• Evaluation
• Commands and Dot-Commands
• Command Files
• Scripts

(poke)

(poke) 23
23
(poke) [1,2,3]
[1,2,3]
(poke) Packet @ 0#B
Packet {i=1179403647,j=65794L}

(poke) fun do_foo = void: {}
(poke) do_foo
(poke) for (i in [1,2,3]) printf "elem %i32d\n", i;
elem 1
elem 2
elem 3

(poke) [1,2,3 + "foo"]
<stdin>:1:6: error: invalid operands in expression
[1,2,3 + "foo"];
     ^~~~~~~~~

(poke) .vm disassemble expression int[] @ 0#B
(poke) .vm disassemble function (dump)

(poke) .vm disa e int[] @ 0#B
(poke) .vm disa f (dump)

(poke) .vm disassemble expression/n int[] @ 0#B
(poke) .vm disassemble function/n (dump)

(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0100 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 8001 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ....@.....@.....
00000040: 1800 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 7900 0000 0000 0000 b701 0000 9a02 0000  y...............
00000060: 7b10 0000 0000 0000 1800 0000 0000 0000  {...............
00000070: 0000 0000 0000 0000 7900 0000 0000 0000  ........y.......

(poke) dump :from 0#B :size 8#B
(poke) dump :from 0#B :size 8#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100                      .ELF....

(poke) dump :from 0#B :size 8#B :ascii 0 :ruler 0
00000000: 7f45 4c46 0201 0100
(poke) dump :ruler 0 :from 0#B :size 8#B :ascii 0
00000000: 7f45 4c46 0201 0100

# The following two lines are dot commands
.load my-pickle.pk
.set obase 16

# The following line is a Poke statement
dump :size 0x100#B :from 0x10#B

# The following line is a Poke expression statement without any side effect.
# Consequently it is valid, but rather useless.
4 == 4

type foo = struct {int this; int that;}

type foo = struct
{
 int this;
 int that;
}

#!/usr/bin/poke -L
!#

print "Hello world!\n";

#!/usr/bin/poke -L
!#

for (arg in argv)
  printf ("Argument: %s\n", arg);

#!/usr/bin/env sh
exec poke -L "$0" "$@"
!#

load elf;
open (argv[0]);
printf ("%v\n", Elf64_Ehdr @ 0#B);

2.1 Setting up Hyperlinks ¶

GNU poke uses terminal hyperlinks in order to improve the interactive usage of the tool: clicking on terminal hyperlinks requests poke to execute certain actions. This is used to implement buttons and other interactive goodies.

Many terminal emulators support terminal hyperlinks. However, we are using a very simple protocol called app:// that is not supported (yet) on GNU/Linux distros. Fortunately, it is very easy to set your system to use this protocol, and this chapter shows you how.

• Make sure your poke speaks hyperlinks
• Use a terminal emulator that supports hyperlinks
• Get and install the app-client utility
• GNOME Terminal

hserver listening in port 43713.

$ git clone https://gitlab.com/darnir/hyperlink-app-client
$ cd hyperlink-app-client
$ make
$ sudo make install

x-scheme-handler/app=app-client.desktop

2.2 Simple Init File ¶

GNU poke is a spartan program that tries to be as simple as possible by default, without fancy displays. Therefore, before exploring poke you may want to configure it minimally. This section contains a few recommendations in that respect.

First we must say that poke reads a per-user configuration from the ~/.pokerc file. See .pokerc.

We recommend new users to set the following options:

.set endian little
.set omode tree
.set oacutoff 5
.set pretty-print yes
.set pager yes

.set endian little will make poke to use little-endian by default when accessing IO spaces, such as files or memory buffers. See Big and Little Endians.

.set omode tree will make poke to format composite data structures, such as arrays and struct, in several lines in a nice tree-like format. By default the program will format these values in a single line (what we call the “plain” output mode) which can be difficult to read.

.set oacutoff 5 will make poke to output at most five elements in array values, followed by a dieresis ....

.set pretty-print yes will make poke to use pretty-printers when printing out values of struct types that have a pretty-printer defined for them. You can recognize pretty-printed values because, by convention, they are emitted between #< and > markers, like for example #<leb128:1234>. If you see a pretty-printed value and you need to look at its internal structure, you can disable pretty-printing using a .set pretty-print no dot-command at any time. See Pretty-printers.

.set pager yes will make poke to page the output of commands that emit output larger than the visible terminal. This of course only has an effect when running poke interactively.

These options are fully explained later in this manual. See .set.

3.1 Binary Files ¶

GNU poke is an editor for binary files. Right, so what is a binary file? Strictly speaking, every file in a computer’s file system is binary. This is because, in a very fundamental level, files are just sequences of bytes.

Colloquially, however, it is very common to talk about “binary files” as opposed to “text files”. In this informal meaning, a text file is basically a file composed, mostly, of bytes (and byte sequences) that can be translated into printable characters in some character set, such as ASCII, EBCDIC or Unicode. It follows that binary files would then be files composed, mostly, of bytes not intended to be interpreted as encoded characters.

Some text files contain non-printable characters, such as form feed characters, and many binary files contain printable strings, such as a string table in an ELF object file. That is why we used the word “mostly” in the definitions above. In practice, however, the distinction is almost always clear and there is common consensus on whether a given file format can be considered as a binary format, or not.

GNU poke can edit any file, and as we shall see, it provides some nice features to manipulate sequences of bytes interpreted as character strings. However, it is called a “binary editor” because it is especially designed to be particularly useful editing binary files, in the sense of the term defined above.

In this chapter, we will be using ELF object files as the experiment subject in most of the examples. ELF files are good for this purpose, because they are eminently binary, highly structured, and still strings play a role in them, encoding names of entities like sections and symbols. You don’t need to have a perfect knowledge of the ELF format in order to follow the examples, but being familiarized with the concept of object file formats should surely help.

You will need to install the ELF pickle. You can get it at https://jemarch.net/poke-elf.

Obtaining a simple ELF object file is easy, if you have a C compiler installed:

$ echo 'int foo () { return 0; }' | gcc -c -xc -o foo.o -

The command above compiles a very simple ELF object file that contains the compiled form of a little dummy function. This object file will be our companion for a while, and will be the subject of much analysis and abuse, as we poke it.

3.2 Files as IO Spaces ¶

Now that we have a binary file (foo.o) it is time to open it with poke. There are two ways to do that.

One way is to pass the name of the file in the poke invocation. The program will start, open the file, and present you with the REPL, like in:

$ poke foo.o
[...]
(poke)

The other way is to fire up poke without arguments, and then use the .file dot-command to open the file:

$ poke
[...]
(poke) .file foo.o
The current IOS is now `./foo.o'.
(poke)

Note how poke replies to the dot-command, stating that the current IOS is now the file we opened.

You may be wondering, what is this IOS thing? It is an acronym for Input/Output Space, often written IO Space. This is the denomination used to refer to the entities being edited with poke. In this case the IO space being edited is a file, but we will see that is not always the case: poke can also edit other entities such as memory buffers and remote block-oriented devices over the network. For now, let’s keep in mind that IOS, or IO space, refers to the file being edited.

And why “current”? GNU poke is capable of editing several files (more generally, several IO spaces) simultaneously. At any time, one of these files is the “current one”. In our case, the current IO space is the file foo.o, since it is the only file poke knows about:

(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o

The command .info ios gives us information about all the IO spaces that are currently open. The first column tells us a positive integer that identifies the IOS. This IO space identifier is unique at any given time. In this example, the id corresponding to foo.o is 0. The second column tells us the type of IO space. The third column tells us that foo.o allows both reading and writing. The fourth column tells us the size of the file, in hexadecimal. The fifth column is the name of the IO space; in this case, it is the path to the file being edited.

You may wonder what is that weird suffix #B. It is a unit, and tells us that the size 0x398 is measured in bytes, i.e. the size of foo.o is 0x398 bytes (or, in decimal, 920 bytes.)

Finally, the asterisk character at the left of the entry for foo.o identifies it as the current IO space. To see this more clearly, let’s open another file:

(poke) .file bar.o
The current IOS is now `./bar.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 1	FILE	rw	0x00000000#B	0x00000398#B	./bar.o
  0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o

Ah, there we have both foo.o and bar.o. Now the current IO space (the entry featuring the asterisk at the left) is the file that we just opened, bar.o. This is because poke always sets the most recently open file as the current one. We can switch back to foo.o using yet another dot-command, .ios, which gets an IO space as an argument:

(poke) .ios 0
The current IOS is now `./foo.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
  1	FILE	rw	0x00000000#B	0x00000398#B	./bar.o
* 0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o

We are back to foo.o. Since we are not really interested in bar.o, let’s close it:

(poke) .close 1
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 0	FILE	rw	0x00000000#B	0x00000398#B	./foo.o

Note how in the examples above we used the IO space identifiers (0 and 1) in order to refer to them. This can be inconvenient, as these ids are not memorable at all. Fortunately it is also possible to refer to an IO space by name using the $<name> construct, like this:

(poke) .ios $<./foo.o>
The current IOS is now `./foo.o'.
(poke) .close $<./bar.o>

Note that the $<name> construct in lenient in the sense that it will accept any non-empty sub string that identifies an existing IO space in an unambiguous way. This is useful in order to avoid having to write annoying paths. For example:

(poke) .file /bin/ls
(poke) .file /bin/cat
(poke) .ios $<ls>
The current IOS is now `/bin/ls'.

Auto completion also works. Try it!

3.3 Dumping File Contents ¶

Data stored in modern computers, in both volatile memory and persistent files, is fundamentally a sequence of entities called bytes. The bytes can be addressed by its position in the sequence, starting with zero:

+--------+--------+--------+ ... +--------+
| byte 0 | byte 1 | byte 2 |     | byte N |
+--------+--------+--------+ ... +--------+

Each byte has capacity to store a little unsigned integer in the range 0..255. Therefore, the IO spaces that we edit with poke (like the file foo.o) can be seen as a sequence of little numbers, like depicted in the figure above.

GNU poke provides a command whose purpose is to display the values of these bytes: dump¹ . It is called like that because it dumps ranges of bytes to the terminal, allowing the user to inspect them.

So let’s use our first poke command! Fire up poke, open the file foo.o as explained above, and execute the dump command:

(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 9801 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ..............
00000040: 2564 0a00 0000 0000 0000 0000 0000 0000  %d..............
00000050: b702 0000 0100 0000 1801 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 8510 0000 ffff ffff  ................
00000070: b700 0000 0000 0000 9500 0000 0000 0000  ................
(poke)

What are we looking at?

The first line of the output, starting with 76543210, is a ruler. It is there to help us to visually determine the location (or offset) of the data.

The rest of the lines show the values of the bytes that are stored in the file, 16 bytes per line. The first column in these data lines shows the offset, in hexadecimal and measured in number of bytes, from which the row of data starts. For example, the offset of the first byte shown in the third data line has offset 0x20 in the file, the second byte has offset 0x21, and so on. Note how the data rows show the values of the individual bytes, in hexadecimal. Generally speaking, when dealing with bytes (and binary data in general) it is useful to manipulate magnitudes in hexadecimal, or octal. This is because it is easy to group digits in these bases to little groups of bits (four and three respectively) in the equivalent binary representation. In this case, each couple of hexadecimal digits denote the value of a single byte². For example, the value of the first byte in the third data row is 0x01, the value of the second byte 0x02, and so on.

Using the ruler and the column of offsets, locating bytes in the data is very easy. Let’s say for example we are interested in the byte at offset 0x68: we use the first column to quickly find the row starting at 0x60, and the ruler to find the column marked with 88. Cross column and row and… voila! The byte in question has the value 0x85. The reverse process is just as easy. What is the offset of the first 0x40 in the file? Try it!

The section at the right of the output is the ASCII output. It shows the row of bytes at the left interpreted as ASCII characters. Non-printable characters are shown as . to avoid scrambling the terminal, and yes, there is actually way to customize what character to use, so they are not confused from real ASCII dot characters (0x2e) :P In this particular dump we can see that near the beginning of the file there are three bytes whose value, if interpreted as ASCII characters, conform the string “ELF”. As we shall see, this is part of the ELF magic number. Again, the ruler is very useful to locate the byte corresponding to some character in the ASCII section, or the other way around. What is the value of the byte corresponding to the F in ELF? Try it!

Something to notice in the dump output above is that these are not, by any mean, the complete contents of the file foo.o. The .info ios dot-command informed us in the last section that foo.o contains 920 bytes, of which the dump command only showed us… 0x80 bytes, or 128 bytes in decimal.

dump is certainly capable of showing more (and less) than 128 bytes. We can ask dump to display some given amount of data by specifying its size using a command argument. For example:

(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 9801 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ..............

The command above asks poke to “dump 64 bytes”. In this example :size is the name of the argument, and 64#B is the argument’s value. Again, the suffix #B tells poke we want to dump 64 bytes, not 64 kilobits nor 64 potatoes.

Another interesting aspect of our first dump (ahem) is that the dumped bytes start from the beginning of the file, i.e. the offset of the first byte is 0x0. Certainly there should be other areas of the file with interesting contents for us to inspect. To that purpose, we can use yet another option, :from:

(poke) dump :size 64#B :from 128#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000080: 1400 0000 0000 0000 0000 0000 0000 0000  ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000a0: 0000 0000 0300 0100 0000 0000 0000 0000  ................
000000b0: 0000 0000 0000 0000 0000 0000 0300 0300  ................

The command above asks poke to “dump 64 bytes starting at 128 bytes from the beginning of the file”. Note how the first row of bytes start at offset 0x80, i.e. 128 in decimal.

Passing options to commands is easy and natural, but we may find ourselves passing the same values again and again to certain command options. For example, if the default size of dump of 128 bytes is not what you prefer, because you have a particularly tall monitor, or you are one of these people using sub-atomic sized fonts, it can be tiresome and error-prone to pass :size to dump every time you use it. Fortunately, the default size can be customized by setting a global variable:

(poke) pk_dump_size = 160#B

This tells poke to set 160 bytes as the new value for the pk_dump_size variable. This is a global variable that the dump command uses to determine how much data to show if the user doesn’t specify an explicit value with the :size option. Many other commands use the same strategy in order to alter their default behavior, not just dump.

And now that we are talking about that, it is also cumbersome to have to set the default size used by dump every time we run poke. But no problem, just set the variable in a file called .pokerc in your home directory, like this:

pk_dump_size = 160#B

Every time poke starts, it reads ~/.pokerc and executes the commands contained in it. See .pokerc.

The dump command is very flexible, and accepts a lot of options and customization variables that we won’t be covering in this chapter. For a complete description of the command, see dump.

3.4 Poking Bytes ¶

Let’s look again at the first bytes of the file foo.o:

(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 9801 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ..............

At this point we know how to use the ruler to localize specific bytes just by looking at the displayed data. If we wanted to operate on the values of some given bytes, we could look at the dump and type the values in the REPL. For example, if we wanted to add the values of the bytes at offsets 0x2 and 0x4, we could look at the dump and then type:

(poke) 0x4c + 0x02
0x4e

GNU poke supports many operators that take integers as arguments, to perform arithmetic, relational, logical and bit-wise operations on them (see Integers). Since bytes are no more (and no less) than little unsigned integers, we can use these operators to perform calculations on bytes.

For example, this is how we would calculate whether the highest bit in the second byte in foo.o is set:

(poke) 0x45 & 0x80
0

Note how booleans are encoded in Poke as integers, 0 meaning false, any other value meaning true.

Looking at the output of dump and writing the desired byte value in the prompt is cumbersome. Fortunately, there is a much more convenient way to access the value of a byte, given its offset in the file: it is called mapping a byte value. This operation is implemented by a binary operator, called the map operator.

This is how it works. Assuming we were interested in the byte at 64 bytes from the beginning of the file, this is how we would refer to it (or “map” it):

(poke) byte @ 64#B
37UB

This application of the map operator tells poke to map a byte at the offset 64 bytes. It can be read as “byte at 64 bytes”. Note how poke replies with the value 37UB. The suffix UB means “unsigned byte”, and is an indication for the user about the nature of the preceding number: it is unsigned, and it occupies a byte when stored.

As we can see in this example, poke uses decimal by default when showing values in the REPL. We already noted how it is usually better to work in hexadecimal when dealing with byte values. Fortunately, we can change the numeration base used by poke when printing numbers, using the .set obase (“set output base”) dot-command as this:

(poke) .set obase 16

After this, we can map the byte again, this time getting the result expressed in hexadecimal:

(poke) byte @ 64#B
0x25UB

Again, you may find it useful to add the .set obase 16 command to your .pokerc file, if you want the customization to be persistent between poke invocations.

Going back to the example of calculating whether the highest bit in the second byte in foo.o is set, this is how we would do it with a map:

(poke) (byte @ 2#B) & 0x80
0

Turns out the answer is no.

The map operator can also be used at the left side of an assignment operator:

(poke) byte @ 0x28#B = 0xff

Which reads “assign 0xff to the byte at offset 0x4a bytes”. Dumping again, we can verify that the byte actually changed:

76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 f700 0102 0000 0000 0000 0000 0000  ................
00000020: 0102 0000 0000 0000 ff01 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0800 0700  ..............

Does this mean that foo.o changed accordingly, in disk? The answer is yes. poke always commits changes immediately to the file being edited. This, that is an useful feature, can also be a bit tricky if you forget about it, leading to data corruption, so please be careful.

Incidentally, altering the byte at offset 0x28 most probably have caused foo.o to stop being a valid ELF file, but since we are just editing bytes (and not ELF structures) we actually don’t care much.

3.5 Values and Variables ¶

Up to now we have worked with byte values, either writing them in the REPL or mapping them at the current IO space. Often it is useful to save values under meaningful names, and access to them by name. In poke we do that by storing the values in variables.

Before being used, variables shall be defined using the var construction. Let’s get the byte at offset 64 bytes and save it in a variable called foo:

(poke) var foo = byte @ 64#B

This defines a new variable (foo) and initializes it to the value of the byte at offset 64 bytes. This results on foo to hold the value 37.

Once defined, we can get the value of a variable by just giving its name to poke:

(poke) foo
37UB

Several variables can be defined in a single var declaration. Each definition is separated by a comma. This is equivalent of issuing several vars:

(poke) var a = 10, b = 20

In general, a variable containing a byte value can be used in any context where the contained value would be expected. If we wanted to check the highest bit in the byte value stored in foo we would do:

(poke) foo & 0x80
0x0

Assigning a value to a variable makes the value the new contents of the variable. For example, we can increase the value of foo by one like this:

(poke) foo = foo + 1

At this point, an important question is: when we change the value of the variable foo, are we also changing the value of the byte stored in foo.o at offset 64 bytes? The answer is no. This is because when we do the mapping:

(poke) var foo = byte @ 64#B

The value stored in foo is a copy of the value returned by the map operator @. You can imagine the variable as a storage cell located somewhere in poke’s memory. After the assignment above is executed there are two copies of the byte value 0x25: one in foo.o at offset 64 bytes, and the other in the variable foo.

It follows that if we wanted to increase the byte in the file, we would need to do something like:

(poke) var foo = byte @ 64#B
(poke) foo = foo + 1
(poke) byte @ 64#B = foo

Or, more succinctly, omitting the usage of a variable:

(poke) byte @ 64#B = (byte @ 64#B) + 1

Or, even more succinctly:

(poke) (byte @ 64#B) += 1

Note how we have to use parenthesis around the map at the right hand side, because the map operator @ has less precedence than the plus operator +.

3.6 From Bytes to Integers ¶

The bytes we have been working with are unsigned whole numbers (or integers) in the range 0..255. We saw how poke sees the contents of the files as a sequence of bytes, and how each byte can be addressed using an offset. Mapping bytes using the map operator @ gives us these values, which are denoted in poke with literals like 10UB or 0x0aUB.

This very limited range of values have consequences when it comes to do arithmetic with bytes. Suppose for example we wanted to calculate the average of the first three byte values stored in foo.o. We could do something like:

(poke) a0 = byte @ 0#B
(poke) a1 = byte @ 1#B
(poke) a2 = byte @ 2#B
(poke) a0
0x7fUB
(poke) a1
0x45UB
(poke) a2
0x4cUB
(poke) (a0 + a1 + a2) / 3UB
5UB

That is obviously the wrong answer. What happened? Let’s do it step by step. First, we add the first two bytes:

(poke) a0 + a1
0xc4UB

Which is all right. 0xc4 is 0x7f plus 0x45. But, let’s add now the third byte:

(poke) a0 + a1 + a2
0x10UB

That’s no good. Adding the value of the third byte (0x4c) we overflow the range of valid values for a byte value. The calculation went banana at this point.

Another obvious problem is that we surely will want to store integers bigger than 255 in our files. Clearly we need a way to encode them somehow, and since all we have in a file are bytes, the integers will have to be composed of them.

Integers bigger than 255 can be encoded by interpreting consecutive byte values in a certain way. First, let’s consider a single byte. If we print a byte value using binary rather than decimal or hexadecimal, we will observe that eight bits are what it takes to encode the numbers between 0 and 0xff (255) using a natural binary encoding:

(poke) .set obase 2
(poke) 0UB
0b00000000UB
(poke) 0xFFUB
0b11111111UB

This is the reason why people say bytes are “composed” of eight bits, or that the width of a byte is eight bits. But this way of talking doesn’t really reflect the view that the operating system has of devices like files or memory buffers: both disk and memory controllers provide and consume bytes, i.e. little unsigned numbers in the range 0..255. At that level, bytes are indivisible. We will see later that poke provides ways to work on the “sub-byte” level, but that is just really an artifact to make our life easier: underneath, all that goes in and out are bytes.

Anyhow, if we were to “concatenate” the binary representation of two consecutive bytes, we would end with a much bigger range of possible numbers, in the range 0b00000000_00000000..0b11111111_11111111³, or 0x0000..0xffff in hexadecimal. poke provides a bit-concatenation operator ::: that does exactly that:

(poke) 0x1UB
0b00000001UB
(poke) 0x1UB ::: 0x1UB
0b0000000100000001UH

Note how the suffix of the resulting number is now UH. This indicates that the number is no longer a byte value: it is too big for that. The H in this new suffix means “half”, and it is a traditional way to call an integer that is encoded using two bytes, or 16 bits.

So, using our method of encoding bigger numbers concatenating bytes, what would be the “half” integer composed of two bytes at the beginning of foo.o?

(poke) .set obase 16
(poke) (byte @ 0#B):::(byte @ 1#B)
0x7f45UH

Now, let’s go back to the syntax we used to map a byte value. In the invocation of the map operator byte @ 0#B the operand at the left (in this case byte) tells the operator what kind of value to map. This is called a type specifier; byte is the type specifier for a single byte value, and byte[3] is the type specifier for a group of three byte values arranged in an array.

As it happens, byte is a synonym for another slightly more interesting type specifier: uint<8>. You can probably infer the meaning already: a byte is an unsigned integer which is 8 bits big. We can of course use this alternate specifier in a mapping operation, achieving exactly the same result than if we were using byte:

(poke) uint<8> @ 0#B
0x7fUB

You may be wondering: is it possible to use a similar type specifier for mapping bigger integers, like these “halves” that are composed of two bytes? Yeah, it is indeed possible:

(poke) uint<16> @ 0#B
0x7f45UH

Mapping an unsigned integer of 16-bits at the offset 0 gives us an unsigned “half” value, as expected.

You can easily build bigger and bigger numbers concatenating more and more bytes. Three bytes? sure:

(poke) uint<24> @ 0#B
(uint<24>) 0x7f454c

Note that in this case poke uses a prefix instead of a suffix to indicate that the given value is 24-bits long. This is because only a limited number of suffixes (which are more concise and more readable than the prefix form) are available, corresponding to common or typical widths.

Four bytes?

(poke) uint<32> @ 0#B
0x7f454c46U

Certain integer widths are so often used that easier-to-type synonyms for their type specifiers are provided. We already know byte for uint<8>. Similarly, ushort is a synonym for uint<16>, uint is a synonym for uint<32> and ulong is a synonym for uint<64>. Try them!

GNU poke supports integers up to eight bytes, i.e. up to 64-bits. This may change in the future, as we are planning to support arbitrarily large integers.

3.7 Big and Little Endians ¶

When talking about whole numbers (integers) we should distinguish between their value (such as 123) and their written form that we would use when writing the number on a piece of paper, such as 123.

The written form of a number is composed of digits, arranged in certain order. We all know that the ordering of the digits in the written form of a number is important: if we write 123 we are referring to a different value than if we write 321. The mathematical reason for this is that depending on the position they occupy in the written form, each digit contributes with a different “weight” to the total value of the number. This is always the case, regardless of the numerical base used to denote the number.

For example, the value of the number 123 (whose written form is 123) is calculated as 1*10^2+2*10^1+3*10^0. If we swap the last two digits in the written form of the number, 132, we have 1*10^2+3*10^1+2*10^0, which results in a different value: 132. When we consider other numerical bases, the bases in the polynomial change accordingly, but the correspondence between written form and value stands: for example, the value of 0x123 is calculated as 1*16^2+2*16^1+3*16^0.

The “higher” a digit is in the polynomial, the more significant it is, i.e. the more weight it has on the value of the number where it appears. In the written number 123, for example, the digit 1 is the most significant digit of the number, and the digit 3 is the least significant digit.

This distinction between the written form of a number and its value is very important. Just like in certain languages letters are read right-to-left (Arabic) or even down-to-up (Japanese) we could certainly conceive a language in which the digits of numbers were arranged from right-to-left instead of left-to-right. In such a language the written representation of 123 would be 321, not 123. In other words: the least significant digit would come first, not last, in the written form of the number.

Now when it comes to store numbers in computers, rather than writing them on a paper, the role of the paper is played by the computer’s memory, be it ephemeral (like RAM) or persistent (like a spinning hard disk or a Flash memory), which is organized as a sequence of bytes. Since we are composing numbers with bytes, it makes sense to have each byte to play the role of a digit in the written form of the bigger number. Since bytes can have values from 0 to 255, the base is 256. But what is the “written form” for our byte-composed numbers?

In the last section we tried to compose bigger integers by concatenating bytes together and interpreting the result. In doing so, we assumed (quite naturally) that in the written form of the resulting integer the bytes are ordered in the same order than they appear in the file, i.e. we assume that the written form of the number b1*256^2+b2*256^1+b3*256^0 would be b1b2b3, where b1, b2 and b3 are bytes. In other words, given a written form b1b2b3, b1 would be the most significant byte (digit) and b3 would be the least significant byte (digit). In our world of IO spaces, the “written form” is the disposition of the bytes in the IO space (file, memory buffer, etc) being edited.

That interpretation of the written form is exactly what the bit-concatenation operator implements:

(poke) dump :from 0#B :size 3#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c                                  .EL
(poke) var b1 = byte @ 0#B
(poke) var b2 = byte @ 1#B
(poke) var b3 = byte @ 2#B
(poke) b1:::b2:::b3
(uint<24>) 0x7f454c

However, much like in certain human languages the written form is read from right to left, some computers also read numbers from right to left in their “written form”. Actually, turns out that most modern computers do it like that. This means that, in these computers, given the written form b1b2b3 (i.e. given a file where b1 comes first, followed by b2 and then b3) the most significant byte is b3 and the least significant byte is b1. Therefore, the value of the number would be b3*256^2+b2*256^1+b3*256^0.

So, given the written form of a bigger number b1b2b3 (i.e. some ordering of bytes implied by the file they are stored in) there are at least two ways to interpret them to calculate the value of the number. When the written form is read from left to right, we talk about a big endian interpretation. When the written form is read from right to left, we talk about a little endian interpretation.

Given the first three bytes in foo.o, we can determine the value of the integer composed of these three bytes in both interpretations:

(poke) b1:::b2:::b3
(uint<24>) 0x7f454c
(poke) b3:::b2:::b1
(uint<24>) 0x4c457f

Remember how the type specifier byte is just a synonym of uint<8>, and how we can use type specifiers like uint<24> and uint<32> to map bigger integers? When we do that, like in:

(poke) uint<24> @ 0#B
(uint<24>) 0x7f454c

Poke should somehow decide what kind of interpretation to use, i.e. how to read the “written form” of the number. As you can see from the example, poke uses the left-to-right interpretation, or big-endian, by default. But you can change it using a new dot-command: .set endian:

(poke) .set endian little
(poke) uint<24> @ 0#B
(uint<24>) 0x4c457f

The currently used interpretation (also called endianness) is shown if you invoke the dot-command without an argument⁴:

(poke) .set endian
little

Different systems use different endianness. Into a given system, it is to be expected that most files will be encoded following the same conventions. Therefore poke provides you a way to set the endianness to whatever endianness is in the system. You do it this way:

(poke) .set endian host

3.8 Negative Integers ¶

• Negative Encodings
• Signed Integers
• Mixing Signed and Unsigned Integers

(poke) 0UB - 1UB
0xffUB

(poke) .set obase 10
(poke) int<16> @ 0#B
28515H

(poke) var h = int<16> @ 0#B
(poke) h - h - 1H
-1H

(poke) 1 + 2
3

(poke) 1U + 2U
3U

(poke) 1U + 2
3U

(poke) 2 as uint<32>
2U

(poke) 1U + 2 as uint<32>
3U

3.9 Weird Integers ¶

Up to this point we have been playing with integers that are built using a whole number of bytes. However, we have seen that the type specifier for an integer has the form int<N> or uint<N> for signed and unsigned variants, where N is the width of the integer, in bits. We have used bit-sizes that are multiple of 8, which is the size of a byte. So, why is this so? Why is N not measured in bytes instead?

The reason is that poke is not limited to integers composed of a whole number of bytes. You can actually have integers composed of any number of bits, between 1 and 64. So yes, int<3> is a type specifier for signed 3-bit integers, and uint<17> is a type specifier for unsigned 17-bit integers.

We call integers like this weird integers.

The vast majority of programming languages do not provide any support for weird integers. In the few cases they do, it is often in a very limited and specific way, like bitmap fields in C structs. Such constructions are often vague, obscure, and often their semantics depend on the specific implementation of the language, and/or the characteristics of the system where you run your program.

In poke, on the contrary, weird numbers are first class citizens, and they don’t differ in any way from “normal” integers which are composed of a whole number of bytes. Their interpretation is also well defined, and they keep the same semantics regardless of the characteristics of the computer on which poke is running.

• Incomplete Bytes
• Quantum Bytenics
• Signed Weird Numbers

  byte 0  |  byte 1
+---------+----+----+
|::::::::::::::|    |
+---------+----+----+
|   uint<12>   |

           byte
+-------------------------+
| b7 b6 b5 b4 b3 b2 b1 b0 |
+-------------------------+

           byte 0         |           byte 1
+-------------------------+-------------+-------------+
| a7 a6 a5 a4 a3 a2 a1 a0   b7 b6 b5 b4 :             |
+-------------------------+-------------+-------------+
|                uint<12>               |

0b a7 a6 a5 a4 a3 a2 a1 a0 b7 b6 b5 b4

0b b7 b6 b5 b4 a7 a6 a5 a4 a3 a2 a1 a0

(poke) .set obase 2
(poke) byte @ 0#B
0b01111111UB
(poke) byte @ 1#B
0b01000101UB

        byte @ 0#B       |        byte @ 1#B
+-------------------------+-------------+-------------+
|  0  1  1  1  1  1  1  1    0  1  0  0 :  0  1  0  1
+-------------------------+-------------+-------------+
|                uint<12>               |

(poke) .set endian big
(poke) uint<12> @ 0#B
(uint<12>) 0b011111110100

(poke) uint<12> @ 0#B
(uint<12>) 0b010001111111

    byte
+-----+----+
|:::::|    |
+-----+----+
uint<5>

             byte
+----------------+----------+
| b7 b6 b5 b4 b3 |          |
+----------------+----------+
|    uint<5>     |

0b b7 b6 b5 b4 b3

(poke) .set obase 2
(poke) .set endian big
(poke) byte @ 0#B
0b01111111UB
(poke) uint<5> @ 0#B
(uint<5>) 0b01111
(poke) .set endian little
(poke) uint<5> @ 0#B
(uint<5>) 0b01111

(poke) .set endian big
(poke) uint<12> @ 0#B
(uint<12>) 0b011111110100

(poke) int<12> @ 0#B
(int<12>) 0b010001111111
(poke) .set obase 10
(poke) int<12> @ 0#B
(int<12>) 1151

(poke) .set obase 2
(poke) byte @ 0#B = 0b1111_1111
(poke) int<12> @ 0#B
(int<12>) 0b111111110100
(poke) .set obase 10
(poke) int<12> @ 0#B
(int<12>) -12

(poke) .set endian little
(poke) .set obase 2
(poke) int<12> @ 0#B
(int<12>) 0b010011111111
(poke) .set obase 10
(poke) int<12>  0#B
(int<12>) 1279

3.10 Unaligned Integers ¶

We have mentioned above that the data stored in computers, that we edit with poke, is arranged as a sequence of bytes. The entities we edit with poke (that we call IO devices) are presented to us as IO spaces. Up to now, we have accessed this IO space in terms of bytes, in commands like dump :from 32#B and in expressions like 2UB + byte @ 0#B. We said that mapped integers are built from bytes read from the IO space.

However, the IO space that poke offers to us is actually a space of bits, not a space of bytes, and the poke values are mapped on this space of bits. The following figure shows this:

poke values      |        uint<16> @ 2#b        |
-----------      |                               |
IO space     |b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|b|
-----------  |               |               |               |
IO device    |     byte0     |     byte1     |     byte2     |

The main consequence of this, that you can see in the figure above, is that we can use offsets in mapping operations that are not aligned to bytes. You can specify an offset in bits, instead of bytes, using the #b suffix instead of #B. Little b means bits, and big B means bytes.

Let’s map an unaligned 16 bit unsigned integer in foo.o:

(poke) dump :from 0#B :size 3#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c                                  .EL
(poke) .set obase 2
(poke) byte @ 0#B
0b01111111UB
(poke) byte @ 1#B
0b01000101UB
(poke) byte @ 2#B
0b01001100UB
(poke) .set endian big
(poke) uint<16> @ 2#b
0b1111110100010101UH

Graphically:

poke values      |        uint<16> @ 2#b        |
-----------      |                               |
IO space     |0|1|1|1|1|1|1|1|0|1|0|0|0|1|0|1|0|1|0|0|1|1|0|0|
-----------  |               |               |               |
IO device    |     0x7f      |      0x45     |      0x4c     |

These three levels of abstractions make it very easy and natural to work with unaligned data. Imagine for example that you are poking packages in a network protocol that is bit-oriented. This means that the packages will generally not be aligned to byte boundaries, but still the payload stored in the packages contains integers of several sizes. Other conventional binary editors or programming languages, that are almost always byte oriented, would require us to “unpack” the network data to a different, byte oriented, representation before messing with it. poke, on the contrary, allows you to directly map these integers as if they were aligned to byte boundaries, and work with them.

However, when one tries to determine the correspondence between a given poke value and the underlying bytes in the IO device, things can get complicated. This is particularly true when we map what we called “weird numbers”, i.e. numbers with partial bytes. As we saw, the rules to build these numbers were expressed in terms of bytes.

In order to ease the visualization of the process used to build integer values (especially if they are weird numbers, i.e. integers with partial bytes) one can imagine an additional layer of “virtual bytes” above the space of bits provided by the IO space. Graphically:

poke values       |        uint<16> @ 2#b        |
-----------       |                               |
Virtual bytes     | virt. byte1   |  virt. byte2  |
-----------       |               |               |
IO space      |0|1|1|1|1|1|1|1|0|1|0|0|0|1|0|1|0|1|0|0|1|1|0|0|
-----------   |               |               |               |
IO device     |     0x7f      |      0x45     |      0x4c     |

It is very important to understand that the IO space is an abstraction provided by poke. The underlying file, or memory buffer, or whatever, is actually a sequence of bytes; poke translates the operations on integers, bits, bytes, etc into the corresponding byte operations, and this translation is far from trivial. Fortunately, we can let poke do the dirty job for us, and abstract ourselves from that complexity.

3.11 Integers of Different Sizes ¶

When integer values of different sizes are passed to an arithmetic or relational operator, the “smaller” operand gets converted into the size of the “bigger” operand. For example:

(poke) 1H + 2
3

The operands are of size 16-bit and 32-bit respectively, and the result is a 32-bit integer. This is equivalent to:

(poke) 1H as int<32> + 2
3

3.12 Offsets and Sizes ¶

Early in the design of what is becoming GNU poke I was struck by a problem that, to my surprise, would prove not easy to fix in a satisfactory way: would I make a byte-oriented program, or a bit-oriented program? Considering that the program in question was nothing less than an editor for binary data, this was no petty dilemma.

Since the very beginning I had a pretty clear idea of what I wanted to achieve: a binary editor that would be capable of editing user defined data structures, besides mere bytes and bits. I also knew I needed some sort of domain specific language to describe these structures and operate on them. How that language would look like, and what kind of abstractions it would provide, however, was not clear to me. Not at all.

So once I sketched an initial language design, barely something very similar to C structs, I was determined to not continue with the poke implementation until I had described as many as binary formats in my language as possible. That, I reckoned, was the only way to make sure the implemented language would be expressive, complete and useful enough to fulfil my requirements.

The first formats I implemented using my immature little language included ELF, FLV, MP3, BSON… all of them describing structures based on whole bytes. Even when they try to be compact, it is always by packing bit-fields in elements that are, invariably, sized as a multiple of bytes. Consequently, the language I was evolving became byte oriented as well. No doubt also influenced by my C inheritance, I would think of bit-fields either as a sort of second class citizen, or as mere results of shifting and masking.

This worked well. The language evolved to be able to express many different aspects of these formats in a very nice way, like variable-length data and holes in structures.

Consider the following definition, which is not valid in today’s Poke:

type Data =
  struct
  {
    byte magic;
    byte count;
    byte dstart;

    byte[count] data @ dstart;
  };

The data starts with a byte that is a magic number. Then the size of the data stored, in bytes, and then the data itself. This data, however, doesn’t start right after dstart: it starts at dstart, which is expressed as an offset, in bytes, since the beginning of the Data. I conceived struct field labels to be any expression evaluating to an integer, which would be interpreted as a number of bytes, obviously.

Then, one day, it was the turn for IETF RFC1951, which is the specification of the DEFLATE algorithm and associated file format. Oh dear. Near the beginning of the spec document it can be read:

This document does not address the issue of the order in which bits of a byte are transmitted on a bit-sequential medium, since the final data format described here is byte- rather than bit-oriented. However, we describe the compressed block format in below, as a sequence of data elements of various bit lengths, not a sequence of bytes.

Then it goes on describing rules to pack the DEFLATE elements into bytes. I was appalled, and certainly sort of deflated as well. The purpose of my program was precisely to edit binary in terms of the data elements described by a format. And in this case, these data elements came in all sort of bit lengths and alignments. This can be seen in the following RFC1951 excerpt, that describes the header of a compressed block:

Each block of compressed data begins with 3 header bits containing the following data:

first bit       BFINAL
next 2 bits     BTYPE

Note that the header bits do not necessarily begin on a byte boundary, since a block does not necessarily occupy an integral number of bytes.

At this point I understood that my little language on the works would never be capable to describe the DEFLATE structures naturally: C-like bit-fields, masking and shifting, all based on byte-oriented containers and boundaries, would never provide the slickness I wanted for my editor. I mean, just use C and get done with it.

This pissed me off. Undoubtedly other formats and protocols would be impacted in a similar way. Even when most formats are byte oriented, what am I supposed to tell to the people hacking bit-oriented stuff? “Sorry pal, this is not supported, this program is not for you”? No way, I thought, not on my watch.

The obvious solution for the problem, is to be general. In this case, to express every offset and every memory size in bits instead of bytes. While this obviously gives the language maximum expressiveness power, and is ideal for expressing the few bit-oriented formats, it has the disadvantage of being very inconvenient for most situations.

To see how annoying this is, let’s revisit the little BitData structure we saw above. In a bit-oriented description language, we would need to write something like:

type BitData =
  struct
  {
    byte magic;
    byte count;
    byte dstart;

    byte[count] data @ dstart * 8;
  };

Yeah… exactly. The * and 8 keys in the keyboards of the poke users would wear out very fast, not to mention their patience as well. Also, should I provide both sizeof and bitsizeof operators? Nasty.

I am very fond of the maxim “Never write a program you would never use yourself”⁵, so I resigned myself to make GNU poke byte oriented, and to provide as many facilities for operating on bit-fields as possible.

Fortunately, I have smart friends…

During one of the Rabbit Herd’s Hacking Weekends⁶ I shared my frustration and struggle with the other rabbits, and we came to realize that offsets and data sizes in Poke should not be pure magnitudes or mere integer values: they should be united. They should have units.

It makes full sense when you come to think about it. For a program like poke, it is only natural to talk about different memory units, like bits, bytes, kilobytes, megabits, and so on. Bits and bytes are just two common units.

The syntax that we eventually used to denote united values is to specify the magnitude part, a hash ('#' character, and then the unit. For example, the size “three bytes” is expressed as 3#B, and “three bits” as 3#b.

Apart from allowing me to express values in different units, this approach also has other benefits as we will see shortly.

I’m really grateful to Bruno Haible, Luca Saiu and Nacho Gonzalez for putting me on the right track.

3.13 Buffers as IO Spaces ¶

We have mentioned already that files are not the only entities that can be edited using poke. Remember the dot-command .file that opened a file as an IO space?

(poke) .file foo.o
The current IOS is now `./foo.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 0	FILE	rw	0x00000000#B	0x000004c8#B	./foo.o

Memory buffers can be created using a similar dot-command, .mem:

(poke) .mem foo
The current IOS is now `*foo*'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 1	MEMORY		0x00000000#B	0x00001000#B    *foo*
  0	FILE	rw	0x00000000#B	0x000004c8#B    ./foo.o

Note how the name of the buffer is built by prepending and appending asterisks. Therefore, the name of the buffer created by the command .mem foo is *foo*. Note also that the new buffer is created with a default size of 0x1000 bytes, or 4096 bytes. The contents of the buffer are zeroed:

(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Memory buffer IO spaces grow automatically when a value is mapped beyond their current size. This is very useful when populating newly created buffers. However, for security reasons, there is a limit: the IO spaces are only allow to grow 4096 bytes at a time.

When it comes to map values, there is absolutely no difference between an IO space backed by a file and an IO space backed by a memory buffer. Exactly the same rules apply in both cases.

3.14 Copying Bytes ¶

Memory buffer IO spaces are cheap, and they are often used as “scratch” areas.

Suppose for example we want to experiment with the ELF header of foo.o. We could open it in poke:

(poke) .file foo.o
The current IOS is now `./foo.o'.

The header of an ELF file comprises the first 64 bytes in the file:

(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 3e00 0100 0000 0000 0000 0000 0000  ..>.............
00000020: 0000 0000 0000 0000 0802 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0b00 0a00  ..............

We know that as soon as we poke something on an IO space, the underlying file is immediately modified. So if we start playing with foo.o’s header, we may corrupt the file. We could of course make a copy of foo.o and work on the copy, but it is much better to create a memory IO space and copy the ELF header there:

(poke) .mem scratch
The current IOS is now `*scratch*'.
(poke) copy :from_ios 0 :from 0#B :size 64#B
(poke) dump :size 64#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0100 3e00 0100 0000 0000 0000 0000 0000  ..>.............
00000020: 0000 0000 0000 0000 0802 0000 0000 0000  ................
00000030: 0000 0000 4000 0000 0000 4000 0b00 0a00  ..............

The command copy above copies 64 bytes starting at byte 0 from the IO with id 0 (the file foo.o) to the current IO space (the buffer *scratch*).

Once we are done working with the copy of the ELF header, and satisfied, we can copy it back to the file and close the memory IO space:

(poke) copy :from 0#B :size 64#B :to_ios 0
(poke) .close
The current IOS is now `./foo.o'.

Note how the command arguments :from_ios and :to_ios are assumed to be the current IO space if they are not explicitly specified in the command invocation. For detailed information on the copy command, see copy.

3.15 Saving Buffers in Files ¶

Another useful command when working with buffer IO spaces (and in general, with any IO space) is save. Let’s say we want to save a copy of the header of an ELF file in another file. We could do it the pokeish way:

$ poke foo.o
[...]
(poke) save :from 0#B :size 64#B :file "header.dat"

The command above saves the first 64 bytes in the current IO space (which is backed by the file foo.o) into a new file header.dat in the current working directory.

Let’s now consider again the scenario where we are using a memory IO space as a scratch area. It is late in the night and we are tired, so we would like to save the contents of the scratch buffer to a file, so we can continue our work tomorrow. This is how we would do that:

(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 1	MEMORY		0x00000000#B	0x00001000#B    *scratch*
  0	FILE	rw	0x00000000#B	0x000f4241#B    ./foo.o
(poke) save :from 0#B :size iosize (1) :file "scratch.dat"

Here we used the built-in function iosize, that given an IO space identifier returns its size.

3.16 Character Sets ¶

Computers understand text as a sequence of codes, which are numbers identifying some particular character. A character can represent things like letters, digits, ideograms, word separators, religious symbols, etc. Collections of character codes are called character sets.

Some character sets try to cover one or a few similar written languages. This is the case of ASCII and ISO Latin-1, for example. These character sets are small, i.e. just a few hundred codes.

Other character sets are much more ambitious. This is the case of Unicode, that tries to cover the entire totality of human languages in the globe, including the fictitious ones, like klingon. Unicode is a really big character set.

In order to store character codes in a computer’s memory, or a file, we need to encode each character code in one or more bytes. The number of bytes needed to encode a given character code depends on the range of codes in the containing set.

ASCII, for example, defines 128 character codes: a single byte is enough to encode every possible ASCII character. It is very easy to encode ASCII.

Unicode, on the contrary, defines many thousand of character codes, and has room for many more: we would need 31 bits in order to encode any conceivable Unicode character code. However, it would be wasteful to use that many bits per character: most used character codes tend to be in lower regions of the code space. For example, the code corresponding to the Latin letter 'a' is a fairly small number, whereas the codes corresponding to the Klingon alphabet are really big numbers. Consequently, some systems opt to just encode a subset of Unicode, like the first 16 bits of the Unicode space, which is called the Basic Multilingual Plane, and contains all the characters that most people will ever need. There are also variable-length encodings of Unicode, that try to use as less bytes as possible to encode any given code. A particularly clever encoding of Unicode, designed by Rob Pike, is backwards compatible with the ASCII encoding, i.e. it encodes all the ASCII codes in one byte each, and the values of these byte are the same than in ASCII. This clever encoding is called UTF-8.

3.17 From Bytes to Characters ¶

• Character Literals
• Classifying Characters
• Non-printable Characters

(poke) 'a'
0x61UB

(poke) 'a' == 0x61UB
1
(poke) 'a' + 1
0x62U

(poke) var b = byte @ 23#B
(poke) b >= '0' && b <= '9'
1

(poke) b
0x37UB
(poke) b - '0'
7UB

(poke) (b >= 'a' && b <= 'z') || (b >= 'A' && b <= 'Z')
0

(poke) '\002'
0x2UB

(poke) '\n'
0xaUB
(poke) '\t'
0x9UB

3.18 ASCII Strings ¶

• String Values
• Poking Strings
• From Characters to Strings

(poke) "word"
"word"

(poke) .mem scratch
The current IOS is now `*scratch*'.
(poke) dump :size 48#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................

(poke) char @ 0x12#B = 'w'
(poke) char @ 0x13#B = 'o'
(poke) char @ 0x14#B = 'r'
(poke) char @ 0x15#B = 'd'
(poke) dump :size 48#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 776f 7264 0000 0000 0000 0000 0000  ..word..........
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................

(poke) string @ 0x12#B
"word"

(poke) "word"'length
4UL
(poke) "word"'size
40UL#b

(poke) string @ 0x22#B = "WORD"
(poke) dump :size 48#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 776f 7264 0000 0000 0000 0000 0000  ..word..........
00000020: 0000 574f 5244 0000 0000 0000 0000 0000  ..WORD..........

(poke) "foo" + "bar"
"foobar"

(poke) "foo"'size + "bar"'size
0x40UL#b
(poke) ("foo" + "bar")'size
0x38UL#b

(poke) "Putin" + 'a'
<stdin>:1:1: error: invalid operands in expression
"Putin" + 'a';
^~~~~~~~~~~~~

(poke) 'a' as string
"a"

(poke) "Putin" + 'a' as string
"Putina"

3.19 From Strings to Characters ¶

Despite being atomic values, poke strings can be indexed in order to retrieve individual characters:

(poke) "word"[2]
0x72UB

Note how the indexing is zero-based, i.e. the first position in the string is referred as [0], the second position with [1], and so on.

If you specify a negative index, or an index that is too big, you will get an error:

(poke) "word"[-1]
<stdin>:1:8: error: index is out of bounds of string
"word"[-1];
       ^
(poke) "word"[5]
<stdin>:1:8: error: index is out of bounds of string
"word"[5];
       ^

3.20 Strings are not Arrays ¶

Arrays are collections of homogeneous data organized in a sequence. We have already seen (briefly) arrays of bytes, which use the type specifier byte[].

Similarly, it is possible to arrange characters (which are basically little numbers) in an array, like in:

(poke) ['a','b','c']
[0x61UB,0x62UB,0x63UB]

However, the array above is not equivalent to the string "abc". The later is a simple value, whereas an array is a composite value, and also is implicitly terminated with a NULL character, i.e. a 0 byte.

The Poke standard library provides a couple of utility functions to convert between string values and character arrays: catos and stoca.

catos gets an array of characters and returns an equivalent string. For example:

(poke) catos (['a','b','c'])
"abc"

stoca gets a string and an array and sets the element of the array to the characters composing the string. For example:

(poke) stoca("abc")
[0x61UB,0x62UB,0x63UB]
(poke) var a = char[3]()
(poke) stoca ("abc", a)
(poke) a
[0x61UB,0x62UB,0x63UB]

4.1 The SBM Format ¶

• Images as Maps of Pixels
• SBM header
• SBM data

 <--- line_width  --->
| pixel | pixel | ... | pixel | pixel | ... | ...
|        line 1       |         line 2      |

                SBM header
+-------+-------+-------+-------+-------+
|  'S'  |  'B'  |  'M'  |  ppl  | lines |
+-------+-------+-------+-------+-------+
  byte0   byte1   byte2   byte3   byte4

        SBM pixel
+-------+-------+-------+
|  Red  | Green | Blue  |
+-------+-------+-------+
  byte0   byte1   byte2

4.2 Poking a SBM Image ¶

• P is for poke
• Preparing the Canvas
• Poking the Header
• Poking the Pixels
• Poking Lines
• Poking Images
• Saving the Image

  | 0 | 1 | 2 | 3 | 4 |
  +---+---+---+---+---+
0 |   | * | * |   |   |
1 |   | * |   | * |   |
2 |   | * |   | * |   |
3 |   | * | * |   |   |
4 |   | * |   |   |   |
5 |   | * |   |   |   |
6 |   | * |   |   |   |

$ poke
[...]
(poke) .mem image
The current IOS is now `*image*'.
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................

(poke) byte @ 0#B = 'S'
(poke) byte @ 1#B = 'B'
(poke) byte @ 2#B = 'M'

(poke) byte @ 3#B = 5
(poke) byte @ 4#B = 7

(poke) dump :size 16#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 0700 0000 0000 0000 0000 0000  SBM.............

(poke) byte @ offset = 255
(poke) byte @ offset+1#B = 255
(poke) byte @ offset+2#B = 255

(poke) byte @ offset = 255
(poke) byte @ offset+1#B = 99
(poke) byte @ offset+2#B = 71

(poke) var bg1 = 255
(poke) var bg2 = 255
(poke) var bg3 = 255
(poke) var fg1 = 255
(poke) var fg2 = 99
(poke) var fg3 = 71

(poke) byte @ offset = fg1
(poke) byte @ offset+1#B = fg2
(poke) byte @ offset+2#B = fg3

(poke) var bg = 255UB:::255UB:::255UB
(poke) var fg = 255UB:::99UB:::71UB
(poke) bg
(uint<24>) 0xffffff
(poke) fg
(uint<24>) 0xff6347

(poke) uint<24> @ 10#B = fg
(poke) dump :from 10#B :size 4#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
0000000a: 4763 ff00                                Gc..

(poke) var bga = [255UB, 255UB, 255UB]
(poke) var fga = [255UB, 99UB, 71UB]

(poke) [1,"foo"]
<stdin>:1:1: error: array initializers should be of the same type
[1,"foo"];
^~~~~~~~~

(poke) bga'length
3UL

(poke) byte[3] @ 10#B = fga
(poke) dump :from 10#B :size 4#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
0000000a: ff63 4700                                 .cG.

(poke) byte[3] @ 10#B
[255UB,99UB,71UB]

(poke) [[1,2],[3,4]]

(poke) int[2][2] @ 100#B
[[0,0],[0,0]]

(poke) var l0 = [bga,fga,fga,bga,bga]
(poke) l0
[[255UB,255UB,255UB],[255UB,99UB,71UB],...]

(poke) var l0 = [bga,fga,bga,fga,bga]
(poke) var l1 = [bga,fga,bga,fga,bga]
(poke) var l2 = [bga,fga,fga,bga,bga]
(poke) var l3 = [bga,fga,bga,bga,bga]
(poke) var l4 = l3
(poke) var l5 = l4

(poke) byte[3][5] @ 5#B = l0
(poke) byte[3][5] @ 10#B = l1
(poke) byte[3][5] @ 15#B = l2
(poke) byte[3][5] @ 20#B = l3
(poke) byte[3][5] @ 25#B = l4
(poke) byte[3][5] @ 30#B = l5
(poke) byte[3][5] @ 35#B = l6

(poke) var image_data = [l0,l1,l2,l3,l4,l5]
(poke) image_data
[[[255UB,255UB,255UB],[255UB,99UB,71UB],[255UB,99UB,71UB]...]...]

(poke) byte[3][5][6] @ 5#B = image_data

(poke) var ppl = byte @ 3#B
(poke) var lines = byte @ 4#B
(poke) save :from 0#B :size 5#B + ppl#B * lines :file "p.sbm"

(poke) .set obase 10
(poke) [1,2,3]'size
96UL#b

(poke) image_data'size
720UL#b

(poke) save :from 0#B :size 5#B + image_data'size :file "p.sbm"

4.3 Modifying SBM Images ¶

• Reading a SBM File
• Painting Pixels
• Cropping the R
• Shortening and Shifting Lines
• Updating the Header
• Saving the Result

$ poke p.sbm
[...]
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 07ff ffff ff63 47ff 6347 ffff  SBM......cG.cG..
00000010: ffff ffff ffff ffff 6347 ffff ffff 6347  ........cG....cG
00000020: ffff ffff ffff ff63 47ff ffff ff63 47ff  .......cG....cG.
00000030: ffff ffff ffff 6347 ff63 47ff ffff ffff  ......cG.cG.....
00000040: ffff ffff ff63 47ff ffff ffff ffff ffff  .....cG.........
00000050: ffff ffff 6347 ffff ffff ffff ffff ffff  ....cG..........
00000060: ffff ff63 47ff ffff ffff ffff ffff       ...cG.........

(poke) var ppl = byte @ 3#B
(poke) ppl
5UB
(poke) var lines = byte @ 4#B
(poke) lines
7UB

(poke) var image_data = byte[3][ppl][lines] @ 5#B
(poke) image_data
[[[255UB,255UB,255UB],[255UB,99UB,71UB], ...]...]

  | 0 | 1 | 2 | 3 | 4 |
  +---+---+---+---+---+
0 |   | * | * |   |   |
1 |   | * |   | * |   |
2 |   | * |   | * |   |
3 |   | * | * |   |   |
4 |   | * | * |   |   |
5 |   | * |   | * |   |
6 |   | * |   | * |   |

$ poke p.sbm
[...]
(poke) .mem scratch
The current IOS is now `*scratch*'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 1	MEMORY		0x00000000#B	0x00001000#B	*scratch*
  0	FILE	rw	0x00000000#B	0x0000006e#B	./p.sbm
(poke) copy :from_ios 0 :from 0#B :to 0#B :size iosize (0)
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 07ff ffff ff63 47ff 6347 ffff  SBM......cG.cG..
00000010: ffff ffff ffff ffff 6347 ffff ffff 6347  ........cG....cG
00000020: ffff ffff ffff ff63 47ff ffff ff63 47ff  .......cG....cG.
00000030: ffff ffff ffff 6347 ff63 47ff ffff ffff  ......cG.cG.....
00000040: ffff ffff ff63 47ff ffff ffff ffff ffff  .....cG.........
00000050: ffff ffff 6347 ffff ffff ffff ffff ffff  ....cG..........
00000060: ffff ff63 47ff ffff ffff ffff ffff 0000  ...cG...........
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................

(poke) var ppl = byte @ 3#B
(poke) var lines = byte @ 4#B
(poke) var image_data = byte[3][ppl][lines] @ 5#B

(poke) var n = int @ offset
(poke) n = n + 1
(poke) int @ offset = n

(poke) var a = [1,2,3]

   +---+---+---+
 a | 1 | 2 | 3 |
   +---+---+---+

(poke) var b = a

   +---+---+---+      +---+---+---+
 a | 1 | 2 | 3 |    b | 1 | 2 | 3 |
   +---+---+---+      +---+---+---+

(poke) a[1] = 5

   +---+---+---+      +---+---+---+
 a | 1 | 5 | 3 |    b | 1 | 2 | 3 |
   +---+---+---+      +---+---+---+

(poke) var a = [[1,2],[3,4],[5,6]]

   +---+---+---+
 a |   |   |   |
   +---+---+---+
     |   |   |   +---+---+
     |   |   +---| 5 | 6 |
     |   |       +---+---+
     |   |
     |   |       +---+---+
     |   +-------| 3 | 4 |
     |           +---+---+
     |
     |           +---+---+
     +-----------| 1 | 2 |
                 +---+---+

(poke) var b = a

   +---+---+---+           +---+---+---+
 a |   |   |   |         b |   |   |   |
   +---+---+---+           +---+---+---+
     |   |   |   +---+---+   |   |   |
     |   |   +---| 5 | 6 |---+   |   |
     |   |       +---+---+       |   |
     |   |                       |   |
     |   |       +---+---+       |   |
     |   +-------| 3 | 4 |-------+   |
     |           +---+---+           |
     |                               |
     |           +---+---+           |
     +-----------| 1 | 2 |-----------+
                 +---+---+

(poke) a[1][1] = 9
(poke) a
[[1,2],[3,9],[5,6]]
(poke) b
[[1,2],[3,9],[5,6]]

   +---+---+---+           +---+---+---+
 a |   |   |   |         b |   |   |   |
   +---+---+---+           +---+---+---+
     |   |   |   +---+---+   |   |   |
     |   |   +---| 5 | 6 |---+   |   |
     |   |       +---+---+       |   |
     |   |                       |   |
     |   |       +---+---+       |   |
     |   +-------| 3 | 9 |-------+   |
     |           +---+---+           |
     |                               |
     |           +---+---+           |
     +-----------| 1 | 2 |-----------+
                 +---+---+

(poke) var a = byte[3] @ offset
(poke) a[1] = 10

(poke) ppl'mapped
0
(poke) lines'mapped
0

(poke) image_data'mapped
1

(poke) image_data'ios
1
(poke) image_data'offset
40UL#b

(poke) var bga = [255UB, 255UB, 255UB]
(poke) var fga = [255UB, 99UB, 71UB]

(poke) image_data[4][2] = fga
(poke) image_data[5][3] = fga
(poke) image_data[6][3] = fga
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d05 07ff ffff ff63 47ff 6347 ffff  SBM......cG.cG..
00000010: ffff ffff ffff ffff 6347 ffff ffff 6347  ........cG....cG
00000020: ffff ffff ffff ff63 47ff ffff ff63 47ff  .......cG....cG.
00000030: ffff ffff ffff 6347 ff63 47ff ffff ffff  ......cG.cG.....
00000040: ffff ffff ff63 47ff 6347 ffff ffff ffff  .....cG.cG......
00000050: ffff ffff 6347 ffff ffff 6347 ffff ffff  ....cG....cG....
00000060: ffff ff63 47ff ffff ff63 47ff ffff 0000  ...cG....cG.....
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................

  | 0 | 1 | 2 |
  +---+---+---+
0 | * | * |   |
1 | * |   | * |
2 | * |   | * |
3 | * | * |   |
4 | * | * |   |
5 | * |   | * |
6 | * |   | * |

     +----+----+----+----+----+
line | p0 | p1 | p2 | p3 | p4 |
     +----+----+----+----+----+

     +----+----+----+
line | p1 | p2 | p3 |
     +----+----+----+

(poke) var l0 = image_data[0]
(poke) l0
[[255UB,255UB,255UB],[255UB,99UB,71UB],...]

(poke) var cl0 = [l0[1],l0[2],l0[3]]

(poke) l0[1:4]
[[255UB,99UB,71UB],[255UB,99UB,71UB]]

(poke) l0[1:3]
[[255UB,99UB,71UB]]

(poke) var l0 = image_data[0]
(poke) var l1 = image_data[1]
(poke) var l2 = image_data[2]
(poke) var l3 = image_data[3]
(poke) var l4 = image_data[4]
(poke) var l5 = image_data[5]

(poke) image_data[0] = l0[1:4]
(poke) image_data[0] = l1[1:4]
(poke) image_data[1] = l2[1:4]
(poke) image_data[2] = l3[1:4]
(poke) image_data[3] = l4[1:4]
(poke) image_data[4] = l5[1:4]

(poke) byte[] @ 0#B = ['S','B','M']
(poke) byte @ 3#B = 3
(poke) byte @ 4#B = 7

(poke) save :from 0#B :size 5#B + image_data'size :file "r.sbm"

4.4 Defining Types ¶

• Naming your Own Abstractions
• Abstracting the Structure of Entities

(poke) byte[3] @ 5#B

(poke) type RGB_Color_Beam = byte

(poke) RGB_Color_Beam[3] @ 5#B

(poke) var pixel = byte[3] @ 5#B

(poke) var pixel = RGB_Color_Beam[3] @ 5#B

(poke) var image_data = RGB_Color_Beam[4][ppl][lines] @ 5#B

(poke) type SBM_Pixel = RGB_Color_Beam[3]

(poke) var image_data = SBM_Pixel[ppl][lines] @ 5#B

4.5 Pickles ¶

• poke Commands versus Poke Constructions
• Poke Files
• Pickling Abstractions
• Exploring Pickles
• Startup

(poke) 10 + 2
12

(poke) var bga = [255UB, 255UB, 255UB]
(poke) var fga = [255UB, 99UB, 71UB]

var white = [255UB, 255UB, 255UB];
var tomato = [255UB, 99UB, 71UB];

var matrix = [[10, 20, 30],
              [40, 50, 60],
              [70, 80, 90]];

(poke) load colors

(poke) tomato
[0xffUB,0x63UB,0x47UB]

(poke) load my-colors
<stdin>:1:8: error: syntax error, unexpected '-', expecting ';'
load my-colors;
       ^

(poke) load "my-colors.pk"

(poke) .load my-colors.pk

var rgb24_white = [255UB, 255UB, 255UB];
var rgb24_tomato = [255UB, 99UB, 71UB];

type RGB_Color_Beam = byte;
type RGB24_Color = RGB_Color_Beam[3];

var rgb24_white = [255UB, 255UB, 255UB];
var rgb24_tomato = [255UB, 99UB, 71UB];

load rgb24;

[... SBM definitions ...]

(poke) load elf

(poke) load elf

(poke) .info variables elf
Name            Declared at
elf_stb_names   elf-common.pk:53
elf_stt_names   elf-common.pk:72
elf_stv_names   elf-common.pk:84

(poke) .info types Elf64
Name                 Declared at
Elf64_Ehdr           elf-64.pk:184
Elf64_Off            elf-64.pk:26
Elf64_SectionFlags   elf-64.pk:130
Elf64_Shdr           elf-64.pk:152
Elf64_RelInfo        elf-64.pk:36
Elf64_Chdr           elf-64.pk:85
Elf64_Rela           elf-64.pk:61
Elf64_Phdr           elf-64.pk:169
Elf64_Sxword         elf-64.pk:24
Elf64_Sym            elf-64.pk:71
Elf64_Rel            elf-64.pk:45
Elf64_File           elf-64.pk:207
Elf64_Addr           elf-64.pk:25
Elf64_Group          elf-64.pk:121
Elf64_Dyn            elf-64.pk:98
Elf64_Xword          elf-64.pk:23

(poke) .info type Elf64_File
Class:      struct
Name:       Elf64_File
Complete:   no
Fields:
  Name     Type
  ehdr     Elf64_Ehdr
  shdr     Elf64_Shdr[]
  phdr     Elf64_Phdr[]
Methods:
  Name                     Type
  get_section_name         (offset<Elf_Word,8>)string
  get_symbol_name          (Elf64_Shdr,,offset<Elf_Word,8>)string
  get_sections_by_name     (string)Elf64_Shdr[]
  get_sections_by_type     (Elf_Word)Elf64_Shdr[]
  section_name_p           (string)int
  get_string               (offset<Elf_Word,8>)string
  get_group_signature      (Elf64_Shdr)string
  get_group_signatures     ()string[]
  get_section_group        (string)Elf64_Shdr[]

(poke) var elf = Elf64_File @ 0#B
(poke) elf.get_sections_by_name (".text")
[Elf64_Shdr {
  sh_name=0x1bU#B,
  sh_type=0x1U,
  sh_flags=#<ALLOC,EXECINSTR>,
  sh_addr=0x0UL#B,
  sh_offset=0x40UL#B,
  sh_size=0xbUL#B,
  sh_link=0x0U,
  sh_info=0x0U,
  sh_addralign=0x1UL,
  sh_entsize=0x0UL#b
}]

# My poke configuration - jemarch
[...]
.load ~/.poke.d/mydefs.pk

4.6 Poking Structs ¶

• Heterogeneous Related Data
• Mapping Structs
• Modifying Mapped Structs

                SBM header
+-------+-------+-------+-------+-------+
|  'S'  |  'B'  |  'M'  |  ppl  | lines |
+-------+-------+-------+-------+-------+
  byte0   byte1   byte2   byte3   byte4

type SBM_Header = byte[5];

type SBM_Header =
 struct
 {
   byte[3] magic;
   uint<8> ppl;
   uint<8> lines;
 }

(poke) SBM_Header @ 0#B
SBM_Header {
  magic=[0x53UB,0x42UB,0x4dUB],
  ppl=0x5UB,
  lines=0x7UB
}

(poke) var header = SBM_Header @ 0#B
(poke) header.ppl * header.lines
35UB

(poke) header.ppl as uint * header.lines
35U

(poke) byte @ 3#B = 3
(poke) byte @ 4#B = 7

(poke) header.ppl = 3
(poke) header.lines = 7

(poke) dump :size 5#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 5342 4d03 07                             SBM..

4.7 How Structs are Built ¶

First we need to define some structure to use as an example. Let’s say we are interesting in poking Packets, as defined by the Packet Specification 1.2 published by the Packet Foundation (none less).

In a nutshell, each Packet starts with a byte whose value is always 0xab, followed by a byte that defines the size of the payload. A stream of bytes conforming the payload follows, themselves followed by another stream of the same number of bytes with “control” values.

We could translate this description into the following Poke struct type definition:

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;
    byte[size] payload;
    byte[size] control;
  };

There are some details described by the fictitious Packet Specification 1.2 that are not covered in this simple definition, but we will be attending to that later in this manual.

So, given the definition of a struct type like Packet, there are only two ways to build a struct value in Poke.

One is to map it from some IO space. This is achieved using the map operator:

(poke) Packet @ 12#B
Packet {
  magic == 0xab,
  size = 2,
  payload = [0x12UB,0x30UB],
  control = [0x1UB,0x1UB]
}

The expression above maps a Packet starting at offset 12 bytes, in the current IO space. See the Poke manual for more details on using the map operator.

The second way to build a struct value is to _construct_ one, specifying the value to some, all or none of its fields. It looks like this:

(poke) Packet {size = 2, payload = [1UB,2UB]}
Packet {
  magic == 0xab,
  size = 2,
  payload = [0x1UB,0x2UB],
  control = [0x0UB,0x0UB]
}

In either case, building a struct involves to determine the value of all the fields of the struct, one by one. The order in which the struct fields are built is determined by the order of appearance of the fields in the type description.

In our example, the value of magic is determined first, then size, payload and finally control. This is the reason why we can refer to the values of previous fields when defining fields, such as in the size of the payload array above, but not the other way around: by the time payload is mapped or constructed, the value of size, has already been mapped or constructed.

What happens behind the curtains is that when poke finds the definition of a struct type, like Packet, it compiles two functions from it: a mapper function, and a constructor function. The mapper function gets as arguments the IO space and the offset from which to map the struct value, whereas the constructor function gets the template specifying the initial values for some, or all of the fields; reasonable default values (like zeroes) are used for fields for which no initial values have been specified.

These functions, mapper and constructor, are invoked to create fresh values when a map operator @ or a struct constructor is used in a Poke program, or at the poke prompt.

4.8 Variables in Structs ¶

Fields are not the only entity that can appear in the definition of a struct type.

Suppose that after reading more carefully the Packet Specification 1.2 (that spans for several thousand of pages) we realize that the field size doesn’t really store the number of bytes of the payload and control arrays, like we thought initially. Or not exactly: the Packet Foundation says that if size has the special value 0xff, then the size is zero.

We could of course do something like this:

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;

    byte[size == 0xff ? 0 : size] payload;
    byte[size == 0xff ? 0 : size] control;
  };

However, we can avoid replicating code by using a variable instead:

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;
  };

Note how the variable can be used after it gets defined. In the underlying process of mapping or constructing the struct, the variable is incorporated into the lexical environment. Once defined, it can be used in constraint expressions, array sizes, and the like. We will see more about this later.

Incidentally, it is of course possible to use global variables as well. For example:

var packet_special = 0xff;
type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;

    var real_size = (size == packet_special ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;
  };

In this case, the global packet_special gets captured in the lexical environment of the struct type (in reality in the lexical environment of the implicitly created mapper and constructor functions) in a way that if you later modify packet_special the new value will be used when mapping/constructing new values of type Packet. Which is really cool, but let’s not get distracted from the main topic... :)

4.9 Functions in Structs ¶

Further reading of the Packet Specification 1.2 reveals that each Packet has an additional crc field. The content of this field is derived from both the payload bytes and the control bytes.

But this is no vulgar CRC we are talking about. On the contrary, it is a special function developed by the CRC Foundation in partnership with the Packet Foundation, called superCRC (patented, TM).

Fortunately, the CRC Foundation distributes a pickle supercrc.pk, that provides a calculate_crc function with the following spec:

fun calculate_crc = (byte[] data, byte[] control) int:

So let’s use the function like this in our type, after loading the supercrc pickle:

load supercrc;

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;

    int crc = calculate_crc (payload, control);
  };

However, there is a caveat: it happens that the calculation of the CRC may involve arithmetic and division, so the CRC Foundation warns us that the calculate_crc function may raise E_div_by_zero. However, the Packet 1.2 Specification tells us that in these situations, the crc field of the packet should contain zero. If we used the type above, any exception raised by calculate_crc would be propagated by the mapper/constructor:

(poke) Packet @ 12#B
unhandled division by zero exception

A solution is to use a function that takes care of the extra needed logic, wrapping calculate_crc:

load supercrc;

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;

    fun corrected_crc = int:
    {
      try return calculate_crc (payload, control);
      catch if E_div_by_zero { return 0; }
    }

    int crc == corrected_crc;
  };

Again, note how the function is accessible after its definition. Note as well how both fields and variables and other functions can be used in the function body. There is no difference to define variables and functions in struct types than to define them inside other functions or in the top-level environment: the same lexical rules apply.

4.10 Struct Methods ¶

At this point you may be thinking something on the line of “hey, since variables and functions are also members of the struct, I should be able to access them the same way than fields, right?”.

So you will want to do:

(poke) var p = Packet  12#B
(poke) p.real_size
(poke) p.corrected_crc

But sorry, this won’t work.

To understand why, think about the struct building process we sketched above. The mapper and constructor functions are derived/compiled from the struct type. You can imagine them to have prototypes like:

Packet_mapper (IOspace, offset) -> Packet value
Packet_constructor (template)   -> Packet value

You can also picture the fields, variables and functions in the struct type specification as being defined inside the bodies of Packet_mapper and Packet_constructor, as their contents get mapped/constructed. For example, let’s see what the mapper does:

Packet_mapper:

  . Map a byte, put it in a local `magic'.
  . Map a byte, put it in a local `size'.
  . Calculate the real size, put it in a local `real_size'.
  . Map an array of real_size bytes, put it in a local `payload'.
  . Map an array of real_size bytes, put it in a local `control'.
  . Compile a function, put it in a local `corrected_crc'.
  . map a byte, call the function in the local `corrected_crc',
    complain if the values are not the same, otherwise put the
    mapped byte in a local `crc'.
  . Build a struct value with the values from the locals `magic',
    `size', `payload', `control' and `crc', and return it.

The pseudo-code for the constructor would be almost identical. Just replace "map a byte" with “construct a byte”.

So you see, both the values for the mapped fields and the values for the variables and functions defined inside the struct type end as locals of the mapping process, but only the values of the fields are actually put in the struct value that is returned in the last step.

This is where methods come in the picture. A method looks very similar to a function, but it is not quite the same thing. Let me show you an example:

load supercrc;

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;

    var real_size = (size == 0xff ? 0 : size);

    byte[real_size] payload;
    byte[real_size] control;

    fun corrected_crc = int:
    {
      try return calculate_crc (payload, control);
      catch if E_div_by_zero { return 0; }
    }

    int crc = corrected_crc;

    method c_crc = int:
    {
      return corrected_crc;
    }
  };

We have added a method c_crc to our Packet struct type, that just returns the corrected superCRC (patented, TM) of a packet. This can be invoked using dot-notation, once a Packet value is mapped/constructed:

(poke) var p = Packet  12#B
(poke) p.c_crc
0xdeadbeef

Now, the important bit here is that the method returns the corrected crc of a Packet. That’s it, it actually operates on a Packet value. This Packet value gets implicitly passed as an argument whenever a method is invoked.

We can visualize this with the following “pseudo Poke”:

method c_crc = (Packet SELF) int:
{
   return SELF.corrected_crc;
}

Fortunately, poke takes care to recognize when you are referring to fields of this implicit struct value, and does The Right Thing(TM) for you. This includes calling other methods:

method foo = void: { ... }
method bar = void:
{
 [...]
 foo;
}

The corresponding “pseudo-poke” being:

method bar = (Packet SELF) void:
{
 [...]
 SELF.foo ();
}

It is also possible to define methods that modify the contents of struct fields, no problem:

var packet_special = 0xff;

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size;
    [...]

    method set_size = (byte s) void:
    {
      if (s == 0)
        size = packet_special;
      else
        size = s;
    }
  };

This is what is commonly known as a setter. Note, incidentally, how a method can also use regular variables. The Poke compiler knows when to generate a store in a normal variable such as packet_special, and when to generate a set to a SELF field.

Given the different nature of the variables, functions and methods, there are a couple of restrictions:

• Functions can’t set fields defined in the struct type.

  This will be rejected by the compiler:

  type Foo =
    struct
    {
       int field;
       fun wrong = void: { field = 10; }
    };
  

  Remember the construction/mapping process. When a function accesses a field of the struct type like in the example above, it is not doing one of these pseudo SELF.field = 10. Instead, it is simply updating the value of the local created in this step in Foo_mapper:

  Foo_mapper:
  
   . Map an int, put it in a local `field'.
   . [...]
  

  Setting that local would impact the mapping of the subsequent fields if they refer to field (for example, in their constraint expression) but it wouldn’t actually alter the value of the field field in the struct value that is created and returned from the mapper!

  This is very confusing, so we just disallow this with a compiler error “invalid assignment to struct field”, for your own sanity 8-)

• Methods can’t be used in field constraint expressions, nor in variables or functions defined in a struct type.

  How could they be? The field constraint expressions, the initialization expressions of variables, and the functions defined in struct types are all executed as part of the mapper/constructor and, at that time, there is no struct value yet to pass to the method.

  If you try to do this, the compiler will greet you with an “invalid reference to struct method” message.

Something to keep in mind about methods is that they can destroy the integrity of the data stored in a struct. Consider for example the following struct type:

type Packet =
  struct
  {
    byte magic == 0xab;
    byte size : size <= 4096;
    [...]

    method set_size = (byte s) void:
    {
      if (s == 0)
        size = packet_special;
      else
        size = s;
    }
  };

Observe how this new version of Packet has an additional constraint that specifies size should not exceed 4096. However, when the method set_size is executed the constraints are not checked again. This is useful at times, but can also lead to unintended data corruption.

A solution for this problem is to make methods aware of the restrictions. Like in this case:

type Packet =
  struct
  {
    var MAXSIZE = 4096;

    byte magic == 0xab;
    byte size : size <= MAXSIZE;
    [...]

    method set_size = (byte s) void:
    {
      if (s == 0)
        size = packet_special;
      else if (s > MAXSIZE)
        raise E_inval;
      else
        size = s;
    }
  };

Note how we use a variable MAXSIZE in order to avoid hard-coding 4096 twice in the struct definition.

Occasionally it is useful to access to additional properties of a containing struct or union in a method, other than its fields. For this purpose it is possible to refer explicitly to the struct using the SELF keyword. Note that the value referred has type any.

For example, this is how we could print the name of the current alternative in a pretty-printer for an union:

type Foo =
  union int<32>
    {
      int<32> One == 1;
      int<32> Two == 2;
      method _print = void:
      {
        printf ("%s (%v)", SELF'ename (0), SELF'elem (0));
      }
    };

4.11 Padding and Alignment ¶

It is often the case in binary formats that certain elements are separated by some data that is not really used for any meaningful purpose other than occupy that space. The reason for keeping that space varies from case to case; sometimes to reserve it for future use, sometimes to make sure that the following data is aligned to some particular alignment. This is known as padding. There are several ways to implement padding in GNU poke. This article shows these techniques and discusses their advantages and disadvantages.

• Esoteric and exoteric padding
• Reserved fields
• Payloads
• Aligning struct fields
• Padding array elements

type Elf64_Chdr =
  struct
  {
    Elf_Word ch_type;
    Elf_Word ch_reserved;
    offset<Elf64_Xword,B> ch_size;
    offset<Elf64_Xword,B> ch_addralign;
  };

type Elf64_Chdr =
  struct
  {
    Elf_Word ch_type;
    Elf_Word ch_reserved == 0;
    offset<Elf64_Xword,B> ch_size;
    offset<Elf64_Xword,B> ch_addralign;
  };

(poke) Elf64_Chdr { ch_reserved = 23 }
unhandled constraint violation exception

type Elf64_Chdr =
  struct
  {
    Elf_Word ch_type;
    Elf_Word;
    offset<Elf64_Xword,B> ch_size;
    offset<Elf64_Xword,B> ch_addralign;
  };

(poke) Elf64_Chdr {}
Elf64_Chdr {
  ch_type=0x0U,
  0x0U,
  ch_size=0x0UL#B,
  ch_addralign=0x0UL#B
}

type Packet =
  struct
  {
    offset<uint<32>,B> payload_size;
    byte[payload_size] payload;
    int flags;
  };

(poke) packet.payload
[23UB, ...]
(poke) packet.payload[0] = 0

(poke) var numbers = ULEB128[packet.payload'size] @ packet.payload'offset

(poke) Packet { payload_size = 23#B }
Packet {
  payload_size=0x17U#B,
  payload=[0x0UB,0x0UB,0x0UB,0x0UB,0x0UB,...],
  flags=0x0
}

type Packet =
  struct
  {
    offset<uint<32>,B> payload_size;
    int flags @ OFFSET + payload_size;
  };

(poke) .set omaps yes
(poke) Packet { payload_size = 500#Mb }
Packet {
  payload_size=62500000U#B @ 0UL#b,
  flags=0 @ 4000000032UL#b
} @ 0UL#b

(poke) 4000000032UL #b/#MB
500UL

type Packet =
  struct
  {
    offset<uint<32>,B> payload_size;:
    var payload_offset = OFFSET;
    int flags @ OFFSET + payload_size;

    method get_payload_offset = off64:
    {
      return payload_offset;
    }
  };

var numbers = ULEB128[packet.payload_size @ packet.get_payload_offset

(poke) Packet {}
Packet {
  payload_size=0x0U#B,
  flags=0x0
}

type Packet =
  struct
  {
    offset<uint<32>,B> payload_size;
    byte[payload_size] payload;
    byte[alignto (OFFSET, 4#B)] padding;
    int flags;
  };

fun alignto = (uoff64 offset, uoff64 to) uoff64:
{
  return (to - (offset % to)) % to;
}

type Packet =
  struct
  {
    offset<uint<32>,B> payload_size;:
    var payload_offset = OFFSET;
    int flags @ OFFSET + payload_size + alignto (payload_size, 4#B);

    method get_payload_offset = off64:
    {
      return payload_offset;
    }
  };

+-----------------+
|      inode      |
+-----------------+
:                 :
:      data       :
:                 :
+-----------------+
|      inode      |
+-----------------+
:                 :
:      data       :
:                 :
+-----------------+
|      ...        |

type Inode =
  struct
  {
    string filename;
    int perms;
    offset<uint<32>,B> data_size;
  };

(poke) Inode[]  0#B

type Inode =
  struct
  {
    string filename;
    int perms;
    offset<uint<32>,B> data_size;
    byte[data_size] data;
    byte[alignto (data_size, 128#B)] padding;
  };

type Inode =
  struct
  {
    string filename;
    int perms;
    offset<uint<32>,B> data_size;

    byte[0] @ OFFSET + data_size + alignto (data_size, 128#B);
  };

(poke) var inodes = Inode[] @ 0#B

4.12 Dealing with Alternatives ¶

• BSON
• Unions are Tagged

type BSON_Doc =
  struct
  {
    offset<int<32>,B> size;
    BSON_Elem[size - size'size - 1#B] elements;
    byte endmark : endmark == 0x0;
  };

type BSON_Elem =
  struct
  {
    byte tag;
    string name;

    union
    {
      int32 integer32 : tag == 0x10;
      int64 integer64 : tag == 0x12;
      BSON_String str : tag == 0x02;
    } value;
  };

{
    "name" : "Jose E. Marchesi",
    "age" : 40,
    "big" : 1076543210012345
}

(poke) .load bson.pk
(poke) var d = BSON_Doc  0#B
(poke) d.elements'length
0x3UL
(poke) d.elements[0]
BSON_Elem {
  tag=0x2UB,
  name="name",
  value=struct {
    str=BSON_String {
      size=0x11,
      value="Jose E. Marchesi",
      chars=[0x4aUB,0x6fUB,0x73UB,0x65UB...]
    }
  }
}
(poke) d.elements[1]
BSON_Elem {
  tag=0x10UB,
  name="age",
  value=struct {
    integer32=0x27
  }
}
(poke) d.elements[2]
BSON_Elem {
  tag=0x12UB,
  name="big",
  value=struct {
    integer64=0x3d31c3f9e3eb9L
  }
}

(poke) d.elements[1].value.integer64
unhandled invalid element exception

type BSON_Elem =
  union
  {
    struct
    {
      byte tag : tag == 0x10;
      string name;
      int32 value;
    } integer32;

    struct
    {
      byte tag : tag == 0x12;
      string name;
      int64 value;
    } integer64;

    struct
    {
      byte tag : tag == 0x12;
      string name;
      BSON_String value;
    } str;
  };

(poke) d.elements[0]
BSON_Elem {
  str=struct {
    tag=0x2UB,
    name="name",
    value=BSON_String {
      size=0x11,
      value="Jose E. Marchesi",
      chars=[0x4aUB,0x6fUB,0x73UB,0x65UB...]
    }
  }
}
(poke) d.elements[1]
BSON_Elem {
  integer32=struct {
    tag=0x10UB,
    name="age",
    value=0x27
  }
}
(poke) d.elements[2]
BSON_Elem {
  integer64=struct {
    tag=0x12UB,
    name="big",
    value=0x3d31c3f9e3eb9L
  }
}

4.13 Structured Integers ¶

When we structure data using Poke structs, arrays and the like, we often use the same structure than a C programmer would use. For example, to model ELF RELA structures, which are defined in C like:

typedef struct
{
  Elf64_Addr   r_offset;  /* Address */
  Elf64_Xword  r_info;    /* Relocation type and symbol index */
  Elf64_Sxword r_addend;  /* Addend */
} Elf64_Rela;

we could use something like this in Poke:

type Elf64_Rela =
  struct
  {
    Elf64_Addr r_offset;
    Elf64_Xword r_info;
    Elf64_Sxword r_addend;
  };

Here the Poke struct type is pretty equivalent to the C incarnation. In both cases the fields are always stored in the given order, regardless of endianness or any other consideration.

However, there are situations where stored integral values are to be interpreted as composite data. This is the case of the r_info field above, which is a 64-bit unsigned integer (Elf64_Xword) which is itself composed by several fields, depicted here:

 63                                          0
+----------------------+----------------------+
|       r_sym          |      r_type          |
+----------------------+----------------------+
MSB                                         LSB

In order to support this kind of composition of integers, C programmers usually resort to either bit masking (most often) or to the often obscure and undefined behaviour-prone C bit fields. In the case of ELF, the GNU implementations define a few macros to access these “sub-fields”:

#define ELF64_R_SYM(i)         ((i) >> 32)
#define ELF64_R_TYPE(i)        ((i) & 0xffffffff)
#define ELF64_R_INFO(sym,type) ((((Elf64_Xword) (sym)) << 32) + (type))

Where ELF64_R_SYM and ELF64_R_TYPE are used to extract the fields from an r_info, and ELF64_R_INFO is used to compose it. This is typical of C data structures.

We could of course mimic the C implementation in Poke:

fun Elf64_R_Sym = (Elf64_Xword i) uint<32>:
   { return i .>> 32; }
fun Elf64_R_Type = (Elf64_Xword i) uint<32>:
   { return i & 0xffff_ffff; }
fun Elf64_R_Info = (uint<32> sym, uint<32> type) Elf64_Xword:
   { return sym as Elf64_Xword <<. 32 + type; }

However, this approach has a huge disadvantage: since we are not able to encode the logic of these “sub-fields” in proper Poke fields, they become second class citizens, with all that implies: no constraints on their own, can’t be auto-completed, can’t be assigned individually, etc.

But we can use the so-called integral structs! These are structs that are defined exactly like your garden variety Poke structs, with a small addition:

type Elf64_RelInfo =
  struct uint<64>
  {
    uint<32> r_sym;
    uint<32> r_type;
  };

Note the uint<64> addition after struct. This can be any integer type (signed or unsigned). The fields of an integral struct should be integral themselves (this includes both integers and offsets) and the total size occupied by the fields should be the same size than the one declared in the struct’s integer type. This is checked and enforced by the compiler.

The Elf64 RELA in Poke can then be encoded like:

type Elf64_Rela =
  struct
  {
    Elf64_Addr r_offset;
    struct Elf64_Xword
    {
      uint<32> r_sym;
      uint<32> r_type;
    } r_info;
    Elf64_Sxword r_addend;
  };

When an integral struct is mapped from some IO space, the total number of bytes occupied by the struct is read as a single integer value, and then the values of the fields are extracted from it. A similar process is using when writing. That is what makes it different with respect a normal Poke struct.

It is possible to obtain the integral value corresponding to an integral struct using a cast to an integral type:

(poke) type Foo = struct int<32> { int<16> hi; uint<16> lo; };
(poke) Foo { hi = 1 } as int<32>
0x10000

An useful idiom, that doesn’t require to specify an explicit integral type, is this:

(poke) type Foo = struct int<32> { int<16> hi; uint<16> lo; };
(poke) var x = Foo @ 0#B;
(poke) +x
0xfe00aa

These casts allow to “integrate” struct values explicitly, but the compiler also implicitly promotes integral struct values to integers in all contexts where an integer is expected:

• Operator to an addition, subtraction, multiplication, division, ceil division, bit left shift, bit right shift, bit-wise not, bit-wise and, bit-wise or, bit-wise xor, logical and, logical or, logical not, logical implication, positive, negative, bit-concatenation, condition in conditional expression.
• Argument to a function whose formal expects an integer.
• Value returned from a function that returns an integer.
• Condition in loop statement.
• Constraint expression in a struct field.
• Expression in a conditional struct field.

Note that the above list doesn’t include relational operators, since these operators work with struct values.

4.14 Working with Incorrect Data ¶

We have seen how Poke type definitions very often include constraint expressions reflecting the integrity of the data stored in these types.

Consider for example the following abbreviated description of the header of an ELF file:

type Elf64_Ehdr =
  struct
  {
    struct
    {
      byte[4] ei_mag = [0x7fUB, 'E', 'L', 'F'];
      byte ei_class;
      [...]
    } e_ident;

    Elf_Half e_type;
    Elf_Half e_machine;
    Elf_Word e_version = EV_CURRENT;
    [...]
    Elf_Half e_shstrndx : e_shnum == 0 || e_shstrndx < e_shnum;
  };

There are three constraint expressions in the definition above:

• The constraint expression in the field ei_mag makes sure that a right magic number begins the header.
• The constraint expression in e_version checks that the ELF version is the current version.
• The constraint expression in e_shstrndx checks that the index stored in the field doesn’t overflow the section header table of the file.

Every time we construct or map an Elf64_Ehdr value the constraint expressions are checked. In case any of the constraints are not satisfied we get a “constraint violation exception”. This is useful to realize that we are facing invalid data (when mapping) or that we are trying to build an invalid ELF header (when constructing.)

However, the exception avoids the completion of the mapping or construction operation. Sometimes this is inconvenient:

• When we are given invalid data that we want to explore or fix. Imagine for example a programmer investigating a possible assembler or linker bug that results in an invalid ELF file.
• When we are discovering or inferring the structure of some data: we suspect at some point there may be something that is similar to an ELF header.
• When we want to create invalid data. Suppose for example a programmer who is building a test case for the linker, that requires a corrupted ELF file.

The way poke provides support for these situation is using the concept of map strictness. Suppose we map an ELF header in some IO space:

(poke) load elf
(poke) var ehdr = Elf64_Ehdr @ 0#B

The mapping operator @ performs a strict mapping. This means that constraint expressions are evaluated and the operation is interrupted if any constraint fails, as described above. Also, this means the integrity will be checked when the data is modified:

(poke) ehdr.e_version = 666
unhandled constraint violation exception

So the mapping operator generates “strict” values. We can check the strictness of a mapped value using the 'strict attribute:

(poke) ehdr'strict
1

If we know that we are facing corrupted data, or if we want to corrupt the ELF header, we perform a non-strict mapping using a variant of the mapping operator:

(poke) var ehdr_ns = Elf64_Ehdr @! 0#B
(poke) ehdr_ns'strict
0

When mapping is performend in non-strict mode, the constraint expressions are still executed, but their resulting value is discarded. The constraint expressions are executed regardless of strictness because it is fairly common for them to include side-effects that have a calculated impact in subsequent operations, such as for example setting the endianness or some other global state.

We can now corrupt the header in the example above:

(poke) ehdr_ns.e_version = 666

5.1 Editing using Variables ¶

Editing data with GNU poke mainly involves creating mapped values and storing them in Poke variables. For example, if we were interested in altering the fields of the header in an ELF file, we would map an Elf64_Ehdr struct at the beginning of the underlying IO space (the file), like in:

(poke) .file foo.o
(poke) load elf
(poke) var ehdr = Elf64_Ehdr @ 0#B

At this point the variable ehdr holds an Elf64_Ehdr structure, which is mapped. As such, altering any of the fields of the struct will update the corresponding bytes in foo.o. For example:

(poke) ehdr.e_entry = 0#B

A Poke value has three mapping related attributes: whether it is mapped, the offset at which it is mapped in an IO space, and in which IO space. This information is accessible for both the user and Poke programs using the following attributes:

(poke) ehdr'mapped
1
(poke) ehdr'offset
0UL#b
(poke) ehdr'ios
0

That’s it, ehdr is mapped at offset zero byte in the IO space 0, which corresponds to foo.o:

(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 0	FILE	rw	0x00000000#B	0x000004c8#B   ./foo.o

Now that we have the ELF header, we may use it to get access to the ELF section header table in the file, that we will reference using another variable shdr:

(poke) var shdr = Elf64_Shdr[ehdr.e_shnum] @ ehdr.e_shoff
(poke) shdr[1]
Elf64_Shdr {
  sh_name=0x1bU#B,
  sh_type=0x1U,
  sh_flags=#<ALLOC,EXECINSTR>,
  sh_addr=0x0UL#B,
  sh_offset=0x40UL#B,
  sh_size=0xbUL#B,
  sh_link=0x0U,
  sh_info=0x0U,
  sh_addralign=0x1UL,
  sh_entsize=0x0UL#b
}

Variables are convenient entities to manipulate in Poke. Let’s suppose that the file has a lot of sections and we want to do some transformation in every section. It is a time consuming operation, and we may forget which sections we have already processed and which not. We could create an empty array to hold the sections already processed:

(poke) var processed = Elf64_Shdr[] ()

And then, once we have processed some given section, add it to the array:

... edit shdr[23] ...
(poke) processed += [shdr[23]]

Note how the array =processed= is not mapped, but the sections contained in it are mapped: Poke uses copy by shared value. So, after we spend the day carefully poking our ELF file, we can ask poke, are we done with all the sections in the file?

(poke) shdr'length == processed'length
1

Yes, we are. This can be made as sophisticated as desired. We could easily write a function that saves the contents of processed in files, so we can continue hacking tomorrow, for example.

We can then concluding that using mapped variables to edit data structures stored in IO spaces works well in common and simple cases like the above: we make our ways mapping here and there, defining variables to hold data that interests us, and it is easy to remember that the variables ehdr and shdr are mapped, where are they mapped, and that they are mapped in the file foo.o.

However, GNU poke allows to edit more than one IO space simultaneously. Let’s say we now want to poke the sections of another ELF file: bar.o. We would start by opening the file:

(poke) .file bar.o
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 1	FILE	rw	0x00000000#B	0x000004c8#B	./bar.o
  0	FILE	rw	0x00000000#B	0x000004c8#B	./foo.o

Now that bar.o is the current IO space, we can map its header. But now, what variable to use? We would rather not redefine ehdr, because that is already holding the header of foo.o. We could adapt our naming schema on the fly:

(poke) var foo_ehdr = ehdr
(poke) var bar_ehdr = Elf64_Ehdr @ 0#B

But then we would need to do the same for the other variables too:

(poke) var foo_shdr = shdr
(poke) var bar_shdr = Elf64_Shdr[bar_ehdr.e_shnum] @ bar_ehdr.e_shoff

However, we can easily see how this can degenerate quickly: what about processed, for example? In general, as the number of IO spaces being edited increases it becomes more and more difficult to manage our mapped variables, which are associated to each IO space.

5.2 poke Maps ¶

As we have seen mapping variables is a very powerful, general and flexible mean to edit stored binary data in one or more IO spaces. However it is easy to lose track of where the variables are mapped and, ideally speaking, we would want to have a mean to refer to, say, the “ELF header”, and get the header as a mapped value regardless of what specific file we are editing. Sort of a “meta variable”. GNU poke provides a way to do this: maps.

A map can be conceived as a sort of “view” that can be applied to a given IO space. Maps have entries, which are values mapped at some given offset, under certain conditions. For example, we have seen an ELF file contains, among other things, a header at the beginning of the file and a table of section headers of certain size and located at certain location determined by the header. These would be two entries of a so-called ELF map.

poke maps are defined in map files. These files use the .map extension. A map file self.map (for sectioned/simple elf) defining the view of an ELF file as a header and a table of section header would look like this:

/* self.map - map file for a simplified view of an ELF file.  */

load elf;

%%

%entry
%name ehdr
%type Elf64_Ehdr
%offset 0#B

%entry
%name shdr
%type Elf64_Shdr[(Elf64_Ehdr @ 0#B).e_shnum]
%condition (Elf64_Ehdr @ 0#B).e_shnum > 0
%offset (Elf64_Ehdr @ 0#B).e_shoff

This map file defines a view of an ELF file as a header entry ehdr and an entry with a table of section headers shdr.

The first section of the file, which spans until the separator line containing %%, is arbitrary Poke code which as we shall see, gets evaluated before the map entries are processed. This is called the map "prologue". In this case, the prologue contains a comment explaining the purpose of the file, and a single statement load that loads the elf.pk pickle, since the entries below use definitions like Elf64_Ehdr that are defined by that pickle. The prologue is useful to define Poke functions and other entities that are then used in the definitions of the entries.

A separator line containing only %% separates the prologue from the next section, which is a list of entries definitions. Each entry definition starts with a line %entry, and has the following attributes:

%name

Like ehdr and shdr. These names should follow the same rules than Poke variables, but as we shall see later, map entries are not Poke variables. This attribute is mandatory.

%type

This can be any Poke expression denoting a type, like int, Elf64_Ehdr or Elf64_Shdr[(Elf64_Ehdr @ 0#B).e_shnum]. This attribute is mandatory.

%condition

If specified, will determine whether to include the entry in the map. In the example above, the map will have an entry shdr only if the ELF file has one or more sections. Any Poke expression evaluating to a boolean can be used as conditions. This attribute is optional: entries not having a condition will always be included in the map.

%offset

Offset in the IO space where the entry will be mapped. Any Poke expression evaluating to an offset can be used as entry offset. This attribute is mandatory.

5.3 Loading Maps ¶

So we have written our =self.map=, which denotes a view or structure of ELF files we are interested on, and that resides in the current working directory. How to use it?

The first step is to fire up poke and open some object file. Let’s start with foo.o:

(poke) .file foo.o

Now, we can load the map using the .map load dot-command:

(poke) .map load self
[self](poke)

The .map load self command makes poke to look in certain directories for a file called self.map, and to load it. The list of directories where poke looks for map files is encoded in the variable map_load_path as a string containing a maybe empty list of directories separated by : characters. Each directory is tried in turn. This variable is initialized with suitable defaults:

(poke) map_load_path
"/home/jemarch/.poke.d:.:/home/jemarch/.local/share/poke:[...]"

Once a map is loaded, observe how the prompt changed to contain a prefix [self]. This means that the map self is loaded for the current IO space. You can choose to not see this information in the prompt by setting the prompt-maps option either at the prompt or in your .pokerc:

poke) .set prompt-maps no

By default prompt-maps is yes. This prompt aid is intended to provide a cursory look of the "views" or maps loaded for the current IO space. If we load another IO space and switch to it, the prompt changes accordingly:

(poke) [self](poke) .mem foo
The current IOS is now `*foo*'.
(poke) .ios $<*foo*>
The current IOS is now `./foo.o'.
[self](poke)

At any time the .info maps dot-command can be used to obtain a full list of loaded maps, with more information about them:

(poke) .info maps
IOS   Name   Source
0    self   ./self.map

In this case, there is a map self loaded in the IO space 0, which corresponds to foo.o.

Once we make foo.o our current IO space, we can ask poke to show us the entries corresponding to this map using another dot-command:

    : (poke) .map show self
    : Offset     Entry
    : 0x0UL#B    $self::ehdr
    : 0x208UL#B  $self::shdr

This tells us there are two entries for self in foo.o: $self::ehdr and $self::shdr= Note how map entries use names that start with the $ character, then contain the name of the map an the name of the entry we defined in the map file, separated by ::.

We can now use these entries at the prompt like if they were regular mapped variables:

[self](poke) $self::ehdr
Elf64_Ehdr {
  e_ident=struct {
    ei_mag=[0x7fUB,0x45UB,0x4cUB,0x46UB],
    [...]
  },
 e_type=0x1UH,
 e_machine=0x3eUH,
 [...]
}
(poke) $self::shdr'length
11UL

It is important to note, however, that map entries like $foo::bar are not part of the Poke language, and are only available when using poke interactively. Poke programs and scripts can’t use them.

Let’s now open another ELF file, and the =self= map in it:

(poke) .file /usr/local/lib/libpoke.so.0.0.0
(poke) .map load self
[self](poke)

So now we have two ELF files loaded in poke: foo.o and libpoke.so.0.0.0, and in both IO spaces we have the self map loaded. We can easily see that the map entries are different depending on the current IO space:

[self](poke) .map show self
Offset       Entry
0UL#B        $self::ehdr
3158952UL#B  $self::shdr
[self](poke) .ios 0
The current IOS is now `./foo.o'.
[self](poke) .map show self
Offset   Entry
0UL#B    $self::ehdr
520UL#B  $self::shdr

foo.o is an object file, whereas libpoke.so.0.0.0 is a DSO:

(poke) .ios 0
The current IOS is now `./foo.o'.
[self](poke) $self::ehdr.e_type
1UH
[self](poke) .ios 2
The current IOS is now `/usr/local/lib/libpoke.so.0.0.0'.
[self](poke) $self::ehdr.e_type
3UH

The interpretation of the map entry $self::ehdr is different depending on the current IO space. This makes it possible to refer to the “ELF header” of the current file.

Underneath, poke implements this by defining mapped variables and “redirecting” the entry names $foo::bar to the right variable depending on the IO space that is currently selected. It hides all that complexity from us.

5.4 Multiple Maps ¶

It is perfectly possible (and useful!) to load more than one map in the same IO space. It is very natural for a single file, for example, to contain data that can be interpreted in several ways, or of different nature.

Let’s for example open again an ELF file, this time compiled with -g:

(poke) .file foo.o

We now load our self map, to get a view of the file as a collection of sections:

(poke) .map load self
[self](poke)

And now we load the dwarf map that comes with poke, to get a view of the file as having debugging information encoded in DWARF:

[self(poke) .map load dwarf
[dwarf,self](poke)

See how the prompt now reflects the fact that the current IO space contains DWARF info! Let’s take a look:

[dwarf,self](poke) .info maps
IOS   Name    Source
0     dwarf   /home/jemarch/gnu/hacks/poke/maps/dwarf.map
0     self    ./self.map
[dwarf,self](poke) .map show dwarf
Offset    Entry
0x5bUL#B  $dwarf::info

Now we can access entries from any of the loaded maps, i.e. access the file in terms of different perspectives. As an ELF file:

[dwarf,self](poke) $self::shdr[1]
Elf64_Shdr {
  sh_name=0xb5U#B,
  sh_type=0x11U,
  sh_flags=#<>,
  sh_addr=0x0UL#B,
  sh_offset=0x40UL#B,
  sh_size=0x8UL#B,
  sh_link=0x18U,
  sh_info=0xfU,
  sh_addralign=0x4UL,
  sh_entsize=0x4UL#b
}

And as a file containing DWARF info:

[dwarf,self](poke) $dwarf::info
Dwarf_CU_Header {
  unit_length=#<0x0000004eU#B>,
  version=0x4UH,
  debug_abbrev_offset=#<0x00000000U#B>,
  address_size=0x8UB#B
}

If you are curious about how the DWARF entries are defined, look at maps/dwarf.map in the poke source distribution, or in your installed poke (.info maps will tell you the file the map got loaded from.)

It is possible to unload or remove a map from a given IO space using the .map remove dot-command. Say we are done looking at the DWARF in foo.o, and we are no longer interested in it as a file containing debugging info. We can do:

[dwarf,self](poke) .map remove dwarf
[self](poke)

Note how the prompt was updated accordingly: only self remains as a loaded map on this file.

5.5 Auto-map ¶

Certain maps make sense when editing certain types of data. For example, dwarf.map is intended to be used in ELF files. In order to ease using maps, poke provides a feature called auto mapping, which is disabled by default.

You can set auto mapping like this:

(poke) .set auto-map yes

When auto mapping is enabled, poke will look to the value of the pre-defined variable auto_map, which must contain an array of pairs of strings, associating a regular expression with a map name.

For example, you may want to initialize auto_map like this in your .pokerc file:

auto_map = [[".*\\.mp3$", "mp3"],
            [".*\\.o$", "elf"],
            ["a\\.out$", "elf"]];

This will make poke to load =mp3.map= for every file whose name ends with .mp3, and elf.map for files having names like foo.o and a.out.

Following the usual pokeish philosophy of being as less as intrusive by default as possible, the default value of auto_map is the empty string.

5.6 Constructing Maps ¶

As we have seen, we can define our own maps using map files like self.map, which contain a prologue and a set of map entries. However, sometimes it is useful to create maps “on the fly” while we explore some data with poke.

To make this possible, poke provides a suitable set of dot-commands. Let’s say we are poking some data, and we want to create a map for it. We can do that like this:

(poke) .map create mymap

This creates an empty map named mymap, with no entries:

[mymap](poke) .map show mymap
Offset   Entry

Adding entries is easy. First, we have to map some variable, and then use it as the base for the new entry:

[mymap](poke) var foo = int[3]  0#B
[mymap](poke) .map entry add mymap, foo
[mymap](poke) .map show mymap
Offset   Entry
0x0UL#B  $mymap::foo

Note how the entry $mymap::foo gets created, associated to the current IO space and mapped at the same offset than the variable foo.

We can remove entries from existing maps using the .map entry remove dot-command:

[mymap](poke) .map entry remove mymap, foo
[mymap](poke) .map show mymap
Offset   Entry
[mymap](poke)

We plan to add an additional command to save maps to map files. The idea is that you can create your maps on the fly, save them, and then load them back some other day when you are ready to continue poking. This is not implemented yet though.

5.7 Predefined Maps ¶

GNU poke comes with a set of useful pre-written maps, which get installed in a system location. We want to expand this collection, so please send us your map files!

6.1 Pretty-printers ¶

• Convention for pretty-printed Output
• Pretty Printing Optional Fields

ID3V1_Tag {
  id=[0x54UB,0x41UB,0x47UB],
  title=[0x30UB,0x31UB,0x20UB,0x2dUB,0x20UB,...],
  artist=[0x4aUB,0x6fUB,0x61UB,0x71UB,0x75UB,...],
  album=[0x4dUB,0x65UB,0x6eUB,0x74UB,0x69UB,...],
  year=[0x20UB,0x20UB,0x20UB,0x20UB],
  data=struct {
    extended=struct {
      comment=[0x20UB,0x20UB,0x20UB,0x20UB,0x20UB,...],
      zero=0x0UB,
      track=0x1UB
    }
  },
  genre=0xffUB
}

(poke) .set pretty-print yes

BPF_Insn = {
  ...
  regs=BPF_Regs {
     src=#<%r3>,
     dst=#<%r0>
  }
  ...
}

#<
  genre: 255
  title: 01 - Eclipse De Mar
  artist: Joaquin Sabina
  album: Mentiras Piadosas
  year:
  comment:
  track: 1
>

type Packet =
  struct
  {
    byte magic : magic in [MAGIC1,MAGIC2];
    byte n;
    byte[n] payload;
    if (magic == MAGIC2)
     PacketTrailer trailer;
  }

  struct
  {
    byte magic : magic in [MAGIC1,MAGIC2];
    [...]
    PacketTrailer trailer if packet_magic2_p (magic);

    method _print = void:
    {
      [...]
      if (magic == MAGIC2)
        pretty_print_trailer;
    }
  }

fun packet_magic2_p = int: { return magic == MAGIC2; }
type Packet =
  struct
  {
    byte magic : magic in [MAGIC1,MAGIC2];
    [...]
    if (packet_magic2_p (magic))
    PacketTrailer trailer;

    method _print = void:
    {
      [...]
      if (packet_magic2_p (magic))
        pretty_print_trailer;
    }
  }

type Packet =
  struct
  {
    byte magic : magic in [MAGIC1,MAGIC2];
    [...]
    if (packet_magic2_p (magic))
      PacketTrailer trailer;

    method _print = void:
    {
      [...]
      try pretty_print_trailer;
      catch if E_elem {}
    }
  }

type ID3V1_Tag =
  struct
  {
    [...]
    union
    {
      /* ID3v1.1  */
      struct
      {
        char[28] comment;
        byte zero = 0;
        byte track : track != 0;
      } extended;
      /* ID3v1  */
      char[30] comment;
    } data;
    [...]

    method _print = void:
    {
      [...]
      try print "  comment: " + catos (data.comment) + "\n";
      catch if E_elem
      {
        print "  comment: " + catos (data.extended.comment) + "\n";
        printf "  track: %u8d", data.extended.track;
      }
    }
  }

6.2 Setters and Getters ¶

Given a struct value, the obvious way to access the value of a field is to just refer to it using dot-notation.

For example, for the following struct type:

type ID3V1_Tag =
  struct
  {
    [...]
    char[30] title;
    char[30] artist;
    char[30] album;
    char[4] year;
    [...]
  }

Suppose the find out the year in the tag is wrong, off by two years: the song was release in 1980, not in 1978!. Unfortunately, due to the bizarre way the year is stored in the file (as a sequence of digits encoded in ASCII, non-NULL terminated) we cannot just write:

(poke) tag.year = tag.year + 2
error

Instead, we can use facilities from the standard library and a bit of programming:

(poke) stoca (format ("%d", atoi (catos (tag.year)) + 2), tag.year)

The above line basically transforms the data we want to operate on (the tag year) from the stored representation into a more useful representation (from an array of character digits to an integer) then operates with it (adds two) then converts back to the stored representation. Let’s call this “more useful” representation the “preferred representation”.

A well written pickle should provide getter and setter methods for fields in struct types for which the stored representation is not the preferred representation. By convention, getter and setter methods have the following form:

method get_field preferred_type: { ... }
method set_field (preferred_type val) void: { ... }

Using the get_ and set_ prefixes consistently is very important, because the pokist using your pickle can easily find out the available methods for some given value using tab-completion in the REPL.

For example, let’s add setter and getter methods for the field year in the ID3V1 tag struct above:

type ID3V1_Tag =
  struct
  {
    [...]
    char[4] year;

    method get_year = int: { return atoi (catos (year)); }
    method set_year = (int val) void:
    {
      var str = format ("%d", val);
      stoca (str, year);
    }
    [...]
  }

What constitutes the preferred representation of a field is up to the criteria of the pickle writer. For the tag above, I would say the preferred representations for the title, artist, album and year are string for title, artist and album, and an integer for the year.

7.1 Poke Scripts ¶

In interactive usage, there are two main ways to execute Poke code: at the interactive prompt (or REPL), and loading “pickles”.

Executing Poke code at the REPL is as easy as introducing a statement or expression:

(poke) print "Hello\n"
Hello

Executing Poke code in a pickle is performed by loading the file containing the code:

(poke) .load say-hello.pk
Hello

Where say-hello.pk contains simply:

print "Hello\n";

However, we would like to have Poke scripts, i.e. to be able to execute Poke programs as standalone programs, from the shell. In other words, we want to use GNU poke as an interpreter. This is achieved by using a shebang, which should appear at the top of the script file. The poke shebang looks like this:

#!/usr/bin/poke -L
!#

The -L command line option tells poke that it is being used as an interpreter. Additional arguments for poke can be specified before -L (but not after). The #! ... !# is an alternative syntax for multi-line comments, which allows to have the shebang at the top of a Poke program without causing a syntax error. This nice trick has been borrowed from guile.

Therefore, we could write say-hello as a Poke script like this:

#!/usr/bin/poke -L
!#

print "Hello\n";

And then execute it like any other program or script:

$ ./say-hello

Sometimes it is necessary or useful to know whether our Poke code is being run interactively or not. For these cases, we can check the poke_interactive_p pre-defined variable:

if (!poke_interactive_p)
  {
    /* Set default endianness.  */
  }

7.2 Command-Line Arguments ¶

When a Poke script is executed, the command line arguments passed to the script become available in the array argv. Example:

#!/usr/bin/poke -L
!#

for (arg in argv)
  print "Argument: " + arg + "\n";

Executing this script results in:

$ ./printargs foo bar 'baz quux'
Argument: foo
Argument: bar
Argument: baz quux

Note how it is not needed to have an argc variable, since the number of elements stored in a Poke array can be queried using an attribute: argv'length.

Note also that argv is only defined when poke runs as an interpreter:

$ poke
[...]
(poke) argv
<stdin>:1:1: error: undefined variable 'argv'
argv;
^~~~

Accessing the argv is more than enough for many simple programs. However, we may need a more sophisticated handling of command-line options: support for both short and long style options, adherence to the GNU coding standards, and the like. For these cases poke provides a pickle argp. See argp.

7.3 Exiting from Scripts ¶

By default a Poke script will communicate a successful status to the environment, upon exiting:

$ cat hello
#!/usr/bin/poke -L
!#

print "hello\n";
$ ./hello && echo $?
0

In order to exit with some other status code, most typically to signal an erroneous situation, the Pokeish way is to raise an E_exit exception with the desired exit status code:

raise Exception { code = EC_exit, exit_status = 1 };

This can be a bit cumbersome to write, so poke provides a more conventional syntax in the form of an exit function:

fun exit = (int<32> exit_code = 0) void:
{
  raise Exception { code = EC_exit, exit_status = exit_code };
}

Using exit, the above raise statement becomes the much simpler:

exit (1);

7.4 Loading pickles as Modules ¶

Suppose we want to write a Poke program to extract the contents of sections in a given ELF object file. Extracting sections requires dealing with several data structures encoded in the ELF file, such as the header, the section header table, the string table (that contains the names of the sections) and so on. It would be of course possible to define Poke types for these structures in the script itself but, as it happens, GNU poke ships with an already written pickle that describes the ELF structures. It is called elf.pk.

So a script needing to mess with ELF data structures can just make use of elf.pk using the load construction:

load elf;

This looks for a file called elf.pk in a set of directories, which are predefined by poke, and loads it. The list of directories where poke looks for pickles is stored in the load_path variable as a colon separated list of directory names, and can be customized:

$ poke
[...]
(poke) load_path
"/home/jemarch/.poke.d:.:/home/jemarch/.local/share/poke:..."

The default value of load_path contains both user-specific directories and system-wide directories. This assures that all the pickles installed by poke are available, and that the user can load her own pickles in her scripts.

Once a pickle is loaded in a script the types, functions and variables defined in it (either directly or indirectly by loading its own pickles) become available.

7.5 elfextractor ¶

As an example, in this section we will be hacking a very simple utility called elfextractor, that extracts the contents of the sections of an ELF file, whose name is provided as an argument in the command line, into several output files. This is the synopsis of the program:

elfextractor file [section_name]

Where file is the name of the ELF file from which to extract sections, and an optional section_name specifies the name of the section to extract.

Say we have a file foo.o and we would like to extract its text section. We would use elfextractor like:

$ elfextractor foo.o .text

Provided foo.o indeed has a section named .text, the utility will create a file foo.o.text with the section’s contents. Note how the names of the output files are derived concatenating the name of the input ELF file and the name of the extracted section.

If no section name is specified, then all sections are extracted. For example:

$ elfextractor foo.o
$ ls foo.o*
foo.o          foo.o.eh_frame       foo.o.shstrtab  foo.o.symtab
foo.o.comment  foo.o.rela.eh_frame  foo.o.strtab    foo.o.text

This is a possible implementation of elfextractor:

#!/usr/bin/poke -L
!#

/* elfextractor - Extract sections from ELF64 files. */

load elf;

if (!(argv'length in [1UL,2UL]))
  {
    print "Usage: elfextractor FILE [SECTION_NAME]\n";
    exit (1);
  }

var file_name = argv[0];
var section_name = (argv'length > 1) ? argv[1] : "";

try
  {
    var fd = open (file_name, IOS_M_RDONLY);
    var elf = Elf64_File @ fd : 0#B;

    for (shdr in elf.shdr where shdr.sh_type != 0x0)
      {
        var sname = elf.get_section_name (shdr.sh_name);

        if (section_name == "" || sname == section_name)
          save :ios elf'ios :file file_name + sname
               :from shdr.sh_offset :size shdr.sh_size;
      }

    close (fd);
  }
catch (Exception e)
  {
    if (e == E_constraint)
      printf ("error: `%s' is not a valid ELF64 file\n", file_name);
    else if (e == E_io)
      printf ("error: couldn't open file `%s'\n", file_name);
    else
      raise e;

    exit (1);
  }

First the command line arguments are handled. The script checks whether the right number of arguments have been passed (either 1 or 2) exiting with an error code otherwise. The file name and the section name are then extracted from the argv array.

Once we have the file name and the optional desired section name, it is time to do the real work. The code is enclosed in a try-catch block statement, because some of the operations may result on exceptions being raised.

First, the ELF file whose name is specified in the command line is opened for reading:

var fd = open (file_name, IOS_M_RDONLY);

The built-in function open returns a file descriptor that can be subsequently used in mapping operations. If the provided file name doesn’t identify a file, or if the file can’t be read for whatever reason, an E_io exception is raised. Note how the exception is handled in the catch block, emitting an appropriate diagnostic message and exiting with an error status.

Once the ELF file is open for reading, we map an Elf64_File on it, at the expected offset (zero bytes from the beginning of the file):

var elf = Elf64_File @ fd : 0#B;

If the file doesn’t contain valid ELF data, this map will fail and raise an E_constraint exception. Again, the catch block handles this situation.

At this point the variable elf contains an Elf64_File. Since we want to extract the sections contained in the file, we need to somehow iterate on them. The section header table is available in elf.shdr. A for-in-where loop is used to iterate on all the headers, skipping the “null” ELF sections which are always empty, and are characterized by a shdr.sh_type of 0. An inner conditional filters out sections whose name do not match the provided name in the command line, if it was specified at all.

For each matching section we then save its contents in a file named after the input ELF file, by calling a function save, which is provided by poke:

save :ios elf'ios :file file_name + sname
     :from shdr.sh_offset :size shdr.sh_size;

The above is exactly what we would have written at the poke REPL! (modulus trailing semicolon). How is this supposed to work? Thing is, GNU poke commands are implemented as Poke functions. Let’s consider save, for example. It is defined as a function having the following prototype:

fun save = (int ios = get_ios,
            string file = "",
            off64 from = 0#B,
            off64 size = 0#B,
            int append = 0,
            int verbose = 0) void:
{ ... }

Once a Poke function is defined in the environment, it becomes available as such. Therefore, in a poke session we could call it like:

(poke) save (get_ios, "filename", 0#B, 12#B, 0, 1)

However, this is cumbersome and error prone. To begin with, we should remember the name, position and nature of each argument accepted by the command. What is even more annoying, we are forced to provide explicit values for them, like in the example above we have to pass the current IOS (the default), and 0 for append (the default) just to being able to set verbose. Too bad.

To ease commanding poke, the Poke language supports an alternative syntax to call functions, in which the function arguments are referred by name, can be given in any order, and can be omitted. The command above can be thus written like:

(poke) save :from 0#B :size 12#B :verbose 1

This syntax is mostly intended to be used interactively, but nothing prevents to use it in Poke programs and scripts whenever it is deemed appropriate, like we did in elfextractor. We could of course have used the more conventional syntax:

if (section_name == "" || sname == section_name)
  save (elf'ios, file_name + sname,
        shdr.sh_offset, shdr.sh_size, 0, 0);

What style to use is certainly a matter of taste.

Anyhow, once the sections have been written out, the file descriptor is closed and the program exits with the default status, which is success. Should the save function find any problem saving the data, such as a full disk, not enough permissions or the like, exceptions will be raised, caught and maybe handled by our catch block.

And this is it! The complete program is 44 lines long. This is a good example that shows how, given a pickle providing a reasonable description of some binary-oriented format (ELF in this case) poke can be leveraged to achieve a lot in a very concise way, free from the many details involved in the encoding, reading and writing of binary data.

7.6 Filters ¶

The classic notion of filter in Unix-like systems and other Operating Systems is of a program that reads certain input using its standard input, realized some transformations on it, and writes the result on its standard output. The filter doesn’t need to operate on all the data at once, nor to buffer it: it reads input in chunks once it becomes available and is needed, and write output as necessary.

GNU poke provides the so called stream IO spaces in order to implement binary utilities that act like filters.

• Stream IO Spaces
• Reading from Streams
• Writing to Streams
• pk-strings

var stdin = open ("<stdin>");

var stdin = open ("<stdin>");

var b = byte @ stdin : 28#B;

var c = byte @stdin : 0#B;

var stdin = open ("<stdin>");

var b = byte @ stdin : 28#B;
var c = byte @ stdin : 29#B;
flush (stdin, 29#B);
var d = byte @ stdin : 30#B;
var e = byte @ stdin : 20#B; /* Exception.  */

var stdout = open ("<stdout>");

byte @ stdout : 28#B = 0xff;

byte @ stdout : 27#B = 0xab;

#!/usr/local/bin/poke -L
!#

/* Printable ASCII characters: 0x20..0x7e */

var stdin = open ("<stdin>");
var stdout = open ("<stdout>");

var offset = 0#B;

try
{
  flush (stdin, offset);

  var b = byte @ stdin : offset;
  if (b >= 0x20 && b <= 0x7e)
    byte  stdout : iosize (stdout) = b;

  offset = offset + 1#B;
}
until E_eof;

close (stdin);
close (stdout);

12.1 ios_dump_bytes ¶

The ios_dump_bytes function prints a byte dump of an area in a given IO space.

It has the following prototype:

fun ios_dump_bytes = (int<32> ios,
                      offset<uint<64>,b> from,
                      offset<uint<64>,b> size,
                      any val = -1,
                      offset<uint<64>, b> group_by = 2#B,
                      int<32> cluster_by = 8,
                      int<32> ruler = 0,
                      int<32> ascii = 0,
                      string unknown_byte = "??",
                      uint<8> nonprintable_char = '.') void

Where ios is the IO space form which dump bytes.

from is the offset from which start dumping bytes. This location is truncated to bytes.

size is an offset specifying the amount of bytes to dump. The offset is rounded up to the next byte.

val Instead of an explicit from+size range, a mapped value can be specified using the val argument. In this case, dump will show the bytes corresponding to the given value, with visual indications on what bytes correspond to what element contained in the value.

group_by determines how the bytes are grouped together in the output. Defaults to two bytes.

cluster_by specifies to display additioanl space after the specified number of groups have been displayed. Defaults to 8.

ruler is a boolean determining whether to include a ruler line in the output. Defaults to 0.

ascii_p is a boolean determining whether to include an ASCII dump. Defaults to 0.

unknown_byte is the string to use to denote bytes that can’t be read from the IO space. Defaults to "??". Note this string is expected to be of length two characters.

nonprintable_char is the character code to use to denote non-printable characters. Defaults to '.'.

This function may raise E_io, E_perm, and E_eof on several error conditions.

12.2 ios_copy_bytes ¶

The ios_copy_bytes copies a range of bytes between IO spaces.

It has the following prototype:

fun ios_copy_bytes = (int<32> from_ios, int<32> to_ios,
                      offset<uint<64>,B> from,
                      offset<uint<64>,B> to,
                      offset<uint<64>,B> size) void

Where from_ios is the origin IO space, i.e. the IO space from which bytes are to be copied.

to_ios is the destination IO space, i.e. the IO space to which bytes are to be copied.

from is a byte offset in from_ios with the beginning of the range to copy.

to is a byte offset in to_ios where the bytes will be copied.

size is a byte offset specifying the amount of data to be copied.

12.3 ios_save_bytes ¶

The function ios_save_bytes saves a range of bytes from an IO space to a file.

It has the following prototype:

fun ios_save_bytes = (int<32> ios, string file,
                      offset<uint<64>,B> from,
                      offset<uint<64>,B> size,
                      int<32> append_p) void

Where ios is the IO space from which save bytes.

file is the name of the file where to save the bytes.

to and size are byte offsets in the IO space determining the range of bytes to save.

By default the output file is truncated if it already exists. If append_p is true, then the data is instead appended at the end of the file.

13.1 base64 ¶

The base64 implements base64 encoding and decoding as described by RFC 4648¹⁰. It provides the following pair of functions:

fun base64_encode = (uint<8>[] src) string
fun base64_decode = (string src) uint<8>

14.1 POSIX Time ¶

In POSIX systems time is encoded as a certain number of seconds elapsed since some origin date, namely the first of January 1970. The time pickle provides two types for 32-bit and 64-bit POSIX time encodings:

POSIX_Time32
POSIX_Time64

A function ptime is also provided, that prints a human-readable representation of a POSIX date, given a number of seconds.

15.1 The Color Registry ¶

The color pickle provides a registry of standard colors, organized as a space of integers. Each integer identifies a standard color, which are accessible as variables named color_* after the pickle is loaded. Example:

(poke) load color
(poke) color_tomato
114

The purpose of having this register is to have a global namespace for colors that can be used in different pickles. The position of each color in the registry is totally unrelated to how the color may be encoded. Other pickles, as we shall see below, contain tables associating standard colors with their encoding, such as RGB.

If you want to add a new color that is not in the standard collection, you can use the color_register function, which gets no arguments:

(poke) var mycolor = color_register
(poke) mycolor
490

The total number of registered colors is recorded in the variable color_num_colors. You can use it to iterate on all the colors in the register, from 0 to color_num_colors - 1.

The index in the registry of the first user-defined color is hold in the variable color_LAST. For example, if you wanted to store a frob per standard color, you would do it like:

(poke) type Frobs = Frob[color_LAST]

The pickle also provides a function color_name that, given a color code, returns a printable name for the color, i.e. a string describing it. For user defined colors, this string is fixed:

(poke) color_name (23)
"lavender blush"
(poke) color_name (color_register)
"user-defined color"

If a non-existent color code is passed to color_name the function raises E_out_of_bounds:

(poke) color_name (color_num_colors)
unhandled out of bounds exception

15.2 RGB24 Encoding ¶

The RGB24 encoding encodes each color as a triplet of color beans, each beam indicating a level of red, green and blue respectively.

Types are provided for both the color beams, and the triplets:

type RGB24_Color_Beam = uint<8>;
type RGB24_Color = RGB24_Color_Beam[3];

The indexes RGB24_RED, RGB24_GREEN and RGB24_BLUE can be used to access to a specific beam of a given color:

(poke) rgb24_color[color_tomato][RGB24_GREEN]
99UB

The rgb24 pickle also provides a table associating poke standard colors (see The Color Registry) with their RGB24 encodings. Once the pickle is loaded, this table is available in the variable rgb24_color. The table is to be indexed by color codes:

(poke) load rgb24
(poke) rgb24_color[color_tomato]
[255UB,99UB,71UB]

16.1 MP3 ¶

• ID3V1 Tags

(poke) id3v1_genres[14]
"rhythm and blues"

fun id3v1_search_genre = (string name) uint<8>:

(poke) id3v1_search_genre ("rock")
17UB

['P','i','c','a','s','s','o',' ',' ', ..., ' ']

(poke) tag.title
[48UB,49UB,32UB,45UB,32UB,...]
(poke) tag.get_title
"01 - Eclipse De Mar"

(poke) tag.set_title ("Join us Now")
(poke) tag.title
[74UB,111UB,105UB,110UB,32UB,...]

(poke) tag.year
[0x31UB,0x39UB,0x38UB,0x30UB]
(poke) tag.get_year
1980
(poke) tag.set_year (1988)

17.1 ELF ¶

poke-elf (https://jemarch.net/poke-elf) is a full-fledged pickle for editing ELF object files, executables, shared libraries and core dumps. It supports many architectures and extensions.

17.2 Dwarf ¶

XXX

18.1 argp ¶

Like any other program, Poke scripts often need to handle options passed in the command line. The argp pickle provides a convenient and easy way to handle these arguments.

The main entry point of the pickle is the function argp_parse, with the following prototype:

fun argp_parse = (string program,
                  string version = "",
                  string summary = "",
                  Argp_Option[] opts = Argp_Option[](),
                  string[] argv = string[](),
                  int allow_unknown = 0) string[]

Where program is the name of the program providing the command-line options. version is either a string identifying the version of the program, or the empty string. summary is a short summary of what the program does. opts is an array of Argp_Option structs, each of which describe a command line option and, in particular, a handler for that option. argv is an array of string containing the command line elements to process. Finally, allow_unknown specifies whether unknown options constitute an error or not. This last option is useful in order to support sub-parsers.

Once invoked, argp_parse analyzes the command line provided in argv, recognizes options, performs checks making sure that all specified options are well formed, and that every option requiring an argument gets one, invokes the handlers registered for each option, and finally returns an array of non-option arguments in argv, for further processing by the user.

Each Argp_Option struct describes an option. They have this form:

type Argp_Option =
  struct
  {
    string name;
    string long_name;
    string summary;
    string arg_name;
    int arg_required;
    Argp_Option_Handler handler;
  };

Where name is a string of size one character specifying the short name for the option. long_name is a non-empty string of any size specifying the long name for the option. summary is a short paragraph with a summary of that the option does. arg_name, if specified, is a string to be used to describe the argument of the option in the --help output. If arg_name is not specified then "ARG" is used. arg_required is a boolean specifying whether the option accepts and expects an argument. Defaults to 0. Finally, handler is a function that will be invoked to process the option. The function has the following prototype:

type Argp_Option_Handler = (string)void

The argp_parse function provides default handlers for several standard options:

--help

The default handler for this option prints out an usage message in the standard GNU way, and exits. In particular, this output is compatible with the help2man utility.

--version

The default handler for this option prints out the name and the version of the program, and exits.

If the user provides handlers for the options above, these take precedence wrt the default handlers.

Several short options can be accumulated this way:

foo -qyz

The special option --, if found in the command line, stops the processing of options. Everything found about it is considered non-option arguments.

21.1 poke-mode ¶

poke-mode is a major mode for editing Poke source files, i.e. .pk files. It provides font-lock, auto-completion and indentation features.

poke-mode is no longer part of the poke source, it is available from GNU ELPA instead and can be installed via M-x package-install ret poke-mode. You can automatically load poke-mode by adding it to auto-mode-alist:

(add-to-list 'auto-mode-alist '("\\.pk\\'" . poke-mode))

21.2 poke-map-mode ¶

poke-map-mode is a major mode for editing poke map-files. It provides font-lock. See Maps and Map-files.

21.3 poke-ras-mode ¶

poke-ras-mode is a major mode for editing PVM assembly. It is used in the development of GNU poke. See the file libpoke/ras in the source distribution for more information about RAS, our Retarded Assembler.

22.1 poke.vim ¶

etc/vim/syntax/poke.vim is a syntax highlighter for Poke source i.e. .pk files. To use it, first place the file in your ~/.vimrc/syntax/ folder on Unix based systems, or in $HOME/vimfiles/syntax/ on Windows. There are a few ways to tell Vim to use poke.vim for .pk files, but the quickest is to add the following line to your .vimrc:

au BufRead,BufNewFile *.pk set filetype=poke

23.1 .load ¶

The .load command loads a file containing Poke code and compiles and executes it. These files usually have the extension .pk.

If a relative path is provided, then prefix/share/poke is tried first as a base directory to find the specified file. If it is not found, then the current directory is tried next.

If the environment variable POKEDATADIR is defined, it replaces prefix/share/poke. This is mainly intended to test a poke program before it gets installed in its final location.

If an absolute path is provided, it is used as-is.

23.2 .source ¶

The .source command executes command lines from some given file. The syntax is:

.source path

where path is a path to the file containing the commands. See Command Files.

23.3 .file ¶

The .file command opens a new IO space backed by a file, or switches to a previously opened file. When reading or writing the file is accessed via the read and write syscalls. An alternative using the mmap syscall is implemented as .mmap command. See .file The syntax is:

.file path

where path is a path to a file to open, which can be relative to poke’s current working directory or absolute.

Tilde expansion is performed in path, much like it’s done in the shell. This means you can include special characters like ~ (which will expand to your home directory) delimit the file name with " in case it includes leading or trailing blank characters, etc.

When a new file is opened it becomes the current IO space. From that point on, every map executed in the REPL or while loading a Poke program will operate on that IO space:

(poke) .file foo.o
The current file is now `foo.o'.

When .file opens a file, it is opened in whatever mode makes more sense: if the file allows to be written and read then the IO space is open in read/write mode, for example.

If the specified file path doesn’t exist then poke emits an error. However, the flag /c ( for “create”) can be passed to the command to tell poke it should create a new file.

23.4 .mem ¶

The .mem command opens a new IO space backed by a memory buffer. The syntax is:

.mem name

where name is the name of the buffer to create. Note that poke adds prefix and trailing asterisk characters, to differentiate file names from buffer names.

When invoked with no name, poke generates a new unique name for the buffer.

When a new memory buffer IOS is opened it becomes the current IO space. See .file.

23.5 .mmap ¶

The .mmap command opens a new IO space where the content is memory mapped from the device driver or file system. This is especially useful when a kernel driver offers a mmap function for mapping the content of kernel memory into userspace. The most famous example is the mem device with its device node /dev/mem. But there are also many drivers especially in embedded systems using it.

In the case of a regular file the mapping is done by the file system driver. For regular files the .file command would be an alternative and should deliver the same result. The difference is that .file is using read and write system calls to retrieve or set the content of the file whereas .mmap is using the mmap system call which is copying the data from the file into ram memory. This is done by the file system driver and on nowadays file systems .file and .mmap are offered by them. When having enough free ram and accessing the file very often the .mmap is supposed to be faster because there are less system calls to be executed. See .file

In the case of a device driver the behavior depends on the implementation of the driver. It particularly depends on the offered read, write and mmap functions. With /dev/mem for example .file and .mmap should deliver the same results. But there are many drivers used in embedded systems especially for customer hardware which are offering either read and write or mmap. In such a case only the corresponding command can be used. For example when dealing with large amounts of data (video stream) which should be visible to the userspace for further processing it’s almost impossible to copy every single picture to userspace with a driver read function. In such cases device drivers often only offer a mmap function which avoids the extra copying operation.

The syntax is:

.mmap filename, base, size

where filename is the name of the device node or the file name, base is the starting offset (or base address) of the mapping and size is the length of the mapped area.

The base has to be a multiple of the MMU page size on the system. It can be retrieved via the C function getpagesize(). On most systems it’s 4096 Bytes.

The following example is mapping the GPIO registers via /dev/mem on the Beaglebone black board and reading the output enable (GPIO_OE) and data out (GPIO_DATAOUT) registers of the IO memory:

$ poke
(poke) .mmap /dev/mem, 0x4804c000, 0x1000
(poke) int  0x13C#B
(poke) int  0x134#B

When using int on 32 bit systems the full address bus width is used when reading and the same applies to 64 bit systems with the long datatype. All accesses with less bytes than the address bus width are done via a byte by byte copy.

Please write to the poke-devel mailing list if you need a different alignment.

Warning: One should be aware that when writing to mapped registers one is writing to the hardware. Therefore one should be aware of the meaning of the registers as well as the electrical circuit connected. Otherwise the hardware can easily be destroyed. To give an example this could happen when setting a GPIO as output with a high signal while in fact it is connected as an input on the electrical circuit with low signal.

23.6 .nbd ¶

The .nbd command opens a new IO space backed by an external NBD server. The syntax is:

.nbd uri

where uri is the name of the newly created buffer, matching the NBD URI specification.

When a new NBD IOS is opened, it becomes the current IO space. See .file.

NBD support in GNU poke is optional, depending on whether poke was compiled against libnbd.

For an example of connecting to the guest-visible content of a qcow2 image, with the default export name as exposed by using qemu as an NBD server:

$ qemu-nbd --socket=/tmp/mysock -f qcow2 image.qcow2
$ poke
(poke) .nbd nbd+unix:///socket=?/tmp/mysock
The current file is now `nbd+unix:///socket=?/tmp/mysock'.

23.7 .proc ¶

The .proc command opens a new IO space backed by the memory mapped for some live process, or switches to a previously opened process. The syntax is:

.proc pid

where pid is a process identifier. In most systems this is a number.

23.8 .sub ¶

The .sub command opens a new IO sub-space. The syntax is:

.sub ios, base, size[, name]

where ios identifies some existing IO space, base and size are integers denoting the base and size of the range covered by the sub-space (in bytes) and name is an optional name.

23.9 .ios ¶

A list of open IO spaces, and their corresponding identifiers and names, can be obtained using the .info ios command. You can use the .ios command to switch back to a particular IO space by providing an expression that evaluates to its identifier:

(poke) .ios 1
The current IOS is now `foo.o'.
(poke) .ios $</bin/ls>
The current IOS is now `/bin/ls'.

23.10 .close ¶

The .close command closes the selected IO space. The syntax is:

.close ios

Where ios is a Poke expression that evaluates to an IO space identifier. Examples:

(poke) .close 1
(poke) .close $<*foo*>

Note that closing an IO space with this dot-command implies closing any sub IO space having it as a base.

23.11 .doc ¶

The .doc command is used to display this manual in poke’s REPL. The syntax is:

.doc [node]

where node is an optional parameter which indicates the chapter or section at which the manual should be opened.

This command uses whatever documentation viewer configured using .set doc-viewer whose valid options are either info and less.

Unless doc-viewer is set to less, this command uses the info program (The GNU Texinfo Manual) to interactively present the manual. If info is not installed, then less is tried next.

When using less to display the documentation, the entry in the Table of Contents corresponding to the requested entry is highlighted. Just press / and then RET to jump to the corresponding section in the manual.

If neither info nor less are installed, the .doc command will fail. If poke is not running interactively then .doc does nothing.

23.12 .editor ¶

The .editor command (usually abbreviated as .edit) invokes an external text editor on a temporary file. You can then put contents on that file, save it and exit the editor. At that point poke will read the file contents, turn them into a single line and execute them in the repl. If poke is not running interactively, then .editor does nothing.

The editor used is identified by the EDITOR environment variable.

23.13 .info ¶

The .info command provides information about several kinds of entities. The recognized sub commands are:

.info ios

Display a list of open files.

(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
  1	FILE	r	0x00000000#B	0x0000df78#B	foo.o
* 0	FILE	rw	0x00000000#B	0x00000022#B	foo.bson

The file acting as the current IO space is marked with an asterisk character * at the beginning of the file. The mode in which the file is open is also specified. The Id field is the tag of the file that can be passed to the .file command in order to switch to it as the new current IO space:

(poke) .ios 1
The current file is now `foo.o'.
(poke) .info ios
  Id	Type	Mode	Bias		Size		Name
* 1	FILE	r	0x00000000#B	0x0000df78#B	foo.o
  0	FILE	rw	0x00000000#B	0x00000022#B	foo.bson

.info variables [regexp] ¶

Shows a list of defined variables along with their current values and the location where the variables were defined. If a regular expression is provided then only the types whose name match the expression are listed. If a regular expression is provided then only the types whose name match the expression are listed.

.info functions [regexp]

Shows a list of defined functions along with their prototypes and the location where the functions were defined. If a regular expression is provided then only the types whose name match the expression are listed. If a regular expression is provided then only the types whose name match the expression are listed.

.info types [regexp]

Shows a list of defined types along with the locations where the types were defined. If a regular expression is provided then only the types whose name match the expression are listed.

.info type expr|type_specifier

Prints a description of the type of the expression expr, or of the type with specifier with name type_specifier.

23.14 .set ¶

The .set command allows you to inspect and set the value of global settings. The following invocations are valid:

.set

Prints all settings along with their values.

.set setting

Prints the value of the given setting.

.set setting value

Sets the value of setting to value.

The following settings can be handled with .set:

pretty-print

This setting determines whether poke should use pretty-printers while printing out the written representation of struct values.

Pretty-printers are defined as methods named ‘_print’.

auto-map

This setting determines whether to automatically load map files when opening IO spaces.

The decision of what maps to load is driven by the contents of the array ‘auto_map’. Each entry in the array is an array whose first element is a regular expression, and the second element is the name of a map.

Example:

auto_map = [["\.o", "elf"], ["\.o", "dwarf"]];

The entry above makes poke to automatically load the maps ‘elf.map’ and ‘dwarf.map’ when opening any file whose name ends in ‘.o’, provided ‘auto-map’ is set to ‘yes’.

prompt-maps

This setting determines whether a list of open maps is displayed before the poke prompt.

Example of a prompt with map information and ‘prompt-maps’ set to ‘yes’:

(poke) .file foo.o [dwarf,elf](poke)

Which indicates that the opening of the IO space ‘foo.o’ resulted in the ‘elf.map’ and ‘dwarf.map’ maps to be automatically loaded.

omaps

This sort of misnamed setting determines whether to show the mapped offsets and element/field offsets of composite values like arrays and structs.

Example:

(poke) .set omaps yes (poke) [1,2,3] [0x1 @ 0x0UL#b,0x2 @ 0x20UL#b,0x3 @ 0x40UL#b] @ 0x0UL#b

This setting defaults to ‘no’.

obase

This setting determines the output numerical base to use when printing integral values like integers, offset magnitudes, and the like. The supported bases are 2 for binary, 8 for octal, 10 for decimal and 16 for hexadecimal.

oacutoff

This setting determines the maximum number of elements to output when printing out array values. Once the maximum number of elements is reached, an ellipsis ‘...’ is printed to represent the rest of the elements.

Example:

(poke) [1,2,3,4,5] [1,2,3,4,5] (poke) .set oacutoff 3 [1,2,3,...]

The default value is 0, which means no limit.

odepth

This setting determines the maximum number of nested structs to output when printing struct and union values. Once the maximum number of nested values is reached, a collapsed form ‘Type {...}’ is printed.

Example:

(poke) type Foo = struct { struct { int i; } bar; } (poke) Foo {} Foo { bar=struct {...} }

The default value for ‘odepth’ is 0, which means infinite depth.

oindent

This setting determines the width (in blank characters) of each indentation level when printing out struct values in ‘tree’ mode.

Example:

(poke) .set oindent 4 (poke) Foo {} Foo { i=0x0, l=0x0L }

This setting defaults to 2. See also ".help omode".

omode

This setting determines the way binary struct data is displayed.

In ‘flat’ mode data is not formatted in any special way. Each value, be it simple or composite, is printed on one line.

In ‘tree’ mode composite values like structs and arrays are displayed in an hierarchical way, using newlines and indentation.

Examples:

(poke) .set omode flat (poke) Foo {} Foo {i=0x0,l=0x0L}

(poke) .set omode tree (poke) Foo {} Foo { i=0x0, l=0x0L }

The default value of ‘omode’ is ‘flat’.

endian

This setting determines the byte endianness used when accessing IO spaces. Valid values are ‘big’, ‘little’, ‘host’ and ‘network’.

The meaning of ‘big’ and ‘little’ is the obvious one: the endianness is set to big-endian and little-endian respectively.

The ‘host’ endianness is the endianness used by the computer and operating system that is running poke in order to handle it’s own memory and files.

The ‘network’ endianness is the endianness used by the host machine in order to communicate with the network.

doc-viewer

This setting determines what mechanism to use to open the online user manual.

If ‘info’ is selected, the GNU info standalone program is launched to show the info version of the user manual.

If ‘less’ is selected, the less utility is launched to show a plain text version of the user manual.

See also ".help .doc".

pager

This setting determines whether the output of poke commands will be paged or not. Each page will have the height of the terminal.

This setting is ‘no’ by default.

tracer

This setting determines whether the Poke compiler and runtime will generate tracing events or not. This is mainly useful for debugging purposes.

Beware that enabling tracer will undoubtly have an impact in performance, and that every type compiled when tracing is enabled will contain instrumentalization. This means you will most likely want to re-compile the types with tracing disabled once the debugging is done.

This setting is ‘no’ by default.

autoremap

This setting determines whether poke will remap mapped values automatically when they are referred to. This assures that the values are always up to date with respect the contents of the IO space.

Turning auto-remap off may be useful in situations where it is very slow to keep all the data structures up to date. In these case, you can ‘.set autoremap off’ and then use the ‘remap’ Poke operator to force the re-mapping of any given value, explicitly.

This setting is ‘yes’ by default.

23.15 .vm ¶

The Poke Virtual Machine (PVM) executes the programs that are the result of the compilation of what you write in the REPL or the pickles you load. The .vm command provides sub-commands to interact with the PVM.

• .vm disassemble
• .vm profile
• .vm dispatch

23.16 .compiler ¶

The .compiler family of dot-commands provide access to auxiliary features of the Poke incremental compiler. This is mostly used for debugging poke itself.

• .compiler ast

23.17 .exit ¶

The .exit command exits poke. The syntax is:

.exit [status]

Poke will terminate, returning the exit status status. If status is omitted, then the exit status zero will be returned.

23.18 .quit ¶

The .quit command behaves exactly like .exit. See .exit.

24.1 dump ¶

At the most basic level, memory can be examined byte by byte. To do this, use the dump command.

This command has the following synopsis.

dump [:from offset] [:size offset] [:val val]
     [:ios int] [:ruler bool]
     [:ascii bool] [:group_by int] [:cluster_by int]

All arguments are optional, which means the simplest use of the command is to simply type dump:

(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 9b07 5a61 4783 f306 4897 f37c fe39 4cd3  ..ZaG...H..|.9L.
00000010: b6a2 a578 8d82 7b7f 2076 374c 3eab 7150  ...x..{. v7L>.qP
00000020: 31df 8ecb 3d33 ee12 429b 2e13 670d 948e  1...=3..B...g...
00000030: 86f1 2228 ae07 d95c 9884 cf0a d1a8 072e  .."(...\........
00000040: f93c 5368 9617 6c96 3d61 7b92 9038 a93b  .<Sh..l.=a{..8.;
00000050: 3b0d f8c9 efbd a959 88d0 e523 fd3b b029  ;......Y...#.;.)
00000060: e2eb 51d5 cb5b 5ba9 b890 9d7a 2746 72ad  ..Q..[[....z'Fr.
00000070: 6cbd 6e27 1c7f a554 8d2e 77f9 315a 4415  l.n'...T..w.1ZD.

The first row is the ruler which serves as a heading for each subsequent row. On the left hand side is the offset of the io space under examination. The centre block displays the hexadecimal representation of each byte, and on the right hand side is their ascii representation. If a byte is not representable in ascii, then the byte will be displayed as a dot.

By default the dump command reads from the currently selected IO space. However, it is possible to specify an explicit IO space using the ios option:

(poke) var myfile = open ("/path/to/file")
(poke) dump :ios myfile

• Information dump shows
• Presentation options for dump

(poke) dump :from 0x10#B :size 0x40#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000010: b6a2 a578 8d82 7b7f 2076 374c 3eab 7150  ...x..{. v7L>.qP
00000020: 31df 8ecb 3d33 ee12 429b 2e13 670d 948e  1...=3..B...g...
00000030: 86f1 2228 ae07 d95c 9884 cf0a d1a8 072e  .."(...\........
00000040: f93c 5368 9617 6c96 3d61 7b92 9038 a93b  .<Sh..l.=a{..8.;

(poke) dump :ruler 0 :size 0x40#B
00000000: b1fd 1608 2346 759c 46a6 aa94 6fcd 846a  ....#Fu.F...o..j
00000010: e39f 473f 3247 415f 174d a32b ed89 a435  ..G?2GA_.M.+...5
00000020: d2c6 2c52 bc82 e0a7 e767 31ea 84de 41e5  ..,R.....g1...A.
00000030: 2add 2869 e9c2 226b e222 8c74 4b94 af24  *.(i.."k.".tK..$

(poke) dump :ruler 0 :size 0x40#B :ascii 0
00000000: 4393 85e7 0b0c 3921 5a26 39ec 2f5f 5f15
00000010: cc46 e6f3 d50f 6ae6 8988 d50e f8c4 d1c6
00000020: 5a2f 7c3e 490b 18d8 d867 4b6f 2549 1f6c
00000030: 34a9 a0d7 24d2 e9ac 9240 8247 10cb 4ba1

(poke) type Foo = struct { int<32> one; int<64> two; }
(poke) dump :val Foo @ 0#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000            ............     one two

(poke) dump :val int[3] @ 2#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000 0000       ..............   [0] [1] [2]

(poke) type Foo = struct { int<32> one; struct { int<64> three; } two; }
(poke) dump :val Foo @ 0#B
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000            ............     one 1=two

(poke) dump :val 1
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0000 0000 0000 0000 0000            ............     three

dumpval (int<32> val = -1) any

(poke) dump :ascii 0 :size 0x40#B :group_by 4#B
00000000: 68f19a63 df2a8886 c466631c a7fdd5c7  h..c.*...fc.....
00000010: 3075746a 0adb03ca f5b1ff14 6166fa07  0utj........af..
00000020: 0dd3cfbd 8eff46a2 4152a81d 471beddf  ......F.AR..G...
00000030: a0501cae 8bfcec6f 7a4f5701 45ba9fc3  .P.....ozOW.E...

(poke) dump :size 0x40#B :group_by 2#B :cluster_by 4
76543210  0011 2233 4455 6677  8899 aabb ccdd eeff  01234567 89ABCDEF
00000000: 91b8 540d d4dc 49ae  3320 ba7d efd1 16ab  ..T...I. 3 .}....
00000010: b1a8 5ea0 5846 8bea  f741 3f80 42bc 201f  ..^.XF.. .A?.B. .
00000020: 6e5e fa50 23fb f16a  d380 be8c fc98 d195  n^.P#..j ........
00000030: 7bbf fa3e 3fc2 43a4  2a1e 9763 2bd6 5d24  {..>?.C. *..c+.]$

24.2 copy ¶

The command copy allows to copy regions of data inside an IO space, or between different IO spaces.

This command has the following synopsis.

copy [:from offset] [:to offset] [:size offset]
     [:from_ios ios] [:to_ios ios]

All arguments are optional. When invoked with no arguments, copy does nothing.

The arguments from and size determine the region to copy. By default, this region is taken from the current IO space, but this can be overwritten using the optional from_ios argument.

The argument to tells copy where to copy the stuff. It defaults to from. Again, this offset is applied to the current IO space by default, but this can be overwritten using the to_ios argument.

Note that it is allowed for the source and destination ranges to overlap.

24.3 save ¶

Use the save command in order to write a region from an IO space into a file in your file system.

This command has the following synopsis.

save [:ios ios] [:from offset] [:size offset] [:file string]
     [:append bool] [:verbose bool]

All arguments are optional. When invoked with no arguments, save does nothing.

The arguments from and size are used to determine the region of the IO space to save. This is how you would extract and save the contents of an ELF section to a file out.text:

(poke) var s = elf_section_by_name (Elf64_Ehdr @ 0#B, ".text")
(poke) save :file "out.text" :from s.sh_offset :size s.sh_size

Both from and size are arbitrary offsets. You should keep in mind however that files are byte oriented. Therefore saving, say, nine bits to a file will actually write two bytes.

By default save will extract the data from the current IO space. However, it is possible to specify an alternative IOS by using the ios argument.

By default save will truncate the output file, if it exists, before starting writing. If the argument append is set as true, however, it will append to the existing contents of the file. In this case, the file should exist.

24.4 sdiff ¶

The sdiff command provides structured byte differences between two mapped values.

This command has the following synopsis:

sdiff :a any :b any [:group_by offset] \
      [:values bool]

Where :a and :b are two mapped values of the same type.

By default, the bytes are shown in the diff separated by white characters. You can alter this behaviour using the group_by parameter.

If values is true, then include field interpreted values in the diff output.

If the two values are not mapped, this command raises E_no_map. If the two values are not of the same type, the output will probably not be meaningful.

24.5 extract ¶

Use the extract command in order to create a temporary memory IO space with the contents of a mapped value.

This command has the following synopsis.

extract :val val [:to ios_name]

Where :val is a mapped value, and :to is the name of the memory IOS to create (or use). The contents of value are copied to the beginning of the memory IOS *ios_name*.

If no :to argument is specified, or if it is the empty string, then the extract command will generate a suitable unique name and use it.

24.6 scrabble ¶

The scrabble command rearranges portions of an IO space based on a transformation defined by a pair of patterns.

This command has the following synopsis:

scrabble [:from offset] [:size offset]
         [:from_pattern string] [:to_pattern string]
         [:ent_size offset]
         [:from_ios ios] [:to_ios ios]

The optional arguments are documented below.

:from offset

Beginning of the range in the origin IO space from where to start rearranging units. Defaults to 0#B.

:size offset

Size of the range to scrabble in the origin IO space. Defaults to 0#B.

:from_pattern string

String specifying a sequence of units. Each character is an unit. Defaults to "".

:to_pattern string

String specifying the scrabbled units. Each character is an unit. If a character that doesn’t appear in the from_pattern is found in this string, it is ignored. Defaults to from_pattern.

:ent_size offset

Size of the entities to scrabble around. Defaults to 1#B.

:from_ios ios

IO space from where to obtain the data to scrabble. Defaults to the current IO space.

:to_ios ios

The IO space to where write the scrabbled data. Defaults to from_ios.