项目作者: 25A0

项目描述 :
A simple helper utility for parsing binary data in Lua
高级语言: Lua
项目地址: git://github.com/25A0/blob.git
创建时间: 2017-02-21T12:05:15Z
项目社区:https://github.com/25A0/blob

开源协议:The Unlicense

下载


Binary Lunar OBjects library

This is a simple helper utility for parsing binary data in Lua.

Blob is primarily a wrapper around a struct unpack function, spiced up with some
quality of life improvements:

  • No need to keep track of your reading offset
  • Dealing with padding becomes a breeze
  • Define custom types to not repeat yourself
  • Handle uncertainty about what to expect by rolling back changes and putting down markers

It preferably uses the string.unpack function introduced in Lua 5.3,
but can also use Roberto Ierusalimschy’s struct library (http://www.inf.puc-rio.br/~roberto/struct/)
as a fall-back for Lua 5.1 and 5.2.

Quick tour

Blob is designed to help you write sane code for parsing binary data,
but does not try to hide details where precision is necessary.
You will still need to specify things like endianness and the way in which
Strings are represented, but Blob tries to take care of all the tedious bits.

Next we will have a look at my-file.bin, a made-up piece of binary data that
we will parse in this example:

  1. -- [[ my-file.bin
  2. (offset) ( data ) ( ASCII )
  3. 00000000 42 4c 4f 42 71 00 41 55 54 48 67 75 79 40 68 6f |BLOBq.AUTHguy@ho|
  4. 00000010 73 74 2e 63 6f 6d 00 00 00 00 00 00 00 00 00 00 |st.com..........|
  5. 00000020 03 00 80 d3 20 00 c0 69 e0 3e 76 ac 21 f1 3a d6 |.... ..i.>v.!.:.|
  6. 00000030 c0 3f d6 5b 70 36 eb 2d e8 3f 0c b5 3a 15 86 5a |.?.[p6.-.?..:..Z|
  7. 00000040 dd 3f dc 18 a2 e0 6d 0c e1 3f b6 0d 38 c8 da 06 |.?....m..?..8...|
  8. 00000050 cc 3f 77 2c 30 60 3b 16 a8 3f 85 72 ab 7f 42 b9 |.?w,0`;..?.r..B.|
  9. 00000060 e5 3f 98 79 eb d0 cb bc e5 3f 02 d2 7b 13 01 e9 |.?.y.....?..{...|
  10. 00000070 ed 3f 99 16 31 4c 4c 8b d8 3f 1e 3e 61 15 0f 9f |.?..1LL..?.>a...|
  11. 00000080 e0 3f 89 2e 35 a3 44 97 ea 3f df 66 23 88 6f b3 |.?..5.D..?.f#.o.|
  12. 00000090 a1 3f a6 be 36 cc 52 5f ab 3f 0a |.?..6.R_.?.|
  13. 0000009b
  14. ]]
  15. local Blob = require("Blob")
  16. -- load the content of a binary file
  17. local blob = Blob.load("my-file.bin")
  18. -- The first four bytes should contain the string "BLOB"
  19. assert(blob:bytes(4) == "BLOB")
  20. -- This is followed by the version, stored as a 2 byte unsigned integer
  21. local version = blob:unpack("I2")
  22. local author
  23. if version >= 110 then
  24. -- Since version 1.1 of this file format, there might be a field tagged
  25. -- "AUTH", followed by the email-address of the author.
  26. if blob:bytes(4) == "AUTH" then
  27. -- the author's email address is a zero-terminated String
  28. author = blob:zerostring()
  29. else
  30. -- there was no author field, so we want to go back to where we left off
  31. -- before we checked the four bytes
  32. blob:rollback()
  33. end
  34. end
  35. -- We want to skip padding bytes to the next 16 byte boundary
  36. blob:pad(16)
  37. -- Create a custom type that can parse 2D or 3D vectors
  38. blob.types.vector = function(dimensions)
  39. -- The vector has one double value per dimension
  40. return string.rep("d", dimensions)
  41. end
  42. -- Parse a list of pairs of 2D coordinates and a three-dimensional color vector.
  43. -- The number of elements is stored as a two byte unsigned integer
  44. local count = blob:unpack("I2")
  45. -- Now parse the list
  46. local list = blob:array(count, function(blob)
  47. return {
  48. -- The format string in `vector` captures multiple values at once.
  49. -- By surrounding the call with curly braces, we make sure that all
  50. -- captured values are stored in `pos` and `color`.
  51. pos = {blob:vector(2)},
  52. color = {blob:vector(3)},
  53. -- The elements are word-aligned.
  54. blob:pad("word"),
  55. }
  56. end)

Usage

Instantiating

  • local blob = Blob.new(string) creates a new instance from a binary string.
    You can safely use multiple blobs in parallel.

  • local blob = Blob.load(filename) creates a new instance from the content of a file

  • local blob = other_blob:split(length) branch off a shallow copy of the
    other_blob. The new copy will have its initial reading position at the current
    reading position of other_blob. The underlying binary data will not be copied,
    but the reading position and markers of the two blobs are independent once
    the new blob is created. If a length is given, then the other_blob will
    be advanced by that many bytes.

    This function is useful in cases where you want to keep an explicit reference
    to a portion of the blob.

Parsing

  • blob:unpack(formatstring) unpacks a bunch of bytes according to the given
    format string, and advance the offset by the number of consumed bytes.
    See http://www.lua.org/manual/5.3/manual.html#6.4.2 for valid format
    strings, or http://www.inf.puc-rio.br/~roberto/struct/ if you are using
    Lua 5.1 or 5.2.
  • blob:unpack(formatstring, ...) calls string.format(formatstring, ...) and uses
    the resulting formatstring to unpack bytes.
    This is useful for generating format strings on the fly, without having to
    write the string.format boilerplate code every time. Example:
  1. -- Check how many bytes of data are available to read
  2. local bytes = blob:unpack("I2")
  3. -- Read that many bytes
  4. local data = blob:unpack("c%d", bytes)

Both variants of blob:unpack can be used with format strings that capture
multiple values (e.g. "c8I4"). In these cases, the calls will return all those
values. They can be captured in various ways:

  1. -- assign the captured values to individual variables
  2. local r, g, b, a = blob:unpack("BBBB")
  3. -- store all captured values in a table
  4. local coords = {blob:unpack("dd")}

Arrays

  • blob:array(count, fun) Parse a list of count elements by repeatedly parsing
    the blob using fun. The passed function should accept a blob and return
    whatever it parsed. See the tour above for an example.
  • blob:array(count, formatstring) Parse a list of count elements by repeatedly
    unpacking data with formatstring.
  • blob:array(count, formatstring, ...) Apply string formatting with
    string.format(formatstring, ...), then parse a list of count elements
    by repeatedly unpacking data with formatstring.

Bits

  • blob:bits(numbits) Parse a number of bits from the beginning of the next
    byte(s) (that is, the most-significant bit of the next byte will be parsed
    first). All numbits will be returned as boolean values.

    This also works for more than eight bits at once, and advances the offset
    by the minimum number of bytes that are necessary
    to capture all bits (e.g. reading 12 bits will advance the offset by two
    bytes). Thus, each call of bits will advance the offset by at least one
    byte.

    Example:

  1. --[[
  2. Datagram:
  3. 00 01 02 03 04 05 06 07 bit
  4. |- flags -| 00 00 00
  5. ]]
  6. -- Capture 5 bits from the beginning of a byte
  7. local f1, f2, f3, f4, f5 = blob:bits(5)
  • blob:bits(numbits, offset) Similar to the previous function, but skip
    offset bits first, and then starts capturing numbits bits from most
    to least significant.

    Example:

  1. --[[
  2. Datagram:
  3. 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 bit
  4. |- padding -| |- flags -|
  5. ]]
  6. -- Capture 4 bits after skipping 7 padding pits on the left
  7. local f1, f2, f3, f4 = blob:bits(4, 7)

Size

  • blob:size(formatstring) returns the size of the given format string.
    This function does not work for format strings containing zero-terminated
    or size-prefixed strings.
  • blob:size(formatstring, ...) similar to the simple size call, but does
    in-place string formatting with string.format(formatstring, ...) and
    calculates the size of the resulting string.

Offset

If you need to change the offset manually, use blob:seek(pos) to move to a
certain position. You can also directly read or write blob.pos, which contains
the current offset.

The offset is handled in Lua’s 1-based indexing, so that the very first byte is
at offset 1.

Custom types

The module Blob contains an array types where custom types are stored.
These custom types are stored either as a valid formatting string for simple cases,
or as a function, for more complex cases. The default types are:

  1. Blob.types = {
  2. byte = "c1",
  3. bytes = function(count)
  4. return string.format("c%d", count)
  5. end,
  6. word = "c2",
  7. dword = "c4",
  8. zerostring = lib.zerostring, -- see Pitfalls
  9. prefixstring = lib.prefixstring, -- see Pitfalls
  10. }

These types can be used for parsing: If mytype is defined in Blob.types,
then you can use blob:mytype() as a method on a blob to generate a format string.
blob:word() is equivalent to blob:unpack("c2"),
and blob:bytes(i) is equivalent to blob:unpack("c%d", i).

If mytype is a function, then this function should return a valid
format string. If mytype requires arguments, you can specify them when
calling the corresponding method: blob:mytype(a, b) will call mytype(a, b),
and use the returned formatstring to unpack data.

Custom types can also be used for padding (see below).

Custom types are always shared between instances, no matter if they are stored
in blob.types or Blob.types.

Use custom types in size or array like so:

  1. local s1 = blob:size(Blob.types.bytes(16))
  2. local s2 = blob:size(Blob.types.dword)
  3. local list = blob:array(10, Blob.types.bytes(4))

Rollback and Markers

If you parsed some data but then realize that you advanced too far, you can
return to the previous position with blob:rollback(). This will put the reading
position back to where it was before the most recent unpack or comparable
function was called.

By default, up to 64 of these rollback points are saved before old ones are
overridden. You can change this limit by changing Blob.max_rollbacks.

You can also use markers to easily navigate between special positions in the blob.

  • blob:mark() creates an anonymous marker and pushes it to a stack.
  • blob:restore() removes the topmost anonymous marker from the stack, and moves the reading position to that marker
  • blob:drop() removes the tompost anonymous marker from the stack

Use named markers if the stack isn’t enough

  • blob:mark(name) creates a marker named name
  • blob:restore(name) moves the reading position to the marker called name
  • blob:drop() removes the marker called name

Padding

Use blob:pad(size, position) to skip padding bytes in various ways.

Here, size can be:

  • either a size specified in bytes,
  • a formatting string (e.g. “I4”), or
  • a custom type, defined in Blob.types.

The position is optional, and can be one of the following:

  • a numeric value that defines the position relative to which padding should be applied,
  • the string “absolute”, to simply skip a fixed number of padding bytes, or
  • a string that refers to a marker, in which case padding will be aligned
    relative to the position of that marker.

If no position is given, then padding is applied relative to the start of the
blob.

Examples

  • You have finished reading the options field of a TCP packet and want to skip
    to the data field, which starts at the next multiple of 4 bytes:
  1. -- Your current position is 23 (using Lua's indexing). The next byte after
  2. -- a 4 byte boundary is at index 25.
  3. print(blob.pos) -- 23
  4. -- Apply padding to dword boundary ("dword" is a double-word, or 4 bytes)
  5. blob:pad("dword")
  6. -- This skipped two bytes, as expected
  7. print(blob.pos) -- 25
  • You want to skip padding bytes equivalent to the size of a somewhat complex
    struct:
  1. blob:pad("c16I4I4I4")
  • You want to skip to the next boundary of 1024 bytes, but the padding is not
    aligned to the beginning of the blob, but to some other position:
  1. blob:pad(1024, 16)
  • The padding does not follow any alignment; it’s just a fixed number of bytes:
  1. blob:pad(32, "absolute")

Pitfalls

Blob will automatically detect whether string.unpack is available. If not, it
will try to use struct as a fall-back.
However, the APIs of these two libraries are not fully compatible.
In particular, the following restrictions apply:

  • struct offers the format string "c0", which reads a number of characters
    equal to the last parsed numeric value. This does not exist in sting.unpack.
  • string.unpack offers the format string "s[n]" to unpack a string that is
    prefixed by an n-byte unsigned integer. This does not exist in struct.
  • struct allows the format string "c" to read one character. However this
    is not a legal format string in string.unpack. The equivalent format string
    for string.unpack is "c1".
  • A zero-terminated string is decoded as "s" in struct, but as "z" in
    string.unpack.

Blob offers a few workarounds that might help to avoid these problems:

  • Use the custom type prefixstring(n) to decode a string that is prefixed by
    an n-byte unsigned integer. This behaves similar to "s[n]" in
    string.unpack, and will in some cases be enough to replace "c0" in struct.
  • Always specify the number of characters in the format string "c", but make
    sure that this number is never 0.
  • Use the custom type zerostring() to decode a zero-terminated string.
    This will automatically use either "z" or "s", depending on which
    library is used.

The string Blob.backend exposes which library is used internally. It contains
"lua" when Blob uses the string functions of Lua 5.3,
and "struct" when Blob uses the struct library.

Example: Parsing RIFF

Here is how you could parse a generic RIFF file with Blob:

  1. local Blob = require("Blob")
  2. -- define type for four-character codes
  3. Blob.types.fourcc = "c4"
  4. local function parse_chunk(blob)
  5. local chunk = {}
  6. chunk.id = blob:fourcc()
  7. chunk.size = blob:unpack("I4")
  8. -- Both RIFF and LIST chunks contain a four character type
  9. if chunk.id == "RIFF" or chunk.id == "LIST" then
  10. chunk.form_type = blob:fourcc()
  11. chunk.nested = {}
  12. local begin = blob.pos
  13. while blob.pos < begin + chunk.size do
  14. table.insert(chunk.nested, parse_chunk(blob))
  15. end
  16. else
  17. chunk.content = blob:split(chunk.size) -- split off a blob of `size` bytes
  18. blob:pad("word") -- Skip padding to the next word boundary
  19. end
  20. return chunk
  21. end
  22. local function parse_riff(blob)
  23. local riff = parse_chunk(blob)
  24. assert(riff.id == "RIFF")
  25. return riff
  26. end
  27. -- Create a new Blob with the content of the file
  28. local blob = Blob.load("some-file.riff")
  29. local riff = parse_riff(blob)

TODO

  • stateful endianness