Chapter 2

Bytevectors

Many applications deal with blocks of binary data by accessing them in various ways—extracting signed or unsigned numbers of various sizes. Therefore, the (rnrs bytevectors (6))library provides a single type for blocks of binary data with multiple ways to access that data. It deals with integers and floating-point representations in various sizes with specified endianness.

Bytevectorsare objects of a disjoint type. Conceptually, a bytevector represents a sequence of 8-bit bytes. The description of bytevectors uses the term byte for an exact integer object in the interval { - 128, ..., 127} and the term octet for an exact integer object in the interval {0, ..., 255}. A byte corresponds to its two’s complement representation as an octet.

The length of a bytevector is the number of bytes it contains. This number is fixed. A valid index into a bytevector is an exact, non-negative integer object less than the length of the bytevector. The first byte of a bytevector has index 0; the last byte has an index one less than the length of the bytevector.

Generally, the access procedures come in different flavors according to the size of the represented integer and the endianness of the representation. The procedures also distinguish signed and unsigned representations. The signed representations all use two’s complement.

Like list and vector literals, literals representing bytevectors must be quoted:

’#vu8(12 23 123)         ⇒ #vu8(12 23 123)

2.1  Endianness

Many operations described in this chapter accept an endianness argument. Endianness describes the encoding of exact integer objects as several contiguous bytes in a bytevector [4]. For this purpose, the binary representation of the integer object is split into consecutive bytes. The little-endian encoding places the least significant byte of an integer first, with the other bytes following in increasing order of significance. The big-endian encoding places the most significant byte of an integer first, with the other bytes following in decreasing order of significance.

This terminology also applies to IEEE-754 numbers: IEEE-754 describes how to represent a floating-point number as an exact integer object, and endianness describes how the bytes of such an integer are laid out in a bytevector.

Note:   Little- and big-endianness are only the most common kinds of endianness. Some architectures distinguish between the endianness at different levels of a binary representation.

2.2  General operations

The name of <endianness symbol> must be a symbol describing an endianness. An implementation must support at least the symbols big and little, but may support other endianness symbols. (endianness <endianness symbol>) evaluates to the symbol named <endianness symbol>. Whenever one of the procedures operating on bytevectors accepts an endianness as an argument, that argument must be one of these symbols. It is a syntax violation for <endianness symbol> to be anything other than an endianness symbol supported by the implementation.

Note:   Implementors should use widely accepted designations for endianness symbols other than big and little.

Note:   Only the name of <endianness symbol> is significant.

Returns the endianness symbol associated implementation’s preferred endianness (usually that of the underlying machine architecture). This may be any <endianness symbol>, including a symbol other than big and little.

Returns #t if obj is a bytevector, otherwise returns #f.

Returns a newly allocated bytevector of k bytes.

If the fill argument is missing, the initial contents of the returned bytevector are unspecified.

If the fill argument is present, it must be an exact integer object in the interval { - 128, ... 255} that specifies the initial value for the bytes of the bytevector: If fill is positive, it is interpreted as an octet; if it is negative, it is interpreted as a byte.

Returns, as an exact integer object, the number of bytes in bytevector.

Returns #t if bytevector₁ and bytevector₂ are equal—that is, if they have the same length and equal bytes at all valid indices. It returns #f otherwise.

The fill argument is as in the description of the make-bytevector procedure. The bytevector-fill! procedure stores fill in every element of bytevector and returns unspecified values. Analogous to vector-fill!.

target target-start k)

Source and target must be bytevectors. Source-start, target-start, and k must be non-negative exact integer objects that satisfy

where l_source is the length of source and l_target is the length of target.

The bytevector-copy! procedure copies the bytes from source at indices

to consecutive indices in target starting at target-index.

This must work even if the memory regions for the source and the target overlap, i.e., the bytes at the target location after the copy must be equal to the bytes at the source location before the copy.

This returns unspecified values.

(let ((b (u8-list->bytevector ’(1 2 3 4 5 6 7 8))))
  (bytevector-copy! b 0 b 3 4)
  (bytevector->u8-list b))         ⇒ (1 2 3 1 2 3 4 8)

Returns a newly allocated copy of bytevector.

2.3  Operations on bytes and octets

K must be a valid index of bytevector.

The bytevector-u8-ref procedure returns the byte at index k of bytevector, as an octet.

The bytevector-s8-ref procedure returns the byte at index k of bytevector, as a (signed) byte.

(let ((b1 (make-bytevector 16 -127))
      (b2 (make-bytevector 16 255)))
  (list
    (bytevector-s8-ref b1 0)
    (bytevector-u8-ref b1 0)
    (bytevector-s8-ref b2 0)
    (bytevector-u8-ref b2 0))) 
                ⇒ (-127 129 -1 255)

K must be a valid index of bytevector.

The bytevector-u8-set! procedure stores octet in element k of bytevector.

The bytevector-s8-set! procedure stores the two’s-complement representation of byte in element k of bytevector.

Both procedures return unspecified values.

(let ((b (make-bytevector 16 -127)))

  (bytevector-s8-set! b 0 -126)
  (bytevector-u8-set! b 1 246)

  (list
    (bytevector-s8-ref b 0)
    (bytevector-u8-ref b 0)
    (bytevector-s8-ref b 1)
    (bytevector-u8-ref b 1))) 
                ⇒ (-126 130 -10 246)

List must be a list of octets.

The bytevector->u8-list procedure returns a newly allocated list of the octets of bytevector in the same order.

The u8-list->bytevector procedure returns a newly allocated bytevector whose elements are the elements of list list, in the same order. It is analogous to list->vector.

2.4  Operations on integers of arbitrary size

Size must be a positive exact integer object. K, ..., k + size - 1 must be valid indices of bytevector.

The bytevector-uint-ref procedure retrieves the exact integer object corresponding to the unsigned representation of size size and specified by endianness at indices k, ..., k + size - 1.

The bytevector-sint-ref procedure retrieves the exact integer object corresponding to the two’s-complement representation of size size and specified by endianness at indices k, ..., k + size - 1.

For bytevector-uint-set!, n must be an exact integer object in the interval {0, ..., 256^mathitsize - 1}.

The bytevector-uint-set! procedure stores the unsigned representation of size size and specified by endianness into bytevector at indices k, ..., k + size - 1.

For bytevector-sint-set!, n must be an exact integer object in the interval { - 256^mathitsize/2, ..., 256^mathitsize/2 - 1}. bytevector-sint-set! stores the two’s-complement representation of size size and specified by endianness into bytevector at indices k, ..., k + size - 1.

The ...-set! procedures return unspecified values.

(define b (make-bytevector 16 -127))

(bytevector-uint-set! b 0 (- (expt 2 128) 3)
                     (endianness little) 16)

(bytevector-uint-ref b 0 (endianness little) 16)
                ⇒
    #xfffffffffffffffffffffffffffffffd

(bytevector-sint-ref b 0 (endianness little) 16)
                ⇒ -3

(bytevector->u8-list b)
                ⇒ (253 255 255 255 255 255 255 255
               255 255 255 255 255 255 255 255)

(bytevector-uint-set! b 0 (- (expt 2 128) 3)
                 (endianness big) 16)

(bytevector-uint-ref b 0 (endianness big) 16) 
                ⇒
    #xfffffffffffffffffffffffffffffffd

(bytevector-sint-ref b 0 (endianness big) 16) 
                ⇒ -3

(bytevector->u8-list b) 
                ⇒ (255 255 255 255 255 255 255 255
               255 255 255 255 255 255 255 253))

Size must be a positive exact integer object. For uint-list->bytevector, list must be a list of exact integer objects in the interval {0, ..., 256^mathitsize - 1}. For sint-list->bytevector, list must be a list of exact integer objects in the interval { - 256^mathitsize/2, ..., 256^mathitsize/2 - 1}. The length of bytevector or, respectively, of list must be divisible by size.

These procedures convert between lists of integer objects and their consecutive representations according to size and endianness in the bytevector objects in the same way as bytevector->u8-list and u8-list->bytevector do for one-byte representations.

(let ((b (u8-list->bytevector ’(1 2 3 255 1 2 1 2))))
  (bytevector->sint-list b (endianness little) 2)) 
                ⇒ (513 -253 513 513)

(let ((b (u8-list->bytevector ’(1 2 3 255 1 2 1 2))))
  (bytevector->uint-list b (endianness little) 2)) 
                ⇒ (513 65283 513 513)

2.5  Operations on 16-bit integers

K must be a valid index of bytevector; so must k + 1. For bytevector-u16-set! and bytevector-u16-native-set!, n must be an exact integer object in the interval {0, ..., 2¹⁶ - 1}. For bytevector-s16-set! and bytevector-s16-native-set!, n must be an exact integer object in the interval { - 2¹⁵, ..., 2¹⁵ - 1}.

These retrieve and set two-byte representations of numbers at indices k and k + 1, according to the endianness specified by endianness. The procedures with u16 in their names deal with the unsigned representation; those with s16 in their names deal with the two’s-complement representation.

The procedures with native in their names employ the native endianness, and work only at aligned indices: k must be a multiple of 2.

The ...-set! procedures return unspecified values.

(define b
  (u8-list->bytevector
    ’(255 255 255 255 255 255 255 255
      255 255 255 255 255 255 255 253)))

(bytevector-u16-ref b 14 (endianness little)) 
                ⇒ 65023
(bytevector-s16-ref b 14 (endianness little)) 
                ⇒ -513
(bytevector-u16-ref b 14 (endianness big)) 
                ⇒ 65533
(bytevector-s16-ref b 14 (endianness big)) 
                ⇒ -3

(bytevector-u16-set! b 0 12345 (endianness little))
(bytevector-u16-ref b 0 (endianness little)) 
                ⇒ 12345

(bytevector-u16-native-set! b 0 12345)
(bytevector-u16-native-ref b 0)         ⇒ 12345

(bytevector-u16-ref b 0 (endianness little)) 
                ⇒ unspecified

2.6  Operations on 32-bit integers

K, ..., k + 3 must be valid indices of bytevector. For bytevector-u32-set! and bytevector-u32-native-set!, n must be an exact integer object in the interval {0, ..., 2³² - 1}. For bytevector-s32-set! and bytevector-s32-native-set!, n must be an exact integer object in the interval { - 2³¹, ..., 2³² - 1}.

These retrieve and set four-byte representations of numbers at indices k, ..., k + 3, according to the endianness specified by endianness. The procedures with u32 in their names deal with the unsigned representation; those with s32 with the two’s-complement representation.

The procedures with native in their names employ the native endianness, and work only at aligned indices: k must be a multiple of 4.

The ...-set! procedures return unspecified values.

(define b
  (u8-list->bytevector
    ’(255 255 255 255 255 255 255 255
      255 255 255 255 255 255 255 253)))

(bytevector-u32-ref b 12 (endianness little)) 
                ⇒ 4261412863
(bytevector-s32-ref b 12 (endianness little)) 
                ⇒ -33554433
(bytevector-u32-ref b 12 (endianness big)) 
                ⇒ 4294967293
(bytevector-s32-ref b 12 (endianness big)) 
                ⇒ -3

2.7  Operations on 64-bit integers

K, ..., k + 7 must be valid indices of bytevector. For bytevector-u64-set! and bytevector-u64-native-set!, n must be an exact integer object in the interval {0, ..., 2⁶⁴ - 1}. For bytevector-s64-set! and bytevector-s64-native-set!, n must be an exact integer object in the interval { - 2⁶³, ..., 2⁶⁴ - 1}.

These retrieve and set eight-byte representations of numbers at indices k, ..., k + 7, according to the endianness specified by endianness. The procedures with u64 in their names deal with the unsigned representation; those with s64 with the two’s-complement representation.

The procedures with native in their names employ the native endianness, and work only at aligned indices: k must be a multiple of 8.

The ...-set! procedures return unspecified values.

(define b
  (u8-list->bytevector
    ’(255 255 255 255 255 255 255 255
      255 255 255 255 255 255 255 253)))

(bytevector-u64-ref b 8 (endianness little)) 
                ⇒ 18302628885633695743
(bytevector-s64-ref b 8 (endianness little)) 
                ⇒ -144115188075855873
(bytevector-u64-ref b 8 (endianness big)) 
                ⇒ 18446744073709551613
(bytevector-s64-ref b 8 (endianness big)) 
                ⇒ -3

2.8  Operations on IEEE-754 representations

K, ..., k + 3 must be valid indices of bytevector. For bytevector-ieee-single-native-ref, k must be a multiple of 4.

These procedures return the inexact real number object that best represents the IEEE-754 single precision number represented by the four bytes beginning at index k.

K, ..., k + 7 must be valid indices of bytevector. For bytevector-ieee-double-native-ref, k must be a multiple of 8.

These procedures return the inexact real number object that best represents the IEEE-754 single precision number represented by the eight bytes beginning at index k.

k x endianness)

K, ..., k + 3 must be valid indices of bytevector. For bytevector-ieee-single-native-set!, k must be a multiple of 4.

These procedures store an IEEE-754 single-precision representation of x into elements k through k + 3 of bytevector, and return unspecified values.

k x endianness)

K, ..., k + 7 must be valid indices of bytevector. For bytevector-ieee-double-native-set!, k must be a multiple of 8.

These procedures store an IEEE-754 double precision representation of x into elements k through k + 7 of bytevector, and return unspecified values.

2.9  Operations on strings

This section describes procedures that convert between strings and bytevectors containing Unicode encodings of those strings. When decoding bytevectors, encoding errors are handled as with the replace semantics of textual I/O (see section 8.2.4): If an invalid or incomplete character encoding is encountered, then the replacement character U+FFFD is appended to the string being generated, an appropriate number of bytes are ignored, and decoding continues with the following bytes.

Returns a newly allocated (unless empty) bytevector that contains the UTF-8 encoding of the given string.

If endianness is specified, it must be the symbol big or the symbol little. The string->utf16 procedure returns a newly allocated (unless empty) bytevector that contains the UTF-16BE or UTF-16LE encoding of the given string (with no byte-order mark). If endianness is not specified or is big, then UTF-16BE is used. If endianness is little, then UTF-16LE is used.

If endianness is specified, it must be the symbol big or the symbol little. The string->utf32 procedure returns a newly allocated (unless empty) bytevector that contains the UTF-32BE or UTF-32LE encoding of the given string (with no byte mark). If endianness is not specified or is big, then UTF-32BE is used. If endianness is little, then UTF-32LE is used.

Returns a newly allocated (unless empty) string whose character sequence is encoded by the given bytevector.

endianness endianness-mandatory)

Endianness must be the symbol big or the symbol little. The utf16->string procedure returns a newly allocated (unless empty) string whose character sequence is encoded by the given bytevector. Bytevector is decoded according to UTF-16BE or UTF-16LE: If endianness-mandatory? is absent or #f, utf16->string determines the endianness according to a UTF-16 BOM at the beginning of bytevector if a BOM is present; in this case, the BOM is not decoded as a character. Also in this case, if no UTF-16 BOM is present, endianness specifies the endianness of the encoding. If endianness-mandatory? is a true value, endianness specifies the endianness of the encoding, and any UTF-16 BOM in the encoding is decoded as a regular character.

Note:   A UTF-16 BOM is either a sequence of bytes #xFE, #xFF specifying big and UTF-16BE, or #xFF, #xFE specifying little and UTF-16LE.

endianness endianness-mandatory)

Endianness must be the symbol big or the symbol little. The utf32->string procedure returns a newly allocated (unless empty) string whose character sequence is encoded by the given bytevector. Bytevector is decoded according to UTF-32BE or UTF-32LE: If endianness-mandatory? is absent or #f, utf32->string determines the endianness according to a UTF-32 BOM at the beginning of bytevector if a BOM is present; in this case, the BOM is not decoded as a character. Also in this case, if no UTF-32 BOM is present, endianness specifies the endianness of the encoding. If endianness-mandatory? is a true value, endianness specifies the endianness of the encoding, and any UTF-32 BOM in the encoding is decoded as a regular character.

Note:   A UTF-32 BOM is either a sequence of bytes #x00, #x00, #xFE, #xFF specifying big and UTF-32BE, or #xFF, #xFE, #x00, #x00, specifying little and UTF-32LE.