Kim Encoding

Kim is a character and count encoding designed by Douglas Crockford. It encodes Unicode characters and variable-length integers using continuation bytes. Kim is simpler and more compact than UTF-8 for most text.

Continuation Bytes

The fundamental idea in Kim is the continuation byte:

C  D  D  D  D  D  D  D

• C — continue bit. If 1, read another byte. If 0, this is the last byte.
• D (7 bits) — data bits.

To decode: shift the accumulator left by 7 bits, add the 7 data bits. If the continue bit is 1, repeat with the next byte. If 0, the value is complete.

To encode: take the value, emit 7 bits at a time from most significant to least significant, setting the continue bit on all bytes except the last.

Character Encoding

Kim encodes Unicode codepoints directly as continuation byte sequences:

Every Unicode character — including emoji and characters from extended planes — is encoded whole, in at most 3 bytes, and stands on its own.

Examples

'A'       (U+0041)  →  41
'é'       (U+00E9)  →  81 69
'💩'      (U+1F4A9)  →  87 E9 29

Count Encoding

Kim is also used for encoding counts (lengths, sizes). The same continuation byte format represents non-negative integers of arbitrary size:

Comparison with UTF-8

Kim trades self-synchronization (the ability to find character boundaries from any position) for simplicity and compactness. In practice, Kim text is accessed sequentially, so self-synchronization is not needed.

Usage in ƿit

Kim is used internally by blobs, by the Nota message format, and — on kim8-profile builds — as the heap text body itself.

As a Text Body (the kim8 profile)

The runtime’s heap-text representation is a build profile axis (axis C in the recipe’s stamp.text). The default utf32 profile packs two UTF-32 characters per 64-bit word (O(1) indexing; fash64 word hash). The kim8 profile (-Dtext=kim8, source/pit_text_kim.c) stores heap text as kim-encoded bytes with the character count and used byte count on the object, and hashes the bytes with FNV-1a.

Language semantics are identical across both profiles. The complexity is not: kim is not random-access, so indexing and slicing seek from the start of the text in O(n) — accepted by ruling for the small-target profiles kim8 serves (per-character loops over long texts degrade to O(n²); the runtime’s own text operations iterate sequentially and stay O(n) total). The payoff is density: ASCII-heavy text costs 1 byte per character instead of 4, and any codepoint costs at most 3. The profile string folds the choice into every artifact and cache key, so builds never cross text profiles.

In Blobs

The blob.write_text and blob.read_text functions use Kim to encode text into binary data:

def blob = use('blob')
var b = blob.make()
blob.write_text(b, "hello")  // Kim-encoded length + characters
stone(b)
var text = blob.read_text(b, 0)  // "hello"

In Nota

Nota uses Kim for two purposes:

1. Counts — array lengths, text lengths, blob sizes, record pair counts
2. Characters — text content within Nota messages

The preamble byte of each Nota value incorporates the first few bits of a Kim-encoded count, with the continue bit indicating whether more bytes follow.

See Nota Format for the full specification.