Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Encoding reference

JSON (std::json), the codec-agnostic wire layer (std::wire), the binary codec (std::binary), raw bytes (std::bytes), and base64 (std::base64).

The short version: derive Json for JSON in/out at app boundaries, derive Wire for rpc payloads, and let the codecs do the rest. Everything below “Derives” here is plumbing you only meet when building custom transports or parsers.

JSON

trait Json { fun to_json(self): str; }                     // encode
trait FromJson {                                           // decode
	fun from_json(text: str): Result<Self, str>;
	fun from_json_value(value: JsonValue): Result<Self, str>;
}

[derive(Json)] implements both from a struct/enum’s shape; scalars, List, Option, Result and fixed arrays nest.

A fixed array [T; n] is a JSON array, as a List is, and decoding it checks the count: [1] read as an [i32; 2] is the error “expected an array of 2 elements, found 1”. The array type heads the decode’s path as any type does — [i32; 2]::from_json(text), and the derive writes [i32; 2]::from_json_value(..) for an array field — and from_json_value_of<[i32; 2]>(value) is the same call spelled generically.

Result<T, E> is Json when both arms are, and it is spelled the way the codecs spell it: externally tagged by variant NAME, Ok(7) as {"Ok":7} and Err("nope") as {"Err":"nope"}. That is the same text [derive(Json)] gives a one-payload variant and the same text the wire layer’s Result writes, so one value crosses to_json, encode_json and the binary codec under one encoding. Decoding checks the shape before the tag: a document that is not an object, and a tag the type does not declare, are both decode errors.

Encoding (to_json) is total, but decoding is fallible: the input is untrusted, so a missing field, a wrong-shaped value, or text that isn’t JSON is a decode error rather than silent garbage or a crash. Both from_json(text) and from_json_value(value) return Result<Self, str>; handle it with !, match, or is Ok(..).

The two decode methods differ in what they take, not in what they answer: from_json parses the text (non-crashing) and hands off to from_json_value, which validates an already-parsed value’s shape. That is the one to call when a value is nested inside another decode, and the one to write when implementing the trait by hand.

The scalar lanes check the VALUE and not only the JSON kind, and it is the same rule the codec reader applies: an integer lane wants a whole number (i32::from_json("1.5") is Err), and an unsigned one wants a non-negative one (u32::from_json("-1") is Err). A derived type’s field decodes through its own type’s from_json_value, so a struct field gets the rule too. The magnitude is deliberately not checked: i32/u32 are the runtime’s numeric lanes rather than hardware widths.

import std::json::{ Json, FromJson, JsonValue, parse_json_value };
import std::result::Result::{ self, Ok, Err };

[derive(Json)]
struct Point {
	x: i32,
	y: i32,
}

fun main() {
	let point = Point { x = 1, y = 2 };
	let text = point.to_json();
	print(text); // {"x":1,"y":2}

	match Point::from_json(text) {
		Ok(let back) => print(back.x), // 1
		Err(let reason) => print(reason),
	}

	// The already-parsed form, same `Result`.
	match Point::from_json_value(parse_json_value("{\"x\":3,\"y\":4}")) {
		Ok(let back) => print(back.y), // 4
		Err(let reason) => print(reason),
	}

	// A missing field is a decode error naming the field.
	match Point::from_json("{\"x\":1}") {
		Ok(_) => print("decoded"),
		Err(let reason) => print(reason), // missing field y
	}

	// So is text that isn't JSON at all.
	match Point::from_json("not json") {
		Ok(_) => print("decoded"),
		Err(let reason) => print(reason), // not valid JSON
	}
}

Written by hand — for a type whose encoding isn’t its shape — the signatures are the trait’s, Self included:

import std::json::{ Json, FromJson, JsonValue };
import std::result::Result::{ self, Ok, Err };

// A newtype that encodes as the bare string it wraps, not as an object.
struct Tag {
	name: str,
}

impl Tag with Json {
	fun to_json(self): str {
		self.name.to_json()
	}
}

impl Tag with FromJson {
	fun from_json(text: str): Result<Self, str> {
		Tag::from_json_value(text.try_parse_json().ok_or("not valid JSON")!)
	}
	fun from_json_value(value: JsonValue): Result<Self, str> {
		Ok(Tag { name = str::from_json_value(value)! })
	}
}

fun main() {
	print(Tag { name = "ada" }.to_json()); // "ada"
	match Tag::from_json("\"ada\"") {
		Ok(let tag) => print(tag.name),    // ada
		Err(let reason) => print(reason),
	}
}

Untyped inspection, when the shape isn’t known up front:

external struct JsonValue;
fun parse_json_value(text: str): JsonValue  // throws on malformed text
str.try_parse_json(): Option<JsonValue>     // the safe form

enum JsonKind { Null, Bool, Number, String, Array, Object }

value.kind(): JsonKind
value.is_null(): bool
value.field(name: str): JsonValue
value.has_field(name: str): bool
value.elements(): List<JsonValue>
value.tag(): str         // an enum discriminator, NOT a type — see below

kind() is the value’s JSON type, normalized: the host’s typeof calls both an array and null an "object", so the intrinsic names those two itself. JsonKind is a backed enum carrying exactly those strings, so value.kind() == JsonKind::Number is one comparison against "number" — the set is closed in the type system rather than in a doc comment, and Object and Null are as usable as the other four. is_null() is a separate intrinsic: it tests the value against null directly.

The set is closed over JSON, not over every JsonValue. value.field(name) for a key the object does not have is the host’s undefined, whose kind is none of the six — == answers false for it, as it always did, and an exhaustive match over JsonKind panics naming it. has_field is the check that keeps you out of that case.

tag() answers a different question and is not a spelling of kind(): it reads an externally-tagged enum’s discriminator — the string itself for a bare "Variant", the single key for a {"Variant":…} object. That is what a derived decoder calls to pick a variant, so it only means anything on those two shapes: on a number or a bool it yields "undefined", and on null it throws. Reach for kind() unless you are decoding an enum by hand.

import std::json::{ JsonKind, parse_json_value };

fun main() {
	let value = parse_json_value("{\"name\":\"ada\",\"tags\":[\"x\",\"y\"]}");
	print(value.kind() == JsonKind::Object);        // true
	print(value.field("name").kind().value());      // string
	print(value.has_field("age"));                  // false

	for element in value.field("tags").elements() {
		if element.kind() == JsonKind::String {
			print(element.kind().value()); // string, twice
		}
	}

	// `tag()` is the other question: an externally-tagged enum's discriminator.
	print(parse_json_value("\"Start\"").tag());         // Start
	print(parse_json_value("{\"Text\":\"hi\"}").tag()); // Text
}

json_codec(): Codec is the JSON wire codec for rpc (see below).

The wire layer (std::wire)

The codec-agnostic serialization protocol under derive(Wire) and rpc:

  • trait Serialize / trait Deserialize: visitor-style value description (begin_struct/field/str_value/i53_value/…), every method on &mut self — a visitor is written to as a value narrates itself. The wire scalars are str, bool and the whole sized numeric family — i8, u8, i16, u16, i32, u32, i53, u53, usize, f32, f64 — plus lists, options, results, maps, structs and enum variants. The visitor itself has six scalar lanes and the sized widths ride the one that holds them exactly (i8/i16 on i32, u8/u16 on u32, u53 on i53, f32 on f64), so every round trip is lossless and no codec grows a method per width. usize is the exception, and a deliberate one: it rides the i32 lane, the width a length or a position has always had on the wire, so a frame does not grow by four bytes a position when a position is respelled usize. A usize past i32::MAX cannot be described.
  • Frame: one encoded message.
  • Codec: a matched writer/reader pair, json_codec() (std::json, readable) or binary_codec() (std::binary, compact). Client and server must agree.

[derive(Wire)] requires every field to be Wire, recursively, checked at the derive site. You implement Serialize/Deserialize by hand only for types with a custom encoding. Neither [derive(Wire)] nor [derive(Json)] takes a trait object among a type’s members (source: dyn Src, a List<dyn Src>, a variant’s dyn payload): the decoder rebuilds every member by its type, and a dyn names none — hold the concrete type, or an enum of the concrete types, there instead.

A reader ENFORCES the type it is asked for. The frame is text a stranger chose, so a read whose value is of the wrong JSON kind poisons the reader with a reason naming what was expected and what was found — expected a number, found a string — and every later read answers its zero value. The integer lanes additionally want a WHOLE number, and the unsigned ones a non-negative one, because 1.5 typed i32 is a value outside its own type. A list opened with begin_list and closed with end_list must have had all of its elements read; a list SHORTER than the reads is caught by the same gate, since the read runs into the enclosing value. failed() is where that shows up, and it is what the rpc route consults between decoding a call’s arguments and running the handler — so a malformed call answers a Decode failure rather than running an implementation on zero values.

Writing a Wire impl by hand

Wire is two methods, and the visitor is a &mut view in both:

trait Wire {
	fun describe<S: Serialize>(self, serializer: &mut S);
	fun rebuild<D: Deserialize>(deserializer: &mut D): Self;
}

The VALUE is by-value self — describing reads it. The VISITOR is &mut, because narrating into it is what moves its cursor. Inside the body the parameter already is the view, so a nested describe or rebuild forwards it bare:

impl Pair with Wire {
	fun describe<S: Serialize>(self, serializer: &mut S) {
		serializer.begin_struct(2);
		serializer.field("left");
		self.left.describe(serializer);     // bare: already a view
		serializer.field("right");
		self.right.describe(serializer);
		serializer.end_struct();
	}

	fun rebuild<D: Deserialize>(deserializer: &mut D): Pair {
		deserializer.begin_struct();
		deserializer.field("left");
		let left = i32::rebuild(deserializer);
		deserializer.field("right");
		let right = str::rebuild(deserializer);
		deserializer.end_struct();
		Pair { left = left, right = right }
	}
}

&mut serializer appears only where you hold the visitor by VALUE — a Serializer/Deserializer record handed to you by a Codec, or one of rpc’s |Serializer| void describer closures. There the record is a bag of closures over the codec’s own writer, so a mut re-binding costs a copy of the bag and writes through to the same writer:

// An rpc describer closure: by value in, `&mut` at the call.
[|mut serializer: Serializer| id.describe(&mut serializer)]

encode_json/decode_json/encode_binary/decode_binary do this for you and are the paths to prefer.

It gives you the wire codec and nothing else: a type that also needs to_json/from_json asks for both, [derive(Json, Wire)]. The two are separate trait families with separate field rules — HashMap and any hand-written impl … with Wire type are Wire and are not Json — so a Wire derive that quietly emitted a JSON codec as well would refuse fields the wire boundary admits, in to_json’s vocabulary rather than Wire’s.

Wire is a trait like any other, and what counts as Wire is what an impl … with Wire applies to — the derive is one way to get one, not the definition. A conditional impl’s own binder bounds are what recurse into the arguments, so impl Pair<type A: Wire, type B: Wire> with Wire demands both and impl Handle<type T> with Wire demands neither.

A GENERIC [derive(Wire)] type gets that same shape from the derive: every parameter a field or payload reaches is bound under Wire, and a parameter no member mentions (Handle<T>’s T, which names an arena and never travels) is left bare. So Page<T> below derives impl Page<type T: Wire> with Wire: Page<i32> is Wire, Page<|| i32> is refused wherever a Wire type is asked for, and a generic function that sends a Page<T> declares T: Wire itself (the refusal names the bound to add).

import std::io::print;
import std::json::{ encode_json, decode_json };
import std::result::Result::{ self, Ok, Err };
import std::option::Option::{ self, Some, None };
import std::wire::Wire;

[derive(Wire)]
struct Page<T> { items: List<T>, next: Option<T> }

fun resend<T: Wire>(page: Page<T>): Result<Page<T>, str> {
	decode_json(encode_json(page))
}

fun main() {
	match resend(Page { items = [1, 2], next = Some(3) }) {
		Ok(let page) => print(page.items.len()),
		Err(let reason) => print(reason),
	}
}

Result<T, E> is Wire when both arms are, and narrates in Option’s vocabulary: an externally-tagged {"Ok": …} / {"Err": …} over JSON, a tagged variant over the binary codec. That makes a fallible reply an ordinary payload — [rpc] fun lookup(self, id: u53): Result<Row, str> needs nothing hand-written. An unrecognized tag is a sticky decode failure, so decode answers Err(reason) rather than panicking on a malformed frame. std::json’s direct pair writes the same tags, so the two spellings of a Result are one encoding.

HashMap<K, V> is Wire when both its key and its value are (the key is already Hashable by the type’s own bound). It narrates as a list of {key, value} pairs — codec-neutral, readable in JSON, and insertion-ordered in both directions.

Keyed collections

Two more names carry the keyed half of the reactive protocol ([expose(keyed)], see the services guide):

trait Keyed<K> {
	fun key(self): K;
}

enum Delta<K, T> {
	Reset(List<T>),      // the collection BECOMES this
	Insert(K, T, usize), // a new element at an index (on the wire at i32's width); a key already held is replaced
	Update(K, T),        // the element under this key takes a new value
	Remove(K),           // the element under this key is gone
}

Keyed is what makes two snapshots of a collection comparable element by element: without it, the only thing a channel can say is “here is the whole value again”. Delta is what it says instead. Update and Remove name a key that must already be present — an absent one is a protocol error, not a silent no-op.

Backed enums on the wire

A backed enum — one whose variants carry an explicit value, enum Align { Start = "flex-start", … } — encodes as that value rather than as its variant name, for both Json and Wire. Align::Start is "flex-start" on the wire, not "Start", and it decodes through Align::parse, so a peer sending a value outside the set is a decode error rather than a confidently-wrong variant. One explicit value backs the whole enum, so enum Level { Low = 0, Mid, High } puts 1 on the wire for Mid, not "Mid".

Adding a backing value to an existing derived enum is a wire-format break, and so is removing one. An enum with no backing value keeps the externally-tagged form ("Start", or {"Text":…} with a payload), so the two shapes are not interchangeable across a version.

Bytes

An immutable-length byte array (Uint8Array underneath), the currency of the binary codec, crypto, and websockets:

impl Bytes {
	fun alloc(size: i32): Bytes
	fun len(self): usize
	fun get(self, index: usize): i32
	fun set(self, index: usize, value: i32)
	fun slice(self, from: usize, to: usize): Bytes
	fun fill(self, value: i32, from: usize, to: usize): Bytes
	fun copy_into(self, source: Bytes, offset: i32)
	fun concat(a: Bytes, b: Bytes): Bytes     // static
	fun to_hex(self): str
}

// UTF-8 text ↔ bytes
fun encode_utf8(text: str): Bytes
fun decode_utf8(bytes: Bytes): str

Lower still: ByteBuffer/DataView (host ArrayBuffer access, read_f64/write_f64), the binary codec’s float channel.

Binary codec (std::binary)

fun binary_codec(): Codec
fun encode_binary<T: Wire>(value: T): Bytes
fun decode_binary<T: Wire>(bytes: Bytes): Result<T, str>
struct BinaryWriter { … }   // write_byte / write_i32 / write_str / finish(): Bytes

Its writers take &mut self and keep their state in plain fields, like std::json’s.

Same model as JSON, compact layout. i53 values ride as f64 bit patterns, exact to 2^53.

The reader validates too, and the format is schema-ORDERED, so what it validates is the bytes rather than a kind: a read past the buffer, a length prefix claiming more than the frame holds, an Option marker or a bool byte that is neither 0 nor 1, and a frame LONGER than the value it declares — bytes left over are bytes the two sides disagree about, and reading a prefix as the whole is how a caller gets a value that was never sent. Each is a sticky failure naming what was expected and what was found, exactly as the JSON reader’s are.

Base64 (std::base64)

URL-safe alphabet, no padding (the JWT flavor):

fun encode_url(bytes: Bytes): str
fun decode_url(text: str): Option<Bytes>