Wit Components

[sevki]

Hey folks,

I've been trying to add wit support for loro.

Since loro is split between the loro_internal and loro crates, generating the facade from a wit file works quite well.

for instance the following

wit file

package loro:crdt@1.0.0;

/// Core Loro CRDT component
world loro {
    export document;
    export events;
    export errors;
}

/// Document management interface
interface document {
    use types.{loro-value, container-id, container-type, export-mode, value-or-container};
    use errors.{error};

    /// Document resource handle
    resource document {
        /// Create a new Loro document
        constructor();

        /// Set the peer ID for this document
        set-peer-id: func(peer-id: u64) -> result<_, error>;

        /// Commit pending changes
        commit: func(doc: borrow<document>) -> result<_, error>;

        /// Export document in specified mode
        export-doc: func(mode: export-mode) -> result<list<u8>, error>;

        /// Import updates into document
        import-doc: func(data: list<u8>) -> result<_, error>;

        /// Get document state as JSON string
        to-json: func() -> result<string, error>;

        /// Fork document at current version
        fork: func() -> result<document, error>;

        /// Get a text container
        get-text: func(id: string) -> result<text, error>;

        /// Get a list container
        get-list: func(id: string) -> result<%list, error>;

        /// Get a map container
        get-map: func(id: string) -> result<map, error>;

        /// Get a tree container
        get-tree: func(id: string) -> result<tree, error>;

        /// Get a movable list container
        get-movable-list: func(id: string) -> result<movable-list, error>;

        /// Get a counter container
        get-counter: func(id: string) -> result<counter, error>;
    }

    enum postype {
        /// The index is based on the length of the text in bytes(UTF-8).
        bytes,
        /// The index is based on the length of the text in Unicode Code Points.
        unicode,
        /// The index is based on the length of the text in UTF-16 code units.
        utf16,
        /// The index is based on the length of the text in events.
        /// It is determined by the `wasm` feature.
        event,
        /// The index is based on the entity index.
        entity,
    }

    /// Text container resource handle
    resource text {
        /// Insert text at position
        insert: func(pos: u32, text: string, postype: postype) -> result<_, error>;

        /// Delete text at position
        delete: func(pos: u32, len: u32) -> result<_, error>;

        /// Get text content
        to-string: func() -> result<string, error>;

        /// Get text length
        len: func() -> result<u32, error>;
    }

    /// List container resource handle
    resource %list {
        /// Insert value at index
        insert: func(index: u32, value: loro-value) -> result<_, error>;

        /// Delete elements at index
        delete: func(index: u32, count: u32) -> result<_, error>;

        /// Get value at index
        get: func(index: u32) -> result<option<loro-value>, error>;

        /// Get list length
        len: func() -> result<u32, error>;

        /// Push value to end
        push: func(value: loro-value) -> result<_, error>;
    }

    /// Map container resource handle
    resource map {
        /// Insert key-value pair
        insert: func(key: string, value: loro-value) -> result<_, error>;

        /// Delete key
        delete: func(key: string) -> result<_, error>;

        /// Get value by key
        get: func(key: string) -> result<option<loro-value>, error>;

        /// Check if key exists
        contains: func(key: string) -> result<bool, error>;

        /// Get all keys
        keys: func() -> result<list<string>, error>;
    }

    /// Tree container resource handle
    resource tree {
        /// Create a new tree node
        create: func(parent: option<borrow<tree-node-id>>) -> result<tree-node-id, error>;

        /// Move node to new parent
        move-node: func(node: borrow<tree-node-id>, parent: option<borrow<tree-node-id>>) -> result<_, error>;

        /// Delete node
        delete: func(node: borrow<tree-node-id>) -> result<_, error>;
    }

    /// Movable list container resource handle
    resource movable-list {
        /// Insert value at index
        insert: func(index: u32, value: loro-value) -> result<_, error>;

        /// Delete elements at index
        delete: func(index: u32, count: u32) -> result<_, error>;

        /// Move element from one position to another
        move-item: func(from-index: u32, to-index: u32) -> result<_, error>;

        /// Get value at index
        get: func(index: u32) -> result<option<loro-value>, error>;

        /// Get list length
        len: func() -> result<u32, error>;
    }

    /// Counter container resource handle
    resource counter {
        /// Increment the counter by the given value
        increment: func(value: f64) -> result<_, error>;

        /// Decrement the counter by the given value
        decrement: func(value: f64) -> result<_, error>;

        /// Get the current value of the counter
        get-value: func() -> result<f64, error>;
    }

    /// Tree node identifier resource
    resource tree-node-id;

    /// Container variant type
    variant container {
        text(text),
        %list(%list),
        map(map),
        tree(tree),
        movable-list(movable-list),
        counter(counter),
    }
}

/// Event types for document change notifications
interface events {
    use types.{loro-value, container-id, value-or-container, tree-id};

    /// How an event was triggered
    enum event-trigger-kind {
        /// Triggered by a local change
        local,
        /// Triggered by importing remote changes
        %import,
        /// Triggered by a checkout operation
        checkout,
    }

    /// A diff event containing all changes from an operation
    record diff-event {
        /// How the event was triggered
        triggered-by: event-trigger-kind,
        /// The origin string of the event
        origin: string,
        /// The current target container (if any)
        current-target: option<container-id>,
        /// List of container diffs
        events: list<container-diff>,
    }

    /// A diff for a specific container
    record container-diff {
        /// The target container id
        target: container-id,
        /// The path from root to this container
        path: list<path-item>,
        /// Whether this is from an unknown container
        is-unknown: bool,
        /// The actual diff content
        diff: diff,
    }

    /// A path item representing a step in the container path
    record path-item {
        /// The container ID at this path step
        container: container-id,
        /// The index within the container
        index: index,
    }

    /// Index type for path navigation
    variant index {
        /// Key index for maps
        key(string),
        /// Numeric index for lists/text
        seq(u32),
    }

    /// The main diff variant type
    variant diff {
        /// List diff operations
        %list(list<list-diff-item>),
        /// Text diff operations
        text(list<text-delta>),
        /// Map diff operations
        map(map-delta),
        /// Tree diff operations
        tree(tree-diff),
        /// Counter diff (the change in value)
        counter(f64),
        /// Unknown container diff
        unknown,
    }

    /// A list diff item - insert, delete, or retain
    variant list-diff-item {
        /// Insert new elements
        insert(list-diff-insert),
        /// Delete elements
        delete(list-diff-delete),
        /// Retain elements (no change)
        retain(list-diff-retain),
    }

    /// List insert operation
    record list-diff-insert {
        /// Values to insert
        insert: list<value-or-container>,
        /// Whether this insert is from a move operation
        is-move: bool,
    }

    /// List delete operation
    record list-diff-delete {
        /// Number of elements to delete
        delete: u32,
    }

    /// List retain operation
    record list-diff-retain {
        /// Number of elements to retain
        retain: u32,
    }

    /// Text delta operations
    variant text-delta {
        /// Retain text
        retain(text-retain),
        /// Insert text
        insert(text-insert),
        /// Delete text
        delete(text-delete),
    }

    /// Text retain operation
    record text-retain {
        /// Number of characters to retain
        retain: u32,
        /// Optional attributes for the retained range
        attributes: option<list<text-attribute>>,
    }

    /// Text insert operation
    record text-insert {
        /// The text to insert
        insert: string,
        /// Optional attributes for the inserted text
        attributes: option<list<text-attribute>>,
    }

    /// Text delete operation
    record text-delete {
        /// Number of characters to delete
        delete: u32,
    }

    /// A text attribute key-value pair
    record text-attribute {
        /// The attribute key
        key: string,
        /// The attribute value
        value: loro-value,
    }

    /// Map delta containing updated entries
    record map-delta {
        /// List of updated key-value pairs
        updated: list<map-delta-entry>,
    }

    /// A map delta entry
    record map-delta-entry {
        /// The key that was updated
        key: string,
        /// The new value (None if deleted)
        value: option<value-or-container>,
    }

    /// Tree diff containing tree operations
    record tree-diff {
        /// List of tree diff items
        diff: list<tree-diff-item>,
    }

    /// A single tree diff item
    record tree-diff-item {
        /// The target node being modified
        target: tree-id,
        /// The action performed
        action: tree-external-diff,
    }

    /// Tree external diff action
    variant tree-external-diff {
        /// Create a new node
        create(tree-create-diff),
        /// Move an existing node
        move(tree-move-diff),
        /// Delete a node
        delete(tree-delete-diff),
    }

    /// Tree create diff details
    record tree-create-diff {
        /// The parent of the new node
        parent: tree-parent-id,
        /// Index among siblings
        index: u32,
        /// Fractional index for ordering
        position: string,
    }

    /// Tree move diff details
    record tree-move-diff {
        /// The new parent
        parent: tree-parent-id,
        /// New index among siblings
        index: u32,
        /// New fractional index position
        position: string,
        /// The old parent before the move
        old-parent: tree-parent-id,
        /// The old index before the move
        old-index: u32,
    }

    /// Tree delete diff details
    record tree-delete-diff {
        /// The parent the node was deleted from
        old-parent: tree-parent-id,
        /// The index the node had before deletion
        old-index: u32,
    }

    /// Tree parent identifier
    variant tree-parent-id {
        /// A regular node as parent
        node(tree-id),
        /// The root of the tree (no parent)
        root,
        /// The deleted nodes root
        deleted,
        /// Node doesn't exist yet (for creation)
        unexist,
    }

    /// A batch of diffs for multiple containers
    record diff-batch {
        /// List of container diffs in order
        entries: list<diff-batch-entry>,
    }

    /// A single entry in a diff batch
    record diff-batch-entry {
        /// The container ID
        container-id: container-id,
        /// The diff for this container
        diff: diff,
    }
}

/// Core types used across interfaces
interface types {
    /// Loro value types
    variant loro-value {
        /// Null value
        null,
        /// Boolean value
        %bool(bool),
        /// 64-bit signed integer
        i64(s64),
        /// 64-bit floating point
        %f64(f64),
        /// UTF-8 string
        %string(string),
        /// Binary data
        binary(list<u8>),
        /// Container reference
        container(container-id),
    }

    /// Value or container - used in diffs and list operations
    /// Uses container-id instead of container resource to avoid circular dependency
    variant value-or-container {
        /// A primitive value
        value(loro-value),
        /// A container reference (by ID, resolve via document)
        container(container-id),
    }

    /// Container identifier
    record container-id {
        /// Container ID string
        id: string,
        /// Type of container
        container-type: container-type,
    }

    /// Tree node identifier
    record tree-id {
        /// Peer ID that created this node
        peer: u64,
        /// Counter value for this node
        counter: s32,
    }

    /// Container types
    enum container-type {
        /// Text container
        text,
        /// List container
        %list,
        /// Map container
        %map,
        /// Tree container
        tree,
        /// Movable list container
        movable-list,
        /// Counter container
        counter,
    }

    /// Export modes for document serialization
    enum export-mode {
        /// Full document state with compressed history
        snapshot,
        /// All updates since document creation
        updates,
        /// Incremental updates from a specific version
        updates-from-version,
    }
}

can be implemented with

rust file

use loro_common::{ContainerID, ContainerType, LoroEncodeError, LoroValue, TreeID};
use loro_internal::cursor::PosType;
use loro_internal::loro::ExportMode as InternalExportMode;
use loro_internal::{
    handler::counter::CounterHandler, ListHandler, LoroDoc, MapHandler, MovableListHandler,
    TextHandler, ToJson, TreeHandler, TreeParentId,
};

use crate::bindings::exports::loro::crdt::document::{
    Counter, Document, DocumentBorrow, Error, ExportMode, Guest, GuestCounter, GuestDocument,
    GuestList, GuestMap, GuestMovableList, GuestText, GuestTree, GuestTreeNodeId, List, Map,
    MovableList, Postype, Text, Tree, TreeNodeId,
};
use crate::bindings::loro::crdt::types::{
    ContainerId as WitContainerId, ContainerType as WitContainerType, LoroValue as WitValue,
};
use crate::error::map_error;

/// Empty enum to implement Guest trait on
pub enum Loro {}

impl Guest for Loro {
    type Document = LoroDoc;

    type Text = TextHandler;

    type List = ListHandler;

    type Map = MapHandler;

    type Tree = TreeHandler;

    type MovableList = MovableListHandler;

    type Counter = CounterHandler;

    type TreeNodeId = TreeID;
}

impl GuestDocument for LoroDoc {
    fn new() -> Self {
        LoroDoc::new_auto_commit()
    }

    fn set_peer_id(&self, peer_id: u64) -> Result<(), Error> {
        self.set_peer_id(peer_id).map_err(map_error)
    }

    fn commit(&self, _doc: DocumentBorrow<'_>) -> Result<(), Error> {
        self.commit_then_renew();
        Ok(())
    }

    fn export_doc(&self, mode: ExportMode) -> Result<Vec<u8>, Error> {
        let mode = match mode {
            ExportMode::Snapshot => InternalExportMode::snapshot(),
            ExportMode::Updates => InternalExportMode::all_updates(),
            ExportMode::UpdatesFromVersion => InternalExportMode::all_updates(),
        };

        self.export(mode).map_err(map_encode_error)
    }

    fn import_doc(&self, data: Vec<u8>) -> Result<(), Error> {
        self.import(&data).map(|_| ()).map_err(map_error)
    }

    fn to_json(&self) -> Result<String, Error> {
        Ok(self.get_deep_value().to_json())
    }

    fn fork(&self) -> Result<Document, Error> {
        Ok(Document::new(self.fork()))
    }

    fn get_text(&self, id: String) -> Result<Text, Error> {
        Ok(Text::new(self.get_text(id)))
    }

    fn get_list(&self, id: String) -> Result<List, Error> {
        Ok(List::new(self.get_list(id)))
    }

    fn get_map(&self, id: String) -> Result<Map, Error> {
        Ok(Map::new(self.get_map(id)))
    }

    fn get_tree(&self, id: String) -> Result<Tree, Error> {
        Ok(Tree::new(self.get_tree(id)))
    }

    fn get_movable_list(&self, id: String) -> Result<MovableList, Error> {
        Ok(MovableList::new(self.get_movable_list(id)))
    }

    fn get_counter(&self, id: String) -> Result<Counter, Error> {
        Ok(Counter::new(self.get_counter(id)))
    }
}

impl GuestText for TextHandler {
    fn insert(&self, pos: u32, text: String, postype: Postype) -> Result<(), Error> {
        self.insert(pos as usize, &text, PosType::from(postype))
            .map_err(map_error)
    }

    fn delete(&self, pos: u32, len: u32) -> Result<(), Error> {
        self.delete_utf16(pos as usize, len as usize)
            .map_err(map_error)
    }

    fn to_string(&self) -> Result<String, Error> {
        let value = self.get_richtext_value();
        if let LoroValue::String(s) = value {
            Ok(s.to_string())
        } else {
            Ok(value.to_json())
        }
    }

    fn len(&self) -> Result<u32, Error> {
        len_to_u32(self.len_utf16())
    }
}

impl GuestList for ListHandler {
    fn insert(&self, index: u32, value: WitValue) -> Result<(), Error> {
        let value = to_internal_value(value)?;
        self.insert(index as usize, value).map_err(map_error)
    }

    fn delete(&self, index: u32, count: u32) -> Result<(), Error> {
        self.delete(index as usize, count as usize)
            .map_err(map_error)
    }

    fn get(&self, index: u32) -> Result<Option<WitValue>, Error> {
        self.get(index as usize).map(to_wit_value).transpose()
    }

    fn len(&self) -> Result<u32, Error> {
        len_to_u32(self.len())
    }

    fn push(&self, value: WitValue) -> Result<(), Error> {
        let value = to_internal_value(value)?;
        self.push(value).map_err(map_error)
    }
}

impl GuestMap for MapHandler {
    fn insert(&self, key: String, value: WitValue) -> Result<(), Error> {
        let value = to_internal_value(value)?;
        self.insert(&key, value).map_err(map_error)
    }

    fn delete(&self, key: String) -> Result<(), Error> {
        self.delete(&key).map_err(map_error)
    }

    fn get(&self, key: String) -> Result<Option<WitValue>, Error> {
        self.get(&key).map(to_wit_value).transpose()
    }

    fn contains(&self, key: String) -> Result<bool, Error> {
        Ok(self.get(&key).is_some())
    }

    fn keys(&self) -> Result<Vec<String>, Error> {
        let mut keys = Vec::new();
        self.for_each(|k, _| keys.push(k.to_string()));
        Ok(keys)
    }
}

impl GuestMovableList for MovableListHandler {
    fn insert(&self, index: u32, value: WitValue) -> Result<(), Error> {
        let value = to_internal_value(value)?;
        self.insert(index as usize, value).map_err(map_error)
    }

    fn delete(&self, index: u32, count: u32) -> Result<(), Error> {
        self.delete(index as usize, count as usize)
            .map_err(map_error)
    }

    fn move_item(&self, from_index: u32, to_index: u32) -> Result<(), Error> {
        self.mov(from_index as usize, to_index as usize)
            .map_err(map_error)
    }

    fn get(&self, index: u32) -> Result<Option<WitValue>, Error> {
        self.get(index as usize).map(to_wit_value).transpose()
    }

    fn len(&self) -> Result<u32, Error> {
        len_to_u32(self.len())
    }
}

impl GuestTree for TreeHandler {
    fn create(
        &self,
        parent: Option<crate::bindings::exports::loro::crdt::document::TreeNodeIdBorrow<'_>>,
    ) -> Result<TreeNodeId, Error> {
        let parent_id = parent.map(|p| *p.get::<TreeID>());
        self.create(TreeParentId::from(parent_id))
            .map(TreeNodeId::new)
            .map_err(map_error)
    }

    fn move_node(
        &self,
        node: crate::bindings::exports::loro::crdt::document::TreeNodeIdBorrow<'_>,
        parent: Option<crate::bindings::exports::loro::crdt::document::TreeNodeIdBorrow<'_>>,
    ) -> Result<(), Error> {
        let node_id = *node.get::<TreeID>();
        let parent_id = parent.map(|p| *p.get::<TreeID>());
        self.mov(node_id, TreeParentId::from(parent_id))
            .map_err(map_error)
    }

    fn delete(
        &self,
        node: crate::bindings::exports::loro::crdt::document::TreeNodeIdBorrow<'_>,
    ) -> Result<(), Error> {
        self.delete(*node.get::<TreeID>()).map_err(map_error)
    }
}

impl GuestTreeNodeId for TreeID {}

impl GuestCounter for CounterHandler {
    fn increment(&self, value: f64) -> Result<(), Error> {
        CounterHandler::increment(self, value).map_err(map_error)
    }

    fn decrement(&self, value: f64) -> Result<(), Error> {
        CounterHandler::decrement(self, value).map_err(map_error)
    }

    fn get_value(&self) -> Result<f64, Error> {
        use loro_internal::HandlerTrait;
        Ok(HandlerTrait::get_value(self).into_double().unwrap_or(0.0))
    }
}

impl From<Postype> for PosType {
    fn from(value: Postype) -> Self {
        match value {
            Postype::Bytes => PosType::Bytes,
            Postype::Unicode => PosType::Unicode,
            Postype::Utf16 => PosType::Utf16,
            Postype::Event => PosType::Event,
            Postype::Entity => PosType::Entity,
        }
    }
}

fn to_internal_container_type(ty: WitContainerType) -> ContainerType {
    match ty {
        WitContainerType::Text => ContainerType::Text,
        WitContainerType::List => ContainerType::List,
        WitContainerType::Map => ContainerType::Map,
        WitContainerType::Tree => ContainerType::Tree,
        WitContainerType::MovableList => ContainerType::MovableList,
        WitContainerType::Counter => ContainerType::Counter,
    }
}

fn to_wit_container_type(ty: ContainerType) -> Result<WitContainerType, Error> {
    match ty {
        ContainerType::Text => Ok(WitContainerType::Text),
        ContainerType::List => Ok(WitContainerType::List),
        ContainerType::Map => Ok(WitContainerType::Map),
        ContainerType::Tree => Ok(WitContainerType::Tree),
        ContainerType::MovableList => Ok(WitContainerType::MovableList),
        ContainerType::Counter => Ok(WitContainerType::Counter),
        _ => Err(Error::NotImplemented(format!(
            "Unsupported container type for WIT mapping: {:?}",
            ty
        ))),
    }
}

fn to_internal_container_id(id: WitContainerId) -> Result<ContainerID, Error> {
    match ContainerID::try_from(id.id.as_str()) {
        Ok(cid) => Ok(cid),
        Err(_err) => {
            let container_type = to_internal_container_type(id.container_type);
            Ok(ContainerID::new_root(&id.id, container_type))
        }
    }
}

fn to_wit_container_id(id: &ContainerID) -> Result<WitContainerId, Error> {
    Ok(WitContainerId {
        id: format!("{}", id),
        container_type: to_wit_container_type(id.container_type())?,
    })
}

fn to_internal_value(value: WitValue) -> Result<LoroValue, Error> {
    match value {
        WitValue::Null => Ok(LoroValue::Null),
        WitValue::Bool(v) => Ok(LoroValue::Bool(v)),
        WitValue::I64(v) => Ok(LoroValue::I64(v)),
        WitValue::F64(v) => Ok(LoroValue::Double(v)),
        WitValue::String(v) => Ok(LoroValue::String(v.into())),
        WitValue::Binary(v) => Ok(LoroValue::Binary(v.into())),
        WitValue::Container(c) => to_internal_container_id(c).map(LoroValue::Container),
    }
}

fn to_wit_value(value: LoroValue) -> Result<WitValue, Error> {
    match value {
        LoroValue::Null => Ok(WitValue::Null),
        LoroValue::Bool(v) => Ok(WitValue::Bool(v)),
        LoroValue::Double(v) => Ok(WitValue::F64(v)),
        LoroValue::I64(v) => Ok(WitValue::I64(v)),
        LoroValue::Binary(v) => Ok(WitValue::Binary((*v).clone())),
        LoroValue::String(v) => Ok(WitValue::String(v.to_string())),
        LoroValue::Container(id) => Ok(WitValue::Container(to_wit_container_id(&id)?)),
        other => Err(Error::NotImplemented(format!(
            "Cannot convert LoroValue variant {:?} to WIT value",
            other
        ))),
    }
}

fn len_to_u32(len: usize) -> Result<u32, Error> {
    u32::try_from(len).map_err(|_| Error::ArgError("Length exceeds u32".into()))
}

fn map_encode_error(err: LoroEncodeError) -> Error {
    Error::Unknown(format!("Encode error: {err}"))
}

The public facade in loro maps so well to the types, I could tab tab auto complete all most of it, (which is why this looks like it was vibe coded, whoops, but the point about how mapping a wit generted interfaces to loro_internal types stands).

Now the actual problem I have is this bit of code here

[target.'cfg(all(target_arch = "wasm32", not(features = "wasm")))'.dependencies]
wasm-bindgen = "0.2.100"

which breaks building things. Would you folks be up for me sending some wit compat patches or would that be out of scope?

[canadaduane]

I'm curious about wit! What is this?

[nightscape]

I guess this: https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md