{
  "id": "2026-05-01-hcs-streaming-wire-protocol-b4e2ed96c9",
  "scope": "redkey",
  "source_of_truth": "repo",
  "source_path": "docs/specs/2026-05-01-hcs-streaming-wire-protocol.md",
  "source_kind": "markdown",
  "visibility": "internal",
  "renderer_id": "design_doc.dreamborn-forge.generated.v1",
  "design_system": "dreamborn-design-system:forge",
  "generated_at": "2026-05-09T13:00:55.699Z",
  "artifact_type": "design_doc",
  "schema_version": "design_doc.generated.v1",
  "title": "HCS Streaming Wire Protocol",
  "summary": "HCS Streaming Wire Protocol Date: 2026 05 01 Status: Draft Related: docs/specs/2026 04 28 agent wire protocol.md, docs/specs/2026 04 27 hcs first claim path.md, docs/specs/2026 04 21 orchestration design.md Core Idea Agent Wire is the typed envelope. HCS is the canonical stream. Instead of treating a Wire as only a package sent at the end of work, the platfo...",
  "format_source": "markdown",
  "sections": [
    {
      "title": "HCS Streaming Wire Protocol",
      "level": 1,
      "body": "**Date:** 2026-05-01\n**Status:** Draft\n**Related:** `docs/specs/2026-04-28-agent-wire-protocol.md`, `docs/specs/2026-04-27-hcs-first-claim-path.md`, `docs/specs/2026-04-21-orchestration-design.md`\n\n---"
    },
    {
      "title": "Core Idea",
      "level": 2,
      "body": "Agent Wire is the typed envelope. HCS is the canonical stream.\n\nInstead of treating a Wire as only a package sent at the end of work, the platform treats every meaningful coordination event as a Wire. Wires that share a `stream_id` form a stream. HCS provides ordering, timestamping, immutability, and replay. Supabase materializes the current state for fast queries and Studio rendering.\n\n```text\nWire = typed event envelope\nStream = ordered chain of Wires\nHCS = canonical append-only stream log\nSupabase = reducer/read model\nStudio = human rendering layer\n```\n\nThis does not replace the existing Wire protocol. It extends it.\n\n---"
    },
    {
      "title": "Why This Matters",
      "level": 2,
      "body": "The current model coordinates mostly by completion:\n\n```text\ntask.available -> task.claim -> task.complete\n```\n\nThat works for simple work, but project-level coordination has more live state than a final package can express:\n\n- who has claimed the work\n- whether the agent started\n- what plan the agent is following\n- whether an artifact is ready before the whole task is complete\n- whether another agent has a blocking concern\n- whether a lease is stale\n- whether runner capacity is saturated or idle\n\nStreaming changes coordination from handoff-based to live orchestration.\n\n```text\ntask.available\ntask.claimed\ntask.started\ntask.plan.created\nartifact.claimed\nartifact.ready\nreview.concern\ntask.complete\n```\n\nThe package still exists. It becomes the terminal snapshot or commitment. The stream is the process record.\n\n---"
    },
    {
      "title": "Design Principle",
      "level": 2,
      "body": "HCS is the source of truth. Supabase is a mirror.\n\nIf Supabase is wrong, stale, or unavailable, the system can replay HCS and rebuild the read model. Agents should coordinate from HCS-originated events, not from Supabase as a second work queue.\n\nThis matches the HCS-first claim path:\n\n```text\nHCS role topic -> task.available\nrunner -> task.claimed Wire on HCS\nlistener -> materializes claim to Supabase\nrunner -> lifecycle Wires on HCS\nlistener -> reduces stream to current state\nEngine -> releases downstream work from materialized state\n```\n\n---"
    },
    {
      "title": "HCS As Stream Transport",
      "level": 2,
      "body": "Each HCS topic is an append-only event log.\n\n```text\nHCS topic = stream transport\nHCS sequence number = canonical topic ordering\nHCS consensus timestamp = canonical event time\nHCS message body = Wire envelope\n```\n\nThe stream itself is a logical grouping over Wires:\n\n```text\nstream_id = task:0.0.8830671:attempt:1\n```\n\nMultiple streams can live on the same HCS topic. The listener groups them by `stream_id`.\n\n---"
    },
    {
      "title": "Practical Starting Point: Shared Topics With Stream IDs",
      "level": 3,
      "body": "Keep the existing role and task topic architecture. Add stream metadata to each Wire.\n\n```text\nroles.developer topic\n  stream_id=task:A:attempt:1\n  stream_id=task:B:attempt:1\n  stream_id=task:C:attempt:1\n\ntask topic\n  stream_id=task:A:attempt:1\n```\n\nThis is the lowest-friction path because it keeps the current role-topic claim model intact.\n\nPros:\n\n- fewer HCS topics\n- compatible with current runners and listener\n- easy incremental adoption\n- stream grouping happens in the read model\n\nTrade-off:\n\n- HCS sequence is topic-global, not stream-local, so `stream_seq` must be included or derived by the reducer."
    },
    {
      "title": "Future Option: Topic Per Long-Running Stream",
      "level": 3,
      "body": "For high-value or long-running streams, the platform may create dedicated topics.\n\n```text\nproject topic\nphase topic\ntask topic\n```\n\nPros:\n\n- clean isolation\n- easy replay per project/task\n- HCS sequence can directly serve as stream order\n\nTrade-off:\n\n- more topics to create, track, and secure\n- more operational overhead\n\n---"
    },
    {
      "title": "Stream-Enabled Wire Envelope",
      "level": 2,
      "body": "Current Wire envelope:\n\n```json\n{\n  \"wire\": \"1.0\",\n  \"type\": \"message_type\",\n  \"sender\": \"agent_slug | system | human\",\n  \"ts\": \"ISO8601\",\n  \"payload\": {}\n}\n```\n\nStreaming adds identity, ordering, and causality fields:\n\n```json\n{\n  \"wire\": \"1.1\",\n  \"wire_id\": \"wire_01\",\n  \"stream_id\": \"task:0.0.8830671:attempt:1\",\n  \"stream_seq\": 12,\n  \"correlation_id\": \"project:knowledgevault-m01r\",\n  \"causation_id\": \"wire_00\",\n  \"type\": \"task.started\",\n  \"sender\": \"agent:quinn-03\",\n  \"ts\": \"2026-05-01T15:12:03Z\",\n  \"payload\": {\n    \"task_id\": \"0.0.8830671\",\n    \"attempt\": 1\n  }\n}\n```\n\n| Field | Meaning |\n|---|---|\n| `wire_id` | Unique ID for this event envelope |\n| `stream_id` | The running process this Wire belongs to |\n| `stream_seq` | Monotonic order within the stream |\n| `correlation_id` | Broader project, phase, client, or workflow this relates to |\n| `causation_id` | The Wire that directly caused this event |\n| `type` | Typed event name |\n| `payload` | Event-specific data |\n\nThe listener should also persist the HCS metadata:\n\n| Field | Meaning |\n|---|---|\n| `hcs_topic_id` | Topic that carried the Wire |\n| `hcs_sequence_number` | Canonical topic order |\n| `consensus_timestamp` | Canonical event time |\n\n---"
    },
    {
      "title": "Event Discipline",
      "level": 2,
      "body": "The stream should contain operational facts, not raw model output.\n\nGood stream events:\n\n- `task.claimed`\n- `task.started`\n- `task.plan.created`\n- `artifact.claimed`\n- `artifact.ready`\n- `contract.ready`\n- `review.concern`\n- `task.blocked`\n- `task.complete`\n- `agent.heartbeat`\n- `agent.capacity.updated`\n\nBad stream events:\n\n- raw model thoughts\n- token-by-token prose\n- large file contents\n- verbose logs\n- secrets\n- temporary scratch notes\n\nThe rule:\n\n**Stream only facts that help scheduling, recovery, review, or visibility.**\n\n---"
    },
    {
      "title": "Large Data And Artifacts",
      "level": 2,
      "body": "HCS is not a blob store. HCS carries the coordination event and integrity proof. Large artifacts live off-chain.\n\n```json\n{\n  \"wire\": \"1.1\",\n  \"wire_id\": \"wire_artifact_ready_01\",\n  \"stream_id\": \"task:0.0.8830671:attempt:1\",\n  \"stream_seq\": 8,\n  \"correlation_id\": \"project:knowledgevault-m01r\",\n  \"type\": \"artifact.ready\",\n  \"sender\": \"agent:quinn-03\",\n  \"ts\": \"2026-05-01T15:30:00Z\",\n  \"payload\": {\n    \"artifact_type\": \"migration\",\n    \"artifact_ref\": \"supabase-storage://task-runs/0.0.8830671/005_experts_auth_alignment.sql\",\n    \"sha256\": \"abc123\",\n    \"ready_for\": [\"review\", \"downstream_tasks\"]\n  }\n}\n```\n\nThe artifact can be in Supabase Storage, HFS, Git, S3, or another durable store. The Wire stores the reference and hash.\n\n---"
    },
    {
      "title": "Project Coordination With Streams",
      "level": 2,
      "body": "A project becomes multiple related streams:\n\n```text\nproject:<project_id>\nphase:<phase_id>\ntask:<task_id>:attempt:<n>\nagent:<agent_id>\nartifact:<artifact_id>\n```\n\nThe project stream records the high-level graph:\n\n```text\nproject.created\nphase.created\ntask.created\ndependency.created\ngate.opened\ngate.passed\nproject.complete\n```\n\nTask streams record execution:\n\n```text\ntask.available\ntask.claimed\ntask.started\ntask.plan.created\nartifact.claimed\nartifact.ready\ntask.blocked\ntask.complete\n```\n\nAgent streams record worker health and capacity:\n\n```text\nagent.started\nagent.heartbeat\nagent.capacity.updated\nagent.lease.accepted\nagent.lease.released\nagent.stopped\n```\n\nThis lets Engine coordinate while work is happening instead of waiting for terminal packages.\n\n---"
    },
    {
      "title": "Earlier Dependency Release",
      "level": 2,
      "body": "Without streaming, downstream work waits for `task.complete`.\n\nWith streaming, downstream work can release when a specific dependency is ready.\n\nExample:\n\n```text\nR1 task: migration/auth contract\nR2 task: API route\nR3 task: middleware\n```\n\nR2 and R3 may not need all of R1 complete. They may only need the auth contract.\n\n```text\nR1 -> contract.ready(auth_expert_id_mapping)\nEngine -> releases R2 and R3\nR1 -> task.complete later\n```\n\nThis improves parallelism without weakening auditability because the release condition is itself an HCS Wire.\n\n---"
    },
    {
      "title": "Multi-Agent Role Pools",
      "level": 2,
      "body": "With many agents per role, the stream is the coordination and backpressure layer.\n\n```text\n15 developer agents watch roles.developer\ntask.available appears\nagents race to claim\nlowest valid claim wins\nwinner emits task.started\nlosers keep watching\n```\n\nClaims should include a lease and optional capacity reservation:\n\n```json\n{\n  \"wire\": \"1.1\",\n  \"wire_id\": \"wire_claim_01\",\n  \"stream_id\": \"task:0.0.9001:attempt:1\",\n  \"stream_seq\": 2,\n  \"correlation_id\": \"project:kv-m02\",\n  \"type\": \"task.claimed\",\n  \"sender\": \"agent:quinn-07\",\n  \"ts\": \"2026-05-01T16:00:00Z\",\n  \"payload\": {\n    \"task_id\": \"0.0.9001\",\n    \"agent\": \"quinn-07\",\n    \"role\": \"developer\",\n    \"lease_expires_at\": \"2026-05-01T16:30:00Z\",\n    \"capacity_units_reserved\": 3\n  }\n}\n```\n\nAgents also emit capacity:\n\n```json\n{\n  \"wire\": \"1.1\",\n  \"wire_id\": \"wire_capacity_01\",\n  \"stream_id\": \"agent:quinn-07\",\n  \"stream_seq\": 44,\n  \"type\": \"agent.capacity.updated\",\n  \"sender\": \"agent:quinn-07\",\n  \"ts\": \"2026-05-01T16:01:00Z\",\n  \"payload\": {\n    \"available_units\": 4,\n    \"max_units\": 6,\n    \"skills\": [\"typescript\", \"nextjs\", \"supabase\"]\n  }\n}\n```\n\nEngine uses claims, leases, heartbeats, and capacity to decide how much work to release.\n\n---"
    },
    {
      "title": "Reducers And Read Models",
      "level": 2,
      "body": "The listener becomes a stream processor:\n\n```text\npoll HCS topic\ndecode Wire\ninsert raw event\nupdate stream offset\nrun reducer for affected stream_id\nupdate materialized read model\n```\n\nSuggested raw event table:\n\n```text\nwire_events\n- id\n- hcs_topic_id\n- hcs_sequence_number\n- consensus_timestamp\n- wire_id\n- stream_id\n- stream_seq\n- correlation_id\n- causation_id\n- type\n- sender\n- payload\n- received_at\n```\n\nSuggested stream state table:\n\n```text\nstream_state\n- stream_id\n- stream_type\n- status\n- last_wire_id\n- last_event_type\n- last_event_at\n- correlation_id\n- reducer_version\n- state jsonb\n```\n\nSuggested task run state:\n\n```text\ntask_run_state\n- stream_id\n- task_id\n- attempt\n- status\n- claimed_by\n- lease_expires_at\n- last_heartbeat_at\n- plan_ref\n- blocker\n- output_ref\n- verification_status\n```\n\nSuggested agent state:\n\n```text\nagent_runtime_state\n- agent_id\n- role\n- status\n- available_units\n- max_units\n- active_leases\n- last_heartbeat_at\n```\n\nThese tables are derived. HCS remains canonical.\n\n---"
    },
    {
      "title": "Level 1: Current Package Mode",
      "level": 3,
      "body": "Keep the existing events:\n\n```text\ntask.available\ntask.claim\ntask.complete\ntask.blocked\n```"
    },
    {
      "title": "Level 2: Stream Metadata",
      "level": 3,
      "body": "Add stream fields to existing Wires:\n\n```text\nwire_id\nstream_id\nstream_seq\ncorrelation_id\ncausation_id\n```\n\nNo major behavior change yet."
    },
    {
      "title": "Level 3: Runner Lifecycle",
      "level": 3,
      "body": "Add a small set of lifecycle events:\n\n```text\ntask.claimed\ntask.started\ntask.plan.created\nagent.heartbeat\ntask.complete\ntask.blocked\n```\n\nThis enables live task state, stale lease detection, and better Studio visibility."
    },
    {
      "title": "Level 4: Coordination Events",
      "level": 3,
      "body": "Add artifact and review events:\n\n```text\nartifact.claimed\nartifact.ready\ncontract.ready\nreview.concern\ngate.opened\ngate.passed\n```\n\nThis enables earlier review and partial dependency release."
    }
  ],
  "html_path": "artifacts/2026-05-01-hcs-streaming-wire-protocol-b4e2ed96c9.html",
  "json_path": "artifacts/2026-05-01-hcs-streaming-wire-protocol-b4e2ed96c9.json"
}