SPATIAL
ABSTRACT
Kona
Kona is a native macOS viewport with Metal surfaces designed around the Genesis embodied-AI simulator. Python owns physics and scene state; Swift owns rendering, camera orbit, and joint control over a shared-memory + TCP transport.
Overview
Kona has a driver layer for Genesis that is a split-process robotics viewport. A Python producer owns the Genesis scene, physics stepping, and camera/renderer, while a native SwiftUI/Metal application renders the live camera feed and drives joint targets — without embedding a Genesis GUI window. The split keeps simulation numerically isolated in Python (where Genesis and PyTorch live) while giving the interactive surface a real macOS UI: gesture-driven camera orbit, an XYZ axes HUD, and per-joint sliders for the robot's degrees of freedom.
The reference scene is a Franka Emika Panda arm loaded from its MJCF definition. There is the ability to parameterize the model loaded such as the Ant, from the MuJoCo Menagerie to shake out model-specific assumptions in the DOF pipeline. A Python-side supervisor process wraps the producer with crash detection and rate-limited auto-restart, so the simulation survives iterative development without a manual relaunch.
GenesisDrive macOS Viewport — Franka Panda, live joint control
Architecture
Component Graph
genesis_producer.py [Python]
│
┌─────┴──────────────────────────────────────────┐
│ Genesis Scene / Camera ──► SharedFrameRing │
└───────┬────────────────────────────────────────┘
│ NDJSON, one dictionary per line
▼
GenesisDrive.app [Swift, macOS]
┌──────────────┐ ┌─────────────────┐ ┌────────────────┐
│ ControlPlane │ │ SharedFrameRing │ │ ViewportBridge │
│Client (actor)│ │ (mmap reader) │ │ @unchecked │
└──────┬───────┘ └────────┬────────┘ │ Sendable) │
│ │ FramePayload └──────────┬─────┘
┌──────▼─────────┐ └───────────────────────▶│
│ GenesisSession │ │ signal()
│ camera / DOF │ ┌────────▼──────────────────┐
│ / connection │ │ MetalViewport Renderer │
└──────┬─────────┘ │ MTKView, axes gizmo (MSL) │
│ SwiftUI └───────────────────────────┘
┌──────▼───────────────────────────────┐
│ ContentView — viewport + AxesTray + │
│ JointControlTray (drag / sliders) │
└──────────────────────────────────────┘
genesis-supervisor acts as RootSupervisor and ProcessLifecycleActor,
auto-restarts genesis_producer on crash. There is a sliding-window rate limit (max 3 restarts / 60 s); orchestrates shutdown
order - TCP SHUTDOWN before SIGTERM.
Components
| Component | Role |
|---|---|
genesis_producer | Owns the Genesis Scene, robot entity, and render camera; steps physics, renders RGBA8 frames into the shared-memory ring, and runs the TCP ControlServer |
SharedFrameRing | File-backed mmap at /tmp/genesis_viewport_rgb0 — 64-byte control block (magic GSHM) + 3 triple-buffered slots (magic GFRM per frame header); lockless read of latest_completed_index |
ControlPlaneClient | Swift actor wrapping NWConnection to 127.0.0.1:7654; newline-delimited JSON framing; exposes an AsyncStream<ControlEvent> |
GenesisSession | @MainActor @Observable — connection state, spherical camera (azimuth/elevation/distance), robotDofs / dofPositions, gamepad bridge, joint velocity + sim-time state (Track 7) |
ViewportBridge + MetalViewportRenderer | Bridges the mmap-read FramePayload to an MTKView; uploads the RGBA8 texture, blits full-screen, and draws an inline-MSL XYZ axes gizmo each frame at 30 fps |
AxesTray / JointControlTray | Collapsible SwiftUI overlays (.ultraThinMaterial, TrayHandle ported from Indigo) — live az/el/dist legend, and 7 arm sliders (J1–J7) + 1 symmetric grip slider driving both finger DOFs |
GameControllerBridge | @MainActor @Observable — Xbox/MFi controller input via GameController.framework, 30 Hz tick, "Option B" page-select scheme for mapping sticks/triggers to joint groups |
genesis-supervisor | Standalone Python process manager — RootSupervisor + ProcessLifecycleActor auto-restart genesis_producer on crash with a sliding-window rate limit (max 3 restarts/60s); coordinates shutdown ordering |
Data Flows
| Connection | Carries | Transport |
|---|---|---|
| User → GenesisSession | Drag deltas (camera orbit), joint slider values, gamepad axes | SwiftUI gestures / DragGesture · GameControllerBridge tick, throttled to 30 Hz |
| GenesisSession → ControlPlaneClient | SET_CAMERA (azimuth, elevation, distance, lookAt), SET_DOF_POSITION (positions[]) | NDJSON over NWConnection, camelCase fields |
| ControlServer → Genesis robot | Target joint positions for full DOF array (0–8 for Franka) | robot.control_dofs_position(np.array(msg["positions"]), dofs_idx_local=...) — Genesis PD position control |
| Genesis camera → SharedFrameRing | RGBA8 frame, row-major, row-0 = top | cam.render() → mmap slot write → FRAME_AVAILABLE notify (slotIndex, frameIndex, timestampNs) |
| SharedFrameRing → ViewportBridge | FramePayload (width, height, bytesPerRow, pixels) | Lockless mmap read of latest completed slot, copied into Data |
| ViewportBridge → MetalViewportRenderer | Uploaded texture + axes overlay lines matching current camera | MTKViewDelegate draw call, inline MSL shaders |
| ControlServer → GenesisSession (Track 7, in progress) | STATE: frameIndex, simTime, running, jointPositions[], jointVelocities[] | Broadcast co-emitted with FRAME_AVAILABLE after every sim step; closes the open-loop gap between commanded and actual joint state |
{index, name, lower, upper}) are discovered from the live Genesis robot entity at connect time and sent in HELLO_ACK, rather than hard-coded per model. The Franka/Ant validation pass exists specifically to prove the joint-control UI and wire protocol generalize across MJCF models instead of assuming a 9-DOF arm.