Introduction

Common Sense Labs records how real people do obvious things and turns it into training data every robot can use. One person shows that a full mug travels upright. Every robot on the open layer stops carrying it sideways.

This page covers the whole pipeline: first-person capture from smart glasses, exocentric LiDAR grounding, cross-device sync, fusion, and the open layer where lessons live. Every code sample on this page imports a real module served from this site, /lib/commonsense.js. It has no dependencies, so you can read all of it in five minutes.

Quick start

Import the module straight from the site. The example below aligns three devices on one clock using events all of them saw.

import { sync } from "https://commonsenselabs.vercel.app/lib/commonsense.js";

const result = sync([
  { device: "glasses", events: [{ t: 1000, label: "grip" }, { t: 1420, label: "lift" }] },
  { device: "iphone",  events: [{ t: 962.3, label: "grip" }, { t: 1382.6, label: "lift" }] },
  { device: "ipad",    events: [{ t: 1021.8, label: "grip" }, { t: 1441.5, label: "lift" }] },
]);

console.log(result.drift);   // { before: 59.5, after: 0.4 }
console.log(result.locked);  // true

# grab the module and read it, it is short
curl -O https://commonsenselabs.vercel.app/lib/commonsense.js
wc -l commonsense.js

<script type="module">
  import { lesson } from "https://commonsenselabs.vercel.app/lib/commonsense.js";
  console.log(lesson("kettle", "Pour from the spout, not the lid"));
</script>

Try it

Move the sliders to add clock error to the iPhone and iPad, then run the real sync() from this site.

How it works

Every capture goes through five steps. They run on one person's devices at home and finish in the open layer.

1. Capture. Smart glasses record what the person sees and a 26-joint hand pose at 90 Hz.
2. Ground. A LiDAR phone on a shelf scans the same room from the outside, so every grip gets coordinates.
3. Sync. Devices agree on one clock using events all of them saw, like a grip or a cup touching the table.
4. Fuse. Ego and exo frames that describe the same instant are paired into one spatial record.
5. Share. The record is packed as an episode and lands in the open layer as a lesson any team can train on.

Capture

First-person capture is what the human sees and what their hands do. It is the part robots never get from simulation: the tiny pause before lifting a full pan, the second hand that steadies the lid.

Ground

The exocentric view comes from an iPhone 12 Pro or newer, or an iPad Pro, standing somewhere it can see the work. It gives the room a volume and the objects a size, so the robot learns the kettle is 24 cm tall and not just yellow.

Sync

sync(streams, opts) takes one stream per device. For each device it looks at every event it shares with the reference device and takes the median time difference as that device's offset. The median ignores the one event where somebody sneezed.

const { offsets, drift, locked, streams } = sync(input, { reference: "glasses", tolerance: 1 });
// offsets: { glasses: 0, iphone: 37.7, ipad: -21.65 }  milliseconds to add
// locked:  true when the worst remaining spread is within tolerance

Fuse

fuse(ego, exo, opts) walks both streams in time order and pairs each ego frame with the nearest exo frame. Pairs further apart than window milliseconds are dropped instead of guessed.

import { fuse } from "https://commonsenselabs.vercel.app/lib/commonsense.js";
const pairs = fuse([{ t: 0 }, { t: 11 }, { t: 22 }], [{ t: 1 }, { t: 12.5 }, { t: 40 }], { window: 8 });
// 2 pairs, dt = [1, 1.5]. The frame at 22 ms has no partner and is left out.

Share

episode(frames, meta) packs fused frames as an RLDS-style episode with is_first and is_last flags, so it loads into the same loaders your team already uses.

const ep = episode(pairs, { task: "pour tea", device: "quest3" });
// { format: "rlds/1", episode_metadata: { task, device, steps, toolkit }, steps: [...] }

Lessons

A lesson is the short human sentence behind an episode: an object and a rule. lesson(object, rule) gives it a stable id, so the same lesson captured by a thousand people lands in one place and counts up instead of duplicating.

lesson("Kettle", "Pour from the spout");
// { id: "cs_0a7450c6", object: "kettle", rule: "Pour from the spout", captures: 1, obvious: true }

API

Formats & devices

Errors

Limits

• Sync needs shared events. Two devices that never saw the same moment cannot be aligned, by us or by anyone.
• The fuse window trades coverage for honesty. Wider windows pair more frames and lie a little more about each one.
• Lessons are only as good as the person capturing them. If you pour tea into your shoe, that is also a lesson, just a different one.
• The module runs in the browser and in Node 18+. It does no network calls and keeps nothing.

$COMMON

$COMMON is the token of the commons. One billion supply, nothing minted after, zero tax, no admin key. The contract and links below are read from the main page, so they are always the same as there.

FAQ