Closing the Human–AI LoopModular Architectures for Autonomous Teaming
and Physiological Sensing

A two-session hands-on tutorial: MOSAIC and SHaSTAIEEE SMC 2026 · Bellevue, WA

Organizers: Hemanth Manjunatha · Ehsan Esfahani · Elahe Oveisi
Kristian Dalland · Bennett Kwaku Dogbey

iHuman Lab · Oklahoma State University
Human in the Loop Systems Lab (HILS Lab) · University at Buffalo

October 4, 2026

Today’s Sessions

Two hands-on sessions, back to back, in the same room.

8:30–10:30 AM Session 1 · MOSAIC

A configurable testbed for human–AI teaming. Play a search-and-rescue mission in your browser, then change the task, the AI teammate and the sensing.

11:00 AM–1:00 PM Session 2 · SHaSTA

A simulator for human–swarm teaming. Put a human in the loop with a swarm, and log what they do on one clock.

Evergreen C Where

@Hyatt_Meeting Wi-Fi network

SMC26 Wi-Fi password

Welcome. This is the tutorial “Closing the Human-AI Loop: Modular Architectures for Autonomous Teaming and Physiological Sensing”, in two sessions in Evergreen C. Session 1, 8:30 to 10:30, is MOSAIC; Session 2, 11:00 to 1:00, is SHaSTA. Wi-Fi: network @Hyatt_Meeting, password SMC26. Leave this slide up while people settle and connect.

Session 1 · 8:30–10:30 AM

MOSAIC
Modular System for Adaptive Human–AI Collaboration

A configurable testbed for human–AI teaming · github.com/iHuman-Lab/mosaic

Meet the iHuman Lab

We study how people and intelligent systems work together.

Hemanth Manjunatha Principal Investigator

Elahe Oveisi PhD Candidate

Bennett Kwaku Dogbey PhD Student

Our work spans brain–computer interfaces, human–robot interaction, human-AI interaction, and cognitive augmentation.

iHuman Lab · Oklahoma State University

Studying Human–AI Teaming

When an AI teammate advises people in dynamic, high-stakes tasks, researchers ask questions like these:

Trust and reliance

When do people follow AI advice, and when is that reliance appropriate?

Decisions under pressure

How do uncertainty and time constraints affect reliance?

Situation awareness

What happens when the person and AI have access to different information?

Cognitive state

How does AI advice affect attention and workload?

AI teammate design

Which teammate behaviors and communication strategies improve team performance?

Adaptive assistance

When should assistance change with task demands or human performance?

Answering these questions requires observing the interaction as it unfolds over time.

These are the field’s core questions. Each one plays out during the interaction, as decisions, advice, and consequences build on each other.

Why Do We Need a Configurable Testbed?

That takes four capabilities in one experiment. Existing tools rarely offer all four together.

01

Interactive decision making

People make a sequence of decisions, and each outcome changes what happens next.

03

Configurable task conditions

Researchers need to vary information access, uncertainty, time pressure, and the consequences of mistakes.

02

A customizable AI teammate

Researchers need to vary what the AI knows, how reliable it is, and when and how it communicates.

04

Synchronized data

Task events, human actions, AI advice, physiological signals, and sensor measurements must share a common timeline.

To study the interaction over time, researchers need an interactive task, a customizable teammate, conditions they can vary, and synchronized data. The next slide shows how MOSAIC brings them together.

What Is MOSAIC?

MOSAIC is a modular testbed for human–AI teaming research. Configure its components for each study.

01

Interactive task

Search and rescue, one decision at a time

03

Configurable conditions

Map, view, timing, and scoring per study

MOSAIC

modular · configurable

02

Customizable AI teammate

Scripted or LLM, with behavior you set

04

Synchronized data

Actions, advice, and signals on one clock

MOSAIC lets researchers vary the task and AI teammate, then record the interaction.

MOSAIC stands for Modular System for Adaptive Human–AI Collaboration: an open-source Python platform for studying human–AI teaming through configurable tasks, built on MiniGrid and extending its BabyAI level generator. Like a mosaic built from tiles, it assembles separate parts (task, teammate, interface, sensing) into one study. Each box answers the need in the same position on the last slide: the task, a teammate you can customize, conditions you set per study, and one synchronized record. Modular: the environment, AI teammate, interface, sensing, and analytics are separate components, so you replace one without touching the others. Configurable: map, hazards, view, scoring, and the teammate are set per study, not hard-coded. Search and rescue is the first testbed; the same pieces can support other collaborative tasks (ihuman-lab.github.io/mosaic).

How MOSAIC Works

Each part is a separate module: the participant acts on the task, the AI teammate advises from task context, and LSL aligns every stream into one session record.

Human participant

Observes, decides, and acts.

actions

observations

MOSAIC task

Runs the task and the human-facing interface.

task context

advice

AI teammate

Interprets task context and provides advice.

physiological signals

task and interaction stream

Study sensors

Physiological data: EEG, eye tracking, pupil diameter

Time-stamped stream

Lab Streaming Layer (LSL)

Aligns streams on a shared clock

Synchronized record

Session data

Ready for analysis and replay

Here is MOSAIC. You’re the participant on the left. You act, the task updates, and when you ask, the teammate reads the task context and answers. Everything the researcher varies sits in configuration, and everything that happens, including gaze and physiology, lands on one clock. Underneath, the reusable parts live in the mosaic package; the notebook you will use today is built on it, and the code map at the end shows where each part lives.

The Task: Search and Rescue

Mission

Search a multiroom building and rescue the real victims.

Real victims and decoys look alike, so an AI teammate’s advice can help or mislead.

Part Three plays the mission.

Search and rescue provides a controlled but consequential setting for studying human–AI decisions.

The Human–AI Teaming Loop

At every decision, the participant can act alone or ask for advice.

next decision

1 Observe

Read the room and the mission state.

2 Decide

Act alone, or ask the AI teammate.

act alone

3 Act

The participant always chooses the final action.

4 Outcome

The environment updates and the score changes.

Optional advice path

ask

use or ignore

Request advice

MOSAIC sends task context to the AI teammate.

advice

Evaluate advice

Compare it with what the participant can see.

Recorded throughout Task state · advice request · AI response · participant action · outcome

The central question is when and how AI advice changes human decisions.

Next, we open the notebook, play a mission, and change the task ourselves.

Part Two

Open the Notebook

Nothing to install: everything runs in your browser

Why Open Experience Configure

Tutorial notebook On the tutorial server: the whole tutorial, run in your browser Sign in with your password, then open 02_mosaic_human_ai.ipynb jupyter.ihuman-lab.work

Nobody installs anything today: MOSAIC, its game window and the recording tools all run on the tutorial server, and the game is drawn in the browser. Leave this slide up while people scan the code and sign in. Anyone whose laptop cannot reach the server pairs with a neighbor; the appendix has the install for a laptop.

Open the Notebook

Four steps, and nothing to install.

1 Go to the server

Scan the code on the last slide, or open jupyter.ihuman-lab.work

2 Sign in

With the password you were given

3 Open the notebook

02_mosaic_human_ai.ipynb, in the file list on the left

4 Pick the kernel

Python (smc), at the top right of the notebook

Notebook 02_mosaic_human_ai.ipynb: the whole tutorial, in eight steps

Game live_play.py shows the game in your browser and takes your keys

Settings config.yaml: the task as a text file

Recording lsl_tools.py: game state and gaze into one file

The top right says Python (smc). Run each cell with Shift+Enter, from the top.

Prefer your own laptop? The appendix has the install, in one slide.

Everything attendees need is already in the folder on the server: the notebook, config.yaml, live_play.py and lsl_tools.py. If someone sees “No module named ‘mosaic’”, the kernel is wrong: pick Python (smc). The first code cell of the notebook must run first (it silences warnings and sets pygame up to draw off-screen); everything after it can run in order.

Your First Level

The task is a text file. Run Step 1 and MOSAIC builds it.

config.yaml the game: section

game:
  num_rows: 2             # rooms down
  num_cols: 2             # rooms across
  room_size: 8            # tiles per room side
  num_real_victims: 2     # per room
  lava_per_room: 1
  locked_room_prob: 0.35  # chance a room is locked
  max_time: 5             # minutes on the clock

Step 1 one call builds the whole task

env = build_sar_env(
    screen_size=640, num_rows=2,
    num_cols=2, room_size=8,
    victim_placer=VictimPlacer(2),
    lava_placer=LavaPlacer(1),
    locked_room_placer=LockedRoomPlacer(0.35),
)
obs, info = env.reset(seed=0)

Change a number in the file, run again, and you have a different task.

Step 1 reads config.yaml and builds the level with build_sar_env. Three placers decide what goes where: victims, lava and locked doors. The seed makes a level reproducible: everyone who uses seed 0 plays the same building (the notebook seeds Python’s random too, because the placers draw from it). Counts are per room, so a bigger building has more of everything and a longer clock.

First Checkpoint

Confirm

The kernel says Python (smc)

Step 1 printed game config: and mission: pick up all 8 victims

No red error text under any cell

You can scroll on to Step 2

Symptom	Recovery
No module named 'mosaic'	Wrong kernel: pick Python (smc) at the top right
No module named 'live_play'	live_play.py is not in the notebook’s folder
The picture does not move	Click the picture once, then press the keys
A blank picture	Run the cell again and wait a second
The page asks you to sign in again	Sign in; your work is kept
The server does not open	Ask a helper; share a neighbor’s screen meanwhile

Everyone with a level built is ready for Part Three.

This is the only checkpoint before the break. Helpers circulate during Part Three too. The most common problem is the kernel; the second is the picture not receiving keys until it is clicked.

Part Three

Experience the SAR Mission

Play one mission in the notebook and learn to read the screen

Why Open Experience Configure

The Search-and-Rescue Mission

Explore, tell real victims from decoys, and rescue them before time runs out.

Explore

Move room to room. A locked door needs the key of the same color.

Identify

Both are red T shapes. A real victim’s stem is centered; a decoy’s is off-center.

Decide

A real victim earns a point, a decoy costs one, and the clock keeps running.

Every choice has a consequence you can see and score.

Almost no code in this part: the goal is to know the mission the way a participant does. The clip is the real interface: a real victim (green flash, +1), a decoy (red flash, -1), then another real victim. In the notebook the clock only counts down: at 0:00 the timer stops and the mission keeps going (appendix slide “Time Limit”).

Game Interface

What each tile means, and where to read the mission.

Game viewport — the green flash is rescue feedback

Information panel — rescued, remaining, steps, inventory

Mission status and score — reward and the clock

Controls, always on screen

Chat panel — where AI teammate advice arrives

You the red triangle

Victim rescue with Tab or E

Decoy costs a point

Lava ends the mission

Door Space opens it

Locked door needs the key of its color

Key Tab or E picks it up

Victims and Decoys

Real victims and decoys are both red T shapes. Only real victims earn points.

REAL VICTIM DECOY

Symbols A real victim’s stem sits in the middle of the bar. A decoy’s is off-center.

Rescue Stand next to a victim, face it, and press Tab or E. The score updates at once.

Customize Part Four changes how many decoys there are, and what rescuing one costs.

+1.0 Real victim rescued

−1.0 Decoy rescued

Real victims earn points and decoys cost them. Telling them apart is where advice matters.

The reusable VictimPlacer places real victims only; decoys come from a small placer written in the notebook’s Step 5. The scores are the defaults in RescueRewards (src/mosaic/sar/actions.py): +1 for a real victim, -1 for a decoy. Point at one of each on screen.

Controls and Gameplay

Four actions. In the browser, E also rescues and Q also asks for advice.

Navigate

↑ Move forward one tile

←→ Turn to face a new direction

Open

Space Open the door you are facing

Locked doors open only while you carry the matching key.

Pick up or rescue

Tab E Take the key, or rescue the victim, you are facing

Request advice

Alt Q Ask the AI teammate

The reply appears in the chat panel. Today’s placeholder teammate answers “Currently, no commands are available.” Step 6 gives it one.

The advice clip was recorded with a teammate that reads the map. In the notebook’s Step 2 game the placeholder teammate answers “Currently, no commands are available.” Step 6 gives the game a stand-in teammate.

Play It Yourself

Open Step 2 of the notebook, the cell called Play it yourself

Run Shift+Enter, then click the picture so it receives your keys

Move Arrow keys turn and walk, Space opens a door, Tab or E rescues

Stop Press Stop to end the game and see your score

You should see something like this: the game on the left, the panels on the right.

Give everyone a few minutes of free play. The picture is the real MOSAIC window, drawn off-screen on the server and sent to the browser; the keys you press go back to it. Browsers keep Tab and Alt for themselves in some setups, so E and Q do the same jobs. Anyone whose picture does not respond clicks it once.

No Human Required

You just played it by hand. The same game runs with nobody at the keyboard.

notebook · Step 2 a random agent plays one episode

for steps in range(1, env.max_steps + 1):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        break

Run Step 2, the cell called No human needed.

Would a learning agent change this loop?

What you get

1 The Gym API reset() and step(action), the interface RL libraries expect

2 The same task Same rooms, rules and rewards a human plays

3 No window Headless and fast: an episode takes a fraction of a second

A human, an AI teammate, or an RL agent can play: one task, one interface.

MOSAIC is built on the Gymnasium (OpenAI Gym) API: env.reset() starts an episode and env.step(action) returns (observation, reward, terminated, truncated, info), with a 7-action Discrete space. So an agent with no human in the loop can run the game, and the notebook’s cell does exactly that with a random agent: no window, a whole episode in a fraction of a second. A learning agent uses the same loop, with its own policy instead of action_space.sample(). Be clear that this tutorial does not train an agent. One practical note: the observation is a richer dictionary than the declared observation_space, so an RL library that validates spaces needs a small wrapper that picks the fields the policy uses. The point for the room: the human is one kind of player, not part of the environment.

Break · 15 minutes

Take a Break

Next · Part Four: Configure MOSAIC

Fifteen minutes. Leave this slide up for the whole break. Helpers use the time to help anyone who could not sign in or whose kernel is wrong. Before starting Part Four, check that everyone can run the Step 2 game; anyone who lost their session reruns the notebook’s first cells. Part Four changes the notebook’s cells, so the game has to be working first.

Part Four

Configure MOSAIC

The notebook in four parts: SAR, GUI, LLM, Sensing

Why Open Experience Configure

Tutorial notebook Steps 1 to 8: every change in this part is a cell you run Sign in with your password, then open 02_mosaic_human_ai.ipynb jupyter.ihuman-lab.work

Runtime Architecture

Four parts you can change, built on Gymnasium, MiniGrid and BabyAI.

↑ ← → Space Tab Alt

Human participant

Watches the screen and presses keys.

keys

frames

4.2 GUI mosaic.gui

Game view, panels, chat, and feedback.

action

observation

4.1 SAR mosaic.sar

Rooms, victims, rules, and rewards.

physiological signals

context · advice

built on

4.4 Sensing lsl_tools.py

Gaze and EEG on the game’s clock, through LSL.

4.3 LLM mosaic.llm

Reads the task, replies in the chat.

obs, reward, terminated, truncated, info = env.step(action)

Gymnasium + MiniGrid

The step API and BabyAI’s room grids.

Point at the attendee’s own window: the keys they press, the GUI that draws it, the environment that decides what happens. Each numbered block is one section of Part Four. The chat image is a real reply from the stand-in teammate on “An AI Teammate Without a Key”. Sensing sits beside the game: in the notebook the gaze is synthetic, and a real tracker streams in the same way. Sensors never talk to the game directly: each one streams into Lab Streaming Layer, which puts game and sensor samples on one clock. The env.step line is src/mosaic/gui/user.py:53. MOSAIC’s levels extend BabyAI’s LevelGen, which ships inside MiniGrid (BabyAI: Chevalier-Boisvert et al., ICLR 2019): room grids, doors, keys, and missions written as instructions. The search-and-rescue task adds victims, lava, cameras and rewards on top.

One Notebook, Four Parts

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

the notebook’s cells, assembled · Steps 1 to 7

env = build_sar_env(
    screen_size=640, num_rows=2, num_cols=2, room_size=8,
    victim_placer=VictimsAndDecoys(num_real_victims=2, num_decoys=3),
    lava_placer=LavaPlacer(lava_per_room=1),
    locked_room_placer=LockedRoomPlacer(locked_room_prob=0.35),
    camera_strategy=AgentFOVCamera(),
    action=RescueAction(rewards=RescueRewards(fake_victim=-5.0)),
)
teammate = StandInTeammate()
gui = SAREnvGUI(env, config=GAME, llm_client=teammate)
gui.user.on_step = record_step
play_mosaic(gui, on_stop=finish)

4.1 SAR

The building, what fills it, the camera, the rewards

4.2 GUI

The participant’s window

4.3 LLM

Who answers Q

4.4 Sensing

What every step records

This is the whole task as a few lines, assembled from the notebook’s cells; the slide is a map, not a cell to copy. Step through the highlights, in the order of Part Four: SAR (the building, what fills it, the camera, the rewards), GUI (the window and how it is shown), LLM (who answers Q), and Sensing (what every step records). VictimsAndDecoys is a placer written in Step 5, StandInTeammate a teammate written in Step 6, and record_step and finish are Step 7.

4.1 · SAR

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

build_sar_env builds the task. Notebook Steps 1 and 5.

notebook · Step 5 inside make_level

build_sar_env(
    screen_size=640, num_rows=rows, num_cols=cols,
    room_size=room_size,
    victim_placer=VictimsAndDecoys(real, decoys),
    lava_placer=LavaPlacer(lava_per_room=lava),
    locked_room_placer=LockedRoomPlacer(locked),
    camera_strategy=camera,
    action=RescueAction(rewards=RescueRewards(
        fake_victim=decoy_cost)),
)

Three things to change

1 The building Notebook

num_rows · room_size · decoys · lava · locks

How big, crowded, and locked

2 Camera Next

camera_strategy=

What the participant sees

3 Rewards Try it

action=RescueAction(...)

What a mistake costs

Everything on the left is one call. Counts are per room, so a bigger building has more victims, more lava and a longer clock. The next two slides are the notebook’s Step 5: what a level can contain, and how little code a new placer takes. Then the camera and the rewards. The code lives in src/mosaic/sar/ and src/mosaic/core/.

What You Can Customize

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Step 5. Five levels, each changing one setting of the default, drawn with the whole-building camera.

Rooms, decoys, lava and locked doors are numbers you choose.

Real victims, decoys and lava are counted per room, so the totals grow with the building, and so does max_steps, the number of steps the game allows. Decoys are look-alike victims: rescuing one costs points. Locked rooms hide their key in another room. The next slide shows where the decoys come from.

Write Your Own Placer

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Try it MOSAIC’s victim placer has no decoys. Add them with a small class of your own.

notebook · Step 5 abridged

class VictimsAndDecoys(VictimPlacer):
    def __init__(self, num_real_victims=2, num_decoys=0):
        super().__init__(num_real_victims)
        self.num_decoys = num_decoys
    def place_all(self, level_gen, rows, cols):
        super().place_all(level_gen, rows, cols)
        for i in range(rows):
            for j in range(cols):
                room = level_gen.get_room(i, j)
                for _ in range(self.num_decoys):
                    x, y = random.choice(free_cells(room))
                    decoy = FakeVictim("left", "up")
                    level_gen.put_obj(decoy, x, y)

Run Change DECOYS in the build-your-own cell and play.

Can you tell the decoys from the victims?

What it does

1 Real victims first The normal placer does its job

2 Then decoys Look-alikes in free cells of every room, never next to a door

3 One method A placer only has to implement place_all

The slide shortens the cell: free_cells(room) stands for the base class’s helpers that list free cells away from doors, and the notebook picks the decoy’s shape at random. The point is how little a placer is. Placer in src/mosaic/core/placers.py has one method, place_all, so a new kind of level content is a short class. The lava placer and the locked-room placer are the same shape.

What the Participant Sees

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

camera_strategy= sets the camera. One walk through a door, three cameras:

Edge-following

EdgeFollowCamera · scrolls with the agent

Whole room

AgentFOVCamera · the whole room

Forward cone

AgentConeCamera · only what is ahead

Agent same tile in all three   ·   Room the agent’s room

Same task. The camera sets what the participant can see.

Step 3 of the notebook draws the same level through five cameras, with env.switch_camera(). To use one in your own level, import the class from mosaic.core.camera and pass it as camera_strategy=. Cameras are a strategy interface: CameraStrategy in src/mosaic/core/camera.py returns an image from get_crop(). Use FullviewCamera and AgentCenteredCamera only with switch_camera: as the camera a level is built with they raise AttributeError on env reset, because they have no reset() method.

Rescue Rewards

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Try it Step 5: make rescuing a decoy cost 5 points instead of 1.

notebook · Step 5 the build-your-own cell, then make_level

# a real victim is +1, a decoy -1 by default
DECOY_COST = -5.0

# inside make_level(...)
action=RescueAction(rewards=RescueRewards(fake_victim=decoy_cost))

Run Play, rescue a decoy, then press Stop.

What will the reward read?

Before · the default

After · fake_victim=-5.0

action= goes inside the build_sar_env(...) call. The defaults in RescueRewards (src/mosaic/sar/actions.py) are +1 for a real victim and -1 for a decoy. Every rescue also reports an event in info["events"]. The score under the game when you press Stop includes what the decoys cost.

The Challenge: Beat the Room

Same level, two minutes, one leaderboard.

notebook · Step 9 before you play, and when you stop

PREDICTION = 3       # your guess: set it BEFORE you play

# when the game ends, your score is saved:
submit_score("mosaic", NICKNAME, score, steps=steps,
             prediction=PREDICTION)

Run Step 9: set PREDICTION, run the cell, and play.

What score will you get?

What you will see

1 Your result Your score, and how close your guess was

2 The agent A random agent plays the same level 50 times: where do you rank?

3 The board Everyone’s scores, and who predicted their own best

A real victim is +1 and a decoy is −5: telling them apart is the whole game.

Everyone plays the same level (a fixed seed, so the buildings, victims, decoys and lava are identical), for two minutes. The score is the reward: +1 for each real victim, −5 for each decoy. The game ends at Stop, when the two minutes are up, or when the episode ends by itself. Before they play, everyone sets PREDICTION to the score they expect: commit first, then see. Afterwards the notebook shows how far off they were and how a random agent does on the same level. Put 03_leaderboard.ipynb on the projector: it refreshes the board every few seconds, ranks by score with fewer steps breaking a tie, and lists the best forecasters. Only each person’s best score is kept, so they can play again.

4.2 · GUI

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

SAREnvGUI is the participant’s window. Notebook Step 4.

notebook · Step 4 the window, then the game

gui = SAREnvGUI(env, config=GAME)
play_mosaic(gui)

Three things to change

1 Time limit Appendix

max_time: in config.yaml

How much time pressure

2 Nudges Step 6

llm_nudge_interval

When the teammate speaks unasked

3 Browser view Notebook

play_mosaic(gui, width=…)

How large the picture is

The GUI lives in src/mosaic/gui/. SAREnvGUI takes the env and a config dict, and optionally a teammate, a prompt builder, and replacements for the panels and the feedback flash. In the notebook the window is drawn off-screen and shown as a picture; play_mosaic (in live_play.py) runs the real game loop and passes your keys to it, and its width, fps and quality settings control the picture.

4.3 · LLM

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

A teammate is any object with one method, query. Notebook Step 6. Today it is a placeholder.

notebook · Step 6 the contract

class StandInTeammate(LLMClient):
    def query(self, prompt):
        return "<START> Head East. <END>"

Today, after Q: the placeholder

Three things to change

1 The teammate Try it · next

llm_client=StandInTeammate()

Who answers Q

2 Prompt style Try it

prompt_type: "sparse" | "detailed"

How long the answers are

3 Nudges Notebook

llm_nudge_interval

When it speaks unasked

The contract lives in src/mosaic/llm/. Any teammate implements one method of LLMClient, query(prompt) -> str: a hosted model, a script, or a person. MOSAIC turns the observation into a prompt, calls query, cleans the reply, and shows it in the chat panel. llm_nudge_interval: N makes the teammate speak every N steps without being asked. The notebook uses a stand-in, so no API key is needed.

An AI Teammate Without a Key

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Try it Swap in a stand-in teammate that answers in the style its prompt asks for.

notebook · Step 6 the last cell

-play_mosaic(SAREnvGUI(env, config=GAME))
+play_mosaic(SAREnvGUI(env, config=GAME,
+                      llm_client=StandInTeammate()))

Run Press Q in the game.

Does it know where you are?

Before · the placeholder

After · StandInTeammate

Be clear with the room: the stand-in does not read the level. It looks at its prompt and answers in that style, the way a real model would be asked to, with a fixed sentence. What it shows is the whole path: MOSAIC builds the prompt, the teammate answers, the reply is cleaned (the START and END markers the prompt asks for are removed) and appears in the chat. A real model replaces the stand-in by implementing the same method, query.

Two Prompt Styles

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Try it Switch the prompt type, press Q again, and compare the answers.

notebook · Step 6 the last cell

PROMPT_TYPE = "sparse"            # or "detailed"

config={**GAME, "prompt_type": PROMPT_TYPE}

sparse One short radio sentence, at most 20 words

detailed A short briefing, 3 to 4 sentences, at most 40 words

How would each change what a participant does?

sparse

detailed

The two prompts describe the level in the same way and differ only in how the teammate should answer: one sentence, or a short briefing. Step 6 of the notebook prints those answer rules for each. The prompts live in src/mosaic/llm/prompts.yaml. Ask the room which style they would rather get under time pressure, and what the longer answer costs them.

4.4 · Sensing

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Each step produces an observation. Step 3 shows it as you play; Step 7 records it with gaze on one clock.

notebook · Step 3 play with a live readout

player = User(make_env())
play_view(player)

obs → agent_x, agent_y, agent_dir, grid,
      saved_victims, remaining_victims,
      victim_health, step_count, …
    + action, reward
    → Step 7: one marker per action on LSL

Three things to change

1 Observation Step 3

player.obs

What each step holds

2 Recording Step 7

user.on_step = record_step

What each action writes

3 Sensors Demo · next

eye_tracker: source: live

Gaze on the game’s clock

Observations are built in src/mosaic/sar/observations.py. In Step 3 the box beside the game shows them updating after every action you take. In Step 7 the game calls a function after each action (user.on_step), and that function writes the state and any rescue to an LSL marker stream; a second stream carries the gaze, and one recorder writes both to a single XDF file. In the notebook the gaze is synthetic, so no hardware is needed; the next slide uses a real tracker. Change what each step records by subclassing ObservationProcessor and passing it as observation=.

Live Demo: Eye Tracking

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Presenter demo A Tobii eye tracker streams gaze into the same recording, while a volunteer plays Step 7.

tobii_to_lsl.py on the laptop the tracker is plugged into

python tobii_to_lsl.py --width 1920 --height 1080

config.yaml the eye_tracker: section

 eye_tracker:
-  source: synthetic
+  source: live

What lands next to the game data?

Watch for

1 Stream The gaze appears as TobiiEyeTracker, 60 Hz

2 Play A short mission in Step 7

3 Two streams, one clock TobiiEyeTracker (gaze) and MOSAIC-State (one marker per action), side by side in the recording

Presenter only. The tracker is plugged into one laptop, and LSL streams are found on the local network, so this demo runs the notebook on that laptop (a local Jupyter), not on the server. Setup is in EYETRACKER_SETUP.md. On the day: start tobii_to_lsl.py, set eye_tracker: source: live in config.yaml, run Step 7, let the volunteer play, press Stop, then run Step 8 on the file it saved. The script does not calibrate; calibrate in Tobii Eye Tracker Manager first. Seat the volunteer 60 to 65 cm from the screen. If the device fails, go to the next slide, which shows a rehearsal recording.

The Synchronized Record

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

One recorded mission gaze and game on the same clock

What to look for

Where Which part of the screen holds the gaze?

When What do the eyes do in the second after a rescue?

Advice Where does gaze go when the teammate replies?

Gaze and game share one clock, so every look lines up with what happened.

Step 8 of the notebook draws figures like this from the file Step 7 saves, with the same helpers (lsl_tools.py). The figure on the slide is from a rehearsal recording of a real tracker. Top: each circle is a fixation, bigger for longer, on the screen’s three areas. Bottom: one clock, the victims saved and under it the area the gaze was in; dark gaps are moments the tracker lost the eyes. In this recording the gaze stayed on the game view (99%) and did not go to the info panel after any of the three rescues. The three rows are the questions a researcher asks of a record like this. The third needs a teammate that gives advice during the mission, which this recording does not have. If the tracker fails, this slide is the fallback. After the venue rehearsal, redraw it with python tools/make_gaze_figure.py <recording.xdf> and render again.

Where to Go Next

The parts you changed today, and where they live.

notebooks/ The tutorial notebook and its helpers. Copy it to start yours.

02_mosaic_human_ai.ipynb

config.yaml

live_play.py

lsl_tools.py

tobii_to_lsl.py

API play_mosaic · play_live · MarkerOutlet

composes

src/mosaic/core/ 4.1

Level generation and cameras

level.py

placers.py

camera.py

API Placer · CameraStrategy

src/mosaic/sar/ 4.1 4.4

The search-and-rescue task

env.py

placers.py

actions.py

objects.py

observations.py

API build_sar_env · RescueAction

src/mosaic/gui/ 4.2

What the participant sees

main.py

user.py

chat.py

info.py

feedback.py

API SAREnvGUI · EdgeVignette

src/mosaic/llm/ 4.3

The teammate contract

client.py

process_prompts.py

prompts.yaml

pathfinding.py

parser.py

API LLMClient · ask

Have the repository open in an editor beside this slide. mosaic/ is the reusable package. The notebook folder is the part attendees copy and change for their own work: its helpers play the game in a browser and record it. The badges match the four parts of Part Four. Every file named here exists in the MOSAIC repository. To run all of this on a laptop instead of the server, see the first appendix slide.

MOSAIC in a Study

What you configured today, put to work in a study of people and an LLM teammate.

LLM-Mediated Human-AI Interaction in Search and Rescue: Impact of Expertise on Attentional Allocation Elahe Oveisi · Hemanth Manjunatha

Task The search-and-rescue mission, with victims whose health runs down.

Teammate Two LLM teammates and a no-LLM baseline, three trials per participant.

Sensing A Tobii eye tracker and the game on one clock, through LSL.

Question Does guidance change where people look, and does expertise matter?

Monday, October 5 · 14:00–14:15 Grand C · Session MoA10 Scan for the talk’s slides

Come and hear what they found: Monday at 14:00, Grand C.

An invitation, and an application of what attendees just configured: say what the study did, and leave the results for the talk. Paper MoA10.3 in the regular session Decision Support System I, Monday October 5, 14:00 to 14:15, room Grand C. Thirteen participants each played three trials in random order: two with an LLM teammate and one no-LLM baseline, up to 15 minutes or 1,000 moves. Victim health deteriorates, and Alt asks for advice and reveals victim health for five seconds. Gaze and pupil data came from a Tobii eye tracker, on one clock with the game through LSL: the same pieces as Part Four. Participants were grouped into experts and novices. The code opens the talk’s slides (ihuman-lab.github.io/presentations/ieee-smc-eo/); the program entry is conf.papercept.net, SMC26 program, Monday, MoA10.3. Check the time and room against the final program on the day.

Open Discussion

1 Human–AI Teaming What human–AI teaming question would you ask first, and what would you change in the rooms, victims or rewards to ask it?

2 An LLM as teammate What should it see, what should it be allowed to do, and how would you tell a good teammate from a fluent one?

3 Sense the human Which physiological signals, and which game events, would you use to close the loop?

Open on purpose: the three questions are prompts, not an agenda, so let the room choose. Give two minutes in pairs first, then take whatever comes. The third question reaches across to the SHaSTA session, so people who attend only one can still join. If it goes quiet, ask who guessed their score best and what changed how they played. Write down what people say is missing; it becomes the issue list.

Thanks!

Questions?

MOSAIC github.com/iHuman-Lab/mosaic

Appendix

More changes and reference

Each one is a line to change in the notebook

Run MOSAIC on Your Own Laptop

Everything here runs on a laptop too, with Python 3.10 or 3.11 and Git.

• macOS / Linux
• Windows PowerShell
• Conda · Any Platform

git clone https://github.com/iHuman-Lab/mosaic.git
cd mosaic
python3 -m venv mosaic_env
source mosaic_env/bin/activate

git clone https://github.com/iHuman-Lab/mosaic.git
cd mosaic
python -m venv mosaic_env
.\mosaic_env\Scripts\Activate.ps1

git clone https://github.com/iHuman-Lab/mosaic.git
cd mosaic
conda create -n mosaic_env python=3.11 -y
conda activate mosaic_env

python -m pip install --upgrade pip
python -m pip install -e .
python -m pip uninstall -y pygame
python -m pip install --force-reinstall "pygame-ce>=2.5.2"
python -m pip install matplotlib scipy pandas pylsl pyxdf
python -m pip install ipywidgets ipyevents notebook
cd notebooks
python -m jupyter notebook 02_mosaic_human_ai.ipynb

1 Clone and isolate

The code, and a private Python environment

2 Install MOSAIC

-e means your edits apply without reinstalling

3 Switch to pygame-ce

The interface needs it; plain pygame hides it

4 Notebook packages

For plots, the browser game, and the recording

5 Open the notebook

Start Jupyter from the notebooks folder

The notebook asks for the kernel: choose Python 3 (ipykernel). The full guide is SETUP.md in the tutorial repository.

The two pygame lines work around a packaging gap: MOSAIC’s pyproject.toml declares pygame, but pygame_gui needs pygame-ce, and both install into the same folder. Without them the import fails with cannot import name 'DIRECTION_LTR' from 'pygame'. --force-reinstall is required. The notebook draws the game off-screen and shows it in the browser, so a laptop needs no display setup beyond a browser. A tracker demo on a laptop also needs tobii-research; see EYETRACKER_SETUP.md.

Locked Rooms

Set the chance that a room is locked: 0.5 locks about half, 0.9 most.

notebook · Step 5 make_level or config.yaml

-make_level(rows=3, cols=3, locked=0.5)
+make_level(rows=3, cols=3, locked=0.9)

Run Play the level and walk to the nearest door.

How much of the building can you still reach?

Before · 0.5: 4 locked doors

After · 0.9: 8 locked doors

Keep locked_room_prob below 1.0: at 1.0 generation never finishes. The count is max(1, int(rooms × prob)), so a 3×3 building always locks 4 rooms at 0.5 and 8 at 0.9; which rooms changes run to run. LockedRoomPlacer lives in src/mosaic/sar/placers.py; its place_all() picks which rooms lock and hides each key in an unlocked room. The notebook’s gallery in Step 5 shows 90% locked on a 2×2 building.

Time Limit

Give the mission 2 minutes instead of 5.

config.yaml the game: section

 game:
-  max_time: 5
+  max_time: 2

Run Run Step 1 again, then play and watch the timer.

How will you play with only 2 minutes?

Before · the default

After · max_time: 2

At 0:00 the timer stops; the mission keeps going.

SAREnvGUI reads max_time (minutes) from the config it is given, which is the game: section of config.yaml in the notebook (GAME); the play cells pass their own value, such as {**GAME, "max_time": 3}. On its own the timer only counts down: at 0:00 the mission keeps going.