LLM-Mediated Human–AI Interaction in Search and Rescue:
Impact of Expertise on Attentional Allocation

Human performance, visual attention, and expertise under LLM guidance

Elahe Oveisi and Dr. Hemanth Manjunatha

Oklahoma State University

2026-10-05

Human–AI Setting in Critical Situation

Operators must locate and recover people while conditions change quickly.

  • Missions occur in challenging environments
  • Decisions must be made under time pressure
  • Need to quickly process high volumes of information from multiple sources and translate it into commands that are clear and understandable for humans 1
  • LLMs provide natural-language guidance

Drones supporting search-and-rescue operations

Research Questions and Contribution

  • Performance: Does LLM guidance improve task outcomes?
  • Attention: How does guidance change visual attention and eye-tracking metrics?
  • Expertise: Does expertise moderate these effects?
  • Testbed: A controlled simulation supports repeatable human–AI teaming research.

Participant using an eye tracker while playing the search-and-rescue game

We study how humans process and respond to information from LLMs, using eye tracking to inform future adaptive LLM-based assistance.

Study Design In Brief

Search-and-rescue simulation interface with annotated game elements

13 Participants

3 Trials each (randomized)

2 + 1 LLM + baseline

15 min or 1,000 moves

Task: rescue victims efficiently in a partially observable grid while victim health deteriorates.

ALT requests advice and reveals victim health for 5 seconds.

Game Play

Expertise Level

Grid of black T-shaped symbols in various rotations
  • Expertise labeling: K-means clustering (K = 2) grouped participants into expert vs. novice 2

How The LLM Was Prompted

Guidance Style

  • You are a search-and-rescue coordinator giving calm, human-like radio guidance to the operator.
  • Instructions must be directional and action-focused, always using North, South, East, West, with no coordinates, step counts, or visible reasoning.
  • Output either a sparse one-sentence instruction or a detailed 3–4 sentence radio briefing, always concise, natural, and wrapped strictly as <START> … <END>.

Priority

  • Prioritize visible reachable survivors who are still saveable, nearest first.
  • From the current state {obs}, choose one immediate objective by prioritizing the nearest reachable visible saveable survivor, then the next saveable survivor if relevant.
  • Otherwise choose the best remaining alive victim, then the nearest reachable survivor or key, or finally explore visible areas that may open a new path.

Mission Rules

  • Follow mission and environment rules: rescue all survivors.
  • Use matching keys for locked doors, unlocked doors can be opened, carry only one key, avoid lava.
  • Never mention unreachable objects. Final instruction must be rule-compliant.
  • Grid-based world: each cell represents an object such as empty space, wall, victim, lava, door, or key. 3
  • Reachability check: BFS determines which targets are reachable and their distance from the player. 4
  • Models: OpenAI and Gemini.
Select Play to explore each part of the prompt.

What We Investigate

WHAT WE VARY
WE TEST
WHAT WE MEASURE
LLM guidance
LLM
No LLM
Expertise
Novice
Expert
➔
➔
Effect of
guidance?
by expertise?
➔
➔
Performance
Victims per step
Total reward
Saved victims
Visual behavior
Fixations & saccades
Pupil diameter
Game vs. chat attention

Task and Eye-Tracking Features

Feature What it shows
Victims per step Rescue efficiency over time
Total reward Overall task performance
Saved victims Mission success
Fixation count How often the participant focused on an area
Fixation duration How long the participant looked at one area
Saccade duration Duration of fast eye movements
Pupil diameter SD Variation in pupil size, related to workload
Game-area fixation time Attention to the SAR environment
Chat-panel fixation time Attention to the LLM guidance

Data Collection and Analysis

  • Synchronized data streams via LSL (Lab Streaming Layer)
  • Gaze and pupil data recorded with a Tobii eye tracker

Mixed-effects model: outcome = LLM condition + LLM × expertise + expertise + participant random intercept.

Participant at the game and eye-tracker setup, with an example LLM guidance chat message shown alongside

Results

Slope plots of the reported LLM, Expertise and LLM × Expertise coefficients for victims per step, total rewards and saved victims

Slope plots of the reported LLM, Expertise and LLM × Expertise coefficients for pupil-size SD, game-area fixation and chat-area fixation

Slope plots of the reported LLM, Expertise and LLM × Expertise coefficients for fixation duration, fixation count and saccade count

Expertise Changes How Guidance Is Used

Experts

Maintained more engagement with the game under LLM support and showed stronger changes in visual scanning.

Novices

Shifted more attention away from the game, consistent with greater reliance on the guidance panel.

LLM guidance received

↓

Expert

Cross-checks guidance against the environment (scans, verifies)

Novice

Follows guidance directly

Limitations and Future Work

Limitations

  • Sample size
  • Simulated grid-world task
  • Incorrect/conflicting LLM guidance was not tested
  • Additional physiological measures were not included

Future work

Trust & reliability Test how LLM errors affect operator responses and trust.

Adaptive guidance Tailor help using performance and physiological signals to reduce mental workload.

Situation awareness Use this data to infer workload, situational awareness, and human–AI interaction patterns.

Other domains Apply this approach to other human–AI teaming domains beyond search and rescue.

The next step is moving from static guidance toward reliable, adaptive human–AI teaming.

References

[1] Handler, Abram, Kai R. Larsen, and Richard Hackathorn. “Large language models present new questions for decision support.” International Journal of Information Management 79 (2024): 102811.

[2] J. P. Distefano, H. Manjunatha, S. Chowdhury, K. Dantu, D. Doermann and E. T. Esfahani, “Using Physiological Information to Classify Task Difficulty in Human-Swarm Interaction,” 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Melbourne, Australia, 2021, pp. 1198-1203, doi: 10.1109/SMC52423.2021.9658653.

[3] Chevalier-Boisvert, M., Dai, B., Towers, M., Perez-Vicente, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., & Terry, J. (2023). Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. Advances in Neural Information Processing Systems (NeurIPS), 36.

[4] Siklóssy, Laurent, A. Rich, and Vesko Marinov. “Breadth-first search: some surprising results.” Artificial Intelligence 4.1 (1973): 1-27.

Thank you

Questions?