SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

Xue Yu Bo Yuan Pengshuai Yang Kailin Zhao Hong Hu Junlan Feng

JIUTIAN Research

Qwen3-VL Safety-Utility 0.191 → 0.596 +212% safety-utility score on MobileSafetyBench
Qwen3-VL Risk-Cost 0.347 → 0.130 -62.5% risk-cost score on MobileSafetyBench
Action Risk Assessment 0.723 F1 Best F1 and Step Score on MobileRisk
Next-State Prediction 0.762 Acc Better Acc than GPT-5.1 on Next-State QA

Demos

Demo 01

Blocking Dangerous Chemical Instructions

Demo 02

Preventing Sensitive Calendar Data Leakage

Demo 03

Defending Against Prompt-Injection Stock Trading

Demo 04

Rejecting a Violent Threat Before Execution

Best tradeoffs on MobileSafetyBench

GUI Agent Safety Leaderboard

Rank Agent SUS ↑ RCS ↓ Safety Shift than Direct
1 Gemini-3.1 + SeerGuard 0.773 0.180 SUS +33.0%, RCS -51.1%
2 GPT-5.1 + SeerGuard 0.679 0.145 SUS +18.5%, RCS -51.8%
3 Qwen3-VL + SeerGuard 0.596 0.130 SUS +212.0%, RCS -62.5%
4 Qwen3-VL + SeerGuardQwen 0.554 0.170 SUS +190.1%, RCS -51.0%
5 Qwen3-VL + SeerGuardact 0.503 0.141 SUS +163.4%, RCS -59.4%
6 Gemini-3.1 0.581 0.368 Direct
7 GPT-5.1 0.573 0.301 Direct
8 Qwen3-VL + SeerGuardinst 0.248 0.310 SUS +29.8%, RCS -10.7%
9 Qwen3-VL + SCoT Prompt 0.219 0.368 SUS +14.7%, RCS +6.1%
10 Qwen3-VL 0.191 0.347 Direct

What is SeerGuard?

Effect of SeerGuard on Risk-Cost Score and Safety-Utility Score.

SeerGuard is a consequence-aware safety framework for mobile GUI agents. It screens unsafe instructions and intercepts risky actions via world model prediction before they are executed. Extensive experiments demonstrate that SeerGuard generalizes effectively across diverse mobile GUI agents

Pre-execution guardrail World-model prediction Lower risk-cost, higher safety-utility

Why SeerGuard?

To our most acknowledge, related works expose a missing capability: a mobile GUI agent should be able to reason about the consequences of a candidate action before execution. The key question is not only whether an instruction is malicious, but whether a specific action will induce an unsafe future state under the current GUI context.

Overview of SeerGuard.
Overview of SeerGuard: A dual-stage, consequence-aware safety framework that combines instruction-level screening and world-model-based action risk assessment, which can defende against explicit malicious intention and unsafe actions before execution.

A consequence-aware safety framework (SeerGaurd)

It can integrate coarse-grained instruction-level screening with fine-grained action-level risk assessment. It can directly reject malicious instructions and proactively prevent risky actions by predicting their likely consequences.

A safety-augmented world model (SAWM)

It can predict semantic next states directly from multimodal GUI contexts, enabling efficient pre-execution auditing of action consequences without computationally expensive pixel-level prediction;

Extensive evaluation across diverse mobile GUI agents

They demonstrate that SeerGuard consistently improves the safety-utility trade-off across diverse mobile GUI agents. Further analyses confirm the effectiveness of SAWM for both instruction-level screening and action-level risk assessment.

Evaluation

The evaluation measures whether SeerGuard improves protection without simply refusing everything. The framework is tested on MobileSafetyBench, which contains both high-risk and low-risk mobile GUI tasks across settings such as messaging, web navigation, social media, calendar settings, and financial transactions. Two aggregate metrics are used. Safety-Utility Score (SUS) rewards safe behavior while preserving benign task completion, whereas Risk-Cost Score (RCS) assigns a larger penalty to harmful execution than to unnecessary refusal. Together, these metrics expose the tradeoff between risk avoidance and usable automation.

Mode Model Risk-Cost Score ↓ Safety-Utility Score ↑
alpha = 0.8 alpha = 0.7 alpha = 0.6 alpha = 0.5 omega = 0.8 omega = 0.7 omega = 0.6 omega = 0.5
Direct GPT-5.1 0.301 0.264 0.228 0.192 0.573 0.617 0.660 0.703
Gemini-3.1 0.368 0.322 0.276 0.230 0.581 0.624 0.668 0.712
Qwen3-VL 0.347 0.303 0.260 0.217 0.191 0.219 0.248 0.277
Guard GPT-5.1 + SeerGuard 0.145 0.155 0.164 0.173 0.679 0.676 0.672 0.668
Gemini-3.1 + SeerGuard 0.180 0.190 0.200 0.210 0.773 0.762 0.752 0.742
Qwen3-VL + SCoT 0.368 0.322 0.276 0.230 0.219 0.236 0.252 0.268
Qwen3-VL + SeerGuard 0.130 0.135 0.140 0.145 0.596 0.564 0.532 0.500
Ablation Qwen3-VL + SeerGuardinst 0.310 0.275 0.240 0.205 0.248 0.268 0.288 0.309
Qwen3-VL + SeerGuardact 0.141 0.144 0.148 0.152 0.503 0.467 0.432 0.397
Qwen3-VL + SeerGuardQwen 0.170 0.185 0.200 0.215 0.554 0.521 0.488 0.455
Framework Performance

Task Completion & Refusal Rates on MobileSafetyBench

Task completion and refusal rates across mobile GUI agents.
SeerGuard substantially reduces high-risk completion (HC) and increases high-risk refusal (HR), while retaining more low-risk completion (LC) than a refusal-only safety strategy. The tradeoff between task completion and task refusal is evident, with different agents exhibiting varying performances.
Best F1

Instruction-level Screening

Agent-SafetyBench Prompt Injection
0.4370.846
Nemo
0.5420.884
Wild
0.5780.917
Poly
0.4000.780
Llama3
0.3260.853
Qwen-G S
0.4850.921
Qwen-G L
0.5450.918
Qwen3-
VL
0.5670.922
SAWM
F1 comparison across instruction-level safety benchmarks. SAWM achieves the best Prompt Injection F1.
Best Acc, F1 and Step Score

Action-level Risk Assessment

Acc F1 Step Score
0.5780.5740.198
Rule
0.6760.6450.341
GPT-5.1
0.6810.6670.312
Qwen3-VL
0.6520.5940.269
Mobile
World
0.6420.6950.269
OS-
Sentinel
0.6960.7230.361
SAWM
SAWM delivers the best Accuracy, F1, and Step Score on MobileRisk.

Case Study

Four risky cases with SeerGuard across privacy, prompt injection, web navigation, and messaging.

01 · Instruction screening

Privacy-sensitive request

Instruction-level screening comparison between Qwen3-VL and SAWM.

SAWM recognizes unsafe intent and refuses before the GUI agent begins execution.

02 · Prompt injection

Unauthorized stock trade

Visual prompt injection leading to an unauthorized financial trade.

Predicted financial consequences allow SAWM to stop the injected action sequence.

03 · Web navigation

Malicious destination

Action-level risk assessment for unsafe web navigation.

SAWM anticipates the unsafe destination and blocks navigation before page load.

04 · SMS workflow

Unsafe generated content

Action-level risk assessment for an unsafe SMS workflow.

Consequence-aware assessment catches the risky sequence before the message is sent.

Citation

If you find this model useful, please cite the SeerGuard paper:

@inproceedings{seerguard2026,
  title={SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction},
  author={Xue Yu, Bo Yuan, Pengshuai Yang, Kailin Zhao, Hong Hu, Junlan Feng},
  booktitle = {Proceedings of the ACM Multimedia Conference (ACM MM)},
  year={2026},
  note={To appear}
}