Lychee logo KnowAct-GUIClaw

KnowAct-GUIClaw: Know Deeply, Act Perfectly

Lychee Team, RICI, Harbin Institute of Technology, Shenzhen

Personal GUI assistant with self-evolving memory and skill for long-horizon, cross-application GUI workflows.

View demos Evaluation Technical Report Hugging Face Code

Abstract

Know deeply,Act perfectly.

KnowAct-GUIClaw is a Know-Route-Act-Reflect framework for personal GUI assistants. It turns accumulated interaction experience into better task decomposition, cross-platform GUI execution, and self-evolving memory and skills. Guided by the Know Deeply, Act Perfectly paradigm, the system connects what the assistant knows with how it acts across Android, iOS, HarmonyOS, and Windows.

Extensive cross-platform experiments validate our model's leading UI manipulation precision and cross-system generalization. GUIClaw + open-source Kimi-2.6 achieves a state-of-the-art 64.1% on the long-horizon MobileWorld benchmark, outperforming all open agent frameworks and closed agents (Seed-2.0-Pro, GPT-5.5). The framework's knowledge memory and execution capabilities generalize to various base models: +8.5% on Kimi-2.6 and +16.2% on Qwen3.5-35B-A3B.

Architecture

Know-Route-Act-Reflect in one loop.

The framework routes long-horizon tasks from high-level intent to executable GUI actions, then feeds execution traces and feedback back into memory and skills.

KnowAct-GUIClaw Know-Route-Act-Reflect architecture diagram
Figure 1: Overview of the KnowAct-GUIClaw execution loop. Two persistent stores - a memory and history store and a skill and shortcut store - supply advisory context to every stage. Know gathers evidence and assembles a reasoning context; Route ranks app candidates and turns the request into either a single GUI task or an ordered multi-app workflow whose subtasks exchange typed values through a blackboard; Act runs GUIClaw's observe-reason-act loop over the hybrid action space of GUI primitives, skills, deeplink/intent shortcuts, and intervention actions; and Reflect distills each trajectory into updated skills and experience memory that feed back into the stores.

Evaluation

Designed for efficient long-horizon GUI workflows.

The leaderboard summarizes MobileWorld GUI-only success rates. The same values appear in Table 1 of the technical report; the bar chart is used here for faster visual comparison on the project page.

Figure 2: MobileWorld GUI-only leaderboard bar chart
Figure 2: Success rate comparison on the MobileWorld GUI-only benchmark. Bar statistics are aggregated from Table 1, supplemented with additional Kimi-based evaluations of KnowAct-GUIClaw. Gray bars represent dedicated GUI agents, colored bars correspond to general-purpose foundation model families, and highlighted bars mark KnowAct-GUIClaw variants equipped with memory modules and reusable skill libraries.

Case Study

Multi-task tool and GUI collaboration.

A representative case shows how KnowAct-GUIClaw combines tool use, GUI interaction, and task progress tracking in a unified execution flow.

English case study for multi-task tool and GUI collaboration
Figure 3: Host-mediated recovery on CheckConferenceLocationTask. The first GUI task reads the MCFT email but returns only the hotel name, which is insufficient for a later SMS and Maps query that require a concrete address. The host therefore uses web search to resolve the full address, then starts two downstream GUI tasks: Messages sends the address to Tom, and Maps searches walking directions from the MIT Stata Center to that address and reads the 43-minute walking time. The trace illustrates how the host can combine partial GUI evidence with external tools while preserving the typed-output contract between subtasks.

Demos

Interactive demos.

Citation

Cite the technical report.

@misc{li2026knowactguiclawknowdeeplyact,
  title={KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill},
  author={Yunxin Li and Jinchao Li and Shibo Su and Zhenran Xu and Chenrui Zhao and Tongshu Bian and Xiaoman Liang and Meishan Zhang and Baotian Hu and Min Zhang},
  year={2026},
  eprint={2607.12625},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2607.12625},
}