Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation

Sunjae Lee1, Junyoung Choi1, Jungjae Lee1, Munim Hasan Wasi1, Hujun Choi1, Steve Ko2, Sangeun Oh3, Insik Shin1,
1 Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea  2 Simon Fraser University, Canada   3 Ajou University, Republic of Korea  
Paper Code arXiv
MY ALT TEXT

System Overview and Workflow of MobileGPT.

Abstract

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app—explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task’s procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 160 user instructions across 8 widely used mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.5% accuracy, and is able to adapt them to different contexts with near perfect (98.75%) accuracy while reducing both latency and cost by 62.5% and 68.8%, respectively, compared to the GPT-4 powered baseline.

Demo Video

Gmail: Schedule an email for tomorrow
afternoon to Alex

Telegram: Send a message to Steve
saying this is Alex

Book hotel in Tokyo from November 24
to November 26

Key Experiment Results

MobileGPT makes task automatoin faster and cheaper by remembering its past execution and adapting to new context.

MobileGPT's human-in-the loop repair system enables users to interact intuitively with the autonomous agent, allowing them to repair and collaboratively build upon the task automation process.

BibTeX


      @misc{lee2024explore,
            title={Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation}, 
            author={Sunjae Lee and Junyoung Choi and Jungjae Lee and Munim Hasan Wasi and Hojun Choi and Steven Y. Ko and Sangeun Oh and Insik Shin},
            year={2024},
            eprint={2312.03003},
            archivePrefix={arXiv},
            primaryClass={cs.HC}
      }