Mobile AI is no longer just “call an LLM API from the app.”
Modern iOS and Android apps can choose between:
on-device model
cloud model
hybrid routing
and each option changes latency, privacy, cost, device support, offline behavior, and product reliability.
Apple's current Foundation Models framework exposes on-device and Private Cloud Compute models through native Swift APIs, while Android's Gemini Nano stack runs through AICore and ML Kit GenAI APIs. Android also now provides ADK support for building agents that can run locally, in the cloud, or as hybrid systems.
That means the architecture question has changed from:
> “Which AI API should the mobile app call?”
into:
> Which tasks belong on the device, which belong in the cloud, and how should the app switch between them safely?
Start with the task, not the model
A mobile feature might need:
summarization
text rewrite
image understanding
voice transcription
search
assistant / agent behavior
Each task has different requirements.
For example:
rewrite a short private message
is a strong on-device use case.
But:
research a 200-page document and reason across the web
is usually better suited to a cloud model.
The three architecture choices
On-device
app
→ local model runtime
→ device CPU/GPU/NPU
Cloud
app
→ backend / model API
→ cloud model
Hybrid
app
→ route simple/private work locally
→ route hard/large-context work to cloud
Hybrid is increasingly the most practical choice.
On-device AI: where it shines
Advantages include:
low latency
offline operation
no per-request cloud inference cost
better data locality
Android's Gemini Nano documentation explicitly positions on-device inference for use cases where privacy, low cost, and offline operation matter.
Apple similarly emphasizes native on-device models for private, low-latency intelligence.
On-device models are not universal replacements
Constraints include:
smaller model capability
limited context
RAM / storage
hardware compatibility
battery / thermal load
A local model may be excellent for short summarization but weak at deep reasoning.
The architecture should route around those limits instead of pretending they do not exist.
Apple: Foundation Models framework
Apple's Foundation Models framework gives Swift apps native access to language models for tasks such as:
summarization
entity extraction
text/image understanding
structured output
tool calling
The framework supports on-device Apple models and can also work with other providers through the LanguageModel protocol.
This is important because the application can abstract the task while changing which model executes it.
Structured generation is valuable on mobile
Instead of returning free text:
“Looks like a reminder for tomorrow at 9.”
you may want:
{
"type": "reminder",
"time": "09:00",
"date": "2026-09-18"
}
Apple supports guided structured generation through Swift data structures.
Android's current Prompt API also supports structured-output workflows.
Machine interfaces should prefer structured results.
Android: Gemini Nano through AICore
Gemini Nano runs through Android's AICore system service.
ML Kit GenAI APIs currently expose capabilities such as:
prompting
summarization
proofreading
rewriting
image description
speech recognition
Because the model runs locally on supported devices, the app can use AI without sending every input to a server.
Device support must be checked at runtime
Not every Android device has the same:
model availability
NPU / accelerator
RAM
OS version
Your UI should gracefully handle:
feature available locally
feature requires model download
feature unavailable
cloud fallback available
Do not crash or silently change behavior because the local model is missing.
Android agents are now a real option
Google's ADK for Android supports Kotlin/Java agents and can use Gemini Nano through ML Kit for local inference.
A hybrid architecture can use:
cloud root agent
→ local privacy-sensitive subagent
or the reverse, depending on the product.
That creates interesting new mobile designs, but it also raises tool-permission and lifecycle questions.
Do not let the agent bypass the app permission model
An AI agent may want access to:
camera
contacts
files
microphone
calendar
It should still go through normal OS permissions and application authorization.
The model should never create a new privilege simply because it decided the data would be useful.
Use narrow local tools
A mobile agent can expose app capabilities such as:
create_note(text)
search_local_messages(query)
set_timer(duration)
rather than unrestricted access to app storage.
This keeps agent behavior auditable and safer.
Cloud models: when they are the better choice
Use cloud inference when the task needs:
larger context
stronger reasoning
web / external tools
large multimodal input
cross-device consistency
Cloud APIs also make model updates easier because the app does not need to ship new weights.
Never put privileged cloud API keys directly in the app
Mobile apps can be reverse engineered.
Do not embed long-lived model-provider secrets inside the client binary.
A production architecture should use:
mobile app
→ authenticated backend
→ provider API
or a provider-supported secure client integration designed for mobile.
The backend can enforce quotas, user identity, and abuse controls.
Hybrid routing is usually task-based
Do not ask the small local model to perfectly decide whether it is capable.
Route using known properties:
input length
task type
privacy class
network availability
required tool
Example:
short rewrite
→ local
long research request
→ cloud
This is easier to test than confidence-based routing alone.
Offline-first behavior needs explicit product design
If the user is offline, decide which capabilities remain available.
Example:
rewrite → available
local search → available
cloud research → unavailable / queued
Do not make every AI button fail with a generic network error.
The UI should explain capability clearly.
Local RAG can power private personal features
A mobile app can index a bounded local dataset such as:
notes
saved documents
app content
Then run:
query
→ local search
→ local model
This can preserve privacy and work offline.
The difficult parts are indexing, synchronization, storage size, and deletion—not just embeddings.
Keep canonical data separate from embeddings
If a note changes:
source record updated
→ old chunks invalidated
→ local index refreshed
Do not let stale vectors survive forever.
Local AI still needs normal data lifecycle management.
Streaming matters for user experience
Cloud responses can take seconds.
Use streaming when available so the interface can show progress.
But design for:
user closes screen
app backgrounded
network changes
request cancelled
Mobile lifecycle events are part of AI correctness.
Cancellation must propagate
If the user leaves the task:
cancel network request
stop unnecessary inference
prevent follow-up tool action
Do not keep expensive cloud runs executing when the result will never be shown.
Background execution needs durable state
Some AI tasks may continue after the screen closes.
Store:
job ID
status
result location
and use platform-appropriate background execution.
Avoid keeping critical state only in one view model or process memory.
Model downloads need product UX
On-device models may require:
download time
storage space
Wi-Fi
The app should communicate this.
Useful pattern:
feature card
→ “Download AI model: 1.2 GB”
→ show progress
→ allow removal
Large hidden downloads destroy trust.
Battery and thermals are real constraints
Continuous local inference can heat the device and drain battery.
Measure:
single-run latency
sustained latency
energy use
thermal throttling
A feature that is fast for 30 seconds may degrade after several minutes.
Privacy should be visible in the architecture
Classify each input:
public
account data
private user content
highly sensitive
Then define whether it may go to:
on-device only
approved cloud provider
never AI
Do not ask the model to decide privacy policy dynamically.
Minimize content logging
AI debugging often tempts teams to store every prompt.
Mobile prompts can contain:
messages
photos
personal notes
voice transcripts
Use operational metadata where possible:
request ID
model
latency
status
and restrict raw-content traces.
Model updates need regression tests
On-device models can change with OS updates.
Apple's current documentation explicitly tells developers to test prompts against updated system models because behavior can change.
The same principle applies on Android.
Do not assume a prompt that worked last year remains stable forever.
Build an AI feature eval set
Test real mobile scenarios:
short input
long input
offline
low-memory device
slow network
unsupported device
image input
ambiguous request
Measure:
task success
latency
battery
fallback behavior
Mobile AI quality is a system metric.
A strong hybrid architecture
Mobile App
│
├── Local AI Runtime
│ ├── private rewrite
│ ├── summarization
│ └── local tools
│
├── Capability Router
│
└── Secure Backend
├── auth / quotas
├── cloud LLM
├── RAG / tools
└── long-running jobs
The router chooses based on task and policy, not hype.
Common mistakes
Sending every task to the cloud
You lose privacy/offline advantages.
Forcing everything on-device
Small models have limits.
Hardcoding provider secrets in the app
Use secure backend mediation.
Ignoring device fragmentation
Test minimum supported hardware.
No cancellation
Mobile users leave screens constantly.
Assuming system models never change
Run regressions after OS/model updates.
Production checklist
Before shipping AI in a mobile app, verify:
- Each task has an explicit local/cloud routing policy
- Device/model availability is checked at runtime
- Sensitive data policy is enforced outside the model
- Cloud secrets are not embedded in the client
- Structured outputs are validated
- Agent tools follow app/OS permissions
- Offline behavior is defined
- Model downloads have visible UX
- Battery and sustained thermal behavior are measured
- Cancellation and background execution are correct
- Model/OS updates trigger regression testing
- Minimum supported devices are included in evals
Final takeaway
Mobile AI architecture is now about placement as much as model choice.
Run small, privacy-sensitive, latency-sensitive work on-device when the hardware supports it. Use cloud models for harder reasoning, large context, and broad tooling. Combine both through explicit routing rules.
> The best mobile AI feature is not the one using the largest model. It is the one that feels fast, reliable, private, and available on the devices your users actually have.
Official references
Discussion (0)