Reachy Mini conversation app

---

title: Reachy Mini Conversation App emoji: 🎤 colorFrom: red colorTo: blue sdk: static pinned: false short_description: Talk with Reachy Mini! suggested_storage: large tags:

• reachy_mini
• reachy_mini_python_app

---

Reachy Mini conversation app

Conversational app for the Reachy Mini robot combining realtime voice, vision, personality-aware tools, and choreographed motion.

Table of contents

• Overview
• Architecture
• Installation
• Configuration
• Running the app
• LLM tools
• Creating and adding tools
• Advanced features
• Contributing
• License

Overview

• Low-latency audio conversation through the Hugging Face realtime backend, using the built-in server or a local endpoint.
• Vision is handled by the realtime backend when the camera tool is used.
• Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble.
• Async tools integrate motion, camera capture, and MCP Tool Spaces. The optional web UI (--ui) manages conversations, personalities, tools, and settings, and its Talk view can show the live transcript. The backend's per-turn latencies appear there when Settings → Developer options → Show response latencies is on; the choice is saved across launches.

Architecture

The app connects the user, AI services, and robot hardware:

Installation

[!IMPORTANT] Install Reachy Mini's SDK (opens in a new tab) before using this app.

Windows support is currently experimental and has not been extensively tested. Use with caution.

Set up with uv (opens in a new tab):

# macOS (Homebrew)
uv venv --python /opt/homebrew/bin/python3.12 .venv

# Linux / Windows (Python in PATH)
uv venv --python python3.12 .venv

source .venv/bin/activate
uv sync

Include dev dependencies:

uv sync --group dev

[!NOTE] Run uv sync --frozen to install the exact dependency set from uv.lock without re-resolving versions.

python -m venv .venv
source .venv/bin/activate
pip install -e .

Install dev dependencies:

pip install -e .[dev]                   # Development tools

Configuration

The default setup uses the Hugging Face backend and does not require an API key.

Copy .env.example to .env when you want to point Hugging Face at your own local endpoint.

These variables are the starting value. What you change through the settings UI (or from the mobile app) is written to settings.json in the instance directory and wins over them, so a toggle always does what it says.

Pick the conversation language with the flag button next to the personality on the Talk view. Save for chooses between the active personality only and all personalities; saving for all also clears the languages saved for single personalities. A personality's own language wins over the shared one and applies from the startup greeting on, so switching personality switches the language with it. Saving reconnects the conversation, and Reachy greets you again in the new language.

LOCAL_VAD_ENABLED defaults to true. Change Local speech detection in the gear-wheel Settings → Speech behavior view. Saved choices take precedence over environment defaults and persist across launches; saving reconnects the conversation to apply the change. Local Silero VAD and server VAD both control physical listening: either detector freezes the antennas and suppresses idle movement. Movement resumes smoothly once both report silence. Local detection holds through 400 ms of silence and clears after 500 ms without microphone frames. Model history clears after each completed speech segment and after 500 ms of consecutive low-confidence idle audio (512 ms at the 32 ms inference cadence). These resets preserve pending microphone samples and context during speech and within-segment pauses. It uses the SDK's 16 kHz microphone audio; conversation turns continue to use server VAD. If local detection fails, the app logs a warning and uses server VAD alone.

Local barge-in defaults to enabled. Change Local barge-in in Settings → Speech behavior, or set LOCAL_BARGE_IN_ENABLED=false to disable it when no saved preference exists. It requires local speech detection; disabling that switch also disables local interruption while preserving the saved barge-in preference. While audio is buffered or being submitted for playback (plus a 100 ms trailing margin), a Silero speech candidate lowers upcoming playback to 25% amplitude. Playback only stops once server VAD confirms the user turn, so loud background speech ducks the robot without dropping its answer; late audio from the interrupted response is discarded. A rejected candidate restores normal volume after 64 ms of silence. Server interruptions remain enabled, and microphone audio continues streaming to the backend. Playback is fed to the SDK in paced 20 ms chunks; already-submitted audio and SDK/transport latency delay audible volume changes. This is experimental: speaker echo, motor noise, audible latency and CPU overhead still need robot testing. With --debug, open /static/vad-monitor.html on the app settings server for a read-only live monitor with completed user and robot transcripts. The transcript view keeps the last 100 turns received while the page is open and clears on refresh. The /rpc WebSocket emits conversation.vad diagnostics with a Unix timestamp, the latest Silero speech probability, local/server/combined listening states, barge-in candidate and playback gain. Probability updates are limited to 10 Hz; state changes are reported immediately. Diagnostics reuse existing inference and do not change the inactivity timer.

On the first startup with local VAD enabled, the app downloads the Silero VAD v6.2.1 model and MIT license (opens in a new tab) directly from the pinned upstream commit and verifies their SHA-256 checksums. They are stored in the Hugging Face assets cache (HF_HOME/assets, or HF_ASSETS_CACHE when set), like the cached emotion recordings. Later starts reuse the files without a network request. The model runs on CPU with ONNX Runtime. If the download or model load fails, the app logs a warning and continues with server VAD alone.

Hugging Face Connection Modes

Use the built-in Hugging Face server through the app-managed Space proxy. This is the default for a new install; set it explicitly only when you want to switch back from a saved local endpoint:

HF_REALTIME_CONNECTION_MODE=deployed

Deployed session allocation falls back to cached hf auth login credentials and reports the daemon-provided hardware ID when available. Cached credentials and the hardware ID are not sent to local endpoints.

Run your own realtime voice backend using speech-to-speech (opens in a new tab) on the same machine as the conversation app:

HF_REALTIME_CONNECTION_MODE=local
HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime

Run your own Hugging Face backend on your laptop and connect to it from Reachy Mini Wireless over the same Wi-Fi network:

HF_REALTIME_CONNECTION_MODE=local
HF_REALTIME_WS_URL=ws://<your-laptop-lan-ip>:8765/v1/realtime

For that LAN setup, make sure the backend listens on an address reachable from the robot, not only on 127.0.0.1.

If the backend stays bound to loopback on your laptop, you can forward it into the robot over SSH instead:

ssh -N -R 8765:127.0.0.1:8765 <robot-user>@<robot-host>

Then set this on the robot:

HF_REALTIME_CONNECTION_MODE=local
HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime

In the web UI's Settings view, the Connection section lets you choose either the built-in server or a local host:port target. The UI writes HF_REALTIME_CONNECTION_MODE for you, and the local path writes HF_REALTIME_WS_URL with a default of localhost:8765.

Running the app

Activate your virtual environment, then launch:

reachy-mini-conversation-app

[!TIP] Make sure the Reachy Mini daemon is running before launching the app. If you see a TimeoutError, it means the daemon isn't started. See Reachy Mini's SDK (opens in a new tab) for setup instructions.

The app runs in console mode. Add --ui to serve the web interface at 127.0.0.1http://127.0.0.1:7860/ (opens in a new tab).

CLI options

Examples

# Audio-only conversation (no camera)
reachy-mini-conversation-app --no-camera

# Launch with the minimal web UI for personality/mic/settings control
reachy-mini-conversation-app --ui

LLM tools exposed to the assistant

The default profile exposes these tools. Use Tools → Tool access to customize any profile. Every bundled profile enables head_tracking by default; users can still disable it per personality.

[!NOTE] remember/forget facts are stored in memory.v1.json inside the app's instance data directory (~/.local/share/reachy_mini_conversation_app/ by default, or the instance path used by the desktop launcher). forget only removes facts matched by query. To reset all remembered facts, delete this file.

Creating and adding tools

Tools can run locally as Python code or remotely in an MCP-compatible Hugging Face Space. Keep robot, camera, and local-data operations in local tools. A Space is a better fit for shareable, stateless services such as search and external API lookups.

Local tools

Create one Python module per tool, with the file name matching the tool's unique name. See idle_do_nothing.py (opens in a new tab) for a minimal implementation.

Each tool subclasses Tool and defines name, a model-facing description, an object-shaped JSON Schema in parameters_schema, and an async __call__ method. Use ToolDependencies for runtime services, and set needs_response = False for actions that should not trigger a spoken follow-up. Catch expected operational failures, log them with the module logger, and return {"error": "..."} so the conversation can continue.

Restart the app after adding the module. Use Tools → Tool access to enable it for a personality, or add its name to that profile's default_tools in profile.md. See External profiles and tools for external directories and autoload behavior.

Hugging Face Space tools

To publish a remote tool, create a Gradio Space, expose its API as MCP with mcp_server=True, and give each function clear type hints and docstrings. Verify that https://<space-subdomain>.hf.space/gradio_api/mcp/schema lists the expected tools before installing the Space.

Use the maintained weather (opens in a new tab), time (opens in a new tab), and search (opens in a new tab) Spaces as examples. See Gradio's MCP server guide (opens in a new tab) for additional publishing guidance and Installing Hugging Face Space tools for this app's installation steps.

Advanced features

Built-in motion content is published as open Hugging Face datasets:

• Emotions: pollen-robotics/reachy-mini-emotions-library (opens in a new tab)
• Dances: pollen-robotics/reachy-mini-dances-library (opens in a new tab)

Create custom profiles with dedicated instructions and per-profile tool access.

Select and save a startup profile in the UI. The choice is stored in startup_settings.json. Before one is saved, REACHY_MINI_CUSTOM_PROFILE=<name> can select profiles/<name>/; otherwise the app uses default.

Every profile directory contains one strict schema-version-1 profile.md. TOML metadata is enclosed by +++; the remaining Markdown body is the realtime assistant prompt:

+++
schema_version = 1
voice = "Aiden"
greeting = "Greet me warmly in one sentence, in character, and vary the wording each time."
hidden = false
default_tools = [
  "dance",
  "camera",
  "sweep_look",
]
+++

## Identity

You are a concise, friendly robot guide.

schema_version, default_tools, and a non-empty Markdown body are required. voice, greeting, and hidden are optional. Set hidden = true to omit a profile from the UI. An empty default_tools list is valid and inherits nothing.

default_tools is the authored baseline. Tools → Tool access stores overrides in instance-local profile_toolsets.json without changing bundled profiles. Restoring defaults removes the override. Active-profile changes reconnect the conversation; other changes apply when selected.

Profile directories are data-only. Python tool implementations belong in src/reachy_mini_conversation_app/tools/, or in REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY for external tools. Each enabled tool ID must resolve to a shared tool, an external tool, or a tool from an installed Hugging Face Space.

See Creating and adding tools for the local tool interface and a maintained example.

To manage personalities in the UI:

With --ui, Home lists the available profiles and the built-in default:

• Tap a card to apply that personality and start talking.
• Tap "Manage tools" on a saved personality to open its tool access directly.
• Tap "Custom" to create a personality with a name, instructions, and optional greeting. It inherits the default tools, which can be changed under "Manage tools". Managed instances store it at user_personalities/<name>/profile.md; standalone runs use external_content/user_personalities/<name>/profile.md.

Switching a personality reloads its prompt and effective tools through a quick backend reconnect. Editing profile.md directly requires re-selecting the profile or restarting the app.

To create a locked variant of the app that cannot switch profiles, edit src/reachy_mini_conversation_app/config.py and set the LOCKED_PROFILE constant to the desired profile name:

LOCKED_PROFILE: str | None = "mars_rover"  # Lock to this profile

When set, the app ignores saved startup settings, REACHY_MINI_CUSTOM_PROFILE, and UI selection. The UI marks the profile as locked and disables editing.

You can extend the app with profiles/tools stored outside the repository defaults.

• Core profiles are under profiles/.
• Core tools are under src/reachy_mini_conversation_app/tools/.

Recommended layout:

external_content/
├── external_profiles/
│   └── my_profile/
│       └── profile.md
├── external_tools/
│   └── my_custom_tool.py
├── user_personalities/
│   └── my_custom_profile/
│       └── profile.md
├── installed_tool_spaces.json
└── profile_toolsets.json

Environment variables:

Set these values in your .env when you want env-driven external profile/tool selection:

# Optional fallback/manual profile selector:
REACHY_MINI_CUSTOM_PROFILE=my_profile
REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=./external_content/external_profiles
REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY=./external_content/external_tools
# Optional convenience mode:
# AUTOLOAD_EXTERNAL_TOOLS=1

Loading rules:

• Profiles: each directory requires a schema-version-1 profile.md with explicit default_tools; there is no cross-profile fallback.
• Default mode: enabled IDs must resolve to a shared, external, or installed Tool Space tool.
• Autoload: AUTOLOAD_EXTERNAL_TOOLS=1 adds every valid *.py module from REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY.
• Web UI: Tools → Tool access enables external modules per profile; it does not upload or edit Python.
• Separation: profile directories contain data only; external Python belongs in REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY.
• Tool names: every loaded class needs a unique Tool.name; duplicates fail fast.

You can install MCP-compatible Hugging Face Spaces as remote tool sources for this app. Private Spaces work too, as long as HF_TOKEN is set (or you have run hf auth login) for an account that can access them. To publish a new Space, follow Creating and adding tools.

Tools → Tool Spaces installs or refreshes a global source. Its tools then appear under Tools → Tool access for per-profile selection. Removing a Space removes its tools from every profile. Active-profile changes reconnect the conversation; other changes apply when selected.

The app accepts Hugging Face Spaces exposing the standard /gradio_api/mcp/ endpoint, not arbitrary MCP URLs. Installation discovers the Space's tools and assigns namespaced local IDs, so do not guess or hard-code those IDs beforehand.

# install + enable in active profile
reachy-mini-conversation-app tool-spaces add <owner/space-name>

# enable in a specific profile
reachy-mini-conversation-app tool-spaces add <owner/space-name> --profile NAME

# install without enabling
reachy-mini-conversation-app tool-spaces add <owner/space-name> --install-only

# list installed spaces
reachy-mini-conversation-app tool-spaces list

# remove an installed space
reachy-mini-conversation-app tool-spaces remove owner/space-name

Bundled Pollen Spaces use static specs and are enabled by the default profile. Newly bundled Spaces are added to existing installations without restoring older tools the user removed. Custom Spaces are validated through the Hugging Face Hub; HF tokens are sent only to private Spaces. Tool metadata is cached in:

• installed_tool_spaces.json in the managed app instance directory
• external_content/installed_tool_spaces.json in terminal mode

Startup and profile switching read this cache without discovery or MCP probing. Network access occurs only during install, refresh, or remote tool calls. Per-profile access is stored in profile_toolsets.json beside the manifest, or under external_content/ in terminal mode.

Recommended tags for discoverability on Hugging Face:

• reachy-mini-tool
• mcp

Tags are advisory; installation still requires successful MCP validation.

[!NOTE] Preinstalled Pollen Spaces can be removed like any other (tool-spaces remove pollen-robotics/reachy-mini-weather-tool). To restore access, reinstall the Space and restore or update the relevant profile under "Tool access".

If you run multiple Reachy Mini daemons on the same network, use:

reachy-mini-conversation-app --robot-name <name>

<name> must match the daemon's --robot-name value so the app connects to the correct robot.

Contributing

See CONTRIBUTING.md (opens in a new tab) for the development workflow and AGENTS.md (opens in a new tab) for coding-agent standards.

License

Apache 2.0