Early Language Learning Agent (ELLA)

early_language_learning_agent is a ROS 2 (Python / ament_python) package that runs an interactive, story-based early language learning experience for preschool-aged children.

At runtime, ELLA alternates between:

• Narrative chunks: pre-generated story narration with synchronized facial expressions + body motion.
• Interaction chunks: short, child-facing questions designed to elicit and reinforce a target vocabulary word, with LLM-driven scaffolding.

This repository folder is intended to be open-sourced as the early_language_learning_package (ROS package name: early_language_learning_agent).

---

Key features

• Story session orchestration (wake-up → in-session chunk loop → wind-down)
• Pre-recorded narrative playback (base64 MP3 decoded to WAV for ROS playback)
• Vocabulary scaffolding via Groq-hosted LLMs (JSON-structured multi-utterance replies)
• Turn-taking using VAD + turn-end detection + ASR transcript buffering
• Robot behavior coupling (facial expressions + timed gesture primitives + body poses)
• Optional start/stop recording hooks for session capture

---

Package layout

• early_language_learning_agent/
  • session_manager.py — top-level ROS node that runs a story session
  • story_models.py — story document + chunk schema
  • narrative_delivery.py — plays narrative chunk audio + animations
  • conversation_manager.py — runs interactive chunks (questions, ASR, scaffolding)
  • interaction_handlers.py — per-interaction-type response generation
  • scaffold_response_generator.py — unified scaffold + transition generation
  • response_delivery.py — TTS → audio playback + expressions/gestures
  • turn_manager.py — VAD/turn-end driven turn completion
  • animation.py — scheduling utilities for face/body programs
  • groq_llm_provider.py — minimal Groq chat-completions client
• stories/ — story JSON files (often large due to embedded base64 audio)
• prompts/ — prompt templates (scaffolds/, greeting/, goodbye/)
• config/config.yaml — session configuration
• launch/ella_session.launch.py — launches the full “robot stack” + ELLA session

---

Requirements

Core

• ROS 2 (Python rclpy)
• m_interfaces (custom action/service definitions)
• Python deps (installed via setup.py):
  • pyyaml
  • pydub
  • groq

System dependency for MP3 decoding

pydub typically requires FFmpeg to decode MP3:

• macOS: brew install ffmpeg
• Ubuntu: sudo apt-get install ffmpeg

External services

• Groq API for LLM scaffolding
  • set GROQ_API_KEY

Model selection

The default Groq model is currently set in code (see interaction_handlers.py / scaffold_response_generator.py), defaulting to:

• openai/gpt-oss-20b

To change it, adjust the model= argument when constructing Groq_LLMProvider / handlers.

---

Installation (ROS 2 workspace)

From your ROS 2 workspace:

# from <ws>
colcon build --packages-select early_language_learning_agent
source install/setup.bash

If you use rosdep, make sure dependencies like m_interfaces, audio_processing, motor_control, m_commons, and m_logging are also available in your workspace.

---

Quickstart

1) Configure environment

source /root/m_ws/.env

2) Run the full session launch

This launch file brings up the surrounding robot/audio stack and the ELLA session manager.

ros2 launch early_language_learning_agent ella_session.launch.py
# or choose a story
ros2 launch early_language_learning_agent ella_session.launch.py story:=quest.json

3) Run only the ELLA node (advanced) [not reccomended unless testing]

You can run just the session manager node, but it expects several ROS interfaces (TTS, audio playback, pose control, ASR/VAD topics) to exist:

ros2 run early_language_learning_agent session_manager --ros-args -p story_filename:=quest.json

---

Configuration

ELLA loads config/config.yaml in this order:

1. ELLA_CONFIG_PATH environment variable
2. Installed package share directory (ament index)
3. Source checkout (relative to the early_language_learning package)

config/config.yaml keys:

• story_directory (default: "stories")
• prompts_directory (default: "prompts")
• wake_up_delay, cooldown_between_chunks, wind_down_delay (seconds)
• initial_wait_ms, vad_silence_timeout_ms, grace_period_ms (milliseconds)
• child_name
• max_interaction_turns
• skip_wake_up, skip_wind_down, skip_interactions

Relative story_directory / prompts_directory values are resolved relative to the installed share directory (preferred) or the source checkout.

---

Stories: JSON schema

Stories are JSON objects with:

• story: full story text (often includes <interaction ...> tags for authoring)
• chunks: ordered list of chunk objects

Chunk types:

Narrative chunk

{
  "type": "narrative",
  "content": "...",
  "audio": "<base64 mp3>",
  "face_programs": [["{...json...}", 0.0]],
  "body_programs": [["[{...timeline...}]", 0.0]]
}

Interaction chunk

{
  "type": "interaction",
  "content": {
    "interaction_type": "greeting" | "vocab" | "goodbye",
    "vocab": "quest",
    "question": "What do you think makes a quest different...?"
  },
  "audio": "<base64 mp3>"
}

Notes:

• audio is expected to be MP3 encoded as base64; ELLA converts it to WAV for ROS playback.
• face_programs and body_programs are optional and are scheduled relative to chunk start.
• The loader automatically attaches prior and upcoming narrative context to interaction chunks for better transitions.

---

Prompts and LLM replies

Prompts live under:

• prompts/scaffolds/scaffold.txt — unified scaffolding prompt
• prompts/scaffolds/transition_vocab.txt — transition back into story after vocab interaction
• prompts/greeting/greeting.txt — greeting interaction
• prompts/goodbye/goodbye.txt — goodbye interaction

ELLA expects the LLM to return a JSON array. The code is tolerant, but best results come from returning:

• (optional) a first object like { "strategy": "<name>" }
• followed by one or more utterance objects:

[
  {"strategy": "eliciting"},
  {"utterance": "Wow!", "expression": "excitement:high", "gesture": "beat:start"},
  {"utterance": "Can you say quest?", "expression": "interested:low", "gesture": "invite:mid"}
]

Each utterance is delivered via:

1. tts_synthesize (TextToSpeech service → MP3 base64)
2. MP3→WAV conversion
3. audio_play action

While audio plays, ELLA publishes facial expressions and schedules gesture primitives.

---

ROS interfaces (what ELLA publishes/subscribes)

Subscribed topics

• asr_text (std_msgs/String) — streaming ASR text fragments
• vad_result (std_msgs/Bool) — VAD speech/not-speech
• turn_completion (std_msgs/String) — turn-end detector results (expects complete / incomplete)

Published topics

• session_active (std_msgs/Bool) — session lifecycle status
• robot_speaking (std_msgs/Bool) — whether the robot is currently speaking
• asr_enable (std_msgs/Bool) — enables/disables ASR buffering
• face_expression (std_msgs/String) — face controller commands (e.g., talking, reset)
• robot_utterance (std_msgs/String) — emitted for logging each spoken utterance

Services / actions used

• tts_synthesize (m_interfaces/srv/TextToSpeech)
• set_pose (m_interfaces/srv/SetPose)
• audio_play (m_interfaces/action/AudioPlay)
• /start_recording + /stop_recording (std_srvs/srv/SetBool) — used by the launch file / session end hook

---

Development

Run lint/tests (ament)

colcon test --packages-select early_language_learning_agent
colcon test-result --verbose

Common issues

• FFmpeg missing: MP3→WAV conversion will fail.
• No Groq key: set GROQ_API_KEY.
• Missing ROS interfaces: running session_manager without the surrounding stack will fail to synthesize/play speech.

---

License

Package metadata declares CC BY 4.0 (see package.xml / setup.py).