I built a voice-controlled light switch a while back using Alexa, and it worked great — right up until my internet went out for an afternoon and suddenly my "smart" light couldn't be turned on by voice at all, because the command had to round-trip through Amazon's servers first. That bugged me more than it probably should have. So I rebuilt it using an ESP32-S3 with offline voice recognition, meaning the whole thing works completely locally. No internet, no cloud account, no company listening in — just a chip that recognizes a handful of specific words on its own.
This guide walks through building a genuinely offline voice-controlled relay, using the ESP32-S3's onboard processing power instead of relying on Alexa or Google's servers.
Why This Is Different From "Regular" Voice Control
Most consumer voice assistants (Alexa, Google Home, Siri) work by sending your audio to a cloud server, where far more powerful hardware does the actual speech recognition, then sends a simple command back to your device. That's genuinely necessary for understanding open-ended conversation — but it means an internet outage kills the feature entirely, and it also means your voice recordings are, at some level, leaving your house.
Offline voice recognition on something like the ESP32-S3 works differently. It doesn't understand full sentences or open conversation — instead, it's trained to recognize a small, fixed set of specific words or phrases directly on the chip itself, using a lightweight machine learning model that runs locally. You lose the flexibility of "ask it anything," but you gain something genuinely valuable: it works with zero internet connection, and nothing ever leaves your device.
What You'll Need
- 1x ESP32-S3 development board (specifically the S3 — the extra processing power and AI-oriented instructions matter here; a regular ESP32 will struggle with this)
- 1x I2S microphone module (INMP441 is a common, reliable choice)
- 1x Relay module
- Jumper wires and a breadboard
- The device you want to control (a lamp, fan, etc.)
How the Recognition Actually Works
Espressif (the company behind the ESP32 chip family) provides a speech recognition framework called ESP-SR, which includes pre-trained wake-word and command-word models designed specifically to run on ESP32-S3 hardware. You typically get a wake word (like "Hi ESP" by default, though this can sometimes be customized) followed by a small vocabulary of command words it can recognize — things like "turn on," "turn off," or numbers, depending on which command set you load.
The chip is constantly listening for just the wake word using a very lightweight always-on detection model. Once it hears that, it switches to a slightly more intensive mode to recognize the actual command word that follows. This two-stage approach is what keeps power consumption reasonable — it's not running full speech recognition constantly, just a simple "did I hear my name" check most of the time.
Step 1: Wiring the Microphone
The INMP441 communicates over I2S, a protocol designed specifically for digital audio:
- VDD → 3.3V
- GND → GND
- SD → GPIO 32
- WS → GPIO 25
- SCK → GPIO 26
- L/R → GND (this sets the microphone to left channel; tie to VDD instead if you want right channel)
Step 2: Wiring the Relay
- Relay VCC → 5V
- Relay GND → GND
- Relay IN → GPIO 4
Step 3: Setting Up ESP-SR
This is the part that's genuinely different from a typical Arduino IDE project. ESP-SR is built for ESP-IDF (Espressif's official development framework), not the standard Arduino IDE, though there are Arduino-compatible wrappers available if you'd rather stay in familiar territory. For the cleanest experience and access to the latest pre-trained models, I'd recommend working directly in ESP-IDF for this one, even if it's a bit more setup than you're used to.
The general workflow looks like this:
- Install ESP-IDF and set up your development environment (Espressif's official getting-started guide covers this well, and it's a one-time setup)
- Clone or reference the
esp-srcomponent in your project - Select a wake word model and a command word model (English command sets typically include basic on/off style vocabulary out of the box)
- Write your application logic so that when a command word is recognized, it triggers your relay function
A simplified version of the recognition callback logic looks roughly like this in concept:
void onCommandDetected(int commandID) {
if (commandID == COMMAND_TURN_ON) {
digitalWrite(RELAY_PIN, HIGH);
} else if (commandID == COMMAND_TURN_OFF) {
digitalWrite(RELAY_PIN, LOW);
}
}The actual implementation involves more setup around initializing the audio pipeline and the recognition engine, but conceptually, this is the core of what's happening — a recognized command ID triggers a simple action, exactly like a button press, just triggered by your voice instead.
Step 4: Testing It
Once flashed, say your wake word clearly from a reasonable distance (start close, like half a meter, before testing further away), followed by your command word. There's a brief pause after the wake word where the device is listening for the command — don't rush straight into it, but don't leave a huge gap either. It takes a few tries to get a feel for the natural rhythm the recognition model expects.
Common Problems (And How I Actually Fixed Them)
It never seems to hear the wake word. Microphone placement matters more than you'd expect. Keep it away from the ESP32-S3's own power regulator and any switching components, since electrical noise can genuinely interfere with a sensitive I2S microphone. A small foam wind cover or physical separation from the board helps too.
It works close up but fails from across the room. This is expected behavior, not a bug — these lightweight on-device models trade some range and noise-robustness for the ability to run locally at all. If you need reliable recognition from further away, positioning the microphone more centrally in the room (rather than tucked inside an enclosure) makes a noticeable difference.
False triggers happen randomly, even with no one talking. Background noise like TV audio or music can occasionally trigger a false wake-word detection. This is a known trade-off with lightweight offline models — if it's a persistent problem in your space, look into adjusting the detection sensitivity threshold in your configuration.
Command recognition works, but I want more command words than the default set includes. Espressif's tools do support training custom command word models, though it's a more involved process than just using pre-trained sets. For most home automation use cases, sticking with a well-tested pre-trained vocabulary and mapping multiple relays to combinations of existing words is far simpler than training new ones from scratch.
Why This Trade-Off Is Worth It (For Some Use Cases)
I'm not going to pretend this replaces Alexa for genuinely flexible voice control — if you want to ask about the weather or set a timer with natural phrasing, cloud-based assistants are still miles ahead. But for a specific, fixed task like "turn this light on or off," offline recognition genuinely delivers something cloud assistants structurally can't: it keeps working when your internet doesn't, and it doesn't require trusting a company's servers with an always-on microphone feed from your house.
Final Thoughts
This project pushed me further into ESP-IDF than most of my usual Arduino-based builds, and honestly, that learning curve is the real cost here — not the hardware, which is cheap and simple. But once it's running, there's something quietly satisfying about a voice command that works entirely inside a box the size of a matchbook, with nothing leaving the room. It's not flashier than the Alexa version. It's just more honestly yours.
%20Down%20Today.jpg)
%20.jpg){kind=tracking}
