I've been adding more and more virtual devices to my home automation server to keep track of personal data, like whether I'm home, which room I'm in, if I am asleep or awake, sleep quality, sleeping heart rate, snoring length (Withings Bluetooth sleep pad notifies the server when I'm on it or not as well as sleep quality/snoring etc.), weight (Withings Bluetooth scale),  calendar info (Google calendars) including work days/meetings, work PTO holidays, whether the upcoming weekend is 2, 3 or 4 days long... not just typical home automation devices like lights/fans/motion sensors.  Now I can use all of this information to start working on a personal assistant, similar to JARVIS from Iron Man.

This poorly recorded audio (held my iphone up to a speaker) is the Text To Speech output of my home automation servers, automatically giving me my morning status report when I get out of bed, similar to what JARVIS does in either Iron Man 1 or 2.  It takes into consideration if today is a work day, a paid time off holiday, a vacation day or a sick day and the name & time of my first meeting.  It knows what type of day tomorrow is as well as the upcoming weekend.  It knows when my next PTO holiday or vacation day is and mentions them if they occur today/tomorrow or the weekend.  If I will be working in the office, it will mention the current drive time from home to work by hitting the Waze public API.  If I'm working from home, it skips that & tells me how long I've been WFH. So far, it's mostly basic if/then statements and a randomizer for picking various greetings/sayings per day. It will grow in ability & complexity in the coming months.

I'm using HomeSeer Pro home automation server installed on a Windows 10 Intel NUC.  It uses the old SAPI Text To Speech system built into Windows and I purchased the "Brian" Ivona TTS voice (It sounded closest to JARVIS from the movies)  I will soon be moving the Speech Recognition & Text to Speech functionality over to a Raspberry Pi 4 running HomeSeer as well via the Mono .NET emulator.  I have the functionality working, but need to wire up the audio in/out to my main audio matrix.  On that Pi, I created a python script for Speech to Text voice recognition that listens locally for the wake word, "computer", then sends everything else I say after that out to the Google voice recognition servers which returns the text I spoke in under a second.  I pass that text into HomeSeer and it reacts as if I spoke directly to it.  I'll still be using the Brian voice, but I'll be sending any HomeSeer response as text out to AWS's Polly TTS service (Polly also has the Brian voice) which will promptly return an .mp3 audio file to play locally.  Same 2-way voice interaction, but it's now handled by Google & AWS instead of windows SAPI.  

I have speakers setup in every room with a few Clear One XAP800 audio matrix processing units so the TTS voice can follow me around as I go from room to room.  If I go offsite, the TTS voice is routed  to the Zello push to talk app on the computer, directly to my iPhone.  So I'll be able to interact with the TTS system anywhere I go.  If I get into my car, it switches over to my car speakers.  Getting the mechanics of Speech to Text / Text to Speech, basic virtual device queries & concatenation of TTS text responses is kind of a messy, large step one.  Next up is trying to integrate Machine Learning/AI to programmatically understand the context of what I say as well as coming up with appropriate responses.  If step one is building the body, then step two is like giving it a brain.

Morning Status

Ex-musician from Michigan masquerading as a Sr. Software Engineer in Los Angeles... who is trying to get back to his songwriting roots.

RandyInLA