Background

Automotive Grade Linux (AGL) is a collaborative open source project that is bringing together automakers, suppliers and 
technology companies to accelerate the development and adoption of a fully open software stack for the connected car. 
Being a part of speech expert group, Amazon (Alexa Automotive) team intends to collaborate to help define the voice 
service APIs in AGL platform.

Definitions

Voice Services

These are standard set of APIs defined by the AGL Speech EG for users and applications to interact with voice 
assistant software of their choice running on the system. The API is flexible and attempts to provide solution 
for uses cases that span, running one or multiple voice assistants on the AGL powered car head units at the 
same time.

Voice Agent

Voice agent is a virtual voice assistant that takes audio utterances from user as input and runs It’s own speech 
recognition and natural language processing algorithms on the audio input , generates intents for applications 
on the system or for applications in their own cloud to perform user requested actions. This is vendor-supplied 
software that needs to comply with a standard set of APIs defined by the AGL Speech EG to run on AGL.

Purpose

The purpose of this document is to propose the Voice Service and Voice Agent APIs for AGL Speech framework, and 
evolve the same into a full blown multi-agent architecture that car OEMs use to enable voice experiences of their choice. 

Assumptions

• One voice agent will not possess all the capabilities that customer desire. So, more than one voice agent needs
  to exist on the platform and they need to work together at times to fulfill a customer ask. 
• One voice agent will not be able to properly detect all the explicit invocation words (wake words) of other
  voice agents. For example, Amazon best optimizes the machine learning models for the Alexa wake word.  Same goes with other voice agent vendors.

Customer Requirements

Customer A: As a IVI head unit user, I would like to have the following experiences,

• Experience A: With only one active voice agent.

  • I should be able to select the active voice service agent.
  • "Alexa, whats the weather" and "whats the weather" should be routed to the user selected voice agent.
    
    

• Experience B: Multiple active voice agents use cases. 
  • Invocation
    • Explicit Voice : Speech framework will pick the appropriate agent based on keyword detection to perform a task.
      For this approach to work neatly all the voice agents should have a wake word.
      • If I say, "Alexa, whats the weather",  my utterance should be routed to Alexa voice agent on the device.

• • • Implicit Voice : Speech framework will pick the appropriate agent based on intent detection to perform a task.
      The speech request should be routed to the best agent responsible for the handling the intent.
      • I should be able to assign appropriate agents to select set of tasks like Navigation, Calling, News, Car control, Local Music, Calendar etc.
      • If I don't explicitly assign an agent to a task, then the system should route my utterance to the best agent capable of handling the request.

• • • Multi Modal Interaction
      • (U) Agent A, Route me Starbucks nearby
      • A list Starbucks nearby are presented to user.
      • (U) Agent A says Select first one OR Clicks on the first item in the list
      • (Agent A) Launches Navigation App with geocode of selected Starbucks

• • • Fallback : Speech framework will fallback to a different voice agent if the chosen one fails to fulfill a request.
      
      • Silently route my utterance if the chosen agent fails to perform a task.
        
      • Inform me using speech that Agent A failed to fulfill the request and so its trying Agent B.

• • • Proactive : Any voice agent should be able to initiate a dialog or perform an action without a corresponding implicit or explicit user invocation.
      • Based on system behavioral changes. Agent A warns user that "Tire pressure is low" or " Maintenance is due, do you want to schedule it ? ".
      • When Agent A does a restaurant reservation, then Agent B can offer a parking spot near that restaurant if its assigned and if it possess that capability.

• • Multi-turn Dialog Use cases
    • No switching to a different voice agent when I am in an active dialog with some other agent.
    
    
    • Agent switching based on keyword detection
      
      • (User) Agent A book flight ticket to Seattle
      • (Agent A) What time you want to leave?
      • (User) Agent B, what time my last meeting ends on Mon
      • (Agent B) 5:30
      • (User) Agent A book ticket after 6:30 pm
        
        
    • Agent switching based on intent detection
      • (User) Book flight ticket to Seattle
      • (Agent A) What time you want to leave?
      • (User) What time my last meeting ends on Mon
      • (Agent B) 5:30
      • (User) Alright, book ticket after 6:30 pm
    
    

Customer B: As a car OEM,

• I should be able control the agents that run on my system, their life times and their responsibilities.

Customer C: As an AGL Application developer,

• I should be able to use AGL Speech framework's to voice enable my application.

Customer D: As a 3rd party Voice Agent Vendor,

• I should be able to follow the guidelines of the AGL speech framework to plugin my voice agent.
• I should be able to follow the guidelines of the AGL speech framework to plugin my wake word solution.
• I should be able to follow the guidelines of the AGL speech framework to plugin my NLU engine.

High Level Architecture

Quoting AGL documentation,
http://docs.automotivelinux.org/docs/apis_services/en/dev/reference/signaling/architecture.html#architecture“
“Good practice is often based on modularity with clearly separated components assembled within a common framework. Such modularity ensures separation of duties, robustness, resilience, achievable long term maintenance and security.”

High Level Components

Architecture

The Voice Services architecture in AGL is layered into two levels. They are High Level Voice Service layer and vendor software layer. In the above architecture, the high-level voice service is composed of multiple bindings APIs (colored in green) that abstract the functioning of all the voice assistants running on the system. The vendor software layer composes of vendor specific voice agent software implementation that complies with the Voice Agent Binding APIs.

High Level Work Flow

Experience A: With only one active voice agent at a time. User selected the active agent.

Assumption: Voice agents are initialized and running, and are discoverable and registered with afbvoiceservice-highlevel. afb-voiceservice-highlevel binding has subscribed to all the events of active voiceagent-binding and vice versa. afb-voiceservice-highlevel is assigned the audio input role by the audio high level binding.

• afb-voiceservice-highlevel startListening API will be triggered by Push-To-Talk button invocation.

• afb-voiceservice-highlevel signals afb-voiceservice-wakeword-detector binding running in same binder context to startListening. The wakeword detector binding uses PCM APIs to read the audio input.
• afb-voiceservice-highlevel will move from IDLE to LISTENING state. The Voice Dialog UI app would show a voice chrome or similar UI indicating the user to start speaking. 
• User starts speaking. afb-voiceservice-highlevel will wait for a few milliseconds for the afb-voiceservice-wakeword-detector to come back with onWakeWordDetected event.
  • If afb-voiceservice-wakeword-detector doesn't detect wake word within the aforementioned timeout, then afb-voiceservice-highlevel will pick the active agent selected by user to process the audio input.
  • If afb-voiceservice-wakeword-detector detects the wake word, then afb-voiceservice-highlevel will get onWakeWordDetected event with the offset of position from which the voice agent should start buffering the audio & with the wake word string that was detected. 
    • Then afb-voiceservice-highlevel  will call the startListening API of the active voice-agent-binding. Along with that it also passes the offset buffering location and the detected wakeword.
• voice-agent-binding, upon receiving startListening call, will transition between LISTENING, THINKING, and SPEAKING states and will regularly publish onDialogStateChanged event with its current state to afb-voiceservice-highlevel.
• voice-agent-binding will start reading audio input until stopListening is called or until end of speech is detected. Once end of speech is detected it responds with onEndOfSpeechDetected event.
• Voice agent will perform either cloud based or onboard ASR, NLU and triggers different actionable responses. 
  • If the response is music audio playback, voice-agent-binding will interface with AGL framework's media player binding to create an audio channel and stream the audio. 
  • If voice-agent-binding needs to present a TTS as response or initiate a multi turn dialog, it will do so by calling audio 4a binding and moving to SPEAKING state. In this case, the afb-voiceservice-highlevel will also move to SPEAKING state. The Voice Dialog UI will present the UI representing the SPEAKING state of the voice agent.
  • If the response needs to command a system application to perform an action like NAVIGATE_TO or CALL, then it will send a topic based message/intent using domain specific multimodal interaction manager APIs of afb-voiceservice-highlevel binding.
  • If there is an associated UI card to be displayed,  voice-agent-binding will send another message to the card rendering application using tmultimodal interaction manager APIs of afb-voiceservice-highlevel binding.

Bindings

The Voice Services architecture in AGL is layered into two levels. They are High Level Voice Service component and vendor software components with Voice Agents and Wake Word detection solutions. In the above architecture, the high level voice service is composed of multiple bindings (colored in green) that will be part of the AGL framework. And the vendor software layer composes of voice agent binding and wake word binding that hosts the vendor specific voice assistant software. The system provides flexibility to voice assistant vendors to provide their software as code or binary as long as they abide by the Voice Agent API specifications.

Below is a technical description of each of these binding in both the levels.

High Level Voice Service

High level voice services primarily runs in following two well known modes.

• Tap-To-Talk Mode
  • In this mode, the user will need to press the tap-to-talk button on the car steering wheel to talk to the voice agents. 
  • If user doesn't mention the wake word then the utterance will be routed to the default voice agent.
  • If user mentions a wake word, then the utterance will be routed the appropriate agent.
• Always listening Mode
  • In this mode, upon wake word detection the utterance will be routed to the appropriate voice agent.

This design makes no assumptions on the mode in which the high level voice service component is configured and running. 

1) agl-service-voice-high

This binding has following responsibilities.

• Structurally follows a bridge pattern to abstract the functioning of the specific voice agent software from the application layer.
• The request arbitrator is main entry point to the system. It is responsible for routing the utterance to the correct voice agent based on various parameters like configuration, wake word detection etc.
• Registers for dialog, connection, auth etc events from voice agents. Maintains the latest and the greatest state of the voice agents.
• Audio/Visual Focus management. Provides an interface using which the voice agents can request audio or visual focus before actual rendering the content. In multiple active voice agent scenario, we can imagine that each agent would be competing for audio and visual focus. Based on the priority of the content, the core should grant or deny focus to an agent. In cases where it grants the focus, it has to inform the agent currently rendering the content to duck or stop. And make audio and visual focus decisions on behalf of the voice agents its managing.

Current Architecture

The following diagrams dives a litter deeper into the low level components of high level voice service (VSHL)

and their dependencies depicted by directional arrows. The dependency in this case can be either through an

association of objects between components or through an interface implementation relationships.

For e.g.,

A depends on B if A aggregates or composes B.

A depends on B if A implements an interface that is used by B to talk to A.

VoiceAgents Module

Core Module

vshl/startListening


Starts listening for speech input. As a part of request, common configuration related information is passed. 


Note: The config inputs below are just examples and not the final list of configurations. 


Request: { 

}


Responses: { 
  "jtype":"afb-reply", 
  "request": { 
    "status":"string" // success or bad-state or bad-request
  }
  "response":{
    "request_id": "string" // Request created by this call.
    "agent_id": "string" // Agent to which the request has been proxied.
  }
}

vshl/cancelListening


Cancels the speech recognition processing in the chosen agent.
If agent id is not passed then the cancel request is sent to the default voice agent.


Request:
{

}

Responses:
{
  "jtype":"afb-reply",
  "request":{
    "status":"string" // success or bad-state or bad-request
  }
}

vshl/subscribe

Subscribe/Unsubscribe to voice service high level events.

"permission": "urn:AGL:permission:speech:public:accesscontrol"

Request:
{
  {
    "type":"array",
    "items" : [{
        "type":"string" // List of events to subscribe to
      }
    ]
  },
  {
    "subscribe":"boolean"
  }  
}

Responses:
{
  "jtype":"afb-reply",
  "request":{
    "status":"string" // success or bad-state or bad-request
  }
}

Dialog state describes the state of the currently active voice agent's dialog interaction. 

Event Data:
{
  "name" : "voice_dialogstate_event"
  "state":"string"
  "agent_id": "integer"
}


Values for state are
1) IDLE
High level voice service is ready for speech interaction.

2) LISTENING
High level voice service is currently listening.

3) THINKING
A customer request has been completed and no more input is accepted. In this state, Voice service is working on a response.

4) SPEAKING
Responding to a request with speech.

Connection state describes the state of the voice agent along with errors. 

Event Data:
{
  "name" : "voice_connectionstate_event"
  "state":"string"
  "agent_id": "integer"
}

1) DISCONNECTED
Voice agent is not connected to its voice service endpoint.

2) PENDING
Voice agent is attempting to establish connection to its endpoint.

3) CONNECTED
Voice agent is connected to its endpoint.

4) CONNECTION_TIMEDOUT
Voice agent connection attempt failed due to excessive load on its server endpoint.

5) CONNECTION_ERROR
Captures other network related errors.

Auth state describes the state of the authorization of the voice agent with its cloud endpoint. 

Event Data:
{
  "name" : "voice_authstate_event"
  "state":"string"
  "agent_id": "integer"
}

1) UNINITIALIZED
Authorization not yet acquired.


2) REFRESHED
Authorization has been refreshed.

3) EXPIRED
Authorization has expired.


4) ERROR
Authorization error has occurred.

vshl/enumerateVoiceAgents


"permission": "urn:AGL:permission:vshl:voiceagents:public"


Enumerates and return an array of voice agents running in the system. This might be need for the applications like settings to be able to present some UI with a list of agents to enable/disable, show status etc.


Request: 
{ } 


Responses: { 
  "jtype":"afb-reply",
  "request": { 
    "status":"string" // success or bad-state or bad-request
  } 
  "response": { 
    "type":"array", 
    "items" : 
      [ 
        { 
          "name":"string", 
          "description":"string", 
          "agent_id":"integer" // Voice agent ID 
          "status":"string" // enabled, disabled 
        } 
      ] 
  }
}

vshl/setDefaultVoiceAgent

Activate or deactivate a voice agent.

"permission": "urn:AGL:permission:vshl:voiceagents:public"


Request:
{ 
  "agent_id":"integer"
  "is_active":"boolean"
}


Responses: { 
  "jtype":"afb-reply",
  "request":
  {
    "status":"string" // success or bad-state or bad-request }
  }
} 

voiceagent/setup

This API is exposed to high level voice service to pass any setup or high level config information like agent_id to the voice agent.

"permission": "urn:AGL:permission:speech:public:accesscontrol"

Request:
{ 
  "agent_id":"integer"
  "language":"string"
}


Responses:
{
  "jtype":"afb-reply",
  "request":{
    "status":"string" // success or bad-state or bad-request
  }
}

voiceagent/cancel

Stop the voice agent and its currently running speech recognition processes.


"permission": "urn:AGL:permission:speech:public:accesscontrol"


Request:
{ 

}


Responses:
{
  "jtype":"afb-reply",
  "request":{
    "status":"string" // success or bad-state or bad-request
  }
}

voiceagent/startListening

Start the listening for speech input. As a part of request, common configuration related information is passed.
Note: The config inputs below are just examples and not the final list of configurations.

"permission": "urn:AGL:permission:speech:public:audiocontrol"
Request:
{
  "request_id": "string" // Request ID assigned by the high level voice service.
  "language":"string"
  "location":"string"
  "preferred_network_mode":"string" // online, offline, hybrid
  "audio_input_device": "string" // ID of the alsa device to read the input
}


Responses:
{
  "jtype":"afb-reply",
  "request":{
    "status":"string" // success or bad-state or bad-request
  }
}

Voice agent will notify its clients that end of speech is detected.
Event Data:
{
  "name" : "voiceagent_endofspeechdetected_event"
  "agent_id": "integer"
  "request_id": "integer" // the request for which the end of speech is detected
}

Dialog state describes the state of the currently active voice agent's dialog interaction. 

Event Data:
{
  "name" : "voiceagent_dialogstate_event"
  "state":"string"
  "agent_id": "integer"
  "request_id": "string" // The request that caused this dialog state transition.
}


Values for state are
1) IDLE
High level voice service is ready for speech interaction.

2) LISTENING
High level voice service is currently listening.

3) THINKING
A customer request has been completed and no more input is accepted. In this state, Voice service is working on a response.

4) SPEAKING
Responding to a request with speech.

Connection state describes the state of the voice agent along with errors. 

Event Data:
{
  "name" : "voiceagent_connectionstate_event"
  "state":"string"
  "agent_id": "integer"
}

1) DISCONNECTED
Voice agent is not connected to its voice service endpoint.

2) PENDING
Voice agent is attempting to establish connection to its endpoint.

3) CONNECTED
Voice agent is connected to its endpoint.

4) CONNECTION_TIMEDOUT
Voice agent connection attempt failed due to excessive load on its server endpoint.

5) CONNECTION_ERROR
Captures other network related errors.

Auth state describes the state of the authorization of the voice agent with its cloud endpoint if any. 

Event Data:
{
  "name" : "vshl_authstate_event"
  "state":"string"
  "agent_id": "integer"
}

1) UNINITIALIZED
Authorization not yet acquired.


2) REFRESHED
Authorization has been refreshed.

3) EXPIRED
Authorization has expired.


4) ERROR
Authorization error has occurred.