LinTO Agents on Webpages

linto-web-client

A full figured LinTO client designed for building custom voice interactions on a webpage.

See demo here : linto.ai

Note : LinTO Web client relies on WebVoiceSDK for handling everything related to audio input, hotword triggers, recordings... See LinTO WebVoiceSDK on NPM for more informations

Note : Any LinTO web client needs to have a token registered towards a LinTO server. See more information in LinTO's official documentation.

Note : This library might cause issues (crashes) on your webpage as some browser's implementation of WebAssembly runtimes is still experimental

Usage

With bundler :

npm i @linto.ai/linto-web-client

Static script from CDN :

<script src="https://cdn.jsdelivr.net/gh/linto-ai/linto-web-client@master/dist/linto.min.js"></script>

Test right away :

• Tweak content in tests/index.js (your token, LinTO server endpoint etc)

npm run test

Instanciante

try {
    window.linto = await new Linto(
        `${My_linto_stack_domain}/overwatch/local/web/login`,
        `${my_app_token}`,
        `${ms_timeout_delay_for_commands}`
    );
} catch (lintoError) {
    // handle the error
}

Handling errors

This command might throw an error if something bad occurs

Instance user methods

- startAudioAcquisition(true, "linto", 0.99) // Uses hotword built in WebVoiceSDK by name / model / threshold
- startCommandPipeline() // Start to listen to hotwords and binds a publisher for acquired audio when speaking stop
- stopCommandPipeline()
- startStreamingPipeline() // Start to listen to hotwords and binds streaming start/stop event when audio acquired
- stopStreamingPipeline()
- triggerHotword(dummyHotwordName = "dummy") // Manualy activate a hotword detection, use it when commandPipeline is active.
- pauseAudioAcquisition()
- resumeAudioAcquisition()
- stopAudioAcquisition()
- startStreaming(metadata = 1) // Tries to initiate a streaming transcription session with your LinTO server. The LinTO server needs a streaming skill and a streaming STT service
- addEventNlp() // Bind the event nlp to handle only linto answer
- removeEventNlp()
- stopStreaming()
- login() // Main startup command to initiate connexion towards your LinTO server
- loggout()
- startHotword()
- stopHotword()
- sendCommandText("blahblah") // Use chatbot pipeline
- sendWidgetText("blahblah") // Publish text to linto (bypass transcribe)
- triggerAction(payload, skillName, eventName) // Publish payload to the desired skill/event
- say("blahblah") // Use browser text to speech
- ask("blahblah ?") // Uses browser text to speech and immediatly triggers hotword when audiosynthesis is complete
- stopSpeech() // Stop linto current speech

Instance events

Use events with :

linto.addEventListener("event_name", customHandlingFunction);

Available events :

• "mqtt_connect"
• "mqtt_connect_fail"
• "mqtt_error"
• "mqtt_disconnect"
• "speaking_on"
• "speaking_off"
• "command_acquired"
• "command_published"
• "command_timeout"
• "hotword_on"
• "say_feedback_from_skill"
• "ask_feedback_from_skill"
• "streaming_start"
• "streaming_stop"
• "streaming_chunk"
• "streaming_final"
• "streaming_fail"
• "action_acquired"
• "action_published"
• "action_feedback"
• "action_error"
• "text_acquired"
• "text_published"
• "chatbot_acquired"
• "chatbot_published"
• "chatbot_feedback"
• "chatbot_error"
• "custom_action_from_skill"

NOTE : See proposed implementation in ./tests/index.js

linto-web-client Widget (UI)

Building sources

npm install
npm run build-widget

Those commands will build linto.widget.min.js file in the /dist folder

Using library

Import linto.widget.min.js file to your web page. Once it's done, you can create a new LintoUI() object and set custom parameters.

<script type="text/javascript" src="YOUR_PATH/linto.widget.min.js"></script>
<script type="text/javascript">
    window.LintoUI = new LintoUI({
        debug: false,
        containerId: CONTAINER_BLOCK_HTML_ID,
        lintoWebToken: LINTO_APPLICATION_TOKEN,
        lintoWebHost: LINTO_APPLICATION_HOST,
        widgetMode: WIDGET_MODE,
    })
</script>

Parameters

Parameter	type	values	description
debug	boolean	true / false	Enable or disable console informations when events are triggered
containerId	string	"div-wrapper-id"	ID of the block that will contain the widget
lintoWebHost	string	"https://my-host.com"	Url of the host where the application is deployed
lintoWebToken	string	"yourToken"	Authorization token to connect the application
widgetMode	string	"multi-modal" (default) / "minimal-streaming"	Set the widget mode
hotwordEnabled	string	"true" / "false"	Enable or disable hotword detection
hotwordValue	string	"linto"	Value of the hotword. Change it if you use an other hotword model than "linto"
audioResponse	string	"true" / "false"	Enable or disable linto widget audio response
streamingStopWord	string	"stop"	Set stop-word for streaming "infinite" mode
lintoCustomEvents	array of objects	{"flag": "event_name": “func”: function(){} }	Bind custom functions to events
widgetMicAnimation	string	"/path/to/animationfile.json"	Set a custom animation file for "widget microphone animation"
widgetThinkAnimation	string	"/path/to/animationfile.json"	Set a custom animation file for "widget thinking animation"
widgetSleepAnimation	string	"/path/to/animationfile.json"	et a custom animation file for "widget sleeping animation"
widgetTalkAnimation	string	"/path/to/animationfile.json"	Set a custom animation file for "widget talking animation"
widgetAwakeAnimation	string	"/path/to/animationfile.json”	Set a custom animation file for "widget awaken animation"
widgetErrorAnimation	string	"/path/to/animationfile.json"	Set a custom animation file for "widget error animation"
widgetValidateAnimation	string	"/path/to/animationfile.json”	Set a custom animation file for "widget validation animation"
widgetTitle	string	"LinTO Widget"	Widget Title value
cssPrimarycolor	string	"#59bbeb"	Value of the widget primary color (default = "#59bbeb")
cssSecondaryColor	string	"#055e89"	Value of the widget secondary color (default = "#055e89")

Testing

You can try the library localy by running the following command:

npm run test-widget

You can change widget parameteres for your tests by updating parameters in the following file: /tests/widget/index.js

Custom handlers

To set custom handlers on events, you can write your own functions and attach it to the widget events. Here is an example:

const myCustomFunction = (event) => {
    console.log('Here is the code')
}
window.LintoUI = new LintoUI({
    ...,
    lintoCustomEvents: [{
        flag: 'my_custom_event',
        func: (event) => {
            myCustomFunction(event)
        }
    }]
})

Work with your own wakeup-word model

As mentionned before, “linto-web-client” bundler works with “webVoiceSdk”. If you want to use your own wakeup-word model, you’ll have to clone both repositories and update “linto-web-client” package.json as following:

Cloning repositories

#Cloning repositories
cd /your/local/repo
git clone git@github.com:linto-ai/WebVoiceSDK.git
git clone git@github.com:linto-ai/linto-web-client.git

Update package.json

cd /linto-web-client
nano package.json

Change “@linto-ai/webvoicesdk” devDependencie path

#package.json
{
    ...,
    "devDependencies": {
        "@linto-ai/webvoicesdk": "../WebVoiceSDK",
        ...
    }
}