REPOSITORY OVERVIEW

Live repository statistics

★ 32Stars

⑂ 28Forks

◯ 3Open issues

◉ 32Watchers

README preview

VAD

VAD is a Flutter library for Voice Activity Detection (VAD) across iOS, Android, Web, macOS, Windows, and Linux platforms. This package allows applications to start and stop VAD-based listening and handle various VAD events seamlessly.

Under the hood, the VAD Package uses direct FFI bindings to ONNX Runtime for native platforms (iOS, Android, macOS, Windows, Linux) and dart:js_interop for Web. All platforms utilize the Silero VAD models with full-feature parity across platforms.

The package provides a simple API to start and stop VAD listening, configure VAD parameters, and handle VAD events such as speech start, speech end, errors, and misfires.

Note: Echo cancellation is not available on Windows and Linux platforms due to limitations in the underlying audio capture library.

Live Demo

Check out the VAD Package Example App to see the VAD Package in action on the Web platform.

Features

Cross-Platform Support: Works seamlessly on iOS, Android, Web, macOS, Windows, and Linux.
Event Streams: Listen to events such as speech start, real speech start, speech end, speech misfire, frame processed, and errors.
Silero V4 and V5 Models: Supports both Silero VAD v4 and v5 models.
16KB Page Size Support: Native Android libraries are properly aligned for 16KB page sizes, meeting Google Play requirements for Android 15+ devices.
Custom Audio Streams: Provide your own audio stream for advanced use cases like custom recording configurations or processing audio from non-microphone sources.

Getting Started

Prerequisites

Before integrating the VAD Package into your Flutter application, ensure that you have the necessary configurations for each target platform.

Web

To use VAD on the web, include the following scripts within the head and body tags respectively in the web/index.html file to load the necessary VAD libraries:

Option 1: Using CDN (Default)

<head>
  ...
  <script src="https://cdn.jsdelivr.net/npm/onnxruntime-web@1.22.0/dist/ort.wasm.min.js"></script>
  ...
</head>

Option 2: Using Local Assets (Offline/Self-hosted) If you prefer to bundle all assets locally instead of using CDN:

Download the required files to your assets/ directory:
Update your web/index.html:

<head>
  ...
  <script src="assets/ort.wasm.min.js"></script>
  ...
</head>

Configure your VAD handler to use local assets:

await vadHandler.startListening(
  baseAssetPath: '/assets/',        // For VAD model files
  onnxWASMBasePath: '/assets/',     // For ONNX Runtime WASM files
  // ... other parameters
);

You can also refer to the VAD Example App for a complete example.

Tip: Enable WASM multithreading (SharedArrayBuffer) for performance improvements

For Production, send the following headers in your server response:

Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

For Local, refer to the workaround applied in the GitHub Pages demo page for the example app. It is achieved with the inclusion of enable-threads.js and loading it in the web/index.html#L24 file in the example app.

iOS

For iOS, you need to configure microphone permissions and other settings in your Info.plist file.

Add Microphone Usage Description: Open ios/Runner/Info.plist and add the following entries to request microphone access:

<key>NSMicrophoneUsageDescription</key>
<string>This app needs access to the microphone for Voice Activity Detection.</string>

Configure Build Settings: Ensure that your Podfile includes the necessary build settings for microphone permissions:

post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
        '$(inherited)',
        'PERMISSION_MICROPHONE=1',
      ]
    end
  end
end

Android

For Android, configure the required permissions and build settings in your AndroidManifest.xml and build.gradle files.

Add Permissions: Open android/app/src/main/AndroidManifest.xml and add the following permissions:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>

Configure Build Settings: Open android/app/build.gradle and add the following settings:

android {
    compileSdkVersion 34
    ...
}

Installation

Add the VAD Package to your pubspec.yaml dependencies:

dependencies:
  flutter:
    sdk: flutter
  vad: ^0.0.5
  permission_handler: ^11.3.1

Then, run flutter pub get to fetch the packages.

Usage

Example

Below is a simple example demonstrating how to integrate and use the VAD Package in a Flutter application. For a more detailed example, check out the VAD Example App in the GitHub repository.

// main.dart
import 'package:flutter/material.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:vad/vad.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(title: const Text("VAD Example")),
        body: const MyHomePage(),
      ),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key});

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  final _vadHandler = VadHandler.create(isDebug: true);
  bool isListening = false;
  final List<String> receivedEvents = [];

  @override
  void initState() {
    super.initState();
    _setupVadHandler();
  }

  void _setupVadHandler() {
    _vadHandler.onSpeechStart.listen((_) {
      debugPrint('Speech detected.');
      setState(() {
        receivedEvents.add('Speech detected.');
      });
    });

    _vadHandler.onRealSpeechStart.listen((_) {
      debugPrint('Real speech start detected (not a misfire).');
      setState(() {
        receivedEvents.add('Real speech start detected (not a misfire).');
      });
    });

    _vadHandler.onSpeechEnd.listen((List<double> samples) {
      debugPrint('Speech ended, first 10 samples: ${samples.take(10).toList()}');
      setState(() {
        receivedEvents.add('Speech ended, first 10 samples: ${samples.take(10).toList()}');
      });
    });

    _vadHandler.onFrameProcessed.listen((frameData) {
      final isSpeech = frameData.isSpeech;
      final notSpeech = frameData.notSpeech;
      final firstFewSamples = frameData.frame.take(5).toList();

      debugPrint('Frame processed - Speech probability: $isSpeech, Not speech: $notSpeech');
      debugPrint('First few audio samples: $firstFewSamples');

      // You can use this for real-time audio processing
    });

    _vadHandler.onVADMisfire.listen((_) {
      debugPrint('VAD misfire detected.');
      setState(() {
        receivedEvents.add('VAD misfire detected.');
      });
    });

    _vadHandler.onError.listen((String message) {
      debugPrint('Error: $message');
      setState(() {
        receivedEvents.add('Error: $message');
      });
    });
  }

  @override
  void dispose() {
    _vadHandler.dispose(); // Note: dispose() is called without await in Widget.dispose()
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Padding(
      padding: const EdgeInsets.all(16),
      child: Column(
        mainAxisSize: MainAxisSize.min,
        children: [
          ElevatedButton.icon(
            onPressed: () async {
              if (isListening) {
                await _vadHandler.stopListening();
              } else {
                await _vadHandler.startListening();
              }
              setState(() {
                isListening = !isListening;
              });
            },
            icon: Icon(isListen