Skip to content

feat!: major performance & accuracy improvements in speech-to-text module #1132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
IgorSwat wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat!: major performance & accuracy improvements in speech-to-text module #1132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b8276eb
Optimal streaming algorithm
IgorSwat May 7, 2026
5b1c7b6
Merge branch '@is/speech-to-text-ultimate' of https://github.com/soft…
IgorSwat May 7, 2026
a82e6fc
Revert back to 100ms refresh rate
IgorSwat May 7, 2026
eccda55
Add CoreML whisper models
IgorSwat May 7, 2026
0f77124
Update model urls
IgorSwat May 8, 2026
6c6545d
Change default model for iOS devices
IgorSwat May 8, 2026
1dfafbe
Add explicit timeout parameter
IgorSwat May 10, 2026
66fde0e
Concurrency fixes & automatic cleaunp
IgorSwat May 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions apps/llm/app/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,6 @@ export default function Home() {
>
<Text style={styles.buttonText}>LLM Structured Output</Text>
</TouchableOpacity>
<TouchableOpacity
style={styles.button}
onPress={() => router.navigate('voice_chat/')}
>
<Text style={styles.buttonText}>Voice Chat</Text>
</TouchableOpacity>
<TouchableOpacity
style={styles.button}
onPress={() => router.navigate('multimodal_llm/')}
Expand Down
311 changes: 0 additions & 311 deletions apps/llm/app/voice_chat/index.tsx
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion apps/speech/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"metro-config": "^0.83.0",
"react": "19.2.5",
"react-native": "0.83.4",
"react-native-audio-api": "0.12.0",
"react-native-audio-api": "0.11.5",
Copy link
Copy Markdown
Member

@msluszniak msluszniak May 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey, why is that? We virtually never want to downgrade packages in demo apps.

Copy link
Copy Markdown
Contributor Author

@IgorSwat IgorSwat May 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

audio-api 0.12.0 causes build fails on iOS, and I think it's the same issue @benITo47 had when testing the 1.2.0 binaries some time ago.

Copy link
Copy Markdown
Member

@msluszniak msluszniak May 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please name when you have these fails? I don't have any on iOS simulator.

Copy link
Copy Markdown
Contributor Author

@IgorSwat IgorSwat May 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Tested on physical device.

Copy link
Copy Markdown
Member

@msluszniak msluszniak May 8, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mdydek could you look at this one? Maybe you have an intuition behind this error? @IgorSwat do you have iOS 26.2 on your physical device?

Copy link
Copy Markdown
Contributor Author

@IgorSwat IgorSwat May 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I even have 26.4.

"react-native-device-info": "^15.0.2",
"react-native-executorch": "workspace:*",
"react-native-executorch-expo-resource-fetcher": "workspace:*",
Expand Down
22 changes: 15 additions & 7 deletions apps/speech/screens/SpeechToTextScreen.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,25 @@ import { SafeAreaProvider, SafeAreaView } from 'react-native-safe-area-context';
import {
useSpeechToText,
WHISPER_TINY_EN,
WHISPER_TINY_EN_QUANTIZED,
WHISPER_TINY_EN_COREML,
WHISPER_BASE_EN,
WHISPER_BASE_EN_COREML,
WHISPER_SMALL_EN,
TranscriptionResult,
SpeechToTextProps,
WHISPER_SMALL_EN_COREML,
Copy link
Copy Markdown
Member

@msluszniak msluszniak May 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this is added after TranscriptionResult and SpeechToTextProps? ;p

} from 'react-native-executorch';
import { ModelPicker, ModelOption } from '../components/ModelPicker';

type STTModelSources = SpeechToTextProps['model'];

const MODELS: ModelOption<STTModelSources>[] = [
{ label: 'Whisper Tiny', value: WHISPER_TINY_EN },
{ label: 'Whisper Tiny Q', value: WHISPER_TINY_EN_QUANTIZED },
{ label: 'Whisper Base', value: WHISPER_BASE_EN },
{ label: 'Whisper Small', value: WHISPER_SMALL_EN },
{ label: 'Whisper Tiny EN (XNNPACK)', value: WHISPER_TINY_EN },
{ label: 'Whisper Tiny EN (CoreML)', value: WHISPER_TINY_EN_COREML },
{ label: 'Whisper Base EN (XNNPACK)', value: WHISPER_BASE_EN },
{ label: 'Whisper Base EN (CoreML)', value: WHISPER_BASE_EN_COREML },
{ label: 'Whisper Small EN (XNNPACK)', value: WHISPER_SMALL_EN },
{ label: 'Whisper Small EN (CoreML)', value: WHISPER_SMALL_EN_COREML },
];
import FontAwesome from '@expo/vector-icons/FontAwesome';
import {
Expand All @@ -45,9 +49,12 @@ import ErrorBanner from '../components/ErrorBanner';

const isSimulator = DeviceInfo.isEmulatorSync();

const DEFAULT_MODEL =
Platform.OS === 'ios' ? WHISPER_BASE_EN_COREML : WHISPER_TINY_EN;

export const SpeechToTextScreen = ({ onBack }: { onBack: () => void }) => {
const [selectedModel, setSelectedModel] =
useState<STTModelSources>(WHISPER_TINY_EN);
useState<STTModelSources>(DEFAULT_MODEL);

const model = useSpeechToText({
model: selectedModel,
Expand Down Expand Up @@ -148,7 +155,7 @@ export const SpeechToTextScreen = ({ onBack }: { onBack: () => void }) => {
recorder.current.onAudioReady(
{
sampleRate,
bufferLength: 0.1 * sampleRate,
bufferLength: 0.1 * sampleRate, // 100 ms
channelCount: 1,
},
({ buffer }) => {
Expand Down Expand Up @@ -178,6 +185,7 @@ export const SpeechToTextScreen = ({ onBack }: { onBack: () => void }) => {
try {
const streamIter = model.stream({
verbose: enableTimestamps,
timeout: 100,
});

for await (const { committed, nonCommitted } of streamIter) {
Expand Down
Loading
Loading