feat!: multilingual text-to-speech

[IgorSwat]

Description

Introduces major changes to the text-to-speech module based on Kokoro model, including:

• Multilingual text-to-speech - a set of complete pipelines & voices for different languages. A complete list of (currently) supported languages can be found below.
• Improved phonemization & speech quality - utilizing neural phonemization model as a fallback for the old lexicon-base phonemization significantly improves speech quality, particularly for non-standard, out of dictionary words.
• Timestamp-based audio cutting - an improve postprocessing algorithm, eliminates artifacts introduced by .pte model, resulting in cleaner, more natural speech.
• API changes: prepared for voice-cloning & custom, fine-tuned versions of Kokoro model.

Supported language current status:

• 🇺🇸 American English: ✅
• 🇬🇧 British English: ✅
• 🇫🇷 French: ✅
• 🇪🇸 Spanish: ✅
• 🇵🇹/🇧🇷 Portugese: ✅
• 🇮🇹 Italian: ✅
• 🇵🇱 Polish: ✅
• 🇩🇪 German: ✅
• 🇮🇳 Hindi: ✅
• 🇯🇵 Japanese: ❌ (coming soon)
• 🇨🇳 Mandarin Chinese: ❌ (coming soon)

Introduces a breaking change?

• Yes
• No

There are 2 major breaking changes introduced by this PR:

• Changed "synthezation from phonemes" API.

  Old API:

   const audioData = await tts.forwardFromPhonemes({
     phonemes:
       'ɐ mˈæn hˌu dˈʌzᵊnt tɹˈʌst hɪmsˈɛlf, kæn nˈɛvəɹ ɹˈiᵊli tɹˈʌst ˈɛniwˌʌn ˈɛls.',
   });
  

  New API:

  const audioData = await tts.forward({
    text:
      'ɐ mˈæn hˌu dˈʌzᵊnt tɹˈʌst hɪmsˈɛlf, kæn nˈɛvəɹ ɹˈiᵊli tɹˈʌst ˈɛniwˌʌn ˈɛls.',
     phonemize: false,  # Disables phonemization and treats text as phonemes
  });
  

• Changed predefined model - voice setups. Now both model files & voice/phonemization files are bundled together, due to languages like Polish or German having fine-tuned model weights.

  Old API:

  const model = useTextToSpeech({
    model: KOKORO_MEDIUM,
    voice: KOKORO_VOICE_AF_HEART,
  });
  

  New API:

  const model = useTextToSpeech(KOKORO_AMERICAN_ENGLISH_FEMALE_HEART);
  

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

Play around demo speech apps.

Unit tests for RNE-specific code will be added later on.
Phonemis package has it's own, wide range of unit tests implemented (see Phonemis repo)

Screenshots

Related issues

#712

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

[msluszniak]

You should also update the code in documentation and documentation in general. Also address lint warnings, there are plenty of them that you need to add to cspell ignore.

[msluszniak]

Also if this PR adds breaking change, please describe it directly below Introduces a breaking change? section in PR body.

[chmjkb]

a bunch of magic numbers, please explain

[IgorSwat]

Come on, this is just a frontend of a demo app. Those values are indeed random numbers which make it look good 😅

[chmjkb]

xD

[chmjkb]

i wonder who the fella is

[chmjkb]

this is very hard to read

[IgorSwat]

I agree, but there are 2 things which prevent me from making it more readable:

• Each time I try to improve it by just re-formatting indentation, the linter reverts it during committing phase
• It's hard to simplify the code logic itself, since we always need these "empty path" checks to check what comes from the JS side, and these checks generate most of the complexity of this piece of code.

[msluszniak]

the linter reverts it during committing phase

This is because we have clang-format call in pre-commit hook with LLVM default format enforced.

[chmjkb]

do we need those assertions?

[IgorSwat]

Yes, we do. Tagger, lexicon and neural model are all theoretically optional, so we need some conditional to decide whether we should pass en empty string or an existing value.