Grapheme-to-Phoneme DBP
Contents
Grapheme-to-Phoneme DBP#
This tutorial is available as an IPython notebook at Malaya/example/phoneme.
This module only trained on standard language structure, so it is not save to use it for local language structure.
This module only trained on 600 samples from https://prpm.dbp.gov.my/ Glosari Dialek.
Explanation#
Phonemizer is a Grapheme-to-phoneme (G2P) conversion is the process of generating pronunciation for words based on their written form, for an example, https://prpm.dbp.gov.my/Cari1?keyword=acaq&d=150348&#LIHATSINI,
acaq
-> [A.tSAâÖ]
[1]:
%%time
import malaya
CPU times: user 6.15 s, sys: 1.52 s, total: 7.66 s
Wall time: 11.3 s
Use deep learning model#
Load LSTM + Bahdanau Attention phonemizer model.
If you are using Tensorflow 2, make sure Tensorflow Addons already installed,
pip install tensorflow-addons U
def deep_model(quantized=False, **kwargs):
"""
Load LSTM + Bahdanau Attention phonetic model,
originally from https://prpm.dbp.gov.my/ Glosari Dialek.
Original size 10.4MB, quantized size 2.77MB .
Parameters
----------
quantized : bool, optional (default=False)
if True, will load 8-bit quantized model.
Quantized model not necessary faster, totally depends on the machine.
Returns
-------
result: malaya.model.tf.Seq2SeqLSTM class
"""
[2]:
model = malaya.phoneme.deep_model()
Load Quantized model#
To load 8-bit quantized model, simply pass quantized = True
, default is False
.
We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine.
[3]:
quantized_model = malaya.phoneme.deep_model(quantized = True)
Load quantized model will cause accuracy drop.
Predict#
def predict(self, strings: List[str], beam_search: bool = False):
"""
Convert to target string.
Parameters
----------
strings : List[str]
beam_search : bool, (optional=False)
If True, use beam search decoder, else use greedy decoder.
Returns
-------
result: List[str]
"""
If want to speed up the inference, set beam_search = False
.
[4]:
model.predict(['saya suka makan ayam', 'ayaq acaq kotoq'])
[4]:
['sA.jA su.kA mA.kAn A.jAm', 'A.jAâÖ A.tSAâÖ kot.S\x8d)Ö']
[5]:
quantized_model.predict(['saya suka makan ayam', 'ayaq acaq kotoq'])
[5]:
['sA.jA su.kA mA.kAn A.jAm', 'A.jAâÖ A.tSAâÖ kot.S\x8d)Ö']
Limitation#
Not able to convert numbers to phoneme.
[10]:
model.predict(['123'])
[10]:
['A']
you have to use normalization like https://malaya.readthedocs.io/en/latest/load-num2word.html
[9]:
model.predict([malaya.num2word.to_cardinal(123)])
[9]:
['s«.ÒAt du.wA pu.luh ti.gA']