EN to MS HuggingFace#

This tutorial is available as an IPython notebook at Malaya/example/en-ms-translation-huggingface.

This module trained on standard language and augmented local language structures, proceed with caution.

[1]:

import os

os.environ['CUDA_VISIBLE_DEVICES'] = ''

[2]:

%%time

import malaya
import logging

logging.basicConfig(level=logging.INFO)

CPU times: user 3.54 s, sys: 3.2 s, total: 6.74 s
Wall time: 2.65 s

/home/husein/dev/malaya/malaya/tokenizer.py:208: FutureWarning: Possible nested set at position 3372
  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))
/home/husein/dev/malaya/malaya/tokenizer.py:208: FutureWarning: Possible nested set at position 3890
  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))

List available HuggingFace models#

[3]:

malaya.translation.en_ms.available_huggingface()

INFO:malaya.translation.en_ms:tested on FLORES200 EN-MS (eng_Latn-zsm_Latn) pair `dev` set, https://github.com/facebookresearch/flores/tree/main/flores200

[3]:
Size (MB) BLEU SacreBLEU Verbose SacreBLEU-chrF++-FLORES200 Suggested length
mesolitica/finetune-translation-t5-super-tiny-standard-bahasa-cased 50.7 39.188342 72.6/48.3/33.5/23.6 (BP = 0.960 ratio = 0.961 ... 64.03 256
mesolitica/finetune-translation-t5-tiny-standard-bahasa-cased 139 41.625536 73.4/50.1/35.7/25.7 (BP = 0.971 ratio = 0.972 ... 65.7 256
mesolitica/finetune-translation-t5-small-standard-bahasa-cased 242 43.937298 74.9/52.2/37.9/27.7 (BP = 0.976 ratio = 0.977 ... 67.43 256
mesolitica/finetune-translation-t5-base-standard-bahasa-cased 892 44.173559 74.7/52.3/38.0/28.0 (BP = 0.979 ratio = 0.979 ... 67.6 256

These huggingface models trained on:

  1. EN-MS dataset, https://huggingface.co/datasets/mesolitica/en-ms

  2. MS-EN dataset, https://huggingface.co/datasets/mesolitica/ms-en

  3. NLLB eng_Latn-zsm_Latn, https://github.com/huseinzol05/malay-dataset/tree/master/translation/laser

Load Transformer models#

def huggingface(model: str = 'mesolitica/finetune-translation-t5-small-standard-bahasa-cased', **kwargs):
    """
    Load HuggingFace model to translate EN-to-MS.

    Parameters
    ----------
    model: str, optional (default='mesolitica/finetune-translation-t5-small-standard-bahasa-cased')
        Check available models at `malaya.translation.en_ms.available_huggingface()`.

    Returns
    -------
    result: malaya.torch_model.huggingface.Generator
    """

[5]:

transformer = malaya.translation.en_ms.transformer()

INFO:malaya_boilerplate.frozen_graph:running home/husein/.cache/huggingface/hub/models--huseinzol05--translation-en-ms-base/snapshots/a2f02ffbb51f5c2226126d4fa9a02f7aa36d20be using device /device:CPU:0
2022-10-04 21:35:57.018780: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-04 21:35:57.022347: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-10-04 21:35:57.022374: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: husein-MS-7D31
2022-10-04 21:35:57.022378: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: husein-MS-7D31
2022-10-04 21:35:57.022491: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program
2022-10-04 21:35:57.022523: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.141.3

[6]:

transformer_huggingface = malaya.translation.en_ms.huggingface()

Translate#

def generate(self, strings: List[str], **kwargs):
    """
    Generate texts from the input.

    Parameters
    ----------
    strings : List[str]
    **kwargs: vector arguments pass to huggingface `generate` method.
        Read more at https://huggingface.co/docs/transformers/main_classes/text_generation

    Returns
    -------
    result: List[str]
    """

For better results, always split by end of sentences.

[7]:

from pprint import pprint

[8]:

# https://www.malaymail.com/news/malaysia/2020/07/01/dr-mahathir-again-claims-anwar-lacks-popularity-with-malays-to-be-pakatans/1880420

string_news1 = 'KUALA LUMPUR, July 1 - Datuk Seri Anwar Ibrahim is not suitable to as the prime minister candidate as he is allegedly not "popular" among the Malays, Tun Dr Mahathir Mohamad claimed. The former prime minister reportedly said the PKR president needs someone like himself in order to acquire support from the Malays and win the election.'
pprint(string_news1)

('KUALA LUMPUR, July 1 - Datuk Seri Anwar Ibrahim is not suitable to as the '
 'prime minister candidate as he is allegedly not "popular" among the Malays, '
 'Tun Dr Mahathir Mohamad claimed. The former prime minister reportedly said '
 'the PKR president needs someone like himself in order to acquire support '
 'from the Malays and win the election.')

[9]:

# https://edition.cnn.com/2020/07/06/politics/new-york-attorney-general-blm/index.html

string_news2 = '(CNN)New York Attorney General Letitia James on Monday ordered the Black Lives Matter Foundation -- which she said is not affiliated with the larger Black Lives Matter movement -- to stop collecting donations in New York. "I ordered the Black Lives Matter Foundation to stop illegally accepting donations that were intended for the #BlackLivesMatter movement. This foundation is not affiliated with the movement, yet it accepted countless donations and deceived goodwill," James tweeted.'
pprint(string_news2)

('(CNN)New York Attorney General Letitia James on Monday ordered the Black '
 'Lives Matter Foundation -- which she said is not affiliated with the larger '
 'Black Lives Matter movement -- to stop collecting donations in New York. "I '
 'ordered the Black Lives Matter Foundation to stop illegally accepting '
 'donations that were intended for the #BlackLivesMatter movement. This '
 'foundation is not affiliated with the movement, yet it accepted countless '
 'donations and deceived goodwill," James tweeted.')

[10]:

# https://www.thestar.com.my/business/business-news/2020/07/04/malaysia-worries-new-eu-food-rules-could-hurt-palm-oil-exports

string_news3 = 'Amongst the wide-ranging initiatives proposed are a sustainable food labelling framework, a reformulation of processed foods, and a sustainability chapter in all EU bilateral trade agreements. The EU also plans to publish a proposal for a legislative framework for sustainable food systems by 2023 to ensure all foods on the EU market become increasingly sustainable.'
pprint(string_news3)

('Amongst the wide-ranging initiatives proposed are a sustainable food '
 'labelling framework, a reformulation of processed foods, and a '
 'sustainability chapter in all EU bilateral trade agreements. The EU also '
 'plans to publish a proposal for a legislative framework for sustainable food '
 'systems by 2023 to ensure all foods on the EU market become increasingly '
 'sustainable.')

[11]:

# https://jamesclear.com/articles

string_article1 = 'This page shares my best articles to read on topics like health, happiness, creativity, productivity and more. The central question that drives my work is, “How can we live better?” To answer that question, I like to write about science-based ways to solve practical problems.'
pprint(string_article1)

('This page shares my best articles to read on topics like health, happiness, '
 'creativity, productivity and more. The central question that drives my work '
 'is, “How can we live better?” To answer that question, I like to write about '
 'science-based ways to solve practical problems.')

[12]:

%%time

pprint(transformer.greedy_decoder([string_news1, string_news2, string_news3, string_article1]))

['KUALA LUMPUR 1 Julai - Datuk Seri Anwar Ibrahim tidak sesuai menjadi calon '
 'Perdana Menteri kerana beliau didakwa tidak "popular" dalam kalangan orang '
 'Melayu, Tun Dr Mahathir Mohamad mendakwa, bekas Perdana Menteri itu '
 'dilaporkan berkata Presiden PKR itu memerlukan seseorang seperti dirinya '
 'bagi mendapatkan sokongan daripada orang Melayu dan memenangi pilihan raya.',
 '(CNN) Peguam Negara New York Letitia James pada hari Isnin memerintahkan '
 'Black Lives Matter Foundation - yang menurutnya tidak berafiliasi dengan '
 'gerakan Black Lives Matter yang lebih besar - untuk berhenti mengumpulkan '
 'sumbangan di New York. "Saya memerintahkan Black Lives Matter Foundation '
 'untuk berhenti menerima sumbangan secara haram yang bertujuan untuk gerakan '
 '#BlackLivesMatter. Yayasan ini tidak berafiliasi dengan gerakan itu, namun '
 'ia menerima banyak sumbangan dan muhibah yang ditipu," tweet James.',
 'Di antara inisiatif luas yang diusulkan adalah kerangka pelabelan makanan '
 'yang berkelanjutan, reformulasi makanan yang diproses, dan bab keberlanjutan '
 'dalam semua perjanjian perdagangan dua hala EU. EU juga berencana untuk '
 'menerbitkan proposal untuk kerangka perundangan untuk sistem makanan lestari '
 'pada tahun 2023 untuk memastikan semua makanan di pasar EU menjadi semakin '
 'lestari.',
 'Halaman ini berkongsi artikel terbaik saya untuk dibaca mengenai topik '
 'seperti kesihatan, kebahagiaan, kreativiti, produktiviti dan banyak lagi. '
 'Soalan utama yang mendorong kerja saya adalah, "Bagaimana kita dapat hidup '
 'lebih baik?" Untuk menjawab soalan itu, saya suka menulis mengenai kaedah '
 'berasaskan sains untuk menyelesaikan masalah praktikal.']
CPU times: user 11.4 s, sys: 2.17 s, total: 13.6 s
Wall time: 6.37 s

[13]:

%%time

pprint(transformer_huggingface.generate([string_news1, string_news2, string_news3, string_article1],
                                       max_length = 1000))

['KUALA LUMPUR 1 Julai - Datuk Seri Anwar Ibrahim tidak sesuai sebagai calon '
 'perdana menteri kerana beliau didakwa tidak "popular" dalam kalangan orang '
 'Melayu, kata Tun Dr Mahathir Mohamad. Bekas perdana menteri itu dilaporkan '
 'berkata presiden PKR memerlukan seseorang seperti dirinya untuk mendapatkan '
 'sokongan daripada orang Melayu dan memenangi pilihan raya.',
 '(CNN) Peguam Negara New York Letitia James pada hari Isnin memerintahkan '
 'Black Lives Matter Foundation - yang menurutnya tidak berafiliasi dengan '
 'gerakan Black Lives Matter yang lebih besar - untuk berhenti mengumpulkan '
 'sumbangan di New York. "Saya memerintahkan Black Lives Matter Foundation '
 'untuk berhenti menerima sumbangan secara haram yang ditujukan untuk gerakan '
 '#BlackLivesMatter. Yayasan ini tidak berafiliasi dengan gerakan itu, namun '
 'ia menerima banyak sumbangan dan menipu muhibah," tweet James.',
 'Di antara inisiatif yang dicadangkan adalah kerangka pelabelan makanan '
 'lestari, reformasi makanan yang diproses, dan bab kelestarian dalam semua '
 'perjanjian perdagangan dua hala EU. EU juga merancang untuk menerbitkan '
 'cadangan untuk kerangka perundangan untuk sistem makanan lestari pada tahun '
 '2023 untuk memastikan semua makanan di pasaran EU menjadi semakin lestari.',
 'Halaman ini berkongsi artikel terbaik saya untuk dibaca mengenai topik '
 'seperti kesihatan, kebahagiaan, kreativiti, produktiviti dan banyak lagi. '
 'Soalan utama yang mendorong karya saya adalah,  Bagaimana kita dapat hidup '
 'lebih baik? "Untuk menjawab soalan itu, saya ingin menulis mengenai kaedah '
 'berasaskan sains untuk menyelesaikan masalah praktikal.']
CPU times: user 13.8 s, sys: 16 ms, total: 13.9 s
Wall time: 1.21 s

compare with Google translate using googletrans#

Install it by,

pip3 install googletrans==4.0.0rc1

[14]:

from googletrans import Translator

translator = Translator()

[15]:

strings = [string_news1, string_news2, string_news3, string_article1]

[16]:

for t in strings:
    r = translator.translate(t, src='en', dest = 'ms')
    print(r.text)

KUALA LUMPUR, 1 Julai - Datuk Seri Anwar Ibrahim tidak sesuai sebagai calon Perdana Menteri kerana dia tidak "popular" di kalangan orang Melayu, Tun Dr Mahathir Mohamad mendakwa.Bekas Perdana Menteri dilaporkan berkata presiden PKR memerlukan seseorang seperti dirinya untuk memperoleh sokongan daripada orang Melayu dan memenangi pilihan raya.
(CNN) Peguam Negara New York, Letitia James pada hari Isnin mengarahkan Yayasan Black Lives Matter - yang dikatakannya tidak bergabung dengan gerakan Black Lives Matter yang lebih besar - untuk berhenti mengumpul sumbangan di New York."Saya mengarahkan Yayasan Black Lives Matter untuk berhenti menerima sumbangan secara haram yang dimaksudkan untuk gerakan #BlackLivesMatter.
Di antara inisiatif yang luas yang dicadangkan adalah rangka kerja pelabelan makanan yang mampan, pembaharuan makanan yang diproses, dan bab kemampanan dalam semua perjanjian perdagangan dua hala EU.EU juga merancang untuk menerbitkan cadangan untuk rangka kerja perundangan untuk sistem makanan lestari menjelang 2023 untuk memastikan semua makanan di pasaran EU menjadi semakin mampan.
Halaman ini berkongsi artikel terbaik saya untuk membaca topik seperti kesihatan, kebahagiaan, kreativiti, produktiviti dan banyak lagi.Soalan utama yang mendorong kerja saya adalah, "Bagaimana kita dapat hidup lebih baik?"Untuk menjawab soalan itu, saya ingin menulis tentang cara berasaskan sains untuk menyelesaikan masalah praktikal.