{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# MS to EN" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This tutorial is available as an IPython notebook at [Malaya/example/ms-en-translation](https://github.com/huseinzol05/Malaya/tree/master/example/ms-en-translation).\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This module only trained on standard language structure, so it is not save to use it for local language structure.\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This interface deprecated, use HuggingFace interface instead.\n", " \n", "
" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 3.65 s, sys: 3.07 s, total: 6.71 s\n", "Wall time: 2.8 s\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/husein/dev/malaya/malaya/tokenizer.py:208: FutureWarning: Possible nested set at position 3372\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n", "/home/husein/dev/malaya/malaya/tokenizer.py:208: FutureWarning: Possible nested set at position 3890\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n" ] } ], "source": [ "%%time\n", "\n", "import malaya\n", "import logging\n", "\n", "logging.basicConfig(level=logging.INFO)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load dictionary\n", "\n", "```python\n", "def dictionary(**kwargs):\n", " \"\"\"\n", " Load dictionary {MS: EN} .\n", "\n", " Returns\n", " -------\n", " result: Dict[str, str]\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "dictionary = malaya.translation.ms_en.dictionary()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'chicken'" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dictionary.get('ayam')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### List available Transformer models" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:malaya.translation.ms_en:tested on FLORES200 MS-EN (zsm_Latn-eng_Latn) pair `dev` set, https://github.com/facebookresearch/flores/tree/main/flores200\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Size (MB)Quantized Size (MB)BLEUSacreBLEU VerboseSacreBLEU-chrF++-FLORES200Suggested length
small42.713.435.39256168.4/43.4/29.5/20.5 (BP = 0.967 ratio = 0.967 ...59.64256
base23482.740.35382371.4/48.1/34.5/25.1 (BP = 0.971 ratio = 0.972 ...63.24256
bigbird24663.738.87758870.2/46.3/32.5/23.2 (BP = 0.983 ratio = 0.983 ...62.491024
small-bigbird50.413.136.33139368.9/44.1/30.3/21.2 (BP = 0.973 ratio = 0.973 ...60.571024
noisy-base23482.740.38461171.4/48.2/34.6/25.1 (BP = 0.971 ratio = 0.971 ...63.31256
\n", "
" ], "text/plain": [ " Size (MB) Quantized Size (MB) BLEU \\\n", "small 42.7 13.4 35.392561 \n", "base 234 82.7 40.353823 \n", "bigbird 246 63.7 38.877588 \n", "small-bigbird 50.4 13.1 36.331393 \n", "noisy-base 234 82.7 40.384611 \n", "\n", " SacreBLEU Verbose \\\n", "small 68.4/43.4/29.5/20.5 (BP = 0.967 ratio = 0.967 ... \n", "base 71.4/48.1/34.5/25.1 (BP = 0.971 ratio = 0.972 ... \n", "bigbird 70.2/46.3/32.5/23.2 (BP = 0.983 ratio = 0.983 ... \n", "small-bigbird 68.9/44.1/30.3/21.2 (BP = 0.973 ratio = 0.973 ... \n", "noisy-base 71.4/48.2/34.6/25.1 (BP = 0.971 ratio = 0.971 ... \n", "\n", " SacreBLEU-chrF++-FLORES200 Suggested length \n", "small 59.64 256 \n", "base 63.24 256 \n", "bigbird 62.49 1024 \n", "small-bigbird 60.57 1024 \n", "noisy-base 63.31 256 " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "malaya.translation.ms_en.available_transformer()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Transformer models\n", "\n", "```python\n", "def transformer(model: str = 'base', quantized: bool = False, **kwargs):\n", " \"\"\"\n", " Load Transformer encoder-decoder model to translate MS-to-EN.\n", "\n", " Parameters\n", " ----------\n", " model : str, optional (default='base')\n", " Model architecture supported. Allowed values:\n", "\n", " * ``'small'`` - Transformer SMALL parameters.\n", " * ``'base'`` - Transformer BASE parameters.\n", " * ``'bigbird'`` - BigBird BASE parameters.\n", " * ``'small-bigbird'`` - BigBird SMALL parameters.\n", " * ``'noisy-base'`` - Transformer BASE parameters trained on noisy dataset.\n", "\n", " quantized : bool, optional (default=False)\n", " if True, will load 8-bit quantized model.\n", " Quantized model not necessary faster, totally depends on the machine.\n", "\n", " Returns\n", " -------\n", " result: model\n", " List of model classes:\n", "\n", " * if `bigbird` in model, return `malaya.model.bigbird.Translation`.\n", " * else, return `malaya.model.tf.Translation`.\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "transformer = malaya.translation.ms_en.transformer()\n", "transformer_small = malaya.translation.ms_en.transformer(model = 'small')\n", "transformer_large = malaya.translation.ms_en.transformer(model = 'large')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Quantized model\n", "\n", "To load 8-bit quantized model, simply pass `quantized = True`, default is `False`.\n", "\n", "We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:root:Load quantized model will cause accuracy drop.\n" ] } ], "source": [ "quantized_transformer = malaya.translation.ms_en.transformer(quantized = True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Translate\n", "\n", "#### Using greedy decoder\n", "\n", "```python\n", "def greedy_decoder(self, strings: List[str]):\n", " \"\"\"\n", " translate list of strings.\n", "\n", " Parameters\n", " ----------\n", " strings : List[str]\n", "\n", " Returns\n", " -------\n", " result: List[str]\n", " \"\"\"\n", "```\n", "\n", "#### Using beam decoder\n", "\n", "```python\n", "def beam_decoder(self, strings: List[str], beam_size: int = 3, temperature: float = 0.5):\n", " \"\"\"\n", " translate list of strings using beam decoder. \n", " Currently only `noisy` models supported `beam_size` and `temperature` parameters.\n", "\n", " Parameters\n", " ----------\n", " strings : List[str]\n", " beam_size: int, optional (default=3)\n", " temperature: float, optional (default=0.5)\n", "\n", " Returns\n", " -------\n", " result: List[str]\n", " \"\"\"\n", "```\n", "\n", "**For better results, always split by end of sentences**." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from pprint import pprint" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('TANGKAK - Tan Sri Muhyiddin Yassin berkata, beliau tidak mahu menyentuh '\n", " 'mengenai isu politik buat masa ini, sebaliknya mahu menumpukan kepada soal '\n", " 'kebajikan rakyat serta usaha merancakkan semula ekonomi negara yang terjejas '\n", " 'berikutan pandemik Covid-19. Perdana Menteri menjelaskan perkara itu ketika '\n", " 'berucap pada Majlis Bertemu Pemimpin bersama pemimpin masyarakat Dewan '\n", " 'Undangan Negeri (DUN) Gambir di Dewan Serbaguna Bukit Gambir hari ini.')\n" ] } ], "source": [ "# https://www.sinarharian.com.my/article/89678/BERITA/Politik/Saya-tidak-mahu-sentuh-isu-politik-Muhyiddin\n", "\n", "string_news1 = 'TANGKAK - Tan Sri Muhyiddin Yassin berkata, beliau tidak mahu menyentuh mengenai isu politik buat masa ini, sebaliknya mahu menumpukan kepada soal kebajikan rakyat serta usaha merancakkan semula ekonomi negara yang terjejas berikutan pandemik Covid-19. Perdana Menteri menjelaskan perkara itu ketika berucap pada Majlis Bertemu Pemimpin bersama pemimpin masyarakat Dewan Undangan Negeri (DUN) Gambir di Dewan Serbaguna Bukit Gambir hari ini.'\n", "pprint(string_news1)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('ALOR SETAR - Kemelut politik Pakatan Harapan (PH) belum berkesudahan apabila '\n", " 'masih gagal memuktamadkan calon Perdana Menteri yang dipersetujui bersama. '\n", " 'Ahli Parlimen Sik, Ahmad Tarmizi Sulaiman berkata, sehubungan itu pihaknya '\n", " 'mencadangkan mantan Pengerusi Parti Pribumi Bersatu Malaysia (Bersatu), Tun '\n", " 'Dr Mahathir Mohamad dan Presiden Parti Keadilan Rakyat (PKR), Datuk Seri '\n", " 'Anwar Ibrahim mengundurkan diri daripada politik sebagai jalan penyelesaian.')\n" ] } ], "source": [ "# https://www.sinarharian.com.my/article/90021/BERITA/Politik/Tun-Mahathir-Anwar-disaran-bersara-untuk-selesai-kemelut-politik\n", "\n", "string_news2 = 'ALOR SETAR - Kemelut politik Pakatan Harapan (PH) belum berkesudahan apabila masih gagal memuktamadkan calon Perdana Menteri yang dipersetujui bersama. Ahli Parlimen Sik, Ahmad Tarmizi Sulaiman berkata, sehubungan itu pihaknya mencadangkan mantan Pengerusi Parti Pribumi Bersatu Malaysia (Bersatu), Tun Dr Mahathir Mohamad dan Presiden Parti Keadilan Rakyat (PKR), Datuk Seri Anwar Ibrahim mengundurkan diri daripada politik sebagai jalan penyelesaian.'\n", "pprint(string_news2)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Menteri Kanan (Kluster Keselamatan) Datuk Seri Ismail Sabri Yaakob berkata, '\n", " 'kelonggaran itu diberi berikutan kerajaan menyedari masalah yang dihadapi '\n", " 'mereka untuk memperbaharui dokumen itu. Katanya, selain itu, bagi rakyat '\n", " 'asing yang pas lawatan sosial tamat semasa Perintah Kawalan Pergerakan (PKP) '\n", " 'pula boleh ke pejabat Jabatan Imigresen yang terdekat untuk mendapatkan '\n", " 'lanjutan tempoh.')\n" ] } ], "source": [ "string_news3 = 'Menteri Kanan (Kluster Keselamatan) Datuk Seri Ismail Sabri Yaakob berkata, kelonggaran itu diberi berikutan kerajaan menyedari masalah yang dihadapi mereka untuk memperbaharui dokumen itu. Katanya, selain itu, bagi rakyat asing yang pas lawatan sosial tamat semasa Perintah Kawalan Pergerakan (PKP) pula boleh ke pejabat Jabatan Imigresen yang terdekat untuk mendapatkan lanjutan tempoh.'\n", "pprint(string_news3)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Selain itu, pameran kerjaya membantu para pelajar menentukan kerjaya yang '\n", " 'akan diceburi oleh mereka. Seperti yang kita ketahui, pasaran kerjaya di '\n", " 'Malaysia sangat luas dan masih banyak sektor pekerjaan di negara ini yang '\n", " 'masih kosong kerana sukar untuk mencari tenaga kerja yang benar-benar '\n", " 'berkelayakan. Sebagai contohnya, sektor perubatan di Malaysia menghadapi '\n", " 'masalah kekurangan tenaga kerja yang kritikal, khususnya tenaga pakar '\n", " 'disebabkan peletakan jawatan oleh doktor dan pakar perubatan untuk memasuki '\n", " 'sektor swasta serta berkembangnya perkhidmatan kesihatan dan perubatan. '\n", " 'Setelah menyedari hakikat ini, para pelajar akan lebih berminat untuk '\n", " 'menceburi bidang perubatan kerana pameran kerjaya yang dilaksanakan amat '\n", " 'membantu memberikan pengetahuan am tentang kerjaya ini')\n" ] } ], "source": [ "# https://qcikgubm.blogspot.com/2018/02/contoh-soalan-dan-jawapan-karangan.html\n", "\n", "string_karangan = 'Selain itu, pameran kerjaya membantu para pelajar menentukan kerjaya yang akan diceburi oleh mereka. Seperti yang kita ketahui, pasaran kerjaya di Malaysia sangat luas dan masih banyak sektor pekerjaan di negara ini yang masih kosong kerana sukar untuk mencari tenaga kerja yang benar-benar berkelayakan. Sebagai contohnya, sektor perubatan di Malaysia menghadapi masalah kekurangan tenaga kerja yang kritikal, khususnya tenaga pakar disebabkan peletakan jawatan oleh doktor dan pakar perubatan untuk memasuki sektor swasta serta berkembangnya perkhidmatan kesihatan dan perubatan. Setelah menyedari hakikat ini, para pelajar akan lebih berminat untuk menceburi bidang perubatan kerana pameran kerjaya yang dilaksanakan amat membantu memberikan pengetahuan am tentang kerjaya ini'\n", "pprint(string_karangan)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Subfasal 6(b) bertujuan untuk memasukkan subseksyen baharu 39(3) dan (4) ke '\n", " 'dalam Akta 452. Subseksyen (3) yang dicadangkan bertujuan untuk menjadikan '\n", " 'suatu kesalahan bagi mana-mana orang yang meninggalkan Malaysia tanpa '\n", " 'membayar caruman yang tertunggak dan kena dibayar atau mengemukakan jaminan '\n", " 'bagi pembayarannya. Subseksyen (4) yang dicadangkan memperuntukkan bahawa '\n", " 'bagi maksud seksyen 39 Akta 452, “caruman” termasuklah apa-apa dividen atau '\n", " 'caj lewat bayar yang kena dibayar ke atas mana-mana caruman.')\n" ] } ], "source": [ "# https://www.parlimen.gov.my/bills-dewan-rakyat.html?uweb=dr#, RUU Kumpulan Wang Simpanan Pekerja (Pindaan) 2019\n", "\n", "string_parlimen = 'Subfasal 6(b) bertujuan untuk memasukkan subseksyen baharu 39(3) dan (4) ke dalam Akta 452. Subseksyen (3) yang dicadangkan bertujuan untuk menjadikan suatu kesalahan bagi mana-mana orang yang meninggalkan Malaysia tanpa membayar caruman yang tertunggak dan kena dibayar atau mengemukakan jaminan bagi pembayarannya. Subseksyen (4) yang dicadangkan memperuntukkan bahawa bagi maksud seksyen 39 Akta 452, “caruman” termasuklah apa-apa dividen atau caj lewat bayar yang kena dibayar ke atas mana-mana caruman.'\n", "pprint(string_parlimen)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('saya menikmati filem mengenai makhluk asing yang menyerang bumi. <> Saya '\n", " 'fikir fiksyen sains adalah genre yang luar biasa untuk apa sahaja. Sains '\n", " 'masa depan, teknologi, perjalanan masa, perjalanan FTL, semuanya adalah '\n", " 'konsep yang menarik. <> Saya sendiri peminat fiksyen sains!')\n" ] } ], "source": [ "string_random1 = 'saya menikmati filem mengenai makhluk asing yang menyerang bumi. <> Saya fikir fiksyen sains adalah genre yang luar biasa untuk apa sahaja. Sains masa depan, teknologi, perjalanan masa, perjalanan FTL, semuanya adalah konsep yang menarik. <> Saya sendiri peminat fiksyen sains!'\n", "pprint(string_random1)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Fiksyen sains <> saya menikmati filem mengenai makhluk asing yang menyerang '\n", " 'bumi. <> Fiksyen sains (sering dipendekkan menjadi SF atau sci-fi) adalah '\n", " 'genre fiksyen spekulatif, biasanya berurusan dengan konsep khayalan seperti '\n", " 'sains dan teknologi futuristik, perjalanan angkasa, perjalanan waktu, lebih '\n", " 'cepat daripada perjalanan ringan, alam semesta selari, dan kehidupan di luar '\n", " 'bumi .')\n" ] } ], "source": [ "string_random2 = 'Fiksyen sains <> saya menikmati filem mengenai makhluk asing yang menyerang bumi. <> Fiksyen sains (sering dipendekkan menjadi SF atau sci-fi) adalah genre fiksyen spekulatif, biasanya berurusan dengan konsep khayalan seperti sains dan teknologi futuristik, perjalanan angkasa, perjalanan waktu, lebih cepat daripada perjalanan ringan, alam semesta selari, dan kehidupan di luar bumi .'\n", "pprint(string_random2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Translate transformer base" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['TANGKAK - Tan Sri Muhyiddin Yassin said he did not want to touch on '\n", " 'political issues at the moment, instead focusing on the welfare of the '\n", " \"people and efforts to revitalize the affected country's economy following \"\n", " 'the Covid-19 pandemic. The prime minister explained the matter when speaking '\n", " 'at a Leadership Meeting with Gambir State Assembly (DUN) leaders at the '\n", " 'Bukit Gambir Multipurpose Hall today.',\n", " 'ALOR SETAR - Pakatan Harapan (PH) political turmoil has not ended when it '\n", " \"has failed to finalize the Prime Minister's candidate agreed upon. Sik MP \"\n", " 'Ahmad Tarmizi Sulaiman said he had suggested former United Nations (UN) '\n", " \"Indigenous Party chairman Tun Dr Mahathir Mohamad and People's Justice Party \"\n", " '(PKR) president Datuk Seri Anwar Ibrahim resign from politics as a solution.',\n", " 'Senior Minister (Security Cluster) Datuk Seri Ismail Sabri Yaakob said the '\n", " 'relaxation was given as the government was aware of the problems they had to '\n", " 'renew the document. He added that for foreigners who had passed the social '\n", " 'visit during the Movement Control Order (CPP) they could go to the nearest '\n", " 'Immigration Department office for further extension.']\n", "CPU times: user 23.9 s, sys: 14.5 s, total: 38.4 s\n", "Wall time: 9.95 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer.greedy_decoder([string_news1, string_news2, string_news3]))" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['TANGKAK - Tan Sri Muhyiddin Yassin said he did not want to touch on '\n", " 'political issues at the moment, instead focusing on the welfare of the '\n", " \"people and efforts to revitalize the affected country's economy following \"\n", " 'the Covid-19 pandemic. The prime minister explained the matter when speaking '\n", " 'at a Leadership Meeting with Gambir State Assembly (DUN) leaders at the '\n", " 'Bukit Gambir Multipurpose Hall today.',\n", " 'ALOR SETAR - Pakatan Harapan (PH) political turmoil has not ended when it '\n", " \"has failed to finalize the Prime Minister's candidate agreed upon. Sik MP \"\n", " 'Ahmad Tarmizi Sulaiman said he had suggested former United Nations (UN) '\n", " \"Indigenous Party chairman Tun Dr Mahathir Mohamad and People's Justice Party \"\n", " '(PKR) president Datuk Seri Anwar Ibrahim resign from politics as a solution.',\n", " 'Senior Minister (Security Cluster) Datuk Seri Ismail Sabri Yaakob said the '\n", " 'relaxation was given as the government was aware of the problems they had to '\n", " 'renew the document. He added that for foreigners who had passed the social '\n", " 'visit during the Movement Control Order (CPP) they could go to the nearest '\n", " 'Immigration Department office for an extension.']\n", "CPU times: user 23.5 s, sys: 13.9 s, total: 37.5 s\n", "Wall time: 9.58 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(quantized_transformer.greedy_decoder([string_news1, string_news2, string_news3]))" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['In addition, career exhibitions help students determine their careers. As we '\n", " 'know, the career market in Malaysia is very broad and there are still many '\n", " 'job sectors in the country that are still vacant because it is difficult to '\n", " 'find a truly qualified workforce. For example, the medical sector in '\n", " 'Malaysia is facing a critical shortage of labor, especially specialists due '\n", " 'to the resignation of doctors and physicians to enter the private sector and '\n", " 'develop health and medical services. Upon realizing this fact, students will '\n", " 'be more interested in medicine because the exhibition careers are very '\n", " 'helpful in providing general knowledge of this career.',\n", " 'Subclause 6 (b) seeks to introduce new subsections 39 (3) and (4) into Act '\n", " '452. Subsection (3) proposed to make an offense for any person leaving '\n", " 'Malaysia without paying a deferred and payable contribution or filing a '\n", " 'guarantee for payment. Subsection (4) proposed provides that for the purpose '\n", " 'of section 39 of Act 452, the \"contribution\" includes any dividend or late '\n", " 'payment charge payable on any contribution.']\n", "CPU times: user 30.8 s, sys: 17 s, total: 47.7 s\n", "Wall time: 10.6 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer.greedy_decoder([string_karangan, string_parlimen]))" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['In addition, career exhibitions help students determine their careers. As we '\n", " 'know, the career market in Malaysia is very broad and there are still many '\n", " 'job sectors in the country that are still vacant because it is difficult to '\n", " 'find a truly qualified workforce. For example, the medical sector in '\n", " 'Malaysia is facing a critical shortage of labor, especially specialists due '\n", " 'to the resignation of doctors and physicians to enter the private sector and '\n", " 'develop health and medical services. Upon realizing this fact, students will '\n", " 'be more interested in the medical field as the career exhibitions are very '\n", " 'helpful to provide general knowledge of this career.',\n", " 'Subclause 6 (b) seeks to introduce new subsections 39 (3) and (4) into Act '\n", " '452. Subsection (3) proposed to make an offense for any person leaving '\n", " 'Malaysia without paying a deferred and payable contribution or to submit a '\n", " 'guarantee for his payment. Subsection (4) proposed provides that for the '\n", " 'purpose of section 39 of Act 452, the \"contribution\" includes any dividend '\n", " 'or late payment charge payable on any contribution.']\n", "CPU times: user 30.9 s, sys: 16.8 s, total: 47.7 s\n", "Wall time: 10.8 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(quantized_transformer.greedy_decoder([string_karangan, string_parlimen]))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['I enjoy movies about aliens attacking the earth. <> I think science fiction '\n", " 'is an incredible genre for anything. Future science, technology, time '\n", " \"travel, FTL travel, everything is an exciting concept. <> I'm a science \"\n", " 'fiction fan!',\n", " 'Science fiction <> I enjoy movies about aliens invading the earth. <> '\n", " 'Science fiction (often shortened to SF or sci-fi) is a genre of speculative '\n", " 'fiction, usually dealing with imaginary concepts such as science and '\n", " 'futuristic technology, space travel, time travel, faster than light travel, '\n", " 'parallel universe, and life abroad.']\n", "CPU times: user 19.1 s, sys: 10.3 s, total: 29.5 s\n", "Wall time: 6.77 s\n" ] } ], "source": [ "%%time\n", "\n", "result = transformer.greedy_decoder([string_random1, string_random2])\n", "pprint(result)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['I enjoy movies about aliens attacking the earth. <> I think science fiction '\n", " 'is an incredible genre for anything. Future science, technology, time '\n", " \"travel, FTL travel, everything is an exciting concept. <> I'm a science \"\n", " 'fiction fan!',\n", " 'Science fiction <> I enjoy movies about aliens invading the earth. <> '\n", " 'Science fiction (often shortened to SF or sci-fi) is a genre of speculative '\n", " 'fiction, usually dealing with imaginary concepts such as science and '\n", " 'futuristic technology, space travel, time travel, faster than light travel, '\n", " 'parallel universe, and life abroad.']\n", "CPU times: user 19.2 s, sys: 11.7 s, total: 30.9 s\n", "Wall time: 6.66 s\n" ] } ], "source": [ "%%time\n", "\n", "result = quantized_transformer.greedy_decoder([string_random1, string_random2])\n", "pprint(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Translate transformer small" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['TANGKAK - Tan Sri Muhyiddin Yassin said he did not want to touch on '\n", " 'political issues at this time, instead focusing on the welfare of the people '\n", " \"and efforts to revitalize the country's economy affected following the \"\n", " 'Covid-19 pandemic. The Prime Minister explained the matter when speaking at '\n", " 'the Leaders Meeting with the leaders of the Gambir State Assembly (DUN) '\n", " 'community at the Bukit Gambir Multipurpose Hall today.',\n", " 'ALOR SETAR - Pakatan Harapan (PH) political turmoil has not been expected '\n", " \"when it still fails to finalize the Prime Minister's candidate agreed \"\n", " 'together. Sik MP Ahmad Tarmizi Sulaiman said the party had suggested former '\n", " 'United Nations Indigenous Party (UN) chairman Tun Dr Mahathir Mohamad and '\n", " \"President of the People's Justice Party (PKR), Datuk Seri Anwar Ibrahim \"\n", " 'resigned from politics as a solution.',\n", " 'Senior Minister (Security Cluster) Datuk Seri Ismail Sabri Yaakob said the '\n", " 'relaxation was given as the government was aware of the problems they faced '\n", " 'to renew the document. He said in addition, for foreigners who had passed '\n", " 'the social visit expired during the Movement Control Order (PKP) could go to '\n", " 'the nearest Immigration Department office for further time.']\n", "CPU times: user 3.46 s, sys: 871 ms, total: 4.33 s\n", "Wall time: 1.5 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer_small.greedy_decoder([string_news1, string_news2, string_news3]))" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['In addition, career exhibitions help students determine their careers. As we '\n", " 'know, the career market in Malaysia is very broad and many employment '\n", " 'sectors in the country are still vacant because it is difficult to find a '\n", " 'truly qualified workforce. For example, the medical sector in Malaysia is '\n", " 'facing critical labor shortages, especially specialists as specialists '\n", " 'resign as well as medical professionals to enter the private sector and the '\n", " 'development of health and medical services. After realizing this fact that '\n", " 'students will be more interested in medicine as the exhibition of careers is '\n", " 'helping to provide general knowledge of this career in this career as this '\n", " 'career is especially in providing general knowledge of this career as this '\n", " 'career as this career is difficult to gain knowledge of this career as it is '\n", " 'difficult to gain knowledge of this career is difficult to find the career '\n", " 'as it is difficult to find the career as it is difficult to enter this '\n", " 'career as it is difficult to enter this career as it is difficult to gain '\n", " 'through this career is difficult to enter the private sector and medical '\n", " 'career as it is difficult to find the field of career as it is difficult to '\n", " 'gain knowledge of this career as it is difficult to find the field of career '\n", " 'as it is difficult to find the field of career as it is especially in the '\n", " 'field of career as it is difficult to find the field of this career as it is '\n", " 'difficult to find the field of career as it is difficult to find that it is '\n", " 'difficult to find that it is difficult to',\n", " 'Subclaal 6 (b) aims to include a new subsection of 39 (3) and (4) into Act '\n", " '452. Subsection (3) proposed aimed at making a mistake for any person who '\n", " 'leaves Malaysia without paying for outstanding contributions and paid or to '\n", " 'provide guarantees for his payment. Subsection (4) proposed provides that '\n", " 'for the purpose of section 39 of Act 452, \"contributions\" including any '\n", " 'dividend or late payment charge to be paid to any contribution.']\n", "CPU times: user 9.94 s, sys: 1.8 s, total: 11.7 s\n", "Wall time: 3.14 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer_small.greedy_decoder([string_karangan, string_parlimen]))" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['I enjoy movies about aliens attacking the earth. <> I think science fiction '\n", " 'is a great genre for whatever future science, technology, travel, FTL '\n", " 'travel, all of which is an interesting concept. <> I personally love science '\n", " 'fiction!',\n", " 'science fiction <> I enjoy movies about aliens who attack the earth. <> The '\n", " 'science fiction (often shortened to SF or sci-fi) is a speculative fiction '\n", " 'genre, usually dealing with the concept of imaginary science and futuristic '\n", " 'technology, space travel, travel, faster than light travel, parallel '\n", " 'universe, and outer life.']\n", "CPU times: user 2.64 s, sys: 402 ms, total: 3.04 s\n", "Wall time: 773 ms\n" ] } ], "source": [ "%%time\n", "\n", "result = transformer_small.greedy_decoder([string_random1, string_random2])\n", "pprint(result)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### compare with Google translate using googletrans\n", "\n", "Install it by,\n", "\n", "```bash\n", "pip3 install googletrans==4.0.0rc1\n", "```" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "from googletrans import Translator\n", "\n", "translator = Translator()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TANGKAK - Tan Sri Muhyiddin Yassin said he did not want to touch on political issues at this time, instead of focusing on the welfare of the people and efforts to regenerate the country's economy following the Covid -19 pandemic.The prime minister explained the matter when speaking at a ceremony with a leader of the Gambir State Assembly (DUN) community leader at the Bukit Gambir Multipurpose Hall today.\n" ] } ], "source": [ "r = translator.translate(string_news1, src='ms', dest = 'en')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ALOR SETAR - The Pakatan Harapan (PH) political turmoil has not ended when it fails to finalize the agreed prime ministerial candidate.Sik Member of Parliament Ahmad Tarmizi Sulaiman said he had suggested former United Indigenous Party (UN) chairman Tun Dr Mahathir Mohamad and the People's Justice Party (PKR) president Datuk Seri Anwar Ibrahim resigned from politics as a solution.\n" ] } ], "source": [ "r = translator.translate(string_news2, src='ms', dest = 'en')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Senior Minister (Security Cluster) Datuk Seri Ismail Sabri Yaakob said the relaxation was given as the government was aware of the problem they had to renew the document.He said, for foreigners, the social visit ended during the Movement Control Order (CPP) could go to the nearest Immigration Department's office for extension.\n" ] } ], "source": [ "r = translator.translate(string_news3, src='ms', dest = 'en')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "In addition, career exhibitions help students determine the careers they will be involved in.As we know, the career market in Malaysia is very broad and there are still many employment sectors in the country that are still vacant because it is difficult to find a truly qualified workforce.For example, the medical sector in Malaysia is facing critical workforce problems, especially experts due to the resignation of doctors and physicians to enter the private sector as well as the growth of health and medical services.Upon realizing this fact, students will be more interested in getting into medicine because their career exhibitions are very helpful in providing general knowledge of this career\n" ] } ], "source": [ "r = translator.translate(string_karangan, src='ms', dest = 'en')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Subfasal 6 (b) aims to include new subsections 39 (3) and (4) into Act 452. Subsection (3) proposed to make an offense for any person leaving Malaysia without paying an outstanding and payable contributionor submit a guarantee for his payment.Subsection (4) proposed providing that for the purposes of section 39 of Act 452, \"Contribution\" includes any dividends or late payments payable on any contribution.\n" ] } ], "source": [ "r = translator.translate(string_parlimen, src='ms', dest = 'en')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }