{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# EN to MS" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This tutorial is available as an IPython notebook at [Malaya/example/en-ms-translation](https://github.com/huseinzol05/Malaya/tree/master/example/en-ms-translation).\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This module only trained on standard language structure, so it is not save to use it for local language structure.\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This interface deprecated, use HuggingFace interface instead.\n", " \n", "
" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ['CUDA_VISIBLE_DEVICES'] = ''" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import logging\n", "\n", "logging.basicConfig(level=logging.INFO)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 3.14 s, sys: 3.42 s, total: 6.56 s\n", "Wall time: 2.28 s\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/husein/dev/malaya/malaya/tokenizer.py:208: FutureWarning: Possible nested set at position 3372\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n", "/home/husein/dev/malaya/malaya/tokenizer.py:208: FutureWarning: Possible nested set at position 3890\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n" ] } ], "source": [ "%%time\n", "import malaya" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load dictionary\n", "\n", "```python\n", "def dictionary(**kwargs):\n", " \"\"\"\n", " Load dictionary {EN: MS} .\n", "\n", " Returns\n", " -------\n", " result: Dict[str, str]\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:malaya_boilerplate.huggingface:downloading frozen huseinzol05/v23-preprocessing/english-malay-200k.json\n" ] } ], "source": [ "dictionary = malaya.translation.en_ms.dictionary()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'ayam'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dictionary.get('chicken')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### List available Transformer models" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import warnings\n", "warnings.filterwarnings('default')" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/husein/dev/malaya/malaya/translation/en_ms.py:159: DeprecationWarning: `malaya.translation.en_ms.available_transformer` is deprecated, use `malaya.translation.en_ms.available_huggingface` instead\n", " warnings.warn('`malaya.translation.en_ms.available_transformer` is deprecated, use `malaya.translation.en_ms.available_huggingface` instead', DeprecationWarning)\n", "INFO:malaya.translation.en_ms:tested on FLORES200 EN-MS (eng_Latn-zsm_Latn) pair `dev` set, https://github.com/facebookresearch/flores/tree/main/flores200\n", "INFO:malaya.translation.en_ms:for noisy, tested on noisy augmented FLORES200 EN-MS (eng_Latn-zsm_Latn) pair `dev` set, https://github.com/huseinzol05/malay-dataset/tree/master/translation/nllb-noisy-dev-augmentation\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Size (MB)Quantized Size (MB)BLEUSacreBLEU VerboseSacreBLEU-chrF++-FLORES200Suggested length
small42.713.439.80538780.2/63.8/52.8/44.4 (BP = 0.997 ratio = 0.997 ...64.46256
base23482.742.21071386.3/73.3/64.1/56.8 (BP = 0.985 ratio = 0.985 ...66.28256
bigbird24663.739.09071770.5/46.7/32.4/22.9 (BP = 0.989 ratio = 0.989 ...63.961024
small-bigbird50.413.136.9019567.0/43.8/30.1/21.0 (BP = 1.000 ratio = 1.028 ...62.851024
noisy-base23482.741.82783173.1/49.7/35.3/25.4 (BP = 0.985 ratio = 0.985 ...66.46256
\n", "
" ], "text/plain": [ " Size (MB) Quantized Size (MB) BLEU \\\n", "small 42.7 13.4 39.805387 \n", "base 234 82.7 42.210713 \n", "bigbird 246 63.7 39.090717 \n", "small-bigbird 50.4 13.1 36.90195 \n", "noisy-base 234 82.7 41.827831 \n", "\n", " SacreBLEU Verbose \\\n", "small 80.2/63.8/52.8/44.4 (BP = 0.997 ratio = 0.997 ... \n", "base 86.3/73.3/64.1/56.8 (BP = 0.985 ratio = 0.985 ... \n", "bigbird 70.5/46.7/32.4/22.9 (BP = 0.989 ratio = 0.989 ... \n", "small-bigbird 67.0/43.8/30.1/21.0 (BP = 1.000 ratio = 1.028 ... \n", "noisy-base 73.1/49.7/35.3/25.4 (BP = 0.985 ratio = 0.985 ... \n", "\n", " SacreBLEU-chrF++-FLORES200 Suggested length \n", "small 64.46 256 \n", "base 66.28 256 \n", "bigbird 63.96 1024 \n", "small-bigbird 62.85 1024 \n", "noisy-base 66.46 256 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "malaya.translation.en_ms.available_transformer()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Transformer models\n", "\n", "```python\n", "def transformer(model: str = 'base', quantized: bool = False, **kwargs):\n", " \"\"\"\n", " Load Transformer encoder-decoder model to translate EN-to-MS.\n", "\n", " Parameters\n", " ----------\n", " model : str, optional (default='base')\n", " Model architecture supported. Allowed values:\n", "\n", " * ``'small'`` - Transformer SMALL parameters.\n", " * ``'base'`` - Transformer BASE parameters.\n", " * ``'bigbird'`` - BigBird BASE parameters.\n", " * ``'small-bigbird'`` - BigBird SMALL parameters.\n", " \n", " quantized : bool, optional (default=False)\n", " if True, will load 8-bit quantized model. \n", " Quantized model not necessary faster, totally depends on the machine.\n", "\n", " Returns\n", " -------\n", " result: model\n", " if `bigbird` in model, return malaya.model.bigbird.Translation\n", " else, return malaya.model.tf.Translation\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/husein/dev/malaya/malaya/translation/en_ms.py:209: DeprecationWarning: `malaya.translation.en_ms.transformer` is deprecated, use `malaya.translation.en_ms.huggingface` instead\n", " warnings.warn(\n", "INFO:malaya_boilerplate.frozen_graph:running home/husein/.cache/huggingface/hub/models--huseinzol05--translation-en-ms-base/snapshots/a2f02ffbb51f5c2226126d4fa9a02f7aa36d20be using device /device:CPU:0\n", "2022-10-21 12:33:35.533093: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n", "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2022-10-21 12:33:35.537338: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n", "2022-10-21 12:33:35.537359: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: husein-MS-7D31\n", "2022-10-21 12:33:35.537363: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: husein-MS-7D31\n", "2022-10-21 12:33:35.537427: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program\n", "2022-10-21 12:33:35.537449: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.141.3\n", "INFO:malaya_boilerplate.frozen_graph:running home/husein/.cache/huggingface/hub/models--huseinzol05--translation-en-ms-small/snapshots/154b07d08054ad5ad65c7dba4e1a5d49762dce85 using device /device:CPU:0\n" ] } ], "source": [ "transformer = malaya.translation.en_ms.transformer()\n", "transformer_small = malaya.translation.en_ms.transformer(model = 'small')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Quantized model\n", "\n", "To load 8-bit quantized model, simply pass `quantized = True`, default is `False`.\n", "\n", "We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "quantized_transformer = malaya.translation.en_ms.transformer(quantized = True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Translate\n", "\n", "#### Using greedy decoder\n", "\n", "```python\n", "def greedy_decoder(self, strings: List[str]):\n", " \"\"\"\n", " translate list of strings.\n", "\n", " Parameters\n", " ----------\n", " strings : List[str]\n", "\n", " Returns\n", " -------\n", " result: List[str]\n", " \"\"\"\n", "```\n", "\n", "#### Using beam decoder\n", "\n", "```python\n", "def beam_decoder(self, strings: List[str], beam_size: int = 3, temperature: float = 0.5):\n", " \"\"\"\n", " translate list of strings using beam decoder. \n", " Currently only `noisy` models supported `beam_size` and `temperature` parameters.\n", "\n", " Parameters\n", " ----------\n", " strings : List[str]\n", " beam_size: int, optional (default=3)\n", " temperature: float, optional (default=0.5)\n", "\n", " Returns\n", " -------\n", " result: List[str]\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from pprint import pprint" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('KUALA LUMPUR, July 1 - Datuk Seri Anwar Ibrahim is not suitable to as the '\n", " 'prime minister candidate as he is allegedly not \"popular\" among the Malays, '\n", " 'Tun Dr Mahathir Mohamad claimed. The former prime minister reportedly said '\n", " 'the PKR president needs someone like himself in order to acquire support '\n", " 'from the Malays and win the election.')\n" ] } ], "source": [ "# https://www.malaymail.com/news/malaysia/2020/07/01/dr-mahathir-again-claims-anwar-lacks-popularity-with-malays-to-be-pakatans/1880420\n", "\n", "string_news1 = 'KUALA LUMPUR, July 1 - Datuk Seri Anwar Ibrahim is not suitable to as the prime minister candidate as he is allegedly not \"popular\" among the Malays, Tun Dr Mahathir Mohamad claimed. The former prime minister reportedly said the PKR president needs someone like himself in order to acquire support from the Malays and win the election.'\n", "pprint(string_news1)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('(CNN)New York Attorney General Letitia James on Monday ordered the Black '\n", " 'Lives Matter Foundation -- which she said is not affiliated with the larger '\n", " 'Black Lives Matter movement -- to stop collecting donations in New York. \"I '\n", " 'ordered the Black Lives Matter Foundation to stop illegally accepting '\n", " 'donations that were intended for the #BlackLivesMatter movement. This '\n", " 'foundation is not affiliated with the movement, yet it accepted countless '\n", " 'donations and deceived goodwill,\" James tweeted.')\n" ] } ], "source": [ "# https://edition.cnn.com/2020/07/06/politics/new-york-attorney-general-blm/index.html\n", "\n", "string_news2 = '(CNN)New York Attorney General Letitia James on Monday ordered the Black Lives Matter Foundation -- which she said is not affiliated with the larger Black Lives Matter movement -- to stop collecting donations in New York. \"I ordered the Black Lives Matter Foundation to stop illegally accepting donations that were intended for the #BlackLivesMatter movement. This foundation is not affiliated with the movement, yet it accepted countless donations and deceived goodwill,\" James tweeted.'\n", "pprint(string_news2)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Amongst the wide-ranging initiatives proposed are a sustainable food '\n", " 'labelling framework, a reformulation of processed foods, and a '\n", " 'sustainability chapter in all EU bilateral trade agreements. The EU also '\n", " 'plans to publish a proposal for a legislative framework for sustainable food '\n", " 'systems by 2023 to ensure all foods on the EU market become increasingly '\n", " 'sustainable.')\n" ] } ], "source": [ "# https://www.thestar.com.my/business/business-news/2020/07/04/malaysia-worries-new-eu-food-rules-could-hurt-palm-oil-exports\n", "\n", "string_news3 = 'Amongst the wide-ranging initiatives proposed are a sustainable food labelling framework, a reformulation of processed foods, and a sustainability chapter in all EU bilateral trade agreements. The EU also plans to publish a proposal for a legislative framework for sustainable food systems by 2023 to ensure all foods on the EU market become increasingly sustainable.'\n", "pprint(string_news3)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('This page shares my best articles to read on topics like health, happiness, '\n", " 'creativity, productivity and more. The central question that drives my work '\n", " 'is, “How can we live better?” To answer that question, I like to write about '\n", " 'science-based ways to solve practical problems.')\n" ] } ], "source": [ "# https://jamesclear.com/articles\n", "\n", "string_article1 = 'This page shares my best articles to read on topics like health, happiness, creativity, productivity and more. The central question that drives my work is, “How can we live better?” To answer that question, I like to write about science-based ways to solve practical problems.'\n", "pprint(string_article1)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Fuzzy matching at scale. From 3.7 hours to 0.2 seconds. How to perform '\n", " 'intelligent string matching in a way that can scale to even the biggest data '\n", " 'sets. Data in the real world is messy. Dealing with messy data sets is '\n", " 'painful and burns through time which could be spent analysing the data '\n", " 'itself.')\n" ] } ], "source": [ "# https://towardsdatascience.com/fuzzy-matching-at-scale-84f2bfd0c536\n", "\n", "string_article2 = 'Fuzzy matching at scale. From 3.7 hours to 0.2 seconds. How to perform intelligent string matching in a way that can scale to even the biggest data sets. Data in the real world is messy. Dealing with messy data sets is painful and burns through time which could be spent analysing the data itself.'\n", "pprint(string_article2)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Emmerdale is the debut studio album,songs were not released in the U.S <> '\n", " 'These songs were not released in the U.S. edition of said album and were '\n", " 'previously unavailable on any U.S. release.')\n" ] } ], "source": [ "random_string1 = 'i am in medical school.'\n", "random_string2 = 'Emmerdale is the debut studio album,songs were not released in the U.S <> These songs were not released in the U.S. edition of said album and were previously unavailable on any U.S. release.'\n", "pprint(random_string2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Translate transformer base" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['KUALA LUMPUR 1 Julai - Datuk Seri Anwar Ibrahim tidak sesuai menjadi calon '\n", " 'Perdana Menteri kerana beliau didakwa tidak \"popular\" dalam kalangan orang '\n", " 'Melayu, Tun Dr Mahathir Mohamad mendakwa, bekas Perdana Menteri itu '\n", " 'dilaporkan berkata Presiden PKR itu memerlukan seseorang seperti dirinya '\n", " 'bagi mendapatkan sokongan daripada orang Melayu dan memenangi pilihan raya.',\n", " '(CNN) Peguam Negara New York Letitia James pada hari Isnin memerintahkan '\n", " 'Black Lives Matter Foundation - yang menurutnya tidak berafiliasi dengan '\n", " 'gerakan Black Lives Matter yang lebih besar - untuk berhenti mengumpulkan '\n", " 'sumbangan di New York. \"Saya memerintahkan Black Lives Matter Foundation '\n", " 'untuk berhenti menerima sumbangan secara haram yang bertujuan untuk gerakan '\n", " '#BlackLivesMatter. Yayasan ini tidak berafiliasi dengan gerakan itu, namun '\n", " 'ia menerima banyak sumbangan dan muhibah yang ditipu,\" tweet James.',\n", " 'Di antara inisiatif luas yang diusulkan adalah kerangka pelabelan makanan '\n", " 'yang berkelanjutan, reformulasi makanan yang diproses, dan bab keberlanjutan '\n", " 'dalam semua perjanjian perdagangan dua hala EU. EU juga berencana untuk '\n", " 'menerbitkan proposal untuk kerangka perundangan untuk sistem makanan lestari '\n", " 'pada tahun 2023 untuk memastikan semua makanan di pasar EU menjadi semakin '\n", " 'lestari.']\n", "CPU times: user 24.3 s, sys: 14 s, total: 38.3 s\n", "Wall time: 11.6 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer.greedy_decoder([string_news1, string_news2, string_news3]))" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Halaman ini berkongsi artikel terbaik saya untuk dibaca mengenai topik '\n", " 'seperti kesihatan, kebahagiaan, kreativiti, produktiviti dan banyak lagi. '\n", " 'Soalan utama yang mendorong kerja saya adalah, \"Bagaimana kita dapat hidup '\n", " 'lebih baik?\" Untuk menjawab soalan itu, saya suka menulis mengenai kaedah '\n", " 'berasaskan sains untuk menyelesaikan masalah praktikal.',\n", " 'Pemadanan kabur pada skala. Dari 3.7 jam hingga 0.2 saat. Cara melakukan '\n", " 'pemadanan rentetan pintar dengan cara yang dapat meningkatkan bahkan set '\n", " 'data terbesar. Data di dunia nyata tidak kemas. Berurusan dengan set data '\n", " 'yang tidak kemas menyakitkan dan terbakar sepanjang masa yang dapat '\n", " 'dihabiskan untuk menganalisis data itu sendiri.']\n", "CPU times: user 15.9 s, sys: 9.21 s, total: 25.2 s\n", "Wall time: 6.32 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer.greedy_decoder([string_article1, string_article2]))" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['saya di sekolah perubatan.',\n", " 'Emmerdale adalah album studio debut, lagu-lagu tidak dikeluarkan di A.S <> '\n", " 'Lagu-lagu ini tidak dikeluarkan dalam edisi A.S. album tersebut dan '\n", " 'sebelumnya tidak tersedia pada sebarang pelepasan A.S.']\n", "CPU times: user 9.98 s, sys: 5.52 s, total: 15.5 s\n", "Wall time: 4.23 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer.greedy_decoder([random_string1, random_string2]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Translate transformer small" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['KUALA LUMPUR 1 Julai - Datuk Seri Anwar Ibrahim tidak sesuai kerana calon '\n", " 'perdana menteri kerana didakwa tidak \"popular\" dalam kalangan orang Melayu, '\n", " 'Tun Dr Mahathir Mohamad mendakwa. Bekas perdana menteri itu dilaporkan '\n", " 'berkata, presiden PKR itu memerlukan seseorang seperti dirinya sendiri untuk '\n", " 'memperoleh sokongan daripada orang Melayu dan memenangi pilihan raya.hari '\n", " 'ini, Datuk Seri Anwar Ibrahim tidak sesuai untuk menjadi calon',\n", " '(CNN) Peguam Negara New York Letitia James pada hari Isnin memerintahkan '\n", " 'Yayasan Black Lives Matter - yang menurutnya tidak berafiliasi dengan '\n", " 'gerakan Black Lives Matter yang lebih besar - untuk berhenti mengumpulkan '\n", " 'sumbangan di New York. \"Saya memerintahkan Yayasan Black Lives Matter untuk '\n", " 'berhenti menerima sumbangan secara haram yang bertujuan untuk gerakan '\n", " '#BlackLivesMatter. Yayasan ini tidak berafiliasi dengan gerakan itu, namun '\n", " 'ia menerima banyak sumbangan dan muhibah yang menipu,\" tweet James.',\n", " 'Amongst inisiatif luas yang dicadangkan adalah kerangka kerja kerja kerja '\n", " 'makanan yang berkelanjutan, penyusunan semula makanan yang diproses, dan bab '\n", " 'kelestarian dalam semua perjanjian perdagangan dua hala EU. EU juga '\n", " 'merancang untuk menerbitkan cadangan kerangka perundangan untuk sistem '\n", " 'makanan lestari pada tahun 2023 untuk memastikan semua makanan di pasaran EU '\n", " 'semakin lestari.']\n", "CPU times: user 3.69 s, sys: 773 ms, total: 4.46 s\n", "Wall time: 1.61 s\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer_small.greedy_decoder([string_news1, string_news2, string_news3]))" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Halaman ini berkongsi artikel terbaik saya untuk membaca topik seperti '\n", " 'kesihatan, kebahagiaan, kreativiti, produktiviti dan banyak lagi. Soalan '\n", " 'pusat yang mendorong karya saya adalah, \"Bagaimana kita dapat hidup lebih '\n", " 'baik?\" Untuk menjawab soalan itu, saya suka menulis mengenai cara berasaskan '\n", " 'sains untuk menyelesaikan masalah praktikal.',\n", " 'Pemadanan Fuzzy pada skala. Dari 3.7 jam hingga 0.2 saat. Cara melakukan '\n", " 'pemadanan rentetan pintar dengan cara yang dapat meningkatkan set data '\n", " 'terbesar bahkan. Data di dunia nyata tidak kemas. Berurusan dengan set data '\n", " 'yang tidak kemas menyakitkan dan terbakar melalui masa yang dapat dihabiskan '\n", " 'untuk menganalisis data itu sendiri.']\n", "CPU times: user 2.45 s, sys: 384 ms, total: 2.84 s\n", "Wall time: 738 ms\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer_small.greedy_decoder([string_article1, string_article2]))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['saya berada di sekolah perubatan.',\n", " 'Emmerdale adalah album studio sulung, lagu-lagu tidak dikeluarkan di A.S <> '\n", " 'Lagu-lagu ini tidak dikeluarkan di edisi A.S. yang dikatakan album dan '\n", " 'sebelumnya tidak tersedia di mana-mana pelepasan A.S.']\n", "CPU times: user 1.7 s, sys: 291 ms, total: 1.99 s\n", "Wall time: 535 ms\n" ] } ], "source": [ "%%time\n", "\n", "pprint(transformer_small.greedy_decoder([random_string1, random_string2]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### compare with Google translate using googletrans\n", "\n", "Install it by,\n", "\n", "```bash\n", "pip3 install googletrans==4.0.0rc1\n", "```" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "from googletrans import Translator\n", "\n", "translator = Translator()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "KUALA LUMPUR, 1 Julai - Datuk Seri Anwar Ibrahim tidak sesuai sebagai calon Perdana Menteri kerana dia tidak \"popular\" di kalangan orang Melayu, Tun Dr Mahathir Mohamad mendakwa.Bekas Perdana Menteri dilaporkan berkata presiden PKR memerlukan seseorang seperti dirinya untuk memperoleh sokongan daripada orang Melayu dan memenangi pilihan raya.\n" ] } ], "source": [ "r = translator.translate(string_news1, src='en', dest = 'ms')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(CNN) Peguam Negara New York, Letitia James pada hari Isnin mengarahkan Yayasan Black Lives Matter - yang dikatakannya tidak bergabung dengan pergerakan Black Lives Matter yang lebih besar - untuk berhenti mengumpul sumbangan di New York.\"Saya mengarahkan Yayasan Black Lives Matter untuk berhenti menerima sumbangan secara haram yang dimaksudkan untuk gerakan #BlackLivesMatter.\n" ] } ], "source": [ "r = translator.translate(string_news2, src='en', dest = 'ms')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Di antara inisiatif yang luas yang dicadangkan adalah rangka kerja pelabelan makanan yang mampan, pembaharuan makanan yang diproses, dan bab kemampanan dalam semua perjanjian perdagangan dua hala EU.EU juga merancang untuk menerbitkan cadangan untuk rangka kerja perundangan untuk sistem makanan lestari menjelang 2023 untuk memastikan semua makanan di pasaran EU menjadi semakin mampan.\n" ] } ], "source": [ "r = translator.translate(string_news3, src='en', dest = 'ms')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Halaman ini berkongsi artikel terbaik saya untuk membaca topik seperti kesihatan, kebahagiaan, kreativiti, produktiviti dan banyak lagi.Soalan utama yang mendorong kerja saya adalah, \"Bagaimana kita dapat hidup lebih baik?\"Untuk menjawab soalan itu, saya ingin menulis tentang cara berasaskan sains untuk menyelesaikan masalah praktikal.\n" ] } ], "source": [ "r = translator.translate(string_article1, src='en', dest = 'ms')\n", "print(r.text)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pencocokan kabur pada skala.Dari 3.7 jam hingga 0.2 saat.Bagaimana untuk melakukan padanan rentetan pintar dengan cara yang boleh skala ke set data terbesar.Data di dunia nyata adalah kemas.Berurusan dengan set data berantakan adalah menyakitkan dan terbakar melalui masa yang boleh dibelanjakan menganalisis data itu sendiri.\n" ] } ], "source": [ "r = translator.translate(string_article2, src='en', dest = 'ms')\n", "print(r.text)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }