{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EN to MS longer text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "This tutorial is available as an IPython notebook at [Malaya/example/en-ms-long-translation](https://github.com/huseinzol05/Malaya/tree/master/example/en-ms-long-translation).\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "This module only trained on standard language structure, so it is not save to use it for local language structure.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.86 s, sys: 986 ms, total: 6.85 s\n",
      "Wall time: 6.26 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import malaya"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List available Transformer models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Size (MB)</th>\n",
       "      <th>Quantized Size (MB)</th>\n",
       "      <th>BLEU</th>\n",
       "      <th>SacreBLEU Verbose</th>\n",
       "      <th>SacreBLEU-chrF++-FLORES200</th>\n",
       "      <th>Suggested length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>small</th>\n",
       "      <td>42.7</td>\n",
       "      <td>13.4</td>\n",
       "      <td>58.67129</td>\n",
       "      <td>80.2/63.8/52.8/44.4 (BP = 0.997 ratio = 0.997 ...</td>\n",
       "      <td>64.46</td>\n",
       "      <td>256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>base</th>\n",
       "      <td>234</td>\n",
       "      <td>82.7</td>\n",
       "      <td>68.259569</td>\n",
       "      <td>86.3/73.3/64.1/56.8 (BP = 0.985 ratio = 0.985 ...</td>\n",
       "      <td>66.28</td>\n",
       "      <td>256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bigbird</th>\n",
       "      <td>246</td>\n",
       "      <td>63.7</td>\n",
       "      <td>59.863535</td>\n",
       "      <td>82.2/65.9/54.9/46.4 (BP = 0.982 ratio = 0.982 ...</td>\n",
       "      <td>59.64</td>\n",
       "      <td>1024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>small-bigbird</th>\n",
       "      <td>50.4</td>\n",
       "      <td>13.1</td>\n",
       "      <td>56.701338</td>\n",
       "      <td>80.7/63.2/51.6/42.8 (BP = 0.979 ratio = 0.979 ...</td>\n",
       "      <td>58.01</td>\n",
       "      <td>1024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>noisy-base</th>\n",
       "      <td>234</td>\n",
       "      <td>82.7</td>\n",
       "      <td>67.285716</td>\n",
       "      <td>86.1/72.7/63.3/55.8 (BP = 0.981 ratio = 0.981 ...</td>\n",
       "      <td>66.2</td>\n",
       "      <td>256</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Size (MB) Quantized Size (MB)       BLEU  \\\n",
       "small              42.7                13.4   58.67129   \n",
       "base                234                82.7  68.259569   \n",
       "bigbird             246                63.7  59.863535   \n",
       "small-bigbird      50.4                13.1  56.701338   \n",
       "noisy-base          234                82.7  67.285716   \n",
       "\n",
       "                                               SacreBLEU Verbose  \\\n",
       "small          80.2/63.8/52.8/44.4 (BP = 0.997 ratio = 0.997 ...   \n",
       "base           86.3/73.3/64.1/56.8 (BP = 0.985 ratio = 0.985 ...   \n",
       "bigbird        82.2/65.9/54.9/46.4 (BP = 0.982 ratio = 0.982 ...   \n",
       "small-bigbird  80.7/63.2/51.6/42.8 (BP = 0.979 ratio = 0.979 ...   \n",
       "noisy-base     86.1/72.7/63.3/55.8 (BP = 0.981 ratio = 0.981 ...   \n",
       "\n",
       "              SacreBLEU-chrF++-FLORES200 Suggested length  \n",
       "small                              64.46              256  \n",
       "base                               66.28              256  \n",
       "bigbird                            59.64             1024  \n",
       "small-bigbird                      58.01             1024  \n",
       "noisy-base                          66.2              256  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.translation.en_ms.available_transformer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look at `Suggested length`, bigbird 4x longer than normal transformer models, able to infer longer text without need to partition. Let's we do some examples.\n",
    "\n",
    "We are going to compare `small` and `base` models with `small-bigbird` and `bigbird` models for MS-EN translation task. Feel free to test on EN-MS translation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Transformer models\n",
    "\n",
    "```python\n",
    "def transformer(model: str = 'base', quantized: bool = False, **kwargs):\n",
    "    \"\"\"\n",
    "    Load Transformer encoder-decoder model to translate EN-to-MS.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : str, optional (default='base')\n",
    "        Model architecture supported. Allowed values:\n",
    "\n",
    "        * ``'small'`` - Transformer SMALL parameters.\n",
    "        * ``'base'`` - Transformer BASE parameters.\n",
    "        * ``'bigbird'`` - BigBird BASE parameters.\n",
    "        * ``'small-bigbird'`` - BigBird SMALL parameters.\n",
    "        * ``'noisy-base'`` - Transformer BASE parameters trained on noisy dataset.\n",
    "\n",
    "    quantized : bool, optional (default=False)\n",
    "        if True, will load 8-bit quantized model.\n",
    "        Quantized model not necessary faster, totally depends on the machine.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: model\n",
    "        List of model classes:\n",
    "\n",
    "        * if `bigbird` in model, return `malaya.model.bigbird.Translation`.\n",
    "        * else, return `malaya.model.tf.Translation`.\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = malaya.translation.en_ms.transformer()\n",
    "transformer_small = malaya.translation.en_ms.transformer(model = 'small')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigbird = malaya.translation.en_ms.transformer(model = 'bigbird')\n",
    "quantized_bigbird = malaya.translation.en_ms.transformer(model = 'bigbird', quantized = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigbird_small = malaya.translation.en_ms.transformer(model = 'small-bigbird')\n",
    "quantized_bigbird_small = malaya.translation.en_ms.transformer(model = 'small-bigbird', quantized = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Long text examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "string = \"\"\"\n",
    "(Reuters) - The Nicaraguan government ordered the closure of a radio station known for its opposition against President Daniel Ortega, its director said on Friday.\n",
    "\n",
    "Local radio station Radio Dario was shut on Friday, its director Anibal Toruno said on his Twitter account, showing images of the government's decision.\n",
    "\n",
    "Radio Dario was considered one of the last operating news radio stations critical of Ortega, whose government recently shut down seven Catholic radio stations linked to a bishop critical of Ortega.\n",
    "\n",
    "\"Neither turning off equipment nor arbitrarily withdrawing a license will succeed in silencing us or silencing the truth. Far from intimidating us, their actions strengthen us,\" Toruno said.\n",
    "\n",
    "The Nicaraguan government could not be reached immediately for comment.\n",
    "\n",
    "The Nicaraguan telecommunications agency, in the statement posted by Toruno, said the radio station did not meet the technical requirements to be on air.\n",
    "\n",
    "(Reporting by Ismael Lopez; Writing by Carolina Pulice; Editing by William Mallard)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "string2 = \"\"\"\n",
    "KUALA LUMPUR (Aug 13): The government will focus on putting the littoral combat ship (LCS) project back on track as decided by the Cabinet, said Senior Defence Minister Datuk Seri Hishammuddin Hussein in a post on Facebook.\n",
    "\n",
    "He said negotiations with vendors, original equipment manufacturers (OEM) and banks are being carried out.\n",
    "\n",
    "Hishammuddin said this in relation to the procurement of the LCS which became controversial following the Public Accounts Committee (PAC) report revealing that not a single ship had been completed even though the project's contract, worth RM6.083 billion, stipulated that five of the six ships should be completed and handed over by August 2022.\n",
    "\n",
    "In the posting, Hishammuddin said his team has taken media personnel to Boustead Naval Shipyard in Lumut, Perak on Saturday (Aug 13) to take a look at the external and internal conditions of the LCS which is under construction.\n",
    "\n",
    "\"I never get tired of asserting many times that the Defence Ministry and I have nothing to hide. At the same time, the wrongdoers must be brought to justice,\" he added.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "do simple cleaning, remove newline and weird characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from unidecode import unidecode\n",
    "\n",
    "def cleaning(string):\n",
    "    return re.sub(r'[ ]+', ' ', unidecode(string.replace('\\n', ' '))).strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "string = cleaning(string)\n",
    "string2 = cleaning(string2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(154, 176)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(string.split()), len(string2.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Translate using greedy decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Pemerintah Nikaragua memerintahkan penutupan stesen radio yang '\n",
      " 'terkenal dengan penentangannya terhadap Presiden Daniel Ortega, kata '\n",
      " 'pengarahnya pada hari Jumaat. Stesen radio tempatan Radio Dario ditutup pada '\n",
      " 'hari Jumaat, kata pengarahnya Anibal Toruno di akaun Twitternya, menunjukkan '\n",
      " 'gambar keputusan pemerintah. Radio Dario dianggap sebagai salah satu stesen '\n",
      " 'radio berita operasi terakhir yang mengkritik Ortega, yang pemerintahnya '\n",
      " 'baru-baru ini menutup tujuh stesen radio Katolik yang dikaitkan dengan uskup '\n",
      " 'yang mengkritik Ortega. \"Tidak mematikan peralatan atau secara '\n",
      " 'sewenang-wenangnya menarik lesen akan berjaya membungkam kita atau '\n",
      " 'membungkam kebenaran. Jauh dari menakut-nakutkan kita, tindakan mereka '\n",
      " 'menguatkan kita,\" kata Toruno. Kerajaan Nikaragua tidak dapat dihubungi '\n",
      " 'segera untuk mendapatkan komen. Agensi telekomunikasi Nikaragua, dalam '\n",
      " 'kenyataan yang disiarkan oleh Toruno, mengatakan stesen radio itu tidak '\n",
      " 'memenuhi syarat teknikal yang akan disiarkan. (Pelaporan oleh Ismael Lopez; '\n",
      " 'Penulisan oleh Carolina Pulice; Penyuntingan oleh William Mallard)']\n"
     ]
    }
   ],
   "source": [
    "pprint(transformer.greedy_decoder([string]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Pemerintah Nicaraguan memerintahkan penutupan stesen radio yang '\n",
      " 'terkenal dengan penentangannya terhadap Presiden Daniel Ortega, kata '\n",
      " 'pengarahnya pada hari Jumaat. Stesen radio tempatan Radio Dario ditutup pada '\n",
      " 'hari Jumaat, pengarahnya Anibal Toruno mengatakan di akaun Twitternya, '\n",
      " 'menunjukkan gambar keputusan pemerintah. Radio Dario dianggap salah satu '\n",
      " 'stesen radio operasi terakhir yang mengkritik Ortega, yang pemerintahnya '\n",
      " 'baru-baru ini menutup tujuh stesen radio Katolik yang dikaitkan dengan uskup '\n",
      " 'kritikal Ortega. \"Tidak ada yang mematikan peralatan atau sewenang-wenangnya '\n",
      " 'menarik lesen akan berjaya membungkam kami atau membungkam kebenaran. Jauh '\n",
      " 'dari menakutkan kami, tindakan mereka menguatkan kami,\" kata Toruno. '\n",
      " 'Kerajaan Nicaraguan tidak dapat dihubungi segera untuk memberi komen. Agensi '\n",
      " 'telekomunikasi Nicaraguan, dalam kenyataan yang disiarkan oleh Toruno, '\n",
      " 'mengatakan stesen radio tidak memenuhi syarat teknikal untuk disiarkan. '\n",
      " '(Pelaporan oleh Ismael Lopez; Penulisan oleh Carolina Pulice; Penyuntingan '\n",
      " 'oleh William Mallard) ) oleh William Mallard.']\n"
     ]
    }
   ],
   "source": [
    "pprint(transformer_small.greedy_decoder([string]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Normal transformer models not able to infer long texts, we need to partition the text, example, split the text by full stop**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['(Reuters) - The Nicaraguan government ordered the closure of a radio station known for its opposition against President Daniel Ortega, its director said on Friday.',\n",
       " \"Local radio station Radio Dario was shut on Friday, its director Anibal Toruno said on his Twitter account, showing images of the government's decision.\",\n",
       " 'Radio Dario was considered one of the last operating news radio stations critical of Ortega, whose government recently shut down seven Catholic radio stations linked to a bishop critical of Ortega.',\n",
       " '\"Neither turning off equipment nor arbitrarily withdrawing a license will succeed in silencing us or silencing the truth.',\n",
       " 'Far from intimidating us, their actions strengthen us,\" Toruno said.',\n",
       " 'The Nicaraguan government could not be reached immediately for comment.',\n",
       " 'The Nicaraguan telecommunications agency, in the statement posted by Toruno, said the radio station did not meet the technical requirements to be on air.',\n",
       " '(Reporting by Ismael Lopez; Writing by Carolina Pulice; Editing by William Mallard).']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition_string = malaya.text.function.split_into_sentences(string)\n",
    "partition_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Pemerintah Nikaragua memerintahkan penutupan stasiun radio yang '\n",
      " 'dikenal karena oposisi terhadap Presiden Daniel Ortega, kata pengarahnya '\n",
      " 'pada Jumat.',\n",
      " 'Stesen radio tempatan Radio Dario ditutup pada Jumaat, kata pengarahnya '\n",
      " 'Anibal Toruno di akaun Twitternya, menunjukkan imej keputusan kerajaan itu.',\n",
      " 'Radio Dario dianggap sebagai salah satu stesen radio berita operasi terakhir '\n",
      " 'yang kritikal terhadap Ortega, yang kerajaannya baru-baru ini menutup tujuh '\n",
      " 'stesen radio Katolik yang dikaitkan dengan biskop yang kritikal terhadap '\n",
      " 'Ortega.',\n",
      " '\"Tidak mematikan peralatan mahupun dengan sewenang-wenangnya menarik balik '\n",
      " 'lesen akan berjaya membungkam kami atau membungkam kebenaran.',\n",
      " 'Jauh dari menakut-nakutkan kami, tindakan mereka memperkuat kami,\" kata '\n",
      " 'Toruno.',\n",
      " 'Pemerintah Nikaragua tidak dapat dihubungi segera untuk mendapatkan '\n",
      " 'komentar.',\n",
      " 'Agensi telekomunikasi Nicaragua, dalam kenyataan yang dimuat naik Toruno, '\n",
      " 'berkata stesen radio itu tidak memenuhi syarat teknikal untuk berada di '\n",
      " 'udara.',\n",
      " '(Pelaporan oleh Ismael Lopez; Penulisan oleh Carolina Pulice; Penyuntingan '\n",
      " 'oleh William Mallard).']\n"
     ]
    }
   ],
   "source": [
    "pprint(transformer.greedy_decoder(partition_string))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Pemerintah Nicaraguan memerintahkan penutupan stesen radio yang '\n",
      " 'terkenal dengan penentangannya terhadap Presiden Daniel Ortega, kata '\n",
      " 'pengarahnya pada hari Jumaat.',\n",
      " 'Stesen radio tempatan Radio Dario ditutup pada hari Jumaat, pengarahnya '\n",
      " 'Anibal Toruno berkata di akaun Twitternya, menunjukkan gambar keputusan '\n",
      " 'kerajaan.',\n",
      " 'Radio Dario dianggap salah satu stesen radio berita operasi terakhir yang '\n",
      " 'mengkritik Ortega, yang baru-baru ini pemerintahnya menutup tujuh stesen '\n",
      " 'radio Katolik yang dikaitkan dengan seorang uskup yang kritikal terhadap '\n",
      " 'Ortega.',\n",
      " '\"Tidak ada yang mematikan peralatan atau sewenang-wenangnya menarik lesen '\n",
      " 'akan berjaya membungkam kami atau membungkam kebenaran.',\n",
      " 'Jauh dari mengintimidasi kami, tindakan mereka memperkuat kami,\" kata '\n",
      " 'Toruno.',\n",
      " 'Kerajaan Nicaraguan tidak dapat dihubungi segera untuk memberi komen.hari '\n",
      " 'ini.anda tidak',\n",
      " 'Agensi telekomunikasi Nicaraguan, dalam kenyataan yang disiarkan oleh '\n",
      " 'Toruno, berkata stesen radio tidak memenuhi syarat teknikal yang akan '\n",
      " 'disiarkan.',\n",
      " '(Pelaporan oleh Ismael Lopez; Penulisan oleh Carolina Pulice; Penyuntingan '\n",
      " 'oleh William Mallard).']\n"
     ]
    }
   ],
   "source": [
    "pprint(transformer_small.greedy_decoder(partition_string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem with partitioning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problem with partition, the model assumed N + 1 element has no relationship with N element, vice versa. Attention mechanism will not work across partition.\n",
    "\n",
    "So we introduced BigBird to solve this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Pemerintah Nicaraguan memerintahkan penutupan stesen radio yang '\n",
      " 'terkenal dengan penentangannya terhadap Presiden Daniel Ortega, kata '\n",
      " 'pengarahnya pada hari Jumaat. Stesen radio tempatan Radio Dario ditutup pada '\n",
      " 'hari Jumaat, pengarahnya Anibal Toruno mengatakan di akaun Twitternya, '\n",
      " 'menunjukkan gambar keputusan pemerintah. Radio Dario dianggap sebagai salah '\n",
      " 'satu stesen radio berita operasi terakhir yang kritikal terhadap Ortega, '\n",
      " 'yang pemerintahnya baru-baru ini menutup tujuh stesen radio Katolik yang '\n",
      " 'dikaitkan dengan biskop yang kritikal terhadap Ortega. \"Sama ada mematikan '\n",
      " 'peralatan dan sewenang-wenangnya menarik lesen akan berjaya merampas kami '\n",
      " 'atau melepaskan hukuman kebenaran. Jauh dari menakutkan kami, tindakan '\n",
      " 'mereka memperkuat kami,\" kata Toruno. Pemerintah Nicaraguan tidak dapat '\n",
      " 'dihubungi segera untuk memberi komen. Agensi telekomunikasi Nicaraguan, '\n",
      " 'dalam kenyataan yang disiarkan oleh Toruno, mengatakan stesen radio tidak '\n",
      " 'memenuhi syarat teknikal untuk disiarkan. (Pelaporan oleh Ismael Lopez; '\n",
      " 'Penulisan oleh Carolina Pulice; Penyuntingan oleh William Mallard)']\n"
     ]
    }
   ],
   "source": [
    "pprint(bigbird_small.greedy_decoder([string]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Pemerintah Nicaraguan memerintahkan penutupan stesen radio yang '\n",
      " 'terkenal dengan penentangannya terhadap Presiden Daniel Ortega, kata '\n",
      " 'pengarahnya pada hari Jumaat. Stesen radio tempatan Radio Dario ditutup pada '\n",
      " 'hari Jumaat, pengarahnya Anibal Toruno mengatakan di akaun Twitternya, '\n",
      " 'menunjukkan gambar keputusan pemerintah. Radio Dario dianggap sebagai salah '\n",
      " 'satu stesen radio berita operasi terakhir yang kritikal terhadap Ortega, '\n",
      " 'yang pemerintahnya baru-baru ini menutup tujuh stesen radio Katolik yang '\n",
      " 'dikaitkan dengan uskup kritikal terhadap Ortega. \"Tidak dapat mematikan '\n",
      " 'peralatan dan sewenang-wenangnya menarik lesen akan berjaya merampas kami '\n",
      " 'atau merampas kebenaran. Jauh dari menakutkan kami, tindakan mereka '\n",
      " 'memperkuat kami,\" kata Toruno. Pemerintah Nicaraguan tidak dapat dihubungi '\n",
      " 'segera untuk memberi komen. Agensi telekomunikasi Nicaraguan, dalam '\n",
      " 'kenyataan yang disiarkan oleh Toruno, mengatakan stesen radio tidak memenuhi '\n",
      " 'syarat teknikal untuk disiarkan. (Pelaporan oleh Ismael Lopez; Penulisan '\n",
      " 'oleh Carolina Pulice; Penyuntingan oleh William Mallard)']\n"
     ]
    }
   ],
   "source": [
    "pprint(quantized_bigbird_small.greedy_decoder([string]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Kerajaan Nicaraguan memerintahkan penutupan stesen radio yang '\n",
      " 'terkenal dengan penentangannya terhadap Presiden Daniel Ortega, kata '\n",
      " 'pengarahnya pada hari Jumaat. Stesen radio tempatan Radio Dario ditutup pada '\n",
      " 'hari Jumaat, kata pengarahnya Anibal Toruno di akaun Twitternya, menunjukkan '\n",
      " 'gambar keputusan pemerintah. Radio Dario dianggap sebagai salah satu stesen '\n",
      " 'radio berita operasi terakhir yang mengkritik Ortega, yang pemerintahnya '\n",
      " 'baru-baru ini menutup tujuh stesen radio Katolik yang berkaitan dengan uskup '\n",
      " 'yang mengkritik Ortega. \"Tidak perlu mematikan peralatan atau dengan '\n",
      " 'sewenang-wenangnya menarik lesen akan berjaya menampar kita atau '\n",
      " 'menjerumuskan kebenaran. Jauh dari menakutkan kita, tindakan mereka '\n",
      " 'menguatkan kita,\" kata Toruno. Pemerintah Nicaraguan tidak dapat dihubungi '\n",
      " 'segera untuk memberi komen. Agensi telekomunikasi Nicaraguan, dalam '\n",
      " 'pernyataan yang disiarkan oleh Toruno, mengatakan stesen radio itu tidak '\n",
      " 'memenuhi syarat teknikal untuk disiarkan. (Pelaporan oleh Ismael Lopez; '\n",
      " 'Penulisan oleh Carolina Pulice; Penyuntingan oleh William Mallard)']\n"
     ]
    }
   ],
   "source": [
    "pprint(bigbird.greedy_decoder([string]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['(Reuters) - Kerajaan Nicaraguan memerintahkan penutupan stesen radio yang '\n",
      " 'terkenal dengan penentangannya terhadap Presiden Daniel Ortega, kata '\n",
      " 'pengarahnya pada hari Jumaat. Stesen radio tempatan Radio Dario ditutup pada '\n",
      " 'hari Jumaat, kata pengarahnya Anibal Toruno di akaun Twitternya, menunjukkan '\n",
      " 'gambar keputusan pemerintah. Radio Dario dianggap sebagai salah satu stesen '\n",
      " 'radio berita operasi terakhir yang mengkritik Ortega, yang pemerintahnya '\n",
      " 'baru-baru ini menutup tujuh stesen radio Katolik yang berkaitan dengan uskup '\n",
      " 'yang mengkritik Ortega. \"Tidak ada mematikan peralatan atau dengan '\n",
      " 'sewenang-wenangnya menarik lesen akan berjaya menampar kita atau '\n",
      " 'menjerumuskan kebenaran. Jauh dari menakutkan kita, tindakan mereka '\n",
      " 'menguatkan kita,\" kata Toruno. Kerajaan Nicaraguan tidak dapat dihubungi '\n",
      " 'dengan segera untuk memberi komen. Agensi telekomunikasi Nicaraguan, dalam '\n",
      " 'pernyataan yang disiarkan oleh Toruno, mengatakan stesen radio itu tidak '\n",
      " 'memenuhi syarat teknikal untuk disiarkan. (Pelaporan oleh Ismael Lopez; '\n",
      " 'Penulisan oleh Carolina Pulice; Penyuntingan oleh William Mallard)']\n"
     ]
    }
   ],
   "source": [
    "pprint(quantized_bigbird.greedy_decoder([string]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['KUALA LUMPUR (13 Ogos): Kerajaan akan memberi tumpuan untuk meletakkan '\n",
      " 'projek kapal tempur (LCS) yang dipasang semula di landasan tepat seperti '\n",
      " 'diputuskan Kabinet, kata Menteri Pertahanan Kanan, Datuk Seri Hishammuddin '\n",
      " 'Hussein dalam satu hantaran di Facebook. Beliau berkata rundingan dengan '\n",
      " 'vendor, pengeluar peralatan asal (OEM) dan bank sedang dijalankan. '\n",
      " 'Hishammuddin berkata demikian berhubung perolehan LCS yang menjadi '\n",
      " 'kontroversi susulan laporan Jawatankuasa Kira-Kira Wang Negara (PAC) yang '\n",
      " 'mendedahkan bahawa bukan satu kapal sudah selesai walaupun kontrak projek '\n",
      " 'itu, bernilai RM6.083 bilion, menetapkan lima daripada enam kapal itu perlu '\n",
      " 'disiapkan dan diserahkan pada Ogos 2022. Dalam catatan itu, Hishammuddin '\n",
      " 'berkata pasukannya sudah membawa anggota media ke Boustead Naval Shipyard di '\n",
      " 'Lumut, Perak pada Sabtu (13 Ogos) bagi melihat keadaan luaran dan dalaman '\n",
      " 'LCS yang sedang dalam pembinaan. \"Saya tidak pernah jemu dengan menegaskan '\n",
      " 'berkali-kali bahawa Kementerian Pertahanan dan saya tidak ada yang perlu '\n",
      " 'disembunyikan. Pada masa sama, pihak yang zalim mesti dibawa ke muka '\n",
      " 'pengadilan,\" katanya lagi.']\n"
     ]
    }
   ],
   "source": [
    "pprint(bigbird_small.greedy_decoder([string2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['KUALA LUMPUR (13 Ogos): Kerajaan akan memberi tumpuan untuk meletakkan '\n",
      " 'projek kapal tempur (LCS) yang dipasang semula di landasan tepat seperti '\n",
      " 'diputuskan Kabinet, kata Menteri Pertahanan Kanan Datuk Seri Hishammuddin '\n",
      " 'Hussein dalam satu hantaran di Facebook. Beliau berkata rundingan dengan '\n",
      " 'vendor, pengeluar peralatan asal (OEM) dan bank sedang dijalankan. '\n",
      " 'Hishammuddin berkata demikian berhubung perolehan LCS yang menjadi '\n",
      " 'kontroversi susulan laporan Jawatankuasa Kira-Kira Wang Negara (PAC) yang '\n",
      " 'mendedahkan bukan satu kapal itu sudah selesai walaupun kontrak projek itu, '\n",
      " 'bernilai RM6.083 bilion, menetapkan lima daripada enam kapal itu perlu '\n",
      " 'disiapkan dan diserahkan pada Ogos 2022. Dalam catatan itu, Hishammuddin '\n",
      " 'berkata pasukannya sudah membawa anggota media ke Boustead Naval Shipyard di '\n",
      " 'Lumut, Perak pada Sabtu (13 Ogos) bagi melihat keadaan luaran dan dalaman '\n",
      " 'LCS yang sedang dalam pembinaan. \"Saya tidak pernah jemu dengan menegaskan '\n",
      " 'berkali-kali bahawa Kementerian Pertahanan dan saya tidak ada yang perlu '\n",
      " 'disembunyikan. Pada masa sama, pihak yang zalim mesti dibawa ke muka '\n",
      " 'pengadilan,\" katanya lagi.']\n"
     ]
    }
   ],
   "source": [
    "pprint(quantized_bigbird_small.greedy_decoder([string2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['KUALA LUMPUR (13 Ogos): Kerajaan akan memberi tumpuan untuk meletakkan '\n",
      " 'semula projek kapal tempur litatoral (LCS) di landasan seperti yang '\n",
      " 'diputuskan Kabinet, kata Menteri Pertahanan Kanan, Datuk Seri Hishammuddin '\n",
      " 'Hussein dalam hantaran di Facebook. Beliau berkata, rundingan dengan vendor, '\n",
      " 'pengeluar peralatan asal (OEM) dan bank sedang dijalankan. Hishammuddin '\n",
      " 'berkata demikian berhubung perolehan LCS yang menjadi kontroversi berikutan '\n",
      " 'laporan Jawatankuasa Kira-Kira Wang Negara (PAC) mendedahkan tidak ada satu '\n",
      " 'kapal pun yang sudah siap sepenuhnya walaupun kontrak projek itu, bernilai '\n",
      " 'RM6.083 bilion, menetapkan lima daripada enam kapal itu perlu disiapkan dan '\n",
      " 'diserahkan Ogos 2022. Dalam hantaran itu, Hishammuddin berkata, pasukannya '\n",
      " 'sudah membawa anggota media ke Boustead Naval Shipyard di Lumut, Perak pada '\n",
      " 'Sabtu (13 Ogos) untuk melihat keadaan luar dan dalaman LCS yang sedang dalam '\n",
      " 'pembinaan. \"Saya tidak pernah jemu menegaskan berkali-kali bahawa '\n",
      " 'Kementerian Pertahanan dan saya tidak mempunyai apa-apa untuk disembunyikan. '\n",
      " 'Pada masa sama, pihak yang melakukan kesalahan mesti dibawa ke muka '\n",
      " 'pengadilan,\" katanya lagi.']\n"
     ]
    }
   ],
   "source": [
    "pprint(bigbird.greedy_decoder([string2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['KUALA LUMPUR (13 Ogos): Kerajaan akan memberi tumpuan untuk meletakkan '\n",
      " 'semula projek kapal tempur litatoral (LCS) di landasan seperti yang '\n",
      " 'diputuskan Kabinet, kata Menteri Pertahanan Kanan, Datuk Seri Hishammuddin '\n",
      " 'Hussein dalam hantaran di Facebook. Beliau berkata rundingan dengan vendor, '\n",
      " 'pengeluar peralatan asal (OEM) dan bank sedang dijalankan. Hishammuddin '\n",
      " 'berkata demikian berhubung perolehan LCS yang menjadi kontroversi berikutan '\n",
      " 'laporan Jawatankuasa Kira-Kira Wang Negara (PAC) mendedahkan tidak ada satu '\n",
      " 'kapal pun yang sudah siap sepenuhnya walaupun kontrak projek itu, bernilai '\n",
      " 'RM6.083 bilion, menetapkan lima daripada enam kapal itu perlu disiapkan dan '\n",
      " 'diserahkan Ogos 2022. Dalam hantaran itu, Hishammuddin berkata pasukannya '\n",
      " 'sudah membawa anggota media ke Boustead Naval Shipyard di Lumut, Perak pada '\n",
      " 'Sabtu (13 Ogos) untuk melihat keadaan luaran dan dalaman LCS yang sedang '\n",
      " 'dalam pembinaan. \"Saya tidak pernah jemu menegaskan berkali-kali bahawa '\n",
      " 'Kementerian Pertahanan dan saya tidak mempunyai apa-apa untuk bersembunyi. '\n",
      " 'Pada masa sama, pihak yang melakukan kesalahan mesti dibawa ke muka '\n",
      " 'pengadilan,\" katanya lagi.']\n"
     ]
    }
   ],
   "source": [
    "pprint(quantized_bigbird.greedy_decoder([string2]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}