{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spelling Correction using Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "This tutorial is available as an IPython notebook at [Malaya/example/spelling-correction-transformer](https://github.com/huseinzol05/Malaya/tree/master/example/spelling-correction-transformer).\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "logging.basicConfig(level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:numexpr.utils:NumExpr defaulting to 8 threads.\n"
     ]
    }
   ],
   "source": [
    "import malaya"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some text examples copied from Twitter\n",
    "\n",
    "string1 = 'krajaan patut bagi pencen awal skt kpd warga emas supaya emosi'\n",
    "string2 = 'Husein ska mkn aym dkat kampng Jawa'\n",
    "string3 = 'Melayu malas ni narration dia sama je macam men are trash. True to some, false to some.'\n",
    "string4 = 'Tapi tak pikir ke bahaya perpetuate myths camtu. Nanti kalau ada hiring discrimination despite your good qualifications because of your race tau pulak marah. Your kids will be victims of that too.'\n",
    "string5 = 'DrM cerita Melayu malas semenjak saya kat University (early 1980s) and now as i am edging towards retirement in 4-5 years time after a career of being an Engineer, Project Manager, General Manager'\n",
    "string6 = 'blh bntg dlm kls nlp sy, nnti intch'\n",
    "string7 = 'mulakn slh org boleh ,bila geng tuh kena slhkn jgk xboleh trima .. pelik'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List available Transformer models\n",
    "\n",
    "We use custom spelling augmentation,\n",
    "\n",
    "1. `replace_similar_consonants`\n",
    "\n",
    "- mereka -> nereka\n",
    "\n",
    "2. `replace_similar_vowels`\n",
    "\n",
    "- suka -> sika\n",
    "\n",
    "3. `socialmedia_form`\n",
    "\n",
    "- suka -> ska\n",
    "\n",
    "4. `vowel_alternate`\n",
    "\n",
    "- singapore -> sngpore\n",
    "- kampung -> kmpng"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:malaya.spelling_correction.transformer:tested on 10k generated dataset at https://github.com/huseinzol05/malaya/tree/master/session/spelling-correction/t5\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Size (MB)</th>\n",
       "      <th>Quantized Size (MB)</th>\n",
       "      <th>WER</th>\n",
       "      <th>Suggested length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>small-t5</th>\n",
       "      <td>355.6</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.015625</td>\n",
       "      <td>256.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tiny-t5</th>\n",
       "      <td>208.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>0.023712</td>\n",
       "      <td>256.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>super-tiny-t5</th>\n",
       "      <td>81.8</td>\n",
       "      <td>27.1</td>\n",
       "      <td>0.038001</td>\n",
       "      <td>256.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Size (MB)  Quantized Size (MB)       WER  Suggested length\n",
       "small-t5           355.6                195.0  0.015625             256.0\n",
       "tiny-t5            208.0                103.0  0.023712             256.0\n",
       "super-tiny-t5       81.8                 27.1  0.038001             256.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.spelling_correction.transformer.available_transformer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Transformer model\n",
    "\n",
    "```python\n",
    "def transformer(model: str = 'small-t5', quantized: bool = False, **kwargs):\n",
    "    \"\"\"\n",
    "    Load a Transformer Spell Corrector.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : str, optional (default='small-t5')\n",
    "        Model architecture supported. Allowed values:\n",
    "\n",
    "        * ``'small-t5'`` - T5 SMALL parameters.\n",
    "        * ``'tiny-t5'`` - T5 TINY parameters.\n",
    "        * ``'super-tiny-t5'`` - T5 SUPER TINY parameters.\n",
    "\n",
    "    quantized : bool, optional (default=False)\n",
    "        if True, will load 8-bit quantized model.\n",
    "        Quantized model not necessary faster, totally depends on the machine.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: malaya.model.t5.Spell class\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "t5 = malaya.spelling_correction.transformer.transformer(model = 'tiny-t5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predict using greedy decoder\n",
    "\n",
    "```python\n",
    "def greedy_decoder(self, strings: List[str]):\n",
    "    \"\"\"\n",
    "    spelling correction for strings.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    strings: List[str]\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['kerajaan patut bagi pencen awal skt kpd warga emas supaya emosi']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t5.greedy_decoder([string1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Husein suka makan ayam dekat kampung Jawa']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t5.greedy_decoder([string2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Melayu malas ni narration dia sama je macam men are trash . True to some , false to some .']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t5.greedy_decoder([string3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mulakan slh orang boleh , bilah geng tuh kena salahkan juga xboleh terima . . pelik']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t5.greedy_decoder([string7])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}