{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spelling Correction using Symspeller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "This tutorial is available as an IPython notebook at [Malaya/example/spelling-correction-symspell](https://github.com/huseinzol05/Malaya/tree/master/example/spelling-correction-symspell).\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This spelling correction is an improvement version for https://github.com/mammothb/symspellpy to adapt with our local shortform / typos. Before you able to use this spelling correction, you need to install,\n",
    "\n",
    "```bash\n",
    "pip install symspellpy\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8er24wly\n",
      "INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8er24wly/_remote_module_non_scriptable.py\n",
      "/home/husein/dev/malaya/malaya/tokenizer.py:214: FutureWarning: Possible nested set at position 3397\n",
      "  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n",
      "/home/husein/dev/malaya/malaya/tokenizer.py:214: FutureWarning: Possible nested set at position 3927\n",
      "  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n"
     ]
    }
   ],
   "source": [
    "import malaya"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some text examples copied from Twitter\n",
    "\n",
    "string1 = 'krajaan patut bagi pencen awal skt kpd warga emas supaya emosi'\n",
    "string2 = 'Husein ska mkn aym dkat kampng Jawa'\n",
    "string3 = 'Melayu malas ni narration dia sama je macam men are trash. True to some, false to some.'\n",
    "string4 = 'Tapi tak pikir ke bahaya perpetuate myths camtu. Nanti kalau ada hiring discrimination despite your good qualifications because of your race tau pulak marah. Your kids will be victims of that too.'\n",
    "string5 = 'DrM cerita Melayu malas semenjak saya kat University (early 1980s) and now as i am edging towards retirement in 4-5 years time after a career of being an Engineer, Project Manager, General Manager'\n",
    "string6 = 'blh bntg dlm kls nlp sy, nnti intch'\n",
    "string7 = 'mulakn slh org boleh ,bila geng tuh kena slhkn jgk xboleh trima .. pelik'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load symspeller model\n",
    "\n",
    "```python\n",
    "def load(\n",
    "    max_edit_distance_dictionary: int = 2,\n",
    "    prefix_length: int = 7,\n",
    "    term_index: int = 0,\n",
    "    count_index: int = 1,\n",
    "    top_k: int = 10,\n",
    "    **kwargs\n",
    "):\n",
    "    \"\"\"\n",
    "    Load a symspell Spell Corrector for Malay.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: malaya.spelling_correction.symspell.Symspell class\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:malaya_boilerplate.huggingface:downloading frozen huseinzol05/v27-preprocessing/bm_1grams.txt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8653a1e4372349439a1040da7a9f56ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)e/main/bm_1grams.txt:   0%|          | 0.00/2.41M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:malaya_boilerplate.huggingface:downloading frozen huseinzol05/v27-preprocessing/bm_1grams.json\n"
     ]
    }
   ],
   "source": [
    "model = malaya.spelling_correction.symspell.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List possible generated pool of words\n",
    "\n",
    "```python\n",
    "def edit_candidates(self, word):\n",
    "    \"\"\"\n",
    "    Generate candidates given a word.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    word: str\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mahathir',\n",
       " 'mahadhir',\n",
       " 'mahatir',\n",
       " 'mahadzir',\n",
       " 'matahir',\n",
       " 'mahathma',\n",
       " 'hathir',\n",
       " 'mahachi',\n",
       " 'mahathat',\n",
       " 'mahadhika',\n",
       " 'mahavira',\n",
       " 'mhthir']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.edit_candidates('mhthir')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sambang',\n",
       " 'lambang',\n",
       " 'tambang',\n",
       " 'ambang',\n",
       " 'bambang',\n",
       " 'sambung',\n",
       " 'sembang',\n",
       " 'sumbang',\n",
       " 'mambang',\n",
       " 'sumbangan',\n",
       " 'sambungan',\n",
       " 'sembawang',\n",
       " 'sembarang',\n",
       " 'sambaran',\n",
       " 'samarang',\n",
       " 'sambakong',\n",
       " 'sembayang',\n",
       " 'tambalang',\n",
       " 'ambacang',\n",
       " 'sabatang',\n",
       " 'samalanga',\n",
       " 'kabananga',\n",
       " 'smbng']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.edit_candidates('smbng')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To correct a word\n",
    "\n",
    "```python\n",
    "def correct(self, word: str, **kwargs):\n",
    "    \"\"\"\n",
    "    Most probable spelling correction for word.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    word: str\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: str\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'suka'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.correct('suke')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kpd'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.correct('kpd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kerajaan'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.correct('krajaan')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To correct a sentence\n",
    "\n",
    "```python\n",
    "def correct_text(self, text: str):\n",
    "    \"\"\"\n",
    "    Correct all the words within a text, returning the corrected text.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    text: str\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: str\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kerajaan patut bagi pencen awal saat kpd warga emas supaya emosi'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.correct_text(string1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = malaya.tokenizer.Tokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Husein ska mkn aym dkat kampng Jawa'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hussein ska makan ayam dapat kampung Jawa'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized = tokenizer.tokenize(string2)\n",
    "model.correct_text(' '.join(tokenized))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Melayu malas ni narration dia sama je macam men are trash . True to some , false to some .'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized = tokenizer.tokenize(string3)\n",
    "model.correct_text(' '.join(tokenized))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'DrM cerita Melayu malas semenjak saya kat University ( early 1980s ) and now asia i am edging towards retirement cina 4 - 5 years time after a career of being an Engineer , Project Manager , General Manager'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized = tokenizer.tokenize(string5)\n",
    "model.correct_text(' '.join(tokenized))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'blh bintang dlm kelas malaya saya , nnti mintalah'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized = tokenizer.tokenize(string6)\n",
    "model.correct_text(' '.join(tokenized))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mulakan slh org boleh , bila geng tuh kena salahkan jgk boleh trima . . pelik'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized = tokenizer.tokenize(string7)\n",
    "model.correct_text(' '.join(tokenized))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}