{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "This tutorial is available as an IPython notebook at [Malaya/example/segmentation](https://github.com/huseinzol05/Malaya/tree/master/example/segmentation).\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "This module trained on both standard and local (included social media) language structures, so it is save to use for both.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.46 s, sys: 690 ms, total: 5.15 s\n",
      "Wall time: 5.24 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import malaya"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Common problem for social media texts, there are missing spaces in the text, so text segmentation can help you,\n",
    "\n",
    "1. huseinsukamakan ayam,dia sgtrisaukan -> husein suka makan ayam, dia sgt risaukan.\n",
    "2. drmahathir sangat menekankan budaya budakzamansekarang -> dr mahathir sangat menekankan budaya budak zaman sekarang.\n",
    "3. ceritatunnajibrazak -> cerita tun najib razak.\n",
    "4. TunM sukakan -> Tun M sukakan."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Segmentation only,\n",
    "\n",
    "1. Solve spacing error.\n",
    "3. Not correcting any grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "string1 = 'huseinsukamakan ayam,dia sgtrisaukan'\n",
    "string2 = 'drmahathir sangat menekankan budaya budakzamansekarang'\n",
    "string3 = 'ceritatunnajibrazak'\n",
    "string4 = 'TunM sukakan'\n",
    "string_hard = 'IPOH-AhliDewanUndangan Negeri(ADUN) HuluKinta, MuhamadArafat Varisai Mahamadmenafikanmesejtularmendakwa beliau akan melompatparti menyokong UMNO membentuk kerajaannegeridiPerak.BeliauyangjugaKetua Penerangan Parti Keadilan Rakyat(PKR)Perak dalam satumesejringkaskepadaSinar Harian menjelaskan perkara itutidakbenarsama sekali.'\n",
    "string_socialmedia = 'aqxsukalah apeyg tejadidekat mamattu'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viterbi algorithm\n",
    "\n",
    "Commonly people use Viterbi algorithm to solve this problem, we also added viterbi using ngram from bahasa papers and wikipedia.\n",
    "\n",
    "```python\n",
    "def viterbi(max_split_length: int = 20, **kwargs):\n",
    "    \"\"\"\n",
    "    Load Segmenter class using viterbi algorithm.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    max_split_length: int, (default=20)\n",
    "        max length of words in a sentence to segment\n",
    "    validate: bool, optional (default=True)\n",
    "        if True, malaya will check model availability and download if not available.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result : malaya.segmentation.SEGMENTER class\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "viterbi = malaya.segmentation.viterbi()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Segmentize\n",
    "\n",
    "```python\n",
    "def segment(self, strings: List[str]):\n",
    "    \"\"\"\n",
    "    Segment strings.\n",
    "    Example, \"sayasygkan negarasaya\" -> \"saya sygkan negara saya\"\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    strings : List[str]\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 109 ms, sys: 1.04 ms, total: 110 ms\n",
      "Wall time: 110 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['husein suka makan ayam,dia sgt risau kan',\n",
       " 'dr mahathir sangat mene kan kan budaya budak zaman sekarang',\n",
       " 'cerita tu n najib razak',\n",
       " 'Tun M suka kan']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "viterbi.segment([string1, string2, string3, string4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8.45 ms, sys: 157 µs, total: 8.6 ms\n",
      "Wall time: 8.69 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['IPOH - Ahli Dewan Undangan Negeri(ADUN) Hulu Kinta, Muhamad Arafat Varisai Mahamadmenafikanmesejtularmendakwa belia u akan me lompat part i me nyo ko ng UMNO mem bentuk kerajaannegeridi Perak. Beliauyangjuga Ketua Penerangan Parti Keadilan Rakyat(PKR) Perak dalam satumesejringkaskepada Sinar Harian men jel ask an perkara it u tidak benar sama sekali.',\n",
       " 'aq x suka lah ape yg te jadi dekat mama ttu']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "viterbi.segment([string_hard, string_socialmedia])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List available Transformer model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Size (MB)</th>\n",
       "      <th>Quantized Size (MB)</th>\n",
       "      <th>Sequence Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>small</th>\n",
       "      <td>42.7</td>\n",
       "      <td>13.1</td>\n",
       "      <td>0.8217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>base</th>\n",
       "      <td>234.0</td>\n",
       "      <td>63.8</td>\n",
       "      <td>0.8759</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Size (MB)  Quantized Size (MB)  Sequence Accuracy\n",
       "small       42.7                 13.1             0.8217\n",
       "base       234.0                 63.8             0.8759"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "malaya.segmentation.available_transformer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Transformer model\n",
    "\n",
    "```python\n",
    "def transformer(model: str = 'small', quantized: bool = False, **kwargs):\n",
    "    \"\"\"\n",
    "    Load transformer encoder-decoder model to Segmentize.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model : str, optional (default='base')\n",
    "        Model architecture supported. Allowed values:\n",
    "\n",
    "        * ``'small'`` - Transformer SMALL parameters.\n",
    "        * ``'base'`` - Transformer BASE parameters.\n",
    "\n",
    "    quantized : bool, optional (default=False)\n",
    "        if True, will load 8-bit quantized model. \n",
    "        Quantized model not necessary faster, totally depends on the machine.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: malaya.model.tf.Segmentation class\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Load quantized model will cause accuracy drop.\n"
     ]
    }
   ],
   "source": [
    "model = malaya.segmentation.transformer(model = 'small')\n",
    "quantized_model = malaya.segmentation.transformer(model = 'small', quantized = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Load quantized model will cause accuracy drop.\n"
     ]
    }
   ],
   "source": [
    "model_base = malaya.segmentation.transformer(model = 'base')\n",
    "quantized_model_base = malaya.segmentation.transformer(model = 'base', quantized = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predict using greedy decoder\n",
    "\n",
    "```python\n",
    "def greedy_decoder(self, strings: List[str]):\n",
    "    \"\"\"\n",
    "    Segment strings using greedy decoder.\n",
    "    Example, \"sayasygkan negarasaya\" -> \"saya sygkan negara saya\"\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    strings : List[str]\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.12 s, sys: 432 ms, total: 1.55 s\n",
      "Wall time: 959 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['husein suka makan ayam, dia sgt risaukan',\n",
       " 'dr mahathir sangat menekankan budaya budak zaman sekarang',\n",
       " 'cerita tun najib razak',\n",
       " 'Tun M sukakan']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.greedy_decoder([string1, string2, string3, string4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.12 s, sys: 464 ms, total: 1.58 s\n",
      "Wall time: 888 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['husein suka makan ayam, dia sgt risaukan',\n",
       " 'dr mahathir sangat menekankan budaya budak zaman sekarang',\n",
       " 'cerita tun najib razak',\n",
       " 'Tun M sukakan']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model.greedy_decoder([string1, string2, string3, string4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.58 s, sys: 2.88 s, total: 8.46 s\n",
      "Wall time: 4.08 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['husein suka makan ayam, dia sgt risaukan',\n",
       " 'dr mahathir sangat menekankan budaya budak zaman sekarang',\n",
       " 'cerita tun najib razak cerita',\n",
       " 'Tun M sukakan Tun M sukakan']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model_base.greedy_decoder([string1, string2, string3, string4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.73 s, sys: 2.96 s, total: 8.69 s\n",
      "Wall time: 3.81 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['husein suka makan ayam, dia sgt risaukan',\n",
       " 'dr mahathir sangat menekankan budaya budak zaman sekarang',\n",
       " 'cerita tun najib razak cerita tun',\n",
       " 'Tun M sukakan Tun M sukakan']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model_base.greedy_decoder([string1, string2, string3, string4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.52 s, sys: 499 ms, total: 3.02 s\n",
      "Wall time: 768 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['IPOH - Ahli Dewan Undangan Negeri (ADUN) Hulu Kinta, Muhamad Arafat Varisai Mahamad menafikan mesej tular mendakwa beliau akan melompat parti menyokong UMNO membentuk kerajaan negeri di Perak. Beliau yang juga Ketua Penerangan Parti Keadilan Rakyat (PKR) Perak dalam satu mesej ringkas kepada Sinar Harian menjelaskan perkara itu tidak benar sama sekali.',\n",
       " 'aq xsukalah ape yg tejadid dekat mamat tu']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.greedy_decoder([string_hard, string_socialmedia])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.62 s, sys: 447 ms, total: 3.07 s\n",
      "Wall time: 756 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['IPOH - Ahli Dewan Undangan Negeri (ADUN) Hulu Kinta, Muhamad Arafat Varisai Mahamad menafikan mesej tular mendakwa beliau akan melompat parti menyokong UMNO membentuk kerajaan negeri di Perak. Beliau yang juga Ketua Penerangan Parti Keadilan Rakyat (PKR) Perak dalam satu mesej ringkas kepada Sinar Harian menjelaskan perkara itu tidak benar sama sekali.',\n",
       " 'aq xsukalah ape yg tejadid dekat mamat tu']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model.greedy_decoder([string_hard, string_socialmedia])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 17.8 s, sys: 10.2 s, total: 28 s\n",
      "Wall time: 5.84 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['IPOH - Ahli Dewan Undangan Negeri (ADUN) Hulu Kinta, Muhamad Arafat Varisai Mahamad menafikan mesej tular mendakwa beliau akan melompat parti menyokong UMNO membentuk kerajaan negeri di Perak. Beliau yang juga Ketua Penerangan Parti Keadilan Rakyat (PKR) Perak dalam satu mesej ringkas kepada Sinar Harian menjelaskan perkara itu tidak benar sama sekali.',\n",
       " 'aq xsukalah ape yg teja di dekat mamat tu aq xsukalah ape yg teja di dekat mamat tu']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model_base.greedy_decoder([string_hard, string_socialmedia])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 17.6 s, sys: 9.63 s, total: 27.3 s\n",
      "Wall time: 5.85 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['IPOH - Ahli Dewan Undangan Negeri (ADUN) Hulu Kinta, Muhamad Arafat Varisai Mahamad menafikan mesej tular mendakwa beliau akan melompat parti menyokong UMNO membentuk kerajaan negeri di Perak. Beliau yang juga Ketua Penerangan Parti Keadilan Rakyat (PKR) Perak dalam satu mesej ringkas kepada Sinar Harian menjelaskan perkara itu tidak benar sama sekali.',\n",
       " 'aq xsukalah ape yg teja di dekat mamat tu aq xsukalah ape yg teja di dekat mamat tu']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model_base.greedy_decoder([string_hard, string_socialmedia])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Problem with batching string, short string might repeating itself, so to solve this, you need to give a single string only**,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.37 s, sys: 532 ms, total: 1.9 s\n",
      "Wall time: 652 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['aq xsukalah ape yg teja di dekat mamat tu']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model_base.greedy_decoder([string_socialmedia])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 648 ms, sys: 228 ms, total: 876 ms\n",
      "Wall time: 289 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['cerita tun najib razak']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model_base.greedy_decoder([string3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 495 ms, sys: 202 ms, total: 697 ms\n",
      "Wall time: 225 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Tun M sukakan']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model_base.greedy_decoder([string4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predict using beam decoder\n",
    "\n",
    "```python\n",
    "def beam_decoder(self, strings: List[str]):\n",
    "    \"\"\"\n",
    "    Segment strings using beam decoder, beam width size 3, alpha 0.5 .\n",
    "    Example, \"sayasygkan negarasaya\" -> \"saya sygkan negara saya\"\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    strings : List[str]\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    result: List[str]\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.38 s, sys: 1.87 s, total: 3.25 s\n",
      "Wall time: 654 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['aq xsukalah ape yg tejadid dekat mamat tu']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model.beam_decoder([string_socialmedia])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.77 s, sys: 3.71 s, total: 10.5 s\n",
      "Wall time: 2.43 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['aq xsukalah ape yg teja di dekat mamat tu']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "quantized_model_base.beam_decoder([string_socialmedia])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We can expect beam decoder is much more slower than greedy decoder**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}