{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Syllable tokenizer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This tutorial is available as an IPython notebook at [Malaya/example/tokenizer-syllable](https://github.com/huseinzol05/Malaya/tree/master/example/tokenizer-syllable).\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This module only suitable for standard language structure, so it is not save to use it for local language structure.\n", " \n", "
" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ['CUDA_VISIBLE_DEVICES'] = ''\n", "os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/husein/dev/malaya/malaya/tokenizer.py:202: FutureWarning: Possible nested set at position 3361\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n", "/home/husein/dev/malaya/malaya/tokenizer.py:202: FutureWarning: Possible nested set at position 3879\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 3.22 s, sys: 3.44 s, total: 6.66 s\n", "Wall time: 2.3 s\n" ] } ], "source": [ "%%time\n", "import malaya" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load rules based syllable tokenizer\n", "\n", "```python\n", "def rules(**kwargs):\n", " \"\"\"\n", " Load rules based syllable tokenizer.\n", " originally from https://github.com/fahadh4ilyas/syllable_splitter/blob/master/SyllableSplitter.py\n", " - improved `cuaca` double vocal `ua` based on https://en.wikipedia.org/wiki/Comparison_of_Indonesian_and_Standard_Malay#Syllabification\n", " - improved `rans` double consonant `ns` based on https://www.semanticscholar.org/paper/Syllabification-algorithm-based-on-syllable-rules-Musa-Kadir/a819f255f066ae0fd7a30b3534de41da37d04ea1\n", " - improved `au` and `ai` double vocal.\n", "\n", " Returns\n", " -------\n", " result: malaya.syllable.Tokenizer class\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "tokenizer = malaya.syllable.rules()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Tokenize\n", "\n", "```python\n", "def tokenize(self, string):\n", " \"\"\"\n", " Tokenize string into multiple strings using syllable patterns.\n", " Example from https://www.semanticscholar.org/paper/Syllabification-algorithm-based-on-syllable-rules-Musa-Kadir/a819f255f066ae0fd7a30b3534de41da37d04ea1/figure/0,\n", " 'cuaca' -> ['cua', 'ca']\n", " 'insurans' -> ['in', 'su', 'rans']\n", " 'praktikal' -> ['prak', 'ti', 'kal']\n", " 'strategi' -> ['stra', 'te', 'gi']\n", " 'ayam' -> ['a', 'yam']\n", " 'anda' -> ['an', 'da']\n", " 'hantu' -> ['han', 'tu']\n", "\n", " Parameters\n", " ----------\n", " string : str\n", "\n", " Returns\n", " -------\n", " result: List[str]\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['a', 'ngan', '-', 'a', 'ngan']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('angan-angan')" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['cua', 'ca']" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('cuaca')" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['hi', 'dup']" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('hidup')" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['in', 'su', 'ran']" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('insuran')" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['in', 'su', 'rans']" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('insurans')" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['a', 'yam']" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('ayam')" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['stra', 'te', 'gi']" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('strategi')" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['han', 'tu']" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('hantu')" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['hel', 'lo']" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('hello')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Better performance\n", "\n", "Split by words and tokenize it." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "string = 'sememang-memangnya kau sakai siot'" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['se', 'me', 'mang', '-', 'me', 'mang', 'nya', 'kau', 'sa', 'kai', 'siot']" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = []\n", "for w in string.split():\n", " results.extend(tokenizer.tokenize(w))\n", "results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### List available deep learning models\n", "\n", "We are also provide syllable tokenizer using deep learning, trained on DBP dataset." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CERQuantized Size (MB)Size (MB)WER
base0.0077693.111.70.043033
\n", "
" ], "text/plain": [ " CER Quantized Size (MB) Size (MB) WER\n", "base 0.007769 3.1 11.7 0.043033" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "malaya.syllable.available_deep_model()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load deep learning model" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-09-17 18:53:45.493265: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n", "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2022-09-17 18:53:45.496779: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n", "2022-09-17 18:53:45.496804: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: husein-MS-7D31\n", "2022-09-17 18:53:45.496807: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: husein-MS-7D31\n", "2022-09-17 18:53:45.496872: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program\n", "2022-09-17 18:53:45.496891: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.141.3\n" ] } ], "source": [ "model = malaya.syllable.deep_model()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Tokenize\n", "\n", "```python\n", "def tokenize(self, string, beam_search: bool = False):\n", " \"\"\"\n", " Tokenize string into multiple strings using deep learning.\n", "\n", " Parameters\n", " ----------\n", " string : str\n", " beam_search : bool, (optional=False)\n", " If True, use beam search decoder, else use greedy decoder.\n", "\n", " Returns\n", " -------\n", " result: List[str]\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['a', 'ngan', 'a', 'ngan']" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.tokenize('angan-angan')" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['in', 'su', 'ran']" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('insuran')" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['in', 'su', 'rans']" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.tokenize('insurans')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Harder example\n", "\n", "Test set from DBP at https://huggingface.co/datasets/mesolitica/syllable/raw/main/test-syllable.json" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "import requests\n", "import json\n", "\n", "r = requests.get('https://huggingface.co/datasets/mesolitica/syllable/raw/main/test-syllable.json')\n", "test_set = r.json()" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1952" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(test_set)" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "def calculate_wer(actual, hyp):\n", " \"\"\"\n", " Calculate WER using `python-Levenshtein`.\n", " \"\"\"\n", " import Levenshtein as Lev\n", "\n", " b = set(actual.split() + hyp.split())\n", " word2char = dict(zip(b, range(len(b))))\n", "\n", " w1 = [chr(word2char[w]) for w in actual.split()]\n", " w2 = [chr(word2char[w]) for w in hyp.split()]\n", "\n", " return Lev.distance(''.join(w1), ''.join(w2)) / len(actual.split())" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.09016393442622951" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "wers = []\n", "for test in test_set:\n", " t = tokenizer.tokenize(test[0])\n", " t = [t_ for t_ in t if t_ not in ['-']]\n", " wer = calculate_wer(test[1], '.'.join(t))\n", " wers.append(wer)\n", " \n", "sum(wers) / len(wers)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "original: mengilukan\n", "actual: ['me', 'ngi', 'lu', 'kan']\n", "predicted: ['me', 'ngi', 'lu', 'kan']\n", "\n", "original: menjongkok\n", "actual: ['men', 'jong', 'kok']\n", "predicted: ['men', 'jong', 'kok']\n", "\n", "original: tergabas\n", "actual: ['ter', 'ga', 'bas']\n", "predicted: ['ter', 'ga', 'bas']\n", "\n", "original: perunding\n", "actual: ['pe', 'run', 'ding']\n", "predicted: ['pe', 'run', 'ding']\n", "\n", "original: kemahalan\n", "actual: ['ke', 'ma', 'ha', 'lan']\n", "predicted: ['ke', 'ma', 'ha', 'lan']\n", "\n", "original: renggang\n", "actual: ['reng', 'gang']\n", "predicted: ['reng', 'gang']\n", "\n", "original: bersuci\n", "actual: ['ber', 'su', 'ci']\n", "predicted: ['ber', 'su', 'ci']\n", "\n", "original: jelebat\n", "actual: ['je', 'le', 'bat']\n", "predicted: ['je', 'le', 'bat']\n", "\n", "original: rekod\n", "actual: ['re', 'kod']\n", "predicted: ['re', 'kod']\n", "\n", "original: amang\n", "actual: ['a', 'mang']\n", "predicted: ['a', 'mang']\n", "\n", "original: aromaterapi\n", "actual: ['a', 'ro', 'ma', 'te', 'ra', 'pi']\n", "predicted: ['a', 'ro', 'ma', 'te', 'ra', 'pi']\n", "\n", "original: pengkompaunan\n", "actual: ['peng', 'kom', 'pau', 'nan']\n", "predicted: ['peng', 'kom', 'pau', 'nan']\n", "\n", "original: payah\n", "actual: ['pa', 'yah']\n", "predicted: ['pa', 'yah']\n", "\n", "original: menghargai\n", "actual: ['meng', 'har', 'ga', 'i']\n", "predicted: ['meng', 'har', 'gai']\n", "\n", "original: keterpaksaan\n", "actual: ['ke', 'ter', 'pak', 'sa', 'an']\n", "predicted: ['ke', 'ter', 'pak', 'sa', 'an']\n", "\n", "original: kerempagi\n", "actual: ['ke', 'rem', 'pa', 'gi']\n", "predicted: ['ke', 'rem', 'pa', 'gi']\n", "\n", "original: pengancaman\n", "actual: ['pe', 'ngan', 'ca', 'man']\n", "predicted: ['pe', 'ngan', 'ca', 'man']\n", "\n", "original: kedwilogaman\n", "actual: ['ke', 'dwi', 'lo', 'ga', 'man']\n", "predicted: ['ked', 'wi', 'lo', 'ga', 'man']\n", "\n", "original: copeng\n", "actual: ['co', 'peng']\n", "predicted: ['co', 'peng']\n", "\n", "original: antienzim\n", "actual: ['an', 'ti', 'en', 'zim']\n", "predicted: ['an', 'ti', 'en', 'zim']\n", "\n", "original: angkar\n", "actual: ['ang', 'kar']\n", "predicted: ['ang', 'kar']\n", "\n", "original: menjembak\n", "actual: ['men', 'jem', 'bak']\n", "predicted: ['men', 'jem', 'bak']\n", "\n", "original: tanggah\n", "actual: ['tang', 'gah']\n", "predicted: ['tang', 'gah']\n", "\n", "original: berjujuk\n", "actual: ['ber', 'ju', 'juk']\n", "predicted: ['ber', 'ju', 'juk']\n", "\n", "original: nestapa\n", "actual: ['nes', 'ta', 'pa']\n", "predicted: ['nes', 'ta', 'pa']\n", "\n", "original: engku\n", "actual: ['eng', 'ku']\n", "predicted: ['eng', 'ku']\n", "\n", "original: undang-undang\n", "actual: ['un', 'dang', 'un', 'dang']\n", "predicted: ['un', 'dang', '-', 'un', 'dang']\n", "\n", "original: tiket\n", "actual: ['ti', 'ket']\n", "predicted: ['ti', 'ket']\n", "\n", "original: janin\n", "actual: ['ja', 'nin']\n", "predicted: ['ja', 'nin']\n", "\n", "original: pakuk\n", "actual: ['pa', 'kuk']\n", "predicted: ['pa', 'kuk']\n", "\n", "original: betika\n", "actual: ['be', 'ti', 'ka']\n", "predicted: ['be', 'ti', 'ka']\n", "\n", "original: nangoi\n", "actual: ['na', 'ngoi']\n", "predicted: ['na', 'ngo', 'i']\n", "\n", "original: mulato\n", "actual: ['mu', 'la', 'to']\n", "predicted: ['mu', 'la', 'to']\n", "\n", "original: peruasan\n", "actual: ['pe', 'rua', 'san']\n", "predicted: ['pe', 'rua', 'san']\n", "\n", "original: terkajang\n", "actual: ['ter', 'ka', 'jang']\n", "predicted: ['ter', 'ka', 'jang']\n", "\n", "original: menjanda\n", "actual: ['men', 'jan', 'da']\n", "predicted: ['men', 'jan', 'da']\n", "\n", "original: menautkan\n", "actual: ['me', 'naut', 'kan']\n", "predicted: ['me', 'naut', 'kan']\n", "\n", "original: khalayak\n", "actual: ['kha', 'la', 'yak']\n", "predicted: ['kha', 'la', 'yak']\n", "\n", "original: lena\n", "actual: ['le', 'na']\n", "predicted: ['le', 'na']\n", "\n", "original: kesurupan\n", "actual: ['ke', 'su', 'ru', 'pan']\n", "predicted: ['ke', 'su', 'ru', 'pan']\n", "\n", "original: meneriak-neriakkan\n", "actual: ['me', 'ne', 'riak', 'ne', 'riak', 'kan']\n", "predicted: ['me', 'ne', 'ri', 'ak', '-', 'ne', 'ri', 'ak', 'kan']\n", "\n", "original: bergumpal\n", "actual: ['ber', 'gum', 'pal']\n", "predicted: ['ber', 'gum', 'pal']\n", "\n", "original: rodat\n", "actual: ['ro', 'dat']\n", "predicted: ['ro', 'dat']\n", "\n", "original: sepukal\n", "actual: ['se', 'pu', 'kal']\n", "predicted: ['se', 'pu', 'kal']\n", "\n", "original: kerani\n", "actual: ['ke', 'ra', 'ni']\n", "predicted: ['ke', 'ra', 'ni']\n", "\n", "original: mewahyukan\n", "actual: ['me', 'wah', 'yu', 'kan']\n", "predicted: ['me', 'wah', 'yu', 'kan']\n", "\n", "original: berprestij\n", "actual: ['ber', 'pres', 'tij']\n", "predicted: ['ber', 'pres', 'tij']\n", "\n", "original: dingin\n", "actual: ['di', 'ngin']\n", "predicted: ['di', 'ngin']\n", "\n", "original: lipas\n", "actual: ['li', 'pas']\n", "predicted: ['li', 'pas']\n", "\n", "original: berdingkit-dingkit\n", "actual: ['ber', 'ding', 'kit', 'ding', 'kit']\n", "predicted: ['ber', 'ding', 'kit', '-', 'ding', 'kit']\n", "\n" ] } ], "source": [ "for test in test_set[:50]:\n", " print('original:', test[0])\n", " print('actual:', test[1].split('.'))\n", " t = tokenizer.tokenize(test[0])\n", " print('predicted:', t)\n", " print()" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "original: mengilukan\n", "actual: ['me', 'ngi', 'lu', 'kan']\n", "predicted: ['me', 'ngi', 'lu', 'kan']\n", "\n", "original: menjongkok\n", "actual: ['men', 'jong', 'kok']\n", "predicted: ['men', 'jong', 'kok']\n", "\n", "original: tergabas\n", "actual: ['ter', 'ga', 'bas']\n", "predicted: ['ter', 'ga', 'bas']\n", "\n", "original: perunding\n", "actual: ['pe', 'run', 'ding']\n", "predicted: ['pe', 'run', 'ding']\n", "\n", "original: kemahalan\n", "actual: ['ke', 'ma', 'ha', 'lan']\n", "predicted: ['ke', 'ma', 'ha', 'lan']\n", "\n", "original: renggang\n", "actual: ['reng', 'gang']\n", "predicted: ['reng', 'gang']\n", "\n", "original: bersuci\n", "actual: ['ber', 'su', 'ci']\n", "predicted: ['ber', 'su', 'ci']\n", "\n", "original: jelebat\n", "actual: ['je', 'le', 'bat']\n", "predicted: ['je', 'le', 'bat']\n", "\n", "original: rekod\n", "actual: ['re', 'kod']\n", "predicted: ['re', 'kod']\n", "\n", "original: amang\n", "actual: ['a', 'mang']\n", "predicted: ['a', 'mang']\n", "\n", "original: aromaterapi\n", "actual: ['a', 'ro', 'ma', 'te', 'ra', 'pi']\n", "predicted: ['a', 'ro', 'ma', 'te', 'ra', 'pi']\n", "\n", "original: pengkompaunan\n", "actual: ['peng', 'kom', 'pau', 'nan']\n", "predicted: ['peng', 'kom', 'pau', 'nan']\n", "\n", "original: payah\n", "actual: ['pa', 'yah']\n", "predicted: ['pa', 'yah']\n", "\n", "original: menghargai\n", "actual: ['meng', 'har', 'ga', 'i']\n", "predicted: ['meng', 'har', 'ga', 'i']\n", "\n", "original: keterpaksaan\n", "actual: ['ke', 'ter', 'pak', 'sa', 'an']\n", "predicted: ['ke', 'ter', 'pak', 'sa', 'an']\n", "\n", "original: kerempagi\n", "actual: ['ke', 'rem', 'pa', 'gi']\n", "predicted: ['ke', 'rem', 'pa', 'gi']\n", "\n", "original: pengancaman\n", "actual: ['pe', 'ngan', 'ca', 'man']\n", "predicted: ['pe', 'ngan', 'ca', 'man']\n", "\n", "original: kedwilogaman\n", "actual: ['ke', 'dwi', 'lo', 'ga', 'man']\n", "predicted: ['ke', 'dwi', 'lo', 'ga', 'man']\n", "\n", "original: copeng\n", "actual: ['co', 'peng']\n", "predicted: ['co', 'peng']\n", "\n", "original: antienzim\n", "actual: ['an', 'ti', 'en', 'zim']\n", "predicted: ['an', 'tien', 'zim']\n", "\n", "original: angkar\n", "actual: ['ang', 'kar']\n", "predicted: ['ang', 'kar']\n", "\n", "original: menjembak\n", "actual: ['men', 'jem', 'bak']\n", "predicted: ['men', 'jem', 'bak']\n", "\n", "original: tanggah\n", "actual: ['tang', 'gah']\n", "predicted: ['tang', 'gah']\n", "\n", "original: berjujuk\n", "actual: ['ber', 'ju', 'juk']\n", "predicted: ['ber', 'ju', 'juk']\n", "\n", "original: nestapa\n", "actual: ['nes', 'ta', 'pa']\n", "predicted: ['nes', 'ta', 'pa']\n", "\n", "original: engku\n", "actual: ['eng', 'ku']\n", "predicted: ['eng', 'ku']\n", "\n", "original: undang-undang\n", "actual: ['un', 'dang', 'un', 'dang']\n", "predicted: ['un', 'dang', 'un', 'dang']\n", "\n", "original: tiket\n", "actual: ['ti', 'ket']\n", "predicted: ['ti', 'ket']\n", "\n", "original: janin\n", "actual: ['ja', 'nin']\n", "predicted: ['ja', 'nin']\n", "\n", "original: pakuk\n", "actual: ['pa', 'kuk']\n", "predicted: ['pa', 'kuk']\n", "\n", "original: betika\n", "actual: ['be', 'ti', 'ka']\n", "predicted: ['be', 'ti', 'ka']\n", "\n", "original: nangoi\n", "actual: ['na', 'ngoi']\n", "predicted: ['na', 'ngoi']\n", "\n", "original: mulato\n", "actual: ['mu', 'la', 'to']\n", "predicted: ['mu', 'la', 'to']\n", "\n", "original: peruasan\n", "actual: ['pe', 'rua', 'san']\n", "predicted: ['pe', 'rua', 'san']\n", "\n", "original: terkajang\n", "actual: ['ter', 'ka', 'jang']\n", "predicted: ['ter', 'ka', 'jang']\n", "\n", "original: menjanda\n", "actual: ['men', 'jan', 'da']\n", "predicted: ['men', 'jan', 'da']\n", "\n", "original: menautkan\n", "actual: ['me', 'naut', 'kan']\n", "predicted: ['me', 'naut', 'kan']\n", "\n", "original: khalayak\n", "actual: ['kha', 'la', 'yak']\n", "predicted: ['kha', 'la', 'yak']\n", "\n", "original: lena\n", "actual: ['le', 'na']\n", "predicted: ['le', 'na']\n", "\n", "original: kesurupan\n", "actual: ['ke', 'su', 'ru', 'pan']\n", "predicted: ['ke', 'su', 'ru', 'pan']\n", "\n", "original: meneriak-neriakkan\n", "actual: ['me', 'ne', 'riak', 'ne', 'riak', 'kan']\n", "predicted: ['me', 'ne', 'riak', 'ne', 'ri', 'kan']\n", "\n", "original: bergumpal\n", "actual: ['ber', 'gum', 'pal']\n", "predicted: ['ber', 'gum', 'pal']\n", "\n", "original: rodat\n", "actual: ['ro', 'dat']\n", "predicted: ['ro', 'dat']\n", "\n", "original: sepukal\n", "actual: ['se', 'pu', 'kal']\n", "predicted: ['se', 'pu', 'kal']\n", "\n", "original: kerani\n", "actual: ['ke', 'ra', 'ni']\n", "predicted: ['ke', 'ra', 'ni']\n", "\n", "original: mewahyukan\n", "actual: ['me', 'wah', 'yu', 'kan']\n", "predicted: ['me', 'wah', 'yu', 'kan']\n", "\n", "original: berprestij\n", "actual: ['ber', 'pres', 'tij']\n", "predicted: ['ber', 'pres', 'tij']\n", "\n", "original: dingin\n", "actual: ['di', 'ngin']\n", "predicted: ['di', 'ngin']\n", "\n", "original: lipas\n", "actual: ['li', 'pas']\n", "predicted: ['li', 'pas']\n", "\n", "original: berdingkit-dingkit\n", "actual: ['ber', 'ding', 'kit', 'ding', 'kit']\n", "predicted: ['ber', 'ding', 'kit', 'ding', 'kit']\n", "\n" ] } ], "source": [ "for test in test_set[:50]:\n", " print('original:', test[0])\n", " print('actual:', test[1].split('.'))\n", " t = model.tokenize(test[0])\n", " print('predicted:', t)\n", " print()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }