{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Segmentation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This tutorial is available as an IPython notebook at [Malaya/example/segmentation](https://github.com/huseinzol05/Malaya/tree/master/example/segmentation).\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "This module trained on both standard and local (included social media) language structures, so it is save to use for both.\n", " \n", "
" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ['CUDA_VISIBLE_DEVICES'] = ''" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/husein/.local/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.\n", " warn(\"The installed version of bitsandbytes was compiled without GPU support. \"\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "/home/husein/.local/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32\n", "CPU times: user 2.69 s, sys: 4 s, total: 6.69 s\n", "Wall time: 1.94 s\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/husein/dev/malaya/malaya/tokenizer.py:214: FutureWarning: Possible nested set at position 3397\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n", "/home/husein/dev/malaya/malaya/tokenizer.py:214: FutureWarning: Possible nested set at position 3927\n", " self.tok = re.compile(r'({})'.format('|'.join(pipeline)))\n" ] } ], "source": [ "%%time\n", "\n", "import malaya" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Common problem for social media texts, there are missing spaces in the text, so text segmentation can help you,\n", "\n", "1. huseinsukamakan ayam,dia sgtrisaukan -> husein suka makan ayam, dia sgt risaukan.\n", "2. drmahathir sangat menekankan budaya budakzamansekarang -> dr mahathir sangat menekankan budaya budak zaman sekarang.\n", "3. ceritatunnajibrazak -> cerita tun najib razak.\n", "4. TunM sukakan -> Tun M sukakan." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Segmentation only,\n", "\n", "1. Solve spacing error.\n", "3. Not correcting any grammar." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "string1 = 'huseinsukamakan ayam,dia sgtrisaukan'\n", "string2 = 'drmahathir sangat menekankan budaya budakzamansekarang'\n", "string3 = 'ceritatunnajibrazak'\n", "string4 = 'TunM sukakan'\n", "string_hard = 'IPOH-AhliDewanUndangan Negeri(ADUN) HuluKinta, MuhamadArafat Varisai Mahamadmenafikanmesejtularmendakwa beliau akan melompatparti menyokong UMNO membentuk kerajaannegeridiPerak.BeliauyangjugaKetua Penerangan Parti Keadilan Rakyat(PKR)Perak dalam satumesejringkaskepadaSinar Harian menjelaskan perkara itutidakbenarsama sekali.'\n", "string_socialmedia = 'aqxsukalah apeyg tejadidekat mamattu'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### List available HuggingFace model" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'mesolitica/finetune-segmentation-t5-super-tiny-standard-bahasa-cased': {'Size (MB)': 51,\n", " 'WER': 0.030962535,\n", " 'CER': 0.0041129253,\n", " 'Suggested length': 256},\n", " 'mesolitica/finetune-segmentation-t5-tiny-standard-bahasa-cased': {'Size (MB)': 139,\n", " 'WER': 0.0207876127,\n", " 'CER': 0.002146691161,\n", " 'Suggested length': 256},\n", " 'mesolitica/finetune-segmentation-t5-small-standard-bahasa-cased': {'Size (MB)': 242,\n", " 'WER': 0.0202468274,\n", " 'CER': 0.0024325431,\n", " 'Suggested length': 256}}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "malaya.segmentation.available_huggingface" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tested on random generated dataset at https://f000.backblazeb2.com/file/malay-dataset/segmentation/test-set-segmentation.json\n" ] } ], "source": [ "print(malaya.segmentation.info)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load HuggingFace model\n", "\n", "```python\n", "def huggingface(\n", " model: str = 'mesolitica/finetune-segmentation-t5-tiny-standard-bahasa-cased',\n", " force_check: bool = True,\n", " **kwargs,\n", "):\n", " \"\"\"\n", " Load HuggingFace model to segmentation.\n", "\n", " Parameters\n", " ----------\n", " model: str, optional (default='mesolitica/finetune-segmentation-t5-tiny-standard-bahasa-cased')\n", " Check available models at `malaya.segmentation.available_huggingface`.\n", " force_check: bool, optional (default=True)\n", " Force check model one of malaya model.\n", " Set to False if you have your own huggingface model.\n", "\n", " Returns\n", " -------\n", " result: malaya.torch_model.huggingface.Generator\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Loading the tokenizer from the `special_tokens_map.json` and the `added_tokens.json` will be removed in `transformers 5`, it is kept for forward compatibility, but it is recommended to update your `tokenizer_config.json` by uploading it again. You will see the new `added_tokens_decoder` attribute that will store the relevant information.\n", "You are using the default legacy behaviour of the . If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n" ] } ], "source": [ "model = malaya.segmentation.huggingface()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Predict\n", "\n", "```python\n", "def generate(self, strings: List[str], **kwargs):\n", " \"\"\"\n", " Generate texts from the input.\n", "\n", " Parameters\n", " ----------\n", " strings : List[str]\n", " **kwargs: vector arguments pass to huggingface `generate` method.\n", " Read more at https://huggingface.co/docs/transformers/main_classes/text_generation\n", "\n", " Returns\n", " -------\n", " result: List[str]\n", " \"\"\"\n", "```" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "spaces_between_special_tokens is deprecated and will be removed in transformers v5. It was adding spaces between `added_tokens`, not special tokens, and does not exist in our fast implementation. Future tokenizers will handle the decoding process on a per-model rule.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1.72 s, sys: 9.94 ms, total: 1.73 s\n", "Wall time: 164 ms\n" ] }, { "data": { "text/plain": [ "['husein suka makan ayam, dia sgt risikokan',\n", " 'dr mahathir sangat menekankan budaya budak zaman sekarang',\n", " 'cerita tun najib razak',\n", " 'Tun M sukakan']" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "model.generate([string1, string2, string3, string4], max_length = 256)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 5.68 s, sys: 11.6 ms, total: 5.69 s\n", "Wall time: 511 ms\n" ] }, { "data": { "text/plain": [ "['husein suka makan ayam, dia sgt risikokan',\n", " 'dr mahathir sangat menekankan budaya budak zaman sekarang',\n", " 'cerita tun najib razak',\n", " 'Tun M sukakan',\n", " 'IPOH - Ahli Dewan Undangan Negeri (ADUN) Hulu Kinta, Muhamad Ararat Varisai Mahamad menafikan mesej tular mendakwa beliau akan melompat parti menyokong UMNO membentuk kerajaan negeri di Perak. Beliau yang juga Ketua Penerangan Parti Keadilan Rakyat (PKR) Perak dalam satu mesej ringkas kepada Sinar Harian menjelaskan perkara itu tidak benar sama sekali.',\n", " 'aq x sukalah ape yg tejadi dekat mamat tu']" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "model.generate([string1, string2, string3, string4, string_hard, string_socialmedia], max_length = 256)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### able to infer mixed MS and EN" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "string5 = 'i hate chicken, but i like fish'\n", "string6 = 'hi guys i noticed semalam & harini dah ramai yang dapat cookies ni kan. So harini i nak share some post mortem of our first batch:'" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 6.6 s, sys: 6.91 ms, total: 6.61 s\n", "Wall time: 601 ms\n" ] }, { "data": { "text/plain": [ "['husein suka makan ayam, dia sgt risikokan',\n", " 'dr mahathir sangat menekankan budaya budak zaman sekarang',\n", " 'cerita tun najib razak',\n", " 'Tun M sukakan',\n", " 'IPOH - Ahli Dewan Undangan Negeri (ADUN) Hulu Kinta, Muhamad Ararat Varisai Mahamad menafikan mesej tular mendakwa beliau akan melompat parti menyokong UMNO membentuk kerajaan negeri di Perak. Beliau yang juga Ketua Penerangan Parti Keadilan Rakyat (PKR) Perak dalam satu mesej ringkas kepada Sinar Harian menjelaskan perkara itu tidak benar sama sekali.',\n", " 'aq x sukalah ape yg tejadi dekat mamat tu',\n", " 'i hate chicken, but i like fish',\n", " 'hi guys i noticed semalam & harini dah ramai yang dapat cookies ni kan. So harini i nak share some post mortem of our first batch:']" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "model.generate([string1, string2, string3, string4, string_hard, string_socialmedia,\n", " string5, string6], max_length = 256)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }