Preprocessing#

This tutorial is available as an IPython notebook at Malaya/example/preprocessing.

[1]:

%%time
import malaya

CPU times: user 2.74 s, sys: 3.78 s, total: 6.52 s
Wall time: 1.95 s

/home/husein/dev/malaya/malaya/tokenizer.py:214: FutureWarning: Possible nested set at position 3397
  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))
/home/husein/dev/malaya/malaya/tokenizer.py:214: FutureWarning: Possible nested set at position 3927
  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))

Available rules#

We know that social media texts from Twitter, Facebook and Instagram are very noisy and we want to clean as much as possible to make our machines understand the structure of sentence much better. In Malaya, we standardize our text preprocessing,

  1. Malaya can replace special words into tokens to reduce dimension curse. rm10k become <money>.

  2. Malaya can put tags for special words, #drmahathir become <hashtag> drmahathir </hashtag>.

  3. Malaya can expand english contractions.

  4. Expand hashtags, #drmahathir become dr mahathir, required a segmentation callable.

  5. Malaya can put emoji tags if provide demoji object.

normalize#

Supported normalize,

  1. hashtag

  2. cashtag

  3. tag

  4. user

  5. emphasis

  6. censored

  7. acronym

  8. eastern_emoticons

  9. rest_emoticons

  10. emoji

  11. quotes

  12. percent

  13. repeat_puncts

  14. money

  15. email

  16. phone

  17. number

  18. allcaps

  19. url

  20. date

  21. time

You can check all supported list at malaya.preprocessing.get_normalize().

Example, if you set money and number, and input string is RM10k, the output is <money>.

annotate#

Supported annotate,

  1. hashtag

  2. allcaps

  3. elongated

  4. repeated

  5. emphasis

  6. censored

Example, if you set hashtag, and input string is #drmahathir, the output is <hashtag> drmahathir </hashtag>.

[2]:

string_1 = 'CANT WAIT for the new season of #mahathirmohamad ＼(^o^)／!!! #davidlynch #tvseries :))), TAAAK SAAABAAR!!!'
string_2 = 'kecewanya #johndoe movie and it suuuuucks!!! WASTED RM10... rm10 #badmovies :/'
string_3 = "@husein:  can't wait for the Nov 9 #Sentiment talks!  YAAAAAAY !!! :-D http://sentimentsymposium.com/."
string_4 = 'aahhh, malasnye nak pegi keje harini #mondayblues'
string_5 = '#drmahathir #najibrazak #1malaysia #mahathirnajib'

Preprocessing Interface#

def preprocessing(
    normalize: List[str] = [
        'url',
        'email',
        'percent',
        'money',
        'phone',
        'user',
        'time',
        'date',
        'number',
    ],
    annotate: List[str] = [
        'allcaps',
        'elongated',
        'repeated',
        'emphasis',
        'censored',
        'hashtag',
    ],
    lowercase: bool = True,
    fix_unidecode: bool = True,
    expand_english_contractions: bool = True,
    segmenter: Callable = None,
    demoji: Callable = None,
    **kwargs,
):
    """
    Load Preprocessing class.

    Parameters
    ----------
    normalize: List[str], optional (default=['url', 'email', 'percent', 'money', 'phone', 'user', 'time', 'date', 'number'])
        normalizing tokens, can check all supported normalizing at `malaya.preprocessing.get_normalize()`.
    annotate: List[str], optional (default=['hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'])
        annonate tokens <open></open>,
        only accept ['hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'].
    lowercase: bool, optional (default=True)
    fix_unidecode: bool, optional (default=True)
        fix unidecode using `ftfy.fix_text`.
    expand_english_contractions: bool, optional (default=True)
        expand english contractions.
    segmenter: Callable, optional (default=None)
        function to segmentize word.
        If provide, it will expand hashtags, #mondayblues == monday blues
    demoji: object
        demoji object, need to have a method `demoji`.

    Returns
    -------
    result : malaya.preprocessing.Preprocessing class
    """

Load default paramaters#

default parameters able to translate most of english to bahasa malaysia.

[3]:

%%time
preprocessing = malaya.preprocessing.preprocessing()

CPU times: user 15.3 ms, sys: 0 ns, total: 15.3 ms
Wall time: 15.2 ms

/home/husein/dev/malaya/malaya/preprocessing.py:41: FutureWarning: Possible nested set at position 42
  k.lower(): re.compile(_expressions[k]) for k, v in _expressions.items()
/home/husein/dev/malaya/malaya/preprocessing.py:41: FutureWarning: Possible nested set at position 3
  k.lower(): re.compile(_expressions[k]) for k, v in _expressions.items()

[4]:

%%time
' '.join(preprocessing.process(string_1))

CPU times: user 1.3 ms, sys: 0 ns, total: 1.3 ms
Wall time: 1.31 ms

[4]:

'<allcaps> CANT WAIT </allcaps> untuk the new season of <hashtag> #mahathirmohamad </hashtag> \\(^o^)/ <repeated> ! </repeated> <hashtag> #davidlynch </hashtag> <hashtag> #tvseries </hashtag> :))) , <allcaps> TAAAK SAAABAAR </allcaps> <repeated> ! </repeated>'

[5]:

%%time
' '.join(preprocessing.process(string_2))

CPU times: user 279 µs, sys: 0 ns, total: 279 µs
Wall time: 280 µs

[5]:

'kecewanya <hashtag> #johndoe </hashtag> movie and it suuuuucks <repeated> ! </repeated> <allcaps> WASTED </allcaps> <money> <repeated> . </repeated> <money> <hashtag> #badmovies </hashtag> :/'

[6]:

%%time
' '.join(preprocessing.process(string_3))

CPU times: user 253 µs, sys: 0 ns, total: 253 µs
Wall time: 254 µs

[6]:

'<user> : can not wait untuk the <date> <hashtag> #Sentiment </hashtag> talks ! <allcaps> YAAAAAAY </allcaps> <repeated> ! </repeated> :-D <url>'

[7]:

%%time
' '.join(preprocessing.process(string_4))

CPU times: user 160 µs, sys: 0 ns, total: 160 µs
Wall time: 161 µs

[7]:

'aahhh , malasnye nak pergi kerja hari ini <hashtag> #mondayblues </hashtag>'

[8]:

%%time
' '.join(preprocessing.process(string_5))

CPU times: user 200 µs, sys: 0 ns, total: 200 µs
Wall time: 201 µs

[8]:

'<hashtag> #drmahathir </hashtag> <hashtag> #najibrazak </hashtag> <hashtag> #1malaysia </hashtag> <hashtag> #mahathirnajib </hashtag>'

Load default paramaters with segmenter to expand hashtags.#

We saw <hashtag> drmahathir </hashtag> <hashtag> najibrazak </hashtag>, we want to expand to become dr mahathir and najib razak.

[10]:

segmenter = malaya.segmentation.huggingface()

Loading the tokenizer from the `special_tokens_map.json` and the `added_tokens.json` will be removed in `transformers 5`,  it is kept for forward compatibility, but it is recommended to update your `tokenizer_config.json` by uploading it again. You will see the new `added_tokens_decoder` attribute that will store the relevant information.
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565

[11]:

segmenter_func = lambda x: segmenter.generate([x], max_length = 100)[0]
segmenter_func('hellosuka')

spaces_between_special_tokens is deprecated and will be removed in transformers v5. It was adding spaces between `added_tokens`, not special tokens, and does not exist in our fast implementation. Future tokenizers will handle the decoding process on a per-model rule.

[11]:

'hello suka'

[12]:

%%time
preprocessing = malaya.preprocessing.preprocessing(segmenter = segmenter_func)

CPU times: user 135 µs, sys: 122 µs, total: 257 µs
Wall time: 263 µs

[13]:

%%time
' '.join(preprocessing.process(string_1))

CPU times: user 1.29 s, sys: 4.45 ms, total: 1.3 s
Wall time: 116 ms

[13]:

'<allcaps> CANT WAIT </allcaps> untuk the new season of <hashtag> mahathir mohamad </hashtag> \\(^o^)/ <repeated> ! </repeated> <hashtag> david lynch </hashtag> <hashtag> tv series </hashtag> :))) , <allcaps> TAAAK SAAABAAR </allcaps> <repeated> ! </repeated>'

[14]:

%%time
' '.join(preprocessing.process(string_2))

CPU times: user 1.14 s, sys: 3.57 ms, total: 1.15 s
Wall time: 102 ms

[14]:

'kecewanya <hashtag> john doe </hashtag> movie and it suuuuucks <repeated> ! </repeated> <allcaps> WASTED </allcaps> <money> <repeated> . </repeated> <money> <hashtag> bad movies </hashtag> :/'

[15]:

%%time
' '.join(preprocessing.process(string_3))

CPU times: user 460 ms, sys: 0 ns, total: 460 ms
Wall time: 43.7 ms

[15]:

'<user> : can not wait untuk the <date> <hashtag> Sentiment </hashtag> talks ! <allcaps> YAAAAAAY </allcaps> <repeated> ! </repeated> :-D <url>'

[16]:

%%time
' '.join(preprocessing.process(string_4))

CPU times: user 544 ms, sys: 0 ns, total: 544 ms
Wall time: 49.3 ms

[16]:

'aahhh , malasnye nak pergi kerja hari ini <hashtag> isnin blues </hashtag>'

[17]:

%%time
' '.join(preprocessing.process(string_5))

CPU times: user 1.75 s, sys: 0 ns, total: 1.75 s
Wall time: 154 ms

[17]:

'<hashtag> dr mahathir </hashtag> <hashtag> najib razak </hashtag> <hashtag> 1malaysia </hashtag> <hashtag> mahathir najib </hashtag>'