This tutorial is available as an IPython notebook at Malaya/example/preprocessing.

import malaya
CPU times: user 3.5 s, sys: 2.3 s, total: 5.8 s
Wall time: 2.98 s
/home/ubuntu/dev/malaya/malaya/ FutureWarning: Possible nested set at position 3361
  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))
/home/ubuntu/dev/malaya/malaya/ FutureWarning: Possible nested set at position 3879
  self.tok = re.compile(r'({})'.format('|'.join(pipeline)))

Available rules#

We know that social media texts from Twitter, Facebook and Instagram are very noisy and we want to clean as much as possible to make our machines understand the structure of sentence much better. In Malaya, we standardize our text preprocessing,

  1. Malaya can replace special words into tokens to reduce dimension curse. rm10k become <money>.

  2. Malaya can put tags for special words, #drmahathir become <hashtag> drmahathir </hashtag>.

  3. Malaya can expand english contractions.

  4. Expand hashtags, #drmahathir become dr mahathir, required a segmentation callable.

  5. Malaya can put emoji tags if provide demoji object.


Supported normalize,

  1. hashtag

  2. cashtag

  3. tag

  4. user

  5. emphasis

  6. censored

  7. acronym

  8. eastern_emoticons

  9. rest_emoticons

  10. emoji


  12. percent

  13. repeat_puncts

  14. money

  15. email

  16. phone

  17. number

  18. allcaps

  19. url

  20. date

  21. time

You can check all supported list at malaya.preprocessing.get_normalize().

Example, if you set money and number, and input string is RM10k, the output is <money>.


Supported annotate,

  1. hashtag

  2. allcaps

  3. elongated

  4. repeated

  5. emphasis

  6. censored

Example, if you set hashtag, and input string is #drmahathir, the output is <hashtag> drmahathir </hashtag>.

string_1 = 'CANT WAIT for the new season of #mahathirmohamad \(^o^)/!!! #davidlynch #tvseries :))), TAAAK SAAABAAR!!!'
string_2 = 'kecewanya #johndoe movie and it suuuuucks!!! WASTED RM10... rm10 #badmovies :/'
string_3 = "@husein:  can't wait for the Nov 9 #Sentiment talks!  YAAAAAAY !!! :-D"
string_4 = 'aahhh, malasnye nak pegi keje harini #mondayblues'
string_5 = '#drmahathir #najibrazak #1malaysia #mahathirnajib'

Preprocessing Interface#

def preprocessing(
    normalize: List[str] = [
    annotate: List[str] = [
    lowercase: bool = True,
    fix_unidecode: bool = True,
    expand_english_contractions: bool = True,
    segmenter: Callable = None,
    demoji: Callable = None,
    Load Preprocessing class.

    normalize: List[str], optional (default=['url', 'email', 'percent', 'money', 'phone', 'user', 'time', 'date', 'number'])
        normalizing tokens, can check all supported normalizing at `malaya.preprocessing.get_normalize()`.
    annotate: List[str], optional (default=['hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'])
        annonate tokens <open></open>,
        only accept ['hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'].
    lowercase: bool, optional (default=True)
    fix_unidecode: bool, optional (default=True)
        fix unidecode using `ftfy.fix_text`.
    expand_english_contractions: bool, optional (default=True)
        expand english contractions.
    segmenter: Callable, optional (default=None)
        function to segmentize word.
        If provide, it will expand hashtags, #mondayblues == monday blues
    demoji: object
        demoji object, need to have a method `demoji`.

    result : malaya.preprocessing.Preprocessing class

Load default paramaters#

default parameters able to translate most of english to bahasa malaysia.

preprocessing = malaya.preprocessing.preprocessing()
CPU times: user 28.3 ms, sys: 68 µs, total: 28.4 ms
Wall time: 28.2 ms
' '.join(preprocessing.process(string_1))
CPU times: user 576 µs, sys: 1.45 ms, total: 2.02 ms
Wall time: 2.03 ms
'<allcaps> CANT WAIT </allcaps> untuk the new season of <hashtag> #mahathirmohamad </hashtag> \\(^o^)/ <repeated> ! </repeated> <hashtag> #davidlynch </hashtag> <hashtag> #tvseries </hashtag> <happy> , <allcaps> TAAAK SAAABAAR </allcaps> <repeated> ! </repeated>'
' '.join(preprocessing.process(string_2))
CPU times: user 466 µs, sys: 0 ns, total: 466 µs
Wall time: 470 µs
'kecewanya <hashtag> #johndoe </hashtag> movie and it suuuuucks <repeated> ! </repeated> <allcaps> WASTED </allcaps> <money> <repeated> . </repeated> <money> <hashtag> #badmovies </hashtag> <annoyed>'
' '.join(preprocessing.process(string_3))
CPU times: user 408 µs, sys: 0 ns, total: 408 µs
Wall time: 412 µs
'<user> : can not wait untuk the <date> <hashtag> #Sentiment </hashtag> talks ! <allcaps> YAAAAAAY </allcaps> <repeated> ! </repeated> <laugh> <url>'
' '.join(preprocessing.process(string_4))
CPU times: user 214 µs, sys: 0 ns, total: 214 µs
Wall time: 217 µs
'aahhh , malasnye nak pergi kerja hari ini <hashtag> #mondayblues </hashtag>'
' '.join(preprocessing.process(string_5))
CPU times: user 152 µs, sys: 90 µs, total: 242 µs
Wall time: 244 µs
'<hashtag> #drmahathir </hashtag> <hashtag> #najibrazak </hashtag> <hashtag> #1malaysia </hashtag> <hashtag> #mahathirnajib </hashtag>'

Load default paramaters with segmenter to expand hashtags.#

We saw <hashtag> drmahathir </hashtag> <hashtag> najibrazak </hashtag>, we want to expand to become dr mahathir and najib razak.

segmenter = malaya.segmentation.transformer(model = 'small', quantized = True)
segmenter_func = lambda x: segmenter.greedy_decoder([x])[0]
2022-09-01 15:22:19.953149: I tensorflow/stream_executor/cuda/] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
'hello suka'
preprocessing = malaya.preprocessing.preprocessing(segmenter = segmenter_func)
CPU times: user 69 µs, sys: 0 ns, total: 69 µs
Wall time: 73 µs
' '.join(preprocessing.process(string_1))
CPU times: user 113 ms, sys: 0 ns, total: 113 ms
Wall time: 166 ms
'<allcaps> CANT WAIT </allcaps> untuk the new season of <hashtag> mahathir mohamad </hashtag> \\(^o^)/ <repeated> ! </repeated> <hashtag> davidlynch </hashtag> <hashtag> tv series </hashtag> <happy> , <allcaps> TAAAK SAAABAAR </allcaps> <repeated> ! </repeated>'
' '.join(preprocessing.process(string_2))
CPU times: user 62 ms, sys: 3.12 ms, total: 65.2 ms
Wall time: 90.7 ms
'kecewanya <hashtag> johndoe </hashtag> movie and it suuuuucks <repeated> ! </repeated> <allcaps> WASTED </allcaps> <money> <repeated> . </repeated> <money> <hashtag> bad movies </hashtag> <annoyed>'
' '.join(preprocessing.process(string_3))
CPU times: user 27.7 ms, sys: 3.83 ms, total: 31.5 ms
Wall time: 45.1 ms
'<user> : can not wait untuk the <date> <hashtag> Sentiment </hashtag> talks ! <allcaps> YAAAAAAY </allcaps> <repeated> ! </repeated> <laugh> <url>'
' '.join(preprocessing.process(string_4))
CPU times: user 37.9 ms, sys: 6.61 ms, total: 44.6 ms
Wall time: 66.3 ms
'aahhh , malasnye nak pergi kerja hari ini <hashtag> mondayblues </hashtag>'
' '.join(preprocessing.process(string_5))
CPU times: user 123 ms, sys: 6.86 ms, total: 129 ms
Wall time: 182 ms
'<hashtag> dr mahathir </hashtag> <hashtag> najib razak </hashtag> <hashtag> 1malaysia </hashtag> <hashtag> mahathir najib </hashtag>'