Preprocessing
Contents
Preprocessing#
This tutorial is available as an IPython notebook at Malaya/example/preprocessing.
[1]:
%%time
import malaya
CPU times: user 3.5 s, sys: 2.3 s, total: 5.8 s
Wall time: 2.98 s
/home/ubuntu/dev/malaya/malaya/tokenizer.py:202: FutureWarning: Possible nested set at position 3361
self.tok = re.compile(r'({})'.format('|'.join(pipeline)))
/home/ubuntu/dev/malaya/malaya/tokenizer.py:202: FutureWarning: Possible nested set at position 3879
self.tok = re.compile(r'({})'.format('|'.join(pipeline)))
Available rules#
We know that social media texts from Twitter, Facebook and Instagram are very noisy and we want to clean as much as possible to make our machines understand the structure of sentence much better. In Malaya, we standardize our text preprocessing,
Malaya can replace special words into tokens to reduce dimension curse.
rm10k
become<money>
.Malaya can put tags for special words,
#drmahathir
become<hashtag> drmahathir </hashtag>
.Malaya can expand english contractions.
Expand hashtags,
#drmahathir
becomedr mahathir
, required a segmentation callable.Malaya can put emoji tags if provide
demoji
object.
normalize#
Supported normalize
,
hashtag
cashtag
tag
user
emphasis
censored
acronym
eastern_emoticons
rest_emoticons
emoji
quotes
percent
repeat_puncts
money
email
phone
number
allcaps
url
date
time
You can check all supported list at malaya.preprocessing.get_normalize()
.
Example, if you set money
and number
, and input string is RM10k
, the output is <money>
.
annotate#
Supported annotate
,
hashtag
allcaps
elongated
repeated
emphasis
censored
Example, if you set hashtag
, and input string is #drmahathir
, the output is <hashtag> drmahathir </hashtag>
.
[2]:
string_1 = 'CANT WAIT for the new season of #mahathirmohamad \(^o^)/!!! #davidlynch #tvseries :))), TAAAK SAAABAAR!!!'
string_2 = 'kecewanya #johndoe movie and it suuuuucks!!! WASTED RM10... rm10 #badmovies :/'
string_3 = "@husein: can't wait for the Nov 9 #Sentiment talks! YAAAAAAY !!! :-D http://sentimentsymposium.com/."
string_4 = 'aahhh, malasnye nak pegi keje harini #mondayblues'
string_5 = '#drmahathir #najibrazak #1malaysia #mahathirnajib'
Preprocessing Interface#
def preprocessing(
normalize: List[str] = [
'url',
'email',
'percent',
'money',
'phone',
'user',
'time',
'date',
'number',
],
annotate: List[str] = [
'allcaps',
'elongated',
'repeated',
'emphasis',
'censored',
'hashtag',
],
lowercase: bool = True,
fix_unidecode: bool = True,
expand_english_contractions: bool = True,
segmenter: Callable = None,
demoji: Callable = None,
**kwargs,
):
"""
Load Preprocessing class.
Parameters
----------
normalize: List[str], optional (default=['url', 'email', 'percent', 'money', 'phone', 'user', 'time', 'date', 'number'])
normalizing tokens, can check all supported normalizing at `malaya.preprocessing.get_normalize()`.
annotate: List[str], optional (default=['hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'])
annonate tokens <open></open>,
only accept ['hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'].
lowercase: bool, optional (default=True)
fix_unidecode: bool, optional (default=True)
fix unidecode using `ftfy.fix_text`.
expand_english_contractions: bool, optional (default=True)
expand english contractions.
segmenter: Callable, optional (default=None)
function to segmentize word.
If provide, it will expand hashtags, #mondayblues == monday blues
demoji: object
demoji object, need to have a method `demoji`.
Returns
-------
result : malaya.preprocessing.Preprocessing class
"""
Load default paramaters#
default parameters able to translate most of english to bahasa malaysia.
[3]:
%%time
preprocessing = malaya.preprocessing.preprocessing()
CPU times: user 28.3 ms, sys: 68 µs, total: 28.4 ms
Wall time: 28.2 ms
[4]:
%%time
' '.join(preprocessing.process(string_1))
CPU times: user 576 µs, sys: 1.45 ms, total: 2.02 ms
Wall time: 2.03 ms
[4]:
'<allcaps> CANT WAIT </allcaps> untuk the new season of <hashtag> #mahathirmohamad </hashtag> \\(^o^)/ <repeated> ! </repeated> <hashtag> #davidlynch </hashtag> <hashtag> #tvseries </hashtag> <happy> , <allcaps> TAAAK SAAABAAR </allcaps> <repeated> ! </repeated>'
[5]:
%%time
' '.join(preprocessing.process(string_2))
CPU times: user 466 µs, sys: 0 ns, total: 466 µs
Wall time: 470 µs
[5]:
'kecewanya <hashtag> #johndoe </hashtag> movie and it suuuuucks <repeated> ! </repeated> <allcaps> WASTED </allcaps> <money> <repeated> . </repeated> <money> <hashtag> #badmovies </hashtag> <annoyed>'
[6]:
%%time
' '.join(preprocessing.process(string_3))
CPU times: user 408 µs, sys: 0 ns, total: 408 µs
Wall time: 412 µs
[6]:
'<user> : can not wait untuk the <date> <hashtag> #Sentiment </hashtag> talks ! <allcaps> YAAAAAAY </allcaps> <repeated> ! </repeated> <laugh> <url>'
[7]:
%%time
' '.join(preprocessing.process(string_4))
CPU times: user 214 µs, sys: 0 ns, total: 214 µs
Wall time: 217 µs
[7]:
'aahhh , malasnye nak pergi kerja hari ini <hashtag> #mondayblues </hashtag>'
[8]:
%%time
' '.join(preprocessing.process(string_5))
CPU times: user 152 µs, sys: 90 µs, total: 242 µs
Wall time: 244 µs
[8]:
'<hashtag> #drmahathir </hashtag> <hashtag> #najibrazak </hashtag> <hashtag> #1malaysia </hashtag> <hashtag> #mahathirnajib </hashtag>'
Load default paramaters with segmenter to expand hashtags.#
We saw <hashtag> drmahathir </hashtag> <hashtag> najibrazak </hashtag>
, we want to expand to become dr mahathir
and najib razak
.
[17]:
segmenter = malaya.segmentation.transformer(model = 'small', quantized = True)
[10]:
segmenter_func = lambda x: segmenter.greedy_decoder([x])[0]
segmenter_func('hellosuka')
2022-09-01 15:22:19.953149: I tensorflow/stream_executor/cuda/cuda_blas.cc:1760] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
[10]:
'hello suka'
[11]:
%%time
preprocessing = malaya.preprocessing.preprocessing(segmenter = segmenter_func)
CPU times: user 69 µs, sys: 0 ns, total: 69 µs
Wall time: 73 µs
[12]:
%%time
' '.join(preprocessing.process(string_1))
CPU times: user 113 ms, sys: 0 ns, total: 113 ms
Wall time: 166 ms
[12]:
'<allcaps> CANT WAIT </allcaps> untuk the new season of <hashtag> mahathir mohamad </hashtag> \\(^o^)/ <repeated> ! </repeated> <hashtag> davidlynch </hashtag> <hashtag> tv series </hashtag> <happy> , <allcaps> TAAAK SAAABAAR </allcaps> <repeated> ! </repeated>'
[13]:
%%time
' '.join(preprocessing.process(string_2))
CPU times: user 62 ms, sys: 3.12 ms, total: 65.2 ms
Wall time: 90.7 ms
[13]:
'kecewanya <hashtag> johndoe </hashtag> movie and it suuuuucks <repeated> ! </repeated> <allcaps> WASTED </allcaps> <money> <repeated> . </repeated> <money> <hashtag> bad movies </hashtag> <annoyed>'
[14]:
%%time
' '.join(preprocessing.process(string_3))
CPU times: user 27.7 ms, sys: 3.83 ms, total: 31.5 ms
Wall time: 45.1 ms
[14]:
'<user> : can not wait untuk the <date> <hashtag> Sentiment </hashtag> talks ! <allcaps> YAAAAAAY </allcaps> <repeated> ! </repeated> <laugh> <url>'
[15]:
%%time
' '.join(preprocessing.process(string_4))
CPU times: user 37.9 ms, sys: 6.61 ms, total: 44.6 ms
Wall time: 66.3 ms
[15]:
'aahhh , malasnye nak pergi kerja hari ini <hashtag> mondayblues </hashtag>'
[16]:
%%time
' '.join(preprocessing.process(string_5))
CPU times: user 123 ms, sys: 6.86 ms, total: 129 ms
Wall time: 182 ms
[16]:
'<hashtag> dr mahathir </hashtag> <hashtag> najib razak </hashtag> <hashtag> 1malaysia </hashtag> <hashtag> mahathir najib </hashtag>'