tokenizer

Past 30d, sorted by best first

03/03	8	ikawaha/Kagome v2 v2.11.0 Self-contained Japanese morphological analyzer in pure Go that tokenizes text and analyzes parts of speech. Go 954☆ 4272d old #golang #go #japanese #tokenizer #japanese-language
03/15	5	mck89/Peast v1.17.5 PHP 5.4+ library that parses JavaScript code to generate an abstract syntax tree following the ESTree standard. PHP 188☆ 3530d old #javascript #php #ecmascript #ast-generation #tokenizer