Tokenization

This reading categorizes tokenization into its types and is intended to interact with the current state of the art at tokenization level of NLP pipeline. It should also provide some idea of what one to choose.