ctok

C ★ 63 updated 6y ago

Expose CPython's tokenizer as a Python class

No plain-English explanation yet — one is being written right now. Check back in a minute.

Open on GitHub → Fully explain →