Hacker Next
From text to token: How tokenization pipelines workparadedb.com
132 points by philippemnoel 3 days ago | 20 comments