#tokenization

Counting tokens at scale using tiktoken

Using encode_to_numpy to count number of tokens in large strings to save memory and CPU