Skip to content

Add token counting tokens to ITokenizer and TikTokenizer - #58

Open
Peter Groenewegen (pgroene) wants to merge 2 commits into
microsoft:mainfrom
pgroene:main
Open

Add token counting tokens to ITokenizer and TikTokenizer#58
Peter Groenewegen (pgroene) wants to merge 2 commits into
microsoft:mainfrom
pgroene:main

Do not allocate not needed arrays for the sequences in the cache.

26ab79b
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs