dan/llm-tokenizer
Watch
1
0
Fork
You've already forked llm-tokenizer
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
4 commits 1 branch 0 tags 14 KiB
  • Go 100%
Find a file
Exact
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Dan 70fdb5d4c3 when encoding, get merge candidates on text to encode rather than replay all trained encodes in order
2026-10-18 22:35:48 +01:00
cmd/tokenize working bpe tokenizer with crude impl, special cases TODO 2026-10-17 14:09:55 +01:00
go.mod working bpe tokenizer with crude impl, special cases TODO 2026-10-17 14:09:55 +01:00
mise.toml working bpe tokenizer with crude impl, special cases TODO 2026-10-17 14:09:55 +01:00
tokenizer.go when encoding, get merge candidates on text to encode rather than replay all trained encodes in order 2026-10-18 22:35:48 +01:00
tokenizer_test.go when encoding, get merge candidates on text to encode rather than replay all trained encodes in order 2026-10-18 22:35:48 +01:00