Files
adnaan 6ab0a79458 feat(tokenization): add js-tiktoken integration for accurate token counting
- Add js-tiktoken dependency for precise token counting across different languages
- Create new tokenCounter utility module with countTokens and countContentTokens functions
- Replace estimateTokens approximation logic with accurate tiktoken-based counting
- Update CompressionManager to use new token counter utilities for better context management
- Refactor estimateMessagesTokens to leverage accurate token counting with proper overhead calculation
- Add comprehensive test coverage for token counter utility
- Update tsconfig for proper module resolution
- Improve token estimation accuracy for Chinese characters, code, and structured content
- Remove estimateTokens from public exports in favor of new utility functions
2026-01-29 20:00:06 +08:00
..