- Add js-tiktoken dependency for precise token counting across different languages
- Create new tokenCounter utility module with countTokens and countContentTokens functions
- Replace estimateTokens approximation logic with accurate tiktoken-based counting
- Update CompressionManager to use new token counter utilities for better context management
- Refactor estimateMessagesTokens to leverage accurate token counting with proper overhead calculation
- Add comprehensive test coverage for token counter utility
- Update tsconfig for proper module resolution
- Improve token estimation accuracy for Chinese characters, code, and structured content
- Remove estimateTokens from public exports in favor of new utility functions