SSealbook

marcelroed/gigatoken

Language model tokenization at GB/s

llmnlptokenizationtokenizer

License & maintenance

MIT is a permissive license: free to use commercially with attribution.

Last code push 47 days ago.

Similar projects

RepositoryStarsLanguageLicenseActivity
1google/langextractA Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visua★ 39kPythonApache-2.0Active
2toon-format/toon🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmark★ 25.5kTypeScriptMITActive
3AiHubCN/Awesome-Chinese-LLM整理开源的中文大语言模型,以规模较小、可私有化部署、训练成本较低的模型为主,包括底座模型,垂直领域微调及应用,数据集与教程等。★ 22.8k——Slowing
4affaan-m/ECCThe agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Code★ 283.1kJavaScriptMITActive
5firecrawl/firecrawlSupercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥★ 192.7kTypeScriptAGPL-3.0Active
6Significant-Gravitas/AutoGPTAutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on w★ 187.6kPythonOtherActive
7ollama/ollamaGet up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.★ 183.7kGoMITActive
8huggingface/transformers🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, f★ 167.8kPythonApache-2.0Active