Transformers Config Tokenizers Expert

About this skill

Preflight reference for HuggingFace snapshots — what vLLM, sglang, and transformers.generate see at runtime. Covers config-file precedence (tokenizer.json, tokenizerconfig.json, generationconfig.json, chattemplate.jinja), transformers v5 tokenizer-class taxonomy (TokenizersBackend, PythonBackend, MistralCommonBackend, TikTokenTokenizer), special-token discovery (allspecialids, addedtokensdecoder, extraspecialtokens, backendtokenizer.getaddedtokensdecoder), chat-template Jinja contract (ImmutableSandboxedEnvironment, loopcontrols, raiseexception, strftimenow, tojson, addgenerationprompt), and engine knobs (skipspecialtokens, trustrequestchattemplate, chattemplatekwargs allowlist, adjustrequest, incremental detokenizer, EOS merge). Ships verified 2026 hall-of-shame for Kimi-K2.6, GLM-5.1, Gemma-4, Qwen3, DeepSeek-V3, plus drop-in Python for resolving markers to IDs, detecting turn-primer-as-EOS leaks, and cross-referencing tokenizer.json vs tokenizerconfig.json.

Quick start

Copy the install prompt. Use the green button above this README.
Send it to your agent. Paste it into your coding agent’s chat and send.
Review the setup. Your agent checks the skill’s instructions, requirements, and installation scope.

Installation guide →

Review, license & listing details

Review the source before using this skill

This is an imported listing. We haven’t certified its safety, license, or compatibility. Skills may include executable scripts.

Automated security scan: Partial

Only SKILL.md was scanned. Supporting files and scripts were not checked. No safety grade has been assigned.

0 potential findings · Scanned 11/4/2026 · Scanner 0.2.1

Content SHA-256: c7d616727a483b61e1185666d662ad311ac399d27cbe06f1182a1e0efd7b1a54

About scan coverage and limitations

Original listing metadata

Original name

transformers-config-tokenizers-expert

Original description

Preflight reference for HuggingFace snapshots — what vLLM, sglang, and transformers.generate see at runtime. Covers config-file precedence (tokenizer.json, tokenizer_config.json, generation_config.json, chat_template.jinja), transformers v5 tokenizer-class taxonomy (TokenizersBackend, PythonBackend, MistralCommonBackend, TikTokenTokenizer), special-token discovery (all_special_ids, added_tokens_decoder, extra_special_tokens, backend_tokenizer.get_added_tokens_decoder), chat-template Jinja contract (ImmutableSandboxedEnvironment, loopcontrols, raise_exception, strftime_now, tojson, add_generation_prompt), and engine knobs (skip_special_tokens, trust_request_chat_template, chat_template_kwargs allowlist, adjust_request, incremental detokenizer, EOS merge). Ships verified 2026 hall-of-shame for Kimi-K2.6, GLM-5.1, Gemma-4, Qwen3, DeepSeek-V3, plus drop-in Python for resolving markers to IDs, detecting turn-primer-as-EOS leaks, and cross-referencing tokenizer.json vs tokenizer_config.json.

About this listing

Category: Engineering
Reported license: Not specified
Listing language (estimated): English
Review status: Not individually reviewed
Agent compatibility: Atlas installation supports Claude Code, Codex, and Cursor. Skill-specific dependencies and features are checked during setup.