llama-models 仓库 Llama 4 Prompt 格式完全指南从特殊 Token 到图文对话与工具调用【免费下载链接】llama-modelsUtilities intended for use with Llama models.项目地址: https://gitcode.com/GitHub_Trending/ll/llama-models本文以 models/llama4/prompt_format.md 为骨架系统讲解 Llama 4 家族模型的完整提示词Prompt编码规范包括|begin_of_text|、|header_start|、|eot|等特殊 Token 的含义多轮对话、单图/多图视觉输入、以及三种工具调用Tool Calling方式的真实 Prompt 示例。读完后你将能手工构造或程序化生成可直接投喂给 Llama 4 Instruct 模型的输入文本并理解这些格式在 llama_models/llama4/chat_format.py 等源码中的落地实现。Llama 4 的对话骨架特殊 Token 与角色体系Llama 4 是一个原生的多模态模型家族支持文本与图像输入参见 llama_models/llama4/MODEL_CARD.md 对 natively multimodal 的描述。它延续了 Llama 3.x 的基于特殊 Token 的对话模板并新增了图像分块tile/patch相关 Token。理解这些 Token 是读懂一切 Llama 4 Prompt 的前提。特殊 Token 一览根据文档Llama 4 支持的对话相关特殊 Token 如下Token含义与使用场景|begin_of_text|标记 Prompt 的开始。整段输入的最开头必须出现一次|end_of_text|模型停止生成更多 Token。注意该 Token 仅由 base预训练模型生成Instruct 模型不输出它|header_start|/|header_end|一对围栏包裹某条消息的角色名role可选角色为 system、user、assistant|eot|End of Turn。表示模型已确定它与发起响应用户消息的交互结束。用于两种场景① 模型与用户一次直接交互的末尾② 模型与若干可用工具多次交互之后的末尾。它向执行器executor发出模型已完成响应生成的信号|image_start|/|image_end|一对围栏包裹 Prompt 中的图像数据|patch|表示图像 tile 的一个分块patch即视觉编码器输出的一个图像片段 Token|tile_y_separator|/|tile_x_separator|用于分隔图像的 y 方向与 x 方向 tile|image|在新架构中此 Token 将常规尺寸的图像信息与缩小到单个 tile 的版本分隔开计算缩放因子时取较长边其余部分通过填充适配到 tile 中从源码层面看这些 Token 全部注册在 llama_models/llama4/tokenizer.py 中。例如LLAMA4_TEXT_POST_TRAIN_SPECIAL_TOKENS包含|header_start|、|header_end|、|eom|、|eot|、|python_start|、|python_end|等LLAMA4_VISION_SPECIAL_TOKENS则包含|image_start|、|image_end|、|tile_x_separator|、|tile_y_separator|、|image|、|patch|等。Tokenizer 将这些 Token 映射为从num_base_tokens起的整数 ID并设置bos_id |begin_of_text|、eos_id |end_of_text|、eot_id |eot|、eom_id |eom|同时把|end_of_text|、|eom|、|eot|三者的 ID 一并加入stop_tokens列表作为生成阶段的中止条件见下文解码与停止条件。三种消息角色文档明确了 Llama 4 支持三种对话角色system设置与 AI 模型交互的上下文通常包含规则、指导原则或必要信息帮助模型有效响应user代表与模型交互的人类包含给模型的输入、命令与问题assistant代表 AI 模型基于 system、tool 与 user 提示中的上下文生成的响应。在 llama_models/llama4/chat_format.py 中role_str()把枚举角色映射为字符串其中Role.tool被特殊映射为ipython_encode_header()则按|header_start|{role}|header_end|\n\n的模板生成角色头possible_headers字典预先缓存了每种角色的头文本。这意味着每条消息在编码后都形如|header_start|role|header_end| 消息内容|eot|其中|eot|位于每条消息内容的末尾assistant 消息在特定条件下使用|eom|见后文。Llama 4 Instruct 模型的多轮对话格式文档给出的最基础用例是简单的用户与助手对话展示了一个常规多轮 user/assistant 会话如何被格式化。输入 Prompt 格式|begin_of_text||header_start|system|header_end| You are a helpful assistant|eot||header_start|user|header_end| Answer who are you in the form of jeopardy?|eot||header_start|assistant|header_end|模型响应格式What is a helpful assistant?|eot|格式拆解整段输入以|begin_of_text|开头每条消息 |header_start|{role}|header_end|\n\n 消息正文 |eot|关键细节最后一条assistant头之后不跟|eot|而是留给模型自己生成。即模板以|header_start|assistant|header_end|\n\n结尾模型从这里开始续写并在完成时自行输出|eot|。这一过程在ChatFormat.encode_dialog_prompt()中有严格对应先追加|begin_of_text|然后对每条消息调用encode_message()编码角色头、内容并追加|eot|或|eom|最后再_encode_header(assistant)补上待模型完成的 assistant 头。而在生成阶段llama_models/llama4/generation.py 中eos_reached | (~input_text_mask[:, cur_pos]) (torch.isin(next_token, stop_tokens))即一旦模型在非输入位置输出了stop_tokens|end_of_text|、|eom|、|eot|之一便终止解码——这也解释了为什么对话格式要求每个非末条消息都必须以|eot|收尾它既是消息边界也是生成终止信号。结合源码的实测入口如果你想在真实推理中观察这种格式可以直接运行仓库自带的示例脚本 llama_models/llama4/scripts/chat_completion.pypython -m llama_models.llama4.scripts.chat_completion \ --checkpoint_dir checkpoint 目录 \ --world_size 1 \ --max_seq_len 4096 \ --temperature 0.6 \ --top_p 0.9脚本内置了多条对话样本普通问答、带 system 提示的问答、以及多图 文本的多模态用例通过RawMessage(role..., content...)构造消息交由Llama4.build(...)与generator.chat_completion(batch, ...)逐 Token 流式返回。其中 system 提示示例RawMessage(rolesystem, contentAlways answer with Haiku)就对应了文档中 system 角色的作用。图像 Prompt 格式tile 与 patch 的编码规则Llama 4 是原生多模态模型图像在 Prompt 中不是被描述而是被编码一张图被切分成若干 tile分块每个 tile 再进一步切成 patch Token。文档用三个递进示例说明了这一机制。单图 Prompt —— 小图无需分隔符当图像尺寸小于单个 tile 大小时不需要 tile 分隔符。其图像段结构为|image_start||image||patch|...|patch||image_end|完整输入示例省略重复的|patch||begin_of_text||header_start|user|header_end| |image_start||image||patch|×N|image_end|Describe this image in two sentences|eot||header_start|assistant|header_end|模型响应示例The image depicts a dog standing on a skateboard, positioned centrally and facing the camera directly. The dog has a distinctive coat pattern featuring white, black, and brown fur, with floppy ears and a black nose, and is standing on a skateboard with red wheels.|eot|注意此例中图像段以|image_start||image|开头——|image|直接出现在开头是因为图像本身就小于一个 tile无需常规尺寸部分 缩小版的分离。单图 Prompt —— 大图含 tile 分隔符与双分辨率当图像大于 tile 大小时输入会包含 tile 分隔符且|image|出现在末尾区域用于把常规尺寸图像与缩小后的单 tile 版本隔开。其结构为|image_start||patch|...|patch||tile_x_separator||patch|...|patch||tile_y_separator||patch|...|patch||image||patch|...|patch||image_end|模型响应示例The image depicts a dog standing on a skateboard, with the dog positioned centrally and facing forward. The dog has a distinctive coat featuring a mix of white, brown, and black fur, and is wearing a collar as it stands on the skateboard, which has red wheels.|eot|结合 llama_models/llama4/chat_format.py 的_encode_image()可以精确理解这一结构若image_chunks 1单 tile输出|image_start||image| N 个|patch||image_end|否则按aspect_ratio (ratio_h, ratio_w)逐行输出 patch每行内用|tile_x_separator|分隔列行尾追加|tile_y_separator|全部行结束后输出|image|再追加一份缩小版本的|patch|序列最后以|image_end|收尾。|image|之后那一组 patch 正是文档所述downsized version that fits in a single tile它把整张图缩放scale factor 由较长边计算后填进一个 tile让模型在细粒度分块之外始终能看到全图缩略。多图 Prompt 格式多图时每张图像各自用一对|image_start|...|image_end|包裹图像之间直接拼接随后接文本指令|begin_of_text||header_start|user|header_end| |image_start|...|image_end||image_start|...|image_end|Describe these images in two sentences|eot||header_start|assistant|header_end|模型响应示例The first image features a dog standing on a skateboard, while the second image showcases a plate of spaghetti with tomato sauce and cheese. The two images appear to be unrelated, with one depicting a playful scene of a dog on a skateboard and the other presenting a classic Italian dish.|eom|注意此例的响应以|eom|收尾而不是|eot|——这是端到端消息标记End of Message在 assistant 消息携带工具调用或停止原因为 end_of_message 时使用参见 llama_models/llama4/datatypes.py 的StopReason与 llama_models/llama4/chat_format.py 中encode_message()对|eom|与|eot|的选择逻辑。这也是generation.py将|eom|列入stop_tokens的原因。图像如何变成 patch预处理与视觉编码如果你想深入为什么是这么多 patch答案在 llama_models/llama4/preprocess.pyIMAGE_RES 448即单个 tile 的基准分辨率patch 的视觉参数由 llama_models/llama4/args.py 的VisionArgsimage_size、patch_size、pixel_shuffle_ratio等定义VariableSizeImageTransform.__call__()的算法流程代码注释中总结为 6 步① 在max_num_chunks上限内枚举所有可用的画布组合② 依据宽高比挑选最佳画布get_best_fit()优先选择最小放大比例、面积最小的画布尽量少填充③ 无失真缩放④ 填充黑色像素补足画布⑤ 归一化mean/std 均为 0.5⑥ 按ratio_w × ratio_h切块返回(chunks, 3, 224, 224)的张量与宽高比ar在ChatFormat._encode_content()中若分块数大于 1还会用ResizeNormalizeImageTransform生成一张固定 448×448 的全局缩略图并拼接到 tile 序列末尾——这正是文档中|image|之后那组 patch 对应的downsized version编码得到的 tile 张量在 llama_models/llama4/vision/encoder.py 的VisionEncoder中经卷积 patch 化、class token、位置编码与 Transformer 编码后替换 Prompt 中对应的|patch|Token 位置见generation.py中image_mask tokens[:, prev_pos:cur_pos] self.tokenizer.special_tokens[|patch|]的掩码逻辑。工具调用Tool Calling格式文档说明 Llama 4 延续了此前版本 Llama 的 zero-shot 函数调用格式所有可用函数既可以放在 system 消息中也可以放在 user 消息中。下面三种写法逐一给出。方式一零样本函数调用 —— 函数列表放在 system 消息把完整的系统提示与 JSON 函数列表放入 system 消息|begin_of_text||header_start|system|header_end| You are a helpful assistant and an expert in function composition. You can answer general questions using your internal knowledge OR invoke functions when necessary. Follow these strict guidelines: 1. FUNCTION CALLS: - ONLY use functions that are EXPLICITLY listed in the function list below - If NO functions are listed (empty function list []), respond ONLY with internal knowledge or I dont have access to [Unavailable service] information - If a function is not in the list, respond ONLY with internal knowledge or I dont have access to [Unavailable service] information - If ALL required parameters are present AND the query EXACTLY matches a listed functions purpose: output ONLY the function call(s) - Use exact format: [func_name1(param1value1, param2value2), func_name2(...)] Examples: CORRECT: [get_weather(locationVancouver), calculate_route(startBoston, endNew York)] - Only if get_weather and calculate_route are in function list INCORRECT: get_weather(locationNew York) INCORRECT: Let me check the weather: [get_weather(locationNew York)] INCORRECT: [get_events(locationSingapore)] - If function not in list 2. RESPONSE RULES: - For pure function requests matching a listed function: ONLY output the function call(s) - For knowledge questions: ONLY output text - For missing parameters: ONLY request the specific missing parameters - For unavailable services (not in function list): output ONLY with internal knowledge or I dont have access to [Unavailable service] information. Do NOT execute a function call. - If the query asks for information beyond what a listed function provides: output ONLY with internal knowledge about your limitations - NEVER combine text and function calls in the same response - NEVER suggest alternative functions when the requested service is unavailable - NEVER create or invent new functions not listed below 3. STRICT BOUNDARIES: - ONLY use functions from the list below - no exceptions - NEVER use a function as an alternative to unavailable information - NEVER call functions not present in the function list - NEVER add explanatory text to function calls - NEVER respond with empty brackets - Use proper Python/JSON syntax for function calls - Check the function list carefully before responding 4. TOOL RESPONSE HANDLING: - When receiving tool responses: provide concise, natural language responses - Dont repeat tool response verbatim - Dont add supplementary information Here is a list of functions in JSON format that you can invoke: [ { name: get_weather, description: Get weather info for places, parameters: { type: dict, required: [city], properties: { city: { type: string, description: The name of the city to get the weather for }, metric: { type: string, description: The metric for weather. Options are: celsius, fahrenheit, default: celsius } } } } ]|eot||header_start|user|header_end| What is the weather in SF and Seattle?|eot||header_start|assistant|header_end|模型响应并行调用两个函数[get_weather(citySan Francisco), get_weather(citySeattle)]|eot|文档要点输出原生支持多个并行工具调用上述响应即一次并行调用两个get_weather函数定义采用 JSON 格式与 Llama 3.1 一致参数结构为type: dictrequiredproperties。从源码看这类响应正是 llama_models/llama3/tool_utils.py 中python_list格式的编码结果ToolUtils.encode_tool_call()在ToolPromptFormat.python_list分支把每个工具调用渲染为[func_name(kv, ...)]而解码侧maybe_extract_custom_tool_call()会先用正则、JSON、最后用ast解析is_valid_python_list()parse_python_list_for_function_calls()识别这类 Python 列表形式的调用并还原出函数名与关键字参数。ChatFormat.decode_assistant_message_from_content()再把这些信息包装成ToolCall附带call_id与arguments_json。方式二零样本函数调用 —— 函数列表放在 user 消息与方式一相同的是模型端输出格式完全一致区别只是函数列表出现在 user 消息里|begin_of_text||header_start|user|header_end| Questions: Can you retrieve the details for the user with the ID 7890, who has black as their special request? Here is a list of functions in JSON format that you can invoke: [ { name: get_user_info, description: Retrieve details for a specific user by their unique identifier. Note that the provided function is in Python 3 syntax., parameters: { type: dict, required: [ user_id ], properties: { user_id: { type: integer, description: The unique identifier of the user. It is used to fetch the specific user details from the database. }, special: { type: string, description: Any special information or parameters that need to be considered while fetching user details., default: none } } } } ] Should you decide to return the function call(s), put them in the format of [func1(params_nameparams_value, params_name2params_value2...), func2(params)] You SHOULD NOT include any other text in the response.|eot||header_start|assistant|header_end|模型响应[get_user_info(user_id7890, specialblack)]|eot|文档要点无论函数列表在 system 还是 user 消息中模型的工具调用格式完全相同。示例同时展示了required之外的special可选参数带默认值none如何被模型解析并填充。方式三自定义工具调用格式function标签如果不想使用上述默认格式可以通过 system/user 提示引导模型使用自定义的工具调用格式。下面这个例子定义了基于function标签的格式|begin_of_text||header_start|user|header_end| You have access to the following functions: Use the function trending_songs to Returns the trending songs on a Music site: {name: trending_songs, description: Returns the trending songs on a Music site, parameters: {genre: {description: The genre of the songs to return, param_type: str, required: false}, n: {description: The number of songs to return, param_type: int, required: true}}} Think very carefully before calling functions. If you choose to call a function ONLY reply in the following format with no prefix or suffix: functionexample_function_name{example_name: example_value}/function Reminder: - If looking for real time information use relevant functions before falling back to brave_search - Function calls MUST follow the specified format, start with function and end with /function - Required parameters MUST be specified - Only call one function at a time - Put the entire function call reply on one line|eot_id||eot||header_start|user|header_end| Use tools to get latest trending songs|eot||header_start|assistant|header_end|模型响应functiontrending_songs{n: 10}/function|eot|文档要点自定义格式下模型严格遵守以function开头、以/function结尾、参数为 JSON、整行输出的约定函数定义在这里使用非标准 JSONparam_type代替type、单行内联说明模型对函数描述形式有很强的容错性示例中的|eot_id|是该自定义提示自带的旧式分隔符随后仍以标准|eot|结束消息——实际使用时可按需取舍。这一格式在源码中同样有完整对应ToolUtils.encode_tool_call()在ToolPromptFormat.function_tag分支输出function{fname}{json_args}/function解码时CUSTOM_TOOL_CALL_PATTERN re.compile(rfunction(?Pfunction_name[^}])(?Pargs{.*?}))负责提取函数名与 JSON 参数llama_models/llama3/tool_utils.py。此外ToolPromptFormat.json分支会把工具调用编码为{type: function, name: ..., parameters: {...}}的标准 JSON解码侧maybe_extract_custom_tool_call()也支持这种形式。关于ToolCall中arguments_json字段的生成可参考测试 llama_models/llama4/tests/api/test_chat_format.py它用 mock 的maybe_extract_custom_tool_call/maybe_extract_builtin_tool_call验证了自定义工具、内置工具brave_search、code_interpreter以及复杂参数列表、布尔、浮点场景下arguments_json都能与解析出的arguments保持一致。解码与停止条件模型输出如何还原为消息理解了输入编码再看输出侧。ChatFormat.decode_assistant_message_from_content()llama_models/llama4/chat_format.py负责把模型生成的 Token 序列还原为结构化消息主要步骤剥离开头的 assistant 角色头|header_start|assistant|header_end|\n\n检查是否以|python_start|开头代码解释器场景并移除|python_end|根据结尾 Token 判定停止原因以|eot|结尾 →StopReason.end_of_turn以|eom|结尾 →StopReason.end_of_message依次尝试解析自定义工具调用maybe_extract_custom_tool_call与内置工具调用maybe_extract_builtin_tool_call内置工具通过BUILTIN_TOOL_PATTERN r\b(?Ptool_name\w)\.call\(query(?Pquery[^]*)\)匹配例如brave_search.call(query...)会被还原为BuiltinTool.brave_search若命中工具调用则生成带call_id与arguments_json的ToolCall并把消息正文清空否则正文原样保留。生成侧generation.py以stop_tokens|end_of_text|、|eom|、|eot|作为解码终止条件并在对话模板最后预留了 assistant 角色头等待模型续写——这与文档中所有示例以|eot|/|eom|收尾的观察完全自洽。常见问题与格式核对清单基于文档与源码整理一份实用核对清单帮助你在手工构造 Prompt 或排查推理异常时快速定位问题开头与结尾整段输入必须以|begin_of_text|开始每条非末位消息以|eot|结束模板以 assistant 头收尾不要替模型补|eot|角色头配对|header_start|与|header_end|必须成对角色只能取 system / user / assistant工具消息在源码中映射为ipython属于内部使用图像段完整性|image_start|...|image_end|必须成对小图结构为|image|前置大图为 tile 分隔符 末尾|image| 缩小版 patch多图时每张图独立成段工具调用默认格式为[func(kv, ...)]的 Python 列表支持并行多个调用也可在提示中自定义function....../function格式函数列表放 system 或 user 均可模型输出格式不受影响停止 Token 语义base 模型使用|end_of_text|Instruct 模型使用|eot|对话结束或|eom|工具调用后的消息结束视觉能力边界llama_models/llama4/MODEL_CARD.md 指出 Llama 4 官方针对最多 5 张输入图像的图像理解做过测试超出此规模时需自行评估与调优。以上格式细节均可对照 models/llama4/prompt_format.md 原文以及在 llama_models/llama4/ 下的chat_format.py、preprocess.py、tokenizer.py、generation.py与 llama_models/llama3/tool_utils.py 中逐一验证是开发 Llama 4 对话、视觉问答与 Agent 工具调用能力的可靠参考。【免费下载链接】llama-modelsUtilities intended for use with Llama models.项目地址: https://gitcode.com/GitHub_Trending/ll/llama-models创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考