Gemma (Google)

Prompting Gemma

How to prompt Gemma 4 open models for local, edge, multimodal, and agentic workflows without treating them like Gemini in the browser.

Which model to choose

Gemma answers a different question than Gemini: “What should run locally, on edge hardware, or in a self-hosted environment?” Google’s Gemma model overview lists a family that ranges from edge-sized models to larger local deployments. Use E2B or E4B for constrained devices, 12B Unified for medium multimodal local work, 26B A4B for efficient MoE workstation/server deployment, and 31B when maximum Gemma capability matters. The number in names like 12B or 31B is the rough parameter count, not a direct quality score.

Do not route general consumer web-chat guidance here. That belongs in the Gemini guide. Gemma guidance should focus on local prompting, chat templates, quantization-aware deployment, and model-card-specific constraints.

Prompting in GUI/local apps

Most Gemma users encounter it through local tools, hosted playgrounds, Hugging Face demos, LM Studio, Ollama-style runners, or Google AI Studio experiments. The GUI may hide the actual prompt serialization. When output looks unexpectedly off, check whether the app is using the instruction-tuned model and the correct chat template.

For local apps, keep prompts shorter and more explicit than you might with closed frontier models. Smaller open models are more sensitive to ambiguity, missing context, and malformed role formatting.

Prompting through local or hosted APIs

For Gemma, prompt formatting is part of the implementation. Hugging Face’s chat-template docs explain that chat messages are converted into a model-specific token sequence, and different models can use different control tokens.

If you are using transformers, use the tokenizer’s chat template rather than hand-rolling a prompt string. If you are serving through vLLM, SGLang, TGI, or another OpenAI-compatible wrapper, verify that the wrapper applies the expected template for the specific Gemma checkpoint.

Gemma 4 has native system-role support and a native thinking mode: place the <|think|> token at the start of the system prompt to enable reasoning, and do not replay prior thinking content in multi-turn history. It also supports native function calling. Use Google’s default sampling settings (temperature 1.0, top_p 0.95, top_k 64), and for multimodal prompts place image content before text and audio after text.

Local API skeleton:

{
  "model": "google/gemma-4-12B-it",
  "messages": [
    {
      "role": "system",
      "content": "You are a concise UI review assistant."
    },
    {
      "role": "user",
      "content": "Inspect this screenshot for clipped text, overlap, missing labels, and unclear controls."
    }
  ]
}

Current prompting guidance

• Use instruction-tuned checkpoints for chat and assistant behavior.
• Gemma 4 supports the system role natively; put system/developer-style instructions first, user task next.
• Be explicit about output format and constraints.
• For multimodal inputs, describe what the model should inspect rather than assuming it will infer the task from the attachment.
• Prefer concise examples over large few-shot blocks on smaller variants.
• Use Google’s default sampling (temperature 1.0, top_p 0.95, top_k 64) unless you have a reason to change it.
• Toggle reasoning with the <|think|> token in the system prompt; do not carry prior thinking content into later turns.
• Order multimodal content image-before-text and audio-after-text.

Examples

Local app prompt:

Inspect this screenshot and list the visible UI issues. Focus on overlap, clipped
text, missing labels, and controls that would be hard to use on a small screen.
Return only issues that are visible in the image.

API/local serving note:

Use the model's chat template for:
- system: You are a concise code-review assistant.
- user: Review this function for correctness bugs only.

To enable reasoning, put the <|think|> token at the start of the system content.

Prompt upgrade example:

Stale pattern: “Act as a genius multimodal AI and infer everything important from this image.”

Better pattern: “Inspect the image for these exact failure modes: clipped text, overlapping controls, missing labels, and ambiguous affordances.”