Establish a measurable baseline
Define the task, acceptable output, users, constraints, and failure costs. Create a representative evaluation set that includes ordinary inputs, ambiguous requests, edge cases, and unsafe or unanswerable requests. Start with a clear prompt and a suitable base model. Record the model version, prompt, decoding settings, input context, latency, cost, and evaluation results. A few attractive examples do not establish reliability across the real workload.
For a support-ticket classifier, specify the allowed labels, an abstain option, a representative evaluation set, and a metric for each critical class. Explain how you would handle a ticket that belongs to more than one class.
Match the intervention to the failure
Improve instructions or examples when the model misunderstands a task or format. Add authorized retrieval when answers need current or private evidence, and evaluate retrieval separately from generation. Consider supervised fine-tuning when a stable behavior remains difficult to express consistently and you have high-quality representative input-output examples. These approaches can be combined: tuning does not make changing facts current, and retrieval does not by itself guarantee that a model follows the requested format.
A policy assistant correctly follows its response format but cites an old reimbursement limit. Compare updating the retrievable source with fine-tuning on the old limit. Name a case where tuning might still help the assistant.
Prepare tuning data without leaking evaluation
Supervised fine-tuning uses examples of inputs and desired outputs. Review labels for correctness, safety, coverage, duplicates, and sensitive information. Separate training, validation, and final test examples by the relevant unit, such as customer, document family, or time period, so near-duplicates do not leak across splits. Use validation to compare tuning choices and reserve the final test for the selected candidate. Google documents supported dataset formats and model-specific limitations; verify the current requirements before creating a job.
You have 500 support conversations with multiple messages each. Explain why assigning individual messages from the same conversation to different splits can inflate results. Propose a conversation-level split and a separate recent-time evaluation.
Compare quality, risk, and operating cost
Evaluate the tuned candidate and base system on the same held-out cases, including important slices. Check task quality, harmful or unsupported responses, refusal behavior, response time, and total training and serving costs. Automated model-based judges can scale review but should be calibrated against human judgments and checked for systematic bias. Keep the prompt and retrieval configuration under version control; a tuned model is one part of the complete system, not a substitute for access controls or monitoring.
A tuned model improves overall format compliance from 88% to 95% but becomes worse on a rare safety-critical slice. Define a release gate and describe the evidence needed before deployment.
A policy assistant follows the desired style but repeatedly cites a reimbursement amount that changes each month. What should you investigate first?
Keep exploring
- Supervised fine-tuning overview ↗
- Prepare supervised tuning data ↗
- Generative AI evaluation overview ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.