Define the service objective
Specify the user-visible deadline, expected request rate, concurrency, availability, and cost constraints. Measure the complete path: authentication, feature lookup or retrieval, preprocessing, network calls, inference, postprocessing, and response delivery. Record percentiles and error rates under representative load. For streaming generative responses, time to first token and time to last token answer different user-experience questions. Average latency alone can hide slow tails.
An application’s average response time is acceptable but its 99th percentile breaches the objective during a traffic burst. List measurements for queueing, feature lookup, model execution, and errors. Explain why component percentiles cannot simply be added to obtain the end-to-end percentile.
Compare online and asynchronous work
Use online inference when a result must be returned during an interactive request. For work that tolerates a later deadline, evaluate batch inference and include queue time, job failures, retries, storage, and downstream delivery in the decision. For an online server, batching several requests may improve accelerator utilization and throughput, but waiting to form a batch can increase per-request latency. Test the actual request-size and arrival distribution rather than assuming larger batches always help.
A nightly catalog enrichment job has six hours before its output is needed. Design a batch completion and retry check. A checkout request has a 200 ms deadline: explain why the same queueing policy may be unsuitable.
Scale based on measured bottlenecks
More replicas, larger machines, or accelerators can increase capacity, but they also change cost and may not help if the bottleneck is a feature store, network, or serialized preprocessing. Autoscaling needs time to react and can be limited by quotas or cold starts. Load-test representative traffic ramps and burst patterns, and monitor queue length, utilization, timeouts, and saturation. Set minimum capacity and fallback behavior according to the service objective and budget.
At peak, requests wait in a queue while model instances are fully utilized. Contrast that with a case where model instances are idle but feature retrieval is slow. Which component would you investigate in each case?
Measure quality after model optimization
A smaller model, quantization, or another serving configuration may reduce compute or memory needs, but the quality and behavior of the changed system must be tested. Compare versions on the same representative held-out cases, important slices, and operational load. For generative models, also measure output length and token usage, safety, evidence support, and response quality. Verify current platform support, pricing, and regional constraints before selecting an option.
A smaller model reduces median latency by 35% but fails a required safety slice. Explain why the latency gain alone is insufficient and what candidate comparison you would run next.
Online model instances are idle, but request latency is dominated by feature retrieval. What is the best first investigation?
Keep exploring
- Configure inference compute ↗
- Generative inference consumption and latency ↗
- Autoscaling inference workloads ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.