Identify the constraint before distributing
A model that fits on one device but trains too slowly presents a different constraint from one that cannot fit in device memory. Start with a measured single-device baseline where feasible, including examples per second, memory use, input wait, and cost. Data parallelism replicates the model and distributes batches; model parallelism partitions model computation or parameters across devices. Hybrid strategies exist, but additional devices introduce communication and coordination costs.
A model fits on one accelerator but training takes too long. Which strategy is a reasonable first investigation? Now consider a model whose parameters do not fit on a single accelerator. Explain why the second case needs a different memory strategy.
Understand batch size and synchronization
In synchronous data parallelism, replicas process different data and coordinate updates, often aggregating gradients. The effective global batch depends on the per-device batch, number of replicas, and any gradient accumulation. Increasing it can change convergence, memory use, and the number of optimizer updates per epoch; compare quality as well as throughput. Ensure data is sharded correctly so replicas do not unintentionally process the same examples, and handle the final partial batch deliberately.
Four replicas each process 32 examples per step with no accumulation. The global batch is 128 examples. With two accumulation steps before an update it is 256, assuming each replica processes a fresh batch each time. Explain why changing this may require retuning the learning rate.
Measure scaling and communication
Speedup compares throughput or elapsed time against an appropriate baseline for the same work. If one device processes 100 examples per second and four process 300, speedup is 3 times and scaling efficiency is 75%. Investigate input starvation, uneven work, communication, and synchronization before adding more hardware. Agent Platform’s distributed training documentation describes worker pools and a Reduction Server option for supported multi-host GPU configurations; it is not a universal fix for every bottleneck.
A four-device job has low accelerator utilization and high data-loader wait. A second job has high communication time between workers. Identify a different first investigation for each.
Plan for restart and reproducibility
Workers can restart, so save checkpoints to durable storage and test recovery. A useful checkpoint may need model weights, optimizer and scheduler state, training step, and random or data-loader state, depending on the framework and reproducibility requirements. Coordinate writes to avoid conflicting checkpoints. Record code, data versions, topology, parameters, and environment; a fixed seed alone does not guarantee bit-for-bit identical distributed execution. Verify resumed quality and cleanup behavior.
A job resumes from model weights but loses optimizer state and the training step. Describe how the continuation may differ from the original run and what evidence you would capture in a recovery test.
A model fits on one GPU but training is too slow. Which approach is a sensible initial candidate to evaluate?
Keep exploring
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.