Begin with the consumer and its deadline
A daily warehouse report, a model retrained weekly, and a fraud decision made in milliseconds have different freshness requirements. Specify source volume, arrival pattern, output deadline, acceptable staleness, and recovery behavior. SQL in BigQuery is a useful option for warehouse-resident transformations. An existing Spark workload may fit a managed Spark service. Apache Beam pipelines on Dataflow can process supported batch and streaming workloads. Evaluate operational fit, data movement, scale, latency, and cost instead of choosing a tool from its name alone.
A recommendation model refreshes customer interests daily, while a payment risk model needs recent transaction counts. For each, propose a processing cadence and explain what happens if the feature is stale.
Understand event time and watermarks
Event time describes when an event occurred; processing time describes when a pipeline processes it. Streaming data may arrive out of order. A window groups events by time, a watermark estimates how far event-time input has progressed, and triggers determine when results are emitted. A watermark is an estimate, not proof that no earlier event can arrive. Configure allowed lateness and update behavior according to the consumer’s tolerance for incomplete or revised aggregates.
A five-minute window covers 10:00 through 10:05. The watermark passes 10:05, then an event timestamped 10:04 arrives. Explain why it is late and how the configured lateness and trigger policy can affect whether an updated result is emitted.
Keep training and serving definitions aligned
Document the feature formula, entity key, window, timestamp semantics, source versions, default values, and freshness objective. Training examples must reflect values available at each prediction time, including late arrivals and corrections; a recomputed final aggregate may contain information the online system did not have. Use shared transformation logic or parity tests on representative records, and define a fallback when online features are missing or expired.
The online service counts transactions available during the previous hour. An offline training query instead counts all events eventually assigned to that hour. Explain the mismatch and propose what historical timestamps or snapshots are needed to reproduce online behavior.
Design for retries and observability
A distributed job can retry work or receive duplicate events. Choose appropriate deduplication keys, idempotent writes or upserts, and a correction policy for the sink; end-to-end behavior depends on both pipeline and destination. Monitor backlog, data freshness, watermark lag, error rates, and feature coverage. Test out-of-order and late events as well as ordinary input. Record the pipeline version and source snapshot or offsets so an investigation can reconstruct what produced a feature.
A customer event is delivered twice and then corrected. Define an event identifier, a sink update rule, and a test that checks both duplicate handling and the final feature value.
A streaming aggregate changes after an event arrives behind the watermark. What is the best explanation?
Keep exploring
- Dataflow streaming, watermarks, and triggers ↗
- Develop and test Dataflow pipelines ↗
- MLOps and feature processing ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.