Separate job trouble from Dataflow cost
On this page
This check is due for a source refresh. Confirm the current documentation before you rely on provider-specific details.
Compare Dataflow estimates with Cloud Billing actuals before choosing cost-reduction work, or you may rank the wrong jobs because estimates can differ from billed cost. Use errors, delays, and scaling patterns for engineering follow-up, not as proof of spend.
What you need first
- Tool
- Use the Google Cloud console, Google's web interface. Start at Dataflow > Jobs, select a job, then open Cost. For Spark, use Cloud Monitoring, Google's metrics service: in Metrics Explorer, filter by "dataproc" and select a "Cloud Managed Service for Apache Spark" resource.
- Access
- Ask authorized service and billing colleagues for read-only results for the selected project, jobs, Spark resources, and billing account. Have each colleague cover their own service or billing scope.
- If you do not use that tool
- Send a Dataflow or Spark owner and a billing colleague the project, region, resource identifiers, billing account, and review period. Request Dataflow Cost and Autoscaling results, actual Dataflow costs from Cloud Billing reports, and relevant Spark metrics or autoscaler logs.
Why this is worth a look
You can send engineering effort to the wrong workload if you treat estimates or performance problems as proof of spend. Dataflow estimates multiply resource-use metrics by regional resource prices, while contractual discounts and temporary billing adjustments can change actual cost. Spark metrics and autoscaler logs diagnose performance, not billed-cost ranking.
Run this check
CHECKLISTFollow these steps in the Google Cloud console, or ask authorized colleagues to return the results. Record cost and performance separately for the same resources and review period. Make no changes.
READ-ONLY COST AND PERFORMANCE REVIEW
Execution surfaces:
- Google Cloud console: Dataflow job views and Cloud Billing reports.
- Cloud Monitoring: metrics service, using Metrics Explorer for Spark.
- Cloud Logging: log service, for Spark autoscaler logs.
Access: ask authorized service and billing colleagues for read-only results for their respective scopes if you cannot view them.
Scope: record project, Dataflow region and job IDs, relevant Spark cluster, job, batch, or session IDs, and billing account.
Time window: record start, end, and time zone. Mark results with different scopes or periods as unmatched.
Units: keep displayed cost currency and resource units. Worker values are counts. CPU utilization is CPU used divided by CPU available. Streaming Engine backlog time is in seconds.
[ ] In Dataflow > Jobs, select a job and open Cost. Select the review period. Record the estimated total and contributions from vCPUs, memory, Dataflow Shuffle or Streaming Engine data processed, and SSD or HDD usage.
[ ] Obtain actual Dataflow costs from Cloud Billing reports for the billing account. Record the available scope and period beside the estimate. Leave the cost ranking unresolved if they cannot be matched.
[ ] Review job errors, stage progress, and delays. For streaming jobs, also review data freshness and the Autoscaling tab. Record worker counts and mean CPU utilization over the review period. For Streaming Engine jobs, record backlog seconds and autoscaling rationale.
[ ] For Managed Service for Apache Spark, open Metrics Explorer. Type "dataproc" in "Filter by resource or metric name" and select the relevant "Cloud Managed Service for Apache Spark" resource. Record metric names, units, period, and gaps.
[ ] For an autoscaled Spark cluster, ask its owner for autoscaler logs from Cloud Logging covering the same period. Request scaling recommendations, metrics used in decisions, original cluster size, and new target size.
[ ] Record two separate outcomes: a cost question for the billing colleague and a performance question for the workload owner. Attach the relevant results. Do not change jobs, clusters, policies, or alerts.How to confirm it
- 01
Choose a comparable scope
Choose the jobs or Spark resources and one review period. Give service and billing colleagues the same identifiers and time zone. Do not compare a single-job estimate with a broader billing total.
- 02
Confirm the cost question
Compare Dataflow Cost results with Cloud Billing reports before ranking jobs. Use resource contributions to identify what drives the estimate, but leave actual-cost conclusions open when billing scope or time differs.
- 03
Assign performance follow-up
Send errors, delays, and scaling patterns to the workload owner. For Dataflow streaming jobs, compare worker changes with CPU utilization and backlog where available. For Spark, use metrics and autoscaler logs to frame the next diagnostic question.
- 04
Review trade-offs before changes
Ask the workload owner to assess both latency and cost before changing Dataflow autoscaling. A lower worker utilization hint can improve scale-up speed but increase total cost if more workers are provisioned.
Before making changes
Assume the views cover comparable resources and periods. Treat missing metrics as gaps, not zero usage. Dataflow Cost does not support Prime jobs or include additional GPU accelerator cost. It approximates TPU accelerator cost with vCPU and memory costs even though TPU workers do not incur those charges. An empty backlog chart without Streaming Engine does not mean no backlog. Spark custom metrics depend on enabled sources and overrides, and retention limits historical review.