Let us know how we can help
Sales & Pricing
Speak to a rep about your business needs
Help & Support
See our product support options
General inquiries and locations
Contact us
Control-M + GCP Dataproc
Your GCP Dataproc workflow starts before a Spark batch ever executes — and it can fail when something upstream breaks. A Cloud Storage object arrives late. A BigQuery preparation job misses its expected completion window. Control-M orchestrates the full chain: Cloud Storage arrival, BigQuery preparation, Dataproc execution, and downstream delivery — so your Spark batch runs only when the data it depends on is actually ready.
common workflow issues
Does this sound like your week?
These aren’t edge cases. They’re the normal operating conditions for teams running GCP Dataproc jobs across multiple tools. Here’s how Control-M handles each one.
LATE DATA ARRIVAL
Your 2 a.m. Spark job starts. The Cloud Storage object is late.
Control-M makes the upstream data arrival part of the job flow, so Dataproc processing waits for the confirmed arrival of the required input instead of starting on a fixed time schedule. The dependent workload starts only when its prerequisite completes, preventing incomplete-data processing.
FAILED BATCH
Your serverless Spark batch was cancelled. Downstream processing is still waiting.
Control-M monitors Dataproc execution and recognizes a cancelled batch as a failure state. Downstream dependencies remain blocked instead of advancing on incomplete processing, giving operators a controlled recovery path and preventing the failure from cascading through the pipeline.
STATUS POLLING
The Spark job is still running. Your next stage cannot safely start.
Control-M polls the Dataproc job state at a configurable verification interval and applies a defined tolerance before ending the job Not OK. Downstream work follows the actual execution state rather than an estimated completion window.
CROSS-TOOL DEPENDENCY
Dataproc finished successfully. The BigQuery handoff still needs coordinating
Control-M places Dataproc and BigQuery jobs in the same end-to-end workflow, so downstream processing can depend on successful upstream completion. The handoff follows job state rather than separate schedules, reducing timing gaps between processing stages.
DUPLICATE EXECUTION
A serverless batch request is retried. You cannot risk duplicate processing.
For Dataproc Serverless for Spark batches, Control-M supports Batch ID and Requested ID parameters, where the Requested ID identifies the CreateBatch request. Dataproc ignores a second request carrying the same ID and returns the operation associated with the original batch instead.
Control‑M + GCP Dataproc
Control‑M + GCP Dataproc
|
workload.types |
Workflow Templates · single Dataproc jobs · Dataproc Serverless for Spark batches · interactive sessions · big-data processing · machine-learning workloads |
|
trigger.type |
file arrival · upstream job completion · time schedule · Control-M dependency · API-driven execution · cross-application job state |
|
cross_tool.deps |
GCP BigQuery job · GCP Dataflow job · GCP Composer DAG · Cloud Storage file · file transfer completion · REST API call |
|
cloud.platforms |
Google Cloud Platform · Dataproc · Dataproc Serverless for Spark · Control-M SaaS · Control-M Web · Automation API · hybrid enterprise workflows |
|
error_handling |
verification poll interval · configurable tolerance · cancelled-batch failure detection · downstream cascade prevention · dependency-based recovery · Requested ID |
|
throughput |
parallel Dataproc processing · serverless Spark batches · large-scale batch processing · big-data workloads |
|
observability |
Dataproc job status · results and output · dependency status · workflow monitoring · execution history · end-to-end job visibility |
end-to-end orchestration
One production workflow. Every tool in the stack.
Control-M orchestrates workflows across GCP Dataproc, BigQuery, Dataflow, Cloud Composer, Cloud Storage, file transfers, and cloud services in a single job flow — with dependency tracking, SLA visibility, and automated recovery across all of them..
- Cross-tool dependency: Cloud Storage → BigQuery → GCP Dataproc → analytics handoff
- Data-aware triggers: file arrival, API event, upstream job completion, Dataproc completion
|
GCP Dataproc |
Workflow Templates · Dataproc jobs · Serverless for Spark batches · interactive sessions · status polling |
|
GCP BigQuery |
data processing · analytics jobs · upstream preparation · downstream analytics |
|
GCP Dataflow |
batch processing · streaming pipelines · cross-job dependencies · workflow handoffs |
|
GCP Composer |
Airflow DAG execution · DAG rerun (option to retry only failed tasks) · cross-platform dependency · workflow coordination |
|
Cloud Storage |
file arrival · input staging · output delivery · data-driven dependencies |
|
File transfers |
file watching · managed transfer · delivery confirmation · downstream processing trigger |
|
REST APIs |
application calls · workflow handoffs · API-driven automation · cross-platform coordination |
airflow coexistance
Control‑M doesn’t replace your Airflow DAGs. It runs the layer above them.
The objection is common: “We’re already on Airflow.” The issue isn’t what Airflow does – it’s what happens before and after Airflow runs. That’s where pipelines actually fail.
Airflow manages its DAG. Control-M manages everything surrounding it.
airflow handles
DAG-level orchestration inside the data pipeline
- DAG-level task orchestration within data pipelines
- Python operators, sensors, and task dependencies
- Execution graph for jobs that run inside your pipeline
- Manages retries within a single DAG context
control-m adds
The coordination layer around your DAGs
- Coordination layer around DAGs — triggers Airflow based on upstream conditions: file arrivals, API events, other tool completions
- Tracks each DAG’s SLA contribution across the full end-to-end workflow, not just its own routine
- Manages failure recovery when upstream dependencies fail before Airflow even starts
- Existing DAGs don’t need to be rewritten or migrated
MONITOR PIPELINES
Monitor Dataproc execution across your complete data pipeline.
Dataproc reports execution inside its own service, but production pipelines extend across storage, preparation, processing, and delivery. Control-M brings those connected jobs into one operational view so teams can monitor execution in end-to-end context:
-
Dataproc job status
-
Results and job output
-
Upstream and downstream dependencies
-
Cross-platform execution visibility
-
Centralized workflow monitoring
SLA ASSURANCE
Keep Dataproc pipelines aligned to delivery deadlines.
A successful Dataproc job does not guarantee the complete data service finishes on time. Control-M tracks the processing step within its broader workflow, helping teams identify dependency delays and manage delivery against end-to-end service expectations:
-
End-to-end SLA visibility
-
Upstream dependency tracking
-
Downstream delivery monitoring
-
Exception-driven operational response
-
Cross-platform workflow status
Documentation
Bring order to complex workflows
Learn how Control-M helps teams orchestrate complex processes with greater visibility, coordination, and control.
