Let us know how we can help
Sales & Pricing
Speak to a rep about your business needs
Help & Support
See our product support options
General inquiries and locations
Contact us
Control-M + Spark
Your Spark workflow starts before a PySpark script or Java/Scala application ever executes — and it can fail when something upstream breaks. An HDFS partition arrives late or below its expected size. A Hive transformation misses its processing window. Control-M orchestrates the full chain: HDFS file arrival, Hive transformation, Spark execution, and Snowflake delivery.
common workflow issues
Does this sound like your week?
LATE DATA
Your HDFS partition is late. The 2:00 a.m. Spark job is ready.
Control-M watches for the required HDFS file and minimum size before releasing the Spark job. If the arrival criteria are not met within the defined window, the watcher fails instead of launching processing against incomplete data.
JOB FAILURE
Your spark-submit fails. Three downstream jobs are waiting on its output.
Control-M tracks the Spark job as part of the end-to-end workflow, prevents dependent jobs from proceeding after a failure, and applies defined recovery actions so incomplete processing does not cascade into downstream delivery.
AIRFLOW · SPARK
Your Airflow DAG finished. The Spark workload still hasn’t started.
Control-M coordinates the dependency between the Airflow DAG and Spark processing, releasing the Spark job only after its upstream conditions are satisfied — without relying on disconnected schedules or manual handoffs between orchestration layers.
HDFS PREPARATION
Your Spark code is healthy. An HDFS preparation command just failed.
Control-M can run HDFS commands before or after Spark execution and determine whether command failure should fail the job. Preparation and cleanup become governed parts of the workflow instead of hidden scripting dependencies.
SLA RISK
Spark is still processing. Your downstream analytics deadline is approaching.
Control-M tracks the Spark workload inside the complete business service, applies SLA monitoring across dependent jobs, and surfaces delays before the end-to-end workflow misses its required completion time.
Control‑M + Spark
Control‑M + Spark
|
workload.types |
PySpark scripts · Java applications · Scala applications · Spark application JARs · application arguments · Spark command-line options |
|
trigger.type |
HDFS file arrival · minimum file size · upstream job completion · time schedule · workflow dependency · Control-M event |
|
cross_tool.deps |
Apache Airflow DAG · Hive job · HDFS commands · Snowflake job · Databricks job · GCP Dataproc · downstream file delivery |
|
cloud.platforms |
Hadoop/YARN clusters · on-premises and cloud-hosted Hadoop · runs alongside separate Control-M integrations for GCP Dataproc, Azure HDInsight, Amazon EMR, and OCI Data Flow |
|
error_handling |
pre-command failure handling · post-command failure handling · dependency gating · rerun actions · downstream cascade prevention · SLA alerting |
|
throughput |
distributed data processing · cluster-scale workloads · batch Spark applications · job-level orchestration · no per-record workflow scheduling |
|
observability |
job status · job output · YARN aggregated logs · workflow dependencies · SLA monitoring · end-to-end workflow visibility |
end-to-end orchestration
One production workflow. Every tool in the stack.
Control-M orchestrates workflows across Spark, Airflow, HDFS, Hive, Snowflake, file transfers, and cloud services in a single job flow — with dependency tracking, SLA visibility, and automated recovery across all of them.
- Cross-tool dependency: HDFS → Hive → Spark → Snowflake, orchestrated across separate Control-M job types in one flow
- Data-aware triggers: file arrival, API event, Airflow completion, upstream job result
|
Spark |
PySpark execution · Java/Scala applications · arguments · command-line options · job monitoring |
|
HDFS |
file watching · minimum-size criteria · commands · preparation · cleanup |
|
Hive |
script execution · parameters · dependency coordination · job monitoring |
|
Apache Airflow |
DAG trigger · status monitoring · cross-tool dependency coordination |
|
Snowflake |
SQL execution · data loading · stored procedures · downstream workflow coordination |
|
GCP Dataproc |
Spark batch, workflow templates, serverless Spark batches |
|
Azure HDInsight |
Spark batch execution, application parameters, Spark logs |
airflow coexistance
Control‑M doesn’t replace your Airflow DAGs. It runs the layer above them.
Airflow manages its DAG. Control-M manages everything surrounding it.
airflow handles
DAG-level orchestration inside the data pipeline
- DAG-level task orchestration within data pipelines
- Python operators, sensors, and task dependencies
- Execution graph for jobs that run inside your pipeline
- Manages retries within a single DAG context
control-m adds
The coordination layer around your DAGs
- Coordination layer around DAGs — triggers Airflow based on upstream conditions: file arrivals, API events, other tool completions
- Tracks each DAG’s SLA contribution across the full end-to-end workflow, not just its own routine
- Manages failure recovery when upstream dependencies fail before Airflow even starts
- Existing DAGs don’t need to be rewritten or migrated
MONITOR PIPELINES
Monitor Spark execution inside the complete data pipeline.
Spark shows what happens inside an application; the production pipeline usually extends across storage, transformation, orchestration, and delivery systems. Control-M gives data teams one operational view of the jobs and dependencies surrounding Spark:
-
Spark job execution status
-
Upstream and downstream dependencies
-
Job output and logs
-
End-to-end workflow visibility
-
Failure and recovery status
SLA ASSURANCE
Manage Spark against the deadline that actually matters.
A successful Spark application can still be too late for the business process depending on it. Control-M connects Spark execution to the end-to-end service SLA, helping teams identify delays and manage dependencies before delivery is missed:
-
End-to-end SLA tracking
-
Required completion time
-
Dependency-aware workflow monitoring
-
Delay and failure visibility
-
Coordinated recovery actions
