AWS Glue Concurrent Execution: What Happens When Multiple Jobs Write to the Same S3 Location?

Imagine you have two AWS Glue jobs running in production.

Both jobs are processing customer data. Both read different source datasets, perform transformations, and write their results to the same S3 location.

Everything works fine when you execute them individually.

But one day, both jobs start running at almost the same time.

One job finishes successfully. The other also reports success.

However, when you query the final data using Athena, some records are missing, some records appear twice, and the output doesn't look the way you expected.

Where did things go wrong?

This is one of the important problems to understand when building production data pipelines with AWS Glue and Amazon S3.

In this article, let's look at what happens when multiple Glue jobs write to the same S3 prefix, why it creates unexpected results, and how to design a safer approach.

1. Understanding the problem

Consider the following architecture.

You have two Glue jobs:

• customer_daily_job: Processes daily customer records.
• customer_correction_job: Processes corrected customer records.

Both jobs write to:

s3://company-data-lake/silver/customers/ 

The architecture looks like this:

 Source System | ------------------- | | v v Glue Job A Glue Job B Daily Load Correction Load | | | | v v --------------------------- | | v v Same S3 Output Location s3://company-data-lake/silver/customers/ 

The problem is that S3 doesn't automatically coordinate these two jobs.

Amazon S3 provides strong read-after-write consistency for object operations, but it does not provide a transaction or lock for an entire prefix.

In other words, S3 doesn't know that Glue Job A and Glue Job B are supposed to work together.

Each job independently executes its write operation.

The outcome depends on the write mode, partitioning strategy, and output commit protocol.

2. What happens when both jobs use overwrite mode?

Let's start with a common PySpark implementation.

Both Glue jobs use:

output_path = "s3://company-data-lake/silver/customers/" df.write \ .mode("overwrite") \ .parquet(output_path) 

Now imagine this execution sequence:

Timeline:

Time 0:00 Glue Job A starts writing customer data. Time 0:02 Glue Job B starts writing to the same location. Time 0:03 Job A begins committing its output. Time 0:04 Job B performs its own overwrite operation. Time 0:05 Both jobs finish writing. 

What happens to the final output?

Depending on the Spark output committer, write mode, and timing, one job can interfere with the other's output.

Possible outcomes include:

• One job's output replacing the other's.
• Previously written files being removed during an overwrite.
• Unexpected combinations of output files.
• Failed commits or incomplete output.
• A successful job run producing data that doesn't represent the expected business result.

The important point is that overwrite is not a distributed transaction across multiple Glue jobs.

You should never assume that two concurrent overwrite operations against the same S3 prefix are safe.

3. What if both jobs use append mode?

Now let's change the write mode.

df.write \ .mode("append") \ .parquet(output_path) 

Append mode doesn't normally replace existing files.

Instead, Spark writes additional Parquet files into the target location.

This sounds safer, but it introduces another problem.

Imagine Job A processes these records:

At the same time, Job B processes:

Both jobs append their output to S3.

The final dataset can contain:

Customer C101 appears twice.

S3 accepted both sets of files because they were different objects.

But from a business perspective, the dataset now contains duplicate records.

This is an important distinction:

Successful S3 writes do not guarantee correct business data.

Append mode is appropriate when the incoming data is genuinely additive and duplicate handling is part of the pipeline design.

4. Partition-level conflicts are another problem

Let's assume your data is partitioned by business date.

df.write \ .mode("overwrite") \ .partitionBy("business_date") \ .parquet(output_path) 

Your S3 structure looks like this:

silver/customers/ business_date=2026-11-01/ business_date=2026-11-02/ business_date=2026-11-03/ 

Now imagine:

• Glue Job A processes 2026-11-03.
• Glue Job B also processes 2026-11-03.

Both jobs are targeting the same partition.

If both jobs overwrite that partition, they can interfere with each other's output.

This is especially dangerous when the jobs have different source snapshots or business logic.

For example, one job might contain 10,000 customer records while another contains only 8,000 corrected records.

The final partition might not represent either complete dataset.

How to reduce this risk

Assign clear ownership of partitions.

For example:

Glue Job A: business_date = 2026-11-01 Glue Job B: business_date = 2026-11-02 Glue Job C: business_date = 2026-11-03 

When different jobs own different partitions, the chance of conflicting writes is significantly reduced.

However, partition ownership must be enforced by your orchestration or application logic. Merely partitioning the data does not prevent two jobs from writing to the same partition.

5. How to identify this problem in production

Concurrent-write problems aren't always obvious from Glue execution logs.

You might see both jobs marked as SUCCEEDED, while the final S3 data is incorrect.

Here are some checks worth implementing.

Check 1: Compare expected and actual record counts

expected_count = source_df.count() actual_count = spark.read \ .parquet(output_path) \ .count() print("Expected:", expected_count) print("Actual:", actual_count) 

A difference doesn't automatically prove concurrent writes, but it is a useful signal.

Check 2: Check for duplicate business keys

from pyspark.sql.functions import count df = spark.read.parquet(output_path) duplicates = ( df.groupBy("customer_id") .agg(count("*").alias("record_count")) .filter("record_count > 1") ) duplicates.show() 

Check 3: Inspect the S3 prefix

aws s3 ls \ s3://company-data-lake/silver/customers/ \ --recursive 

Check:

• Unexpected file counts.
• Multiple output batches.
• Missing partitions.
• Unexpected file modification patterns.

Check 4: Review Glue execution history

Compare the start and end times of Glue job runs.

If multiple jobs target the same output location and their execution windows overlap, investigate their write operations.

You should also review the job's Spark logs and output committer errors.

6. Solution 1: Use separate S3 staging locations

One of the simplest ways to avoid direct write conflicts is to give every Glue job its own staging location.

Instead of writing directly to the final output prefix:

silver/customers/ 

Use:

staging/customers/job_a/run_id=001/ staging/customers/job_b/run_id=002/ 

Example:

job_run_id = "001" staging_path = ( f"s3://company-data-lake/staging/" f"customers/run_id={job_run_id}/" ) df.write \ .mode("overwrite") \ .parquet(staging_path) 

Each job now writes independently.

The architecture becomes:

 Source Data | ------------------- | | v v Glue A Glue B | | v v Staging/Job A Staging/Job B | | ------------------- | v Validation Step | v Controlled Publisher | v Silver/Customers/ 

The important part is the controlled publisher.

Only one process should publish a given target partition or dataset at a time.

Do not simply copy both staging directories into the final prefix. You still need a defined conflict-resolution and publishing strategy.

7. Solution 2: Control execution using AWS Step Functions

Another practical approach is to control which Glue job can write to a particular S3 location.

AWS Step Functions can orchestrate the execution sequence.

For example:

 Step Functions | v Run Customer Job | v Validate Output | v Run Correction Job | v Validate Output | v Publish Dataset 

Instead of allowing both jobs to write simultaneously, the orchestration ensures that they execute in a controlled order.

This is useful when:

• Multiple jobs update the same dataset.
• Jobs depend on previous transformations.
• Data quality checks must complete before publishing.
• The output requires a single-writer strategy.

If you're using a Step Functions Map state to execute multiple jobs, configure concurrency according to the ownership of their output locations.

For example, jobs targeting the same dataset can be grouped and executed sequentially, while jobs targeting independent datasets can run in parallel.

8. Solution 3: Implement a distributed lock using DynamoDB

Sometimes sequential execution is not enough.

You might have multiple pipelines triggered by EventBridge, S3 events, or external applications.

In this situation, a DynamoDB-based locking mechanism can help coordinate writers.

For example, create a DynamoDB table:

Table: PipelineLocks Partition Key: resource_name 

Before starting a write operation, the Glue job attempts to acquire a lock.

Conceptually:

import boto3 from botocore.exceptions import ClientError dynamodb = boto3.client("dynamodb") def acquire_lock(resource_name, job_id): try: dynamodb.put_item( TableName="PipelineLocks", Item={ "resource_name": {"S": resource_name}, "job_id": {"S": job_id} }, ConditionExpression=( "attribute_not_exists(resource_name)" ) ) return True except ClientError as error: if error.response["Error"]["Code"] == ( "ConditionalCheckFailedException" ): return False raise 

The condition ensures that only one job can acquire the lock for a given resource.

For example:

if acquire_lock("silver/customers", "glue-job-001"): print("Lock acquired. Starting write operation.") else: raise RuntimeError( "Another job is already writing to this dataset." ) 

Important production considerations:

A production lock needs more than an insert operation.

You should also implement:

• Lock expiration or lease management.
• Safe lock release.
• Recovery from failed Glue jobs.
• Ownership verification before releasing a lock.
• Heartbeats for long-running jobs.
• Protection against an expired job continuing to write.

A lease-based design with conditional updates is safer than a permanent lock.

9. Solution 4: Use Apache Iceberg for concurrent table writes

If you're building a large data lake, this is worth considering.

Traditional Parquet files stored directly in S3 don't provide table-level transactions.

Apache Iceberg introduces a metadata layer that manages snapshots and table changes.

Instead of managing individual Parquet files manually, applications commit changes through the Iceberg table metadata.

For example:

df.writeTo( "glue_catalog.silver.customers" ).append() 

Iceberg supports atomic metadata commits and optimistic concurrency control.

This means concurrent writers can detect conflicting changes rather than simply relying on multiple processes writing files into the same directory.

However, concurrent commits can still conflict, and applications may need retry or conflict-resolution logic.

You also need to configure a compatible Iceberg version, Glue Catalog integration, and appropriate locking or commit coordination for your environment.

When should you consider Iceberg?

• Multiple pipelines update the same logical table.
• You need reliable snapshot management.
• You need time travel.
• You need schema evolution.
• You want safer table-level writes.

For a production lakehouse, Iceberg is often a better foundation than treating a raw S3 prefix as a transactional table.

10. What I would implement in a production environment

For a production data lake, I would not allow multiple independent Glue jobs to directly overwrite the same final S3 location.

I would use the following approach:

 EventBridge | v Step Functions | ------------------- | | v v Glue Job A Glue Job B | | v v Staging A Staging B | | ------------------- | v Data Validation | v Controlled Publish | v Apache Iceberg | v Athena 

The responsibilities would be:

1. Glue jobs process data independently.
2. Each job writes to a unique staging prefix.
3. Data quality checks validate the output.
4. Step Functions coordinates publishing.
5. Iceberg manages transactional table updates.
6. CloudWatch monitors execution failures and processing metrics.

For workloads that do not need Iceberg, I would still use unique staging locations and a single controlled publisher for each final dataset or partition.

Final Thoughts

Concurrent execution is useful because it helps reduce pipeline execution time and improves resource utilization.

But parallel processing should not mean uncontrolled writes.

When multiple AWS Glue jobs write to the same S3 location, the outcome depends on how those writes are configured and committed.

Overwrite operations can interfere with one another. Append operations can introduce duplicate business records. Partition-level conflicts can produce unexpected results.

The solution is not to disable parallel execution everywhere.

Instead, design your pipeline so that independent workloads can run in parallel while conflicting writes are coordinated.

One principle I follow when designing data pipelines: Parallelize processing, but control publishing.

That small architectural decision can prevent a lot of unexpected production data issues.

Have you ever encountered a situation where two AWS Glue jobs wrote to the same S3 prefix? How did you handle it? I'd be interested in hearing about your approach.