Upload Systems with Heavy and Long Backend Processing (Asynchronous)

Many enterprise systems have similar needs: users upload large files — could be Excel, CSV, or ZIP — which must then be processed with logic that isn’t simple. Row-by-row data validation, format transformation, enrichment with data from other sources, up to final storage in the database. This kind of process isn’t something that finishes in seconds; it can take minutes, sometimes more, depending on file size and logic complexity. When a process that large is forced to run synchronously inside one HTTP request, problems appear from all directions at once. This article discusses why the synchronous approach isn’t a sensible choice for cases like this, and how an asynchronous processing architecture with a status monitoring mechanism becomes the right solution.

Why Synchronous Isn’t an Option

Imagine an upload endpoint that receives a file, directly validates every row, transforms the data, and stores it in the database — all in one request-response cycle. For small files, this approach might feel fine. But once the file size grows or the processing logic gets more complex, three problems appear simultaneously.

The first problem is that the HTTP request will time out. Load balancers, reverse proxies, API gateways, even browsers all have wait time limits far shorter than five minutes. Once that limit is exceeded, the connection is forcibly cut, even though behind it the process might still be running and resources are still being used for something whose result will never reach the user.

The second problem is very poor user experience. Users are forced to stare at a loading page for five minutes without a clear progress indication. There’s no way to know whether the process is running normally, stuck, or already failed midway. If the user’s internet connection drops for a moment, the entire process can be lost without a trace even though the backend is still working.

The third problem is backend resources are locked for a long time. Each request being processed holds one slot from the HTTP server’s worker pool. If several users upload large files simultaneously, the entire pool can fill up just handling these heavy processes — and other requests that should be light and fast are also affected because no workers remain to serve them.

SYNCHRONOUS APPROACH FOR HEAVY PROCESSES:
  ✗ HTTP request times out before the process finishes
  ✗ Users don't know the progress, just stare at loading
  ✗ Backend resources (worker pool, connections) locked for a long time
  ✗ One large file can affect the performance of the entire API

The right solution for this situation is an asynchronous processing architecture, equipped with a status monitoring mechanism allowing users to track progress without having to wait on the same page.

The Big Picture of the Solution

The basic idea of this architecture is separating one large monolithic process into three independent components, each with clear responsibilities.

flowchart LR
    A[Upload & Acceptance] -->|fast| B[Heavy Processing]
    B -->|asynchronous, long| C[Monitoring & Observability]
    C -->|status to user| D[User]

Upload and acceptance is the first stage that must run fast — receiving the user’s file, storing it to storage, recording it as a task waiting to be processed, then immediately sending a response. This stage should finish in seconds, no matter how heavy the process that follows.

Heavy processing is the second stage running outside the HTTP request cycle — this is where all the heavy logic (validation, transformation, enrichment, storage) actually executes. This stage can run long without timeout risk, because there’s no HTTP connection directly waiting for its result.

Monitoring and observability is the third stage giving users visibility into what’s happening with their file, without needing to wait in the same place. Users don’t need to know the technical details behind the scenes; they only need to know that the file was accepted, and the process is running, done, or failed.

The core philosophy of this architecture is simple: separate “receiving the work” from “doing the work”. Once this principle is held consistently, many subsequent design decisions — data structure, worker choice, error handling strategy — become easier to derive naturally.

High-Level Flow

In broad terms, the entire lifecycle of an uploaded file follows these six steps, from the moment the user presses the upload button until the result can be monitored.

sequenceDiagram
    participant User
    participant Backend
    participant Storage
    participant DB
    participant Worker

    User->>Backend: 1. Upload file
    Backend->>Storage: 2. Save file (object storage)
    Backend->>DB: 3. Record task (queue-like table)
    Backend-->>User: 4. Immediate response
    Note over Worker: Runs independently
    Worker->>DB: 5. Process task asynchronously
    Worker->>DB: 6. Update status
    User->>Backend: Check status anytime

The first and fourth steps are the only parts directly involving the user and must run fast. The user uploads a file, the backend stores it in object storage, records it as a new task in a database acting like a queue table, then immediately responds — all of this ideally finishes in seconds. After that, a worker or processor running independently takes that task and processes it asynchronously, updating the status throughout the process, so the user can monitor progress anytime without needing to stay connected to the same page.

Data Model

A single simple table is enough to represent the entire lifecycle of an upload task, from acceptance to finished processing.

The upload_tasks table generally contains an id column as the unique identifier, user_id to know who owns the task, file_path pointing to the file’s location in object storage, status representing the current state (PENDING, PROCESSING, DONE, or FAILED), progress as an optional number from 0 to 100 for cases where granular progress is indeed relevant to display, error_message filled when the status is FAILED, plus created_at and updated_at for audit and processing time analysis purposes.

stateDiagram-v2
    [*] --> PENDING: task created
    PENDING --> PROCESSING: worker takes the task
    PROCESSING --> DONE: process succeeded
    PROCESSING --> FAILED: process failed
    DONE --> [*]
    FAILED --> [*]

This table actually serves three functions at once, even though its structure is single. First, it functions as a process queue — the worker takes rows with PENDING status to process next. Second, it functions as a monitoring data source — the status endpoint just queries this table to tell the user the current condition of their file. Third, it functions as an audit and history — when the file was uploaded, when processing started, when it finished, and if it failed, why, all stored naturally without needing an additional table.

The progress column is optional because not all types of processing have a natural way to calculate a progress percentage. For processes consisting of discrete steps (for example parsing, then validation, then saving), granular progress is relatively easy to calculate. For processes that are one big block by nature, the binary PENDING/PROCESSING/DONE/FAILED status is usually informative enough for users.

UX and User Flow

The user experience in an asynchronous architecture differs fundamentally from the synchronous approach, and this difference must be clearly reflected in the interface designed.

During Upload

Once the user finishes selecting a file and presses the upload button, the system should immediately display a short message like “File uploaded successfully and is being processed” — not a spinner spinning without a clear end. This message implicitly tells users they’re free to leave this page and come back later, because the process isn’t tied to the currently active browser session.

Status Page

This page displays a list of the user’s upload history, complete with each file’s status, upload time, and last update time. Such a page gives users control to check many files at once without opening them one by one, especially useful when users do several uploads within a close time span.

Detail Page

For one specific file, the detail page shows the status more granularly — including progress if available, and most importantly, a clear error message when the process fails. Informative error messages (not just “an error occurred”) help users understand what they need to fix before trying to re-upload, for example an incorrect column format or invalid data in a particular row.

Don’t let the status page only display raw statuses like PROCESSING without additional context. Non-technical users don’t always understand the meaning of these technical terms — consider translating them into more familiar language, for example “Being processed” accompanied by a time estimate if available, so users don’t feel the system is silent without news.

Backend Processing — Two Approaches

After the task is recorded in the database, there are two common approaches to actually execute the heavy processing. Both are valid, with different trade-offs depending on scale and available infrastructure.

Option 1: CLI Worker (Running Forever)

The first approach uses a program running continuously as a long-running process, periodically checking the database for tasks waiting to be processed.

flowchart TD
    A[Worker starts the loop] --> B{Any PENDING task?}
    B -- No --> C[Wait an interval, e.g. 5 seconds]
    C --> A
    B -- Yes --> D[Lock task, change to PROCESSING]
    D --> E[Process the task]
    E --> F{Successful?}
    F -- Yes --> G[Update to DONE]
    F -- No --> H[Update to FAILED]
    G --> A
    H --> A

The main characteristic of this approach is the program runs without stopping, querying the database at certain intervals (for example every 5 seconds), taking tasks with PENDING status, locking that task by changing its status to PROCESSING, then processing and ending it with DONE or FAILED status.

The advantage of this approach lies in its simplicity. Such a program is relatively simple, can be deployed as a container on EC2, VM, or Cloud Run, and is very suitable for on-premise environments or those not yet fully cloud-native. There’s no external dependency besides the database that already exists.

However, there are disadvantages to consider. Polling is inherently inefficient — the worker keeps asking the database even when there are no new tasks, consuming a little resource even when idle. Scaling must also be done manually; if load increases, you need to consciously add worker instances, not automatically. And when there are no tasks at all for a long period, the resources allocated to the worker remain used without producing useful work.

Option 2: AWS SQS + Lambda (Event-Driven)

The second approach flips the model above: instead of an active worker asking, the upstream system actively notifies whenever there’s new work.

sequenceDiagram
    participant Backend
    participant SQS
    participant Lambda
    participant DB

    Backend->>SQS: Push message (1 task = 1 message)
    SQS->>Lambda: Automatic trigger
    Lambda->>DB: Take & process the task
    Lambda->>DB: Update status
    Note over SQS,Lambda: Auto-scales with the message volume

Its main characteristic: when an upload happens, the backend directly pushes a message to SQS, with each task represented as one message. Lambda or another worker is called automatically when a message is available, and this entire mechanism scales automatically following the incoming message volume.

The advantage of this approach lies in its truly event-driven nature — no polling at all, and the delay between task creation and task processing starting becomes much smaller. Scaling happens automatically without manual intervention, and this model is very cost-effective for non-constant workloads, because Lambda essentially only charges when it’s actually run.

The disadvantage: this approach involves more components that must be understood and configured correctly. You need to understand the concepts of retry, dead-letter queue (DLQ), and visibility timeout — all three are crucial to ensure failed tasks don’t just disappear or get processed repeatedly without control.

EventBridge + Lambda as a Hybrid Alternative

Besides the two main approaches above, there’s also a hybrid approach combining polling elements with serverless infrastructure: EventBridge running a cron job at certain intervals (for example every minute), triggering a Lambda that then scans the database for tasks with PENDING status.

flowchart LR
    A[EventBridge cron, every 1 minute] --> B[Lambda]
    B --> C[Scan DB: find PENDING tasks]
    C --> D[Process the found tasks]

This approach is worth considering in several specific conditions: when you don’t want to add SQS complexity to the stack, when the task occurrence rate is relatively low so the one-minute delay from the cron interval isn’t a significant problem, and when the architecture should stay serverless without having to run containers that live continuously.

However, in principle, SQS is still more appropriate for use cases that are genuinely queue-shaped. EventBridge + Lambda essentially still contains a polling element — only the polling is moved from the application level to the scheduled infrastructure level. For cases where low latency and burst traffic scalability truly matter, SQS provides a purer event-driven model.

Error Handling and Reliability

Asynchronous systems, because of their nature of running outside a directly monitorable request cycle, need extra attention to error handling so tasks don’t “disappear” silently without an investigable trace.

Several best practices worth applying consistently include: the PROCESSING status must be atomic, usually achieved with SELECT FOR UPDATE at the database level, to ensure no two workers take the same task simultaneously. Retries should be limited in count, not repeated infinitely, so consistently failing tasks don’t consume resources repeatedly. After retries reach the maximum count, the task should be moved to a dead-letter condition — marked FAILED permanently — rather than continuously retried forever. And finally, error messages must be stored with enough detail for debugging purposes, not just a boolean flag indicating “failed” without any context.

RELIABILITY CHECKLIST:
  □ The PROCESSING status is changed atomically (SELECT FOR UPDATE / SKIP LOCKED)
  □ Retries are limited in count, not repeated infinitely
  □ Tasks failing after N retries are marked FAILED permanently
  □ Error messages are stored completely, not just a success/fail status
  □ There's visibility (dashboard/log) for tasks stuck too long in PROCESSING

Without an atomic locking mechanism when taking tasks, two workers running simultaneously — whether two CLI Worker instances or two overlapping Lambda invocations — can process the exact same task in parallel. This risks producing duplicate data in the database or other unwanted side effects, especially if the executed process isn’t idempotent.

Conclusion

The asynchronous architecture for uploads with heavy processes isn’t just a nice-to-have design choice — for modern systems with significant workloads, this approach is almost mandatory. The three principles discussed in this article complement each other: separate upload from processing so HTTP requests never wait on heavy processes, use consistent status tracking so users and the system have the same visibility into progress, and choose a worker model matching your current scale needs.

For small scale with relatively stable and predictable traffic, a CLI Worker is adequate without needing extra complexity. For larger production scale with fluctuating traffic, SQS combined with Lambda is the more ideal choice because of its event-driven nature and native auto-scaling.

Summary

• Synchronous processing fails for heavy processes because of three problems at once: request timeout, poor user experience, and backend resources locked for a long time.
• Separate three responsibilities: fast upload/acceptance, asynchronous heavy processing, and monitoring/observability giving users visibility.
• High-level flow: upload → save to storage → record task in DB → fast response → worker processes asynchronously → status updated and monitorable.
• One upload_tasks table is enough to function as a process queue, monitoring data source, and audit history at once.
• CLI Worker (polling) suits small scale and simple infrastructure; SQS + Lambda (event-driven) suits production scale with fluctuating traffic.
• EventBridge + Lambda is a hybrid alternative that stays serverless without SQS, but in principle SQS is more appropriate for pure queue use cases.
• Reliability requires atomic locking, limited retries, dead-letter handling, and informative error messages for debugging.
• The worker model choice depends on scale — there’s no single universal answer fitting all infrastructure conditions and traffic volumes.