File Upload Deduplication with SHA: The Most Sensible Solution for Double Uploads

In any system with a file upload feature, a double upload case will almost certainly happen at some point. The cause isn’t always a race condition or a backend bug — often it’s purely human factors and unclear UI. Users forget they’ve already uploaded the same file before, admins open a new tab without realizing it and re-upload the exact same document, the UI gives no clear feedback so users repeat the action because they’re unsure whether their first upload succeeded, or a manual retry happens after a slow internet connection made the process feel stuck. The core problem is simple: the uploaded files are truly identical, only the time of occurrence differs. If left unhandled, the impact can be quite serious — storage bloats for no clear reason, data becomes redundant in many places, storage costs increase over time, and data relationships between entities become ambiguous because it’s unclear which file should be the primary reference. This article discusses the most appropriate and robust solution for this case: SHA-based deduplication, also known as content-based deduplication.

The Main Problem: Logical Duplicates, Not Race Conditions

Before discussing the solution, it’s important to distinguish between two types of problems often mixed together in discussions about data duplication, because they require completely different approaches. A race condition happens when two requests arrive simultaneously within a very narrow time window — this problem can be solved with locking or atomic update mechanisms at the database level, because what’s being fought is a timing problem. A logical duplicate, by contrast, happens when requests arrive far apart in time — could be minutes, hours, or even days — but the uploaded file content is exactly the same.

flowchart TD
    A[File uploaded] --> B{When did it happen?}
    B -- Nearly simultaneous, milliseconds --> C[Race Condition]
    B -- Time-separated, minutes/hours/days --> D[Logical Duplicate]
    C --> E[Solution: locking, atomic update]
    D --> F[Solution: content-based deduplication]

For logical duplicates, the approaches commonly used for race conditions are completely irrelevant. Locking doesn’t help, because there are no two processes competing for a resource at the same time — both uploads have already been processed without any technical conflict. Debouncing at the UI level isn’t enough either, because debounce only effectively prevents double-clicks within seconds, not users who deliberately come back the next day and upload the same file again. Time-window checks — for example rejecting an upload if a file with a similar name exists within the last five minutes — also aren’t reliable, because heuristics like this can easily err in both false positive and false negative directions.

What’s actually needed is a way to answer one fundamental question: is this file physically identical to something uploaded before, regardless of when it happened, who uploaded it, or what filename was used?

What Is a File SHA?

SHA, short for Secure Hash Algorithm, is a cryptographic hash function that produces a unique fingerprint of a file’s content. The fingerprint concept is quite apt as an analogy — just as human fingerprints are unique to each individual, a file’s SHA hash is essentially unique for the specific byte combination making up that file.

What’s actually hashed is only the file’s raw byte content — the raw sequence of bytes making up the file’s contents. Several things deliberately excluded from the hashing process include the filename, extension, upload time, the identity of the uploading user, and the storage path on the server. All these attributes are metadata around the file, not part of the file content itself.

SHA(file) = hash(byte[0] + byte[1] + ... + byte[N])

Conceptually, this hash function takes the entire byte sequence from the start to the end of the file, then produces a single output string of fixed length — for SHA-256 for example, it always produces a 64-character hexadecimal output, whether the original file is 10 KB or 10 GB.

SHA-256 is widely chosen for deduplication because its collision risk — two different inputs producing the same hash — is practically negligible for common application needs. Older algorithms like MD5 or SHA-1 have been proven vulnerable to deliberately engineered collisions, so they should be avoided for cases involving data integrity.

Important Consequences of File SHA

Understanding SHA’s basic nature helps explain why this approach works so well for the double upload problem, and also where its limits are.

Identical Files Produce Identical SHAs

If two files are byte-for-byte exactly the same, in the same order, then the SHA produced from both is guaranteed identical. This property is deterministic — the same hash function, given the same input, will always produce the same output, no matter how many times it’s run or on which machine.

flowchart LR
    A[File A: invoice.pdf] -->|hash| C[SHA: a3f5b8...]
    B[File B: invoice_copy.pdf] -->|hash| C
    C --> D[Exactly the same, even though filenames differ]

A Single Byte Difference Completely Changes the SHA

On the other hand, SHA is extremely sensitive to the smallest changes. Some examples of seemingly trivial changes that still produce a completely different SHA include PDF metadata changing even though the visual document content is identical, different EXIF timestamps on image files even though the displayed photo is the same, line ending differences between LF and CRLF in text files, or files re-saved by certain editor applications that add hidden metadata without changing anything visible to the eye.

Even though these two files look identical when opened, the resulting SHA can be completely different. This is an important trade-off to understand from the start — SHA detects similarity at the raw byte level, not at the level of “meaning” or the file’s visual appearance.

If your use case involves files frequently re-exported from different applications (for example the same PDF but exported from Word on different computers, or the same image but re-saved through an editing app), pure SHA may not catch duplicates that are actually visually identical. Consider the additional strategies discussed at the end of this article for such cases.

Why SHA Is the Right Solution for Double Uploads

SHA addresses the duplication problem exactly at the level it should — at the file’s actual content, not at the surrounding attributes that are easily changed or unintentionally manipulated.

Filename-based approaches fail because two files with exactly the same content can be given completely different names by different users — for example final_report.pdf and final_report_v2_REVISED.pdf that turn out to have identical contents. File-size-based approaches also aren’t enough, because two files with exactly the same size in bytes can have entirely different contents — this possibility is small but real, especially for files with uniform structures.

Content-based deduplication approaches like this aren’t new or experimental — they’re already widely used in large, production-proven systems, such as Git which uses content hashes to detect file changes, Docker image layers deduplicated by content hash, and content-addressable storage (CAS) which uses the hash as the primary identifier for stored data.

Example Dedup Upload Flow

Broadly speaking, implementing SHA-based deduplication follows six relatively simple steps to understand and implement.

sequenceDiagram
    participant User
    participant Backend
    participant DB
    participant Storage

    User->>Backend: Upload file
    Backend->>Backend: Compute SHA-256 from the file stream
    Backend->>DB: Check whether the SHA already exists?
    alt SHA already exists
        DB-->>Backend: Found
        Backend-->>User: Reject / reuse the old file
    else SHA doesn't exist
        DB-->>Backend: Not found
        Backend->>Storage: Save the new file
        Backend->>DB: Save the SHA to the database
        Backend-->>User: Upload successful
    end

Users upload files as usual without needing to know there’s an extra process behind it. The backend then reads the incoming file stream and computes its SHA, usually using SHA-256 as the standard choice. This hash result is then checked against the database to see whether the same SHA has been recorded before. If found, the system has two options: reject the upload entirely, or reuse the previously stored old file without saving a new duplicate. If not found, the new file is saved to storage as usual, and its SHA is recorded in the database as a reference for future checks.

Example Implementation in Golang

Computing SHA-256 from a File in a Streaming Manner

Computing the hash in a streaming fashion, rather than reading the entire file into memory at once, is an important practice especially for large files.

func ComputeSHA256(r io.Reader) (string, error) {
    hash := sha256.New()

    if _, err := io.Copy(hash, r); err != nil {
        return "", err
    }

    sum := hash.Sum(nil)
    return hex.EncodeToString(sum), nil
}

The io.Copy function here works by reading data from the io.Reader in small sequential chunks, then feeding them directly into the hash state being built — without ever holding the entire file content in memory at once. This approach keeps hash computation efficient even when the processed file is hundreds of megabytes or larger.

Example Usage in an Upload Handler

file, _, err := r.FormFile("file")
if err != nil {
    return err
}
defer file.Close()

sha, err := ComputeSHA256(file)
if err != nil {
    return err
}

// ANTI-PATTERN: saving the file directly without checking for duplicates
// saveFile(file)
// -- the same file could be stored repeatedly without detection

// CORRECT: check the SHA against the database before saving
exists := repo.ExistsBySHA(sha)
if exists {
    return errors.New("file already uploaded")
}

// reset the reader to the start position since it was fully read during hashing
file.Seek(0, io.SeekStart)
saveFile(file)
repo.SaveSHA(sha)

An important technical detail to note here is the file.Seek(0, io.SeekStart) call after the hashing process finishes. Because io.Copy reads the stream from start to end to compute the hash, the file’s read pointer position will be at the end after hashing completes. Without resetting this position back to the start, the subsequent file save attempt will produce an empty file, because there’s no data left to read from that position.

For very large files, streaming hashing like the example above is highly recommended over reading the entire file into a byte slice in memory first. The streaming approach keeps memory usage constant, no matter how large the processed file is.

Mandatory Best Practices

Several practices below need to be applied consistently so the deduplication mechanism is truly robust in production, not just working in simple test scenarios.

Hash Computed on the Server

Never trust a SHA sent from the client. If clients are allowed to send their own hash and the server only verifies without recomputing, a malicious client could send a fake hash for various purposes — either to avoid detecting an actual duplicate, or conversely, to make the system mistake a different file for a duplicate. The hash must always be recomputed server-side from the file content actually received.

Use a Unique Index in the Database

CREATE UNIQUE INDEX uniq_file_sha ON uploaded_files (sha256);

This unique index provides two important benefits at once. First, an atomic guarantee — the database will automatically reject any attempt to store a row with the same SHA, without needing extra logic at the application level to enforce it. Second, protection from future race conditions — if someday two upload requests with identical files arrive almost simultaneously, this database-level constraint will prevent duplicates from being stored, complementing the check already done at the application level.

Store the File Size

Storing the file size alongside its SHA is useful for additional validation and debugging purposes. If you ever find two rows with the same SHA but different recorded file sizes, that’s a strong signal that there’s a bug in the hash computation or metadata storage process that needs immediate investigation.

Determine the Dedup Scope

An important question that must be answered early in the design is at what level this deduplication applies. Does dedup apply globally across the entire system, per individual user, or per tenant in a multi-tenant system? The answer depends heavily on your application’s business context.

-- Global dedup: one SHA may only exist as one row in the entire table
CREATE UNIQUE INDEX uniq_file_sha ON uploaded_files (sha256);

-- Per-tenant dedup: the same SHA may exist in different tenants,
-- but must not be duplicated within the same tenant
CREATE UNIQUE INDEX uniq_tenant_sha ON uploaded_files (tenant_id, sha256);

Multi-tenant systems, for example, almost always need per-tenant dedup scope, not global — because two different tenants are very likely to upload files with identical contents independently with no relationship whatsoever, and both should still be allowed to be stored as separate entities.

Determine the Behavior When a Duplicate Is Detected

Some common choices that can be applied when the system detects a duplicate file include rejecting the upload directly with a clear error message, returning a reference to the existing old file without saving a new file, attaching the old file to the new entity being created without the user needing to know the physical file is actually the same as an existing one, or simply logging the event without blocking the upload process at all — an approach sometimes called soft dedup.

BEHAVIOR OPTIONS WHEN A DUPLICATE IS DETECTED:
  ✓ Reject upload         -- firm, suitable for unique documents per transaction
  ✓ Return the old reference -- saves storage, the old file stays in use
  ✓ Attach to the new entity -- transparent to the user, still saves storage
  ✓ Log only (soft dedup) -- observability without blocking the user

This decision depends entirely on the specific business needs you’re facing — there’s no universally correct answer for all use cases.

When SHA Isn’t Enough

There are situations where files are “logically the same” but their binaries differ, and pure SHA won’t catch this similarity. Examples include images with exactly the same visual content but different EXIF metadata, or PDF documents with identical contents but different internal metadata — like the modification date or the application used to generate them.

flowchart TD
    A[Original file] --> B[Re-saved via an editor application]
    B --> C[Metadata changed, visual content the same]
    C --> D{SHA the same?}
    D -- No --> E[Pure SHA doesn't detect this duplicate]

For cases like this, several advanced solutions can be considered, though their level is far more complex than the pure SHA approach: stripping metadata before hashing so only the core content is computed, canonicalizing the file format so its byte representation is consistent before hashing, or using perceptual hashes like pHash specifically for images, designed to detect visual similarity even when the raw bytes differ.

Advanced solutions like perceptual hashing and canonicalization are advanced-level techniques that aren’t always necessary. Before implementing them, make sure your use case genuinely requires them — for the majority of common file upload systems, pure SHA-based deduplication is more than enough to solve the double upload problem that actually occurs in the field.

Conclusion

A file SHA is essentially a fingerprint of the file content itself, regardless of any accompanying metadata. SHA-based deduplication is the most appropriate solution for double upload problems that are non-race in nature — when two identical files are uploaded at far-apart times, not because of timing conflicts. This approach is proven robust against differences in time, user, or the device used for uploading, and no less important, it’s relatively easy to implement and scalable for large data volumes.

If you’ve ever experienced re-uploading with the exact same file, then SHA-based deduplication isn’t just the technically correct solution, but also a mature, widely proven solution across various large-scale production systems.

Summary

• A logical duplicate differs from a race condition — the problem isn’t timing, but the same file content uploaded at far-apart times.
• SHA is a fingerprint of the file content, computed from raw byte content, unaffected by filenames, upload time, or who uploaded it.
• Identical files always produce identical SHAs, but a single byte difference — including hidden metadata — can completely change the SHA.
• SHA is more reliable than filename, file size, time windows, or UI locks, because it works directly on the file’s actual content.
• The dedup flow: compute the SHA on upload, check the database, reject or reuse if it exists, save if it doesn’t.
• The hash must be computed on the server, not trusted from the client, to prevent manipulation.
• A unique index on the SHA column provides an atomic guarantee and additional protection from future race conditions.
• Dedup scope (global, per user, or per tenant) must be determined up front, because it affects the unique constraint structure in the database.
• Pure SHA has limitations for files that are “visually the same” but have different metadata — advanced solutions like perceptual hashing are only needed for special cases.