Data Deduplication
Also known as:
Dedup
Deduplication
Single-Instance Storage
Definition
Data deduplication stores only one physical copy of identical data, no matter how many times it appears. Systems split data into chunks, fingerprint each chunk with a content hash, and before writing a chunk they check whether that hash already exists. If it does, they simply reference the existing block instead of storing it again. This cuts both storage cost and, when done on the client, upload bandwidth.
Popular reads
View All
Payment System Design: Ledger, Idempotency, and Settlement
Jul 18, 2026
The Complete HTMX Guide: From Zero to Production
Dec 22, 2025
Cursor Skills: How to Create and Use Agent Skills
Jun 23, 2026
Transactional Outbox Pattern: Never Lose an Event Again
Apr 07, 2026
Flash Sale System Design: Architecture, Scale, and Oversell
May 16, 2026
X Algorithm Explained: How the Open Source Recommendation System Works
Jan 22, 2026
Key Takeaways
- Deduplication stores duplicate data once and references it many times, saving storage and bandwidth.
- It falls out naturally from content-addressable storage, where chunks are keyed by content hash.
- Chunk-level dedup beats file-level dedup because files that share parts still share blocks.
- Global (cross-user) dedup saves the most but raises privacy and side-channel concerns, so many systems scope it per account.
How It Works
- Split incoming data into chunks, either fixed-size or content-defined.
- Hash each chunk to get a fingerprint that identifies it by content.
- Look up the fingerprint; if the chunk already exists, store only a reference to it.
- Otherwise store the chunk once and record its fingerprint for future lookups.
Where It Is Used
- Dropbox and similar sync services skip uploading chunks they already hold.
- Backup systems like restic and Borg dedup chunks across every snapshot.
- Storage arrays and filesystems such as ZFS offer inline block-level deduplication.