# singhajit.com - Full Content Index > Technical blog by Ajit Singh, Staff Software Engineer at Agoda. > Last updated: 2026-09-09 ## About This file provides AI systems with structured access to blog content for accurate citation and reference. All content is original and written by Ajit Singh. --- ## Topic Hubs Canonical landing pages for each topic area. The definition and FAQ blocks below are quotable answers tied to each hub URL. ### Database Engineering - URL: https://singhajit.com/database/ - Meta Title: Database Engineering Patterns: Indexing, Sharding & Internals Guide - Also Known As: Database Design Patterns; Database Engineering Patterns; Database Internals; Database Architecture - Description: Database engineering patterns and internals: PostgreSQL, MongoDB, Redis, DynamoDB, B-tree and LSM indexing, sharding, locking, isolation levels, query optimization and SQL vs NoSQL tradeoffs. Definition: Database engineering is the discipline of choosing, designing and tuning the storage layer of an application: picking SQL or NoSQL, designing schemas, building the right indexes (B-tree, hash, GIN, LSM), tuning isolation levels and locks, sharding for scale, and reasoning about replication, consistency and failover. The same patterns appear in PostgreSQL, MySQL, MongoDB, Redis, Cassandra and DynamoDB. Key Terms: - B-tree Index: A self-balancing tree structure that keeps keys sorted on disk pages, providing O(log n) lookups and supporting both equality and range queries; the default index in most relational databases. - LSM Tree: Log-Structured Merge tree, a write-optimized index that buffers writes in memory and periodically flushes sorted runs to disk; used by Cassandra, RocksDB, and LevelDB. - ACID: Atomicity, Consistency, Isolation, Durability: the four guarantees that classical relational transactions provide. - MVCC: Multi-Version Concurrency Control: a technique where writers create new versions of rows so readers never block writers; used by PostgreSQL, Oracle, and MySQL InnoDB. - Sharding: Horizontal partitioning of a dataset across multiple database nodes by a shard key (hash, range, or directory based). - Replication: Copying data from a primary node to one or more replicas for read scaling and failover; can be synchronous, semi-synchronous, or asynchronous. - Isolation Level: A configurable guarantee about what concurrent transactions can observe; ranges from Read Uncommitted to Serializable. - Write-Ahead Log: An append-only log that records every change before it is applied to data files, used for crash recovery and replication. - Connection Pool: A cache of pre-established database connections that application threads borrow and return, avoiding the cost of opening a new TCP and TLS handshake per query. - OLTP vs OLAP: OLTP is transactional workload (many small reads and writes, low latency); OLAP is analytical workload (few large scans and aggregations over historical data). FAQ: - Q: What is the difference between SQL and NoSQL databases? A: SQL databases (PostgreSQL, MySQL) store rows in fixed schemas, support multi-row ACID transactions, and use SQL with joins. NoSQL is an umbrella for document stores (MongoDB), key-value stores (Redis, DynamoDB), wide-column stores (Cassandra), and graph databases (Neo4j). NoSQL trades joins and strict schemas for horizontal scalability, flexible documents, or specialized access patterns. - Q: How do database indexes work? A: An index is a separate data structure that maps column values to row locations so the database can find rows without scanning the whole table. B-tree indexes (the default) keep keys sorted and support equality and range queries in O(log n). Hash indexes support only equality lookups but in O(1). LSM trees (used by Cassandra, RocksDB) optimize writes by buffering in memory and flushing sorted files to disk. - Q: What are database isolation levels? A: Isolation levels define what concurrent transactions can see of each other. Read Uncommitted allows dirty reads; Read Committed blocks dirty reads; Repeatable Read blocks non-repeatable reads; Serializable blocks phantom reads and behaves as if transactions ran one at a time. PostgreSQL defaults to Read Committed, MySQL InnoDB defaults to Repeatable Read, and most distributed databases offer Snapshot Isolation. - Q: When should I shard a database? A: Shard when a single primary node cannot handle the write throughput, when the dataset is too large to fit on one disk, or when you need data residency in specific regions. Before sharding, exhaust simpler options: read replicas, vertical scaling, partitioning within a single node, archiving cold data, and a cache. Sharding adds operational complexity around resharding, cross-shard queries, and distributed transactions. - Q: PostgreSQL vs MongoDB: which should I use? A: Use PostgreSQL when you need joins, multi-row ACID transactions, a strict schema, rich SQL, and proven OLTP performance. Use MongoDB when your data is naturally hierarchical (documents), the schema evolves often, you need horizontal sharding out of the box, or your access pattern is dominated by single-document reads and writes. Modern PostgreSQL also supports JSONB, narrowing the gap considerably. - Q: What is a write-ahead log in a database? A: A write-ahead log (WAL) is an append-only log that records every change before it is applied to the data files. On crash, the database replays the WAL to recover committed transactions and roll back incomplete ones. PostgreSQL, MySQL InnoDB (redo log), and SQLite all use WAL, and replication is typically built by streaming the WAL to followers. ### Design Patterns - URL: https://singhajit.com/design-patterns/ - Meta Title: Design Patterns Explained: Gang of Four Patterns with Java Examples - Also Known As: Gang of Four Design Patterns; GoF Patterns; Object-Oriented Design Patterns; Software Design Patterns - Description: Design patterns explained with Java examples: all 23 Gang of Four creational, structural and behavioral patterns including Singleton, Factory, Strategy, Observer, Decorator, Builder, Adapter and Proxy. Definition: Design patterns are reusable, named solutions to recurring object-oriented design problems. The Gang of Four catalog (Gamma, Helm, Johnson, Vlissides, 1994) groups 23 patterns into three families: Creational (how objects are made: Singleton, Factory, Builder, Prototype, Abstract Factory), Structural (how objects compose: Adapter, Bridge, Composite, Decorator, Facade, Flyweight, Proxy) and Behavioral (how objects communicate: Strategy, Observer, Command, State, Template Method, Iterator, Mediator, Memento, Visitor, Chain of Responsibility, Interpreter). Key Terms: - Singleton: A creational pattern that restricts a class to a single instance and provides a global access point; used for shared configuration, logging, and connection pools. - Factory Method: A creational pattern that defines an interface for creating an object but lets subclasses decide which concrete class to instantiate. - Builder: A creational pattern that constructs a complex object step by step, useful when an object has many optional parameters. - Adapter: A structural pattern that converts the interface of a class into another interface clients expect, letting incompatible classes work together. - Decorator: A structural pattern that attaches new behavior to an object dynamically by wrapping it, providing a flexible alternative to subclassing. - Proxy: A structural pattern that provides a placeholder for another object to control access, add caching, or enable lazy loading. - Strategy: A behavioral pattern that defines a family of interchangeable algorithms and lets the client pick one at runtime. - Observer: A behavioral pattern that defines a one-to-many dependency so when one object changes state, all dependents are notified. - Command: A behavioral pattern that encapsulates a request as an object, allowing it to be queued, logged, undone, or sent across processes. - Template Method: A behavioral pattern that defines the skeleton of an algorithm in a base class and lets subclasses override specific steps. FAQ: - Q: What are design patterns? A: Design patterns are time-tested solutions to common software design problems, expressed as named templates rather than concrete code. They give developers a shared vocabulary (Singleton, Factory, Strategy) and capture the trade-offs of the solution. The most influential catalog is the 23 Gang of Four patterns from the 1994 book Design Patterns: Elements of Reusable Object-Oriented Software. - Q: What are the three categories of Gang of Four design patterns? A: Creational patterns deal with object creation: Singleton, Factory Method, Abstract Factory, Builder, Prototype. Structural patterns deal with how classes and objects are composed: Adapter, Bridge, Composite, Decorator, Facade, Flyweight, Proxy. Behavioral patterns deal with object collaboration and responsibilities: Strategy, Observer, Command, State, Template Method, Iterator, Mediator, Memento, Visitor, Chain of Responsibility, Interpreter. - Q: What is the Singleton pattern? A: Singleton ensures a class has only one instance and provides a global access point to it. It is used for shared resources like configuration, logging, thread pools, and database connection pools. In Java the safest implementations are an enum-based Singleton or a static inner holder class, both of which guarantee thread safety and lazy initialization without explicit synchronization. - Q: What is the difference between Strategy and State pattern? A: Both encapsulate behavior in separate classes, but the intent differs. Strategy lets a client pick an interchangeable algorithm (sort order, payment method) at runtime; the strategies usually do not know about each other. State lets an object change its behavior when its internal state changes; the state objects often know about each other and trigger transitions. - Q: Are design patterns still relevant? A: Yes. Functional and reactive programming have absorbed some patterns (Iterator, Observer, Command) into language features, but the core Gang of Four catalog still describes problems that recur in every codebase: object creation, decoupling, varying behavior. Modern frameworks (Spring, Guice, React, Android) are built almost entirely from these patterns. - Q: Where can I see design patterns in real code? A: The JDK uses Iterator (Collections), Observer (java.util.Observable, listeners), Decorator (BufferedReader wrapping InputStreamReader), Factory (Calendar.getInstance), Singleton (Runtime.getRuntime), and Adapter (java.io.InputStreamReader). Spring is built on Factory, Proxy, Template Method, and Strategy. Android uses Builder (AlertDialog.Builder), Observer (LiveData), and Adapter (RecyclerView.Adapter). ### Distributed Systems - URL: https://singhajit.com/distributed-systems/ - Meta Title: Distributed Systems Patterns: Consensus, Replication & Fault Tolerance Guide - Also Known As: Patterns of Distributed Systems; Distributed System Design Patterns; Distributed Computing Patterns; Distributed Systems Concepts - Description: Distributed systems patterns explained: consensus (Paxos, Raft), replication, gossip, write-ahead log, quorum, heartbeat, two-phase commit. Real production examples from Kafka, Cassandra, DynamoDB. Definition: Distributed systems patterns are reusable solutions to recurring problems that arise when independent computers cooperate over an unreliable network. They cover consensus (Paxos, Raft), replication (write-ahead log, leader-follower), failure detection (heartbeat, gossip), coordination (quorum, two-phase commit), and fault tolerance (circuit breaker, idempotent receiver) used inside Kafka, Cassandra, DynamoDB and ZooKeeper. Key Terms: - Consensus: A protocol that lets a group of nodes agree on a single value (such as the next entry in a replicated log) even when some nodes are slow, crashed, or unreachable. - Paxos: A family of consensus protocols introduced by Leslie Lamport that uses prepare/promise and accept/accepted phases to safely choose a value across a cluster. - Raft: An understandable consensus algorithm built around a strong leader, randomized election timeouts, and append-only log replication; used by etcd, Consul, and CockroachDB. - Write-Ahead Log: An append-only log written before any state mutation, allowing crash recovery and replication by replaying the log in order. - Quorum: The minimum number of nodes that must acknowledge an operation for it to be considered durable; majority quorum (N/2 + 1) guarantees any two quorums overlap. - Heartbeat: A periodic message a node sends to advertise it is alive; missed heartbeats trigger failure detection and leader re-election. - Gossip Protocol: An epidemic-style dissemination algorithm where each node periodically exchanges state with random peers, achieving eventually consistent membership and metadata. - Two-Phase Commit: A blocking atomic commit protocol with a prepare phase (vote) and commit phase (decision) coordinated by a transaction manager across participating resources. - Leader Election: The process by which a cluster picks one node as the coordinator for writes or partition ownership, typically via Raft, Paxos, or ZooKeeper ephemeral nodes. - CAP Theorem: A result by Eric Brewer stating that during a network partition a distributed data store can provide either consistency or availability, but not both. FAQ: - Q: What are distributed systems patterns? A: Distributed systems patterns are battle-tested solutions to problems that show up whenever multiple computers coordinate over a network: agreeing on a value (consensus), keeping data in sync (replication), detecting failed nodes (failure detection), ordering events (clocks), and recovering from partial failure. Examples include Paxos, Raft, write-ahead log, gossip, heartbeat, quorum, two-phase commit, and leader election. - Q: What is the difference between Paxos and Raft? A: Paxos and Raft are both consensus algorithms that let a cluster of nodes agree on the same value despite failures. Paxos was published first (1998) and is mathematically minimal but notoriously hard to implement. Raft was designed in 2014 explicitly for understandability, splitting consensus into leader election, log replication, and safety. etcd, CockroachDB, and Consul use Raft; Google Chubby and Spanner use Paxos variants. - Q: Which distributed systems patterns power Kafka and Cassandra? A: Kafka uses the write-ahead log (segmented log), leader-follower replication, in-sync replica quorum, controller election, and idempotent producer patterns. Cassandra uses gossip dissemination for membership, hinted handoff and read-repair for replication, tunable quorum reads/writes, Merkle trees for anti-entropy, and the Bloom filter pattern for SSTable lookups. - Q: What is the CAP theorem? A: The CAP theorem states that a distributed data store can guarantee at most two of three properties during a network partition: Consistency (every read sees the latest write), Availability (every request receives a response), and Partition tolerance (the system keeps working when messages are dropped). In practice partitions happen, so systems choose CP (MongoDB, HBase) or AP (Cassandra, DynamoDB). - Q: What is a quorum in distributed systems? A: A quorum is the minimum number of nodes that must agree before an operation is considered successful. The most common rule is majority quorum (N/2 + 1), which guarantees that any two quorums overlap by at least one node. This overlap is what lets leader election and replicated logs stay consistent even if some nodes fail or get partitioned. - Q: Where can I learn distributed systems patterns? A: Start with Martin Fowler and Unmesh Joshi's Patterns of Distributed Systems catalog and book, Designing Data-Intensive Applications by Martin Kleppmann, the Raft and Paxos papers, and the Kafka and Cassandra documentation. The articles in this hub break down each pattern with diagrams and code examples drawn from real production systems. ### System Design - URL: https://singhajit.com/system-design/ - Meta Title: System Design Patterns & Case Studies: Scalability, High Availability Guide - Also Known As: System Design Patterns; Scalability Patterns; Software Architecture Patterns; System Design Interview Prep - Description: System design patterns and real-world case studies: how Uber, Stripe, WhatsApp, Cloudflare, Meta and Shopify scale. Load balancing, sharding, caching, messaging, high availability. Definition: System design is the discipline of breaking a large software product into components, choosing the right data stores, communication patterns, caching layers and failure modes so the system stays fast, available and consistent at scale. Common patterns include load balancing, sharding, replication, write-through and write-back caching, message queues, CDC, rate limiting, and circuit breakers. Key Terms: - Load Balancer: A network component that distributes incoming requests across multiple backend servers using algorithms like round-robin, least-connections, or consistent hashing. - Sharding: Partitioning a dataset across multiple database nodes by a shard key so each node stores and serves only a subset of the data. - Replication: Maintaining copies of data on multiple nodes to improve read throughput and survive node failures; can be synchronous or asynchronous. - Cache: An in-memory data store (Redis, Memcached) placed in front of a slower data source to reduce latency and offload reads. - CDN: A Content Delivery Network of geographically distributed edge servers that cache static assets close to end users. - Message Queue: A broker (Kafka, RabbitMQ, SQS) that buffers messages between producers and consumers for asynchronous processing. - Rate Limiter: A component that caps how many requests a client can make within a time window using token bucket, leaky bucket, or sliding window algorithms. - Circuit Breaker: A fault-tolerance pattern that stops sending requests to a failing downstream service after an error threshold is exceeded, then probes for recovery. - CAP Theorem: A constraint stating that during a network partition a distributed data store can guarantee either consistency or availability, but not both. - Microservices: An architectural style that decomposes an application into small, independently deployable services that communicate over the network. FAQ: - Q: What are the most common system design patterns? A: The patterns that show up in almost every system design interview and production architecture are: load balancing, horizontal sharding, leader-follower replication, read-through and write-through caching, message queues for async work, CDN for static assets, rate limiting, circuit breaker for downstream protection, idempotency keys, and event sourcing or CDC for cross-service data sync. - Q: How do I prepare for a system design interview? A: Start with fundamentals: load balancers, databases (SQL vs NoSQL), caches, queues, and CDNs. Then study 8-10 well-known systems like URL shortener, Twitter feed, ride-sharing dispatch, video streaming, payment processing, and chat. Always clarify scope, estimate scale (QPS, storage, bandwidth), draw the high-level diagram, deep-dive one component, and end with bottlenecks, failure modes, and trade-offs. - Q: What is the difference between horizontal and vertical scaling? A: Vertical scaling adds more CPU, RAM, or disk to a single machine. It is simple but bounded by hardware limits and creates a single point of failure. Horizontal scaling adds more machines and distributes load across them with sharding or replication. It scales further and survives node loss but introduces consensus, replication, and partitioning complexity. - Q: When should I use a message queue? A: Use a message queue (Kafka, RabbitMQ, SQS) when producers and consumers need to be decoupled, when work can be processed asynchronously, when you need to absorb traffic spikes with a buffer, when you need at-least-once or exactly-once delivery, or when you need to fan out the same event to multiple downstream services. - Q: How does sharding work? A: Sharding splits a large dataset across multiple database nodes by a shard key. Hash sharding spreads load evenly but makes range queries expensive. Range sharding keeps related rows together but can create hotspots. Consistent hashing minimizes data movement when you add or remove nodes. Resharding strategies include double writes, backfill, and traffic cutover. - Q: What system design patterns does Uber use? A: Uber uses geospatial indexing (H3 hexagons) for nearby driver search, Cell architecture for fault isolation, Kafka for trip events, Cassandra for high-write workloads, MySQL for relational data, Redis for hot caches, schemaless storage on top of MySQL for flexibility, and Ringpop for sharding and routing within services. --- ## System Design Articles ### How Google manages billions of lines of code in one monorepo - URL: https://singhajit.com/how-google-manages-its-monorepo/ - Date: 2026-09-04 - Tags: system-design, git, devops, software-engineering - Description: How Google manages billions of lines of code in one monorepo using Piper, CitC, Bazel, trunk-based development, automated testing, and large refactors. Key Takeaways: - A monorepo is a source layout, not a monolith. Google ships many independent services from one tree. - A traditional full Git clone is impractical at google3's scale. Google built Piper for storage and CitC for cloud workspaces, so developers do not download the full tree. - Trunk-based development and one shared version of each internal library prevent diamond dependency conflicts. Upgrades become whole-tree jobs, which Rosie and automated tests make practical. - Bazel (Blaze inside Google) is an essential enabler: it provides hermetic, incremental, cached builds from a declared dependency graph. - The culture is as important as the database. OWNERS files, Critique, Tricorder, TAP, and a dedicated large-scale change process help maintain code health. - Copy the ideas, not the hardware. Most teams want Git plus Bazel, Nx, or Turborepo, not a global Spanner-backed VCS. FAQ: - Q: Does Google really keep all of its code in one repository? A: Almost, not all. Google's 2016 CACM paper reported that 95 percent of its software developers used the main Piper-backed tree commonly called google3. Android and Chrome used Git outside that repository because those products needed to work with external partners and open source contributors. The paper reported that Android alone was split across more than 800 Git repositories. - Q: Why did Google not just use Git for the monorepo? A: A traditional full Git clone copies the repository and its history to the developer's machine. That model is impractical when the tree has billions of files and tens of millions of commits. Google evaluated commercial and open source systems after running a large Perforce installation for more than a decade, but none supported the required scale as one repository. Piper stores the tree as a distributed service, while CitC presents the tree without syncing it in full. The 2016 paper said a Git migration would have required splitting google3 into thousands of repositories. - Q: What is Piper and what is CitC? A: Piper is the version control system: the source of truth, replicated across Google data centers, originally on Bigtable and later on Spanner, with Paxos keeping replicas consistent. CitC (Clients in the Cloud) is a FUSE filesystem backed by cloud storage. A workspace looks like the entire repository, but only edited files are stored in its overlay. The 2016 paper reported that more than 80 percent of Piper users used CitC and that an average workspace contained fewer than ten modified files. - Q: What is the difference between Blaze and Bazel? A: Blaze is Google's internal build system. Bazel is its open source counterpart, released in 2015. Both support hermetic, incremental, parallel builds from BUILD files and a precise dependency graph. Caching and remote execution avoid rebuilding unaffected targets and distribute the work that remains. - Q: How does Google avoid merge hell in a monorepo this large? A: Engineers work at head and branching is uncommon. Changes are committed in one serial order on the trunk, so there is no long-lived development branch drifting away from mainline. Presubmit checks and post-submit tests detect breakage, while automated tooling can identify and roll back a bad change. Rosie splits large refactors into independently tested and reviewed changes instead of leaving one unmergeable patch. - Q: Should my company use a Google-style monorepo? A: Use a monorepo if your projects share libraries, you want atomic API changes, and you will invest in a real build graph (Bazel, Nx, Turborepo, or equivalent). Do not copy Piper, CitC, or TAP unless you have Google's traffic and headcount. A polyrepo is still the better default when teams release on unrelated cadences, need hard access boundaries, or cannot staff build infrastructure. The layout should follow coupling, not fashion. - Q: How does Google test every commit if there are tens of thousands of commits per day? A: It does not run the entire test corpus on every commit in isolation. Presubmit checks test a pending change before submission. After submission, TAP (Test Automation Platform) batches commits into milestones and selects tests through the dependency graph. A 2017 paper reported about 800,000 builds and 150 million test runs per day. At that scale, flaky tests consume shared compute and delay unrelated changes. - Q: Is a monorepo the same as a monolith? A: No. A monorepo describes where source code lives. A monolith describes how software is built and deployed. One repository can contain many independently built and deployed services, while a single monolithic application can be spread across several repositories. You can use either model, both, or neither. ### Fixed Partitions Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/fixed-partitions/ - Date: 2026-08-31 - Tags: distributed-systems, system-design, database, software-engineering - Description: Learn the Fixed Partitions pattern in distributed systems: why hash(key) % nodeCount reshuffles almost all your data when a node joins, how a fixed number of logical partitions fixes it, and how Kafka, Redis Cluster, and Cassandra use it. Key Takeaways: - Mapping keys directly to nodes with hash(key) % nodeCount is simple but brutal: change the node count and nearly every key moves to a different node. - Fixed Partitions adds a stable middle layer. Keys map to a fixed number of logical partitions; partitions map to nodes. Only the second mapping changes on a resize. - Because the partition count never changes, the key-to-partition math is stable, so rebalancing moves whole partitions, not individual keys. - Pick the partition count once, up front, and make it much larger than your expected node count. Changing it later is painful because it rehashes data. - The partition-to-node assignment lives in a consistent core like ZooKeeper or etcd, so every client and node agrees on who owns what. - Consistent hashing solves the same resize problem a different way. Fixed Partitions uses a fixed count of explicit partitions; consistent hashing uses a ring and virtual nodes. - Kafka, Redis Cluster, Cassandra, Akka Cluster Sharding, Hazelcast, and Ignite all lean on fixed partitions to scale horizontally without reshuffling everything. FAQ: - Q: What is the Fixed Partitions pattern in distributed systems? A: Fixed Partitions is a data partitioning pattern that keeps the number of partitions constant for the life of the cluster. Keys are mapped to a fixed set of logical partitions using a hash, and those partitions are separately assigned to physical nodes. Because the partition count never changes, the key-to-partition mapping stays stable even as nodes are added or removed, so only whole partitions move during a resize instead of rehashing every key. It is documented in Unmesh Joshi's Patterns of Distributed Systems on Martin Fowler's site. - Q: Why is hash(key) % nodeCount a bad way to distribute data? A: Because the node count is part of the formula. The moment you add or remove a node, nodeCount changes, so the modulo result changes for almost every key, and nearly all data has to move to a different node. In a three-node cluster growing to five, the majority of keys get remapped. For large datasets this means a huge, slow, expensive data shuffle every time the cluster resizes, which defeats the point of scaling out smoothly. - Q: What is the difference between Fixed Partitions and consistent hashing? A: Both solve the same problem: keep data movement small when the cluster changes size. Fixed Partitions uses an explicit, fixed number of logical partitions (for example 1024) and a separate table that assigns partitions to nodes. Consistent hashing places nodes and keys on a hash ring and gives each node many virtual node positions to balance load. Fixed Partitions makes the partition count and ownership explicit and easy to reason about; consistent hashing avoids maintaining a large assignment table. Many systems blend the two ideas. - Q: How many partitions should I choose? A: Choose a number well above your expected maximum node count, so each node can hold several partitions and future nodes have partitions to receive, but not so high that per-partition overhead (metadata, open files, connections) becomes a burden. Common real-world choices are Redis Cluster's fixed 16384 hash slots and Kafka topics created with tens to hundreds of partitions. The key constraint is that changing the count later is expensive, so you size it once for future growth. - Q: Can you change the number of partitions later? A: It is possible but painful, which is the whole point of picking well up front. Changing the partition count changes the hash(key) % partitionCount result for most keys, so it triggers the same mass reshuffle that Fixed Partitions was meant to avoid. Kafka lets you increase a topic's partition count but warns that it breaks key-to-partition ordering guarantees. Redis Cluster keeps its 16384 slots fixed forever. Treat the partition count as a long-term decision, not a tuning knob. - Q: Where is the partition-to-node mapping stored? A: In a small, strongly consistent coordination service, often called a consistent core, such as ZooKeeper, etcd, or a Raft-based controller. Every client and node reads the same assignment table from there, so they all agree on which node currently owns each partition. When a rebalance moves a partition, the mapping is updated in one place and the change propagates to everyone, avoiding split views where two nodes think they own the same partition. - Q: What is a hot partition and how does Fixed Partitions relate to it? A: A hot partition is one that receives far more traffic than the others, usually because the partitioning key is skewed (for example, everything keyed by the same tenant or the same day). Fixed Partitions spreads keys across many partitions, which helps, but it cannot fix a bad key. If one key or a small key range dominates, its partition stays hot no matter how many partitions or nodes you have. Choosing a high-cardinality, evenly distributed partition key matters just as much as the pattern itself. ### Dropbox System Design: How Cloud File Storage Works - URL: https://singhajit.com/dropbox-system-design/ - Date: 2026-08-20 - Tags: system-design, distributed-systems, storage, software-engineering - Description: A plain-language Dropbox system design walkthrough. Learn how file chunking, deduplication, delta sync, a metadata service, and object storage combine to build a scalable cloud file storage and file sync service. Key Takeaways: - Split files into chunks and store the raw blocks separately from the metadata. The metadata service is the brain; block storage is the muscle. - Content-addressed chunks (named by their hash) give you free deduplication and integrity checks in one move. - Delta sync uploads only changed chunks, not the whole file, which is the single biggest bandwidth saver for large edited files. - Sync is a pull model triggered by a lightweight notification, not a push of file bytes. Keep the notification channel cheap and the transfer channel separate. - Most of the hard problems are in metadata and conflict handling, not in storing bytes. Object storage already solves durable byte storage for you. - Design for eventual consistency across devices, and make client sync idempotent so a retried upload never corrupts a file. FAQ: - Q: What is the core idea behind Dropbox system design? A: The core idea is to separate file content from file metadata. Files are broken into fixed-size chunks, each chunk is stored as an immutable block in object storage, and a separate metadata service records which chunks and versions make up each file. Clients sync by comparing metadata and transferring only the chunks they are missing. This split lets the storage layer and the metadata layer scale independently. - Q: How does Dropbox save storage and bandwidth? A: Through deduplication and delta sync. Deduplication means a chunk that already exists in storage is never uploaded or stored twice, because each chunk is identified by the hash of its contents. Delta sync means when you edit a large file, only the chunks that actually changed are uploaded, not the entire file. Together they dramatically cut both storage cost and network usage. - Q: How does file sync work across multiple devices? A: Sync uses a notification plus pull model. When a client uploads a change, the metadata service records the new version and a notification service tells other online devices that something changed. Each device then asks the metadata service what changed, compares it to what it has locally, and downloads only the missing chunks from block storage. Offline devices catch up the next time they connect. - Q: How are conflicts handled when two devices edit the same file? A: Most consumer file sync services avoid silent data loss by keeping both versions. When the server detects that two clients edited the same file version concurrently, it accepts one as the next version and stores the other as a conflicted copy, often named with the device or user and a timestamp. This is simpler and safer than trying to automatically merge arbitrary binary files. - Q: Why not just store whole files instead of chunks? A: Chunking unlocks three things whole-file storage cannot. First, deduplication at the chunk level so common data is stored once. Second, delta sync so an edit to a 1GB file transfers only a few kilobytes. Third, parallel and resumable uploads, since chunks upload independently and a failed chunk can be retried on its own. Whole-file storage is simpler but wastes storage and bandwidth. - Q: What databases and storage does a Dropbox-like system use? A: Raw file blocks go into object storage such as Amazon S3 or an in-house equivalent, because it is cheap, durable, and effectively infinite. Metadata (users, files, versions, chunk lists, sharing) goes into a scalable database, often a sharded relational database or a distributed key-value store, because it needs fast lookups, transactions, and relationships. The two layers are deliberately different tools for different jobs. - Q: Is Dropbox system design a common interview question? A: Yes. Design Dropbox or Google Drive is a classic system design interview question because it touches storage, metadata modeling, sync, deduplication, conflict resolution, and scale all at once. Interviewers like it because a strong answer shows you can separate concerns (blocks vs metadata), reason about bandwidth and consistency, and make practical trade-offs. ### Columnar Databases Explained: ClickHouse, BigQuery, and Redshift - URL: https://singhajit.com/columnar-databases-explained/ - Date: 2026-08-05 - Tags: database, analytics, system-design, data-engineering - Description: A clear guide to columnar databases like ClickHouse, BigQuery, and Redshift. Learn how column-oriented storage, compression, and vectorized execution power fast OLAP analytics. Key Takeaways: - A columnar database stores values column by column, so a query reads only the columns it touches instead of every byte of every row. - Columnar is not the same as a wide column store. Columnar is about physical layout for analytics (ClickHouse). Wide column is a NoSQL data model for scale (Cassandra). - Three ideas make columnar fast: column pruning (read fewer columns), heavy compression (5 to 30x on similar values), and vectorized execution (process values in batches with SIMD). - Data skipping uses per-block metadata like min/max and bloom filters to avoid reading blocks that cannot match, so a query often touches a tiny fraction of the data. - Columnar wins for OLAP: scans, aggregates, and filters over millions of rows and a few columns. Row stores win for OLTP: point lookups and frequent updates. - Writes favor big batches, not single rows. Updates and deletes are expensive because data is stored as immutable, sorted, compressed blocks. - The common production shape is hybrid: PostgreSQL for transactions, a columnar store for analytics, kept in sync with change data capture. FAQ: - Q: What is a columnar database? A: A columnar database, or column-oriented database, is an analytical (OLAP) database that stores each column of a table contiguously on disk instead of storing whole rows together. Because a typical analytical query reads only a few columns from a wide table, this layout cuts I/O sharply and lets the engine compress each column heavily. ClickHouse, DuckDB, Google BigQuery, Amazon Redshift, and Snowflake are well known columnar databases. - Q: What is the difference between a columnar database and a wide column store? A: They sound alike but are different concepts. A columnar database (ClickHouse, BigQuery, Redshift) is about physical storage layout: values of one column are stored together to make analytical scans and aggregations fast. A wide column store (Cassandra, Bigtable) is a NoSQL data model where each row can hold a different, flexible set of columns, built for high write throughput and key-based reads. Columnar is for OLAP analytics. Wide column is for OLTP-style access at scale. - Q: Why are columnar databases so fast for analytics? A: Three things compound. Column pruning means the query reads only the columns it references, not the whole row, so it moves far fewer bytes. Compression is very effective because a column holds values of one type, often with long runs of similar data, giving 5 to 30x savings. Vectorized execution processes values in dense batches using CPU SIMD instructions instead of one row at a time. On top of that, data skipping uses block metadata to avoid reading data that cannot match the filter. - Q: What is the difference between OLTP and OLAP? A: OLTP (online transaction processing) is the day-to-day workload of an application: many small, fast transactions that read or update a handful of rows by key, such as placing an order. OLAP (online analytical processing) is the analytics workload: a few large queries that scan and aggregate millions or billions of rows, such as revenue by region over a year. Row stores like PostgreSQL are built for OLTP. Columnar stores like ClickHouse are built for OLAP. - Q: Is ClickHouse a columnar database? A: Yes. ClickHouse is an open-source column-oriented database built for real-time OLAP. It stores each column in its own compressed files, uses a sparse primary index and data-skipping indexes to prune data, and runs vectorized query execution. It is one of the fastest analytical databases available and is widely used for observability, product analytics, and customer-facing dashboards. - Q: Can I use a columnar database for transactions? A: Generally no. Columnar databases are poor at OLTP work. Single-row inserts, frequent updates, and point lookups are slow because data is stored as large, sorted, compressed column blocks that are expensive to modify in place. For transactional workloads you should use a row store like PostgreSQL or MySQL. Many teams run both: a row store for transactions and a columnar store for analytics, synced with change data capture. - Q: What are columnar file formats like Parquet and ORC? A: Apache Parquet and Apache ORC are open, columnar, on-disk file formats. They store data column by column with per-column statistics and compression, so query engines can prune columns and skip row groups. Apache Arrow is the in-memory columnar standard used to move data between systems without repeated serialization. These formats are the backbone of data lakes, Spark jobs, and tools like DuckDB, and they let many engines read the same files. - Q: When should I choose a columnar database over PostgreSQL? A: Choose a columnar database when your queries scan and aggregate large numbers of rows across a few columns of a wide table, when reporting or dashboards on PostgreSQL have become slow, or when you need real-time analytics on high-volume event data. Keep PostgreSQL for transactions and point reads. If dashboard queries take tens of seconds on Postgres and you are aggregating millions of rows, that is the signal to add a columnar store alongside it. ### Wide Column Stores Explained: Cassandra, Bigtable, and ScyllaDB - URL: https://singhajit.com/wide-column-stores-explained/ - Date: 2026-07-29 - Tags: database, nosql, system-design, distributed-systems - Description: A clear guide to wide column stores like Cassandra, Bigtable, and ScyllaDB. Learn the data model, partition key vs clustering key, LSM writes, tunable consistency, and when to use one. Key Takeaways: - A wide column store is not the same as a columnar (OLAP) database. Wide column means rows with flexible columns grouped into partitions; columnar means values of one column stored together for analytics. - The partition key decides which node stores the data, and the clustering key decides the sort order inside that partition. Getting these two right is the whole game. - Writes are fast because they go to an in-memory memtable plus an append-only commit log, then flush to immutable SSTables. Random writes become sequential writes. - Reads can be slower and more variable because a key may live in the memtable plus several SSTables, so bloom filters and compaction do a lot of work behind the scenes. - You design tables around queries, not entities. Denormalize and keep one table per query. Joins do not exist. - Consistency is tunable per request. You trade latency and availability against how many replicas must agree, guided by the CAP theorem. - Wide partitions, piled-up tombstones, and ALLOW FILTERING are the three mistakes that quietly wreck a Cassandra cluster. FAQ: - Q: What is a wide column store? A: A wide column store is a NoSQL database that organizes data into tables of rows, where each row is a partition key plus a sorted set of columns, and different rows can hold different columns. Rows are distributed across a cluster by hashing the partition key and sorted within a partition by a clustering key. This design gives very high write throughput and linear horizontal scaling. Apache Cassandra, ScyllaDB, Apache HBase, and Google Cloud Bigtable are the best known examples. - Q: What is the difference between a wide column store and a columnar database? A: They sound similar but solve different problems. A wide column store (Cassandra, Bigtable) is a row-oriented NoSQL store where each row can have a flexible set of columns, built for high-volume transactional writes and key-based reads. A columnar or column-oriented database (ClickHouse, BigQuery, Amazon Redshift) physically stores all values of one column together to make analytical aggregations fast. Wide column is for OLTP-style access at scale. Columnar is for OLAP analytics. - Q: What is the difference between a partition key and a clustering key in Cassandra? A: The partition key determines which node in the cluster stores a row. Cassandra hashes the partition key and maps the hash onto a ring of nodes, so all rows with the same partition key live together on the same node and its replicas. The clustering key determines the sort order of rows inside a single partition on disk, which is what makes range scans within a partition cheap. Together they form the primary key. - Q: Is Cassandra a wide column store or a columnar database? A: Cassandra is a wide column store, not a columnar analytics database. It is row-oriented under the hood, stores rows grouped by partition, and is optimized for high write throughput and predictable key-based reads. People sometimes call it column-oriented because its data model exposes flexible columns per row, but that is different from a true columnar store like ClickHouse that lays out each column separately on disk for analytics. - Q: When should I use a wide column database? A: Use a wide column database when you have a write-heavy workload at large scale, need linear horizontal scaling across many nodes and regions, can tolerate eventual consistency, and know your query patterns at design time. It fits time-series data, event and audit logs, IoT sensor streams, messaging history, and activity feeds. Avoid it when you need ad hoc queries, joins, aggregations across partitions, or strong multi-row transactions. In those cases a relational database is usually the better choice. - Q: Why are writes so fast in Cassandra and ScyllaDB? A: Because they use an LSM tree. A write is appended to a commit log for durability and added to an in-memory memtable, then acknowledged. The database never seeks around the disk to update a row in place. When the memtable fills, it is flushed to disk as an immutable sorted file called an SSTable. Turning random writes into sequential appends is what gives these systems their very high write throughput. - Q: What is a tombstone in Cassandra? A: A tombstone is a marker that says a row or column was deleted. Because SSTables are immutable, Cassandra cannot erase data in place, so a delete writes a tombstone instead. The real data is removed later during compaction. If you delete or overwrite a lot of data, tombstones pile up and reads have to scan past them, which slows queries. Managing tombstones with TTLs and good data modeling is an important part of running Cassandra. - Q: What is tunable consistency in a wide column store? A: Tunable consistency means you choose, per query, how many replicas must respond before a read or write is considered successful. Common levels are ONE, QUORUM, and ALL. A higher consistency level gives stronger guarantees but costs more latency and is less available if nodes are down. Cassandra and ScyllaDB let you set this on each request, so you can pick strong consistency for critical writes and cheaper eventual consistency for the rest. - Q: Cassandra vs Bigtable: what is the difference? A: Both are wide column stores with the same rough data shape, but the architecture differs. Cassandra and ScyllaDB use a masterless peer-to-peer ring where every node is equal and there is no single coordinator, which makes them highly available and easy to run across regions. Bigtable uses a leader-based design where a master assigns tablets (key ranges) to tablet servers, and it is offered as a fully managed Google Cloud service. Cassandra gives you control and multi-cloud portability; Bigtable gives you zero operational burden on Google Cloud. ### Request Waiting List Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/request-waiting-list/ - Date: 2026-07-24 - Tags: distributed-systems, system-design, networking, software-engineering - Description: Learn the Request Waiting List pattern in distributed systems: why a node cannot answer a client until other nodes respond, how it parks the request against a key and callback, and how Raft, Kafka, and Cassandra use it to wait for a quorum. Key Takeaways: - A node often cannot answer a client alone. It must replicate to other nodes and wait for enough of them to acknowledge before the request is safe to confirm. - The waiting list is a map from a key to a callback. The key matches the responses that will arrive, and the callback decides when the request is complete. - The key is usually a correlation ID for direct messages, or the high watermark log index when waiting for a replicated log entry to commit. - Responses arrive asynchronously and out of order, so the node counts them per entry and fires the callback only when a quorum is reached. - Every entry needs a timeout. Without one, a lost response leaks memory and leaves a client hanging forever. - It pairs naturally with Request Pipeline and Request Batch: those keep the connection full, the waiting list untangles which response answers which request. FAQ: - Q: What is the Request Waiting List pattern in distributed systems? A: The Request Waiting List pattern is a technique where a cluster node stores a client request that it cannot answer immediately because the answer depends on responses from other nodes. The node maintains a map from a key to a callback function. The key is chosen to match the responses that will arrive, such as a correlation ID for a point-to-point message or the high watermark log index when waiting for a replicated log entry to commit. As responses come back asynchronously, the node looks up the matching entry and the callback decides whether the client request can now be fulfilled, usually once a majority quorum of acknowledgements has arrived. - Q: Why can't a node just answer the client right away? A: Because a single node's copy of the data is not safe on its own. If the node confirms a write to the client and then crashes before any other node has the data, the write is lost even though the client was told it succeeded. To avoid that, the node replicates the change to other nodes and waits until enough of them, typically a majority quorum, have stored it durably. Only then is the write safe to confirm. The client request has to be parked somewhere during that wait, and the waiting list is where it lives. - Q: What key does the waiting list use? A: The key is whatever will let the node match an incoming response back to the pending request. For direct request-reply messaging between nodes, it is usually a correlation ID: a unique number stamped on the request and echoed on the response. For a replicated log, the natural key is the log index of the entry the request created; the request completes when the high watermark, the commit index, moves past that index. The rule is simple: pick the key that the arriving responses can be grouped by. - Q: How is the Request Waiting List different from the Request Pipeline pattern? A: They solve different halves of the same problem and are usually used together. The Request Pipeline pattern is about not waiting: a node sends many requests on a connection without blocking for each reply, so the network stays full. The Request Waiting List pattern is about tracking: once responses start coming back out of order, the node needs a place to remember which pending request each response belongs to and what condition completes it. Pipelining keeps the pipe busy; the waiting list makes sense of the replies that come back. - Q: What happens if a response never arrives? A: That is why every entry in the waiting list needs a timeout. A background sweep periodically checks for entries that have been waiting too long and expires them, failing the associated client request with a timeout error and removing the entry. Without expiry, a lost or delayed response would leave the entry in the map forever, leaking memory and leaving the client blocked. In quorum systems the request can still succeed as long as enough other nodes respond in time; the timeout only fires when not enough acknowledgements arrive. - Q: How does Raft use a request waiting list? A: When a Raft leader receives a client command, it appends the command to its log and replicates it to followers, but it cannot answer the client until the entry is committed, meaning a majority of nodes have stored it. The leader keeps the client request pending, keyed by the log index of the new entry. As followers acknowledge, the leader advances its commit index (the high watermark). When the commit index reaches or passes the entry's index, the leader applies the command to its state machine and completes the waiting client request with the result. - Q: Is the waiting list the same as a callback or a future? A: They are closely related. The value stored against each key in the waiting list is effectively a callback: code that runs when the response criteria are met. In many languages this is implemented as a future or promise that the caller awaits, and the receiver completes when the quorum is reached. Whether you call it a callback, a future, or a completion handler, the idea is the same: the waiting list holds deferred work that runs later, driven by responses arriving from other nodes. ### DDoS Attacks: How They Work and How to Protect Your App - URL: https://singhajit.com/ddos-attack-and-protection/ - Date: 2026-07-22 - Tags: security, networking, system-design, devops - Description: What is a DDoS attack and how do you stop one? A developer's guide to how distributed denial-of-service attacks work, the main types, and DDoS protection that actually holds up. Key Takeaways: - A DoS attack comes from one source; a DDoS attack comes from many at once, which is what makes it hard to block by IP. - There are three broad types: volumetric (fill the pipe), protocol (exhaust connection state), and application-layer (exhaust the app itself). - Layer 7 attacks are the scariest because a few thousand realistic requests can be as damaging as terabits of raw traffic, and they look like real users. - You cannot 'firewall away' a large DDoS on your own box. You need a network bigger than the attack, which is why CDNs and scrubbing services exist. - Defense is layered: absorb with a CDN and anycast, filter with a WAF and rate limiting, and stay up with caching, autoscaling, and graceful degradation. - Attacks are getting bigger fast. Cloudflare mitigated a record 31.4 Tbps attack in late 2025, up more than 700% from a year earlier, so plan for scale you cannot handle alone. FAQ: - Q: What is a DDoS attack in simple terms? A: A DDoS (distributed denial-of-service) attack is when an attacker uses many computers at the same time to flood a website or server with so much traffic or so many requests that it slows down or crashes, blocking real users. The 'distributed' part means the traffic comes from thousands of different machines, usually a botnet of hijacked devices, so you cannot stop it by blocking a single address. - Q: What is the difference between a DoS and a DDoS attack? A: A DoS (denial-of-service) attack comes from a single source, so you can often block it by filtering one IP address. A DDoS (distributed denial-of-service) attack comes from many sources at once, often thousands or millions of devices in a botnet. That distribution is the whole point: it multiplies the attacker's firepower and makes simple IP blocking useless, because there is no single address to ban. - Q: What are the main types of DDoS attacks? A: There are three broad categories. Volumetric attacks (like UDP floods and DNS amplification) try to saturate your bandwidth with sheer volume, measured in bits per second. Protocol attacks (like SYN floods) exhaust connection state on servers and firewalls, measured in packets per second. Application-layer or Layer 7 attacks (like HTTP floods) send realistic-looking requests to expensive endpoints, measured in requests per second, and are the hardest to detect because they mimic real users. - Q: How do you stop or prevent a DDoS attack? A: You cannot stop a large DDoS from a single server, because the attack is bigger than your pipe. The practical answer is layered: route traffic through a CDN or DDoS scrubbing service with a network far larger than any attack, use anycast to spread load across many data centers, apply rate limiting and a web application firewall to drop bad requests, and cache aggressively so the origin sees very little traffic. Autoscaling and graceful degradation keep the service usable while the attack is filtered. - Q: Can a firewall stop a DDoS attack? A: A traditional firewall helps against small attacks and specific protocols, but it cannot stop a large volumetric DDoS on its own. If the attack traffic is bigger than your internet connection, the firewall itself becomes the bottleneck and goes down with everything behind it. Large attacks must be absorbed and filtered upstream, in a network with far more capacity than the attack, which is what CDNs and scrubbing centers provide. - Q: What is a botnet and how does it relate to DDoS? A: A botnet is a network of internet-connected devices infected with malware and controlled remotely by an attacker, often without the owners knowing. Each infected device, called a bot or zombie, follows commands from a central server. Botnets are the engine behind most large DDoS attacks: one attacker can direct the combined traffic of thousands or millions of hijacked devices, such as home routers and IoT cameras, at a single target. - Q: How big can DDoS attacks get? A: Very big, and growing fast. In late 2025 Cloudflare mitigated a record 31.4 terabits per second attack that lasted just 35 seconds, powered by the Aisuru-Kimwolf botnet of infected devices. Attack sizes grew more than 700% over the previous year, and hyper-volumetric attacks (over 1 Tbps or a billion packets per second) are now routine. No single server or on-premise firewall can absorb that, which is why cloud-scale protection is essential. - Q: Is DDoS protection worth paying for? A: For any business that loses money or trust when it goes offline, yes. Free tiers from providers like Cloudflare cover common attacks, but managed DDoS protection services and plans like AWS Shield Advanced add always-on monitoring, higher capacity, and cost protection against traffic bills during an attack. The cost of a good DDoS protection service is almost always lower than the cost of extended downtime, lost sales, and reputational damage. ### Payment System Design: Ledger, Idempotency, and Settlement - URL: https://singhajit.com/payment-system-design/ - Date: 2026-07-18 - Tags: system-design, distributed-systems - Description: Learn payment system design end to end: idempotency keys, a double-entry ledger, payment state machines, PSP integration, webhooks, the saga pattern, and reconciliation. A practical guide for the system design interview and production. Key Takeaways: - Idempotency is the foundation. Every write carries an idempotency key, stored before you call the processor, backed by a database unique constraint so a retry replays the first result instead of charging again. - Model money as a double-entry ledger. Every movement is a debit and an equal credit, the table is append-only, and balances are summed rather than mutated, which kills a whole class of race conditions. - Use a payment state machine. Define the states (created, authorized, captured, settled, refunded, failed) and reject any transition that is not allowed, so a charge can never be captured twice. - Never do the slow work on the request. Take the request, return quickly, and let workers talk to the processor, send receipts, and run fraud checks in the background. - Coordinate multi-step flows with a saga, not a distributed lock. Reserve, charge, and fulfill are local steps with compensating actions, published reliably through a transactional outbox. - Webhooks are the source of truth for outcomes. The processor knows whether the charge settled or the card was declined hours later, so verify the signature and process every webhook idempotently. - Reconciliation is not optional. A daily job that compares your ledger against the processor's settlement report is how you catch the money bugs your tests missed. - Wrap every processor call in a circuit breaker and decide fail-open versus fail-closed on purpose. For money, fail-closed is usually right; a declined payment beats a double charge. FAQ: - Q: How do you design a payment system? A: Start with correctness, not scale. Accept each payment request with a client-supplied idempotency key and store it before doing any work, so retries are safe. Drive the payment through an explicit state machine (created, authorized, captured, settled, refunded, failed) and reject invalid transitions. Record every money movement in an append-only double-entry ledger where debits equal credits. Talk to the payment service provider through an adapter with timeouts, retries, and a circuit breaker. Coordinate multi-service flows with a saga and publish events reliably using a transactional outbox. Learn the final outcome from signed webhooks, and run a daily reconciliation job that compares your ledger against the processor's settlement report. - Q: How do payment systems prevent double charging? A: With idempotency keys plus a database unique constraint. The client generates a unique key (usually a UUID) before the request and sends it in an Idempotency-Key header. The server records the key before calling the payment processor. If the same key arrives again, the server returns the stored response instead of charging the card a second time. Redis is used as a fast first check, but the durable unique constraint in the database is the real guarantee, because the cache can be evicted or bypassed. This is the same pattern Stripe uses. - Q: What is a double-entry ledger and why do payment systems use it? A: A double-entry ledger records every money movement as two matching entries, a debit on one account and a credit on another, for the same amount. The books are correct only when total debits equal total credits, which gives you a cheap invariant to check on every transaction. The ledger is append-only, so you never edit or delete a balance; a correction is a new pair of reversing entries. This gives payment systems a complete audit trail, removes update races on balance rows, and makes bugs and fraud easy to detect because the books simply will not balance. - Q: What is the difference between authorization, capture, and settlement in payments? A: Authorization checks that the card is valid and the funds are available, and places a hold, but no money moves yet. Capture tells the processor to actually move the held funds, which usually happens when the order ships or the service is delivered. Settlement is the back-office process, run in batches by the card networks and banks, where the money actually lands in the merchant's account, typically a day or two later. Splitting authorize and capture lets you hold funds now and charge later, and it is why your payment state machine needs distinct authorized and captured states. - Q: How do you handle failures in a distributed payment flow? A: Use the saga pattern instead of a distributed transaction. Break the flow into local steps (reserve inventory, charge the card, create the order, notify the user) where each step has a compensating action that undoes it. If charging succeeds but fulfillment fails, the saga runs the compensations in reverse, for example issuing a refund. Publish the events that drive the saga through a transactional outbox so an event is emitted if and only if the local transaction committed. Every step and every compensation must be idempotent because they will be retried. - Q: Why are webhooks important in payment systems? A: Because the final outcome of a payment often arrives asynchronously. A card authorization can succeed instantly but the settlement, a dispute, or a delayed bank decline can come hours or days later. The processor tells you about these through webhooks. Your webhook handler must verify the signature to confirm the event really came from the processor, process each event idempotently because processors retry and can send duplicates, and update the payment state machine and ledger accordingly. Relying only on the synchronous API response means you miss half the story. - Q: How do you scale a payment system? A: Scale for correctness first, then throughput. Shard the ledger and payment tables by a stable key like merchant id or account id so each shard owns a slice of the traffic and cross-shard transactions are rare. Keep the hot path short by doing only the idempotency check, the ledger write, and an enqueue synchronously, and push everything else (processor calls, receipts, fraud scoring) to background workers reading from a queue like Kafka. Cache read-heavy data such as merchant configuration. The database is usually the bottleneck, so use strong consistency where money is written and eventual consistency for reporting and analytics. - Q: What are the most common payment system design interview questions? A: The canonical prompt is to design a payment system like Stripe or PayPal that processes charges and refunds without ever double charging. Expect follow-ups on how idempotency keys work end to end, why a double-entry ledger is used instead of a balance column, how you model the payment state machine, the difference between authorization and capture, how you coordinate across services with a saga and a transactional outbox, how you handle processor timeouts and retries, how webhooks close the loop, how reconciliation catches discrepancies, and where you choose strong versus eventual consistency. Strong answers lead with the correctness requirements and name the trade-offs explicitly. ### Designing Database Isolation for B2B Multi-Tenant SaaS - URL: https://singhajit.com/multi-tenant-database-isolation/ - Date: 2026-07-14 - Tags: database, postgres, saas, system-design, security - Description: Learn how to design multi-tenant database isolation for B2B SaaS. Compare shared schema, schema-per-tenant, and database-per-tenant, and use PostgreSQL RLS safely. Key Takeaways: - Isolation is a full-stack problem: resolve the tenant at the API edge, pass it through every query, cache key, job, and backup path, and treat the database as the last line of defense. - Shared schema with `tenant_id` plus PostgreSQL RLS is the right default for most B2B SaaS. It is cheap, easy to migrate, and scales to thousands of tenants. - Always set tenant context with `SET LOCAL` inside a transaction so pooled connections cannot leak one tenant into the next request. - Lead every tenant-scoped index with `tenant_id`. Without that, RLS and `WHERE tenant_id = ...` filters turn into expensive scans. - Schema-per-tenant looks clean early, then migration and connection-pool pain grow with every new customer. - Database-per-tenant gives the strongest isolation and simplest per-tenant restore, at the cost of ops automation and cloud database spend. - Design a graduation path from day one: moving one enterprise tenant to a dedicated database should be a tested runbook, not a rewrite. FAQ: - Q: What is multi-tenant database isolation? A: Multi-tenant database isolation is the set of design choices that keep one customer's data invisible and unreachable to every other customer in a shared SaaS product. It covers how you store rows, how queries are filtered, how connections carry tenant context, and how backups, restores, and deletes stay scoped to one tenant. - Q: What are the three main multi-tenant database models? A: The three common models are shared schema (one database, one set of tables, tenant_id on every row), schema-per-tenant (one database, a separate Postgres schema per customer), and database-per-tenant (a dedicated database or cluster per customer). Shared schema is usually the default. Schema-per-tenant is a middle ground with higher ops cost. Database-per-tenant is for regulated or very large enterprise tenants. - Q: Is PostgreSQL Row Level Security enough for tenant isolation? A: No. RLS is a strong safety net, not a replacement for application filters. Your app should still scope every query by tenant. RLS stops a forgotten WHERE clause from leaking data. You still need tenant-aware indexes, SET LOCAL for session context, non-superuser app roles, FORCE ROW LEVEL SECURITY on owner roles when needed, and tests that try to cross tenants on purpose. - Q: When should a B2B SaaS use database-per-tenant? A: Choose database-per-tenant when a customer needs physical isolation for compliance (for example HIPAA with a BAA and dedicated audit trails), when they require independent backup and restore SLAs, when one tenant's load would hurt others on a shared cluster, or when the contract explicitly demands a dedicated database. Automate provisioning first. Manual per-tenant databases do not scale. - Q: How do I stop noisy neighbors in a shared multi-tenant database? A: Use composite indexes led by tenant_id, statement timeouts, per-tenant rate limits at the API, connection pool tiers by plan, and monitoring that attributes CPU, I/O, and lock waits to tenant_id. For the heaviest tenants, move them to a dedicated database so their load cannot starve the shared pool. - Q: Should I use SET or SET LOCAL for app.current_tenant? A: Use SET LOCAL inside a transaction. SET LOCAL resets when the transaction ends, which is what you want with PgBouncer or any pooled connection. A plain SET can stick on the connection and leak tenant context into the next request that reuses that connection. - Q: Can I start with shared schema and move enterprise tenants later? A: Yes, and that hybrid path is what most successful B2B SaaS products end up with. Keep tenant_id in the data model even if some tenants later get their own database. Abstract connection routing behind a tenant registry so graduating one customer is an extract-and-route change, not a rewrite of every query. - Q: Does schema-per-tenant help with compliance? A: It gives stronger logical separation than row-level tenancy and can make per-tenant dumps simpler, but it is still one shared Postgres instance for compute and storage. Many auditors still ask for controls, encryption, and access logs. If the requirement is physical isolation, you usually need database-per-tenant, not just another schema. - Q: How should I identify the tenant on each request? A: Resolve the tenant from something the user cannot forge, such as a signed JWT claim or an authenticated server-side session, and reject any request without a valid tenant. Subdomains and paths are convenient routing hints, but the authoritative tenant id should come from verified auth, never from a raw header or query parameter the client controls. Bind the resolved tenant to a per-request context so every query and background job inherits it automatically. - Q: How do I scale a shared-schema multi-tenant database? A: First tune indexes led by tenant_id, add read replicas for reporting, and move analytics off the primary. When a single node can no longer hold your write or storage load, shard by tenant_id using a distributed Postgres like Citus, which co-locates each tenant's rows on one node so joins and transactions stay local. Sharding by tenant keeps the shared-schema model intact while letting it scale horizontally. ### Request Pipeline Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/request-pipeline/ - Date: 2026-07-07 - Tags: distributed-systems, system-design, networking, software-engineering - Description: Learn the Request Pipeline pattern in distributed systems: why waiting for each response wastes network capacity, how sending multiple requests on one connection without blocking cuts latency, and how HTTP/2, Redis, Kafka, and PostgreSQL use pipelining. Key Takeaways: - One request at a time on a connection wastes almost everything. Most of the time is spent waiting for a round trip, not doing work, so both the network and the server sit idle. - Request pipelining sends many requests without blocking on replies. The connection stays full, so total latency drops from N round trips to roughly one. - The clean design uses two threads: one that writes requests to the socket and one that reads responses. They never block each other. - You need flow control. Cap the number of in-flight requests (a bounded window) or a fast client will overrun a slow server and run it out of memory. - Responses can come back out of order, so each request carries a correlation ID and the client matches replies to a waiting-list of pending requests. - Pipelining exposes head-of-line blocking: one slow request stalls everything behind it on the same connection. HTTP/2 streams and HTTP/3 fix this at different layers. - Ordering plus retries is the sharp edge. Allowing many in-flight writes can reorder them on a retry, which is why Kafka pairs a high in-flight count with idempotence. FAQ: - Q: What is the Request Pipeline pattern in distributed systems? A: The Request Pipeline pattern is a technique where a node sends multiple requests over a single network connection without waiting for the response to each previous request. A separate thread reads responses as they come back and matches them to the pending requests using a correlation ID. This keeps the connection and the receiving server's request queue full, which improves both latency and throughput compared to a strict send-and-wait loop. It is documented in Unmesh Joshi's Patterns of Distributed Systems on Martin Fowler's site and usually works together with the Single Socket Channel and Singular Update Queue patterns. - Q: Why is sending one request at a time slow? A: Because most of the elapsed time is spent waiting, not working. When a client sends a request and blocks until the reply arrives, it pays a full network round trip (RTT) for every single request. If the RTT is 1 millisecond and you send 1000 requests one after another, you spend about 1 second just waiting on the wire, even if the server processes each request in microseconds. The network link is idle in one direction while you wait, and the server's queue is empty when it could be working. Pipelining removes that per-request wait. - Q: What is the difference between pipelining and batching? A: Batching combines many logical operations into one request and one response, so the server sees a single large message. Pipelining keeps the operations as separate requests but sends them back to back on the same connection without waiting for each reply. Batching reduces per-message overhead and is great when you can group work, but it needs all the items up front. Pipelining works even when requests are generated one at a time and preserves independent responses. Many systems combine both: pipeline batches of requests. - Q: What is head-of-line blocking in pipelining? A: Head-of-line blocking happens when responses on a pipelined connection must come back in the same order the requests were sent. If the first request is slow, every response behind it is stuck waiting, even if those requests finished quickly. This is exactly why HTTP/1.1 pipelining failed in practice and stayed disabled in browsers. HTTP/2 fixed the application layer by multiplexing independent streams on one connection, and HTTP/3 (over QUIC) fixed the remaining transport-layer version by removing the single ordered TCP byte stream. - Q: How does Kafka use request pipelining? A: The Kafka producer setting max.in.flight.requests.per.connection controls pipelining. It is the number of unacknowledged produce requests the producer will send to a broker before it must wait for acknowledgements. The default of 5 lets several batches be in flight at once, which raises throughput. The catch is ordering on retry: if an earlier batch fails and is retried while a later one succeeded, records can be reordered. Enabling idempotence (enable.idempotence=true, the default in modern Kafka) makes the broker preserve order and deduplicate even with up to 5 in-flight requests. - Q: What is Redis pipelining? A: Redis pipelining is the Request Pipeline pattern applied to the Redis protocol. Instead of sending one command, waiting for its reply, then sending the next, the client writes many commands to the socket at once and then reads all the replies together. Because Redis processes commands in order on a single connection, this is safe and dramatically faster over a network: it turns N round trips into roughly one. It is different from MULTI/EXEC transactions, which group commands atomically; pipelining is purely about not waiting for each reply. - Q: How do you stop a pipelined client from overwhelming the server? A: With flow control, usually a bounded in-flight window. The client tracks how many requests it has sent that are not yet acknowledged and stops sending once that count hits a limit, resuming as responses come back. This is back pressure: it lets a fast sender adapt to a slower receiver so the receiver's queue and memory stay bounded. TCP already does this at the byte level with its receive window, and application protocols like HTTP/2 add their own flow-control windows on top per stream and per connection. ### CDN System Design: How Content Delivery Networks Work - URL: https://singhajit.com/cdn-system-design/ - Date: 2026-06-30 - Tags: system-design, distributed-systems, caching - Description: A clear, developer-focused guide to CDN system design. Learn how content delivery networks work, how edge caching, anycast routing, and origin shield cut latency, and how to design one in a system design interview. Key Takeaways: - A CDN's whole job is to reduce distance. Serving a file from an edge server 20 km away instead of an origin 8,000 km away is the single biggest latency win you can buy. - The core flow is request routing then caching. Anycast or DNS picks the nearest Point of Presence, then a cache hit or miss decides whether the edge serves locally or fetches from origin. - Tiered caching and an origin shield protect your origin. Edge misses funnel through a regional cache and a single shield node, so the origin sees a tiny fraction of total traffic. - Cache hit ratio is the metric that matters. Good cache keys, sensible TTLs, and request collapsing push it toward 95 percent or higher, which is what makes a CDN cheap and fast. - Pull CDNs fetch content on first request and suit most websites. Push CDNs upload content ahead of time and suit large, infrequently changed files like software downloads and video. - Modern CDNs are more than caches. They terminate TLS, run edge compute, block DDoS and bot traffic with a WAF, and optimize images, all at the edge. FAQ: - Q: What is a CDN and how does it work? A: A CDN (content delivery network) is a network of servers spread across the world that store cached copies of your content close to users. When someone requests a page, image, or video, the CDN routes them to the nearest edge server instead of your origin server. If that edge server already has the content cached, it serves it immediately. If not, it fetches the content from your origin once, caches it, and serves every future request from the edge. This cuts latency, reduces load on your origin, and improves availability. - Q: What is the difference between an edge server and an origin server? A: The origin server is the single source of truth where your real content lives, usually your own web servers or an object store like Amazon S3. Edge servers are the CDN's distributed cache nodes placed in data centers around the world, also called Points of Presence (PoPs). Edge servers hold temporary cached copies of content from the origin and serve users from nearby, while the origin only gets hit on a cache miss or for content that cannot be cached. - Q: What is the difference between a push CDN and a pull CDN? A: A pull CDN fetches content from your origin lazily, the first time a user requests it, then caches it for later requests. You just point the CDN at your origin and it pulls on demand, which suits most websites. A push CDN requires you to upload content to the CDN ahead of time, so it is already in place before the first request. Push works better for large files that change rarely, such as software installers, game assets, and video, where you want full control over what is stored and when. - Q: What is cache hit ratio and why does it matter? A: Cache hit ratio is the percentage of requests served from the CDN cache without contacting the origin. A 95 percent hit ratio means 95 of every 100 requests are answered at the edge. It matters because every cache hit is faster (no origin round trip) and cheaper (no origin bandwidth or compute). A low hit ratio usually points to bad cache keys, short TTLs, or too many content variants, and it means your origin is doing work the CDN was supposed to absorb. - Q: How does a CDN handle cache invalidation? A: CDNs use a mix of time-based expiry and explicit purging. With TTL (time to live), each cached object expires after a set period and is revalidated against the origin. For instant updates, you issue a purge (also called invalidation) that removes an object from the cache by URL, by cache tag, or for the whole site. A common pattern is to set long TTLs and change the file name or add a version query string when content changes, so users always get the newest version without waiting for expiry. - Q: Does a CDN replace web hosting? A: No. A CDN sits in front of your hosting and caches content, but it does not host your application or store your source of truth. You still need an origin server or object store where the real content lives. The CDN improves performance and resilience for that origin, but if the origin has no copy of the content, the CDN has nothing to cache and serve. Think of the CDN as a global cache and shield layer, not a replacement for hosting. - Q: What is anycast routing in a CDN? A: Anycast is a network addressing method where many edge servers share the same IP address, and the internet's routing protocol (BGP) naturally sends each user to the topologically nearest one. This lets a CDN advertise a single IP for a site while serving it from hundreds of locations. Anycast is also what makes CDNs resilient to DDoS attacks, because attack traffic gets spread across many Points of Presence instead of hitting one server. - Q: Which CDN provider should I use? A: It depends on your workload. Cloudflare is the common default thanks to a generous free tier, built-in security, and edge compute. AWS CloudFront is the natural fit if you already run on AWS with S3 and Lambda. Fastly suits teams that need instant cache purging and programmable edge logic. Akamai is the enterprise standard for the largest media and finance workloads. Bunny.net is popular for cost-sensitive, static-heavy, or video delivery. Match the provider to your traffic shape, budget, and how much edge control you need. ### Idempotent Receiver Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/idempotent-receiver/ - Date: 2026-06-04 - Tags: distributed-systems, system-design, microservices, software-engineering - Description: Learn how the idempotent receiver pattern safely handles duplicate requests in distributed systems using a client ID, request number, and a saved response. Key Takeaways: - Duplicate requests are not a bug, they are a guarantee. The moment a client retries on timeout, your server will eventually see the same request twice. - An idempotent receiver identifies each client with a unique ID and each request with a request number, then stores the response so retries return the saved result without redoing the work. - Idempotency is about the effect, not the message. Receiving a message twice is fine as long as the side effect (charge, insert, send) happens once. - Some operations are naturally idempotent (set balance to 100, PUT a full resource). Others are not (add 100, send email) and need explicit deduplication. - You cannot store saved responses forever. Use a sliding window keyed on what the client has already seen, plus session expiry, so the dedup store stays bounded. - Stripe idempotency keys, Kafka idempotent producers, and Raft client sessions are all the same pattern at different layers: a stable ID plus a sequence number plus a stored result. - An at-least-once channel plus an idempotent receiver gives you effectively-once processing. That is the practical answer to exactly-once delivery, which does not really exist end to end. FAQ: - Q: What is an idempotent receiver? A: An idempotent receiver is a server or message consumer that can receive and process the same request multiple times while producing the same effect as processing it once. It works by uniquely identifying each client and each request, storing the result of every request it processes, and returning the stored result when a duplicate arrives instead of executing the operation again. It is one of the patterns documented in Unmesh Joshi's Patterns of Distributed Systems. - Q: Why do distributed systems receive duplicate requests? A: Most networks and message brokers offer at-least-once delivery. When a client sends a request and does not get a response, it cannot tell whether the request was lost on the way, the server crashed before processing, or only the response was lost. To be safe, the client retries. If the server had already processed the original request, the retry becomes a duplicate. Brokers like Kafka, RabbitMQ, and SQS also redeliver messages after a consumer fails to acknowledge in time. - Q: What is the difference between idempotency and exactly-once delivery? A: Exactly-once delivery means the network delivers each message to the receiver exactly one time. This is effectively impossible end to end because of crashes and lost acknowledgements. Idempotency sidesteps the problem: the channel delivers a message at least once, possibly many times, and the receiver is built so that duplicates have no extra effect. The combination of at-least-once delivery plus an idempotent receiver is what people actually mean when they say exactly-once processing. - Q: What is the difference between an idempotent receiver and an idempotent consumer? A: They describe the same idea in different contexts. Idempotent Receiver, from Patterns of Distributed Systems, is the general server-side pattern of deduplicating client requests using a client ID and request number. Idempotent Consumer, from microservices.io and Enterprise Integration Patterns, is the messaging-specific version: a consumer that records the IDs of messages it has processed so it can skip duplicates delivered by the broker. Both rely on a unique message or request identifier and a record of what has already been processed. - Q: How do you make an operation idempotent? A: Two ways. First, choose naturally idempotent semantics where possible, such as setting a value rather than incrementing it, or using PUT with the full resource rather than POST. Second, when the operation has a side effect that cannot be repeated, add explicit deduplication: require a unique idempotency key on each request, store processed keys with their result in a database, and check that store before doing the work. A unique constraint on the key column turns duplicate processing into a safe, detectable conflict. - Q: What is an idempotency key? A: An idempotency key is a unique value, usually a UUID, that a client attaches to a request so the server can recognise retries of that exact request. Stripe popularised this for payment APIs: you send an Idempotency-Key header, and if the same key arrives again, Stripe returns the result of the first call instead of charging the card twice. The key is the practical, API-level form of the client ID plus request number used by the idempotent receiver pattern. - Q: How does Kafka implement idempotency? A: Kafka has an idempotent producer, enabled with enable.idempotence=true (the default since Kafka 3.0). Each producer gets a unique Producer ID, and every message carries a per-partition sequence number. The broker tracks the last sequence number it accepted per producer and partition, so a retried message with a sequence number it has already seen is acknowledged but not written again. This stops producer retries from creating duplicate records, which is the broker-side half of an idempotent receiver. ### ULID Explained: How Sortable Unique IDs Work - URL: https://singhajit.com/ulid-guide/ - Date: 2026-06-02 - Tags: system-design - Description: What is a ULID? Learn how Universally Unique Lexicographically Sortable Identifiers work, the 26-character Crockford Base32 format, ULID vs UUID vs UUID v7, monotonic generation, and how to generate and decode ULIDs in Python, JavaScript, Java, Go, and PostgreSQL. Key Takeaways: - A ULID is 128 bits: a 48-bit millisecond timestamp followed by 80 random bits, encoded as 26 Crockford Base32 characters. - ULIDs are time-sortable as plain strings, so new rows append to the end of a B-tree index instead of scattering like random UUID v4. - ULID and UUID v7 solve the same index-fragmentation problem. UUID v7 is the RFC 9562 standard with native database types, while ULID is shorter (26 chars) and URL-safe. - Monotonic generation keeps same-millisecond IDs in order, but it makes them guessable, so do not use ULIDs as secret tokens, session IDs, or password-reset links. - ULIDs embed a readable creation time, which is convenient for debugging but leaks the row's age to anyone who sees the ID. FAQ: - Q: What is a ULID? A: A ULID (Universally Unique Lexicographically Sortable Identifier) is a 128-bit unique identifier encoded as a 26-character Crockford Base32 string. It combines a 48-bit millisecond timestamp with 80 bits of randomness. The timestamp comes first, so sorting ULIDs as strings sorts them by creation time. ULIDs are case-insensitive and URL-safe. - Q: What is the difference between ULID and UUID? A: ULIDs are 26 characters (Crockford Base32) while UUIDs are 36 characters (hex with hyphens). A ULID always carries a timestamp and is always time-sortable. UUID v4, the most common UUID, is fully random and not sortable. UUID v7 is also time-sortable but stays in the longer 36-character hex format with a native database type. ULIDs are shorter and URL-safe without extra encoding. - Q: Should I use ULID or UUID v7? A: Both place a millisecond timestamp in the high bits, so both give you the same database write performance and avoid the page splits that random UUID v4 causes. Pick UUID v7 if you want the RFC 9562 standard and native uuid columns in PostgreSQL or MySQL. Pick ULID if you want shorter, URL-safe, readable strings or you already use ULIDs. For a brand new project on a database with native uuidv7(), UUID v7 is the safer default. - Q: How do I get the timestamp from a ULID? A: The first 10 characters of a ULID encode a 48-bit Unix timestamp in milliseconds using Crockford Base32. Decode those 10 characters to milliseconds since 1970 and convert to a date. Most ULID libraries expose this directly, and an online ULID decoder will show the creation time in UTC, local time, and Unix milliseconds without writing code. - Q: Are ULIDs safe to use as secret tokens? A: No. ULIDs are designed to be sortable, not secret. With monotonic generation, two ULIDs created in the same millisecond differ by a small increment, so an attacker who sees one can often guess the next. The timestamp is also readable. Use a random UUID v4 or a dedicated cryptographically random token for password resets, session IDs, and API keys. - Q: Can two ULIDs collide? A: It is practically impossible. With 80 bits of randomness per millisecond you would need to generate roughly 1.21 x 10^12 ULIDs in the same millisecond on the same machine to reach a 50 percent chance of a collision. With monotonic generation on a single node, same-millisecond ULIDs never collide because the random part is incremented instead of re-rolled. ### Auth0 vs Okta: How to Pick the Right Identity Platform - URL: https://singhajit.com/auth0-vs-okta/ - Date: 2026-05-30 - Tags: security, system-design, software-engineering - Description: Auth0 vs Okta compared for software developers in 2026. Customer Identity Cloud vs Workforce Identity Cloud, pricing, SSO, SAML, OIDC, MFA, SDKs, Actions, and when to pick Auth0, Okta, Keycloak, or Cognito. Key Takeaways: - Auth0 and Okta are owned by the same company but solve different problems. Auth0 is for your customers, Okta Workforce is for your employees. - Auth0 is priced per Monthly Active User (MAU). Okta Workforce is priced per seat per month. Mixing them up at planning time leads to surprise bills. - Auth0 ships with the largest CIAM developer ecosystem: SDKs for every framework, a generous free tier up to 25,000 MAUs, and the Actions extensibility model. - Okta wins on enterprise breadth. The Okta Integration Network has 8,000+ pre-built apps and SCIM provisioning that IT teams expect out of the box. - Auth0 Rules and Hooks reach end of life on November 18, 2026. Anything still on Rules must be migrated to Actions before then or it stops executing. - Both platforms speak OAuth 2.0, OIDC, and SAML 2.0. Protocol parity is a given. The difference is operational fit, not standards support. - For new B2C or B2B SaaS apps, start with Auth0. For internal SSO across SaaS tools, start with Okta. If you outgrow either on price, look at Keycloak, WorkOS, or Cognito. FAQ: - Q: What is the difference between Auth0 and Okta? A: Auth0 is a Customer Identity and Access Management (CIAM) platform built for developers adding login to their own apps. Okta Workforce Identity Cloud is an Identity and Access Management (IAM) platform built for IT teams managing employee access to SaaS tools. Okta acquired Auth0 in 2021, but the two are sold as separate products with separate pricing, separate dashboards, and separate SDKs. - Q: Did Okta acquire Auth0? A: Yes. Okta completed the acquisition of Auth0 in May 2021 for approximately 6.5 billion dollars in an all-stock deal. Auth0 was later rebranded as Okta Customer Identity Cloud (CIC), but the product, console, and developer-facing brand 'Auth0' are still in active use as of 2026. - Q: Is Auth0 cheaper than Okta? A: It depends on the use case. For customer-facing apps, Auth0's per-MAU pricing is usually cheaper at small to medium scale because you only pay for users who actually log in. For internal employees, Okta's per-seat pricing is more predictable. Auth0 costs can step-function up after the free 25,000 MAUs, and B2B SSO connections beyond the 3 included on Essentials are billed at $100/month each per the official Auth0 pricing page. - Q: Can I use Auth0 and Okta together? A: Yes, and it is a common pattern at larger companies. Okta Workforce handles employees signing into Salesforce, Slack, GitHub, and AWS. Auth0 handles customers signing into the product the company sells. Both can integrate with the same downstream apps using SAML or OIDC, and Okta has tools to federate the two if needed. - Q: Does Auth0 support SAML and OIDC? A: Yes. Auth0 fully supports OAuth 2.0, OpenID Connect, and SAML 2.0 as both an identity provider and a service provider. It also supports WS-Federation, LDAP through enterprise connections, and JWT-based machine-to-machine authentication via the Client Credentials grant. - Q: What replaced Auth0 Rules and Hooks? A: Auth0 Actions replaced Rules and Hooks. Actions provide typed inputs and outputs, npm package support, and a visual flow editor. As of November 18, 2024, Rules and Hooks moved to read-only mode. They will stop executing entirely on November 18, 2026, so any tenant still using them must migrate to Actions before that date. - Q: How many integrations does the Okta Integration Network have? A: The Okta Integration Network (OIN) lists more than 8,000 pre-built integrations as of 2026, with over 1,300 of them supporting SAML 2.0 or OIDC for single sign-on. The rest cover provisioning, lifecycle management, and API integrations for popular SaaS tools. - Q: Auth0 vs Okta for a startup: which one should I use? A: For a startup building a customer-facing product, start with Auth0. The Free tier covers 25,000 MAUs with no credit card required, the SDKs are excellent, and you get login, signup, social providers, passkeys, and 1 enterprise connection in an afternoon. Only adopt Okta Workforce when your company has more than 50 to 100 employees and you need centralized SSO across the SaaS tools your team uses; Okta's Starter Suite begins at $6 per user per month with a $1,500 annual minimum. - Q: What are the alternatives to Auth0 and Okta? A: For CIAM, the main alternatives to Auth0 are Clerk, Stytch, WorkOS, Firebase Authentication, and AWS Cognito. Self-hosted teams often choose Keycloak. For workforce identity, the main alternatives to Okta are Microsoft Entra ID (formerly Azure AD), OneLogin, JumpCloud, and Ping Identity. - Q: Is Auth0 SOC 2 compliant? A: Yes. Auth0 is SOC 2 Type II, ISO 27001, HIPAA BAA-ready, and GDPR-aligned. The same is true of Okta Workforce Identity Cloud. Both platforms publish trust portals with current compliance reports, which is one of the reasons enterprise auditors recognize them on sight. ### Notification System Design: Push, SMS, Email at Scale - URL: https://singhajit.com/notification-system-design/ - Date: 2026-05-29 - Tags: system-design, distributed-systems - Description: How to design a scalable notification system that sends push, SMS, email, and in-app messages to millions of users. Covers Kafka priority queues, idempotency, retries, FCM and APNs, provider fallback, and DLQs. Key Takeaways: - Never send notifications synchronously from the producing service. Accept the request, return 202, and let workers do the real work in the background. - Split traffic into priority lanes. A 10 million row marketing blast must never sit in line behind a one-time password. - Set the idempotency key in Redis **before** you call the provider, not after. That is the only way to stop double-sends when Twilio or SendGrid times out without confirming delivery. - Retry with exponential backoff and jitter, then move to a [dead-letter queue](/role-of-queues-in-system-design/) so one bad message does not poison the worker. - User preferences belong in a fast cache like Redis with the database as the source of truth. Check them at the router stage, not at the channel. - Treat every third-party provider as a flaky dependency. Wrap it in a [circuit breaker](/circuit-breaker-pattern/), keep a fallback, and respect their 429 responses. - Webhook handlers are not optional. The provider knows whether the email bounced or the SMS hit the carrier. Your own logs only know what you sent. - Scheduled notifications need a leader-elected timer service backed by a durable store. A cron job on one box will silently miss messages. - Rate limit per user, per channel, and per provider. Otherwise a bug in one campaign will spam every user on file and earn a global IP block. - Observability is queue depth, provider latency, retry count, and delivery success rate. If you cannot answer 'where is my SMS' in 30 seconds, the system is undebuggable. FAQ: - Q: How do you design a scalable notification system? A: Decouple the producer from the channel. The Notification API accepts an event, validates it, deduplicates it with an idempotency key, and writes it to a durable queue like Kafka with the response status set to 202 Accepted. Priority queues (transactional, social, marketing) keep critical traffic out of the line. Stateless workers read from the queues, resolve user preferences from a fast cache, render templates, and call channel-specific dispatchers (FCM and APNs for push, SendGrid or Amazon SES for email, Twilio or Vonage for SMS). Failures are retried with exponential backoff and routed to a dead-letter queue after the retry budget. A webhook handler closes the loop by recording actual delivery and engagement events. - Q: How do you prevent duplicate notifications? A: Carry an idempotency key on every event end to end. The Notification API generates or accepts an Idempotency-Key header, records (key, status) in Redis with a TTL of a few hours, and rejects duplicate events. Workers also check the key before they call any external provider, because providers can time out after they have already delivered, and a naive retry will send the same SMS or push twice. Pair the cache with a UNIQUE constraint in the notification log database as the last line of defense. - Q: How do you handle push notifications at scale? A: Use Firebase Cloud Messaging for Android and the Apple Push Notification service for iOS. Keep long-lived HTTP/2 connections to both, batch where the provider supports it, and respect their rate limits. FCM publishes a default quota of 600,000 messages per minute per project and returns 429 with a retry-after header when you exceed it. APNs throttles per device token and will signal back with 429 or GOAWAY frames. Treat both providers as flaky dependencies, wrap them in a circuit breaker, and queue retries with at least 10 seconds of backoff plus jitter. - Q: What is the difference between fan-out on write and fan-out on read for notifications? A: Fan-out on write means you expand a single event into per-recipient rows as soon as it arrives and queue each one for delivery. It gives low latency but uses more storage and more queue traffic, which is the right trade for transactional alerts. Fan-out on read means you store one event and resolve recipients when a worker pulls it, which is cheaper for huge broadcasts like 'new feature launched' but slower per user. Most production platforms run a hybrid: fan-out on write for transactional messages and small audiences, staged fan-out on read for marketing blasts and broadcasts. - Q: How do you handle user preferences and quiet hours? A: Store preferences in a relational database keyed by user id, channel, and category. Cache them in Redis with a write-through pattern so the worker check is sub-millisecond. The router stage of the pipeline applies three filters: hard opt-outs (regulatory), category opt-outs ('no marketing'), and time-based gates (quiet hours, do-not-disturb). Critical safety messages bypass quiet hours by design and that exception is documented in the preference schema, not in worker code. - Q: How do you guarantee at-least-once delivery for notifications? A: Persist the event to a durable log like Kafka before you return 202, retry transient failures with exponential backoff, and route exhausted retries to a dead-letter queue. Combine that with idempotent consumers and an idempotency key cached in Redis. Exactly-once is not realistic across third-party providers like APNs, FCM, Twilio, or SendGrid, so the practical goal is at-least-once delivery with idempotent dispatch, which is what platforms like Slack, Uber, and Doordash run in production. - Q: How do you design scheduled and recurring notifications? A: Use a leader-elected scheduler backed by a durable store. The classic pattern is a Redis sorted set keyed by `next_fire_at` with the message id as the value, mirrored to a row in PostgreSQL or ScyllaDB for crash recovery. A single leader pops due entries every second, publishes them to the normal Notification API, and removes them from the set. For recurring notifications, the worker reinserts the next occurrence with the new fire time. Always store the schedule outside the worker process so a single host failure does not lose hours of pending reminders. - Q: How do you implement rate limiting in a notification system? A: Apply rate limits at three layers: per user (no more than N pushes per hour to avoid notification fatigue), per channel (respect the provider rate limit), and per producer (so a buggy upstream service cannot consume the whole queue). A [token bucket](/dynamic-rate-limiter-system-design/) implemented in Redis with INCR and TTL works well for all three. When the limit is hit, the worker either delays, drops, or aggregates depending on category. Marketing rate-limit hits get dropped. Transactional rate-limit hits get delayed. - Q: How do real companies like Slack handle notification fan-out? A: Slack publishes every message to a Kafka topic and lets the notification service consume it as one of several downstream consumers. That decouples notification latency from message delivery, which means a slow APNs call cannot block the chat itself. Slack also runs an internal job queue layer (called JQRelay) that buffers jobs in Kafka before they hit Redis, and uses span-based notification tracing keyed by a notification_id to track every push from creation to open. The architecture is documented in detail on the Slack engineering blog at https://slack.engineering/tracing-notifications/. - Q: What are the most common notification system design interview questions? A: The canonical prompt is: design a notification system that sends push, SMS, and email to 100 million users with at-least-once delivery, opt-outs, and priority handling. Follow-ups include how you prevent duplicates, how transactional traffic stays ahead of marketing, how you handle FCM and APNs rate limits, how you implement fan-out for broadcasts, how the scheduler survives a host crash, how you reconcile delivery status across providers, how you make every step idempotent, how the system degrades when Twilio is down, and how you keep your sender reputation clean. Strong answers always cite capacity numbers up front and pick one delivery guarantee (at-least-once) and own the trade-offs. ### Flash Sale System Design: Architecture, Scale, and Oversell - URL: https://singhajit.com/flash-sale-system-design/ - Date: 2026-05-16 - Tags: system-design, distributed-systems - Description: How to design a flash sale system that handles millions of buyers, prevents overselling, and blocks duplicate orders with Redis, queues, and idempotency keys. Key Takeaways: - Flash sales fail at the inventory row, not at the load balancer. Move the stock counter to Redis, decrement it with a Lua script, and the database stops being the bottleneck. - A virtual waiting room is the single highest-leverage component. It converts a 10 million person spike into a steady stream the rest of the system can plan for. - Idempotency keys make every order operation safe to retry. Pair them with a `UNIQUE(user_id, sku_id, sale_id)` index and duplicate orders become impossible. - Pre-allocate tokens equal to stock. Only token holders can place an order. Overselling becomes mathematically impossible because tokens cannot exceed inventory. - Never write to the order database synchronously. Push the order intent to [Kafka](/kafka-vs-rabbitmq-vs-sqs/), return a 202 to the user, and process payment and fulfillment in the background. - Use a [token bucket rate limiter](/dynamic-rate-limiter-system-design/) per user, per IP, and per device fingerprint to keep bots from draining stock in the first 50 milliseconds. - Cache the sale page on a CDN with a short TTL. The cheapest request is the one that never reaches your servers. - Plan for hot keys. The single SKU on sale becomes the hottest cache key on the planet, so shard the counter, use local read replicas, and fall back to negative caching. - Test with shadow traffic at 5x peak before the sale opens. Every postmortem says the same thing: we did not load test the actual product, only the happy path. - Treat the database `UNIQUE` constraint as your safety net, not your primary defense. If it ever fires under load, something upstream is wrong. FAQ: - Q: How do you design a flash sale system that handles millions of users? A: Layer the system so each tier absorbs an order of magnitude of traffic before the next one sees it. A CDN serves the static sale page, a virtual waiting room admits users in batches with random scoring, a token gate hands out exactly as many tokens as there are units in stock, a Redis Lua script atomically decrements the inventory counter, the order intent is published to a Kafka topic, and a worker pool consumes the topic to write the order to a relational database protected by a UNIQUE constraint. Synchronous work on the user request is kept to a Redis call and a queue enqueue, both of which are sub-millisecond. Everything else (payment, fulfillment, email) is asynchronous. - Q: How do you prevent inventory overselling during a flash sale? A: Overselling happens when two requests read the same stock count, both see one unit available, and both decrement to zero, creating two winners for one item. Three patterns prevent this. The first is a SQL UPDATE with a WHERE clause that requires stock greater than zero, which is atomic at the database level. The second is an optimistic version column with compare-and-swap. The third, and the one most flash sale systems use, is a Redis Lua script that runs DECR on the stock key as a single atomic operation, since Lua scripts on a single Redis node run with no interleaving. A token gate that pre-allocates exactly N tokens for N units is the strongest form: overselling becomes structurally impossible. - Q: How do you prevent duplicate orders from the same user in a flash sale? A: Use an idempotency key on every order create request, plus a database unique constraint as the safety net. The client generates a UUID before clicking Buy, sends it as an Idempotency-Key header, and the server records (idempotency_key, user_id, sale_id) in a fast store like Redis with a short TTL. A duplicate request with the same key returns the cached response. The database also enforces UNIQUE(user_id, sale_id) so even if the cache is bypassed, the second insert fails with a constraint violation. The pattern is the same one [Stripe uses to prevent double payments](/how-stripe-prevents-double-payment/). - Q: What is a virtual waiting room and why do flash sales need one? A: A virtual waiting room is a separate service that holds incoming users in a queue and admits them to the real application in controlled batches. It is built on a Redis sorted set: every arriving user is added with a random score, the system pops the lowest N scores every few seconds, and admitted users receive a short-lived signed token that the real backend trusts. Without a waiting room, a spike of 10 million users hits your origin all at once. With one, the origin sees a steady stream of, say, 20,000 users per second, which it can actually serve. Ticketmaster, Nike SNKRS, and Supreme all run some form of this pattern. - Q: Should I use Redis or a database for flash sale inventory? A: Use Redis for the hot-path stock counter, and the database as the durable source of truth. Pre-load the counter into Redis when the sale starts, decrement it with a Lua script on every order intent, and asynchronously sync the final counts back to the database. Redis on a single node can handle 100,000 plus DECR operations per second with sub-millisecond latency, which a database row under SELECT FOR UPDATE cannot. The database still owns the durable order rows, the audit log, and the final reconciliation, but it never sees the raw spike on the inventory row. - Q: How does Alibaba handle Singles' Day flash sales? A: Alibaba's Double 11 system processes hundreds of thousands of orders per second at peak. The key patterns are pre-warmed inventory caches across many regions, an in-memory token gate per SKU, a heavily customized message queue (RocketMQ) for asynchronous order processing, an active-active multi-region deployment so any single region can fail without taking down checkout, and aggressive load testing weeks before the event. The architecture has evolved over a decade and grew from a single-region monolith to a cloud-native multi-region system handling more than 400 times the peak transaction volume of its first year. - Q: What is the difference between optimistic and pessimistic locking for flash sale inventory? A: Pessimistic locking uses SELECT FOR UPDATE on the inventory row, which holds a row lock for the duration of the transaction. It serializes access and prevents overselling, but every order waits for the lock, which destroys throughput at flash sale scale. Optimistic locking adds a version column and an UPDATE WHERE version = expected_version clause; if another transaction won the race, the UPDATE returns zero rows affected and the application retries. Optimistic is faster under low contention but degrades into a retry storm under flash sale contention. For flash sales, neither pure approach scales: the practical answer is to move the counter to Redis with an atomic Lua script and use the database only for durable order storage. - Q: How do you stop bots from buying up flash sale inventory? A: Bot defenses are a layered problem. Rate limit per IP, per account, and per device fingerprint with a token bucket. Require CAPTCHA or proof-of-work before admission to the waiting room. Score requests with a fraud signal (residential vs datacenter IP, missing headers, JS fingerprint mismatch). Quarantine new accounts so they cannot participate in their first sale. Sign and short-TTL every admission token so it cannot be replayed. None of these defenses are absolute, and large-scale sneaker drops still see significant bot traffic, but each layer materially reduces the share of inventory that bots can win. - Q: How do you scale a flash sale system across multiple regions? A: Sharded inventory is the cleanest pattern. Split the total stock across regions in proportion to expected demand (for example, 60 percent US, 30 percent EU, 10 percent APAC). Each region runs its own Redis counter and decrements locally with no cross-region coordination. A periodic reconciliation job rebalances unsold stock between regions. For globally rare items (limited edition drops where every unit must be available everywhere), a single authoritative region handles the counter and remote regions proxy the DECR call. The trade-off is latency for global consistency versus simplicity for sharded consistency, and most real systems pick sharded with periodic rebalancing. - Q: What are the most common flash sale system design interview questions? A: The classic prompt is: design a flash sale system that sells 10,000 units of a phone to 10 million users at noon. Follow-ups include how you prevent overselling, how you prevent duplicate orders, how you handle the 1000-to-1 read-to-write ratio on the product page, how the virtual waiting room works, how Redis Lua scripts give you atomicity, how you keep the database from being the bottleneck, how you process payments asynchronously, how you defend against bots, how you handle a Redis node failure mid-sale, and how you reconcile inventory across regions. Strong answers always start with capacity numbers and end with named trade-offs. ### Design TinyURL: System Design Interview Guide for URL Shorteners - URL: https://singhajit.com/tinyurl-system-design/ - Date: 2026-05-04 - Tags: system-design, distributed-systems - Description: A practical guide to designing a URL shortener like TinyURL or Bitly. Walk through requirements, capacity estimation, base62 encoding, ID generation, database schema, caching, redirects, analytics, custom aliases, and rate limiting. Built for system design interviews and real production systems. Key Takeaways: - A URL shortener is a high-read, low-write key-value system. Optimize the redirect path first, everything else is secondary. - Use base62 encoding of a unique integer ID. Six characters give 56 billion codes, seven give 3.5 trillion. That is enough for any realistic system. - Pick one of three ID strategies: a counter with base62 encoding, a [Snowflake ID](/snowflake-id-guide/), or a Key Generation Service that pre-mints codes in batches. - The schema is small. Short code as the primary key, long URL, owner, created at, expires at, click count. Index on short code for O(1) lookup. - Cache aggressively. The top 20 percent of links serve 80 percent of traffic, so a Redis layer in front of the database gives a 95 percent hit rate. - Use 301 only when you do not need analytics. Use 302 when you want to count every click. The difference matters on cache, SEO, and click data. - Analytics belongs on a separate async pipeline (Kafka, queue, or stream). Never block a redirect on writing a click record. - Custom aliases need uniqueness checks at insert time. Reserve a namespace for them so they never collide with generated codes. - Rate limit URL creation per user and IP. Most abuse comes from spam, phishing, and malware shorteners, not from organic load. - The system design interview answer is mostly trade-offs: SQL vs NoSQL, counter vs hash, 301 vs 302, sync vs async analytics, single region vs multi region. FAQ: - Q: How do you design a URL shortener like TinyURL? A: Start with the read path. A URL shortener is a key-value lookup where the short code maps to a long URL. The system needs a short code generator (base62 of a unique integer ID, a Snowflake ID, or a Key Generation Service), a primary store keyed by the short code (PostgreSQL with read replicas at small scale, DynamoDB or Cassandra at large scale), a Redis cache in front of the database for hot URLs, and an async analytics pipeline (Kafka or a queue) so a click never blocks the redirect. Add rate limiting on the create endpoint to slow down abuse, and reserve a namespace for custom aliases so they cannot collide with auto-generated codes. The full architecture is a CDN, an API gateway, stateless app servers, the cache, the database, and a separate analytics consumer. - Q: How does TinyURL generate short codes? A: TinyURL and similar services map a unique integer ID to a base62 string of length 6 or 7. Base62 uses the alphabet a-z, A-Z, and 0-9, which gives 62^6 = 56.8 billion codes at length 6 and 62^7 = 3.5 trillion at length 7. The integer ID can come from a database auto-increment column, a Snowflake-style distributed ID generator, or a Key Generation Service that hands out pre-minted batches of codes to app servers. Hash-based approaches (MD5 or SHA-1 of the long URL, then truncate) are simple but cause collisions and force retry logic, so most production systems use the counter plus base62 approach. - Q: What database should I use for a URL shortener? A: For a small to medium service handling tens of millions of URLs, PostgreSQL or MySQL with one or two read replicas is enough. The schema is one table keyed by the short code, with the long URL, owner, created at, and expires at columns. For very large services (billions of URLs and hundreds of thousands of redirects per second), a horizontally partitioned NoSQL store like DynamoDB, Cassandra, or ScyllaDB is a better fit because the workload is a pure key lookup with no complex joins. Either way, the short code is the primary key and you put a hash index or B-tree index on it for O(1) or O(log n) lookups. - Q: How does a URL shortener handle billions of redirects per day? A: Three layers do most of the work. A CDN caches the redirect at the edge for each unique short code, which absorbs the bulk of repeat traffic. A Redis cluster sits in front of the database with the most popular codes in memory, giving sub-millisecond lookup and a 95 percent hit rate because of the long-tail distribution of clicks. The database itself is partitioned by the short code so reads can scale horizontally. Analytics writes go to a separate async pipeline (Kafka or a queue) so the redirect path does only one cache or database lookup before returning a 301 or 302. - Q: Should a URL shortener use 301 or 302 redirects? A: Use 301 (permanent) when you do not need to count clicks and want browsers and proxies to cache the redirect aggressively. This reduces server load but kills click analytics, since most clicks never reach your servers. Use 302 (temporary, found) when click analytics matters because every click goes through your service. Most modern URL shorteners default to 302 to preserve analytics, with the trade-off being more traffic to absorb. SEO impact also differs: 301 passes link equity to the destination, while 302 does not. - Q: How do you avoid collisions in URL shortener codes? A: Avoid collisions by using a counter plus base62 instead of a hash. Each new URL gets the next unique integer ID, encoded into base62. Two URLs cannot share an ID, so they cannot share a short code. If you must use a hash (for example, for content-deduplication so the same long URL always returns the same short code), check for collisions on insert by comparing the long URL on hash matches and either accept the collision (return the existing short URL) or rehash with a salt. A Key Generation Service that pre-mints codes in batches removes collisions entirely because each app server consumes from its own private range. - Q: How long should the short code be? A: Six or seven base62 characters covers any realistic scale. Six characters give 62^6 = 56.8 billion unique codes, which is enough for tens of years of URLs at TinyURL or Bitly volume. Seven characters give 62^7 = 3.5 trillion, which is essentially unlimited. Going shorter than six (say four characters at 14.7 million) makes codes guessable and constrains the system, while going longer than eight wastes characters that users will paste into chat, SMS, and QR codes. Most services start with seven characters to leave headroom. - Q: How does Bitly differ from TinyURL? A: Functionally they are the same product: long URL in, short URL out. Bitly historically added analytics dashboards, branded domains (so you can use your own short domain like brnd.co), team collaboration, and an enterprise API tier with rate limits in the millions per month. TinyURL is closer to the original 2002 service: anonymous shortening, basic custom aliases, and no analytics by default. Architecturally both are read-heavy key-value lookups with a Redis cache, a partitioned database, and an async analytics pipeline. The interesting differences are in commercial features, fraud prevention, and how each service handles billions of historical redirects. - Q: How do you prevent abuse in a URL shortener? A: Rate limit URL creation per IP, per account, and globally so a single attacker cannot mint millions of codes that point to phishing or malware. Run new long URLs through Google Safe Browsing or a similar reputation API at create time. Strip and validate the URL (only http and https, no javascript: schemes, no exotic ports). Add a delay or CAPTCHA on the redirect for newly created codes so an attacker cannot use your service as part of a click-fraud pipeline. Log every click with IP, user agent, and referer so abuse patterns are detectable. Periodically rescan stored URLs and disable any that are now flagged as malicious. - Q: What are the most common system design interview questions for URL shorteners? A: The classic question is: design a URL shortener like TinyURL that handles 100 million URLs per month and 10 billion redirects per month. Common follow-ups are: how do you generate short codes (base62, Snowflake, or KGS), how do you handle collisions, how does the database schema look, how do you cache, how do you scale read traffic, what are 301 versus 302 trade-offs, how do you implement custom aliases, how do you handle expiration, how do you collect analytics without blocking the redirect, how do you prevent abuse, and how do you scale across multiple regions. Strong answers always start with capacity estimation and end with explicit trade-offs. ### Saga Pattern Explained: Distributed Transactions for Microservices - URL: https://singhajit.com/saga-pattern-distributed-transactions/ - Date: 2026-04-29 - Tags: system-design, distributed-systems, microservices, software-engineering - Description: A deep, no-fluff guide to the Saga Pattern for software developers. Learn how choreography and orchestration sagas work, how to design compensating transactions, when to pick saga over two-phase commit, and how to implement sagas with Kafka, Temporal, AWS Step Functions, and the Outbox pattern. Key Takeaways: - A saga is a sequence of local ACID transactions tied together by events or commands, where each step has a paired compensating transaction to undo it. - Sagas exist because two-phase commit does not scale across microservices and most modern brokers like Kafka and SQS do not even support distributed transactions. - Choreography sagas are simple to start, hard to debug at five or more steps. Orchestration sagas add a coordinator that owns the flow and is much easier to operate. - Compensating transactions are not rollbacks. They are real business operations like refunds, restocks, and apology emails, written in normal code that can fail and must be retried. - Every saga step and every compensation must be idempotent, because brokers and orchestrators retry on failure and you will get duplicates. - Sagas give up isolation. You can see dirty intermediate state. Use semantic locks, commutative updates, or pessimistic views to keep that bounded. - Pair sagas with the Transactional Outbox pattern so the local transaction and the event publish never drift apart. - In production, most teams settle on orchestration with Temporal, AWS Step Functions, Camunda, or Eventuate Tram. The hand-rolled choreography saga rarely survives contact with a real on-call rotation. FAQ: - Q: What is the Saga Pattern? A: The Saga Pattern is a way to manage data consistency across services that each own their own database. Instead of one big distributed transaction, you split the work into a sequence of local transactions, one per service. Each step publishes an event or accepts a command, and each step has a compensating transaction that can undo its effect if a later step fails. The result is an eventually consistent business workflow without two-phase commit. - Q: What problem does the Saga Pattern solve? A: It solves the loss of ACID transactions across microservices. With a database-per-service architecture you cannot wrap an order, payment, and inventory write in one BEGIN and COMMIT. The Saga Pattern gives you a structured way to coordinate those writes, and to roll the world back to a sane state when one of the writes fails. - Q: What is the difference between choreography and orchestration in sagas? A: In a choreography saga, each service reacts to events published by others. There is no central brain. In an orchestration saga, a dedicated orchestrator calls each service in turn, decides the next step, and triggers compensation on failure. Choreography is simpler to build and lighter on infrastructure but gets tangled past four or five steps. Orchestration adds a coordinator but gives you a clear state machine, easier debugging, and centralized observability. - Q: What is a compensating transaction? A: A compensating transaction is the business operation that undoes the effect of an earlier saga step. If the earlier step charged a card, the compensating step issues a refund. If it reserved seats, the compensating step releases them. Compensations are written by you in application code. They are not database rollbacks. They must be idempotent and they must be designed to handle the world having moved on. - Q: When should I use the Saga Pattern instead of two-phase commit? A: Use sagas in microservice architectures where each service owns its database and you cannot, or do not want to, run a distributed transaction coordinator across them. Two-phase commit needs all participants to support XA transactions, holds locks for the duration of the transaction, and reduces availability. Most message brokers like Kafka, RabbitMQ, and SQS do not support 2PC at all. Sagas are the practical default for cross-service workflows. - Q: Does the Saga Pattern guarantee strong consistency? A: No. Sagas give you eventual consistency. While the saga is running, other services can read partial intermediate state, like a pending order with no payment yet. You handle that with semantic locks, status fields like PENDING or RESERVED, and clear UI states. If you cannot tolerate any inconsistency window, sagas are not the right pattern. - Q: How does the Saga Pattern work with Kafka? A: Each service performs its local database write and publishes an event to a Kafka topic in the same atomic step, usually via the Transactional Outbox pattern. Other services consume those events and do their own local transactions. For orchestration sagas, the orchestrator either consumes events directly or sends commands to per-service command topics. Idempotent consumers are required because Kafka guarantees at-least-once delivery. - Q: What tools implement the Saga Pattern in production? A: The most common in 2026 are Temporal, AWS Step Functions, Camunda 8 (Zeebe), Azure Durable Functions, Netflix Conductor, Eventuate Tram Sagas for Spring Boot, and Axon Framework for event-sourced systems. Each takes care of state persistence, retries, timeouts, and visibility so you do not have to write a saga engine from scratch. ### How GitHub Stores and Serves Git Repositories - URL: https://singhajit.com/how-github-stores-and-serves-git-repositories/ - Date: 2026-04-25 - Tags: system-design, git, github, distributed-systems - Description: A developer's guide to how GitHub stores and serves Git repositories. Walk through the Spokes replication system, three-replica voting, the Git proxy, pack files, monorepo optimizations like multi-pack-index and reftable, and Git LFS. Learn what really happens between git push and the bytes landing on disk. Key Takeaways: - Every repository on GitHub lives on three independent fileservers chosen at random across the pool. One server can fail and nothing breaks. - Spokes is application-level replication using plain Git, not block-level replication. The replicas are loosely coupled Git repositories kept in sync over Git protocols and rsync. - Writes use a three-phase commit and a quorum vote. A push is only durable once two replicas have committed the same reference transaction. - Reads are routed to the closest in-sync replica. That is how a clone in Singapore can be served from a fileserver in Singapore even though the write happened in Virginia. - Failure detection is driven by real application traffic, not by heartbeats. Three failed RPCs in a row push a server to the back of the routing list. - Git itself is the bottleneck and the secret. Pack files, reachability bitmaps, multi-pack-index, partial clone, and reftable are what let GitHub scale a monorepo to millions of refs without melting. - Large binaries do not live in the Git object store. Git LFS swaps them for tiny pointer files and stores the bytes on object storage. - The mental model that fixes most performance and reliability questions is short: proxy at the front, three plain Git repos on SSD at the back, MySQL for metadata, object storage for binaries. FAQ: - Q: How does GitHub store Git repositories? A: GitHub stores each repository as three independent copies on three different fileservers using a system called Spokes, formerly known as DGit. Each fileserver stores a normal bare Git repository on local SSD. A proxy layer sits in front and replicates every write to all three copies using a three-phase commit so that at least two replicas always agree on the same Git state. Reads are routed to the closest in-sync replica. The repository metadata, like ownership and which fileservers hold which repo, is kept in a sharded MySQL tier. - Q: What is GitHub Spokes (formerly DGit)? A: Spokes is GitHub's distributed replication system for Git repositories. It was originally launched in 2016 under the name DGit and renamed Spokes in late 2016. Spokes keeps three replicas of every repository on three independently chosen fileservers, replicates every push synchronously to all three using a three-phase commit, and self-heals by creating new replicas whenever a server fails. It replaced an older DRBD block-level replication system. Spokes was later extended in the Stretching Spokes work to allow replicas to live in different datacenters across continents. - Q: Why does GitHub keep three replicas of every Git repository? A: Three is the smallest number that lets you tolerate one failure while still making decisions by majority vote. With three replicas, any write that two of them agree on is the official outcome, so a single server can fail and the repository stays both readable and writable. Two-of-three quorum also gives Spokes natural conflict resolution because two concurrent writes cannot both acquire a majority of locks at the same time. - Q: What protocol does GitHub use for git push and git fetch? A: GitHub speaks the standard Git wire protocol, either over SSH on port 22 or over HTTPS on port 443 using the smart HTTP protocol. Both arrive at a proxy layer that authenticates the user, looks up which three fileservers hold the repository in the metadata database, opens connections to those fileservers, and forwards the upload-pack or receive-pack streams. From a developer's perspective it looks like a normal git remote, but every byte is being multiplexed across three Git processes on three machines. - Q: How does Git itself store data on disk? A: Git stores everything as content-addressable objects in the .git/objects directory. There are four object types, blob for file contents, tree for directories, commit for snapshots, and tag for annotated tags. Each object is identified by the SHA-1 (or SHA-256) hash of its contents. New objects start as loose files compressed with zlib. Garbage collection packs many loose objects into a single pack file with a separate index for fast lookup. References, like branches and tags, are tiny files in .git/refs that point to commit object hashes. - Q: How does GitHub handle large files in Git? A: GitHub uses Git Large File Storage, also called Git LFS. When you track a file pattern with git lfs track, Git replaces the actual file content with a small text pointer file containing the SHA-256 hash and size of the real content. The pointer is stored in the normal Git object store, while the actual bytes are uploaded over HTTPS to GitHub's LFS object storage backend, which is built on top of S3-style object storage. On checkout, a smudge filter downloads the real file using the hash in the pointer. - Q: How does GitHub scale to a monorepo with millions of files? A: GitHub scales monorepos with a stack of Git features it actively co-develops with the upstream Git project. Pack files compress the object store. The multi-pack-index lets a single index span many pack files. Reachability bitmaps make commit traversal almost free. Partial clone lets clients download only the objects they need. Sparse checkout limits which files appear on disk. Reftable replaces the slow packed-refs file with a binary format that scales to millions of references. Together these features let pushes that touch hundreds of refs complete in a fraction of a second. - Q: Is GitHub still based on plain Git on the server? A: Yes. Despite the scale, every fileserver still runs the standard Git binary on a normal Linux filesystem with local SSDs. GitHub's reasoning is that Git is highly sensitive to disk and object lookup latency, so any abstraction like a SAN or a distributed filesystem would slow it down. Spokes adds replication and routing on top of plain Git rather than replacing the storage engine. Several of the upstream Git performance features were contributed by GitHub engineers to make this approach keep up with growth. ### REST vs GraphQL vs gRPC: How to Pick the Right API Protocol - URL: https://singhajit.com/rest-vs-graphql-vs-grpc/ - Date: 2026-04-09 - Tags: system-design, distributed-systems, software-engineering - Description: A developer's guide to choosing between REST, GraphQL, and gRPC. Covers architecture, performance benchmarks, serialization formats, streaming, error handling, versioning, security, and real-world use cases at Netflix, GitHub, Uber, and Shopify. Includes decision flowchart and code examples. Key Takeaways: - REST is the safest default for public APIs. It is simple, cacheable, universally supported, and every developer already knows it. - GraphQL solves real problems with over-fetching and under-fetching, but introduces new ones: N+1 queries, query complexity attacks, and caching difficulties. - gRPC uses Protocol Buffers for binary serialization, HTTP/2 for multiplexing, and supports four streaming patterns. It is the fastest of the three but not browser-native. - Most production systems use more than one protocol. gRPC between backend services, GraphQL for client-facing BFF layers, and REST for partner APIs. - Choosing an API protocol is not about which is best. It is about which trade-offs you can live with for each communication boundary in your system. - Schema evolution works differently in each: REST uses URL versioning, GraphQL uses additive schema changes, and gRPC uses Protocol Buffer field numbering rules. FAQ: - Q: What is the difference between REST and GraphQL? A: REST exposes fixed endpoints that return predefined data structures. Each resource has its own URL, and the server decides what fields to include. GraphQL exposes a single endpoint where the client specifies exactly which fields it needs using a query language. REST is simpler to cache and more widely supported. GraphQL eliminates over-fetching and under-fetching but requires more server-side complexity for query parsing, validation, and execution. - Q: What is the difference between REST and gRPC? A: REST uses JSON over HTTP/1.1 with text-based serialization. gRPC uses Protocol Buffers over HTTP/2 with binary serialization. gRPC is significantly faster due to smaller payloads, multiplexed connections, and header compression. REST is browser-native and human-readable. gRPC requires code generation from .proto files and is mainly used for internal service-to-service communication where performance matters more than readability. - Q: What is the difference between GraphQL and gRPC? A: GraphQL is a query language designed for client-facing APIs where different clients need different data shapes. gRPC is a binary RPC framework designed for fast, type-safe communication between backend services. GraphQL gives clients flexibility in what data they request. gRPC gives services performance and strict contracts through Protocol Buffers. They solve different problems and are often used together in the same system. - Q: When should I use REST over GraphQL? A: Use REST when building public APIs for third-party developers, when HTTP caching is important, when your data model is simple with straightforward CRUD operations, or when your team is small and wants the simplest possible API layer. REST is also the better choice when you need maximum compatibility across all clients and programming languages. - Q: When should I use gRPC over REST? A: Use gRPC for internal service-to-service communication in microservices where latency and throughput matter. gRPC delivers 5-10x higher throughput than REST with significantly lower latency due to binary serialization, HTTP/2 multiplexing, and persistent connections. Also use gRPC when you need bidirectional streaming or when you have polyglot services that need auto-generated type-safe clients. - Q: Is GraphQL faster than REST? A: GraphQL is not inherently faster than REST. A single GraphQL request can replace multiple REST calls, which reduces total network round trips. But the GraphQL server does more work to parse, validate, and execute the query. For simple single-resource requests, REST is typically faster. GraphQL wins when a client needs data from multiple related resources that would require several REST calls. - Q: Can I use REST, GraphQL, and gRPC together? A: Yes. Many production systems use all three protocols at different boundaries. A common pattern is gRPC for internal microservice communication, GraphQL as a BFF (Backend for Frontend) layer that aggregates data from gRPC services, and REST for public partner APIs. Netflix, Uber, and Shopify all use multiple protocols across their systems. - Q: What is Protocol Buffers and why does gRPC use it? A: Protocol Buffers (protobuf) is a binary serialization format developed by Google. You define your data structures in .proto files, and the protobuf compiler generates code for your programming language. gRPC uses protobuf because binary encoding is 3-5x smaller than JSON, serialization is 5-10x faster, and the schema provides strict type safety. The tradeoff is that protobuf messages are not human-readable like JSON. - Q: What is the N+1 problem in GraphQL? A: The N+1 problem occurs when a GraphQL query fetches a list of N items and then executes a separate database query for each item to resolve a nested field. For example, fetching 100 users and their posts results in 1 query for users plus 100 queries for posts, totaling 101 database queries. The solution is to use a DataLoader that batches and deduplicates these queries. Without a DataLoader, GraphQL APIs can have serious performance problems. ### Transactional Outbox Pattern: Never Lose an Event Again - URL: https://singhajit.com/transactional-outbox-pattern/ - Date: 2026-04-07 - Tags: system-design, distributed-systems, software-engineering - Description: Complete guide to the transactional outbox pattern for reliable event publishing in microservices. Learn how to solve the dual write problem using an outbox table, polling relay, and change data capture (CDC) with Debezium and Kafka. Includes outbox table schema, code examples, and production best practices. Key Takeaways: - The dual write problem occurs when a service must update its database and publish an event to a message broker. If either fails, the system becomes inconsistent. - The outbox pattern converts a dual write into a single write. Business data and events go into the same database transaction. - A relay process reads unpublished events from the outbox table and publishes them to the broker. Two approaches: polling and change data capture (CDC). - Polling is simple to implement but adds latency. CDC with tools like Debezium reads the database transaction log for near real-time event publishing. - Consumers must be idempotent because the relay guarantees at-least-once delivery, not exactly-once. Duplicate events are possible. - Companies like Uber, Netflix, and Stripe use variations of this pattern to keep their systems consistent at scale. FAQ: - Q: What is the transactional outbox pattern? A: The transactional outbox pattern is a design pattern for microservices that guarantees reliable event publishing. Instead of writing to the database and publishing to a message broker separately (which can fail independently), the pattern stores events in an outbox table within the same database transaction as the business data. A separate relay process then reads from the outbox and publishes events to the broker. This ensures events are published if and only if the database transaction commits. - Q: What is the dual write problem in microservices? A: The dual write problem occurs when a microservice needs to perform two writes that must succeed together: updating a database and publishing a message to a broker like Kafka or RabbitMQ. Since the database and the broker are separate systems, there is no shared transaction. If the database write succeeds but the broker publish fails, or vice versa, the system ends up in an inconsistent state where some services have stale data. - Q: How does the outbox pattern solve the dual write problem? A: The outbox pattern solves the dual write problem by turning two separate writes into one. Instead of writing to the database and publishing to the broker, the service writes both the business data and the event to the same database in a single transaction. A background relay process then reads unpublished events from the outbox table and publishes them to the message broker. Since both writes happen in one ACID transaction, they either both succeed or both fail. - Q: What is the difference between polling and CDC for the outbox pattern? A: Polling uses a scheduled job that periodically queries the outbox table for unpublished events and publishes them to the broker. It is simple to build but adds latency based on the polling interval. Change Data Capture (CDC) reads the database transaction log directly (like PostgreSQL WAL or MySQL binlog) and streams changes to the broker in near real time. CDC is more complex to set up but provides lower latency and does not add query load to the database. - Q: Does the outbox pattern guarantee exactly-once delivery? A: No. The outbox pattern guarantees at-least-once delivery. If the relay publishes an event to the broker but crashes before marking it as published in the outbox, the event will be published again on the next run. Consumers must be idempotent to handle duplicate events safely. You can achieve this using idempotency keys, deduplication tables, or by designing operations to be naturally idempotent. - Q: What should the outbox table schema look like? A: A practical outbox table includes: id (bigint auto-increment for ordering), aggregate_type (the entity type like Order or Payment), aggregate_id (the entity ID, used as the message key for partition routing), event_type (like OrderCreated or PaymentProcessed), payload (JSON with the event data), created_at (timestamp), and a status or published flag. Use bigint IDs instead of UUIDs for better index performance. - Q: When should I use the outbox pattern vs two-phase commit? A: Use the outbox pattern when you need reliable event publishing from a single service without blocking other services. It works well in microservices where services own their databases. Use two-phase commit (2PC) only when you need strong consistency across multiple databases in the same data center and can tolerate the performance hit. 2PC is blocking and reduces availability. The outbox pattern is non-blocking and works with eventual consistency. - Q: What is Debezium and how does it work with the outbox pattern? A: Debezium is an open-source CDC platform that captures database changes by reading the transaction log. For the outbox pattern, Debezium watches the outbox table and streams new rows to Kafka automatically. It includes an Outbox Event Router SMT (Single Message Transform) that routes events to the correct Kafka topic based on the aggregate_type column and uses aggregate_id as the message key. This eliminates the need to build a custom polling relay. ### Kafka vs RabbitMQ vs Amazon SQS: Picking the Right Message Broker - URL: https://singhajit.com/kafka-vs-rabbitmq-vs-sqs/ - Date: 2026-04-03 - Tags: system-design, distributed-systems, software-engineering - Description: A developer's guide to choosing between Apache Kafka, RabbitMQ, and Amazon SQS. Covers architecture differences, throughput benchmarks, delivery guarantees, message ordering, pricing, and real-world use cases at Uber, Stripe, and Netflix. Updated for Kafka 4.0 and RabbitMQ 4.1. Key Takeaways: - Kafka is a distributed commit log, not a message queue. It retains messages for replay. RabbitMQ and SQS delete messages after consumption. - Kafka handles millions of messages per second. RabbitMQ handles around 100K with publisher confirms. SQS Standard has nearly unlimited throughput but higher per-message latency. - RabbitMQ has the most flexible routing with exchanges: direct, topic, fanout, and headers. Kafka and SQS require you to handle routing at the application level. - SQS costs nothing when idle and needs no infrastructure management. Kafka and RabbitMQ require clusters you provision and maintain. - Kafka 4.0 removed ZooKeeper entirely. KRaft mode is now the only way to run Kafka. This makes deployment significantly simpler than before. - All three support dead letter queues for handling failed messages. The implementation is different, but the pattern works everywhere. FAQ: - Q: What is the difference between Kafka and RabbitMQ? A: Kafka is a distributed commit log designed for high-throughput event streaming. It retains messages for a configurable period and supports replay. RabbitMQ is a traditional message broker that implements AMQP with flexible routing through exchanges. RabbitMQ deletes messages after consumption (unless using streams). Kafka excels at millions of messages per second with multiple consumers. RabbitMQ excels at complex routing, lower latency, and request-reply patterns. - Q: When should I use Amazon SQS instead of Kafka or RabbitMQ? A: Use Amazon SQS when you are on AWS and want zero operational overhead. SQS is fully managed, scales automatically, and you pay only for what you use. Choose SQS for simple task queues, Lambda triggers, and decoupling services in a serverless architecture. Avoid SQS if you need message replay, complex routing, or extremely high throughput with low latency. - Q: Which message broker has the highest throughput? A: Kafka has the highest throughput. It can handle millions of messages per second across partitions. RabbitMQ handles around 100-120K messages per second with publisher confirms in production configurations. SQS Standard has nearly unlimited throughput but adds network latency since it is a managed cloud service. Kafka achieves high throughput through sequential disk I/O, batching, and zero-copy transfers. - Q: Does Kafka guarantee message ordering? A: Kafka guarantees message ordering within a partition, not across partitions. If you need strict ordering for a specific entity like a user or order, use the entity ID as the partition key so all messages for that entity go to the same partition. RabbitMQ guarantees ordering within a queue. SQS FIFO queues guarantee ordering within a message group. - Q: What is the difference between SQS Standard and SQS FIFO? A: SQS Standard offers nearly unlimited throughput with at-least-once delivery and best-effort ordering. Messages may be delivered more than once or out of order. SQS FIFO guarantees exactly-once processing and strict ordering within message groups. Default throughput is 3,000 messages per second with batching, and high throughput mode can push this to 9,000+ TPS per API action. Choose Standard for high throughput. Choose FIFO when message order and deduplication matter. - Q: Can RabbitMQ replay messages like Kafka? A: Traditional RabbitMQ queues delete messages after acknowledgment, so replay is not possible. However, RabbitMQ 3.9 introduced Streams, which use an append-only log similar to Kafka. Streams support non-destructive consumption and offset tracking, allowing consumers to replay messages. RabbitMQ Streams are a newer feature and less mature than Kafka's log-based storage. - Q: How much does each message broker cost? A: Kafka and RabbitMQ are open-source and free to run, but you pay for the infrastructure: servers, storage, and operational effort. Managed services like Confluent Cloud or Amazon MSK add a premium. Amazon SQS charges $0.40 per million requests for Standard queues and $0.50 per million for FIFO queues, with a free tier of 1 million requests per month. SQS has no idle cost. Kafka clusters cost money even when idle. - Q: What is a dead letter queue and do all three support it? A: A dead letter queue stores messages that fail processing after a set number of retries. All three support dead letter queues. SQS has built-in DLQ configuration with a maxReceiveCount setting. RabbitMQ supports DLQ through dead letter exchanges. Kafka implements DLQ at the application level by producing failed messages to a separate topic. The pattern prevents poison messages from blocking healthy processing. - Q: Is Kafka overkill for my application? A: If you are processing fewer than a few thousand messages per second, do not need message replay, and have a single consumer per message, Kafka is likely overkill. The operational cost of running a Kafka cluster, even with KRaft mode, is significant. RabbitMQ or SQS would be simpler and cheaper. Kafka pays off when you need high throughput, multiple consumer groups, event sourcing, or stream processing. - Q: What happened to ZooKeeper in Kafka 4.0? A: Kafka 4.0 removed ZooKeeper entirely. KRaft mode, which uses a Raft-based consensus protocol built into Kafka, is now the only way to run a cluster. This simplifies deployment from two systems (Kafka plus ZooKeeper) to one, increases the maximum partition count from around 200,000 to nearly 2 million, and speeds up controller failover. Existing Kafka clusters must migrate to KRaft before upgrading to 4.0. ### OpenTelemetry in Production: A Complete Setup Guide - URL: https://singhajit.com/opentelemetry-production-guide/ - Date: 2026-03-31 - Tags: devops, system-design, software-engineering - Description: A hands-on guide to running OpenTelemetry in production. Covers the OTel Collector, auto-instrumentation for Java, Python, and Go, sampling strategies, Kubernetes deployment patterns, and how to connect traces, metrics, and logs to backends like Prometheus, Jaeger, and Grafana Tempo. Key Takeaways: - OpenTelemetry is not a monitoring tool. It is a vendor-neutral instrumentation and data pipeline standard. You still need backends like Prometheus, Jaeger, or Grafana Tempo. - Auto-instrumentation works for most frameworks out of the box. Start there and add manual spans only for business-specific logic. - The Collector is the most important piece in production. It handles batching, retries, sampling, and format translation between your apps and backends. - Put the memory_limiter processor first in every pipeline. Without it, a traffic spike or slow backend will OOM your Collector. - Tail-based sampling catches errors and slow requests that head sampling misses, but it needs a stateful Collector tier with trace-aware load balancing. - Monitor your observability pipeline. A Collector that silently drops spans is worse than no Collector at all. FAQ: - Q: What is OpenTelemetry? A: OpenTelemetry (OTel) is an open-source, vendor-neutral observability framework for generating, collecting, and exporting telemetry data: traces, metrics, and logs. It is a CNCF project and the second most active CNCF project after Kubernetes. It provides APIs, SDKs, and the OpenTelemetry Collector so you can instrument your applications once and send data to any backend. - Q: What is the difference between OpenTelemetry and Jaeger? A: OpenTelemetry handles instrumentation and data collection. Jaeger is a tracing backend that stores and visualizes traces. They work together: your application uses OpenTelemetry SDKs to generate traces, the OpenTelemetry Collector processes and exports them, and Jaeger receives and stores them. OpenTelemetry replaced Jaeger's client libraries. - Q: What is the difference between OpenTelemetry and Prometheus? A: Prometheus is a metrics backend with its own pull-based collection model. OpenTelemetry is a vendor-neutral instrumentation layer that can push metrics to Prometheus via remote write, or expose a Prometheus scrape endpoint via the Collector. You can use both together: instrument with OpenTelemetry, store in Prometheus. - Q: What is the OpenTelemetry Collector? A: The Collector is a standalone binary that receives telemetry data from your applications, processes it through a pipeline of receivers, processors, and exporters, and sends it to one or more backends. It handles batching, retries, sampling, attribute manipulation, and format translation. You can deploy it as an agent alongside your app or as a central gateway. - Q: What is OTLP? A: OTLP stands for OpenTelemetry Protocol. It is the native wire format for sending traces, metrics, and logs from OpenTelemetry SDKs to the Collector or directly to backends. OTLP supports gRPC on port 4317 and HTTP/protobuf on port 4318. Most modern observability backends accept OTLP natively. - Q: What is the difference between head sampling and tail sampling in OpenTelemetry? A: Head sampling decides at the start of a request whether to record the trace. It is simple and cheap but can miss errors and slow requests. Tail sampling decides after the request completes, so it can keep all errors and high-latency traces. Tail sampling requires buffering complete traces in memory and routing all spans of a trace to the same Collector instance. - Q: How do I deploy OpenTelemetry in Kubernetes? A: The most common pattern is a DaemonSet Collector on every node for receiving telemetry from pods, plus a central gateway Deployment for processing and exporting. The OpenTelemetry Operator can automate this and inject auto-instrumentation into pods. Use Helm charts from the official opentelemetry-helm-charts repository. - Q: Does OpenTelemetry support auto-instrumentation? A: Yes. OpenTelemetry provides zero-code auto-instrumentation for Java (via a Java agent JAR), Python (via the opentelemetry-instrument CLI), .NET (via a NuGet package), and Node.js (via a require flag). Auto-instrumentation adds spans for HTTP requests, database queries, gRPC calls, and cache operations without modifying your code. - Q: What backends work with OpenTelemetry? A: Nearly all modern observability backends support OpenTelemetry. For traces: Jaeger, Grafana Tempo, Zipkin, Datadog, New Relic, Honeycomb. For metrics: Prometheus, Grafana Mimir, Datadog, Dynatrace. For logs: Grafana Loki, Elasticsearch, Splunk. Most accept OTLP natively. The Collector has exporters for all major vendors. - Q: Is OpenTelemetry production ready? A: Yes. Tracing is stable (GA) across all major language SDKs. Metrics are stable in Java, .NET, Python, and Go. Logs are stable in Java and .NET, and reaching stability in other languages. The Collector is production-ready. Companies like GitHub, Shopify, Canva, and eBay run OpenTelemetry at scale. ### Architecting Multi-Agent AI Swarms: A System Design Deep Dive - URL: https://singhajit.com/multi-agent-ai-swarms-system-design/ - Date: 2026-03-19 - Tags: ai, system-design - Description: Learn how to architect multi-agent AI systems. This deep dive covers orchestration patterns (supervisor, pipeline, mesh), inter-agent communication, memory management, framework comparison (LangGraph vs CrewAI vs AutoGen), production failures, and practical lessons for software developers building agentic AI swarms. Key Takeaways: - Multi-agent systems outperform single agents when tasks need different skills, parallel execution, or fault isolation - The Supervisor pattern is the most production-ready orchestration approach for most teams - Always separate the orchestrator from worker agents. The orchestrator plans and delegates but never executes - Agent memory is the hardest unsolved problem. Use scoped context with summarization checkpoints - LangGraph leads for complex stateful workflows. CrewAI wins for fast prototyping. AutoGen has fallen behind - Most agent failures happen at handoff points between agents, not inside individual agents - Start with a single agent. Only move to multi-agent when you hit clear limits in capability or scale FAQ: - Q: What is a multi-agent AI system? A: A multi-agent AI system is a software architecture where multiple AI agents, each with a specific role and set of tools, work together to accomplish complex tasks. Instead of one large agent doing everything, specialized agents handle different parts of a workflow. For example, one agent might research a topic, another might write content, and a third might review and edit. The agents coordinate through an orchestration layer that manages communication, state, and task delegation. - Q: What are the main orchestration patterns for multi-agent systems? A: The five main patterns are: Supervisor (a coordinator agent delegates subtasks to specialists), Sequential Pipeline (agents process work in a linear chain), Peer-to-Peer Mesh (agents communicate directly without central control), Event-Driven (agents react to events on a message bus like Kafka), and Hub-and-Spoke (a central router connects independent agents). Most production systems use the Supervisor pattern because it provides the best balance of control, debuggability, and reliability. - Q: When should I use multi-agent instead of a single agent? A: Use multi-agent systems when your task requires different specialized skills that one agent cannot handle well, when you need parallel execution to reduce latency, when you need fault isolation so one failing component does not bring down everything, or when your context window is too small for a single agent to hold all relevant information. If a single agent with the right tools can solve your problem, stick with that. Multi-agent adds coordination overhead that is not always worth it. - Q: What is the difference between LangGraph CrewAI and AutoGen? A: LangGraph uses graph-based orchestration with explicit state management and checkpointing. It gives you fine-grained control and is best for complex stateful workflows. CrewAI uses role-based agent teams where you define agents with roles and goals that collaborate autonomously. It is faster to set up and best for rapid prototyping. AutoGen by Microsoft is built around agent conversations but has declined in production adoption because its auto speaker selection proved unpredictable in complex workflows. - Q: How do agents share memory and context in multi-agent systems? A: Agents share context through three main approaches: shared state stores where all agents read and write to a common data structure, message passing where agents send structured messages to each other through a bus or queue, and scoped context where the orchestrator passes only the relevant slice of context to each agent. The most reliable approach in production is scoped context managed by the orchestrator, because it prevents context pollution and keeps each agent focused on its specific task. - Q: What are common failure modes in multi-agent AI systems? A: The most common failures are: infinite loops where agents keep calling each other without making progress, context drift where agents gradually lose track of the original goal, budget explosion where agents make too many LLM calls and burn through API credits, handoff failures where information gets lost or corrupted when passing between agents, and conflicting actions where two agents try to modify the same resource simultaneously. These failures are mostly orchestration problems, not model capability problems. - Q: How do you make multi-agent systems production ready? A: Key production requirements include: running each agent in its own container with resource limits, implementing circuit breakers to stop cascading failures, adding human-in-the-loop approval for high-stakes actions, using deterministic workflow engines instead of fully autonomous agents, setting hard budget limits on API calls, implementing structured logging and distributed tracing for debugging, and building graceful degradation paths so the system falls back to simpler behavior when agents fail. ### Circuit Breaker Pattern Explained: The Complete Guide - URL: https://singhajit.com/circuit-breaker-pattern/ - Date: 2026-03-18 - Tags: system-design, distributed-systems - Description: What is the circuit breaker pattern? Learn how it prevents cascading failures in microservices with closed, open, and half-open states. Includes Resilience4j, Go, and Python examples, architecture diagrams, and real-world patterns from Netflix. FAQ: - Q: What is the circuit breaker pattern in microservices? A: The circuit breaker pattern is a fault tolerance design pattern that stops your application from repeatedly calling a failing service. Like an electrical circuit breaker, it trips when too many failures occur and blocks further requests. This prevents cascading failures where one slow or broken service brings down your entire system. It was popularized by Michael Nygard in his book Release It and later by Netflix through their Hystrix library. - Q: What are the three states of a circuit breaker? A: A circuit breaker has three states. Closed is the normal state where all requests pass through and failures are counted. Open is the tripped state where all requests fail immediately without calling the downstream service, giving it time to recover. Half-Open is the recovery test state where a limited number of requests are allowed through to check if the service is healthy again. If they succeed, the breaker closes. If they fail, it opens again. - Q: What is the difference between circuit breaker and retry pattern? A: The retry pattern re-executes a failed request hoping it will succeed on the next attempt. The circuit breaker pattern stops all requests to a failing service for a period of time. Retries handle transient failures like brief network glitches. Circuit breakers handle sustained failures where retrying would only make things worse. In practice, you use both together: retry for the first few failures, then trip the circuit breaker if failures persist. - Q: How does a circuit breaker prevent cascading failures? A: Without a circuit breaker, when Service B goes down, Service A keeps sending requests and waiting for responses that never come. This exhausts Service A's thread pool and connection pool, making it slow. Then Service C, which depends on Service A, also starts failing. One failure cascades through the entire system. A circuit breaker stops this chain by failing fast. The moment Service B is detected as unhealthy, the breaker opens and Service A returns an error or fallback response immediately. - Q: When should I use the circuit breaker pattern? A: Use circuit breakers whenever your service calls an external dependency that could fail or become slow. This includes calls to other microservices, third-party APIs, databases, and message brokers. Do not use circuit breakers for in-memory operations or local function calls where failures are not transient. The pattern is most valuable in distributed systems where network calls are inherently unreliable. - Q: What is the difference between circuit breaker and bulkhead pattern? A: The circuit breaker pattern monitors failure rates and stops requests to failing services. The bulkhead pattern isolates resources so one failing service does not consume all available threads or connections. Think of bulkhead as walls between ship compartments, preventing a leak in one section from flooding the whole ship. Circuit breaker detects the leak. Bulkhead contains it. Use both together for defense in depth. - Q: What are the best circuit breaker libraries? A: For Java and Spring Boot, Resilience4j is the standard choice since Netflix Hystrix entered maintenance mode in 2018. For Go, sony/gobreaker is the most widely used library. For Python, pybreaker provides a simple decorator-based API. In service mesh architectures like Istio, Envoy proxy handles circuit breaking at the infrastructure level without code changes. - Q: How do you configure circuit breaker thresholds? A: Start with these defaults: failure rate threshold of 50 percent, sliding window of 10 to 20 calls, minimum 5 calls before evaluation, wait duration in open state of 30 to 60 seconds, and 3 to 5 permitted calls in half-open state. Then tune based on your service's behavior. Critical services need lower thresholds and shorter timeouts. Services with naturally high error rates need higher thresholds to avoid false trips. ### Distributed Tracing: Jaeger vs Tempo vs Zipkin - URL: https://singhajit.com/distributed-tracing-jaeger-vs-tempo-vs-zipkin/ - Date: 2026-03-09 - Tags: devops, system-design, software-engineering - Description: Compare Jaeger, Grafana Tempo, and Zipkin for distributed tracing in microservices. Covers storage backends, sampling strategies, OpenTelemetry setup, cost at scale, and a practical guide to picking the right tool in 2026. Key Takeaways: - All three support OpenTelemetry. Your instrumentation code does not change when you switch backends. - Zipkin is the oldest and simplest. Good for learning and small projects, not for production scale. - Jaeger is a CNCF graduated project from Uber. It has the richest feature set for standalone use and adaptive tail sampling. - Grafana Tempo stores traces in object storage (S3, GCS, Azure Blob). It is dramatically cheaper per GB at scale than Jaeger with Elasticsearch. - Sampling strategy matters more than which backend you pick. Bad sampling means you miss the traces that actually matter. - Distributed tracing alone is not enough. You need logs and metrics alongside it to get the full picture. FAQ: - Q: What is distributed tracing? A: Distributed tracing tracks a single request as it travels through multiple services in a distributed system. Each service records a span, which is a unit of work with a name, timing data, and optional metadata. All spans from the same request share a trace ID, so you can reconstruct the full path of a request and see exactly where time was spent or errors occurred. - Q: What is the difference between Jaeger and Zipkin? A: Both are open-source distributed tracing systems, but Jaeger was built more recently (2015 by Uber), is written in Go, is a CNCF graduated project, and has features like adaptive sampling, a service dependency graph, and richer UI. Zipkin was created by Twitter in 2012, is written in Java, and is simpler to set up but lacks features for large-scale production use. - Q: What is Grafana Tempo and how is it different from Jaeger? A: Grafana Tempo is a distributed tracing backend from Grafana Labs released in 2020. The key difference from Jaeger is storage: Tempo stores traces in object storage like Amazon S3 or Google Cloud Storage instead of Elasticsearch or Cassandra. This makes Tempo significantly cheaper at high trace volumes. Tempo also integrates natively with Grafana, Loki, and Prometheus for correlated observability. - Q: Do I need OpenTelemetry for distributed tracing? A: You do not need it strictly, but you should use it. OpenTelemetry is the industry-standard observability framework that works with all major tracing backends including Jaeger, Tempo, and Zipkin. Using OpenTelemetry means your instrumentation code is vendor-neutral. You can switch from Jaeger to Tempo without rewriting any application code. - Q: What is trace sampling and why does it matter? A: Sampling is the decision of which requests to record full traces for. Recording every trace at production volume generates enormous storage costs. Head-based sampling decides at the start of a request (simple, low overhead, but misses slow tail events you have not seen yet). Tail-based sampling decides after the request completes (more accurate, captures errors and slow requests, but harder to implement). Jaeger supports adaptive tail sampling natively. Tempo supports tail sampling via the OpenTelemetry Collector. - Q: Is Jaeger free to use? A: Yes. Jaeger is fully open-source under the Apache 2.0 license. You run it yourself and pay for infrastructure: Elasticsearch or Cassandra for storage and compute for the Jaeger backend. Managed Jaeger is available on AWS, and via the Red Hat OpenShift distributed tracing platform. - Q: Can Zipkin handle production workloads? A: For small to medium applications, yes. Zipkin supports Cassandra and Elasticsearch as storage backends, which can handle significant volumes. But Zipkin lacks adaptive sampling, multi-tenancy, and the advanced querying capabilities of Jaeger and Tempo. Most teams outgrow Zipkin when tracing becomes central to debugging production. If you are starting fresh, Jaeger or Tempo is a better long-term choice. - Q: What is the Grafana LGTM stack? A: LGTM stands for Loki (logs), Grafana (dashboards), Tempo (traces), and Mimir or Prometheus (metrics). It is Grafana Labs' open-source observability stack. If your team already uses Grafana for dashboards and Loki for logs, Tempo is a natural fit because you get trace-to-log and trace-to-metric correlation out of the box in the same UI. ### Redis vs DragonflyDB vs KeyDB: Best Redis Alternative in 2026? - URL: https://singhajit.com/redis-vs-dragonflydb-vs-keydb/ - Date: 2026-03-05 - Tags: database, system-design, software-engineering - Description: Redis, DragonflyDB, or KeyDB? Compare architecture, real performance benchmarks, licensing, and when each one is the right choice for your stack in 2026. Key Takeaways: - Redis changed its license in 2024 from BSD to SSPL/RSALv2, then added AGPLv3 with Redis 8 in May 2025. AGPLv3 is open-source but has strong copyleft implications. Valkey remains the BSD-licensed alternative - DragonflyDB uses a shared-nothing multi-threaded architecture and can replace a Redis Cluster with a single node - KeyDB is a multithreaded Redis fork maintained by Snapchat. It is BSD-licensed and fully protocol-compatible with Redis - All three support the RESP protocol, so your existing Redis client code works with all of them without changes - DragonflyDB has gaps in Redis Module support (Search, JSON, TimeSeries) and Lua scripting edge cases. Check compatibility before migrating - For most teams already running Redis, the lowest-risk migration path is Valkey, not DragonflyDB or KeyDB FAQ: - Q: What is the difference between Redis and DragonflyDB? A: Redis uses a single-threaded event loop for command execution and requires Redis Cluster for horizontal scaling. DragonflyDB uses a multi-threaded shared-nothing architecture that scales across all CPU cores on a single machine. DragonflyDB reports 10-25x higher throughput than Redis on the same hardware and 2-4x better memory efficiency. DragonflyDB is API-compatible with Redis but does not support all Redis Modules or every Lua scripting edge case. - Q: Is DragonflyDB production ready in 2026? A: DragonflyDB reached general availability in late 2023 and has been adopted by engineering teams as a Redis replacement. It is not as battle-tested as Redis, which has been in production since 2009. Teams with mission-critical workloads should benchmark DragonflyDB on their specific use case and access patterns before replacing Redis in production. - Q: What is KeyDB and who maintains it? A: KeyDB is a multithreaded fork of Redis originally developed by EQ Alpha Technology. Snapchat acquired KeyDB in May 2022. It runs under the BSD-3-Clause license and is actively maintained on GitHub. KeyDB offers multi-master active replication, FLASH (SSD) storage support, and MVCC for non-blocking concurrent reads. - Q: Why did Redis change its license in 2024? A: Redis Ltd changed Redis from a BSD license to a dual SSPL/RSALv2 license in March 2024. The stated reason was that large cloud providers were offering managed Redis services without contributing back to the project. This triggered the creation of Valkey, a BSD-licensed community fork backed by AWS, Google Cloud, Oracle, and the Linux Foundation. Redis later added AGPLv3 as a third license option with Redis 8 in May 2025, returning it to OSI-approved open-source status, though the AGPLv3 copyleft terms remain a consideration for commercial products. - Q: Should I migrate from Redis to Valkey? A: If you are running open-source Redis and want to stay on a truly open-source codebase, Valkey is the most conservative migration. Valkey is a drop-in replacement with 100% API compatibility, maintained by the Linux Foundation with backing from AWS and Google Cloud. Most managed Redis offerings from AWS and Google have already switched to Valkey under the hood. - Q: Can DragonflyDB replace Redis Cluster? A: For many workloads, yes. DragonflyDB scales to 1TB on a single node and handles millions of operations per second, which would otherwise require a multi-node Redis Cluster. Eliminating the cluster removes hash slot restrictions, simplifies multi-key operations, and lowers infrastructure cost. However, DragonflyDB does not implement the Redis Cluster protocol, so test your client configuration before migrating. - Q: Which is faster: Redis or DragonflyDB? A: DragonflyDB is significantly faster on multi-core hardware. On a 32-core server, DragonflyDB achieves 2-4 million operations per second compared to Redis's 150-200K. Redis command execution is limited to one CPU core regardless of hardware. DragonflyDB reports p99 latency of 0.15ms versus Redis's 0.3ms. On single-core hardware, the gap is much smaller. - Q: Is KeyDB faster than Redis? A: Yes. KeyDB achieves over 1 million operations per second on a single node by using multithreaded command execution instead of Redis's single-threaded model. This is roughly 5-7x faster than Redis on multi-core hardware. KeyDB is generally slower than DragonflyDB, which was built from scratch rather than forked from Redis. - Q: What is the Redis SSPL license? A: SSPL (Server Side Public License) was created by MongoDB Inc and requires that any service offering SSPL-licensed software as a network service must also release the source code of the entire service, including infrastructure code. Redis moved to SSPL/RSALv2 in March 2024, which effectively prevented cloud providers from offering Redis as a managed service. Redis 8 (May 2025) added AGPLv3 as a third option, so users can now choose between AGPLv3 (OSI open-source, strong copyleft), RSALv2, or SSPLv1. ### When to Use PostgreSQL vs MongoDB vs DynamoDB (2026 Guide) - URL: https://singhajit.com/postgresql-vs-mongodb-vs-dynamodb/ - Date: 2026-03-04 - Tags: database, system-design, software-engineering - Description: Learn exactly when to use PostgreSQL, MongoDB, or DynamoDB. Covers ACID vs eventual consistency, horizontal scaling, cost at scale, real-world company choices, and common mistakes developers make when picking a database. Key Takeaways: - PostgreSQL is a safe default for most applications. Battle-tested, flexible, and scales further than most people think - MongoDB shines when your data has variable shape and you do not want to run schema migrations every week - DynamoDB requires you to know your access patterns before you design the schema. Get this wrong and you will regret it - All three can handle large scale. The difference is in how much operational effort it takes to get there - Cost at scale is not obvious. DynamoDB looks cheap until you add Global Secondary Indexes and on-demand mode - You can use more than one database in the same system. Most large applications do FAQ: - Q: PostgreSQL vs MongoDB: which is better in 2026? A: Neither is objectively better. PostgreSQL is better for structured relational data, complex queries, and ACID transactions. MongoDB is better for document-shaped data with variable schema. PostgreSQL has become much more flexible with JSONB support, so many use cases that previously needed MongoDB can now be handled by PostgreSQL. If you are unsure, start with PostgreSQL. - Q: When should I use DynamoDB instead of PostgreSQL? A: Use DynamoDB when you are fully committed to AWS, need serverless auto-scaling with zero operational overhead, and your access patterns are simple and predictable. DynamoDB handles unpredictable traffic spikes better than PostgreSQL out of the box. However, it requires careful upfront data modeling and becomes painful if you need ad hoc queries or complex joins. - Q: Is MongoDB faster than PostgreSQL? A: It depends on the workload. MongoDB can be faster for simple document reads when data is stored denormalized and no joins are needed. PostgreSQL is faster for complex queries with multiple joins and aggregations. For write-heavy workloads, PostgreSQL 17 handles around 19,000 inserts per second under concurrent load. Real-world performance depends heavily on schema design, indexes, and query patterns. - Q: Can DynamoDB replace PostgreSQL? A: For most traditional applications, no. DynamoDB does not support joins, ad hoc queries, or complex aggregations. It is designed for specific high-scale access patterns. If your app relies on complex reporting, cross-entity queries, or evolving business logic, DynamoDB will frustrate you. It is excellent at what it does, but what it does is narrow. - Q: How much does DynamoDB cost compared to PostgreSQL? A: DynamoDB cost depends on read/write capacity units and storage. At low throughput it can be very cheap. At scale, costs can become unpredictable, especially in on-demand mode. Managed PostgreSQL (RDS, Aurora, Supabase, Neon) has predictable monthly pricing. MongoDB Atlas charges per cluster size or serverless compute units. Always model your expected workload before choosing based on cost. - Q: What is DynamoDB single-table design? A: Single-table design is a DynamoDB pattern where multiple entity types (users, orders, products) are stored in a single table using composite partition and sort keys. You pre-compute relationships at write time instead of joining at query time. This enables fast, predictable reads but requires knowing your access patterns upfront. It is the recommended pattern for most DynamoDB applications. - Q: Can I use PostgreSQL as a document database? A: Yes. PostgreSQL has excellent JSONB support with indexing, querying, and aggregation on JSON fields. For many use cases that previously required MongoDB, PostgreSQL JSONB is a practical alternative. You get the flexibility of documents without sacrificing ACID guarantees or the ability to mix relational and document data in the same database. - Q: Which database is easiest to scale horizontally? A: DynamoDB scales horizontally automatically with no configuration. MongoDB has built-in sharding but requires planning. PostgreSQL horizontal scaling requires tools like Citus or Vitess, or sharding at the application level. For most applications, PostgreSQL with read replicas handles enormous traffic without horizontal sharding. ### How to Solve the Thundering Herd Problem in Distributed Systems - URL: https://singhajit.com/thundering-herd-problem/ - Date: 2026-02-19 - Tags: system-design - Description: What is the thundering herd problem? Learn how cache stampedes crash systems, 6 solutions used by Facebook, Twitter, and Netflix, and practical code examples for preventing thundering herd in distributed systems. Complete guide with architecture diagrams. FAQ: - Q: What is the thundering herd problem? A: The thundering herd problem happens when a large number of processes or threads wake up simultaneously in response to the same event, but only one can actually handle it. In web systems, this commonly occurs as a cache stampede: when a popular cache key expires, thousands of concurrent requests bypass the cache and hit the database at the same time, overwhelming it and causing timeouts or outages. - Q: What is a cache stampede? A: A cache stampede (also called dog pile effect or cache miss storm) is a specific form of the thundering herd problem. It happens when a cached value expires and many concurrent requests discover the miss simultaneously. Instead of one request refreshing the cache, hundreds or thousands of requests all query the database at the same time, causing extreme load spikes. - Q: What is request coalescing and how does it prevent thundering herd? A: Request coalescing (also called request collapsing or the singleflight pattern) groups identical concurrent requests and executes only one backend call. When multiple requests arrive for the same cache key during a miss, only the first request hits the database. All other requests wait for that single result and share it. This eliminates duplicate work and prevents database overload. - Q: How does adding jitter prevent cache stampede? A: Jitter adds randomness to cache TTL values and retry intervals. Instead of all cache keys expiring at the same moment (like exactly 1 hour), each key gets a slightly different TTL (55 to 65 minutes). This spreads expirations over time so the database handles a steady stream of individual refreshes rather than a sudden spike. For retries, jitter prevents all failed clients from retrying at the same instant. - Q: How did Facebook solve the thundering herd problem? A: Facebook introduced a lease mechanism in their Memcached infrastructure. When a cache miss occurs, the first request gets a short-lived lease (token). Only the request holding the lease can set the cache value. Other requests that see the miss either wait briefly and retry or receive a stale cached value. This prevents thousands of requests from simultaneously hitting the database for the same key. - Q: What is the difference between thundering herd and hot key problem? A: The thundering herd problem is about synchronized cache misses, where many requests discover a cache miss at the same time and all query the backend simultaneously. The hot key problem is about a single cache key receiving too many reads, overwhelming the cache server itself. They often appear together but require different solutions. Hot keys need replication across cache nodes; thundering herd needs request coalescing or locking. - Q: What is exponential backoff with jitter? A: Exponential backoff with jitter is a retry strategy where each retry waits exponentially longer (1s, 2s, 4s, 8s) plus a random delay. The exponential part prevents rapid retries from overwhelming the system. The jitter (random component) prevents all clients from retrying at the same moment even with backoff. AWS recommends this pattern for all SDK retries. - Q: How do you prevent thundering herd in Redis? A: Prevent thundering herd in Redis using distributed locking (SET with NX and EX flags), request coalescing at the application layer, staggered TTLs with jitter, or probabilistic early recomputation. The most common approach combines a Redis lock (only one request refreshes) with stale data fallback (other requests get the old value while refresh happens). ### How Consistent Hashing Works - URL: https://singhajit.com/consistent-hashing-explained/ - Date: 2026-02-13 - Tags: system-design - Description: What is consistent hashing and how does it work? Learn how consistent hashing distributes data across servers, handles scaling with virtual nodes, and powers systems like DynamoDB, Cassandra, and Memcached. A practical guide for system design with diagrams and code. FAQ: - Q: What is consistent hashing? A: Consistent hashing is a distributed hashing technique that maps both servers and data keys onto a circular hash space called a ring. Each key is assigned to the nearest server in the clockwise direction. When servers are added or removed, only a small fraction of keys need to be remapped, unlike traditional hashing where almost everything moves. This makes it ideal for distributed caches, databases, and load balancers. - Q: How does consistent hashing work in system design? A: In system design, consistent hashing solves the problem of distributing data across multiple servers that can scale up or down. Both servers and keys are hashed onto a ring of size 0 to 2^32-1. Keys are assigned to the next server clockwise on the ring. When a server joins, it takes over a portion of keys from its clockwise neighbor. When a server leaves, its keys move to the next server clockwise. This minimizes data movement during scaling. - Q: What is the difference between consistent hashing and regular hashing? A: Regular hashing uses hash(key) mod N where N is the number of servers. If N changes, nearly every key maps to a different server, causing massive data shuffling. Consistent hashing maps keys and servers onto a ring, so adding or removing a server only affects keys in the segment between the changed server and its predecessor. On average, only K/N keys move instead of almost all keys. - Q: What are virtual nodes in consistent hashing? A: Virtual nodes (vnodes) are multiple positions on the hash ring assigned to each physical server. Instead of one position, a server might have 100 or 200 virtual positions spread across the ring. This solves uneven distribution that happens with few servers and ensures data is spread more uniformly. Cassandra uses 256 virtual nodes per server by default. - Q: Where is consistent hashing used in real systems? A: Consistent hashing is used in Amazon DynamoDB for data partitioning, Apache Cassandra for distributing data across nodes, Memcached for distributed caching, Akamai CDN for content distribution, Discord for routing messages, and load balancers like Nginx and HAProxy for sticky sessions. It is a foundational technique in any system that needs to distribute data across servers that can scale. - Q: How many keys move when a server is added in consistent hashing? A: When a new server is added to a consistent hashing ring, only K/N keys need to move on average, where K is the total number of keys and N is the new number of servers. For example, with 1 million keys and 10 servers, adding an 11th server moves roughly 91,000 keys (1/11 of total). With traditional hashing, nearly all 1 million keys would need to move. - Q: What hash functions are used in consistent hashing? A: Common hash functions for consistent hashing include MD5, SHA-1, SHA-256, MurmurHash, and xxHash. The hash function should distribute values uniformly across the ring. MurmurHash is popular in production systems because it is fast and has good distribution. Cassandra uses Murmur3, and Memcached uses MD5 by default. - Q: How does consistent hashing handle server failures? A: When a server fails in a consistent hashing ring, all keys assigned to that server automatically fall through to the next server clockwise on the ring. No rehashing of other keys is needed. Combined with replication (storing copies on the next N servers clockwise), consistent hashing provides fault tolerance. The gossip protocol is often used alongside to detect server failures. ### How Netflix Video Processing Pipeline Works - URL: https://singhajit.com/netflix-video-processing-pipeline/ - Date: 2026-02-03 - Tags: system-design, distributed-systems - Description: Deep dive into Netflix video encoding pipeline architecture. Learn how Netflix uses microservices, parallel processing, VMAF quality metrics, and the Cosmos platform to process thousands of video titles at scale. Practical lessons for software developers. Key Takeaways: - Netflix rebuilt their monolithic video pipeline into microservices running on the Cosmos platform - Long encoding jobs are split into chunks and processed in parallel across hundreds of EC2 instances - VMAF (Video Multimethod Assessment Fusion) is used to measure perceptual video quality at scale - Each title gets a custom encoding ladder optimized for its specific content characteristics - The pipeline scales elastically on AWS, spinning up for new releases and scaling down when idle FAQ: - Q: How does Netflix encode videos for streaming? A: Netflix uses a microservices-based video processing pipeline running on AWS EC2. Source files are ingested, inspected for quality, split into chunks, and encoded in parallel. Each chunk is processed by the Video Encoding Service (VES) built on their Cosmos platform. The encoded segments are then assembled into multiple quality levels for adaptive streaming. - Q: What is Netflix VMAF and why is it important? A: VMAF (Video Multimethod Assessment Fusion) is Netflix's open-source perceptual quality metric. It predicts how humans perceive video quality better than traditional metrics like PSNR. Netflix uses VMAF to optimize encoding decisions, compare codecs, and ensure consistent quality across their entire catalog. - Q: What is per-title encoding in Netflix? A: Per-title encoding means each movie or show gets a custom encoding ladder based on its content complexity. An animated cartoon needs less bitrate than a fast-action movie to achieve the same perceived quality. Netflix analyzes each title and creates optimal bitrate-resolution pairs, saving bandwidth while maintaining quality. - Q: How does Netflix handle video quality at scale? A: Netflix processes thousands of titles using parallel encoding. Long jobs are split into small chunks that run across hundreds of EC2 instances. Quality is measured at each step using VMAF. The elastic cloud infrastructure scales up for new releases and scales down during quiet periods. - Q: What is Netflix Open Connect? A: Open Connect is Netflix's content delivery network (CDN). It consists of servers placed inside ISP networks worldwide. After videos are encoded by the processing pipeline, they are distributed to Open Connect appliances. When you stream, content comes from a server near you, not from Netflix's main data centers. - Q: How long does it take Netflix to process a new movie? A: Processing time varies by content length and complexity. A 2-hour movie typically takes several hours to fully encode into all quality variants. Netflix parallelizes the work across many machines to reduce wall-clock time. Priority content like new releases gets dedicated resources for faster turnaround. ### How OpenAI Scales PostgreSQL to 800 Million Users - URL: https://singhajit.com/how-openai-scales-postgresql/ - Date: 2026-02-01 - Tags: database, system-design - Description: Learn how OpenAI scales PostgreSQL to handle 800 million ChatGPT users. Deep dive into connection pooling with PgBouncer, read replicas, horizontal sharding, query optimization, and database architecture patterns used at massive scale. Key Takeaways: - Connection pooling with PgBouncer lets thousands of app instances share a smaller pool of database connections - Read replicas handle read-heavy workloads while the primary handles writes - Horizontal sharding distributes data across multiple PostgreSQL instances by user ID or tenant - Query optimization and proper indexing matter more at scale than they do on small datasets - Monitoring query patterns and slow queries is essential for identifying scaling bottlenecks FAQ: - Q: How does OpenAI scale PostgreSQL for 800 million ChatGPT users? A: OpenAI uses several techniques to scale PostgreSQL: connection pooling with PgBouncer to reduce connection overhead, read replicas to distribute read traffic, horizontal sharding to partition data across multiple database instances, aggressive query optimization, and proper indexing strategies. These techniques together allow PostgreSQL to handle the massive concurrent load from ChatGPT. - Q: What is connection pooling and why is it important for database scaling? A: Connection pooling maintains a pool of reusable database connections instead of creating new connections for each request. PostgreSQL connections are expensive (each uses about 10MB of memory). PgBouncer sits between your application and PostgreSQL, letting thousands of app instances share hundreds of actual database connections. This dramatically reduces database memory usage and connection overhead. - Q: What are read replicas and when should you use them? A: Read replicas are copies of your primary database that handle read queries. Writes go to the primary and replicate to the replicas. Use read replicas when your workload is read-heavy (most applications are 80-90% reads). This distributes the read load across multiple servers while the primary focuses on writes. - Q: What is horizontal sharding in PostgreSQL? A: Horizontal sharding splits your data across multiple PostgreSQL instances based on a shard key (like user_id). Each shard contains a subset of the data. This distributes both the data size and query load. For example, users 1-1M go to shard 1, users 1M-2M go to shard 2, and so on. Each shard is a complete PostgreSQL instance. - Q: How do you handle database migrations at scale? A: At scale, you cannot run blocking migrations. Use online schema change tools like pg_online_schema_change or pt-online-schema-change. Add new columns as nullable first, backfill data in batches, then add constraints. Always test migrations on production-sized datasets before running them in production. - Q: What are the signs that you need to scale your PostgreSQL database? A: Key signs include: connection limits being hit frequently, query response times increasing, CPU or memory consistently high on the database server, replication lag growing, and autovacuum struggling to keep up. Monitor these metrics proactively to scale before problems impact users. ### System Design Cheat Sheet: Concepts Every Developer Should Know - URL: https://singhajit.com/system-design-cheat-sheet/ - Date: 2026-01-31 - Tags: system-design, architecture - Description: A practical system design cheat sheet covering scalability, load balancing, caching, database sharding, CAP theorem, and distributed systems patterns. Essential concepts for building systems that scale and preparing for system design interviews. Key Takeaways: - Start every design with requirements. Functional requirements define what the system does, non-functional requirements define how well it does it - Horizontal scaling beats vertical scaling for most production systems. Add more machines instead of bigger machines - Caching is not optional at scale. Know your cache patterns and when to use each one - Database choice matters. SQL for transactions and complex queries, NoSQL for scale and flexibility - Design for failure. Every component will fail eventually. Build redundancy and graceful degradation into your system FAQ: - Q: What is system design and why is it important? A: System design is the process of defining the architecture, components, and data flow of a system to meet specific requirements. It matters because poorly designed systems fail under load, cost too much to run, and become impossible to maintain. Good system design ensures your application scales with users, stays reliable, and remains easy to evolve. - Q: What is the difference between horizontal and vertical scaling? A: Vertical scaling means adding more power to existing machines (more CPU, RAM, storage). Horizontal scaling means adding more machines. Vertical scaling is simpler but has limits and creates single points of failure. Horizontal scaling is more complex but provides better fault tolerance and theoretically unlimited capacity. Most production systems use horizontal scaling. - Q: What is the CAP theorem? A: The CAP theorem states that a distributed system can only guarantee two of three properties: Consistency (all nodes see the same data), Availability (every request gets a response), and Partition Tolerance (system works despite network failures). Since network partitions are inevitable, you must choose between consistency and availability during failures. - Q: When should I use SQL vs NoSQL databases? A: Use SQL databases when you need ACID transactions, complex joins, or data integrity is critical (financial systems, inventory). Use NoSQL when you need horizontal scaling, flexible schemas, or high write throughput (user sessions, social feeds, logs). Many systems use both: SQL for transactions, NoSQL for caching and analytics. - Q: What is database sharding and when should I use it? A: Sharding splits your database across multiple servers, with each shard holding a portion of the data. Use sharding when a single database cannot handle your read or write load, your data exceeds single server storage, or you need geographic distribution. Common sharding strategies include hash-based, range-based, and geographic sharding. - Q: What is a load balancer and how does it work? A: A load balancer distributes incoming traffic across multiple servers to prevent any single server from being overwhelmed. It improves availability (if one server fails, traffic goes to others) and enables horizontal scaling. Common algorithms include round-robin, least connections, and IP hash. Popular options include Nginx, HAProxy, and cloud load balancers. - Q: What caching strategies should I know for system design? A: The main caching strategies are: Cache-Aside (application manages cache, most common), Read-Through (cache fetches from database on miss), Write-Through (writes go to cache and database synchronously), Write-Behind (writes go to cache first, database later), and Write-Around (writes bypass cache). Choose based on your read/write patterns and consistency requirements. - Q: How do I estimate capacity for system design? A: Start with user estimates: daily active users, requests per user, data per request. Calculate reads/writes per second, storage needs, and bandwidth. Use round numbers and powers of 10. For example: 10 million users, 10 requests/day each = 100 million requests/day = about 1,200 requests/second. Always design for 3-5x your expected peak load. ### How Google Docs Works Behind the Scenes - URL: https://singhajit.com/how-google-docs-works/ - Date: 2026-01-29 - Tags: system-design - Description: How does Google Docs let multiple people edit the same document at once? Learn the system design behind real-time collaboration, conflict resolution, and instant syncing across users. FAQ: - Q: How does Google Docs handle multiple users editing at the same time? A: Google Docs uses Operational Transform (OT) to handle concurrent edits. Each keystroke is converted into an operation with a position and content. When two users edit simultaneously, the server receives both operations, transforms them to account for each other's changes, and broadcasts the transformed operations back. This ensures all users see the same final document without losing any edits. - Q: What is Operational Transform (OT)? A: Operational Transform is an algorithm that resolves conflicts in collaborative editing. It transforms operations based on other concurrent operations. For example, if Alice inserts at position 5 and Bob deletes at position 3, Bob's operation shifts Alice's insert position to 4. The transformation ensures both edits are applied correctly regardless of the order they arrive. - Q: What is the difference between OT and CRDT for collaborative editing? A: OT (Operational Transform) requires a central server to order and transform operations. It's simpler to implement but has a single point of coordination. CRDT (Conflict-Free Replicated Data Types) allows each client to apply edits independently and merge them later without a central coordinator. CRDTs work better for offline-first applications but use more memory. Google Docs uses OT while tools like Figma use CRDTs. - Q: How does Google Docs sync changes in real-time? A: Google Docs maintains persistent WebSocket connections between each user's browser and Google's servers. When you type, your browser sends the operation to the server within milliseconds. The server applies the operation, transforms it against any concurrent edits, and broadcasts it to all other connected users. The entire round trip typically takes 50-200 milliseconds. - Q: How does Google Docs handle offline editing? A: Google Docs stores a local copy of the document and queues your edits when offline. Operations are saved with timestamps. When you reconnect, the client sends all queued operations to the server. The server applies them using OT to merge your offline changes with any edits others made while you were disconnected. - Q: How does cursor position syncing work in Google Docs? A: Each user's cursor position is tracked as an index in the document. When you move your cursor or select text, your browser sends the position to the server, which broadcasts it to other users. When edits happen before your cursor position, the server transforms your cursor position just like it transforms text operations, keeping the cursor in the right place relative to the text. - Q: How does Google Docs store version history? A: Google Docs stores document versions using a combination of snapshots and operation logs. Periodically, the system saves a complete snapshot of the document. Between snapshots, it logs all operations. To restore any version, the system loads the nearest snapshot and replays operations. This is more storage-efficient than saving full copies of every version. - Q: What database does Google Docs use? A: Google Docs uses Google's internal distributed storage systems. Documents are stored in Spanner (Google's globally distributed database) or Bigtable for high availability. The actual document structure is stored in a format optimized for operational transform, with indexes for fast access to any position in the document. ### How Meta Handles Millions of Serverless Function Calls Per Second - URL: https://singhajit.com/meta-xfaas-serverless-at-scale/ - Date: 2026-01-24 - Tags: system-design, distributed-systems - Description: Deep dive into Meta's XFaaS serverless platform. Learn how they handle 11.5 million function calls per second with 66% CPU utilization. Practical lessons on cold start elimination, load distribution, congestion control, and building serverless systems at scale. Key Takeaways: - Meta's XFaaS processes 11.5 million function calls per second across 100,000+ servers - They achieve 66% average CPU utilization, far higher than typical cloud FaaS platforms - Cold start elimination through universal workers is the key to consistent performance - Load is spread across both time (defer to off-peak) and space (route to other datacenters) - XFaaS is only used for non-user-facing functions due to variable latency in serverless FAQ: - Q: What is Meta XFaaS? A: XFaaS is Meta's internal serverless platform that processes trillions of function calls per day. It runs on over 100,000 servers across dozens of datacenter regions and achieves 66% average CPU utilization, significantly higher than public cloud FaaS offerings like AWS Lambda. - Q: How many serverless function calls does Meta handle per second? A: At peak capacity, Meta's XFaaS handles approximately 11.5 million function calls per second. During traffic spikes, they can receive 20 million function calls within just 15 minutes, with peak demand reaching 4.3 times higher than off-peak demand. - Q: How does Meta eliminate serverless cold starts? A: Meta uses a universal worker approach where any worker can execute any function immediately without startup overhead. Workers are pre-warmed and kept ready to handle any function type, eliminating the variable latency caused by cold starts. - Q: What is the XFaaS architecture? A: XFaaS has five main components: Submitter (entry point with quota management), Queue Load Balancers (distributes load), DurableQ (persistent storage), Scheduler (orders function calls by priority), and Worker Pool (executes functions across 100,000+ servers). - Q: Does Meta use XFaaS for user-facing features? A: No. Meta explicitly uses XFaaS only for non-user-facing functions like notifications and thumbnail generation. Serverless functions have too much variable latency for consistent user-facing performance, so they keep customer-facing code on more predictable infrastructure. ### X Algorithm Explained: How the Open Source Recommendation System Works - URL: https://singhajit.com/system-design/x-twitter-for-you-algorithm/ - Date: 2026-01-22 - Tags: system-design, machine-learning, software-engineering - Description: The X algorithm explained — a detailed breakdown of how Twitter's open source recommendation algorithm works in 2026. Covers the full GitHub repository (xai-org/x-algorithm), the Grok-based transformer ranking model, Two-Tower retrieval, candidate pipeline architecture, scoring weights, and practical lessons for building recommendation systems at scale. Key Takeaways: - X's recommendation algorithm is fully open source on GitHub (xai-org/x-algorithm), written in Rust (62.9%) and Python (37.1%) - The algorithm uses a multi-stage pipeline: candidate sourcing (Thunder + Phoenix Two-Tower), ranking (Grok-based transformer), and filtering - 500 million daily posts are narrowed to ~1,500 candidates per user, ranked in under 200 milliseconds - Negative signals (block, mute, report) carry far more weight than positive ones (like, reply) — a single block is -3.0 vs. a like at +0.5 - Candidate isolation in the Grok transformer ensures each post's score is independent, enabling aggressive caching and consistent rankings - X eliminated all hand-engineered features and heuristics, letting the transformer model learn patterns directly from engagement data FAQ: - Q: How does the X For You algorithm work? A: The X For You algorithm uses a three-stage pipeline: candidate sourcing (fetching posts from accounts you follow and ML-discovered content), ranking (scoring each post using a Grok-based transformer model), and filtering (removing duplicates, blocked content, and applying diversity rules). The system processes 500 million daily posts to create a personalized feed of around 1,500 candidates. - Q: What programming languages does X use for its recommendation algorithm? A: X's algorithm is primarily written in Rust (62.9%) and Python (37.1%). Rust handles the high-performance serving infrastructure including the Thunder post store and candidate pipeline, while Python is used for machine learning model training and the Phoenix transformer model. - Q: What is the Grok-based ranking model in X's algorithm? A: Phoenix is X's Grok-based transformer model adapted for recommendations. It predicts engagement probabilities for multiple actions (like, reply, repost, click, block, mute) simultaneously. Unlike traditional models, it uses candidate isolation during inference so each post's score is independent of other posts in the batch. - Q: How does X find out-of-network content for the For You feed? A: X uses a Two-Tower retrieval model in Phoenix. A User Tower encodes your engagement history into an embedding, while a Candidate Tower encodes all posts. The system finds relevant out-of-network content by computing dot product similarity between your user embedding and post embeddings, retrieving the top-K most similar posts. - Q: What signals does the X algorithm use for ranking posts? A: The algorithm predicts probabilities for positive actions (favorite, reply, repost, quote, click, video_view, share, dwell time, follow_author) and negative actions (not_interested, block_author, mute_author, report). The final score combines these predictions with different weights, where negative signals heavily penalize content. - Q: Is the X algorithm open source on GitHub? A: Yes. X (formerly Twitter) published the full source code of their recommendation algorithm on GitHub at xai-org/x-algorithm. The repository includes the complete production pipeline: Thunder (in-memory post store), Phoenix (ML retrieval and ranking), and the Candidate Pipeline framework. The codebase is 62.9% Rust and 37.1% Python. - Q: What changed in the X recommendation algorithm in 2026? A: The 2026 version of X's algorithm replaced earlier hand-engineered features and heuristics with a Grok-based transformer model called Phoenix. There are no longer hard-coded boosts for verified accounts, media types, or trending topics. The system also introduced candidate isolation in the ranking model, making scores independent and cacheable. The full source code is available on GitHub at xai-org/x-algorithm, written primarily in Rust and Python. ### How Snowflake IDs Work - URL: https://singhajit.com/snowflake-id-guide/ - Date: 2026-01-14 - Tags: system-design - Description: Learn how Snowflake IDs work, their 64-bit structure, and how to implement them in Java. Understand Discord's snowflake ID length, Twitter's timestamp bits, and why companies choose Snowflake over UUID for distributed systems. FAQ: - Q: What is a Snowflake ID? A: A Snowflake ID is a 64-bit unique identifier used in distributed systems. It combines a 41-bit timestamp, 10-bit machine ID, and 12-bit sequence number. This structure allows multiple servers to generate unique IDs independently without coordination, while keeping IDs time-sortable. Twitter created this approach in 2010. - Q: What is the Discord Snowflake ID length in digits? A: Discord Snowflake IDs are 64-bit integers, which means they can be up to 19 digits long when represented as decimal numbers. For example, a Discord user ID like 123456789012345678 is 18 digits. The exact length varies based on when the ID was generated since older IDs have smaller timestamps. - Q: How do I extract the timestamp from a Twitter Snowflake ID? A: To extract the timestamp from a Twitter Snowflake ID, right-shift the ID by 22 bits to get the milliseconds since Twitter's epoch (1288834974657). Then add Twitter's epoch to get the Unix timestamp. In JavaScript: new Date((snowflakeId >> 22n) + 1288834974657n). Discord uses epoch 1420070400000 (January 1, 2015). - Q: Why use Snowflake IDs instead of UUIDs? A: Snowflake IDs are smaller (64-bit vs 128-bit), time-sortable for chronological queries, and more efficient as database primary keys. UUIDs are random, causing poor B-tree index performance and page splits. Snowflake IDs maintain locality of reference, making database writes faster. - Q: How do I generate Snowflake IDs in Java? A: For Java, use libraries like callicoder/java-snowflake or phxql/snowflake-id. These handle timestamp generation, machine ID assignment, and sequence number management. Example: SnowflakeIdGenerator generator = new SnowflakeIdGenerator(machineId); long id = generator.nextId(); ### CQRS Pattern: Splitting Read and Write Models - URL: https://singhajit.com/cqrs-pattern-guide/ - Date: 2025-12-23 - Tags: architecture, system-design - Description: Learn the CQRS pattern with practical examples. Understand when to use Command Query Responsibility Segregation, see real implementation code, and avoid common mistakes developers make. FAQ: - Q: What is the difference between CQRS and CRUD? A: CRUD uses a single model for all operations (Create, Read, Update, Delete). CQRS separates the model into two parts: Commands for writes and Queries for reads. This separation allows you to optimize each side independently for its specific workload. - Q: When should I use CQRS? A: Use CQRS when your read and write workloads are significantly different, when complex queries are slowing down write operations, when you need to scale reads and writes independently, or when working with event sourcing. Avoid it for simple CRUD applications where the added complexity is not justified. - Q: Does CQRS require two databases? A: No, CQRS does not require two databases. You can start with logical separation using different models within the same database. Physical separation with multiple databases is optional and should be added only when you need independent scaling or different storage technologies for reads and writes. - Q: What is the relationship between CQRS and Event Sourcing? A: CQRS and Event Sourcing are separate patterns that work well together. CQRS separates read and write models, while Event Sourcing stores all changes as a sequence of events. You can use CQRS without Event Sourcing and vice versa, but combining them provides benefits like complete audit trails and the ability to rebuild read models from event history. - Q: What are the main challenges of implementing CQRS? A: The main challenges include handling eventual consistency between read and write models, increased complexity from maintaining two models, and the learning curve for teams new to the pattern. Start with logical separation before adding physical separation to manage complexity. ### How Google Ads Supports 4.8 Billion Users with a SQL Database - URL: https://singhajit.com/how-google-ads-scales-with-spanner/ - Date: 2025-12-19 - Tags: system-design - Description: Deep dive into Google Spanner architecture. Learn how Google Ads handles 4.8 billion users with a globally distributed SQL database. Covers TrueTime, Paxos, automatic sharding, and practical lessons for building scalable systems. FAQ: - Q: What is Google Spanner? A: Google Spanner is a globally distributed SQL database that provides strong consistency, horizontal scaling, and full ACID transactions across data centers worldwide. It powers Google Ads, serving 4.8 billion users. Unlike traditional databases, Spanner offers both SQL flexibility and NoSQL scale. - Q: How does Google Spanner achieve global consistency? A: Spanner uses TrueTime, a system of atomic clocks and GPS receivers in every data center that provides accurate timestamps with known uncertainty bounds. Combined with Paxos consensus for every write, this allows Spanner to order transactions globally and provide strong consistency across continents. - Q: What is TrueTime in Google Spanner? A: TrueTime is Google's globally synchronized clock system. Unlike regular clocks, TrueTime returns a time interval [earliest, latest] with guaranteed bounds. Spanner uses this to ensure transactions are ordered correctly across all replicas worldwide, enabling strong consistency without sacrificing performance. - Q: How does Spanner compare to traditional sharded databases? A: Traditional sharded databases require manual shard management, complex cross-shard transaction logic, and often sacrifice SQL flexibility. Spanner handles sharding automatically, supports full SQL with joins and secondary indexes, and provides ACID transactions across shards without application complexity. ### Role of Queues in System Design - URL: https://singhajit.com/role-of-queues-in-system-design/ - Date: 2025-12-17 - Tags: system-design - Description: Deep dive into message queues in system design. Learn when and why to use queues, popular queue technologies like RabbitMQ, Kafka, and SQS, and real-world patterns from Uber, Slack, and Stripe. Practical guide with architecture diagrams and code examples. FAQ: - Q: Why use message queues in system design? A: Message queues decouple services, enabling asynchronous processing. They absorb traffic spikes (buffering requests during peaks), improve resilience (failed consumers don't crash producers), reduce latency (non-critical work happens in background), and enable horizontal scaling (add more consumers to process faster). - Q: When should I use Kafka vs RabbitMQ vs SQS? A: Use Kafka for high-throughput event streaming, log aggregation, and when you need message replay. Use RabbitMQ for complex routing, request-reply patterns, and traditional task queues. Use SQS for simple AWS-native queuing with minimal operational overhead. Kafka handles millions of messages/second; RabbitMQ excels at routing flexibility; SQS wins on simplicity. - Q: What is a dead letter queue? A: A dead letter queue (DLQ) stores messages that fail processing after multiple retry attempts. Instead of losing failed messages or blocking the queue, they're moved to the DLQ for inspection and manual handling. This prevents poison messages from blocking healthy processing while preserving failed messages for debugging. - Q: What is the difference between pub/sub and point-to-point queuing? A: In point-to-point queuing, each message is consumed by exactly one consumer (like a task queue). In pub/sub (publish-subscribe), messages are broadcast to all subscribers. Use point-to-point for work distribution; use pub/sub for events that multiple services need to react to independently. ### How Amazon S3 Stores 100 Trillion Objects Without Losing One - URL: https://singhajit.com/how-amazon-s3-works/ - Date: 2025-12-10 - Tags: system-design - Description: Deep dive into Amazon S3 architecture. Learn how S3 achieves 11 nines durability, handles massive scale, and why understanding it makes you a better developer. Includes practical examples, diagrams, and real-world insights. FAQ: - Q: How does Amazon S3 achieve 11 nines durability? A: S3 achieves 99.999999999% durability by storing each object redundantly across multiple devices in multiple Availability Zones within a region. Data is checksummed, automatically repaired if corruption is detected, and replicated to maintain redundancy even when hardware fails. This means losing one object out of 10 million would take about 10,000 years. - Q: What is the difference between S3 storage classes? A: S3 Standard is for frequently accessed data with millisecond access. S3 Intelligent-Tiering automatically moves data between tiers based on access patterns. S3 Glacier is for archives with retrieval times from minutes to hours. S3 Glacier Deep Archive is the cheapest, for data accessed once or twice a year with 12-hour retrieval. - Q: How does S3 handle large file uploads? A: S3 uses multipart upload for files larger than 100MB. The file is split into parts (5MB-5GB each), uploaded in parallel, and S3 assembles them. If a part fails, only that part needs re-uploading. This enables resumable uploads, parallel transfers, and handling files up to 5TB. - Q: Is Amazon S3 strongly consistent? A: Yes, since December 2020, S3 provides strong read-after-write consistency for all operations. When you PUT an object, subsequent GET requests immediately return the new data. This applies to new objects, overwrites, and deletes. No more eventual consistency surprises. ### Modular Monolith: The Architecture Most Teams Actually Need - URL: https://singhajit.com/modular-monolith-architecture/ - Date: 2025-12-04 - Tags: system-design - Description: Learn how modular monolith architecture combines the simplicity of monoliths with the organization of microservices. Understand module boundaries, communication patterns, and when to choose this architecture over microservices. FAQ: - Q: What is a modular monolith? A: A modular monolith is a single deployable application divided into well-defined modules with clear boundaries. Each module owns its domain and communicates through explicit interfaces. It combines the simplicity of monoliths (single deployment, no network calls between modules) with the organization of microservices (clear boundaries, team autonomy). - Q: When should I choose a modular monolith over microservices? A: Choose a modular monolith when you have a small-to-medium team (under 50 developers), don't need independent scaling of components, want simpler debugging and deployment, or are building an MVP. Microservices add network latency, distributed debugging complexity, and operational overhead that often outweighs benefits for smaller teams. - Q: How do modules communicate in a modular monolith? A: Modules communicate through explicit public interfaces (APIs), not by directly accessing each other's internals. Options include: direct method calls through interfaces, in-process events/mediator pattern, or shared contracts. The key is that modules can't reach into each other's database tables or private classes. - Q: Can you migrate from a modular monolith to microservices? A: Yes, and this is a key advantage. Because modules already have clear boundaries and communicate through interfaces, extracting a module into a microservice is straightforward. You replace in-process calls with network calls. Companies like Shopify use this approach - starting modular and extracting services only when needed. ### The Complete Guide to Server-Sent Events (SSE) - URL: https://singhajit.com/server-sent-events-explained/ - Date: 2025-12-03 - Tags: system-design - Description: What is SSE? Server-Sent Events is a web standard for real-time server-to-client streaming over HTTP. Learn about the EventSource API, retry field default 3000 ms per MDN and the HTML specification, auto-reconnection behavior, Last-Event-ID, and when to choose SSE over WebSockets. FAQ: - Q: What is SSE (Server-Sent Events)? A: SSE (Server-Sent Events) is a web technology that allows servers to push real-time updates to browsers over a single HTTP connection. Unlike WebSockets, SSE is one-way (server to client only) and uses the text/event-stream format. It's ideal for live dashboards, notifications, stock prices, and any scenario where the server needs to push data without the client requesting it. - Q: What is the default EventSource retry time? A: The default EventSource retry time is 3000 milliseconds (3 seconds). When a connection drops, the browser waits 3 seconds before attempting to reconnect. Servers can override this by sending a retry field with a different value in milliseconds. - Q: What is the default EventSource reconnection time according to the HTML standard? A: According to the HTML Living Standard, the default reconnection time for EventSource is approximately 3 seconds (3000ms). The server can modify this by sending a retry field with the desired milliseconds value. - Q: How do I change the EventSource retry interval? A: The server can change the retry interval by sending a retry field in the event stream. For example, sending 'retry: 5000' sets the reconnection time to 5 seconds. The client cannot directly set this value. - Q: Does EventSource automatically reconnect? A: Yes, EventSource automatically reconnects when the connection drops. The browser handles reconnection without any code needed, waiting the retry time (default 3000ms) before attempting to reconnect. - Q: What is the EventSource reconnection behavior according to MDN? A: According to MDN (Mozilla Developer Network), EventSource automatically attempts to reconnect when the connection is closed. The default retry interval is 3000 milliseconds (3 seconds). The server can customize this by sending a retry field in the event stream. On reconnection, the browser sends the Last-Event-ID header to enable message replay. - Q: What does the server-sent events specification say about the retry field default? A: The HTML Living Standard specification for server-sent events states that the default reconnection time is approximately 3000 milliseconds (3 seconds). The retry field in the event stream format allows servers to override this default by sending a new value in milliseconds (e.g., 'retry: 5000' for 5 seconds). ### Long Polling Explained: Build Real-Time Apps Without WebSockets - URL: https://singhajit.com/long-polling-explained/ - Date: 2025-12-02 - Tags: system-design - Description: Learn how Long Polling enables real-time communication using plain HTTP. Understand the implementation, trade-offs, and when to choose Long Polling over WebSockets or Server-Sent Events. FAQ: - Q: What is long polling? A: Long polling is a technique where the client sends a request to the server and the server holds the connection open until new data is available (or a timeout occurs). When data arrives or timeout hits, the server responds and the client immediately opens a new request. This provides near-real-time updates using standard HTTP. - Q: What is the difference between long polling and WebSockets? A: Long polling uses standard HTTP request-response cycles, opening a new connection for each update. WebSockets maintain a persistent bidirectional connection. Long polling works through any firewall/proxy that supports HTTP; WebSockets need special proxy support. WebSockets are more efficient for high-frequency updates; long polling is simpler to implement and more universally compatible. - Q: When should I use long polling instead of WebSockets? A: Use long polling when WebSockets are blocked by firewalls or proxies, when you need a simple fallback mechanism, when updates are infrequent (less than once per second), or when you're using serverless infrastructure that doesn't support persistent connections. Long polling works everywhere HTTP works. - Q: What are the downsides of long polling? A: Long polling creates more overhead than WebSockets due to HTTP headers on every request, requires server resources to hold connections open, and has slightly higher latency since a new connection is needed after each response. It's also less efficient for high-frequency bidirectional communication. ### How Stock Brokers Push 1 Million Price Updates Per Second to Your Screen - URL: https://singhajit.com/how-stock-brokers-handle-real-time-price-updates/ - Date: 2025-12-01 - Tags: system-design - Description: How stock brokers deliver millions of real-time price updates per second using WebSockets, Kafka, and ticker plants. Complete system design guide covering the fan-out problem, low-latency architecture, and real-time data distribution. FAQ: - Q: How do stock brokers handle millions of real-time price updates? A: Stock brokers use a multi-layered architecture: exchanges broadcast price updates to ticker plants that normalize and filter data, message brokers (like Kafka) distribute updates to broker backends, WebSocket connections push updates to clients, and clients buffer and batch updates for efficient rendering. The entire journey from exchange to user screen takes 300-500 milliseconds. - Q: What is a ticker plant in stock trading systems? A: A ticker plant is a high-performance system that receives raw market data feeds from exchanges, normalizes different exchange formats into a common format, filters and enriches data, and distributes it to downstream systems. It acts as the central hub that processes millions of messages per second before they reach broker systems. - Q: How do WebSockets help in real-time stock price updates? A: WebSockets provide persistent, bidirectional connections that allow servers to push price updates instantly to clients without polling overhead. Unlike HTTP polling, WebSockets maintain open connections, enabling sub-second latency for price updates. Brokers use WebSocket connections to push filtered, normalized price data directly to user devices. - Q: What is the fan-out problem in stock trading systems? A: The fan-out problem occurs when one price update from an exchange needs to reach millions of users watching that stock. A single update must be replicated and delivered to potentially millions of WebSocket connections. Message brokers like Kafka solve this by allowing one producer to publish to a topic, and multiple consumers (one per user connection) subscribe and receive the update. - Q: How fast are stock price updates delivered to users? A: Stock price updates typically reach users within 300-500 milliseconds from when a trade occurs on the exchange. This includes exchange processing, ticker plant normalization, broker backend processing, WebSocket transmission, and client rendering. High-frequency trading systems aim for even lower latency, but for retail users, sub-second delivery is the standard. ### Stop Blocking Your Paying Customers: Build a Smart Rate Limiter - URL: https://singhajit.com/dynamic-rate-limiter-system-design/ - Date: 2025-11-05 - Tags: system-design - Description: Learn how to design and implement a dynamic rate limiter that adapts to system load, user behavior, and traffic patterns. Real-world strategies from Stripe, Twitter, and Netflix. FAQ: - Q: What is a rate limiter in system design? A: A rate limiter controls how many requests a client can make to an API within a time window. It protects servers from overload, prevents abuse, and ensures fair resource distribution. Common implementations include token bucket, leaky bucket, fixed window, and sliding window algorithms. - Q: What is the difference between token bucket and leaky bucket? A: Token bucket allows bursts up to a maximum capacity while refilling at a steady rate - good for APIs that can handle occasional traffic spikes. Leaky bucket processes requests at a constant rate regardless of input - good for systems that need smooth, predictable throughput. Token bucket is more flexible; leaky bucket is more predictable. - Q: How do you implement distributed rate limiting? A: Distributed rate limiting typically uses a centralized store like Redis to track request counts across multiple server instances. Common approaches include Redis INCR with TTL, Lua scripts for atomic operations, or sliding window counters. The key challenge is balancing accuracy with latency overhead. - Q: What HTTP status code should a rate limiter return? A: Rate limiters should return HTTP 429 (Too Many Requests) when limits are exceeded. Include headers like X-RateLimit-Limit (max requests), X-RateLimit-Remaining (requests left), X-RateLimit-Reset (when the window resets), and Retry-After (seconds to wait). This helps clients implement proper backoff. ### How Shopify Powers 5 Million Stores Without Breaking a Sweat - URL: https://singhajit.com/shopify-system-design/ - Date: 2025-10-24 - Tags: system-design - Description: Deep dive into Shopify's system design and architecture. How they handle millions of merchants, billions in sales, and massive traffic spikes. Learn from their modular monolith, pod architecture, and scaling strategies. FAQ: - Q: How does Shopify handle millions of stores? A: Shopify uses a pod architecture where stores are grouped into isolated 'pods' - each pod is a complete copy of the application stack with its own databases. This provides tenant isolation (one store's traffic spike doesn't affect others), enables horizontal scaling by adding more pods, and limits blast radius if issues occur. - Q: Why did Shopify choose a modular monolith over microservices? A: Shopify found that microservices added too much operational complexity for their needs. A modular monolith gives them clear code boundaries and team autonomy while keeping deployment simple and avoiding network latency between services. They can still extract modules into services when truly needed. - Q: How does Shopify handle Black Friday traffic? A: Shopify handles BFCM (Black Friday/Cyber Monday) through extensive caching, pre-scaling pods, load shedding for non-critical features, and queue-based processing for orders. They also run 'flash sale' simulations throughout the year to test their infrastructure under realistic spiky load conditions. - Q: What database does Shopify use? A: Shopify primarily uses MySQL with extensive sharding. Each pod has its own MySQL cluster. They shard by shop_id, ensuring all data for a single store lives on the same shard. This avoids cross-shard queries and enables linear horizontal scaling as they add more shops. ### How DNS Works: The Complete Guide for Developers - URL: https://singhajit.com/how-dns-works-complete-guide/ - Date: 2025-10-21 - Tags: networking, system-design, tutorial - Description: Complete guide to how DNS works for developers. Learn DNS resolution step by step, caching layers, DNS record types (A, CNAME, MX, TXT), TTL, DNS performance optimization, and debugging with dig and nslookup. Includes diagrams and real-world examples. FAQ: - Q: How does DNS work? A: DNS translates domain names (like google.com) to IP addresses. When you visit a website, your browser checks local caches first, then queries a recursive resolver, which contacts root servers, TLD servers (.com, .org), and finally the authoritative nameserver for the domain. The IP address is returned and cached at multiple levels for faster future lookups. - Q: What is DNS TTL? A: TTL (Time To Live) is how long DNS records should be cached before re-querying. A 300-second TTL means caches will hold the record for 5 minutes. Lower TTLs enable faster failover but increase DNS query load. Higher TTLs reduce load but mean changes take longer to propagate. Typical values range from 60 seconds to 24 hours. - Q: What is the difference between A record and CNAME? A: An A record maps a domain directly to an IPv4 address (example.com to 93.184.216.34). A CNAME creates an alias pointing to another domain (www.example.com to example.com). CNAMEs require an additional DNS lookup to resolve the final IP. A records are terminal; CNAMEs chain to other records. - Q: What causes DNS propagation delays? A: DNS propagation delays occur because cached records must expire before new ones take effect. If your old TTL was 24 hours, some users will see old IPs for up to 24 hours after a change. To minimize delays, lower your TTL before making changes, wait for the old TTL to expire, then make the change and restore normal TTL. - Q: What is DNS over HTTPS (DoH)? A: DNS over HTTPS encrypts DNS queries inside HTTPS connections, preventing your ISP from seeing which websites you look up. Firefox uses Cloudflare DoH by default. Chrome and Android also support DoH. It provides privacy and protection from DNS spoofing, but bypasses corporate DNS filters and adds slight latency. ### How Ticket Booking Systems Handle 50,000 People Fighting for One Seat - URL: https://singhajit.com/ticket-booking-system-design/ - Date: 2025-10-13 - Tags: system-design - Description: How ticket booking systems prevent double bookings, handle 50K concurrent users, and process payments. Inside BookMyShow and Ticketmaster architecture. FAQ: - Q: How do ticket booking systems prevent double booking? A: Ticket systems use distributed locks (often Redis-based) to ensure only one user can hold a seat at a time. When you select a seat, the system acquires a lock with a TTL (e.g., 5 minutes). If payment completes, the booking is finalized. If the lock expires, the seat returns to available. Database constraints provide a final safety net. - Q: Why do ticket booking sites show seats that disappear? A: Seats 'disappear' because of temporary holds. When you view available seats, someone else may have just locked one. Or your held seat expired while you were entering payment details. High-traffic events have thousands of concurrent locks expiring and being acquired, creating constantly shifting availability. - Q: How do systems like Ticketmaster handle millions of concurrent users? A: High-traffic ticket systems use virtual queues to control flow, CDNs for static content, aggressive caching for show/venue data, and horizontal scaling of booking services. They separate read operations (browsing) from write operations (booking) and use rate limiting to prevent any single user from overwhelming the system. - Q: What caused the Ticketmaster Taylor Swift crash? A: The 2022 Eras Tour crash happened when 14 million fans attempted to buy 2 million tickets simultaneously. The system couldn't handle the concurrency - locks were timing out, database connections maxed out, and the virtual queue became overwhelmed. The architecture wasn't designed for that level of simultaneous demand. ### Kubernetes Architecture: The Operating System for the Cloud - URL: https://singhajit.com/devops/kubernetes-architecture/ - Date: 2025-09-30 - Tags: system-design, devops - Description: Deep dive into Kubernetes architecture - understand how the control plane, worker nodes, and core components work together to orchestrate containers at scale. Learn from real-world examples and practical insights. FAQ: - Q: What are the main components of Kubernetes architecture? A: Kubernetes has two main parts: the Control Plane (kube-apiserver, etcd, kube-scheduler, kube-controller-manager) which makes decisions about the cluster, and Worker Nodes (kubelet, kube-proxy, container runtime) which run the actual containerized workloads. - Q: What is the Kubernetes control plane? A: The control plane is the brain of Kubernetes that manages the cluster state. It includes the API server (handles all communication), etcd (stores cluster state), scheduler (assigns pods to nodes), and controller manager (maintains desired state). It runs on master nodes separate from worker nodes. - Q: What is etcd in Kubernetes? A: etcd is a distributed key-value store that holds all Kubernetes cluster data including pod specs, service configurations, secrets, and cluster state. It uses the Raft consensus algorithm for reliability. The API server is the only component that directly communicates with etcd. - Q: What is the difference between a Pod and a Container in Kubernetes? A: A container is a single running instance of a Docker/OCI image. A Pod is the smallest deployable unit in Kubernetes that can contain one or more containers sharing the same network namespace, IP address, and storage volumes. Pods provide the execution environment for containers. - Q: How does Kubernetes handle node failures? A: When a node fails, the Node Controller detects the failure (typically after 5 minutes of no heartbeat). It marks the node as NotReady and evicts pods. The ReplicaSet controller then creates new pods on healthy nodes to maintain the desired replica count. This self-healing happens automatically. - Q: What is kubelet and what does it do? A: Kubelet is the primary node agent running on every worker node. It watches for pod assignments from the API server, pulls container images, starts and monitors containers, runs health checks (liveness and readiness probes), and reports node and pod status back to the control plane. ### How Meta Achieves 99.99999999% Cache Consistency - URL: https://singhajit.com/meta-cache-consistency/ - Date: 2025-09-22 - Tags: system-design - Description: Deep dive into Meta's cache consistency architecture - how they handle billions of users with near-perfect cache consistency using TAO, memcache, and distributed invalidation strategies. Learn from their scaling challenges and architectural decisions. FAQ: - Q: How does Meta achieve cache consistency across data centers? A: Meta uses a cache invalidation service that propagates invalidations globally within milliseconds. When data changes, the system invalidates cached copies everywhere before acknowledging the write. They use version numbers and lease mechanisms to prevent stale data from being written back to cache after invalidation. - Q: What is TAO at Meta? A: TAO (The Associations and Objects) is Meta's distributed graph cache system designed for social data. It stores objects (users, posts, photos) and associations (friendships, likes, comments) as a graph. TAO provides read-after-write consistency and handles trillions of queries per day across Meta's global infrastructure. - Q: How does Meta handle cache invalidation at scale? A: Meta's cache invalidation system uses a pub/sub model where database changes trigger invalidation messages that propagate to all cache servers globally. They batch invalidations for efficiency, use version vectors to detect stale writes, and employ lease mechanisms to prevent thundering herd problems during cache misses. - Q: What is the difference between cache-aside and write-through caching? A: In cache-aside (look-aside), the application checks cache first, then database on miss, and populates cache manually. In write-through, writes go to both cache and database simultaneously. Meta uses variations of both - TAO is write-through for consistency, while memcache is cache-aside for flexibility. ### How Slack Built a System That Handles 10+ Billion Messages - URL: https://singhajit.com/slack-system-design/ - Date: 2025-09-19 - Tags: system-design - Description: Deep dive into Slack's system design and architecture - how they handle millions of users, billions of messages, and maintain real-time communication at scale. Learn from their scaling challenges, database design, and microservices architecture. FAQ: - Q: What database architecture does Slack use for storing messages? A: Slack uses MySQL with workspace-based sharding. Messages are stored in tiered storage - hot storage (Redis cache + MySQL) for recent messages (last 30 days), warm storage for older messages (30-365 days), and cold storage (Amazon S3) for messages older than a year. Elasticsearch powers the search layer across all tiers. - Q: How does Slack handle millions of users and channels at scale? A: Slack shards everything by workspace - each workspace gets its own database shard, RTM server, and search index. This provides linear scaling, fault isolation, and data locality. Users within a workspace share resources, but different workspaces are completely isolated. - Q: How does Slack deliver messages in real-time to users? A: Slack uses a two-brain architecture: WebApp servers handle message validation, processing, and database writes, while Real-Time Messaging (RTM) servers manage WebSocket connections and message broadcasting. Messages flow through WebApp for storage, then RTM broadcasts to connected users instantly. - Q: What is Slack's scalability strategy for handling billions of messages? A: Slack's scalability comes from workspace-based sharding, separating read/write concerns (WebApp vs RTM servers), tiered message storage, Redis caching, and solving the thundering herd problem with exponential backoff and jitter during reconnections. - Q: How does Slack store and retrieve channel messages efficiently? A: Channel messages are stored in sharded MySQL databases partitioned by workspace. Recent messages are cached in Redis for instant access. Lazy-loading fetches message history as users scroll, and Elasticsearch enables fast full-text search across all messages. - Q: What technology stack powers Slack's messaging architecture? A: Slack uses PHP for WebApp servers, Java for RTM (real-time) servers, MySQL for sharded databases, Redis for caching and connection state, Elasticsearch for search, AWS infrastructure, HAProxy for load balancing, and CloudFront CDN for file delivery. ### Distributed Counter System Design - URL: https://singhajit.com/distributed-counter-architecture-guide/ - Date: 2025-09-03 - Tags: system-design - Description: How to design a distributed counter for high-traffic systems. Complete system design guide covering sharded counters, sharded counter architecture, local aggregation, CRDTs, and production patterns with code examples. FAQ: - Q: What is a distributed counter? A: A distributed counter is a data structure that tracks counts across multiple servers or nodes in a distributed system. Unlike a simple counter that runs on one machine, distributed counters handle concurrent increments from many sources while maintaining consistency, fault tolerance, and high availability. - Q: How do you design a distributed counter? A: To design a distributed counter, you can use techniques like sharded counters (splitting the count across multiple shards), local aggregation (batch updates locally before syncing), or CRDTs (Conflict-free Replicated Data Types). The choice depends on your consistency requirements, traffic volume, and acceptable read latency. - Q: What are sharded counters? A: Sharded counters split a single logical counter into multiple physical shards. Each shard handles a portion of the increment traffic independently. To get the total count, you sum all shards. This approach eliminates single-point bottlenecks and allows horizontal scaling for high-traffic scenarios like social media likes or view counts. - Q: When should you use sharded counters vs a single counter? A: Use sharded counters when you have high write throughput (thousands of increments per second), need to avoid hotspots, or require horizontal scalability. A single counter works fine for low-traffic scenarios or when strong consistency on every read is critical. ### How Stripe Prevents Double Payments With Idempotency Keys - URL: https://singhajit.com/how-stripe-prevents-double-payment/ - Date: 2025-08-29 - Tags: system-design - Description: Learn how Stripe prevents double payments using idempotency keys. Complete guide to stripe idempotency with code examples using tok_visa, database constraints, and retry logic. Prevent duplicate charges in your payment systems. FAQ: - Q: What are Stripe idempotency keys and how do they prevent duplicate charges? A: Stripe idempotency keys are unique identifiers you send with API requests. When Stripe receives a request with an idempotency key it has seen before, it returns the cached response from the original request instead of processing the payment again. This prevents duplicate charges even if you retry the same request multiple times due to network failures. - Q: How do I use idempotency keys with Stripe tok_visa for testing? A: When testing with tok_visa or other Stripe test tokens, include an Idempotency-Key header with a unique value like a UUID. Example: curl -X POST https://api.stripe.com/v1/charges -H 'Idempotency-Key: unique-key-123' -d source=tok_visa -d amount=1000 -d currency=usd. The same key with the same parameters will return the cached result. - Q: How long does Stripe store idempotency keys? A: Stripe stores idempotency keys for 24 hours. After 24 hours, using the same key will process the request as new. This window is long enough to handle retries during outages but short enough to prevent storage issues. Generate new keys for genuinely new transactions. - Q: What happens if I send different parameters with the same idempotency key? A: Stripe returns a 400 error if you send a request with an existing idempotency key but different parameters. This is a safety feature. It prevents bugs where you accidentally reuse a key for a different transaction. Always generate a new idempotency key for each unique operation. - Q: How do I prevent duplicate payments in Stripe when the network fails? A: Use idempotency keys on every payment request. Generate the key before making the request and store it. If the request fails or times out, retry with the same key. Stripe will either process the payment (if the first request never arrived) or return the cached result (if it did). This makes retries safe. - Q: What is the best format for Stripe idempotency keys? A: Stripe recommends using V4 UUIDs for idempotency keys. Alternatively, use a combination of your order ID and attempt number, like order_12345_v1. The key must be unique per operation. Using predictable patterns like order IDs alone risks key reuse if a customer places multiple orders. ### 55 Million Requests Per Second: Inside Cloudflare's Magic - URL: https://singhajit.com/how-cloudflare-supports-55-million-requests-per-second/ - Date: 2025-08-20 - Tags: system-design - Description: Deep dive into Cloudflare's technical architecture - how 15 PostgreSQL clusters, ClickHouse, and Quicksilver work together to handle 55 million requests per second with millisecond latency. FAQ: - Q: How does Cloudflare handle 55 million requests per second? A: Cloudflare handles 55 million RPS using a combination of connection pooling with PgBouncer, bare metal PostgreSQL servers, HAProxy load balancing, Anycast routing across 330+ data centers, and intelligent caching. They use only 15 PostgreSQL clusters by efficiently managing connections and distributing traffic globally. - Q: What is Anycast and how does Cloudflare use it? A: Anycast is a routing technique where multiple servers share the same IP address. The internet automatically routes requests to the nearest server. Cloudflare uses Anycast across 330+ data centers so users are automatically served from the closest location, reducing latency and providing automatic failover if a data center goes offline. - Q: Why does Cloudflare use bare metal servers instead of cloud? A: Cloudflare runs PostgreSQL on bare metal servers to eliminate virtualization overhead. At 55 million requests per second, every microsecond matters. Bare metal provides predictable performance without the latency penalty of virtualization layers, maximizing throughput from their hardware. - Q: What is PgBouncer and why does Cloudflare use it? A: PgBouncer is a lightweight PostgreSQL connection pooler. Instead of each application opening direct database connections (which are expensive), PgBouncer maintains a smaller pool of connections and shares them among thousands of clients. This prevents connection exhaustion and handles the thundering herd problem. ### How Uber Finds Nearby Drivers at 1 Million Requests per Second - URL: https://singhajit.com/how-uber-finds-nearby-drivers-1-million-requests-per-second/ - Date: 2025-08-16 - Tags: system-design - Description: How does Uber find nearby drivers? Learn how Uber's system for finding you nearby drivers works at 1M+ requests per second using H3 hexagonal grids, geospatial indexing, and real-time matching. FAQ: - Q: How does Uber find nearby drivers? A: When you request a ride, Uber converts your GPS location into an H3 hexagonal cell ID. The system then searches that cell and its neighboring hexagons to find all available drivers nearby. This geospatial indexing approach means Uber doesn't need to calculate distances to every driver in the city - just those in nearby hexagonal cells, making the search extremely fast. - Q: How does Uber find nearby drivers so quickly? A: Uber uses H3 hexagonal grid indexing to convert GPS coordinates into hexagonal cells. When you request a ride, the system converts your location to an H3 cell, searches nearby cells (k-ring neighbors), and finds all available drivers in those cells. This avoids checking every driver in the city, reducing search time from 10-15 seconds to under 3 seconds. - Q: What is H3 hexagonal grid and how does Uber use it? A: H3 is a hexagonal hierarchical spatial indexing system that divides the Earth into millions of hexagonal tiles. Uber uses H3 to convert GPS coordinates into unique 64-bit cell IDs. This allows them to quickly find drivers in nearby hexagons without expensive distance calculations. Hexagons are preferred over squares because they have uniform neighbors and better approximate circles. - Q: How does Uber handle 1 million requests per second? A: Uber handles 1M+ RPS through geographic sharding (data separated by city/region), memory-first architecture (driver locations in RAM), circuit breakers for overload protection, and predictable performance optimization focusing on p99 latency. The real-time index is sharded by H3 prefix, ensuring requests from one city never hit servers storing data from another city. - Q: What technology stack does Uber use for driver matching? A: Uber's driver matching system uses Go for real-time services, Java for business logic, Redis for hot data, Cassandra for persistent storage, Kafka for event streaming, gRPC for service-to-service communication, and WebSocket for mobile connections. The system is deployed using blue-green deployments with canary releases. ### How WhatsApp Scaled to Billions of Users with Just 50 Engineers - URL: https://singhajit.com/whatsapp-scaling-secrets/ - Date: 2025-08-07 - Tags: system-design - Description: Learn how WhatsApp handles 100 billion messages daily with a tiny team. Deep dive into Erlang, the actor model, Mnesia database, and the system design that powers 2 billion users. Key Takeaways: - Erlang's actor model lets each connection run as an isolated lightweight process - One server can handle 2+ million concurrent connections using Erlang processes - Messages are stored only until delivered, then deleted from servers - Hot code swapping enables updates without disconnecting users - FreeBSD was chosen over Linux for superior networking performance FAQ: - Q: How did WhatsApp scale to billions of users with only 50 engineers? A: WhatsApp achieved massive scale through smart technology choices: Erlang for massive concurrency with lightweight processes, Mnesia for fast in-memory data, FreeBSD for superior networking, and a philosophy of extreme simplicity. Each Erlang process handles one client, allowing millions of connections per server with minimal overhead. - Q: Why did WhatsApp choose Erlang over Java or Python? A: Erlang was built for telecom systems that need 99.999% uptime. It offers lightweight processes (2KB each vs 1MB for OS threads), built-in fault tolerance through supervisors, hot code swapping for zero-downtime updates, and native distributed computing. These features aligned perfectly with WhatsApp's needs. - Q: What database does WhatsApp use? A: WhatsApp uses Mnesia (an Erlang-native distributed database) for real-time data like sessions and routing. They also use MySQL shards for user data and RocksDB for fast read/write operations. Messages are stored temporarily in memory until delivered. - Q: How does WhatsApp deliver messages at scale? A: Each connected user has a dedicated Erlang process on the server. When you send a message, your process looks up the recipient's process and forwards the message directly. If offline, messages queue in the sender's process until the recipient reconnects. This direct process-to-process communication is extremely fast. - Q: What is the Signal Protocol and how does WhatsApp use it? A: The Signal Protocol provides end-to-end encryption using the Double Ratchet algorithm. Each message gets a unique encryption key. Even if one key is compromised, past and future messages remain secure. WhatsApp cannot read message content because encryption happens on your device. - Q: How does WhatsApp handle media files like images and videos? A: Media files are encrypted on the device, uploaded to a CDN (content delivery network), and only the encryption key plus CDN URL are sent through the message. Recipients download from the CDN and decrypt locally. This keeps the message servers lightweight. - Q: What is hot code swapping and why does it matter? A: Hot code swapping lets you update running code without stopping the application. WhatsApp can deploy new features and fixes while users stay connected. This is critical for a global service where any downtime affects billions of people. - Q: Why did WhatsApp choose FreeBSD instead of Linux? A: FreeBSD's networking stack handled more concurrent TCP connections per server. The WhatsApp team found they could push over 2 million connections on a single machine with FreeBSD. The kernel's fine-grained tuning options let them optimize specifically for their workload. ### How Flutter Works Under the Hood - URL: https://singhajit.com/flutter-under-the-hood/ - Date: 2020-09-20 - Tags: flutter, mobile-cross-platform, system-design - Description: How does Flutter work under the hood? A deep dive into Flutter architecture, the three-tree rendering system (Widget, Element, RenderObject), Impeller and Skia engines, Dart AOT and JIT compilation, hot reload, platform channels, and how Flutter renders UI at 60-120fps. Written for software developers. FAQ: - Q: What are the three layers of Flutter architecture? A: Flutter has three layers. The Framework layer is written in Dart and contains widgets, rendering, animation, and gesture handling. The Engine layer is written in C++ and handles low-level rendering (via Impeller or Skia), the Dart runtime, text layout, and platform channels. The Embedder layer is platform-specific (Java/C++ on Android, Objective-C/Swift on iOS) and handles the app lifecycle, input events, and rendering surfaces. - Q: How does Flutter render UI without native components? A: Flutter does not use OEM widgets like UIKit on iOS or Android Views. Instead, it renders every pixel itself using a graphics engine (Impeller or Skia). Your widgets describe the UI, Flutter builds three internal trees (Widget, Element, RenderObject), calculates layout, and the engine draws pixels directly to a platform-provided canvas. This is why a Flutter app looks identical on iOS and Android. - Q: What is the difference between the Widget Tree, Element Tree, and RenderObject Tree? A: The Widget Tree holds your immutable UI descriptions. Widgets are cheap to create and get thrown away on every rebuild. The Element Tree is the long-lived layer that manages state and lifecycle. It compares old and new widgets to decide what changed. The RenderObject Tree handles layout, painting, and hit-testing. RenderObjects are expensive, so Flutter reuses them whenever possible instead of creating new ones. - Q: What is Impeller and how is it different from Skia? A: Impeller is Flutter's newer rendering engine that replaced Skia. Skia compiled shaders at runtime (JIT), which caused frame drops the first time new visual effects appeared. Impeller pre-compiles all shaders at build time (AOT), eliminating that jank entirely. Impeller also uses modern GPU APIs like Metal on iOS and Vulkan on Android. Impeller is now the only engine on iOS and the default on Android. - Q: How does Flutter hot reload work? A: Hot reload works because development builds use Dart's JIT compiler running on the Dart VM. When you save a file, Flutter detects the changes, sends updated source code to the VM on your device, recompiles only the changed functions, and reassembles all widgets. Your app state is preserved. The whole cycle takes under 500ms. Production builds use AOT compilation and do not support hot reload. - Q: What is the difference between AOT and JIT compilation in Flutter? A: JIT (Just-In-Time) compilation runs Dart code on the Dart VM during development. It compiles code as needed, enabling hot reload and fast iteration. AOT (Ahead-Of-Time) compilation is used for release builds. It compiles Dart directly to native ARM machine code, removing the need for a VM entirely. AOT builds are faster at runtime but cannot support hot reload. - Q: How does Flutter communicate with native platform code? A: Flutter uses Platform Channels to communicate with native code. Your Dart code sends a message through a MethodChannel with a method name and arguments. The native side (Kotlin/Java on Android, Swift/Objective-C on iOS) receives the message, processes it, and sends a response back. Messages are serialized using a binary codec and passed asynchronously to keep the UI responsive. - Q: How is Flutter different from React Native? A: React Native uses a bridge (or JSI in the new architecture) to control native UI components. Your JavaScript code tells the platform what native views to render. Flutter takes a completely different approach. It compiles Dart to native ARM code and renders its own UI with a graphics engine, bypassing native components entirely. Flutter achieves consistent cross-platform appearance. React Native uses actual platform components, so apps look different on iOS and Android. ### Android Build Process: How Your Code Becomes an APK - URL: https://singhajit.com/android-build-process/ - Date: 2015-10-20 - Tags: android, system-design - Description: Learn how the Android build process works step by step. Covers Gradle, AAPT2, D8, R8, APK signing, build variants, product flavors, build optimization, and how your source code becomes a running app. Complete guide for Android developers. FAQ: - Q: What is the Android build process? A: The Android build process is the series of steps that transforms your source code, resources, and libraries into an installable APK or Android App Bundle. Gradle orchestrates the build. AAPT2 compiles resources and generates R.java. The Java or Kotlin compiler creates .class files. D8 converts them to Dalvik bytecode (.dex). For release builds, R8 shrinks and obfuscates the code. Finally, everything is packaged, aligned, and signed. - Q: What is AAPT2 in Android and how is it different from AAPT? A: AAPT2 (Android Asset Packaging Tool 2) compiles resources in the res/ directory and AndroidManifest.xml. It generates R.java containing integer IDs for all resources. AAPT2 replaced the original AAPT and works in two phases: compile (processes individual resource files into binary format) and link (merges all compiled resources and generates the final resource table). AAPT2 supports incremental resource compilation, making builds faster. - Q: What is the difference between D8 and R8 in Android? A: D8 is the default dex compiler that converts Java bytecode (.class files) into Dalvik bytecode (.dex files) for the Android runtime. R8 replaces both D8 and ProGuard in release builds. R8 does everything D8 does plus code shrinking (removes unused code), obfuscation (renames classes and methods), and optimization (improves bytecode). R8 is enabled by default for release builds. - Q: What is the difference between APK and Android App Bundle (AAB)? A: An APK is a ready-to-install package containing all your app's code, resources, and assets. An Android App Bundle (AAB) is a publishing format that contains all your app's compiled code and resources but defers APK generation to Google Play. Google Play generates optimized APKs for each device configuration, resulting in smaller downloads. Google Play requires AAB for new app submissions. - Q: What are build variants in Android? A: Build variants are combinations of build types (like debug and release) and product flavors (like free and paid). If you have 2 build types and 2 product flavors, you get 4 build variants: freeDebug, freeRelease, paidDebug, paidRelease. Each variant can have different source code, resources, and build settings. Gradle generates a separate APK for each variant. - Q: What is R.java in Android? A: R.java is an auto-generated class that contains integer IDs for every resource in your app. Layouts, drawables, strings, colors, and styles all get unique IDs. When you write R.layout.activity_main or R.drawable.icon in code, you are referencing these IDs. AAPT2 generates R.java during the build. Never edit it manually as it gets regenerated on every build. - Q: What is classes.dex and why does Android use it? A: classes.dex contains your compiled code in Dalvik bytecode format, which is what the Android Runtime (ART) executes. Android does not use standard Java bytecode because Dalvik bytecode is optimized for devices with limited memory and battery. The D8 compiler converts .class files to .dex format. Large apps may have multiple dex files (multidex) because a single dex file can reference at most 65,536 methods. - Q: How do I speed up Android Gradle builds? A: Enable the Gradle daemon (runs by default), use the Gradle build cache, enable parallel execution with org.gradle.parallel=true, increase JVM heap size with org.gradle.jvmargs=-Xmx4g, use incremental compilation, avoid dynamic dependency versions, and modularize your project so unchanged modules are skipped. For large projects, enabling configuration cache can save significant time. --- ## Distributed Systems Articles ### Emergent Leader Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/emergent-leader/ - Date: 2026-08-07 - Description: Learn the Emergent Leader pattern in distributed systems: how peer-to-peer clusters like Akka, Hazelcast, and JGroups pick a coordinator by node age instead of running an election, how gossip and heartbeats keep the choice in sync, and when this is safe versus when you need Raft or a consistent core. Key Takeaways: - Emergent leader means the coordinator falls out of an agreed ordering of nodes instead of being chosen by a vote. Order the members by age and the oldest one is the leader, no election protocol needed. - It works because every node applies the same deterministic rule to the same membership list, so they all reach the same conclusion independently. There is no ballot, no majority, no term number. - Gossip spreads membership changes and heartbeats detect failures. Once the cluster reaches gossip convergence, the leader is obvious to everyone. - This is cheaper and more available than running Raft or Paxos, which is why peer-to-peer systems like Akka, Hazelcast, and JGroups use it for cluster management. - The catch is that it is only as safe as your membership view. During a network partition both sides can think they hold the oldest node, so emergent leaders are used for management tasks, not for data that must never have two writers. - Use an emergent leader for coordination that can tolerate brief disagreement. Use a consistent core or a real consensus-backed election when a split decision would corrupt data. ### Consistent Core Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/consistent-core/ - Date: 2026-07-03 - Description: Learn the Consistent Core pattern in distributed systems: why quorum throughput drops as clusters grow, how a small 3 to 5 node core stores metadata with linearizable consistency, and how ZooKeeper, etcd, Consul, Kafka, and Kubernetes use it for leader election, locks, and group membership. Key Takeaways: - Quorum-based consensus gets slower as you add nodes, because every write must be acknowledged by a majority. You cannot run it across a 300 node data cluster on the hot path. - The fix is to split the problem: a small consistent core (3 or 5 nodes) holds the little bit of state that must be linearizable, and the large data cluster handles the bulk work. - The core stores metadata, not data. Group membership, leader and partition assignments, configuration, locks, and leases live there. User data does not. - Clients talk to the core through sessions with heartbeats, ephemeral keys that vanish when a session dies, and watches that push change notifications instead of forcing you to poll. - Linearizable reads are the subtle part. A follower can serve stale metadata, so the core reads through the leader using a leader lease or a ReadIndex check to stay correct. - You almost never build a consistent core yourself. You run etcd, ZooKeeper, or Consul, or embed a Raft library, because hand-rolled coordination is a famous source of rare data-loss bugs. ### Leader and Followers Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/leader-follower/ - Date: 2026-06-16 - Description: Learn the Leader and Followers pattern in distributed systems: leader election, heartbeats, generation clocks, and log replication. With real examples from Kafka, ZooKeeper, Raft, etcd, MongoDB, Redis, and PostgreSQL, plus how to avoid split brain. Key Takeaways: - One leader makes all the write decisions and replicates them to followers. This turns a hard 'everyone must agree on every write' problem into a simpler 'agree once on who the leader is' problem. - Leader election needs a majority quorum so two halves of a split network can never both elect a leader. The minority side cannot reach a majority, so it cannot make progress. - A generation clock (term or epoch number) tags everything the leader does. Followers reject messages from an older generation, which is what stops a revived old leader from corrupting state. - Heartbeats are how followers notice a dead leader. If a follower misses heartbeats for an election timeout, it starts a new election in a higher generation. - Single leader replication gives you strong consistency and simple reasoning at the cost of write throughput bounded by one node and a short unavailability window during failover. - Leaderless designs like Dynamo and Cassandra trade that simplicity for higher write availability. The leader and followers pattern is the right default when correctness matters more than squeezing out every last write. ### Lease Pattern in Distributed Systems Explained - URL: https://singhajit.com/distributed-systems/lease/ - Date: 2026-05-21 - Description: Learn the Lease pattern in distributed systems: time-bound exclusive access using TTL, heartbeats, and fencing tokens. With real implementations from etcd, Kubernetes, ZooKeeper, Chubby, and HDFS, plus pitfalls around GC pauses and clock drift. Key Takeaways: - A lease is a lock with an expiry date. The holder gets exclusive access for a fixed TTL and must renew it with a heartbeat before the timer runs out. - Leases fix the classic locking problem in distributed systems: a node that crashes or pauses after taking a lock would otherwise hold the resource forever. The TTL is the safety net. - Always pair a lease with a [fencing token](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html). Without it, a paused or slow holder can wake up after expiry and corrupt the resource it no longer owns. - Leases need a strongly consistent store to issue and renew. Most production systems use a [Raft](/distributed-systems/replicated-log/) or [Paxos](/distributed-systems/paxos/) backed Consistent Core such as etcd, ZooKeeper, or Chubby. - TTL is a trade-off. Short TTL means fast failover but more renewal traffic and more risk of spurious expiry on a slow network. Long TTL means cheaper coordination but slower recovery. - Wall clocks lie. Lease durations should be measured against a monotonic clock and a generous safety margin. Process pauses, virtualisation stalls, and GC pauses can eat the entire TTL in one shot. - Kubernetes leader election, HDFS file write coordination, etcd lock APIs, and ZooKeeper ephemeral nodes are all leases under different names. Once you see the shape, you spot it everywhere. ### Debezium and the Outbox Pattern: The Real Impact on Your Postgres Database - URL: https://singhajit.com/debezium-outbox-postgres-database-impact/ - Date: 2026-05-05 - Description: A practical, production-grade look at what Debezium does to your Postgres primary when you use it to stream a transactional outbox table to Kafka. Covers logical decoding, replication slots, WAL retention, walsender CPU, the reorder buffer, max_slot_wal_keep_size, heartbeats, monitoring, and what to tell your DBA team. Key Takeaways: - Debezium for Postgres uses logical replication. It looks like one extra replica to the database, but the cost is mostly CPU for decoding plus a replication slot that holds WAL on the primary. - The biggest operational risk is WAL bloat. If Debezium goes down or lags, `pg_wal/` can fill the disk and crash the primary. Use `max_slot_wal_keep_size` (Postgres 13+) to cap it. - Steady-state CPU overhead for an outbox-only publication is usually 5 to 15 percent on the primary, but only single-digit percent for outbox-shaped workloads with small inserts. - Use a narrow publication (`CREATE PUBLICATION outbox_pub FOR TABLE outbox`). Wildcard publications force the primary to decode every change before filtering, which is pure waste. - Set `REPLICA IDENTITY DEFAULT` (the primary key) on the outbox table. Never use `REPLICA IDENTITY FULL` here. It bloats every WAL record with the full row image. - Partition the outbox table by day or by hour and `DROP PARTITION` the old ones. Row-by-row `DELETE` from a hot CDC table creates tombstones, index bloat, and extra WAL traffic. - Monitor `pg_replication_slots.confirmed_flush_lsn` lag, walsender CPU, and reorder buffer spill files. Alert before the disk fills, not when it fills. - If the outbox table is rarely written to but the rest of the database is busy, add a [Debezium heartbeat](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) so the slot keeps advancing and WAL keeps recycling. - Debezium plus the outbox pattern is the right default. Application-level polling against the outbox table looks simpler but causes index bloat, lock contention, and is hard to make exactly-once. CDC just reads the WAL the database is already writing. ### Lamport Clock in Distributed Systems - URL: https://singhajit.com/distributed-systems/lamport-clock/ - Date: 2026-04-22 - Description: A practical guide to the Lamport Clock distributed systems pattern. Learn the algorithm, the happens-before relation, total ordering with tie-breakers, real implementations in Cassandra and Kafka, and how it compares to vector clocks and Hybrid Logical Clocks. Key Takeaways: - A Lamport Clock is a per node counter that ticks on every local event, send, and receive. It captures the happens-before relation across the cluster without needing synchronized wall clocks. - The receive rule is the whole pattern: `local = max(local, received) + 1`. Everything else is application detail. - Lamport Clocks give a partial order. To get a total order you append the node id and break ties lexicographically. - A Lamport timestamp tells you nothing about wall clock time. It cannot answer 'what happened at 9 AM yesterday' the way an HLC can. - If A and B are concurrent (neither happens-before the other) you cannot tell from their Lamport timestamps. Vector clocks are needed for that. - Real systems use Lamport style counters everywhere: Cassandra last write wins timestamps, Kafka producer epoch, Raft term numbers, Paxos ballot ids, and most write-ahead log sequence numbers are all Lamport descendants. - Lamport Clocks are the building block. Once you understand them, [vector clocks](https://en.wikipedia.org/wiki/Vector_clock) and [Hybrid Logical Clocks](/distributed-systems/hybrid-clock/) fall out as small, focused extensions. ### Hybrid Logical Clock in Distributed Systems - URL: https://singhajit.com/distributed-systems/hybrid-clock/ - Date: 2026-04-18 - Description: Learn how the Hybrid Logical Clock (HLC) pattern works in distributed systems. Complete guide with examples from CockroachDB, MongoDB, and YugabyteDB. Covers the HLC algorithm, 64 bit timestamp format, causal consistency, consistent snapshots, and how HLC compares to Lamport and vector clocks. Key Takeaways: - Lamport clocks give you ordering but no real time. Physical clocks give you real time but no causal ordering. Hybrid Logical Clocks give you both in a single 64 bit value. - An HLC timestamp is a tuple of (physical time, logical counter). The physical part stays within a small bound of NTP time. The counter only grows when events happen faster than the physical clock can distinguish. - On every send, receive, and local event the algorithm takes the maximum of the local physical time, the incoming HLC, and the local HLC, then bumps the counter only when wall time did not advance. - HLC is monotonic across the cluster even when individual node clocks drift, jump backward after an NTP correction, or briefly disagree by a few hundred milliseconds. - CockroachDB uses HLC for transaction ordering and an uncertainty interval equal to the configured max clock offset (default 500 ms) to handle ambiguous reads. - MongoDB exposes HLC as cluster time and operation time. Causally consistent sessions piggyback the latest cluster time on every read so secondaries can wait until they catch up before serving the query. - HLC does not need GPS, atomic clocks, or PTP to work. Plain NTP with a known maximum drift is enough, which is why almost every modern multi region database picks it over TrueTime style hardware clocks. ### Low Watermark: How Distributed Systems Know What's Safe to Delete - URL: https://singhajit.com/distributed-systems/low-watermark/ - Date: 2026-04-15 - Description: Learn how the Low Watermark pattern controls WAL truncation, log compaction, and log cleanup in Kafka, etcd, PostgreSQL, and ZooKeeper with examples. Key Takeaways: - The Low Watermark marks the oldest log entry the system must still keep. Everything below it can be deleted, compacted, or replaced by a snapshot. - Without a Low Watermark, write-ahead logs grow without limit and eventually fill up disk space, bringing the whole system down. - The Low Watermark is calculated as the minimum across all consumers of the log: the slowest replica, the oldest backup cursor, and any active replication slot. - In Kafka, the Low Watermark maps to the Log Start Offset. It advances when time-based retention deletes old segments or log compaction removes outdated keys. - In Raft-based systems like etcd, the Low Watermark advances after a snapshot is taken. The system discards all log entries before the snapshot index. - Advancing the Low Watermark too aggressively breaks follower catch-up and backup recovery. Too conservatively, and you run out of disk. ### High Watermark: How Distributed Systems Know What's Safe to Read - URL: https://singhajit.com/distributed-systems/high-watermark/ - Date: 2026-04-13 - Description: Learn how the High Watermark pattern keeps distributed systems consistent. Complete guide with real-world examples from Kafka, Raft, etcd, and ZooKeeper. Covers commit index, log replication, consumer visibility, ISR, leader election, and the relationship with Low Watermark. Key Takeaways: - The high watermark tracks the last log entry replicated to a majority of nodes. Anything above it is uncommitted and invisible to clients. - In Kafka, the high watermark equals the minimum Log End Offset across all In-Sync Replicas. Consumers can only read up to this point. - In Raft, the commit index serves the same purpose. The leader advances it when a majority of followers confirm an entry. - Without a high watermark, a leader crash could expose data to clients that the new leader doesn't have, causing data to disappear. - The [Low Watermark](/distributed-systems/low-watermark/) is its counterpart. It marks how far back the log can be safely truncated without losing recoverable state. ### How Replicated Log Works in Distributed Systems - URL: https://singhajit.com/distributed-systems/replicated-log/ - Date: 2026-02-16 - Description: Learn how the Replicated Log pattern keeps distributed systems in sync. Complete guide with real-world examples from Raft, Kafka, etcd, and ZooKeeper. Covers log replication, state machine replication, high-water mark, leader-based replication, log compaction, and failure recovery with diagrams. ### How Gossip Protocol Works in Distributed Systems - URL: https://singhajit.com/distributed-systems/gossip-dissemination/ - Date: 2026-02-10 - Description: What is gossip protocol? Learn how gossip dissemination works in distributed systems with real examples from Cassandra, Consul, and DynamoDB. Covers push, pull, push-pull variants, SWIM protocol, failure detection, anti-entropy repair, and tuning parameters with diagrams and code. ### Majority Quorum in Distributed Systems Explained - URL: https://singhajit.com/distributed-systems/majority-quorum/ - Date: 2026-01-03 - Description: Learn Majority Quorum: the consensus pattern behind Cassandra, etcd, and ZooKeeper. Master the W+R>N formula, fault tolerance, and split-brain prevention. ### Two-Phase Commit: The Protocol That Keeps Distributed Transactions Honest - URL: https://singhajit.com/distributed-systems/two-phase-commit/ - Date: 2025-12-06 - Description: What is 2 phase commit? Complete guide to the two phase commit protocol (2PC) for distributed transactions. Learn how the 2 phase commit protocol coordinates atomicity across multiple databases, its phases, failure scenarios, and implementations in PostgreSQL, MySQL, and microservices. ### Heartbeat: How Distributed Systems Know You're Still Alive - URL: https://singhajit.com/distributed-systems/heartbeat/ - Date: 2025-11-15 - Description: Learn how heartbeat mechanisms detect failures in distributed systems. Master failure detection patterns with real-world examples from Kubernetes, Cassandra, HAProxy, and etcd. Complete guide covering push/pull patterns, gossip protocols, split brain prevention, and Phi Accrual detection. ### How Kafka Works: The Engine Behind Real-Time Data Pipelines - URL: https://singhajit.com/distributed-systems/how-kafka-works/ - Date: 2025-10-01 - Description: Deep dive into Apache Kafka architecture - understand topics, partitions, consumer groups, and how Kafka achieves high throughput and fault tolerance. Learn from real-world examples and practical insights for building scalable data pipelines. ### Paxos: The Democracy of Distributed Systems - URL: https://singhajit.com/distributed-systems/paxos/ - Date: 2025-09-18 - Description: Learn how Paxos algorithm achieves consensus in distributed systems. Complete guide with real-world examples, diagrams, and practical implementations covering Google Chubby, Cassandra, and etcd. ### Write-Ahead Log: The Golden Rule of Durable Systems - URL: https://singhajit.com/distributed-systems/write-ahead-log/ - Date: 2025-09-10 - Description: Learn how Write-Ahead Log (WAL) prevents data loss in distributed systems. Complete guide with real-world examples, code samples, and diagrams covering PostgreSQL, Kafka, and custom implementations. --- ## Visual Explainers ### CQRS Design Pattern Explained - URL: https://singhajit.com/explainer/cqrs-design-pattern/ - Description: Master CQRS architecture through visual components and system diagrams. Learn when and how to implement Command Query Responsibility Segregation without heavy code examples. FAQ: - Q: What is CQRS and how does it work? A: CQRS (Command Query Responsibility Segregation) separates read and write operations into different models. Commands handle writes, updates, and deletes with business logic validation. Queries handle reads optimized for fast data retrieval. This separation allows each side to be optimized, scaled, and secured independently. - Q: When should I use CQRS? A: Use CQRS for high-read, low-write applications, complex business logic on writes, when you need different data models for reads vs writes, performance-critical read operations, and event-driven architectures. Avoid it for simple CRUD apps, small teams, or when strong consistency is required. - Q: What is the difference between CQRS and Event Sourcing? A: CQRS is about separating read and write models. Event Sourcing stores all changes as a sequence of events instead of current state. They are often used together but are independent patterns. CQRS can work without Event Sourcing, and Event Sourcing can work without CQRS. - Q: What are the main challenges of implementing CQRS? A: The main challenges are eventual consistency between read and write models (data sync delays), increased complexity with more moving parts, debugging difficulties across separate systems, and additional infrastructure requirements. Start simple and evolve gradually. - Q: How do you handle data consistency in CQRS? A: CQRS typically uses eventual consistency - the read model is updated asynchronously after writes. Handle this with proper event handling, monitoring for sync failures, compensating actions for errors, and designing UIs to accommodate slight delays. Some systems use synchronous updates for critical data. ### cURL Command Explained - URL: https://singhajit.com/explainer/curl-commands/ - Description: Complete cURL command reference for developers. Learn HTTP requests, authentication, file uploads, API testing, and advanced cURL techniques with practical examples and syntax highlighting. FAQ: - Q: What is cURL and what is it used for? A: cURL (Client URL) is a command-line tool for transferring data using various protocols like HTTP, HTTPS, FTP. Developers use it to test APIs, download files, debug web requests, and automate HTTP operations. It's available on Linux, macOS, and Windows. - Q: How do I make a POST request with cURL? A: Use curl -X POST with -d for data: 'curl -X POST -H "Content-Type: application/json" -d '{"name":"value"}' https://api.example.com/endpoint'. Use -d for form data or JSON body. Add -H to set headers like Content-Type. - Q: How do I send headers with cURL? A: Use the -H flag: 'curl -H "Authorization: Bearer token123" -H "Content-Type: application/json" https://api.example.com'. You can add multiple -H flags for multiple headers. Common headers include Authorization, Content-Type, and Accept. - Q: How do I download a file with cURL? A: Use -O to save with the remote filename: 'curl -O https://example.com/file.zip'. Use -o to specify a custom filename: 'curl -o myfile.zip https://example.com/file.zip'. Add -L to follow redirects. - Q: How do I see the response headers with cURL? A: Use -i to include response headers in output: 'curl -i https://example.com'. Use -I (capital i) to fetch only headers without body. Use -v for verbose output showing both request and response headers. ### N+1 Query Problem Explained - URL: https://singhajit.com/explainer/n-plus-one-query-problem/ - Description: Learn about the N+1 query problem through visual examples and practical solutions. Understand why your database queries are slow and how to fix them with eager loading, batching, and other optimization techniques. FAQ: - Q: What is the N+1 query problem? A: The N+1 query problem occurs when code executes 1 query to get a list of N items, then N additional queries to get related data for each item. For 100 users with their posts, that's 1 query for users + 100 queries for posts = 101 database round trips instead of 2 queries with proper loading. - Q: How do I fix the N+1 query problem? A: Use eager loading to fetch related data in one query. In Django use select_related() or prefetch_related(). In Rails use includes(). In SQLAlchemy use joinedload(). For GraphQL, use DataLoader to batch requests. The key is fetching all related data upfront instead of on-demand. - Q: What is the difference between eager loading and lazy loading? A: Lazy loading fetches related data only when accessed - convenient but causes N+1 problems. Eager loading fetches related data immediately with the main query using JOINs or batch queries. Use eager loading when you know you'll need the related data to avoid multiple database round trips. - Q: How do I detect N+1 queries in my application? A: Use query logging to count database queries per request. Tools like Django Debug Toolbar, Bullet gem for Rails, or SQLAlchemy's echo mode show query counts. Look for patterns of similar queries repeated N times. APM tools like New Relic also highlight N+1 issues. - Q: Does N+1 only happen with ORMs? A: N+1 is most common with ORMs due to lazy loading defaults, but it can happen anywhere. GraphQL resolvers, manual loops with database calls, and API calls in loops all suffer from the same pattern. The fix is always batching - fetch data in bulk instead of one at a time. ### Blue-Green vs Canary Deployment Explained - URL: https://singhajit.com/explainer/blue-green-vs-canary-deployment/ - Description: Learn the difference between blue-green and canary deployment strategies. Understand how to deploy your applications safely with zero downtime using practical examples and visual diagrams. FAQ: - Q: What is the difference between blue-green and canary deployment? A: Blue-green deployment runs two identical production environments (blue and green), switching traffic instantly from one to the other. Canary deployment gradually routes a small percentage of traffic to the new version, monitoring for issues before fully rolling out. Blue-green is instant but requires double infrastructure; canary is gradual and safer for catching issues early. - Q: When should I use blue-green deployment vs canary deployment? A: Use blue-green when you need instant rollback capability and have infrastructure to run two full environments. Use canary when you want to minimize risk by testing with real traffic gradually, or when you have limited infrastructure. Canary is better for catching subtle bugs that only appear under production load. - Q: How does blue-green deployment achieve zero downtime? A: Blue-green achieves zero downtime by running the new version (green) alongside the current version (blue) on separate infrastructure. Once green is tested and ready, traffic is switched instantly via load balancer configuration. If issues occur, traffic can be switched back to blue immediately without any downtime. - Q: What are the advantages of canary deployment? A: Canary deployment advantages include: gradual risk reduction by testing with small traffic percentage, early detection of production issues before full rollout, lower infrastructure costs (no need for duplicate full environments), ability to monitor real user behavior, and easy rollback by routing traffic back to old version. - Q: How do you rollback in blue-green vs canary deployment? A: In blue-green, rollback is instant - simply switch traffic back from green to blue via load balancer. In canary, rollback means routing traffic back from the canary (new version) to the stable version, which can be done gradually or instantly depending on your routing configuration. Both allow quick rollback, but blue-green is typically faster. ### WebSockets Explained - URL: https://singhajit.com/explainer/websockets-explained/ - Description: Learn how WebSockets enable real-time, two-way communication between browsers and servers. Understand when to use WebSockets vs traditional HTTP through simple examples and visual diagrams. FAQ: - Q: What is WebSocket and how does it work? A: WebSocket is a protocol that provides persistent, full-duplex (two-way) communication between browser and server over a single TCP connection. Unlike HTTP's request-response model, WebSocket keeps the connection open so both sides can send messages anytime without waiting for the other. - Q: When should I use WebSocket instead of HTTP? A: Use WebSocket for real-time applications: chat apps, live notifications, collaborative editing, gaming, live dashboards, and streaming data. Use HTTP for traditional request-response patterns, RESTful APIs, and when real-time updates aren't needed. WebSocket adds complexity, so only use when necessary. - Q: What is the difference between WebSocket and HTTP polling? A: HTTP polling sends repeated requests to check for updates (wasteful). Long polling keeps requests open until data is available (better but still one-way). WebSocket maintains a permanent two-way connection with minimal overhead. WebSocket is more efficient for frequent updates but requires more infrastructure. - Q: How do I handle WebSocket connection failures? A: Implement automatic reconnection with exponential backoff (1s, 2s, 4s delays). Use heartbeat messages (ping/pong) to detect dead connections. Queue messages during disconnection and resend on reconnect. Consider libraries like Socket.IO that handle reconnection automatically. - Q: Are WebSockets supported by all browsers? A: Yes, all modern browsers support WebSockets (Chrome, Firefox, Safari, Edge). Support has been universal since around 2012. For older environments or when WebSocket is blocked by firewalls, libraries like Socket.IO fall back to HTTP long-polling automatically. ### Kubernetes Resource Units Explained - URL: https://singhajit.com/explainer/kubernetes-resource-units/ - Description: Master Kubernetes resource units - learn the difference between millicores, cores, Mi, Gi, and other CPU and memory notations. Understand what 128Mi, 500m, and other resource values actually mean. FAQ: - Q: What does 500m mean in Kubernetes CPU? A: 500m means 500 millicores, which equals 0.5 CPU cores or 50% of one CPU core. The 'm' suffix stands for milli (1/1000). So 1000m = 1 core, 250m = 0.25 cores (25% of a core), and 100m = 0.1 cores (10% of a core). - Q: What is the difference between Mi and M in Kubernetes memory? A: Mi (Mebibytes) uses binary units (1 Mi = 1024 Ki = 1,048,576 bytes). M (Megabytes) uses decimal units (1 M = 1000 K = 1,000,000 bytes). Kubernetes prefers Mi/Gi (binary) over M/G (decimal). 128Mi is about 134MB. - Q: What is the difference between requests and limits in Kubernetes? A: Requests are guaranteed resources for scheduling - Kubernetes ensures the node has this capacity before placing the pod. Limits are maximum allowed resources - the container gets throttled (CPU) or killed (memory) if exceeded. Set requests for normal operation, limits for burst protection. - Q: What happens if a pod exceeds its memory limit? A: If a container exceeds its memory limit, Kubernetes kills it with an OOMKilled (Out Of Memory) error. The pod may restart based on its restartPolicy. To avoid this, monitor actual memory usage and set limits with appropriate headroom above normal usage. - Q: How do I choose the right resource values for my pod? A: Start by monitoring actual resource usage in development. Set requests to cover normal operation (p50-p75 usage). Set limits 20-50% higher than peak observed usage. Use Vertical Pod Autoscaler recommendations or load testing data. Avoid setting limits equal to requests as this removes burst capacity. ### Change Data Capture (CDC) Explained - URL: https://singhajit.com/explainer/change-data-capture/ - Description: Learn how Change Data Capture (CDC) works to automatically keep your cache updated when database data changes. Understand CDC patterns, use cases, and when to use this approach through simple examples and diagrams. FAQ: - Q: What is Change Data Capture (CDC)? A: Change Data Capture (CDC) is a pattern that tracks changes in your database and automatically updates other systems like caches, search indexes, or data warehouses. Instead of polling for changes, CDC detects changes the moment they happen by reading the database's transaction log. - Q: How does CDC keep cache in sync with database? A: CDC reads the database's transaction log (WAL, binlog) to detect INSERT, UPDATE, and DELETE operations. When a change is detected, CDC sends an event to connected systems like Redis cache, which then updates or invalidates the affected data automatically without any polling. - Q: What are the main CDC methods? A: The three main CDC methods are: Transaction Log reading (reads database WAL/binlog, most efficient), Triggers (database triggers write changes to a table, adds overhead), and Timestamp Polling (queries for rows with updated_at > last_check, not real-time). Transaction log is preferred for production. - Q: What is Debezium and how does it work? A: Debezium is an open-source CDC tool that reads transaction logs from MySQL, PostgreSQL, MongoDB, and SQL Server. It captures row-level changes and streams them to Apache Kafka, allowing downstream systems to consume and react to database changes in real-time. - Q: When should I use CDC vs polling? A: Use CDC when you need real-time data sync, have multiple systems consuming the same data, or want to avoid database load from polling queries. Use polling when data rarely changes, you need simplicity over real-time, or you cannot access database transaction logs. ### Row vs Column Store Explained - URL: https://singhajit.com/explainer/row-vs-column-store/ - Description: Learn the difference between row-oriented and column-oriented database storage. Understand when to use each approach through simple examples and visual diagrams. FAQ: - Q: What is the difference between row store and column store databases? A: Row stores save data by row - each row's columns are stored together (PostgreSQL, MySQL). Column stores save data by column - each column's values are stored together (ClickHouse, Redshift). Row stores are fast for transactional queries fetching entire rows. Column stores are fast for analytical queries aggregating specific columns. - Q: When should I use a columnar database? A: Use columnar databases for analytics, reporting, and data warehousing where you aggregate large datasets across few columns (SUM, AVG, COUNT). They excel at queries like 'total sales by region' that scan millions of rows but only need 2-3 columns. Not ideal for transactional workloads or frequent row updates. - Q: Why are column stores faster for analytics? A: Column stores read only the columns needed for a query, not entire rows. For analytics scanning millions of rows but using 3 columns out of 50, you read 94% less data. Columns also compress better (similar values together) and enable vectorized processing for faster computation. - Q: What are examples of row and column store databases? A: Row stores: PostgreSQL, MySQL, Oracle, SQL Server - traditional OLTP databases. Column stores: ClickHouse, Amazon Redshift, Google BigQuery, Apache Parquet, Snowflake - designed for OLAP and analytics. Some databases like PostgreSQL offer columnar extensions for mixed workloads. - Q: Can I use a column store for my main application database? A: Column stores aren't ideal as primary application databases. They're slow at inserting single rows, updating individual records, and fetching complete rows by ID - common transactional operations. Use a row store for your application and sync data to a column store for analytics. ### Service Discovery Explained - URL: https://singhajit.com/explainer/service-discovery/ - Description: Learn how Service Discovery helps microservices automatically find and talk to each other. Understand the difference between client-side and server-side discovery, when to use each approach, and see real-world examples with simple diagrams. FAQ: - Q: What is Service Discovery in microservices? A: Service Discovery is a mechanism that allows services to automatically find and communicate with each other without hardcoded IP addresses. Services register themselves with a registry (like Consul or Eureka), and other services query the registry to find available instances. This enables dynamic scaling and deployment. - Q: What is the difference between client-side and server-side discovery? A: In client-side discovery, the client queries the service registry directly and chooses which instance to call (e.g., Netflix Eureka). In server-side discovery, the client calls a load balancer/router which queries the registry and forwards requests (e.g., Kubernetes Services, AWS ELB). Server-side is simpler for clients but adds a network hop. - Q: What is a Service Registry? A: A Service Registry is a database of available service instances with their network locations (IP:port). Services register on startup and deregister on shutdown. The registry performs health checks to remove unhealthy instances. Examples include Consul, etcd, ZooKeeper, and Netflix Eureka. - Q: How does Kubernetes handle Service Discovery? A: Kubernetes provides built-in service discovery through DNS and Services. Each Service gets a stable DNS name (service-name.namespace.svc.cluster.local) that resolves to Pod IPs. kube-proxy handles load balancing across healthy pods. No external registry needed - Kubernetes acts as the registry. - Q: When should I use Service Discovery vs a load balancer? A: Service Discovery is essential for dynamic microservices environments where instances frequently scale up/down. Traditional load balancers work for stable services with known endpoints. Modern solutions combine both - Kubernetes Services, Consul Connect, and Istio provide discovery with built-in load balancing. ### Service Mesh Explained - URL: https://singhajit.com/explainer/service-mesh-explained/ - Description: Learn how Service Mesh handles traffic between microservices. Understand what problems it solves, when to use it, and see how it works through simple diagrams and real examples. FAQ: - Q: What is a Service Mesh? A: A Service Mesh is an infrastructure layer that handles service-to-service communication in microservices. It uses sidecar proxies (like Envoy) deployed alongside each service to manage traffic routing, load balancing, encryption, observability, and retry logic - all without changing application code. - Q: What is a sidecar proxy in a service mesh? A: A sidecar proxy is a container that runs alongside your application container in the same pod. All incoming and outgoing traffic flows through it. The proxy handles mTLS encryption, load balancing, retries, circuit breaking, and metrics collection transparently. Envoy is the most common sidecar proxy. - Q: When should I use a Service Mesh? A: Use a service mesh when you have many microservices (20+) needing consistent observability, security (mTLS), and traffic management. It's valuable for canary deployments, circuit breaking, and distributed tracing. Avoid for simple architectures - the complexity overhead isn't worth it for 5-10 services. - Q: What is the difference between Istio and Linkerd? A: Istio is feature-rich with advanced traffic management and security policies but is more complex and resource-heavy. Linkerd is lightweight, simpler to operate, and uses less resources with its Rust-based proxy. Choose Linkerd for simplicity, Istio for advanced features and enterprise requirements. - Q: Does a service mesh replace API Gateway? A: No, they serve different purposes. API Gateways handle north-south traffic (external clients to services) with authentication, rate limiting, and API management. Service Mesh handles east-west traffic (service to service) with mTLS, observability, and traffic control. Many architectures use both together. ### DNS Records Explained - URL: https://singhajit.com/explainer/dns-records-explained/ - Description: Learn about different DNS record types like A, AAAA, CNAME, MX, TXT, and NS records. Understand what each record does, when to use them, and how they work together to make the internet function. FAQ: - Q: What is the difference between A and CNAME records? A: An A record points a domain directly to an IP address (example.com → 192.168.1.1). A CNAME record points a domain to another domain name (www.example.com → example.com). Use A for root domains, CNAME for subdomains that should follow another domain. - Q: What is an MX record used for? A: MX (Mail Exchange) records specify which mail servers handle email for your domain. They include a priority number - lower numbers have higher priority. When someone sends email to your domain, their mail server looks up your MX records to find where to deliver it. - Q: What is a TXT record and why do I need it? A: TXT records store text data for various purposes: SPF records for email authentication, DKIM signatures, domain verification for services like Google Workspace, and DMARC policies. They're commonly required when setting up email or verifying domain ownership. - Q: What is TTL in DNS records? A: TTL (Time To Live) specifies how long DNS resolvers should cache a record before requesting fresh data. Lower TTL (300 seconds) means faster propagation of changes but more DNS queries. Higher TTL (86400 seconds) reduces queries but delays change propagation. - Q: What are NS records and do I need to change them? A: NS (Nameserver) records specify which DNS servers are authoritative for your domain. You typically set these when registering your domain or changing DNS providers. They point to your DNS host (like ns1.cloudflare.com) who manages your other DNS records. ### Passkeys Explained - URL: https://singhajit.com/explainer/passkeys-explained/ - Description: Learn how passkeys replace passwords with a simple, secure way to log in. Understand how they work, why they're safer than passwords, and when to use them. FAQ: - Q: What is a passkey and how does it work? A: A passkey is a cryptographic credential that replaces passwords. It uses public-key cryptography: your device stores a private key (unlocked by biometrics or PIN), and the website stores the public key. During login, your device signs a challenge with the private key, proving identity without sending any secret. - Q: Are passkeys safer than passwords? A: Yes, passkeys are significantly safer. They can't be phished (tied to specific domains), can't be leaked in database breaches (sites only have public keys), can't be guessed or cracked, and don't need to be remembered or typed. Each passkey is unique per site, eliminating password reuse risks. - Q: What happens if I lose my device with passkeys? A: Passkeys can sync across devices via iCloud Keychain, Google Password Manager, or 1Password. If synced, access them from any linked device. If not synced, use account recovery options. Many sites let you register multiple passkeys or keep a backup authentication method. - Q: What is the difference between passkeys and hardware security keys? A: Hardware security keys (like YubiKey) are physical devices you plug in or tap. Passkeys can be stored on phones/computers and synced via cloud. Both use the same WebAuthn/FIDO2 standard. Hardware keys are slightly more secure (no cloud sync), passkeys are more convenient. - Q: Which websites and apps support passkeys? A: Major services supporting passkeys include Google, Apple, Microsoft, GitHub, PayPal, eBay, Best Buy, and many more. Support is growing rapidly. Check passkeys.directory for a current list of supporting services. ### Vector Databases and RAG Explained - URL: https://singhajit.com/explainer/vector-databases-and-rag/ - Description: Learn how vector databases and RAG help AI remember your data. Understand what they are, how they work together, and when to use them through simple examples and diagrams. FAQ: - Q: What is RAG (Retrieval Augmented Generation)? A: RAG is a technique that enhances AI responses by retrieving relevant information from your data before generating answers. Instead of relying only on training data, the AI searches a knowledge base for context, then uses that information to give accurate, up-to-date responses specific to your content. - Q: What is a vector database? A: A vector database stores data as numerical vectors (embeddings) that capture semantic meaning. Unlike traditional databases that match exact keywords, vector databases find similar content by measuring distance between vectors. This enables semantic search - finding content by meaning, not just matching words. - Q: How do embeddings work in vector search? A: Embeddings convert text into numerical vectors (arrays of numbers) using AI models. Similar meanings produce similar vectors. 'dog' and 'puppy' have close vectors, while 'dog' and 'computer' are far apart. Vector databases search by finding vectors closest to your query's embedding. - Q: What is the difference between RAG and fine-tuning? A: RAG retrieves relevant context at query time from external data. Fine-tuning trains the model on your data, changing its weights. RAG is cheaper, faster to update, and works with any data. Fine-tuning is better for teaching new behaviors or styles. Many applications combine both approaches. - Q: Which vector database should I use? A: Popular choices: Pinecone (fully managed, easy to start), Weaviate (open-source, feature-rich), Milvus (high performance, self-hosted), ChromaDB (lightweight, Python-native), pgvector (PostgreSQL extension). For prototypes use ChromaDB. For production, evaluate based on scale, hosting preference, and features needed. ### SLI, SLO, and SLA Explained - URL: https://singhajit.com/explainer/sli-slo-sla-explained/ - Description: Learn what SLI, SLO, and SLA mean and how they work together. Understand how to measure service quality, set targets, and make promises to users through simple examples and diagrams. FAQ: - Q: What is the difference between SLI, SLO, and SLA? A: SLI (Service Level Indicator) is what you measure - like latency or error rate. SLO (Service Level Objective) is your internal target - like 99.9% availability. SLA (Service Level Agreement) is the external promise to customers with consequences - like refunds if uptime drops below 99.5%. - Q: What is an Error Budget and how does it work? A: Error Budget is the allowed amount of unreliability based on your SLO. With 99.9% availability SLO, your error budget is 0.1% downtime (~43 minutes/month). Teams spend error budget on deployments and changes. When budget is exhausted, focus shifts to reliability over new features. - Q: How do I choose good SLIs for my service? A: Choose SLIs that reflect user experience. For APIs: request latency (p99), error rate, and availability. For data pipelines: freshness and correctness. For storage: durability and read latency. Start with 3-5 key SLIs. More isn't better - focus on what users actually care about. - Q: What is the difference between availability and uptime? A: Uptime measures if the service is running. Availability measures if the service is working correctly for users. A service can have 100% uptime but poor availability if it's responding with errors or too slowly. SLOs should measure availability from the user's perspective, not just uptime. - Q: How do I set realistic SLO targets? A: Start by measuring current performance as a baseline. Set SLOs slightly below current performance to allow for variations. Consider dependencies - your SLO can't exceed your dependencies' reliability. Use 9s carefully: 99.9% (43 min/month downtime) is achievable, 99.99% (4 min/month) is very hard. ### Concurrency vs Parallelism Explained - URL: https://singhajit.com/explainer/concurrency-vs-parallelism/ - Description: Learn the difference between concurrency and parallelism. Understand when to use each approach through simple examples, visual diagrams, and real-world analogies. FAQ: - Q: What is the difference between concurrency and parallelism? A: Concurrency is about managing multiple tasks by switching between them on a single CPU core - tasks make progress but don't run simultaneously. Parallelism is about actually executing multiple tasks at the exact same time on different CPU cores. Concurrency deals with structure, parallelism deals with execution. - Q: When should I use concurrency vs parallelism? A: Use concurrency for I/O-bound tasks like web requests, file operations, and database queries where tasks spend time waiting. Use parallelism for CPU-bound tasks like video encoding, data processing, and scientific calculations where you need raw computational power across multiple cores. - Q: Can a single-core CPU do parallel processing? A: No, true parallelism requires multiple CPU cores to execute tasks simultaneously. A single-core CPU can only do concurrency by rapidly switching between tasks, creating the illusion of multitasking. For parallelism, you need multiple cores, each handling a different task at the same time. - Q: What is async/await and is it concurrency or parallelism? A: Async/await is a concurrency pattern. It allows a single thread to manage multiple tasks by switching between them when one task is waiting (like for a network response). The tasks don't run simultaneously - the thread handles one at a time but doesn't block while waiting. - Q: Can you use both concurrency and parallelism together? A: Yes, many applications combine both. For example, a web server might handle 1000 concurrent connections on each of 4 parallel worker processes. Use concurrency for I/O operations and parallelism for CPU-intensive work. Python's asyncio with multiprocessing is a common example. ### Linux Directory Structure Explained - URL: https://singhajit.com/explainer/linux-directory-structure/ - Description: Learn what each directory in Linux is for and why it matters. Understand where programs, config files, and user data live through simple examples and diagrams. FAQ: - Q: What is the difference between /bin and /usr/bin in Linux? A: /bin contains essential system binaries needed for single-user mode and system recovery (ls, cp, cat). /usr/bin contains user command binaries for normal operation (git, python, vim). Historically separated for disk space, modern systems often merge them with symlinks. - Q: Where are configuration files stored in Linux? A: System-wide configuration files are in /etc (like /etc/nginx/nginx.conf, /etc/ssh/sshd_config). User-specific configs are in home directories, often as dotfiles (~/.bashrc, ~/.gitconfig) or in ~/.config/ following XDG standards. - Q: What is the /var directory used for in Linux? A: /var stores variable data that changes during system operation: log files (/var/log), mail (/var/mail), databases (/var/lib/mysql), web content (/var/www), and temporary files (/var/tmp). It grows over time and needs monitoring for disk space. - Q: What is the difference between /tmp and /var/tmp? A: /tmp is for temporary files that can be deleted on reboot - many systems mount it as tmpfs (RAM-based). /var/tmp is for temporary files that should persist between reboots. Programs needing temporary storage that survives restarts use /var/tmp. - Q: Where should I install custom software on Linux? A: Install system-wide custom software to /usr/local (binaries in /usr/local/bin, libs in /usr/local/lib). For user-only software, use ~/bin or ~/.local/bin. Package managers use /usr for their packages. Avoid modifying /usr directly. ### MCP Explained - URL: https://singhajit.com/explainer/mcp-explained/ - Description: Learn what Model Context Protocol (MCP) is and how it helps AI assistants connect to external tools, databases, and services. Understand when to use MCP through simple examples and diagrams. FAQ: - Q: What is Model Context Protocol (MCP)? A: MCP (Model Context Protocol) is an open standard created by Anthropic that allows AI assistants to connect with external tools, databases, and services. It provides a standardized way for LLMs to access real-time data, execute functions, and interact with external systems without custom integrations. - Q: How is MCP different from function calling? A: Function calling requires developers to define functions in each AI application. MCP provides a universal protocol so tools can be built once and work with any MCP-compatible AI assistant. Think of function calling as app-specific, while MCP is a shared standard across the ecosystem. - Q: What can you do with MCP servers? A: MCP servers can provide AI assistants with access to databases, file systems, APIs, browser automation, code execution, and any custom functionality. Examples include querying databases, reading local files, controlling web browsers, running shell commands, and integrating with external services. - Q: Which AI assistants support MCP? A: Claude Desktop and Cursor IDE natively support MCP. The protocol is open-source, so any AI application can implement MCP client support. More tools are adding MCP compatibility as the standard gains adoption. - Q: How do I create my own MCP server? A: You can build MCP servers using the official SDKs for Python or TypeScript. Define your tools as functions with input schemas, implement the logic, and run the server. The MCP SDK handles the protocol communication with AI clients automatically. ### Snowflake IDs Explained - URL: https://singhajit.com/explainer/snowflake-ids-explained/ - Description: Learn how Snowflake IDs work and why they're used in distributed systems. Understand the structure, how they're generated, and why companies like Twitter and Discord use them. FAQ: - Q: What is a Snowflake ID? A: A Snowflake ID is a 64-bit unique identifier used in distributed systems. It combines a timestamp (41 bits), machine/datacenter ID (10 bits), and sequence number (12 bits). This structure allows multiple servers to generate unique IDs independently without coordination, while keeping IDs roughly time-ordered. - Q: Why use Snowflake IDs instead of UUIDs? A: Snowflake IDs are smaller (64-bit vs 128-bit UUID), time-sortable (can sort by creation time), and more efficient as database primary keys. UUIDs are random, causing poor index performance. Snowflake IDs maintain locality of reference and work better with B-tree indexes. - Q: How do Snowflake IDs stay unique across servers? A: Each server gets a unique machine ID (10 bits = 1024 possible IDs). Combined with millisecond timestamp and 12-bit sequence (4096 IDs per millisecond), each server can generate 4 million unique IDs per second without any coordination with other servers. - Q: Can you extract the timestamp from a Snowflake ID? A: Yes, the first 41 bits contain milliseconds since a custom epoch. Right-shift the ID by 22 bits, add the epoch timestamp, and you get the creation time. This is useful for time-based queries and debugging. Discord and Twitter both expose this feature. - Q: What happens when Snowflake ID sequence overflows? A: If a server generates more than 4096 IDs in the same millisecond (sequence exhausted), it waits until the next millisecond to continue. This is rare in practice - 4096 IDs per millisecond per server is extremely high throughput. The wait ensures uniqueness is never compromised. ### Cron Jobs Explained - URL: https://singhajit.com/explainer/cron-jobs-explained/ - Description: Learn how cron jobs work and how to write cron expressions. A simple guide to scheduling automated tasks on Linux and Unix systems with examples. FAQ: - Q: What is a cron job? A: A cron job is a scheduled task that runs automatically at specific times on Linux and Unix systems. You define when to run using a cron expression (5 fields for minute, hour, day, month, weekday) and what command to execute. The cron daemon checks every minute and runs matching tasks. - Q: How do I write a cron expression? A: A cron expression has 5 fields separated by spaces: minute (0-59), hour (0-23), day of month (1-31), month (1-12), and day of week (0-6, Sunday=0). Use * for every value, */n for every n units, comma for lists, and hyphen for ranges. Example: 0 9 * * 1-5 means 9 AM on weekdays. - Q: How do I view and edit my cron jobs? A: Use 'crontab -l' to list your cron jobs, 'crontab -e' to edit them in a text editor, and 'crontab -r' to remove all jobs. Each user has their own crontab. System-wide cron jobs are in /etc/crontab and /etc/cron.d/ directory. - Q: What does */15 mean in a cron expression? A: The /15 is a step value meaning 'every 15 units'. So */15 * * * * means every 15 minutes (at 0, 15, 30, 45 minutes past each hour). You can use steps with ranges too: 0-30/10 means every 10 minutes from 0 to 30 (0, 10, 20, 30). - Q: Why is my cron job not running? A: Common reasons: wrong timezone (cron uses system time), missing execute permissions on script, environment variables not set (cron has minimal PATH), wrong file paths (use absolute paths), or syntax errors. Check /var/log/syslog or /var/log/cron for error messages. ### Regular Expressions Explained - URL: https://singhajit.com/explainer/regex-explained/ - Description: Learn how regular expressions (regex) work with simple examples. Understand pattern matching, special characters, and common use cases. A beginner-friendly guide to regex basics. FAQ: - Q: What is a regular expression (regex)? A: A regular expression is a pattern that describes a set of strings. Think of it as a search query on steroids - instead of searching for exact text, you search for patterns. For example, you can find all email addresses in a document without knowing what they are beforehand. - Q: Why should I learn regex? A: Regex saves hours of manual work. You can validate user input (emails, phone numbers), search through logs, clean up data, and do find-and-replace across thousands of files. Every programming language supports regex. - Q: What does the dot (.) mean in regex? A: The dot matches any single character except newline. So 'c.t' matches 'cat', 'cut', 'c9t', or 'c@t' - any three-character string starting with 'c' and ending with 't'. - Q: What is the difference between * and + in regex? A: Both are quantifiers. The asterisk (*) matches zero or more of the previous character - so 'a*' matches '', 'a', 'aa', 'aaa'. The plus (+) matches one or more - so 'a+' requires at least one 'a'. It won't match an empty string. - Q: How do I match the start or end of a line? A: Use ^ for the start and $ for the end. The pattern ^Hello matches 'Hello world' but not 'Say Hello'. The pattern world$ matches 'Hello world' but not 'world peace'. - Q: What are capture groups in regex? A: Capture groups are created with parentheses (). They save the matched text so you can reference it later. For example, in the pattern (hello)-(world), the first group captures 'hello' and the second captures 'world'. Use groups to extract specific parts of a match. ### Zero-Day Vulnerability Explained - URL: https://singhajit.com/explainer/zero-day-vulnerability/ - Description: Learn what zero-day vulnerabilities are and why they're so dangerous. Understand how attackers exploit them, how they get discovered, and what you can do to stay safe. FAQ: - Q: What is a zero-day vulnerability? A: A zero-day vulnerability is a security flaw in software that the vendor doesn't know about yet. When attackers find it before the vendor does, they can exploit it freely because there's no patch available. The term 'zero-day' refers to the fact that developers have had zero days to fix the problem. - Q: Why are zero-day vulnerabilities so dangerous? A: Zero-day vulnerabilities are dangerous because there's no defense against them. Since the vendor doesn't know the flaw exists, there's no patch or fix available. Antivirus software and security tools may not detect attacks using unknown vulnerabilities. Users are completely exposed until someone discovers the flaw and a patch is released. - Q: How do zero-day vulnerabilities get discovered? A: Zero-day vulnerabilities can be discovered by security researchers, hackers, or sometimes by accident. Security researchers often report them responsibly to vendors. Hackers may sell them on the black market or use them in attacks. Some governments stockpile zero-days for cyber operations. - Q: How can I protect myself from zero-day attacks? A: Keep your software updated so you get patches quickly once released. Use multiple layers of security like firewalls and endpoint protection. Be careful with email attachments and suspicious links. Use browsers with sandboxing. Consider security software that detects unusual behavior, not just known threats. - Q: What happens after a zero-day is discovered? A: Once discovered, the vendor works on a patch. They may release a temporary workaround while developing a fix. After the patch is released, it becomes a 'known vulnerability' and the race begins - defenders rush to patch while attackers target unpatched systems. ### Quartz Cron Expressions Explained - URL: https://singhajit.com/explainer/quartz-cron-explained/ - Description: Learn how Quartz cron expressions work and how they differ from standard cron. Understand the 6-field format, special characters like ?, L, W, #, and where Quartz cron is used. FAQ: - Q: What is a Quartz cron expression? A: A Quartz cron expression is a 6 or 7 field string used to schedule jobs in Java applications. The fields are: second, minute, hour, day-of-month, month, day-of-week, and optionally year. It's used by Java Quartz Scheduler, Spring Boot @Scheduled, and other enterprise schedulers. - Q: How is Quartz cron different from standard cron? A: Standard Unix cron has 5 fields starting from minute. Quartz has 6-7 fields starting from second. Quartz also adds special characters: ? (no specific value), L (last), W (nearest weekday), and # (nth weekday). Day-of-week numbering is 1-7 (Sunday=1) instead of 0-6. - Q: What does the ? mean in Quartz cron? A: The question mark means 'no specific value'. You must use it in either day-of-month or day-of-week (not both). It tells Quartz to not constrain on that day field. For example, 0 0 9 ? * 2-6 means 9 AM on weekdays - the ? says ignore day-of-month. - Q: What does L mean in Quartz cron? A: L means 'last'. In the day-of-month field, L means the last day of the month. In day-of-week, 6L means the last Friday of the month. You can also use L-3 to mean 3 days before the last day. - Q: What does # mean in Quartz cron? A: The # character lets you pick the Nth occurrence of a weekday. For example, 6#3 means the 3rd Friday of the month (Friday is 6 in Quartz where Sunday=1). Valid values are 1-5 for the occurrence number. ### Linux File Permissions Explained - URL: https://singhajit.com/explainer/linux-file-permissions-explained/ - Description: Learn how Linux file permissions work. Understand read, write, execute permissions, owner, group, others, octal notation like 755 and 644, and the chmod command with simple examples. FAQ: - Q: What does chmod 755 mean? A: chmod 755 means the owner can read, write, and execute the file (7). The group can read and execute but not write (5). Others can also read and execute but not write (5). This is the most common permission for scripts and executable files. - Q: What is the difference between chmod 644 and chmod 755? A: chmod 644 gives the owner read and write access, while group and others can only read. chmod 755 gives the owner full access, while group and others can read and execute. Use 644 for regular files and 755 for scripts or programs that need to be executed. - Q: How do I check file permissions in Linux? A: Use the ls -l command. The output shows permissions as a 10-character string like -rwxr-xr-x. The first character is the file type, then three groups of rwx for owner, group, and others. - Q: What does rwx mean in Linux? A: r means read (view file contents), w means write (modify or delete the file), and x means execute (run the file as a program). A dash (-) means that permission is not granted. - Q: How do I make a file executable in Linux? A: Run chmod +x filename to add execute permission for everyone. Or use chmod u+x filename to add it only for the owner. You can also use octal notation like chmod 755 filename. --- ## Developer Tools All tools run 100% client-side in the browser. No data is sent to servers. ### JSON Validator & Formatter - URL: https://singhajit.com/tools/json-validator/ - Purpose: Validate JSON syntax with detailed error messages showing line and column numbers. Format and beautify JSON data. ### CSV to JSON Converter - URL: https://singhajit.com/tools/csv-to-json/ - Purpose: Convert CSV or TSV text and files into pretty-printed JSON arrays of objects or arrays. Auto-detect delimiters, optional headers, copy and download. ### Regex Tester & Debugger - URL: https://singhajit.com/tools/regex-tester/ - Purpose: Test regular expressions with real-time highlighting, pattern explanation, and capture group visualization. ### XPath Tester - URL: https://singhajit.com/tools/xpath-tester/ - Purpose: Test XPath 1.0 expressions against XML or HTML. View matching nodes, paths, snippets, and scalar string/number/boolean results. ### HMAC Generator & Verifier - URL: https://singhajit.com/tools/hmac-generator/ - Purpose: Generate and verify HMAC-SHA1, SHA-256, SHA-384, and SHA-512 digests with hex or Base64 output using the Web Crypto API. ### Cron Expression Generator - URL: https://singhajit.com/tools/cron-expression/ - Purpose: Parse, validate, and build cron job expressions visually. Translate cron expressions to human-readable text. ### Base64 Encoder & Decoder - URL: https://singhajit.com/tools/base64-encoder/ - Purpose: Encode text to Base64 or decode Base64 to text. Supports UTF-8 characters including emojis. ### Base64 to PDF Converter - URL: https://singhajit.com/tools/base64-to-pdf/ - Purpose: Convert Base64 to PDF with in-browser preview and download, or encode uploaded PDF files to Base64 or data URLs. ### Snowflake ID Decoder - URL: https://singhajit.com/tools/snowflake-decoder/ - Purpose: Decode Discord, Twitter, and Instagram Snowflake IDs to timestamps. Visualize the 64-bit structure. ### Epoch Timestamp Converter - URL: https://singhajit.com/tools/epoch-converter/ - Purpose: Convert Unix timestamps to human-readable dates and vice versa. Supports seconds and milliseconds. ### SLA Uptime Calculator - URL: https://singhajit.com/tools/sla-calculator/ - Purpose: Calculate allowed downtime for any SLA uptime percentage. Understand what "five nines" really means. ### URL Encoder & Decoder - URL: https://singhajit.com/tools/url-encoder/ - Purpose: Encode text for URLs using percent encoding or decode URL-encoded strings. --- ## Cheat Sheets ### MongoDB Cheat Sheet: 100+ Commands with Real Examples - URL: https://singhajit.com/mongodb-cheat-sheet/ - Date: 2026-03-20 - Description: Master MongoDB with this practical cheat sheet. Covers mongosh commands, CRUD operations, query operators, aggregation pipeline, indexes, schema validation, replication, sharding, backup and restore, and performance tuning. Real examples for software developers. ### Docker Cheat Sheet: 100+ Commands with Real Examples - URL: https://singhajit.com/devops/docker-cheat-sheet/ - Date: 2026-02-20 - Description: Master Docker with this practical cheat sheet. Covers docker commands, container management, image building, Dockerfile best practices, docker-compose, networking, volumes, Docker Hub, and troubleshooting. Real examples for software developers. ### PostgreSQL Cheat Sheet: 100+ Commands with Real Examples - URL: https://singhajit.com/postgresql-cheat-sheet/ - Date: 2026-02-18 - Description: Master PostgreSQL with this practical cheat sheet. Covers psql commands, database management, table operations, queries, indexes, joins, JSON, backup and restore, performance tuning with EXPLAIN ANALYZE, roles, and troubleshooting. Real examples for software developers. ### Kubernetes Cheat Sheet: Essential kubectl Commands for Developers - URL: https://singhajit.com/kubernetes-cheat-sheet/ - Date: 2026-02-02 - Description: Complete Kubernetes cheat sheet with essential kubectl commands, debugging workflows, and practical examples. Learn pod management, deployments, services, and troubleshooting for container orchestration. ### System Design Cheat Sheet: Concepts Every Developer Should Know - URL: https://singhajit.com/system-design-cheat-sheet/ - Date: 2026-01-31 - Description: A practical system design cheat sheet covering scalability, load balancing, caching, database sharding, CAP theorem, and distributed systems patterns. Essential concepts for building systems that scale and preparing for system design interviews. ### Git Cheat Sheet: Commands Every Developer Should Know - URL: https://singhajit.com/git-cheat-sheet/ - Date: 2026-01-11 - Description: A practical Git cheat sheet with examples. Learn essential Git commands for branching, merging, undoing changes, and working with remote repositories. Includes common workflows and tips for solving everyday problems. ### 50+ Linux Commands Cheat Sheet: The Complete Developer Guide - URL: https://singhajit.com/linux-commands-cheat-sheet/ - Date: 2026-01-02 - Description: Master 50+ essential Linux commands with practical examples. Covers file management, process control, networking, permissions, grep, find, SSH, and system administration for developers. ### Regex Cheat Sheet: Patterns Every Developer Should Know - URL: https://singhajit.com/regex-cheat-sheet/ - Date: 2026-01-01 - Description: A practical regex cheat sheet with real examples. Learn regular expression syntax, character classes, quantifiers, lookaheads, and common patterns for email, URL, and phone validation. --- ## All Articles (Complete Index) ### Dev Weekly Aug 31-Sep 6, 2026: GPT-6 Astra Ships, Claude Fable 5.1, Nvidia Buys Hugging Face - URL: https://singhajit.com/dev-weekly/2026/aug-31-sep-6/gpt-6-astra-fable-51-nvidia-hugging-face/ - Date: 2026-09-06 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for August 31 to September 6, 2026: OpenAI ships GPT-6 Astra, Anthropic launches Claude Fable 5.1, Nvidia buys Hugging Face for $12.93 billion, and VS Code 1.136 previews Agent Merge. On September 3 OpenAI released GPT-6 Astra, its first model at the Critical cybersecurity tier, on ChatGPT, the API as gpt-6-astra, Azure, and Bedrock at $10 per million input tokens and $50 per million output. On September 1 Anthropic shipped Claude Fable 5.1 and Mythos 5.1, cutting cache reads 75% to $0.25 per million tokens, and GitHub Copilot added Fable 5.1 the same day. On September 3 Nvidia signed to acquire Hugging Face for $12.93 billion, with close targeted for the first half of 2027. On September 2 Google launched Gemini 3.8 Flash at $0.75/$3.75 introductory pricing and VS Code 1.136 added Agent Merge. Also this week: Python 3.15.0rc2, Copilot can approve pull requests, CISA added PaperCut and seven more KEVs including Starlette and LiteLLM, Anthropic signed a reported $35 billion Lambda cloud deal, AIR raised $50 million, Lyte raised $165 million, and layoffs hit Uber and The Trade Desk. ### How Google manages billions of lines of code in one monorepo - URL: https://singhajit.com/how-google-manages-its-monorepo/ - Date: 2026-09-04 - Tags: system-design, git, devops, software-engineering - Description: How Google manages billions of lines of code in one monorepo using Piper, CitC, Bazel, trunk-based development, automated testing, and large refactors. - Quick Answer: **Google's published engineering papers describe most of its code living in one monorepo**, commonly called google3. **Piper**, a custom version control system built on [Spanner](/how-google-ads-scales-with-spanner/), stores the tree. Developers access it through **CitC**, a cloud workspace that overlays their changed files on the full tree instead of making a traditional clone. Engineers work at **head** using trunk-based development. **Blaze/Bazel** rebuilds affected targets, **Critique** supports review, **TAP** runs tests, and **Rosie** splits large refactors into manageable changes. Android and Chrome use Git outside the main repository because they work with external partners and open source contributors. The 2016 CACM paper's January 2015 snapshot reported about **2 billion lines of code**, **9 million source files**, **35 million commits**, and **86 TB**. ### Claude Code Skills: How to Create and Use Agent Skills - URL: https://singhajit.com/how-to-create-and-use-skills-in-claude-code/ - Date: 2026-09-01 - Tags: AI, claude-code, developer-tools, software-engineering - Description: Learn Claude Code skills from scratch. This hands-on guide covers SKILL.md, .claude/skills, CLAUDE.md vs skills, slash commands, allowed-tools, dynamic context injection, plugins, and how to write a description Claude actually triggers on. - Quick Answer: A **Claude Code skill** is a folder with a `SKILL.md` file that teaches Claude how to do one job, such as summarizing a diff, reviewing a PR, or deploying. Put project skills in `.claude/skills//SKILL.md` and personal skills in `~/.claude/skills//SKILL.md`. The directory name becomes the slash command (`/summarize-changes`). Write a `description` that states what the skill does and when to use it, using the words you would type. Claude loads the skill automatically when the task matches, or you type `/skill-name`. Keep standing rules in `CLAUDE.md`. Keep procedures in skills so the body loads only when needed. For a live snapshot, add a bang-command `git diff HEAD` line so Claude Code runs the command before the model sees the prompt. ### Fixed Partitions Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/fixed-partitions/ - Date: 2026-08-31 - Tags: distributed-systems, system-design, database, software-engineering - Description: Learn the Fixed Partitions pattern in distributed systems: why hash(key) % nodeCount reshuffles almost all your data when a node joins, how a fixed number of logical partitions fixes it, and how Kafka, Redis Cluster, and Cassandra use it. - Quick Answer: The **Fixed Partitions** pattern keeps the number of partitions constant for the life of a cluster. Instead of mapping keys straight to nodes with `hash(key) % nodeCount`, which remaps almost all data when the node count changes, you map keys to a fixed set of logical partitions (say 1024) that never changes, then map those partitions to physical nodes through a separate table. When a node joins or leaves, whole partitions move between nodes but individual keys never rehash, so a resize moves only a small slice of data. This is how [Kafka](/distributed-systems/how-kafka-works/) topic partitions, Redis Cluster's 16384 hash slots, and Cassandra's token ranges keep scaling cheap. ### Dev Weekly Aug 24-30, 2026: OpenAI Cuts Off Cursor, AWS Buys DuckLabs, Grok Bot on Pro - URL: https://singhajit.com/dev-weekly/2026/aug-24-30/openai-cursor-aws-ducklabs-grok-bot/ - Date: 2026-08-30 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for August 24 to 30, 2026: OpenAI cuts models from Cursor, Grok Bot expands to Cursor Pro, AWS buys DuckLabs, and Next.js patches two unauthenticated RCEs. On August 28 OpenAI said it will wind down OpenAI models in Cursor by November 12 after SpaceX closed the Anysphere acquisition. On August 26 SpaceXAI included Grok Bot with SuperGrok, Cursor Pro, and all Cursor Teams plans, with Bot usage on a separate quota. On August 26 Amazon signed to acquire DuckLabs, the Amsterdam company behind DuckDB, while the DuckDB Foundation keeps the MIT-licensed project. On August 25 Vercel published Next.js 16.3.3 and 15.5.24 for an Image Optimization AVIF RCE and CVE-2026-75604 on Windows-hosted servers. On August 25 AWS Lambda opened public preview managed runtimes for Node.js 26 and Python 3.15. Also this week: GitHub Copilot's Customize tab went GA and Copilot CLI moved to a native Rust runtime, The Information reported Nvidia agreed to buy Hugging Face for $12.9 billion, Z.ai open-sourced GLM-5.3-Flash, Qwen shipped Qwen3.8-Flash-Next, IBM released Granite 4.2, Tencent open-sourced Hy4 preview, CISA added Citrix NetScaler and SQL Server bugs to KEV, PHP 8.4.25 landed, Instinct raised a $250 million Series B at $2.5 billion, a16z launched a $1.1 billion Machine Age Fund, and layoffs hit Apple, PagerDuty, and Kneat. ### Cuckoo Filter: A Better Bloom Filter That Supports Deletion - URL: https://singhajit.com/data-structures/cuckoo-filter/ - Date: 2026-08-27 - Tags: data-structures, algorithms - Description: What is a cuckoo filter and how does it work? A clear guide to this probabilistic data structure that supports deletion, uses cuckoo hashing and fingerprints, and often beats the Bloom filter on space and speed. - Quick Answer: A **cuckoo filter** answers 'is X in the set?' by storing a short **fingerprint** of each item in a compact hash table built on **cuckoo hashing**. Like a Bloom filter it can return a false positive but never a false negative, so **NO** means definitely absent and **YES** means probably present. Unlike a standard Bloom filter it **supports deletion**, gives faster lookups (only two buckets to check), and uses less space when the target false positive rate is below about 3%. The trade-off: inserts can fail once the table is roughly 95% full, and you must only delete items you actually added. RedisBloom ships a cuckoo filter (`CF.ADD`, `CF.EXISTS`, `CF.DEL`). ### Dev Weekly Aug 17-23, 2026: Go 1.27 Generic Methods, Cursor Origin, GitHub's Outage, and Stripe's OpenRouter Deal - URL: https://singhajit.com/dev-weekly/2026/aug-17-23/go-127-cursor-origin-github-outage-stripe-openrouter/ - Date: 2026-08-23 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for August 17 to 23, 2026: Go 1.27 ships generic methods, Cursor launches Origin, GitHub's 8-hour outage, and Stripe agrees to buy OpenRouter. On August 19 the Go team released Go 1.27 with generic methods, encoding/json/v2, a goroutine-leak profile, and faster small-object allocation. On August 17 Cursor began rolling Origin, its Git forge, to paid users, hours before GitHub degraded for 7 hours and 47 minutes after an Istio sidecar hit its concurrency limit and a VS Code retry bug amplified Copilot traffic. On August 17 Redis 8.10.1 closed an RDB loading path that can lead to remote code execution, plus a TLS certificate authentication bypass. On August 19 Stripe agreed to acquire OpenRouter, reported at about $7.5 billion. Also this week: Anthropic took computer use, browser use, Files, and Skills out of beta on the Claude API, Rust 1.98.0 landed, AWS Glue 6.0 went GA at 30% lower price with Iceberg v3, JFrog flagged compromised crates.io packages that ran malware at cargo build, Cloudflare published a remote Spectre attack on Workers that leaked JWTs at 12 bit/s, CISA added exploited bugs in Windows IKE, SharePoint, vCenter, and macOS, Groq raised $350 million, Dash0 bought Polar Signals, Rillet hit a $1 billion valuation, and layoffs hit TikTok Shop and Qualtrics. ### Dropbox System Design: How Cloud File Storage Works - URL: https://singhajit.com/dropbox-system-design/ - Date: 2026-08-20 - Tags: system-design, distributed-systems, storage, software-engineering - Description: A plain-language Dropbox system design walkthrough. Learn how file chunking, deduplication, delta sync, a metadata service, and object storage combine to build a scalable cloud file storage and file sync service. - Quick Answer: A Dropbox-style service splits each file into fixed-size **chunks** (blocks), fingerprints every chunk with a hash, and stores the raw blocks in **object storage** while a separate **metadata service** tracks which chunks make up each file version. **Deduplication** skips uploading chunks the system already has, and **delta sync** uploads only the chunks that changed on edit. A **notification service** tells other devices a change happened, and clients pull the new metadata and only the missing blocks. This separation of block data from metadata is what makes the system scale. ### How Java Debugging Works: Inside JPDA, JDWP, and JDI - URL: https://singhajit.com/how-java-debugging-works/ - Date: 2026-08-17 - Tags: java - Description: How does Java debugging work? A plain-language guide to the Java Platform Debugger Architecture: JVM TI, the JDWP protocol, the JDI API, breakpoints, and remote debugging with -agentlib:jdwp. - Quick Answer: Java debugging is built on the **Java Platform Debugger Architecture (JPDA)**, which has three layers. At the bottom, **JVM TI** is a native interface inside the JVM that can pause threads, read variables, and set breakpoints. In the middle, **JDWP (Java Debug Wire Protocol)** is the format for debug requests and events that travels between the debugged process and your debugger, usually over a socket. At the top, **JDI (Java Debug Interface)** is the Java API that IDEs like IntelliJ and Eclipse use to build breakpoints, stepping, and variable inspection. You turn it on by starting the JVM with `-agentlib:jdwp=...`, which loads the debug agent so a debugger can attach locally or across a network. ### Dev Weekly Aug 10-16, 2026: Gemini 3.7 Flash, Grok 4.6, Patch Tuesday's 400 Fixes, and Anthropic's $2 Trillion IPO - URL: https://singhajit.com/dev-weekly/2026/aug-10-16/gemini-3-7-flash-grok-4-6-patch-tuesday-anthropic-2t-ipo-chaindrop/ - Date: 2026-08-16 - Tags: dev-weekly, tech-news, software-development-news - Description: Google ships Gemini 3.7 Flash at half price, xAI's Grok 4.6 reaches GitHub Copilot, Microsoft patches 400 flaws on Patch Tuesday, the ChainDrop worm rips through npm, and Anthropic's backers model a $2 trillion IPO. On August 13 Google released Gemini 3.7 Flash for coding and agents at an introductory $0.75 per million input tokens, half its predecessor. On August 12 xAI launched Grok 4.6 in Cursor and Grok Build, and on August 14 it arrived in GitHub Copilot across eight surfaces. On August 11 Microsoft's Patch Tuesday fixed 400 vulnerabilities including an actively exploited AFD.sys zero-day used by Lazarus and a wormable DNS bug. The ChainDrop npm worm poisoned 444 packages and burrowed into Claude Code and VS Code configs, with fresh analysis landing August 11 to 15. On August 13 Anthropic's investors modeled a $2 trillion October IPO on projected 2028 revenue, and the company was reported in talks to buy Decart for about $6 billion. OpenAI brought its ChatGPT and Codex desktop app to Linux in preview on August 11, Anthropic detailed how it will watermark Claude's text to comply with the EU AI Act, and OpenAI previewed GPT-5.6 Sol Ultrafast on Cerebras. Also this week: PostgreSQL 18.6 and PG19 Beta 3, Go and Python security releases, a near-autonomous AI cyberattack on Taiwan's nuclear safety agency, Claude Code auto mode becoming the default, Lovable's $400 million Series C at $13.3 billion, River AI's $1.1 billion seed, Dynatrace buying Arize for $915 million, and layoffs at Rapid7, Netflix's game studios, and CD Projekt Red. ### How the JVM Works: From Bytecode to Native Code - URL: https://singhajit.com/how-jvm-works/ - Date: 2026-08-11 - Tags: java - Description: How does the JVM work? A plain-language guide to how the Java Virtual Machine executes code: class loading, bytecode verification, runtime memory, the interpreter, JIT compilation, and garbage collection. - Quick Answer: The **JVM (Java Virtual Machine)** runs the `.class` bytecode produced by `javac`. It works in three stages. First the **class loader** loads, links, and initializes your classes into memory. Then the JVM lays out **runtime data areas** like the heap (objects), the stacks (method calls), and the method area (class metadata). Finally the **execution engine** runs the bytecode: it starts by interpreting instructions one by one, then the **JIT compiler** turns frequently used hot methods into optimized native machine code, while the **garbage collector** reclaims memory from unused objects in the background. This design is what gives Java its Write Once, Run Anywhere promise. ### Dev Weekly Aug 3-9, 2026: Meta Ships Muse Code, GPT-5.6 Luna Becomes the Free ChatGPT Default, Qwen 3.8-Max Goes GA, Next.js 16.3, and a Black Hat GitHub-Issue RCE in Claude Code, Gemini CLI, and Codex - URL: https://singhajit.com/dev-weekly/2026/aug-3-9/muse-code-gpt-56-luna-qwen-38-max-nextjs-16-3-black-hat-ai-coding-rce/ - Date: 2026-08-09 - Tags: dev-weekly, tech-news, software-development-news - Description: Meta ships Muse Code, GPT-5.6 Luna becomes the free ChatGPT default, Qwen 3.8-Max goes GA, Next.js 16.3 lands, and Black Hat researchers turn a public GitHub issue into remote code execution across AI coding agents. On August 5 Meta released Muse Code, a terminal and CI coding agent on the new Muse Spark 1.2 model, with persistent subagents in isolated git worktrees and a crash-safe event log. On August 6 OpenAI retuned GPT-5.6 Sol for chat with a new effort slider and made GPT-5.6 Luna the default for Free and Go users, with unlimited text chats and a Think button following next week. On August 3 Alibaba made Qwen 3.8-Max generally available, a 2.4 trillion parameter model with a promised open-weight release, and Next.js 16.3 shipped Instant Navigations, a lighter dev server, and an experimental Rust-based React Compiler. On August 7 Anthropic said Claude Code auto mode becomes the default for Pro, Max, and Team plans on August 14, and on August 6 Kimi K3 reached general availability in GitHub Copilot. On the security side, Novee Security showed at Black Hat USA 2026 that a single public GitHub issue could drive Claude Code, Gemini CLI, and Codex into remote code execution and credential theft, patched in Claude Code 2.1.163 and Gemini CLI 0.39.1, while CISA added actively exploited flaws in JetBrains TeamCity, IBM Langflow, Apache Tomcat, and N-able N-central to its KEV catalog, and Zenity disclosed trojanized AI agent skills that racked up 1.7 million installs. Also this week: Django 6.1, ClickHouse 26.7, Google's reported $1.5 billion Mechanize deal, AMD's acquisition of Taalas, HappyRobot's $150 million Series C, and layoffs at Etsy, Google, Salesforce, and Zillow. ### Emergent Leader Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/emergent-leader/ - Date: 2026-08-07 - Tags: distributed-systems - Description: Learn the Emergent Leader pattern in distributed systems: how peer-to-peer clusters like Akka, Hazelcast, and JGroups pick a coordinator by node age instead of running an election, how gossip and heartbeats keep the choice in sync, and when this is safe versus when you need Raft or a consistent core. - Quick Answer: The **Emergent Leader** pattern lets a peer-to-peer cluster pick a coordinator without running any election. Every node shares its membership view through a gossip protocol, and once all nodes agree on the member list, they sort it by each node's age in the cluster. The oldest member automatically becomes the coordinator. Nobody votes. Each node computes the same answer locally because they all sort the same list the same way. When the oldest node dies, heartbeats detect it and the next oldest node quietly takes over. Akka Cluster, Hazelcast, JGroups, and Apache Ignite all use this to run cluster management tasks like assigning partitions and tracking membership. ### Columnar Databases Explained: ClickHouse, BigQuery, and Redshift - URL: https://singhajit.com/columnar-databases-explained/ - Date: 2026-08-05 - Tags: database, analytics, system-design, data-engineering - Description: A clear guide to columnar databases like ClickHouse, BigQuery, and Redshift. Learn how column-oriented storage, compression, and vectorized execution power fast OLAP analytics. - Quick Answer: A **columnar database** stores each column of a table together on disk instead of each row. That layout lets an analytical query read only the few columns it needs, skip the rest, and compress heavily because similar values sit next to each other. The result is analytical queries that run 10 to 100 times faster than on a row store. **ClickHouse, DuckDB, Google BigQuery, Amazon Redshift, and Snowflake** are the main examples. Use one for dashboards, reporting, and large scans (OLAP). Do not use one for single-row lookups and frequent updates (OLTP); a row store like PostgreSQL wins there. ### Dev Weekly Jul 27-Aug 2, 2026: Node.js Ships Emergency Security Releases, Redis Patches AI-Found Zero-Days, DeepSeek V4 Flash Gets an Agentic Upgrade, Loco 1.0, Rails Active Storage RCE - URL: https://singhajit.com/dev-weekly/2026/jul-27-aug-2/nodejs-security-redis-kimi-k3-deepseek-v4-flash-loco-1-gemini-agents/ - Date: 2026-08-02 - Tags: dev-weekly, tech-news, software-development-news - Description: Node.js ships emergency security releases, Redis patches AI-found zero-days, DeepSeek upgrades V4 Flash, and Loco 1.0 lands for Rust. On July 29 Node.js published security releases 22.23.2, 24.18.1, and 26.5.1 across its Maintenance LTS, Active LTS, and Current lines with a highest severity of HIGH, and Rails shipped 7.2.3.2, 8.0.5.1, and 8.1.3.1 to fix an arbitrary file read and remote code execution bug in Active Storage variant processing. On July 27 Redis publicly responded to a researcher who said he used Moonshot's Kimi K3 model to find 19 zero-days in Redis 8.8.0 and turn one into a working exploit in 27 minutes, confirming three memory bugs reachable through the RESTORE command and pointing users to the 8.8.1 security release. On July 31 DeepSeek shipped the official V4 Flash 0731, a post-training refresh with much stronger agentic and tool-calling scores served under the unchanged deepseek-v4-flash API name at the same $0.14/$0.28 per million token pricing, with MIT-licensed weights. On July 29 Loco 1.0 landed as the first stable release of the Rails-like Rust web framework, built on Sea-ORM 2.0 with first-class LLM and agent support. On July 28 Google made Gemini 3.6 Flash the default for managed agents in the Gemini API and added environment hooks, budget controls, and a free tier, and GitHub Copilot for JetBrains added OpenTelemetry export, model management, and MCP servers in Claude agent flows on July 27. Also this week: OpenAI and Anthropic formally endorsed the Pacing the Frontier letter, Okta agreed to buy Permiso for a reported $200 million to secure AI agent identities, Safe Superintelligence landed a reported $5 billion Nvidia partnership, Enigma raised $71 million, and Krutrim cut another 20 to 25 staff. ### Wide Column Stores Explained: Cassandra, Bigtable, and ScyllaDB - URL: https://singhajit.com/wide-column-stores-explained/ - Date: 2026-07-29 - Tags: database, nosql, system-design, distributed-systems - Description: A clear guide to wide column stores like Cassandra, Bigtable, and ScyllaDB. Learn the data model, partition key vs clustering key, LSM writes, tunable consistency, and when to use one. - Quick Answer: A **wide column store** is a NoSQL database that stores rows grouped into partitions, where each row can have a different set of columns. Data is placed across a cluster by a **partition key** (hashed to pick a node) and sorted inside each partition by a **clustering key**. Writes are cheap because they use an [LSM tree](/glossary/lsm-tree/) (memtable plus append-only commit log, flushed to immutable SSTables). Cassandra, ScyllaDB, Apache HBase, and Google Cloud Bigtable are the main examples. Use one when you need huge write throughput, linear horizontal scaling, and you know your read patterns up front. Avoid it when you need ad hoc queries, joins, or multi-partition transactions. ### Dev Weekly Jul 20-26, 2026: Anthropic Ships Claude Opus 5 at Half the Price, Google Floods the Market With Three Gemini Flash Models, Cursor Router, OpenAI Voice-Drives Codex, SharePoint Flaw Exploited - URL: https://singhajit.com/dev-weekly/2026/jul-20-26/claude-opus-5-gemini-flash-cursor-router-openai-voice-sharepoint-mrmustard/ - Date: 2026-07-26 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for July 20 to 26, 2026: Anthropic ships Claude Opus 5 at half the price of Fable 5, Google floods the market with three Gemini Flash models while its flagship slips, Cursor launches a model Router, OpenAI voice-drives Codex, and a SharePoint flaw is exploited within hours. On July 24 Anthropic released Claude Opus 5, priced the same as Opus 4.8 at $5/$25 per million tokens but close to frontier Fable 5 intelligence, with a new state of the art on Frontier-Bench and GDPval-AA. On July 21 Google shipped Gemini 3.6 Flash, Gemini 3.5 Flash-Lite, and a restricted Gemini 3.5 Flash Cyber security model, and said it has begun pre-training Gemini 4, while the long-promised Gemini 3.5 Pro stays in partner testing. On July 22 Cursor launched Cursor Router, a request-level classifier that routes each query to the best model and cut costs 30 to 60 percent in its tests for Teams and Enterprise plans. On July 23 OpenAI brought full-duplex GPT-Live voice into Codex and ChatGPT Work on macOS and Windows so developers can steer multiple coding agents hands-free. On July 20 a public proof of concept for SharePoint CVE-2026-50522 was used within hours to steal machine keys for persistent access, and CISA added it to the KEV catalog on July 22. On July 24 a hijacked account published mrmustard 0.7.4 to PyPI with an import-time credential stealer that grabs SSH, AWS, and Kubernetes secrets. Also this week: Anthropic shipped a Claude Security plugin for Claude Code, Vercel patched nine Next.js vulnerabilities in its first scheduled security release, Deno 2.9.4 landed, Travis Kalanick's Atoms raised $1.7 billion led by a16z, Etched raised $300 million at a $10.3 billion valuation, Meshy AI raised $400 million, and Monday.com, Patreon, Uber, and Magic Leap cut jobs. ### Request Waiting List Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/request-waiting-list/ - Date: 2026-07-24 - Tags: distributed-systems, system-design, networking, software-engineering - Description: Learn the Request Waiting List pattern in distributed systems: why a node cannot answer a client until other nodes respond, how it parks the request against a key and callback, and how Raft, Kafka, and Cassandra use it to wait for a quorum. - Quick Answer: The **Request Waiting List** pattern lets a cluster node accept a client request it cannot answer yet, because the answer depends on responses from other nodes. The node replicates the work asynchronously and parks the client request in a waiting list: a map from a key (a [correlation ID](/glossary/correlation-id/) or a log index) to a callback. As acknowledgements arrive out of order from other nodes, the node looks up the matching entry and the callback checks whether the condition is met, usually a majority [quorum](/distributed-systems/majority-quorum/). Once it is, the callback completes the client request. A timeout sweeps entries that never gather enough responses so nothing leaks or hangs forever. It is the mechanism behind how [Raft](/distributed-systems/replicated-log/), Kafka, and Cassandra hold a client reply until replication is safe. ### DDoS Attacks: How They Work and How to Protect Your App - URL: https://singhajit.com/ddos-attack-and-protection/ - Date: 2026-07-22 - Tags: security, networking, system-design, devops - Description: What is a DDoS attack and how do you stop one? A developer's guide to how distributed denial-of-service attacks work, the main types, and DDoS protection that actually holds up. - Quick Answer: A **DDoS (distributed denial-of-service) attack** floods a website or server with fake traffic from many machines at once, so real users cannot get through. Because the traffic comes from thousands of sources, usually a botnet of hijacked devices, you cannot just block one IP. Attacks hit either the network layer (raw bandwidth, like a UDP or SYN flood) or the application layer (expensive requests, like a flood of logins). You protect against them by putting a large network in front of your origin: a CDN or scrubbing service to absorb volume, rate limiting and a web application firewall to filter bad requests, and autoscaling plus caching so the app survives the spike. No single trick is enough; real DDoS protection is layered. ### Dev Weekly Jul 13-19, 2026: Google's Gemini 3.5 Pro Delay Report, SpaceXAI Open-Sources Grok Build After Upload Scandal, Record 570-Flaw Patch Tuesday, AsyncAPI npm Attack, Fireworks AI Raises $1.5B - URL: https://singhajit.com/dev-weekly/2026/jul-13-19/gemini-3-5-pro-delay-grok-build-open-source-patch-tuesday-asyncapi-npm-fireworks/ - Date: 2026-07-19 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for July 13 to 19, 2026: Bloomberg reports Google's Gemini 3.5 Pro is months behind schedule over weak coding performance, SpaceXAI open-sources Grok Build, Microsoft ships a record 570-flaw Patch Tuesday, and a supply chain attack poisons the @asyncapi npm packages. On July 16 Bloomberg reported Gemini 3.5 Pro, promised for June at Google I/O, is still in partner testing because coding results missed internal targets, frustrating engineers as OpenAI and Anthropic pull ahead. On July 15 SpaceXAI open-sourced the Grok Build coding agent under Apache 2.0 days after its grok CLI was caught uploading entire local directories, including SSH keys, to xAI cloud buckets. On July 14 Microsoft's Patch Tuesday fixed a record 570 vulnerabilities with three zero-days, two of them (SharePoint CVE-2026-56164 and ADFS CVE-2026-56155) already exploited, and Microsoft tied the record count to AI-powered bug discovery. Also on July 14 attackers used a pull_request_target pwn request to publish four backdoored @asyncapi packages with roughly 2.9 million weekly downloads, dropping the Miasma RAT at import time despite valid npm OIDC provenance. On July 16 Fireworks AI raised $1.5 billion at a $17.5 billion valuation, AWS Security Hub began monitoring Microsoft Azure and AI workloads, Anthropic detailed migrating Bun from Zig to Rust with Claude Code, Vercel launched a monthly Next.js security release program, and 1Password shipped credential access for Claude agents. On July 17 SAP acquired Prior Labs for over 1 billion euros and Capital One open-sourced VulnHunter. On July 18 Alibaba's T-Head open-sourced the SAIL CUDA alternative at WAIC and Microsoft cut hundreds of security engineers, while Redis cut about 80 Tel Aviv engineers and Polygon Labs ran a second 2026 layoff round. Claude Fable 5 subscription access ended July 19. ### Payment System Design: Ledger, Idempotency, and Settlement - URL: https://singhajit.com/payment-system-design/ - Date: 2026-07-18 - Tags: system-design, distributed-systems - Description: Learn payment system design end to end: idempotency keys, a double-entry ledger, payment state machines, PSP integration, webhooks, the saga pattern, and reconciliation. A practical guide for the system design interview and production. - Quick Answer: A payment system is a **correctness-first** system, not a throughput-first one. The proven shape is: an **API layer** that takes every request with an **idempotency key** so retries never double charge, a **payment state machine** that only allows valid transitions (created, authorized, captured, settled, refunded, failed), a **double-entry ledger** that records money as matching debits and credits and is never edited in place, a **PSP adapter** that talks to Stripe, Adyen, or a bank with timeouts and retries, a **transactional outbox** plus **saga** to coordinate multi-step flows across services, **webhooks** to learn the real outcome from the processor, and a nightly **reconciliation** job that proves your books match the processor's. Get money correctness first, then scale by sharding the ledger on merchant or account id. ### Designing Database Isolation for B2B Multi-Tenant SaaS - URL: https://singhajit.com/multi-tenant-database-isolation/ - Date: 2026-07-14 - Tags: database, postgres, saas, system-design, security - Description: Learn how to design multi-tenant database isolation for B2B SaaS. Compare shared schema, schema-per-tenant, and database-per-tenant, and use PostgreSQL RLS safely. - Quick Answer: **B2B multi-tenant database isolation** means every customer (tenant) can only see and change their own data, even when many tenants share the same infrastructure. Most B2B SaaS teams start with a **shared database and shared schema** plus a `tenant_id` on every table, enforced in the app and backed by **PostgreSQL Row Level Security (RLS)**. Use **schema-per-tenant** sparingly. Use **database-per-tenant** when enterprise contracts, HIPAA, or hard noisy-neighbor limits demand physical separation. Mature products often use a **hybrid**: shared pool for most tenants, dedicated databases for the largest or most regulated ones. ### Dev Weekly Jul 6-12, 2026: OpenAI Ships GPT-5.6 Sol Terra Luna, SpaceXAI Launches Grok 4.5, ChatGPT Work Debuts, npm v12 Locks Down Installs, Apple Sues OpenAI, Microsoft Cuts 4,800 Jobs - URL: https://singhajit.com/dev-weekly/2026/jul-6-12/gpt-56-sol-grok-45-chatgpt-work-npm-v12-apple-openai-microsoft-layoffs/ - Date: 2026-07-12 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for July 6 to 12, 2026: OpenAI ships GPT-5.6 Sol, Terra, and Luna for general availability, SpaceXAI launches Grok 4.5, and ChatGPT Work debuts as an agentic productivity layer. On July 9 OpenAI ends the government-gated GPT-5.6 preview and prices Sol at $5/$30, Terra at $2.50/$15, and Luna at $1/$6 per million tokens across ChatGPT, Codex, and the API, with an ultra multi-agent mode for harder work. The same day it launches ChatGPT Work, an agent that builds docs, sheets, slides, and sites across connected apps, and makes GPT-5.6 the preferred model in Microsoft 365 Copilot. On July 8 SpaceXAI releases Grok 4.5 at $2/$6, trained alongside Cursor and available in Grok Build, Cursor, and the API, with EU access still pending. Also on July 8 npm v12 goes latest with install scripts, git deps, and remote tarballs off by default, TypeScript 7 ships as a native Go port with typical 8x to 12x faster builds, and AWS announces the Claude apps gateway for centralized Claude Code governance. On July 6 Microsoft cuts 4,800 jobs (2.1 percent of staff), with Xbox shedding about one-fifth of its workforce and spinning out four studios. On July 10 Apple sues OpenAI over alleged trade secret theft tied to more than 400 ex-Apple hires and OpenAI's hardware push. Wiz publicly discloses GhostApproval, a symlink trust-boundary flaw across six AI coding assistants, researchers publish Ghostcommit image-based prompt injection that steals .env secrets, SambaNova raises a $1 billion Series F at an $11 billion valuation with JPMorgan as an inference partner, SK hynix lists ADRs on Nasdaq in a $26.5 billion offering, and a viral Reddit post claims a services firm cut about 70 developers to eight while saying Claude Fable 5 is enough. ### Request Pipeline Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/request-pipeline/ - Date: 2026-07-07 - Tags: distributed-systems, system-design, networking, software-engineering - Description: Learn the Request Pipeline pattern in distributed systems: why waiting for each response wastes network capacity, how sending multiple requests on one connection without blocking cuts latency, and how HTTP/2, Redis, Kafka, and PostgreSQL use pipelining. - Quick Answer: The **Request Pipeline** pattern sends multiple requests on a single connection without waiting for the response to each one. Instead of the slow send, wait, receive, repeat loop, the client fires requests back to back while a separate reader thread collects responses as they arrive. This keeps the network link and the server's [request queue](/role-of-queues-in-system-design/) full, so latency stops being dominated by round-trip time and throughput climbs. It is the idea behind [HTTP/2](/how-webtransport-works/) multiplexing, Redis pipelining, Kafka's in-flight requests, and PostgreSQL pipeline mode. The catch is flow control: you must cap how many requests are in flight so a fast sender does not overwhelm a slower receiver. ### Dev Weekly Jun 29-Jul 5, 2026: Claude Sonnet 5 Released, US Lifts Fable 5 Export Controls, GitHub Copilot Native in JetBrains, Meta Watermelon Catches GPT-5.5, Cloudflare Outage Hits npm - URL: https://singhajit.com/dev-weekly/2026/jun-29-jul-5/claude-sonnet-5-fable-5-returns-copilot-jetbrains-meta-watermelon-cloudflare-npm-outage/ - Date: 2026-07-05 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for June 29 to July 5, 2026: Anthropic ships Claude Sonnet 5, the US lifts Fable 5 export controls, and GitHub Copilot goes native in JetBrains. On June 30 Anthropic releases Claude Sonnet 5, its most agentic mid-size model, at $2 per million input tokens through August 31 and as the default for Free and Pro plans. The same day the US Commerce Department lifts the export controls that had kept Claude Fable 5 and Mythos 5 offline for 19 days, and Fable 5 returns globally on July 1. On June 30 GitHub Copilot becomes a first-class native agent in the JetBrains AI Assistant picker, no ACP setup required. On July 2 Meta superintelligence chief Alexandr Wang tells staff the in-training Watermelon model has caught OpenAI's GPT-5.5 on benchmarks. On July 1 and 2 a Cloudflare network performance failure across North America takes npm installs, GitHub, and other services offline for hours. Mastra 1.48 adds heartbeats for scheduling agents on a cron, CloudNativePG 1.30 ships DatabaseRole and lease-based failover for Postgres on Kubernetes, and Google Cloud brings Conversational Analytics in BigQuery to general availability. Together AI raises an $800 million Series C at an $8.3 billion valuation, Kuaishou's Kling AI raises nearly $3 billion at $18 billion, Datadog acquires Adaptive ML, and Schneider Electric buys Cognite for $3.1 billion. CISA adds the SharePoint deserialization RCE CVE-2026-45659 to its Known Exploited Vulnerabilities catalog with a July 4 deadline, and layoffs hit Microsoft, TikTok, and Cisco. ### Consistent Core Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/consistent-core/ - Date: 2026-07-03 - Tags: distributed-systems - Description: Learn the Consistent Core pattern in distributed systems: why quorum throughput drops as clusters grow, how a small 3 to 5 node core stores metadata with linearizable consistency, and how ZooKeeper, etcd, Consul, Kafka, and Kubernetes use it for leader election, locks, and group membership. - Quick Answer: The **Consistent Core** pattern keeps a small cluster of 3 to 5 nodes that provides strong (linearizable) consistency and fault tolerance, and lets a much larger data cluster offload the decisions that must be exactly right, things like [leader election](/distributed-systems/leader-follower/), group membership, configuration, and distributed [locks and leases](/distributed-systems/lease/). The core runs an expensive [consensus algorithm](/distributed-systems/paxos/) over a [replicated log](/distributed-systems/replicated-log/) on a handful of nodes, while the data cluster grows to hundreds of servers without paying [quorum](/distributed-systems/majority-quorum/) costs on every request. ZooKeeper, etcd, and Consul are consistent cores; Kafka, Kubernetes, HBase, and CockroachDB are built on top of them. ### CDN System Design: How Content Delivery Networks Work - URL: https://singhajit.com/cdn-system-design/ - Date: 2026-06-30 - Tags: system-design, distributed-systems, caching - Description: A clear, developer-focused guide to CDN system design. Learn how content delivery networks work, how edge caching, anycast routing, and origin shield cut latency, and how to design one in a system design interview. - Quick Answer: A CDN (content delivery network) is a globally distributed group of servers that cache your content close to users so requests travel a short distance instead of all the way to your origin server. When a user asks for a file, anycast routing or DNS sends them to the nearest edge server (a Point of Presence). If that edge has the file cached, it serves it instantly (a cache hit). If not (a cache miss), it fetches the file from the origin, often through a regional or origin-shield cache, stores a copy, and serves it. The result is lower latency, less load on your origin, higher availability, and built-in protection against traffic spikes and DDoS attacks. ### How to Use Cursor: 12 Tips to 10x Your Productivity - URL: https://singhajit.com/how-to-use-cursor/ - Date: 2026-06-29 - Tags: AI, cursor, developer-tools, software-engineering - Description: Learn how to use Cursor, the AI code editor, like a power user. This hands-on guide covers Tab autocomplete, Cmd+K inline edit, Chat, Agent and Plan modes, @ context, Rules, Skills, MCP, model picks, and the workflow tips that actually make you faster. - Quick Answer: To use **Cursor**, install the editor from cursor.com, sign in, and import your VS Code settings. Then learn its core surfaces in the order you will actually use them: **Tab** for inline autocomplete, **Cmd+K** for surgical single-file edits, **Chat (Cmd+L)** for codebase questions, and the **Agent** for multi-file work that can run terminal commands and tests. Use **Plan Mode (Shift+Tab)** before any large task, pin context with **@-mentions**, encode your conventions once in **Rules** (`.cursor/rules/`), package repeatable workflows as **Skills**, and connect live tools with **MCP**. Pick a fast model for routine edits and a stronger one for hard reasoning, and always review the diff before you accept. ### Dev Weekly Jun 22-28, 2026: OpenAI Previews GPT-5.6 Sol, Anthropic Says Claude Writes 80% of Its Code, Google Ships Gemini Interactions API, Leo npm Worm, Cursor Buys Continue - URL: https://singhajit.com/dev-weekly/2026/jun-22-28/gpt-56-sol-gemini-interactions-anthropic-80-percent-code-leo-npm-cursor-continue/ - Date: 2026-06-28 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly for June 22 to 28, 2026: OpenAI previews GPT-5.6 Sol, Anthropic says Claude now writes 80% of its code, Google ships the Gemini Interactions API, and a worm hits Leo Platform npm packages. On June 26 OpenAI previews the GPT-5.6 family, flagship Sol plus the cheaper Terra and Luna, starting as a limited preview through Codex and the API at the request of the US government. On June 22 Google promotes the Interactions API to general availability and makes it the default interface for Gemini models and agents, freezing new agent features out of the legacy generateContent endpoint. The same day Anthropic publishes When AI Builds Itself, reporting that more than 80 percent of code merged into its codebase in May was written by Claude and that engineers now ship 8x as much code per quarter. On June 24 an attacker uses a compromised maintainer account to poison more than 20 Leo Platform npm packages with a Shai-Hulud and Miasma worm variant that steals cloud credentials and CI secrets. Cursor quietly acquires open-source coding assistant Continue on June 22 in an acqui-hire that shuts the product down, OpenAI expands its Daybreak security program with Patch the Planet and the full release of GPT-5.5-Cyber, and DeepReinforce open-sources the Ornith-1.0 coding model family on June 25. Nx launches Polygraph and GitKraken launches Code Flow and Kepler for multi-agent development, Next.js 16.3 Preview leans into agent-driven development, and Envoy AI Gateway and the Kubernetes Security Profiles Operator both reach v1.0. Baseten raises $1.5 billion at up to a $13 billion valuation, Groq raises $650 million, Superhuman acquires GPTZero, and MoEngage buys Aampe. CISA adds the Cisco Unified Communications Manager SSRF CVE-2026-20230 and the PTC Windchill RCE CVE-2026-12569 to its Known Exploited Vulnerabilities catalog with a June 28 deadline, and layoffs hit Walmart, Cisco, and Amperity. ### Cursor Skills: How to Create and Use Agent Skills - URL: https://singhajit.com/how-to-create-and-use-skills-in-cursor/ - Date: 2026-06-23 - Tags: AI, cursor, developer-tools, software-engineering - Description: Learn how to create and use Cursor Skills, Cursor's implementation of the open Agent Skills standard. This hands-on guide covers the SKILL.md format, frontmatter fields, project vs personal skills, the paths field, scripts, progressive disclosure, and how to write a description the agent actually triggers on. - Quick Answer: A **Cursor Skill** is a folder with a `SKILL.md` file that teaches the AI agent how to do a specific job, like reviewing a PR, writing a commit message, or running a deploy. To create one, make a folder under `.cursor/skills//` (shared with your repo) or `~/.cursor/skills//` (personal, all projects), add a `SKILL.md` with two required frontmatter fields, `name` (must match the folder) and `description` (tells the agent when to use it), then write the instructions below. Cursor discovers it on startup. The agent uses it automatically when the task matches the description, or you can force it by typing `/` in chat. Keep `SKILL.md` short and move long details into `references/` and `scripts/` so the agent loads them only when needed. ### Dev Weekly Jun 15-21, 2026: SpaceX Buys Cursor for $60 Billion, North Korea Backdoors 140+ Mastra npm Packages, GitHub Copilot App Goes GA, AWS Summit Kiro Mobile, OpenAI Codex Record and Replay - URL: https://singhajit.com/dev-weekly/2026/jun-15-21/spacex-cursor-60b-mastra-npm-attack-github-copilot-app-aws-kiro-codex-record-replay/ - Date: 2026-06-21 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for June 15 to 21, 2026. SpaceX signs a definitive agreement on June 16 to acquire Anysphere, the maker of the AI coding assistant Cursor, in a $60 billion all-stock deal that closes in Q3 2026 and puts the model-agnostic editor inside Elon Musk's xAI. North Korean group Sapphire Sleet backdoors more than 140 packages across the Mastra npm scope on June 17 with a malicious easy-day-js dependency, a supply chain attack Microsoft attributes on June 19. GitHub makes the agent-native Copilot desktop app generally available on June 17 for macOS, Windows, and Linux. AWS Summit New York on June 17 launches Kiro for iOS, AWS Context, AWS Continuum, and new AWS DevOps Agent release management. OpenAI adds Record and Replay to the Codex macOS app on June 18, and Anthropic brings shareable Artifacts to Claude Code the same day. Vercel open-sources the eve agent framework on June 17, Node.js ships emergency security releases for 26.x, 24.x, and 22.x on June 18, and Google Cloud launches Cloud Network Insights for cross-cloud observability. Cisco warns of an actively exploited Catalyst SD-WAN Manager zero-day CVE-2026-20262, and Fortinet FortiSandbox flaws are chained for unauthenticated root. DeepSeek raises $7.4 billion at a $50 billion valuation, Odyssey raises $310 million for world models, Anthropic joins the Frontier carbon coalition and opens a Seoul office, and layoffs hit Rackspace, plus Bay Area WARN filings from Ubisoft, Salesforce, and ServiceNow. ### Leader and Followers Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/leader-follower/ - Date: 2026-06-16 - Tags: distributed-systems - Description: Learn the Leader and Followers pattern in distributed systems: leader election, heartbeats, generation clocks, and log replication. With real examples from Kafka, ZooKeeper, Raft, etcd, MongoDB, Redis, and PostgreSQL, plus how to avoid split brain. - Quick Answer: The **Leader and Followers** pattern picks one node in a cluster as the **leader** and makes every other node a **follower**. All writes go through the leader, which orders them and replicates them to the followers through a [replicated log](/distributed-systems/replicated-log/). Followers serve reads and stand ready to take over. The leader is chosen by an election that needs a [majority quorum](/distributed-systems/majority-quorum/), and a [generation clock](https://martinfowler.com/articles/patterns-of-distributed-systems/generation.html) (also called a term or epoch) plus [heartbeats](/distributed-systems/heartbeat/) keep a stale leader from corrupting data after a failover. This is the backbone of [Kafka](/distributed-systems/how-kafka-works/), ZooKeeper, Raft, etcd, MongoDB replica sets, Redis, and PostgreSQL replication. ### Dev Weekly Jun 8-14, 2026: Apple WWDC and Siri AI, Claude Fable 5 Launched Then Pulled, Microsoft's Record Patch Tuesday, OpenAI Files for IPO and Buys Ona, SpaceX Nasdaq Debut - URL: https://singhajit.com/dev-weekly/2026/jun-8-14/wwdc-siri-ai-claude-fable-5-takedown-patch-tuesday-openai-s1-ona/ - Date: 2026-06-14 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for June 8 to 14, 2026. Apple opens WWDC 2026 on June 8 with Siri AI, iOS 27, and the next generation of Apple Intelligence built with Google, then confirms Siri AI will not ship in the EU under the Digital Markets Act. Apple brings agentic coding to Xcode 27 and expands the Foundation Models framework with free Private Cloud Compute, image input, and a single Swift API for Claude and Gemini. Anthropic launches Claude Fable 5 and Claude Mythos 5 on June 9, then disables both for all users on June 12 after a US government export control order. Microsoft's June 9 Patch Tuesday fixes a record 206 vulnerabilities including three publicly disclosed zero-days and an exploited Exchange Server flaw. OpenAI files a confidential S-1 on June 8, partners with Visa and Oracle on June 10, and acquires Ona on June 11 to give Codex secure cloud environments. GitHub opens Agentic Workflows in public preview on June 11 and extends security scanning to third-party coding agents on June 9. Cursor makes Bugbot three times faster with Composer 2.5 on June 10. SpaceX begins trading on the Nasdaq on June 12 in the largest IPO ever, raising about 75 billion dollars at a valuation above 2 trillion dollars, the same day a global Meta outage takes down Facebook, Instagram, and WhatsApp. Python 3.14.6 and Deno 2.8.3 ship, and AWS launches Graviton5 M9g instances. Layoffs hit Shopee, Expeditors, and Veritone. ### Dev Weekly Jun 1-7, 2026: Microsoft Build 2026 and MAI Models, GitHub Copilot Billing Goes Live, Anthropic Files for IPO, npm Miasma Worm, GitLab Cuts 14% - URL: https://singhajit.com/dev-weekly/2026/jun-1-7/microsoft-build-mai-models-copilot-billing-anthropic-ipo-npm-miasma-gitlab-layoffs/ - Date: 2026-06-07 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for June 1 to 7, 2026. Microsoft opens Build 2026 on June 2 in San Francisco with seven in-house MAI models including MAI-Thinking-1 and the coding model MAI-Code-1-Flash, the OpenClaw-based Scout agent, on-device Aion 1.0 Instruct and Plan models, Project Solara for agent devices, Agent 365, Work IQ APIs, and the Majorana 2 quantum chip. GitHub Copilot switches all plans to token-based AI Credits billing on June 1 to heavy developer backlash. GitHub ships the agent-native Copilot desktop app in technical preview and makes the Copilot SDK generally available across Node.js, Python, Go, .NET, Rust, and Java on June 2, plus an Agent tasks REST API on June 4. Anthropic confidentially files a draft S-1 with the SEC on June 1 and picks Goldman Sachs, Morgan Stanley, and JPMorgan as underwriters on June 3. OpenAI makes frontier models and Codex generally available on AWS on June 1, ships the Dreaming memory system for ChatGPT on June 4, and updates GPT-Rosalind on June 3. JetBrains open-sources the 12B Mellum2 MoE model under Apache 2.0 on June 1. The Miasma npm worm hits 32 Red Hat packages on June 1 and 57 more via the Phantom Gyp binding.gyp technique on June 3. Anthropic maps a year of AI-enabled cyber threats to MITRE ATT&CK on June 3 and extends Project Glasswing to 150 organizations on June 2. Google's June Android bulletin patches 124 flaws including the exploited CVE-2025-48595, added to CISA KEV on June 2. Layoffs: GitLab cuts about 350 jobs on June 3, Uber cuts 23% of its People and Places team on June 3, and Oracle completes around 30,000 cuts by June 15. Releases: Cursor 3.7, Elixir 1.20.0, Angular v22, and Go 1.26.4. ### Git Flow vs GitHub Flow - URL: https://singhajit.com/git-flow-vs-github-flow/ - Date: 2026-06-05 - Tags: git, version-control, devops, software-engineering - Description: Git Flow vs GitHub Flow compared in plain language. See how each branching strategy handles features, releases, and hotfixes, and which Git workflow fits your team. - Quick Answer: **Git Flow** uses two long-lived branches (`main` and `develop`) plus short-lived `feature`, `release`, and `hotfix` branches. It suits versioned software with scheduled releases, like mobile apps, desktop tools, or regulated systems. **GitHub Flow** uses one long-lived branch (`main`) and short-lived feature branches that merge back through a pull request and deploy right away. It suits web apps and SaaS that ship continuously. Pick Git Flow when you must support several versions at once. Pick GitHub Flow when you deploy often and keep only one version in production. ### Idempotent Receiver Pattern in Distributed Systems - URL: https://singhajit.com/distributed-systems/idempotent-receiver/ - Date: 2026-06-04 - Tags: distributed-systems, system-design, microservices, software-engineering - Description: Learn how the idempotent receiver pattern safely handles duplicate requests in distributed systems using a client ID, request number, and a saved response. - Quick Answer: An **Idempotent Receiver** is a server that can process the same request more than once without changing the result beyond the first time. The client tags every request with a **unique client ID** and a **request number**. The server stores the result of each request, so when a retry arrives with a request number it has already handled, it returns the **saved response** instead of running the work again. This makes retries safe under **at-least-once** delivery, where duplicates are guaranteed, not rare. ### ULID Explained: How Sortable Unique IDs Work - URL: https://singhajit.com/ulid-guide/ - Date: 2026-06-02 - Tags: system-design - Description: What is a ULID? Learn how Universally Unique Lexicographically Sortable Identifiers work, the 26-character Crockford Base32 format, ULID vs UUID vs UUID v7, monotonic generation, and how to generate and decode ULIDs in Python, JavaScript, Java, Go, and PostgreSQL. - Quick Answer: A ULID (Universally Unique Lexicographically Sortable Identifier) is a 128-bit ID written as **26 Crockford Base32 characters**. The first 10 characters are a **48-bit millisecond timestamp** and the last 16 are **80 bits of randomness**. Because the time comes first, sorting ULIDs as plain strings sorts them by creation time, which makes them faster database primary keys than random UUID v4 and shorter than UUID v7 (26 vs 36 characters). ### Dev Weekly May 25-31, 2026: Anthropic Hits $965B, Claude Opus 4.8 Ships, Cognition Raises $1B, Glassworm Botnet Goes Down, Wix Cuts 1,000 - URL: https://singhajit.com/dev-weekly/2026/may-25-31/anthropic-965b-claude-opus-48-cognition-1b-glassworm-takedown-wix-layoffs/ - Date: 2026-05-31 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for May 25 to 31, 2026. Anthropic closes a $65 billion Series H on May 28 at a $965 billion post-money valuation, overtaking OpenAI to become the world's most valuable AI startup, and ships Claude Opus 4.8 the same day with Dynamic Workflows that orchestrate up to 1,000 parallel subagents, a 2.5x faster Fast Mode, and a 1 million token context window. Cognition raises over $1 billion at a $26 billion valuation on May 27 for its Devin AI software engineer, with annualized revenue at $492 million. CrowdStrike, Google, and Shadowserver dismantle the Glassworm botnet on May 26 and 27, cutting four command-and-control channels behind two years of open source supply chain attacks. GitHub confirms TeamPCP exfiltrated roughly 3,800 internal repositories through a poisoned Nx Console VS Code extension. GitHub also has a global outage on May 27 from 12:10 UTC to 13:16 UTC. Wix announces 1,000 layoffs on May 28, about 20% of headcount, citing AI and a strengthening shekel. OpenAI brings Codex Computer Use and ChatGPT mobile control to Windows on May 29 and releases Codex CLI 0.134.0 on May 26. xAI launches Grok Build coding agent on May 25 at $300 per month. Cursor 3.6 ships Auto-review Run Mode on May 29. Anthropic releases a Claude Code security-guidance plugin on May 26. Microsoft releases the Lens 3.8B text-to-image model on Hugging Face on May 26. Google ADK Python 2.0 GA. Apple seeds iOS 26.6, iPadOS 26.6, macOS 26.6, watchOS 26.6, tvOS 26.6, and visionOS 26.6 developer betas on May 26. Mistral launches Industrial Engineering AI with Airbus, BMW, EDF, and CMA CGM on May 28. Snowflake commits $6 billion to AWS on May 27 for Graviton compute and AI infrastructure. Salesforce reports Q1 FY27 record revenue of $11.1 billion on May 27 with Agentforce crossing $1 billion ARR. Security: Microsoft uncovers two npm supply chain waves on May 28 and 29 with credential stealers targeting AWS, HashiCorp Vault, and CI/CD secrets. CISA KEV adds CVE-2026-48172 LiteSpeed cPanel on May 26, three supply chain CVEs for Daemon Tools, TanStack, and Nx Console on May 27, and CVE-2026-0257 Palo Alto PAN-OS authentication bypass on May 29. Gravity Bridge drained of $5.4 million on May 31. Releases: Puppeteer v25.1.0, Typer 0.26.0, Spring AI 2.0.0-M8, Docker Agent v1.70.0. ### Auth0 vs Okta: How to Pick the Right Identity Platform - URL: https://singhajit.com/auth0-vs-okta/ - Date: 2026-05-30 - Tags: security, system-design, software-engineering - Description: Auth0 vs Okta compared for software developers in 2026. Customer Identity Cloud vs Workforce Identity Cloud, pricing, SSO, SAML, OIDC, MFA, SDKs, Actions, and when to pick Auth0, Okta, Keycloak, or Cognito. - Quick Answer: **Auth0** (now branded as **Okta Customer Identity Cloud**) is a developer-first **CIAM** platform for the people who log into your product. **Okta Workforce Identity Cloud** is for the people who work at your company logging into your tools. Same parent (Okta acquired Auth0 in 2021 for $6.5B), separate products, separate pricing. Pick **Auth0** for B2C and B2B SaaS sign-in, social login, and API authorization. Pick **Okta** for employee SSO, lifecycle management, and 8,000+ pre-built enterprise integrations. Many large companies run both. ### Notification System Design: Push, SMS, Email at Scale - URL: https://singhajit.com/notification-system-design/ - Date: 2026-05-29 - Tags: system-design, distributed-systems - Description: How to design a scalable notification system that sends push, SMS, email, and in-app messages to millions of users. Covers Kafka priority queues, idempotency, retries, FCM and APNs, provider fallback, and DLQs. - Quick Answer: A notification system is a **fan-out and reliability problem** dressed up as a messaging problem. One event ("order shipped") has to become a push notification on the phone, an email in the inbox, an SMS to the family member listed as the backup contact, and a red dot in the app, all without sending the same alert twice and without taking down the producing service when Twilio or SendGrid has a bad afternoon. The proven shape is: a thin **Notification API** that returns `202 Accepted`, **priority queues** in [Kafka](/kafka-vs-rabbitmq-vs-sqs/) that keep one-time passwords away from marketing blasts, a **router** that resolves user preferences and quiet hours, **channel dispatchers** that talk to FCM, APNs, SendGrid, and Twilio, **idempotency keys** in Redis to stop double-sends, a **dead-letter queue** for failures, and a **webhook handler** that closes the loop on actual delivery. ### PostgreSQL MVCC and Autovacuum Explained - URL: https://singhajit.com/postgresql-mvcc-autovacuum/ - Date: 2026-05-25 - Tags: database, postgres, sql - Description: Understand PostgreSQL MVCC, dead tuples, and table bloat. Learn how to tune autovacuum, vacuum cost limit, and scale factor to keep production Postgres fast. - Quick Answer: PostgreSQL uses **Multi-Version Concurrency Control (MVCC)** to allow multiple transactions to read and write simultaneously without locking tables. When you `UPDATE` or `DELETE` a row, Postgres does not modify the old data in-place; instead, it writes a new version (tuple) and marks the old one as **dead**. These dead tuples accumulate and cause **table bloat**, which slows down queries. The **autovacuum** daemon runs in the background to clean up these dead tuples and reclaim space. To keep production databases fast, you must tune autovacuum per-table using thresholds like `autovacuum_vacuum_scale_factor` and cost parameters like `autovacuum_vacuum_cost_limit` rather than relying on default settings. ### Dev Weekly May 18-24, 2026: Google I/O 2026, Anthropic Buys Stainless, SpaceX and OpenAI IPO Filings, Meta Cuts 8,000, Laravel-Lang Attack - URL: https://singhajit.com/dev-weekly/2026/may-18-24/google-io-2026-anthropic-stainless-spacex-ipo-openai-ipo-meta-layoffs-laravel-lang-attack/ - Date: 2026-05-24 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for May 18 to 24, 2026. Google I/O 2026 opens on May 19 with Gemini 3.5 Flash as the new default in the Gemini app and AI Mode, Antigravity 2.0 as a standalone agent-first desktop app with CLI and SDK, Managed Agents in the Gemini API, native Android vibe coding in Google AI Studio, Gemini Spark as a 24/7 personal agent on dedicated Google Cloud VMs, Gemini Omni Flash for video first generation, and Universal Cart for agentic shopping. Anthropic acquires Stainless on May 18 for SDK and MCP server tooling and announces self hosted sandboxes and MCP tunnels for Claude Managed Agents at Code with Claude London on May 19. Andrej Karpathy joins Anthropic on May 19 to use Claude to accelerate Claude pre training. SpaceX files its S-1 on May 20 disclosing 18.7 billion dollars in 2025 revenue, 6.6 billion dollars adjusted EBITDA, a 1.25 billion dollar per month Anthropic compute deal through May 2029, and expansion to Colossus 2. OpenAI prepares a confidential IPO filing with Goldman Sachs and Morgan Stanley as soon as May 22 targeting a September 2026 debut at over 850 billion dollars. Meta begins 8,000 layoffs on May 20 and reassigns 7,000 staff into AI roles. Intuit announces 3,000 layoffs on May 20. Cursor Composer 2.5 ships May 18 as the new default, Cursor in Jira on May 19, and Cursor 3.5 brings Automations to the Agents Window on May 20. Anthropic launches self hosted sandboxes and MCP tunnels for Claude Managed Agents on May 19 and posts a Project Glasswing update on May 22 with 2,100 Claude Security patches. OpenAI partners with Dell on May 18 to bring Codex to hybrid and on premises environments. Codex CLI 0.132.0 lands on May 19. Microsoft open sources RAMPART and Clarity for agent safety on May 20. GitHub Copilot adds Claude Haiku 4.5 and GPT-5.4-mini cloud agent models on May 18, Gemini 3.5 Flash GA on May 19, and Copilot for Eclipse open source on May 21. Node.js v24.16.0 Krypton LTS ships May 21. Google ADK Python v2.0 GA on May 19. FastAPI 0.136.2 on May 23. Security: Laravel-Lang Composer supply chain attack on May 22 and 23 republishes around 700 versions across four packages, LiteSpeed cPanel CVE-2026-48172 CVSS 10.0 with active exploitation, AntV Mini Shai-Hulud npm wave on May 19 with 323 packages and 639 versions, CISA KEV additions for CVE-2026-9082 Drupal Core SQL injection on May 22 and CVE-2025-34291 Langflow plus CVE-2026-34926 Trend Micro Apex One on May 21. Funding: Hark 700 million dollars Series A at 6 billion dollars, Socket 60 million dollars Series C at 1 billion dollars, Viktor 75 million dollars Series A, Status AI 17 million dollars, Unframe 50 million dollars. Qualtrics closes the 6.75 billion dollar Press Ganey Forsta deal on May 18. Coupa acquires Tonkean on May 21. ### Lease Pattern in Distributed Systems Explained - URL: https://singhajit.com/distributed-systems/lease/ - Date: 2026-05-21 - Tags: distributed-systems - Description: Learn the Lease pattern in distributed systems: time-bound exclusive access using TTL, heartbeats, and fencing tokens. With real implementations from etcd, Kubernetes, ZooKeeper, Chubby, and HDFS, plus pitfalls around GC pauses and clock drift. - Quick Answer: A Lease in distributed systems is a time-bound grant that gives a single cluster node exclusive access to a resource for a fixed duration. The holder must keep refreshing the lease through [heartbeats](/distributed-systems/heartbeat/) before its time to live (TTL) expires. If the node crashes, pauses, or gets partitioned away, the lease quietly expires and another node can take over. To protect against a slow or paused holder waking up after expiry and still writing, leases are paired with a monotonically increasing **fencing token** that the storage layer validates. The pattern is used by [Google Chubby](https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems/), [ZooKeeper](/distributed-systems/replicated-log/), [etcd](https://etcd.io/docs/v3.5/learning/api/), [Kubernetes](https://kubernetes.io/docs/concepts/architecture/leases/), HDFS, and Cassandra for leader election, distributed locking, session tracking, and resource coordination. ### Dev Weekly May 11-17, 2026: TanStack npm Attack Hits OpenAI, Patch Tuesday 137 Fixes, Anthropic $900B, Cursor 3.4 - URL: https://singhajit.com/dev-weekly/2026/may-11-17/tanstack-npm-attack-patch-tuesday-anthropic-30b-android-show-cursor-34/ - Date: 2026-05-17 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for May 11 to 17, 2026. The TanStack npm supply chain attack on May 11 compromises 42 @tanstack/* packages with 84 malicious versions in a six minute window, signed using legitimate OIDC trusted publishing, and reaches OpenAI's internal source code repos, forcing a Mac code signing rotation with a June 12 update deadline for ChatGPT Desktop users. Microsoft Patch Tuesday on May 12 ships 137 fixes with 13 rated critical and no zero days under active exploitation, including Netlogon RCE CVE-2026-41089 and DNS Client RCE CVE-2026-41096. Anthropic opens talks on May 12 to raise at least 30 billion dollars at a valuation above 900 billion dollars, ahead of an expected end of May close. Google holds The Android Show I/O Edition on May 12 and unveils Gemini Intelligence on Android, Create My Widget vibe coded widgets, Rambler in Gboard, Adobe Premiere on Android, and the Googlebook AI native laptop line. Cursor 3.4 ships May 13 with cloud agent dev environments, multi repo support, build secrets, Dockerfile config, 70 percent faster layer caching, version history rollback, and audit logs. Cursor Bugbot Effort Levels land May 11. Claude Code v2.1.139 arrives May 11 with the Agent view, the /goal command, the /scroll-speed setting, and CLAUDE_PROJECT_DIR for MCP stdio servers. Anthropic launches Claude for Small Business on May 13 with 15 ready to run agentic workflows for finance, HR, marketing, and ops. OpenAI launches Daybreak on May 11 with Codex Security and three GPT-5.5 tiers including GPT-5.5-Cyber, plus the OpenAI Deployment Company with over 4 billion dollars committed, the Tomoro acquisition, and 19 partner firms. OpenAI brings Codex to the ChatGPT mobile app on May 14 with Hooks, Remote SSH, and HIPAA support. Microsoft cancels Claude Code licenses inside Experiences and Devices and moves engineers to GitHub Copilot CLI by June 30. Microsoft Security ships its MDASH multi model agentic security system on May 12. GitHub introduces Copilot Pro, Pro+, and a new Max plan at 100 dollars a month with flex allotments effective June 1. Fortinet patches critical RCE CVE-2026-44277 in FortiAuthenticator and CVE-2026-26083 in FortiSandbox on May 12. JetBrains TeamCity fixes CVE-2026-44413 privilege escalation in 2026.1. Dirty Frag gets a new Fragnesia variant CVE-2026-46300 by May 14. Cisco patches Catalyst SD-WAN authentication bypass CVE-2026-20182 with CVSS 10.0 on May 14 with active exploitation by UAT-8616, CISA adds it to KEV the same day with a May 17 federal due date. CISA adds Microsoft Exchange CVE-2026-42897 stored XSS to KEV on May 15. Twisted 26.4.0 ships May 11 with DNS DoS fix CVE-2026-42304. GitLab announces an open ended restructuring memo on May 12 with R&D into 60 autonomous teams. GM lays off more than 600 IT employees on May 11 to hire AI skills. Celonis signs to acquire Ikigai Labs on May 12. Coursera and Udemy complete their merger May 11. Funding: Isomorphic Labs 2.1 billion dollars, Exaforce 125 million dollars Series B, White Circle 11 million dollars. ### Flash Sale System Design: Architecture, Scale, and Oversell - URL: https://singhajit.com/flash-sale-system-design/ - Date: 2026-05-16 - Tags: system-design, distributed-systems - Description: How to design a flash sale system that handles millions of buyers, prevents overselling, and blocks duplicate orders with Redis, queues, and idempotency keys. - Quick Answer: A flash sale is a **write-heavy, contention-heavy** problem hiding inside an e-commerce page. Ten million buyers click "Buy" at the same second for ten thousand units. The job of the system is to (1) reduce the wave to a trickle before it reaches the database, and (2) make sure exactly ten thousand orders win and no buyer ever pays twice. The proven stack is: a **static landing page on the CDN**, a **virtual waiting room** that admits batches, a **token gate** that mints one token per available unit, an **atomic Redis Lua script** that decrements stock without race conditions, an **idempotency key** that makes every order create-once, a **Kafka or queue** that decouples checkout from payment and fulfillment, and a **database unique constraint** as the last line of defense. Each layer drops about an order of magnitude of traffic, so the relational store never sees the raw spike. ### What Is a GUID? How to Generate One in C# and SQL Server - URL: https://singhajit.com/guid-explained/ - Date: 2026-05-11 - Tags: database - Description: What is a GUID? Learn how a Globally Unique Identifier works, the 8-4-4-4-12 hex format, GUID vs UUID, GUID versions, and how to generate one in C# with Guid.NewGuid(), in SQL Server with NEWID() and NEWSEQUENTIALID(), and in PowerShell with New-Guid. - Quick Answer: A GUID (Globally Unique Identifier) is Microsoft's name for a UUID: a **128-bit** value written as **32 hexadecimal digits** in the `8-4-4-4-12` pattern, like `550e8400-e29b-41d4-a716-446655440000`. It is the same format and standard (RFC 9562) as a UUID. `Guid.NewGuid()` in C# creates a random version 4 GUID, SQL Server uses `NEWID()` and `NEWSEQUENTIALID()`, and PowerShell uses `New-Guid`. ### Dev Weekly May 4-10, 2026: Anthropic-SpaceX Compute Deal, Cloudflare Cuts 1,100, AWS Agent Payments, Cursor 3.3 - URL: https://singhajit.com/dev-weekly/2026/may-4-10/anthropic-spacex-cloudflare-layoffs-cursor-33-aws-agent-payments/ - Date: 2026-05-10 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for May 4 to 10, 2026. Anthropic signs a SpaceX deal for the entire Colossus 1 data center on May 6, unlocking 220,000 NVIDIA GPUs and 300 megawatts and lifting Claude Code five hour rate limits across Pro, Max, Team, and Enterprise. Cloudflare reports Q1 2026 revenue of 639.8 million dollars on May 7, then announces 1,100 layoffs, a 140 to 150 million dollar restructuring charge, and an agentic AI-first operating model as the stock falls 24 percent. AWS launches Bedrock AgentCore Payments in preview on May 7 with Coinbase and Stripe to let agents pay over the x402 protocol with stablecoins, and the AWS MCP Server reaches general availability on May 6 with the Agent Toolkit for AWS. Cursor 3.3 ships on May 7 with a full PR review experience, Build in Parallel for plans, and Split PRs. OpenAI updates ChatGPT to GPT-5.5 Instant on May 5, ships GPT-Realtime-2, GPT-Realtime-Translate, and GPT-Realtime-Whisper voice models on May 7, scales GPT-5.5-Cyber for cyber defenders, and rolls out Trusted Contact. xAI launches Grok 4.3 on May 6 with a 1 million token context, agentic tool use, and a May 15 deprecation of older Grok models. Google Cloud ships Agent Identity GA, Agent Gateway preview, and 80 Agentic Data Cloud updates on May 6, plus the Gemini API File Search goes multimodal on May 5. Anthropic releases ten financial services agent templates on May 5. Node.js 26.1.0 lands experimental node:ffi on May 7. Next.js 15.5.18 and 16.2.6 patch 13 advisories including CVE-2026-44575 on May 7. React 19.0.6 patches RSC DoS CVE-2026-23870. CISA adds Ivanti EPMM CVE-2026-6973 to the KEV catalog on May 7, BerriAI LiteLLM CVE-2026-42208 on May 8, and Palo Alto PAN-OS CVE-2026-0300 on May 6. Cloudflare ships an emergency WAF release for the Next.js middleware bypass. Snowflake 10.16, Databricks Runtime 18.2 GA, and the Lakeflow Pipelines Editor GA all ship May 4. Kubernetes v1.36 Declarative Validation graduates on May 5. SAP signs deals to acquire Dremio and Prior Labs on May 4. Cisco signs intent to acquire Astrix Security and IREN agrees to acquire Mirantis for 625 million dollars. Google launches the 99 dollar Fitbit Air on May 7. Funding: Blitzy 200 million dollars at 1.4 billion valuation, Corgi 160 million dollars Series B, Quantum Motion 160 million dollars Series C, DeepInfra 107 million dollars Series B, RadixArk 100 million dollars seed for SGLang, Tessera Labs 60 million dollars from a16z. ### Debezium and the Outbox Pattern: The Real Impact on Your Postgres Database - URL: https://singhajit.com/debezium-outbox-postgres-database-impact/ - Date: 2026-05-05 - Tags: database, distributed-systems, software-engineering - Description: A practical, production-grade look at what Debezium does to your Postgres primary when you use it to stream a transactional outbox table to Kafka. Covers logical decoding, replication slots, WAL retention, walsender CPU, the reorder buffer, max_slot_wal_keep_size, heartbeats, monitoring, and what to tell your DBA team. - Quick Answer: Running Debezium against a Postgres outbox table looks to the database like one extra logical replica. The cost is mostly **CPU for logical decoding on the primary** (single-digit percent under steady load), a small amount of memory per `walsender`, and one **replication slot that pins WAL until Debezium acknowledges it**. The biggest production risk is not CPU; it is **unbounded WAL growth** when the connector falls behind or dies, which has crashed primaries in the wild. Cap it with [`max_slot_wal_keep_size`](https://www.postgresql.org/docs/current/runtime-config-replication.html), use a **dedicated publication for the outbox table only**, set `REPLICA IDENTITY DEFAULT`, partition the table by time, and wire alerts to `pg_replication_slots.confirmed_flush_lsn` lag. Done right, the steady-state overhead is small and far cheaper than [polling the outbox table from the application](/transactional-outbox-pattern/). ### Design TinyURL: System Design Interview Guide for URL Shorteners - URL: https://singhajit.com/tinyurl-system-design/ - Date: 2026-05-04 - Tags: system-design, distributed-systems - Description: A practical guide to designing a URL shortener like TinyURL or Bitly. Walk through requirements, capacity estimation, base62 encoding, ID generation, database schema, caching, redirects, analytics, custom aliases, and rate limiting. Built for system design interviews and real production systems. - Quick Answer: A URL shortener is a small **read-heavy key-value lookup**. The write path stores `(short_code, long_url)` once. The read path looks up the short code, returns an HTTP 301 or 302 redirect, and fires an async analytics event. The short code is **6 to 7 characters of base62** (`a-z`, `A-Z`, `0-9`), which gives 56 billion to 3.5 trillion unique codes. The two clean ways to mint codes are a **counter plus base62 encoding** or a [Snowflake ID](/snowflake-id-guide/) truncated to 7 characters. Hot URLs sit in **Redis** with LRU eviction, the source of truth is a partitioned SQL or NoSQL store with the short code as the primary key, and a CDN absorbs most repeat traffic. The hard parts are not the schema, they are collision handling, predictability, abuse, and keeping a 100 to 1 read to write ratio cheap. ### Dev Weekly Apr 27-May 3, 2026: Microsoft-OpenAI Split, AWS Bedrock GPT-5.5, Pentagon AI Deals, Apple $111B - URL: https://singhajit.com/dev-weekly/2026/apr-27-may-3/microsoft-openai-restructure-aws-bedrock-pentagon-ai-cursor-sdk-apple-earnings/ - Date: 2026-05-03 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for April 27 to May 3, 2026: Microsoft and OpenAI end exclusivity, AWS ships GPT-5.5, Codex, and Bedrock Managed Agents on Amazon Bedrock, and the Pentagon signs eight tech vendors (Microsoft, AWS, Google, OpenAI, Nvidia, SpaceX, Oracle, Reflection AI) for classified IL6 and IL7 AI deployment. Apple posts a record $111.2B Q2 with $57B in iPhone revenue and a $100B buyback. Microsoft Azure grows 40 percent. Anthropic opens Claude Security public beta on Opus 4.7. Cursor ships a TypeScript SDK with sandboxed cloud VMs and subagents. Warp open sources its agentic IDE. Sentry launches the Seer Agent. OpenAI publishes the Symphony coding agent spec. IBM ships Granite 4.1, Nvidia drops Nemotron 3 Nano Omni, xAI launches the Grok 4.3 API, and Hippocratic AI releases Polaris 5.0. GitHub Copilot moves to usage based billing on June 1. Critical security: Linux kernel Copy Fail (CVE-2026-31431), cPanel auth bypass (CVE-2026-41940) exploited since February, Mini Shai-Hulud worm hits SAP npm packages, ShinyHunters breaches at Vimeo, Udemy, and Amtrak. Plus David Silver's Ineffable Intelligence $1.1B seed, Meta acquires Assured Robot Intelligence, Stripe Sessions 2026 launches 288 products, and Scout AI raises $100M for defense. ### How Git Stores Data Internally: The Object Model Explained - URL: https://singhajit.com/how-git-stores-data-internally/ - Date: 2026-05-01 - Tags: git, version-control, devops - Description: A developer's guide to how Git stores data internally. Walk through the .git folder, the four object types (blob, tree, commit, tag), the index, refs and HEAD, loose objects vs pack files, delta compression, and the SHA-1 to SHA-256 transition. Learn what really happens behind git add, git commit, and git push. - Quick Answer: Git is a small **content-addressable key-value database** living inside the `.git` folder. Every piece of your project becomes one of four object types: **blob** (file contents), **tree** (a directory listing), **commit** (a snapshot pointer with metadata), or **tag** (a named pointer). Each one is identified by the **SHA-1 hash** of its content. Branches are tiny text files in `.git/refs/heads/` that hold a single commit hash. New objects start as **loose files** zlib compressed under `.git/objects/`, and `git gc` later rolls them into **pack files** with delta compression and a fast index. That is the whole storage model. Run [git cat-file -p HEAD](/git-cheat-sheet/) on any repo to see it for yourself. ### Vector Database Deep Dive: How They Actually Work - URL: https://singhajit.com/vector-database-deep-dive/ - Date: 2026-04-30 - Tags: ai, database, software-engineering - Description: A practical deep dive into vector databases for software developers. Learn how embeddings, approximate nearest neighbor search, HNSW, IVF, and product quantization work, and how to pick between pgvector, Pinecone, Qdrant, Weaviate, Milvus, and Chroma for RAG and semantic search. - Quick Answer: A **vector database** is a system that stores high-dimensional embeddings and answers nearest neighbor queries in milliseconds. It uses an **approximate nearest neighbor (ANN)** index, most commonly **HNSW** (a layered graph) or **IVF** (k-means clusters), to avoid scanning every vector. On top of that index it adds metadata filtering, persistence, replication, and a query API. Pick **pgvector** if you already run PostgreSQL and have under ten million vectors, **Qdrant** for the lowest latency self-hosted option, **Pinecone** for fully managed scale, **Weaviate** for hybrid search, and **Milvus** when you cross the billion-vector mark. ### Saga Pattern Explained: Distributed Transactions for Microservices - URL: https://singhajit.com/saga-pattern-distributed-transactions/ - Date: 2026-04-29 - Tags: system-design, distributed-systems, microservices, software-engineering - Description: A deep, no-fluff guide to the Saga Pattern for software developers. Learn how choreography and orchestration sagas work, how to design compensating transactions, when to pick saga over two-phase commit, and how to implement sagas with Kafka, Temporal, AWS Step Functions, and the Outbox pattern. - Quick Answer: The **Saga Pattern** breaks one large distributed transaction into a sequence of small **local transactions**, one per service, each with a **compensating transaction** that can undo it if a later step fails. There is no global lock and no two-phase commit. You pick between **choreography** (services react to each other's events) and **orchestration** (a central coordinator calls each service in turn). Use sagas when you need to keep data consistent across services that own their own databases, and you can live with eventual consistency. ### Dev Weekly Apr 20-26, 2026: Tim Cook Steps Down, GPT-5.5, Vercel Breach, Anthropic-AWS $100B - URL: https://singhajit.com/dev-weekly/2026/apr-20-26/tim-cook-steps-down-gpt-55-vercel-breach-anthropic-aws-google-cloud-next/ - Date: 2026-04-26 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for April 20 to 26, 2026 covering the biggest software developer news, AI model releases, security breaches, and funding. Apple announced on April 20 that Tim Cook is stepping down as CEO after 15 years and John Ternus, SVP of Hardware Engineering, will become CEO on September 1, 2026, with Cook moving to executive chairman. OpenAI released GPT-5.5 on April 23 with 58.6 percent on SWE-Bench Pro, 82.7 percent on Terminal-Bench 2.0, and 84.9 percent on GDPval. Vercel disclosed a breach via Context.ai on April 20 with environment variables exposed and customer data sold on BreachForums for 2 million dollars. Anthropic and Amazon announced a 100 billion dollar AWS commitment on April 20 with up to 5 gigawatts of Trainium capacity. Google Cloud Next 2026 ran April 22 to 23 with Gemini 3.1 Pro, Gemini 3.1 Flash Image, eighth generation TPUs, and the Gemini Enterprise Agent Platform. Microsoft shipped an emergency out of band patch on April 21 for ASP.NET Core CVE-2026-40372 with a CVSS 9.1 score. Cursor 3.2 launched April 24 with async multitasking, improved worktrees, and multi root workspaces. Moonshot AI promoted Kimi K2.6 to general availability on April 21 with 12 hour autonomous runs and 300 agent swarms. The Eclipse Foundation launched Open VSX Managed Registry on April 21 with a 99.95 percent uptime SLA. GitHub paused new Copilot Pro and Pro+ sign-ups on April 20 due to compute strain from agentic workflows. OpenAI shipped Workspace Agents on April 22 and Codex CLI updates on April 23 and 24. Sundar Pichai said on April 22 that 75 percent of new code at Google is AI generated. Anthropic launched the Economic Index Survey on April 22. CVE-2026-33032 in nginx UI is being actively exploited. A CanisterWorm npm worm hit Namastex Labs packages on April 21. Reliable Robotics raised 160 million dollars, AcuityMD 80 million dollars, Orkes 60 million dollars. ### How GitHub Stores and Serves Git Repositories - URL: https://singhajit.com/how-github-stores-and-serves-git-repositories/ - Date: 2026-04-25 - Tags: system-design, git, github, distributed-systems - Description: A developer's guide to how GitHub stores and serves Git repositories. Walk through the Spokes replication system, three-replica voting, the Git proxy, pack files, monorepo optimizations like multi-pack-index and reftable, and Git LFS. Learn what really happens between git push and the bytes landing on disk. - Quick Answer: GitHub stores every Git repository on **three independent file servers** in a system called [Spokes](https://github.blog/engineering/architecture-optimization/introducing-dgit/) (originally DGit). When you `git push`, a proxy in front of the file servers streams the update to all three replicas, runs a **three-phase commit**, and only acknowledges success once **at least two replicas** have applied the same Git transaction. Reads are routed to the closest in-sync replica for speed. Underneath, every file server still runs **plain Git on local SSDs**, storing objects in **pack files** indexed for fast lookup. A separate MySQL/Vitess tier holds repository metadata, Git LFS holds large binaries on object storage, and aggressive maintenance like [multi-pack-index bitmaps](/distributed-systems/replicated-log/) keeps even gigantic monorepos fast. ### PostgreSQL Internals: How Queries Actually Execute - URL: https://singhajit.com/postgresql-internals-how-queries-execute/ - Date: 2026-04-23 - Tags: database, postgres, sql - Description: A developer's guide to PostgreSQL internals. See how queries actually execute through the parser, planner, and executor, how MVCC, the WAL, and the shared buffer pool work, and how to read EXPLAIN ANALYZE plans to fix slow Postgres queries. - Quick Answer: When you run a SQL query, a Postgres backend process **parses** the text into a tree, the **rewriter** expands views and rules, the **planner/optimizer** picks the cheapest plan from many candidates using table statistics, and the **executor** pulls rows through the plan one at a time. Reads come from the **shared buffer pool** (or disk on a miss), writes go to the **Write-Ahead Log** first for durability, **MVCC** keeps old row versions visible until vacuum reclaims them, and a small army of background processes (checkpointer, bgwriter, autovacuum, WAL writer) keeps the whole thing healthy. Run [EXPLAIN ANALYZE](/postgresql-cheat-sheet/) on any slow query to see exactly which step is costing you. ### Lamport Clock in Distributed Systems - URL: https://singhajit.com/distributed-systems/lamport-clock/ - Date: 2026-04-22 - Tags: distributed-systems - Description: A practical guide to the Lamport Clock distributed systems pattern. Learn the algorithm, the happens-before relation, total ordering with tie-breakers, real implementations in Cassandra and Kafka, and how it compares to vector clocks and Hybrid Logical Clocks. - Quick Answer: A Lamport Clock is a single integer counter kept on every node in a distributed system. Every event bumps the counter by one. Every outgoing message carries the counter. On receive, a node sets its counter to `max(local, received) + 1`. This simple rule guarantees that if event A causally happened before event B then `LC(A) < LC(B)`, even when the underlying wall clocks are skewed. It was introduced by Leslie Lamport in his 1978 paper [Time, Clocks, and the Ordering of Events in a Distributed System](https://lamport.azurewebsites.net/pubs/time-clocks.pdf) and is the foundation behind vector clocks, [Hybrid Logical Clocks](/distributed-systems/hybrid-clock/), Cassandra timestamps, Kafka producer epochs, and Raft terms. ### Dev Weekly Apr 13-19, 2026: Claude Opus 4.7, Codex Computer Use, Patch Tuesday Zero-Day, Snap Layoffs - URL: https://singhajit.com/dev-weekly/2026/apr-13-19/claude-opus-47-codex-computer-use-patch-tuesday-snap-layoffs-cursor-31/ - Date: 2026-04-19 - Tags: dev-weekly, tech-news, software-development-news - Description: Dev Weekly roundup for April 13 to 19, 2026 covering the biggest software developer news, AI model releases, layoffs, security patches, and funding. Anthropic released Claude Opus 4.7 on April 16 with a 64.3 percent SWE-Bench Pro score, vision up to 2,576 pixels, and a Cyber Verification Program. OpenAI updated Codex desktop the same day with Computer Use across Mac and Windows apps, a built-in Chromium browser, gpt-image-1.5 image generation, and 111 new plugins. Microsoft April 2026 Patch Tuesday on April 14 fixed 167 vulnerabilities including actively exploited SharePoint zero-day CVE-2026-32201, a Defender privilege escalation CVE-2026-33825, and a 9.8 CVSS Windows IKE remote code execution flaw. Snap cut 16 percent of staff (1,000 employees) on April 15 and cited rapid AI advances. Cursor 3.1 shipped tiled multi-agent layouts on April 13. Cloudflare expanded Agent Cloud, made Sandboxes generally available, and rebuilt Wrangler CLI around AI agents. OpenAI launched GPT-5.4-Cyber for vetted security professionals. Google DeepMind released Gemini Robotics-ER 1.6 with multi-view spatial reasoning. Nvidia open-sourced Ising for quantum error correction with 2.5x speed and 3x accuracy gains. AWS announced Agent Registry, Interconnect multicloud GA, and Claude Mythos preview on Bedrock. Anthropic debuted Claude Design powered by Opus 4.7. OpenAI shipped a new Agents SDK with a sandbox harness. GitHub added Claude and Codex model selection on github.com. Slash raised $100M, Sygaldry $139M, Helical $10M. Researchers found Anthropic, Google, and Microsoft AI coding agents can leak GitHub credentials through prompt injection. OpenAI confirmed limited exposure from the Axios npm supply chain attack. ### Hybrid Logical Clock in Distributed Systems - URL: https://singhajit.com/distributed-systems/hybrid-clock/ - Date: 2026-04-18 - Tags: distributed-systems - Description: Learn how the Hybrid Logical Clock (HLC) pattern works in distributed systems. Complete guide with examples from CockroachDB, MongoDB, and YugabyteDB. Covers the HLC algorithm, 64 bit timestamp format, causal consistency, consistent snapshots, and how HLC compares to Lamport and vector clocks. - Quick Answer: A Hybrid Logical Clock (HLC) is a 64 bit timestamp made of two parts: a physical wall clock component and a logical counter. The physical part keeps the timestamp close to NTP time so it is human readable. The logical counter advances when two events share the same physical millisecond, which preserves causal ordering across nodes. HLC is monotonic, tolerates NTP drift, fits in a standard 64 bit field, and is used by [CockroachDB](https://www.cockroachlabs.com/blog/clock-management-cockroachdb/){:target="_blank" rel="noopener"}, [MongoDB cluster time](https://www.mongodb.com/blog/post/casual-guarantees-anything-casual){:target="_blank" rel="noopener"}, and [YugabyteDB](https://docs.yugabyte.com/v2024.2/architecture/transactions){:target="_blank" rel="noopener"} to order distributed transactions and produce consistent snapshots without relying on atomic clocks like Google Spanner's TrueTime. ### Low Watermark: How Distributed Systems Know What's Safe to Delete - URL: https://singhajit.com/distributed-systems/low-watermark/ - Date: 2026-04-15 - Tags: distributed-systems - Description: Learn how the Low Watermark pattern controls WAL truncation, log compaction, and log cleanup in Kafka, etcd, PostgreSQL, and ZooKeeper with examples. - Quick Answer: The Low Watermark is an index in the write-ahead log that marks the point below which log entries can be **safely discarded**. It is the counterpart to the [High Watermark](/distributed-systems/high-watermark/). While the High Watermark controls how far forward clients can read (visibility), the Low Watermark controls how far back the system needs to keep data (cleanup). It advances when all nodes have applied entries to their state machines, or when snapshots make old entries redundant. Used by Kafka (log start offset), Raft/etcd (snapshot compaction), PostgreSQL (checkpoint + replication slots), and ZooKeeper (transaction log purging). ### High Watermark: How Distributed Systems Know What's Safe to Read - URL: https://singhajit.com/distributed-systems/high-watermark/ - Date: 2026-04-13 - Tags: distributed-systems - Description: Learn how the High Watermark pattern keeps distributed systems consistent. Complete guide with real-world examples from Kafka, Raft, etcd, and ZooKeeper. Covers commit index, log replication, consumer visibility, ISR, leader election, and the relationship with Low Watermark. - Quick Answer: The High Watermark (also called commit index) is an index in the write-ahead log that marks the last entry safely replicated to a **majority of nodes**. Only entries at or below the high watermark are considered committed and visible to clients. The leader calculates it based on replication acknowledgments and communicates it to followers. Used by Kafka, Raft (etcd, CockroachDB), and ZooKeeper to prevent clients from reading uncommitted data that could be lost during leader failover. ### Dev Weekly Apr 6-12, 2026: Anthropic Mythos Finds Decades-Old Zero-Days, Intel Joins $25B Terafab, Copilot Rubber Duck, Chrome Vertical Tabs - URL: https://singhajit.com/dev-weekly/2026/apr-6-12/anthropic-mythos-glasswing-intel-terafab-copilot-rubber-duck-chrome-vertical-tabs/ - Date: 2026-04-12 - Tags: dev-weekly, tech-news, software-development-news - Description: Developer news for April 6 to 12, 2026. Anthropic Claude Mythos finds thousands of zero-day vulnerabilities in major operating systems and browsers through Project Glasswing, a restricted cybersecurity initiative with Amazon, Apple, Microsoft, Google, and Nvidia. Intel joins Elon Musk's $25 billion Terafab AI chip factory in Austin as foundry partner, contributing its 18A process node for Tesla and SpaceX chips. GitHub Copilot CLI adds Rubber Duck cross-model review using GPT-5.4 to review Claude Sonnet's work, closing 74.7% of the Sonnet-to-Opus performance gap. Chrome 147 ships vertical tabs. VS Code 1.115 launches the Agents app for running multiple agent sessions in parallel. Anthropic revenue run rate passes $30 billion, tripling from $9 billion at end of 2025, and expands Google Cloud TPU capacity by 3.5 gigawatts through Broadcom deal. AWS launches S3 Files for native file system access to S3 buckets. GitHub Dependabot alerts can now be assigned to AI coding agents for automated fix generation. Oracle upgrades AI Database with sub-3-second disaster failover. Docker authorization bypass CVE-2026-34040 silently disables security plugins via oversized requests, affecting 92% of enterprise deployments. Fortinet FortiClientEMS CVE-2026-35616 under active zero-day exploitation. Red Hat shuts down China engineering team and lays off hundreds. Pendo cuts 10% of workforce citing AI. NeuBird AI raises $19.3M. Bolt lays off 30% of staff. ### REST vs GraphQL vs gRPC: How to Pick the Right API Protocol - URL: https://singhajit.com/rest-vs-graphql-vs-grpc/ - Date: 2026-04-09 - Tags: system-design, distributed-systems, software-engineering - Description: A developer's guide to choosing between REST, GraphQL, and gRPC. Covers architecture, performance benchmarks, serialization formats, streaming, error handling, versioning, security, and real-world use cases at Netflix, GitHub, Uber, and Shopify. Includes decision flowchart and code examples. - Quick Answer: **REST** for public APIs, third-party integrations, and simple CRUD operations where HTTP caching matters. **GraphQL** when multiple clients need different data shapes from the same backend, especially mobile apps fighting bandwidth constraints. **gRPC** for internal service-to-service communication where performance is critical, with 5-10x throughput advantage over REST. ### Transactional Outbox Pattern: Never Lose an Event Again - URL: https://singhajit.com/transactional-outbox-pattern/ - Date: 2026-04-07 - Tags: system-design, distributed-systems, software-engineering - Description: Complete guide to the transactional outbox pattern for reliable event publishing in microservices. Learn how to solve the dual write problem using an outbox table, polling relay, and change data capture (CDC) with Debezium and Kafka. Includes outbox table schema, code examples, and production best practices. - Quick Answer: The transactional outbox pattern solves the dual write problem by storing events in an **outbox table** within the same database transaction as your business data. A separate **relay process** reads from the outbox and publishes events to the message broker. This guarantees that events are published if and only if the transaction commits, without needing distributed transactions like 2PC. ### Dev Weekly: Axios npm Supply Chain Attack, Claude Code Source Code Leak, Oracle Lays Off 30,000, Cursor 3, Gemma 4 - URL: https://singhajit.com/dev-weekly/2026/mar-30-apr-5/axios-supply-chain-claude-code-leak-oracle-layoffs-cursor-3-gemma-4/ - Date: 2026-04-05 - Tags: dev-weekly, tech-news, software-development-news - Description: Software developer news for the week of March 30 to April 5, 2026. North Korean state actor Sapphire Sleet hijacks axios on npm, pushing a cross-platform RAT through malicious versions 1.14.1 and 0.30.4 that reached millions of developers. Anthropic accidentally leaks all 512,000 lines of Claude Code source code through an npm source map file, exposing unreleased features including Kairos and Undercover Mode. Oracle lays off up to 30,000 employees in its largest restructuring ever to fund a $50 billion AI data center buildout. Cursor 3 ships with a new agent-first Agents Window, Design Mode, and cloud agent handoff. Google releases Gemma 4 open source models under Apache 2.0. GitHub Copilot SDK enters public preview in five languages. Microsoft open sources an Agent Governance Toolkit covering all 10 OWASP agentic AI risks. Google ADK for Java hits 1.0. JetBrains Rider 2026.1 ships. Docker Offload goes GA. Coder raises $90M. Qodo raises $70M. Depthfirst raises $80M. DigitalOcean acquires Katanemo Labs. Chrome zero-day CVE-2026-5281 patched. Citrix NetScaler critical RCE exploited in the wild. ### Kafka vs RabbitMQ vs Amazon SQS: Picking the Right Message Broker - URL: https://singhajit.com/kafka-vs-rabbitmq-vs-sqs/ - Date: 2026-04-03 - Tags: system-design, distributed-systems, software-engineering - Description: A developer's guide to choosing between Apache Kafka, RabbitMQ, and Amazon SQS. Covers architecture differences, throughput benchmarks, delivery guarantees, message ordering, pricing, and real-world use cases at Uber, Stripe, and Netflix. Updated for Kafka 4.0 and RabbitMQ 4.1. - Quick Answer: **Kafka** if you need high-throughput event streaming, message replay, and multiple consumers reading the same data. **RabbitMQ** if you need flexible routing, low per-message latency, and traditional work queue patterns. **Amazon SQS** if you are on AWS, want zero infrastructure management, and your messaging needs are straightforward. ### OpenTelemetry in Production: A Complete Setup Guide - URL: https://singhajit.com/opentelemetry-production-guide/ - Date: 2026-03-31 - Tags: devops, system-design, software-engineering - Description: A hands-on guide to running OpenTelemetry in production. Covers the OTel Collector, auto-instrumentation for Java, Python, and Go, sampling strategies, Kubernetes deployment patterns, and how to connect traces, metrics, and logs to backends like Prometheus, Jaeger, and Grafana Tempo. - Quick Answer: Start with **auto-instrumentation** to get traces, metrics, and logs without code changes. Deploy the **OpenTelemetry Collector** as a gateway to handle batching, sampling, and export to backends like Prometheus, Jaeger, or Grafana Tempo. Use **head-based sampling** for simplicity, or **tail-based sampling** if you need to keep every error and slow request. In Kubernetes, deploy the Collector as a **DaemonSet** for node-level collection and a **Deployment** for a central gateway. ### Dev Weekly: TeamPCP Supply Chain Attack Grows, Arm Ships First Chip, Codex Gets Plugins, EU Commission Hacked - URL: https://singhajit.com/dev-weekly/2026/mar-23-29/teampcp-supply-chain-arm-agi-cpu-codex-plugins-jetbrains-central/ - Date: 2026-03-29 - Tags: dev-weekly, tech-news, software-development-news - Description: TeamPCP supply chain attack hits LiteLLM and Telnyx on PyPI, stealing credentials from hundreds of thousands of systems. Arm ships its first in-house chip in 35 years with Meta as lead customer. OpenAI adds plugin system to Codex with 20+ integrations. JetBrains launches Central platform for agentic software development. European Commission confirms AWS account breach with 350GB of data stolen. Microsoft and Nvidia partner to use AI for accelerating nuclear reactor deployment. Google unveils TurboQuant to cut AI memory usage by 6x with zero accuracy loss. VS Code 1.113 adds AI reasoning controls and nested subagents. GitHub expands security coverage with AI-powered detections. Amazon acquires Fauna Robotics and the Sprout humanoid robot. Chroma releases Context-1, a 20B parameter model for agentic search. CISA adds Langflow and Trivy to Known Exploited Vulnerabilities catalog. China releases first embodied AI industry standard. Kandou AI raises $225M for copper interconnect technology. Databricks acquires two startups and launches Lakewatch SIEM. Isara raises $94M backed by OpenAI to build AI agent swarms. Black Duck launches Signal for agentic security of AI-generated code. Cloudflare Agents SDK v0.8.0 ships. Google launches Search Live globally. Meta lays off 700 employees. Claude paid subscriptions more than double. March 2026 developer news. ### Model Context Protocol (MCP) Explained - URL: https://singhajit.com/model-context-protocol-mcp-explained/ - Date: 2026-03-25 - Tags: AI, software-engineering - Description: Learn what MCP (Model Context Protocol) is, how it works, and why it matters for AI development. This guide covers MCP architecture (hosts, clients, servers), tools vs resources vs prompts, transport mechanisms, JSON-RPC message flow, building your first MCP server, security best practices, and real-world use cases with Cursor, Claude, and VS Code. - Quick Answer: MCP (Model Context Protocol) is an open standard by Anthropic that gives AI applications a universal way to connect to external tools and data sources. Think of it as **USB-C for AI**: instead of building custom integrations for every tool, you build one MCP server and any MCP-compatible host (Cursor, Claude, VS Code, ChatGPT) can use it. The protocol runs on **JSON-RPC 2.0** and defines three primitives: **Tools** (actions the model can call), **Resources** (data the model can read), and **Prompts** (reusable templates). Communication happens over **stdio** (local) or **Streamable HTTP** (remote). ### What Happens When You Type a URL in the Browser - URL: https://singhajit.com/what-happens-when-you-type-url-in-browser/ - Date: 2026-03-24 - Tags: networking, web-development - Description: What happens when you type a URL in the browser and press Enter? A step-by-step breakdown of the complete journey: URL parsing, DNS lookup, TCP handshake, TLS encryption, HTTP request, server processing, and browser rendering. Explained for developers with diagrams and real-world examples. - Quick Answer: When you type a URL and press Enter, the browser parses the URL, checks its cache, resolves the domain via DNS, opens a TCP connection (three-way handshake), negotiates TLS encryption, sends an HTTP request, receives the response from the server, then parses HTML to build the DOM, builds the CSSOM from CSS, combines them into a render tree, calculates layout, and paints pixels to the screen. The whole process takes under a second on a fast connection. ### Dev Weekly: Nvidia GTC Drops Vera Rubin, Stripe Ships 1,300 PRs/Week with AI, Laravel 13 Arrives - URL: https://singhajit.com/dev-weekly/2026/mar-16-22/nvidia-gtc-vera-rubin-stripe-minions-laravel-13-trivy-attack/ - Date: 2026-03-22 - Tags: dev-weekly, tech-news, software-development-news - Description: Nvidia unveils Vera Rubin platform and DLSS 5 at GTC 2026. Stripe Minions AI agents produce 1,300+ pull requests per week. Microsoft restructures Copilot under new leadership. OpenAI signs AWS deal for US government AI. Java 26 released with HTTP/3 support, Vector API, and ahead-of-time object caching. Laravel 13 ships with AI SDK. Next.js 16.2 delivers 400% faster dev startup. Trivy GitHub Actions compromised in second supply chain attack. GlassWorm malware infects 72 Open VSX extensions. GitLab 18.10 adds passkeys and AI security. March 2026 developer news. ### How Database Locks Work: A Complete Guide - URL: https://singhajit.com/database-locks-explained/ - Date: 2026-03-21 - Tags: database, software-engineering - Description: Learn how database locks work. Covers shared vs exclusive locks, row-level locking, deadlocks, optimistic vs pessimistic locking, MVCC, and SELECT FOR UPDATE with real SQL examples. - Quick Answer: Database locks prevent concurrent transactions from corrupting data. **Shared locks** allow multiple readers. **Exclusive locks** block everyone else for writes. Lock granularity ranges from **row-level** (high concurrency) to **table-level** (low overhead). **Pessimistic locking** grabs locks upfront with `SELECT FOR UPDATE`. **Optimistic locking** uses version columns and checks at commit time. **MVCC** lets readers and writers work without blocking each other. **Deadlocks** happen when transactions wait on each other in a cycle. The database kills one to break the cycle. ### MongoDB Cheat Sheet: 100+ Commands with Real Examples - URL: https://singhajit.com/mongodb-cheat-sheet/ - Date: 2026-03-20 - Tags: database, mongodb, nosql - Description: Master MongoDB with this practical cheat sheet. Covers mongosh commands, CRUD operations, query operators, aggregation pipeline, indexes, schema validation, replication, sharding, backup and restore, and performance tuning. Real examples for software developers. - Quick Answer: Essential mongosh: `show dbs` (list databases), `show collections` (list collections), `db.coll.find()` (query documents), `db.coll.insertOne()` (insert). Key operations: `db.coll.createIndex()` (speed up queries), `db.coll.aggregate()` (data processing), `mongodump` (backup), `mongorestore` (restore). Always create indexes on fields you query often. Use the aggregation pipeline for anything more complex than a simple find. ### Architecting Multi-Agent AI Swarms: A System Design Deep Dive - URL: https://singhajit.com/multi-agent-ai-swarms-system-design/ - Date: 2026-03-19 - Tags: ai, system-design - Description: Learn how to architect multi-agent AI systems. This deep dive covers orchestration patterns (supervisor, pipeline, mesh), inter-agent communication, memory management, framework comparison (LangGraph vs CrewAI vs AutoGen), production failures, and practical lessons for software developers building agentic AI swarms. - Quick Answer: Multi-agent AI swarms distribute work across specialized agents instead of relying on one monolithic agent. The five main orchestration patterns are **Supervisor** (central coordinator delegates to specialists), **Sequential Pipeline** (agents process work in stages), **Peer-to-Peer Mesh** (agents communicate directly via message bus), **Event-Driven** (agents react to event streams), and **Hub-and-Spoke** (central router with independent agents). For production systems, use deterministic workflow engines with LLM decision points, isolate each agent in its own container, and always keep a human in the loop for high-stakes actions. ### Circuit Breaker Pattern Explained: The Complete Guide - URL: https://singhajit.com/circuit-breaker-pattern/ - Date: 2026-03-18 - Tags: system-design, distributed-systems - Description: What is the circuit breaker pattern? Learn how it prevents cascading failures in microservices with closed, open, and half-open states. Includes Resilience4j, Go, and Python examples, architecture diagrams, and real-world patterns from Netflix. - Quick Answer: The circuit breaker pattern prevents cascading failures by wrapping calls to external services. It has three states: **Closed** (requests pass through normally), **Open** (requests fail immediately without calling the service), and **Half-Open** (a few test requests check if the service recovered). When failures cross a threshold, the breaker trips open, giving the failing service time to recover instead of piling on more requests. Use it with retries, bulkheads, and fallbacks for complete fault tolerance. ### Dev Weekly: Anthropic Sues Pentagon, Google Closes $32B Wiz Deal, Atlassian Cuts 1,600 - URL: https://singhajit.com/dev-weekly/2026/mar-9-15/anthropic-sues-pentagon-google-wiz-atlassian-layoffs-nemotron/ - Date: 2026-03-15 - Tags: dev-weekly, tech-news, software-development-news - Description: Anthropic sues Pentagon over supply chain risk label. Google completes $32B Wiz acquisition. Atlassian cuts 1,600 jobs to fund AI. Nvidia launches Nemotron 3 Super open model. Anthropic ships Code Review for Claude Code. DryRun finds 87% of AI-coded PRs have vulnerabilities. Microsoft Patch Tuesday fixes 2 zero-days. March 2026 developer news. ### How to Set Up OpenClaw with Docker - URL: https://singhajit.com/openclaw-docker-setup/ - Date: 2026-03-13 - Tags: ai, docker - Description: Complete guide to setting up OpenClaw with Docker. Covers Docker Compose configuration, prebuilt vs local builds, sandboxing, multi-agent setup, local LLMs with Ollama, production hardening, and troubleshooting common issues like exit code 137 and permission denied errors. - Quick Answer: Clone the repo and run `./docker-setup.sh`, or use a Docker Compose file with the prebuilt image from `ghcr.io/openclaw/openclaw:latest`. The Gateway runs on port **18789** and config persists in `~/.openclaw/` via bind mounts. Use at least **2GB RAM** (1GB hosts will crash with exit code 137). For production, bind to `127.0.0.1`, put a reverse proxy in front, and enable sandboxing to isolate agent tool execution in separate containers. ### How Perplexity Personal Computer Works: Mac Mini as a 24/7 AI Agent - URL: https://singhajit.com/perplexity-computer-explained/ - Date: 2026-03-12 - Tags: ai, software-engineering - Description: How Perplexity Personal Computer turns a Mac mini into a 24/7 AI agent. Architecture, multi-model orchestration, security model, and lessons for developers. - Quick Answer: Perplexity Personal Computer turns a **Mac mini** into an always-on AI worker. The Mac runs locally 24/7 with access to your files and apps, while AI processing happens on Perplexity's cloud using **19 orchestrated models** (Claude Opus 4.6, Gemini, GPT-5.2, Grok, and more). It uses a local-cloud hybrid architecture: the Mac mini is the interface and file access layer, cloud Firecracker VMs handle execution, and a separate cloud browser handles web automation. Available on Perplexity Max at $200/month via waitlist. ### Building a Code Review Assistant with LLMs - URL: https://singhajit.com/building-code-review-assistant-with-llms/ - Date: 2026-03-10 - Tags: ai, software-engineering, tutorial - Description: Learn how to build an AI code review assistant using LLMs. Covers architecture, GitHub webhook integration, prompt engineering for code review, handling false positives, and deploying a production-ready automated PR reviewer. - Quick Answer: An LLM-based code review assistant works in three steps: **1) Capture the PR diff** via GitHub webhooks or Actions, **2) Build context** by combining the diff with repo structure, related files, and your team's coding guidelines, **3) Send to an LLM** with a structured prompt and post the review comments back to the PR. The hard part is not calling the API. It is reducing false positives, managing context windows, and earning developer trust. ### Distributed Tracing: Jaeger vs Tempo vs Zipkin - URL: https://singhajit.com/distributed-tracing-jaeger-vs-tempo-vs-zipkin/ - Date: 2026-03-09 - Tags: devops, system-design, software-engineering - Description: Compare Jaeger, Grafana Tempo, and Zipkin for distributed tracing in microservices. Covers storage backends, sampling strategies, OpenTelemetry setup, cost at scale, and a practical guide to picking the right tool in 2026. - Quick Answer: **Zipkin** if you need something running in under five minutes for learning or a small project. **Jaeger** if you want a battle-tested, standalone tracing backend with rich visualization, adaptive sampling, and a service dependency graph. **Grafana Tempo** if you are already on the Grafana stack and need cheap, high-volume trace retention backed by object storage like S3 or GCS. ### Dev Weekly Mar 2-8, 2026: GPT-5.4 Launch, Cursor AI Agents, US Gov Drops Anthropic, Oracle 30K Cuts - URL: https://singhajit.com/dev-weekly/2026/mar-2-8/gpt-54-cursor-automations-us-agencies-anthropic-oracle/ - Date: 2026-03-08 - Tags: dev-weekly, tech-news, software-development-news - Description: OpenAI GPT-5.4 launches with 1M context and 33% fewer hallucinations. US government drops Anthropic after Trump directive. Cursor Automations, Oracle layoffs, and more. March 2026. ### Prompt Injection: The #1 Security Threat to Your AI Application - URL: https://singhajit.com/prompt-injection-explained/ - Date: 2026-03-07 - Tags: AI, security, software-engineering - Description: Prompt injection is the #1 vulnerability in LLM applications two years running (OWASP LLM01). Learn how direct and indirect attacks work, see real-world incidents, and get practical Python code to defend your application with layered security. - Quick Answer: Prompt injection is when an attacker embeds instructions in content your LLM reads, causing it to ignore your system prompt and do something else. It's OWASP LLM01 for two years running. **Direct injection** comes from user input. **Indirect injection** comes from documents, emails, web pages, or anything the model reads. Defense requires multiple layers: input validation, structured prompts, retrieval hardening, least-privilege tool access, output filtering, and monitoring. No single control is enough. ### Claude Cowork Guide for Software Developers - URL: https://singhajit.com/claude-cowork-guide/ - Date: 2026-03-07 - Tags: AI, software-engineering - Description: A developer-focused Claude Cowork guide. Learn how Anthropic's agentic desktop AI works: VM isolation, observe-plan-act-reflect loop, MCP plugins, and multi-agent orchestration. Setup, use cases, limitations, and when to pair Cowork with Claude Code. - Quick Answer: **Claude Cowork** is Anthropic's agentic desktop AI (released January 12, 2026). It runs in an isolated Linux VM on your Mac or Windows machine, uses an observe-plan-act-reflect loop, and connects to tools via MCP. You give it a folder, describe a task, approve its plan, and come back to finished work. Best for multi-step knowledge work (docs, spreadsheets, reports); pair with **Claude Code** for coding. ### Redis vs DragonflyDB vs KeyDB: Best Redis Alternative in 2026? - URL: https://singhajit.com/redis-vs-dragonflydb-vs-keydb/ - Date: 2026-03-05 - Tags: database, system-design, software-engineering - Description: Redis, DragonflyDB, or KeyDB? Compare architecture, real performance benchmarks, licensing, and when each one is the right choice for your stack in 2026. - Quick Answer: **Redis** if you need maximum ecosystem compatibility, Redis Modules (Search, JSON, TimeSeries), or your team is already running Redis. **DragonflyDB** if you want to eliminate Redis Cluster, reduce memory costs, and get 10-25x higher throughput on the same hardware. **KeyDB** if you want a conservative multithreaded upgrade from Redis with active replication and an open-source license. ### When to Use PostgreSQL vs MongoDB vs DynamoDB (2026 Guide) - URL: https://singhajit.com/postgresql-vs-mongodb-vs-dynamodb/ - Date: 2026-03-04 - Tags: database, system-design, software-engineering - Description: Learn exactly when to use PostgreSQL, MongoDB, or DynamoDB. Covers ACID vs eventual consistency, horizontal scaling, cost at scale, real-world company choices, and common mistakes developers make when picking a database. - Quick Answer: **PostgreSQL** if you have relational data, need complex queries, or want ACID guarantees (finance, SaaS, e-commerce). **MongoDB** if your data is document-shaped and your schema changes often (catalogs, CMS, user profiles). **DynamoDB** if you are on AWS, need massive serverless scale, and know your access patterns upfront (IoT, session stores, high-traffic APIs). ### Dev Weekly Feb 23-Mar 1, 2026: OpenAI $110B, Anthropic vs Pentagon, Claude Used in 150GB Breach - URL: https://singhajit.com/dev-weekly/2026/feb-23-mar-1/openai-110b-anthropic-pentagon-figma-codex-go-126/ - Date: 2026-03-01 - Tags: dev-weekly, tech-news, software-development-news - Description: Developer news Feb 23-Mar 1, 2026: OpenAI raises $110B from Amazon, Nvidia, and SoftBank at $840B valuation. Anthropic refuses Pentagon demands on autonomous weapons safeguards, Claude hits #1 on the App Store. Figma partners with OpenAI to bring Codex into the design workflow. Go 1.26 ships with Green Tea GC on by default. A hacker used Claude to steal 150GB of Mexican government data. Latest software development news. ### Building Your First RAG Application - URL: https://singhajit.com/building-your-first-rag-application/ - Date: 2026-02-24 - Tags: ai, software-engineering, tutorial - Description: Learn how to build your first RAG application step by step. This tutorial covers RAG architecture, document chunking, embeddings, vector databases, and retrieval pipelines. Perfect for developers adding document Q&A to LLM applications. - Quick Answer: Build a RAG application in four steps: **1) Chunk your documents** (split text into meaningful pieces), **2) Create embeddings** (turn chunks into vectors with an embedding model), **3) Store in a vector database** (Chroma, pgvector, or Pinecone), **4) At query time**, embed the question, retrieve the closest chunks, and send them to the LLM as context. The LLM answers using your data instead of guessing. Start with a single document type and simple chunking before scaling. ### How to Build an LLM Application From Scratch - URL: https://singhajit.com/building-your-first-llm-application/ - Date: 2026-02-23 - Tags: ai, software-engineering, tutorial - Description: Learn how to build your first LLM application step by step. This tutorial covers LLM architecture, API integration, conversation memory, error handling, and deployment. Perfect for software developers getting started with LLM-powered applications. - Quick Answer: Build your first LLM application in 4 steps: **1) Choose an LLM API** (OpenAI, Anthropic, or local via Ollama), **2) Create a simple backend** (Node.js/Express or Python/FastAPI), **3) Add conversation memory** (store chat history in database or session), **4) Build a frontend** (HTML/JavaScript or React). Use provider-agnostic code so you can switch models. Add error handling, rate limiting, and cost tracking from day one. ### Dev Weekly Feb 16-22, 2026: India AI Summit $250B, TypeScript 7 Compiler, npm Security Attack - URL: https://singhajit.com/dev-weekly/2026/feb-16-22/india-ai-summit-typescript-7-npm-worm-grok-investigation/ - Date: 2026-02-22 - Tags: dev-weekly, tech-news, software-development-news - Description: Developer news Feb 16-22, 2026: India secures $250B AI investments at New Delhi summit. TypeScript 7 native compiler ships with faster builds. Big Tech cuts buybacks for $500B+ AI infrastructure spending. Critical npm worm attack steals dev secrets. Ireland investigates X Grok AI safety. Latest software development news. ### How an AI Bot Named Kiro Took Down AWS Cost Explorer - URL: https://singhajit.com/aws-outage-kiro-ai-bot/ - Date: 2026-02-21 - Tags: tech-news - Description: Complete analysis of the December 2025 AWS outage caused by Kiro AI bot. The AI tool autonomously decided to delete and recreate a production environment, causing a 13-hour outage. Learn about access control failures, AI automation risks, and critical lessons for developers using AI tools in production. - Quick Answer: AWS Kiro AI bot was given operator-level permissions without mandatory peer review. The AI autonomously decided to delete and recreate a production environment, causing a 13-hour outage affecting AWS Cost Explorer. AWS only introduced safeguards after the incident. Lesson: AI tools need constrained permissions, mandatory approval, and human oversight for production changes. ### Docker Cheat Sheet: 100+ Commands with Real Examples - URL: https://singhajit.com/devops/docker-cheat-sheet/ - Date: 2026-02-20 - Tags: docker, devops, containers - Description: Master Docker with this practical cheat sheet. Covers docker commands, container management, image building, Dockerfile best practices, docker-compose, networking, volumes, Docker Hub, and troubleshooting. Real examples for software developers. - Quick Answer: Core commands: `docker ps` (list containers), `docker images` (list images), `docker run IMAGE` (start container), `docker build -t NAME .` (build image), `docker-compose up` (start services). **Image** = template, **Container** = running instance. Use `-d` for detached mode, `-p HOST:CONTAINER` for ports, `-v HOST:CONTAINER` for volumes. Always use `.dockerignore` to speed up builds. ### How to Solve the Thundering Herd Problem in Distributed Systems - URL: https://singhajit.com/thundering-herd-problem/ - Date: 2026-02-19 - Tags: system-design - Description: What is the thundering herd problem? Learn how cache stampedes crash systems, 6 solutions used by Facebook, Twitter, and Netflix, and practical code examples for preventing thundering herd in distributed systems. Complete guide with architecture diagrams. - Quick Answer: The thundering herd problem occurs when many requests simultaneously hit the same backend resource, usually after a cache key expires. All requests see the miss, all query the database at once, and the database crashes under the spike. **Solutions**: Add jitter to TTLs, use request coalescing (singleflight), distributed locking with stale data fallback, probabilistic early recomputation, or write buffering with queues. ### PostgreSQL Cheat Sheet: 100+ Commands with Real Examples - URL: https://singhajit.com/postgresql-cheat-sheet/ - Date: 2026-02-18 - Tags: database, postgres, sql - Description: Master PostgreSQL with this practical cheat sheet. Covers psql commands, database management, table operations, queries, indexes, joins, JSON, backup and restore, performance tuning with EXPLAIN ANALYZE, roles, and troubleshooting. Real examples for software developers. - Quick Answer: Essential psql: `\l` (list databases), `\dt` (list tables), `\d table_name` (describe table), `\c dbname` (switch database). Key SQL: `EXPLAIN ANALYZE` (query performance), `CREATE INDEX` (speed up queries), `pg_dump` (backup), `COPY` (import/export CSV). Always use transactions for multi-step changes. Use `\x` for readable output on wide tables. ### How Replicated Log Works in Distributed Systems - URL: https://singhajit.com/distributed-systems/replicated-log/ - Date: 2026-02-16 - Tags: distributed-systems - Description: Learn how the Replicated Log pattern keeps distributed systems in sync. Complete guide with real-world examples from Raft, Kafka, etcd, and ZooKeeper. Covers log replication, state machine replication, high-water mark, leader-based replication, log compaction, and failure recovery with diagrams. - Quick Answer: A Replicated Log keeps multiple nodes in sync by maintaining the same ordered sequence of entries across all cluster members. A leader accepts writes, appends them to its log, and replicates entries to followers. Once a **majority** confirms, the entry is committed. Every node applies committed entries in the same order, so they all reach the same state. Used by Raft (etcd, CockroachDB), Kafka (ISR replication), and ZooKeeper (ZAB protocol). ### Dev Weekly Feb 9-15: Anthropic Raises $30B, Seedance 2.0 Goes Viral, Google Launches CodeWiki - URL: https://singhajit.com/dev-weekly/2026/feb-9-15/anthropic-30b-gpt53-codex-gemini-deep-think-interop-2026/ - Date: 2026-02-15 - Tags: dev-weekly, tech-news, software-development-news - Description: Anthropic raises $30B. Seedance 2.0 goes viral — Disney sends cease-and-desist. Google launches CodeWiki and offers exit packages. GPT-5.3 Codex and Gemini Deep Think drop. Dev news Feb 9-15, 2026. ### How Consistent Hashing Works - URL: https://singhajit.com/consistent-hashing-explained/ - Date: 2026-02-13 - Tags: system-design - Description: What is consistent hashing and how does it work? Learn how consistent hashing distributes data across servers, handles scaling with virtual nodes, and powers systems like DynamoDB, Cassandra, and Memcached. A practical guide for system design with diagrams and code. - Quick Answer: Consistent hashing maps both servers and keys onto a circular hash ring (0 to 2^32-1). Each key is assigned to the nearest server clockwise. When a server is added or removed, only **K/N keys** move (K = total keys, N = total servers), instead of rehashing everything. **Virtual nodes** solve uneven distribution by placing each physical server at multiple ring positions. Used by DynamoDB, Cassandra, Memcached, and CDNs. ### Feature Flags and Feature Toggles: Complete Guide for Developers - URL: https://singhajit.com/feature-flags-guide/ - Date: 2026-02-11 - Tags: web-development - Description: Complete guide to feature flags and feature toggles. Learn implementation, best practices, gradual rollouts, A/B testing, and how companies like Meta and eBay use them. - Quick Answer: Feature flags are runtime conditionals that enable or disable code paths without redeployment. They let you deploy code to production while keeping features hidden, enabling gradual rollouts, A/B testing, and instant rollbacks. Use release toggles for new features, experiment toggles for A/B tests, ops toggles for operational control, and permission toggles for user access. Companies like Meta, eBay, and Flickr use feature flags extensively for safe, continuous delivery. ### How Gossip Protocol Works in Distributed Systems - URL: https://singhajit.com/distributed-systems/gossip-dissemination/ - Date: 2026-02-10 - Tags: distributed-systems - Description: What is gossip protocol? Learn how gossip dissemination works in distributed systems with real examples from Cassandra, Consul, and DynamoDB. Covers push, pull, push-pull variants, SWIM protocol, failure detection, anti-entropy repair, and tuning parameters with diagrams and code. - Quick Answer: Gossip dissemination is a peer-to-peer communication pattern where each node randomly picks a few other nodes and shares its state at regular intervals (typically every 1 second). Information spreads exponentially, reaching all N nodes in **O(log N)** rounds. Three variants exist: **push** (send what you know), **pull** (ask what others know), and **push-pull** (exchange both ways). Used by Cassandra, Consul, DynamoDB, and Redis Cluster for membership tracking and failure detection. ### Dev Weekly Feb 2-8, 2026: SpaceX-xAI $1.25T Merger, GitHub Agent HQ, Docker AI Agent, Big Tech AI Spending - URL: https://singhajit.com/dev-weekly/2026/feb-2-8/spacex-xai-merger-anthropic-super-bowl-github-agent-hq/ - Date: 2026-02-08 - Tags: dev-weekly, tech-news, software-development-news - Description: Developer news Feb 2-8, 2026: SpaceX merges with xAI in $1.25T deal. GitHub Agent HQ launches with Claude and Codex support. Docker AI Agent beta released. Big Tech announces $600B AI spending. Anthropic Super Bowl ads challenge OpenAI. Claude Opus 4.6 GA for Copilot. Software development news and AI tools updates. ### How Databases Actually Store Data Internally - URL: https://singhajit.com/how-databases-store-data-internally/ - Date: 2026-02-06 - Tags: database - Description: Learn how databases store data internally. Complete guide covering pages, B-trees, buffer pools, write-ahead logging, and storage engines. Understand heap storage, slotted pages, and why this matters for query performance. - Quick Answer: Databases store data in fixed-size **pages** (typically 8KB or 16KB), not as raw tables. Pages live in files on disk and get cached in a **buffer pool** in memory. Data is organized using **B-trees** for fast lookups. Before any write hits actual data, it goes to the **Write-Ahead Log** for crash recovery. Understanding this helps you write faster queries. ### How to Run Your Own AI Agent with Cloudflare Moltworker - URL: https://singhajit.com/moltworker-self-hosted-ai-agent/ - Date: 2026-02-05 - Tags: AI - Description: Complete Moltworker and OpenClaw guide for developers. Learn to deploy self-hosted AI agents on Cloudflare Workers without hardware. Covers gateway architecture, WhatsApp and Slack integration, and Markdown memory systems. - Quick Answer: Moltworker lets you run OpenClaw (formerly Moltbot) on Cloudflare Workers. It is a self-hosted AI agent that connects to **10+ messaging apps** (WhatsApp, Telegram, Slack, Discord) through a **Gateway** running on port 18789. Memory is stored as **plain Markdown files**, making it debuggable and auditable. Deploy locally with Docker or serverlessly with Cloudflare Workers. ### Django PostgreSQL Setup: From Zero to Production - URL: https://singhajit.com/django-postgresql-setup-from-zero-to-production/ - Date: 2026-02-04 - Tags: database, django, python, postgresql - Description: Django PostgreSQL setup for production: DATABASES in settings.py, migrations, SSL, PgBouncer, managed PostgreSQL on AWS RDS and cloud platforms. Step-by-step developer tutorial. - Quick Answer: Create the project with **uv init**, **uv add django psycopg2-binary**, and **django-admin startproject config .** Create a PostgreSQL database and user, then in **config/settings.py** set DATABASES with ENGINE django.db.backends.postgresql, NAME, USER, PASSWORD, HOST, PORT (use os.environ for credentials). Run **uv run python manage.py migrate** and verify with **manage.py dbshell**. For production, use CONN_MAX_AGE or PgBouncer and SSL in OPTIONS. ### How Netflix Video Processing Pipeline Works - URL: https://singhajit.com/netflix-video-processing-pipeline/ - Date: 2026-02-03 - Tags: system-design, distributed-systems - Description: Deep dive into Netflix video encoding pipeline architecture. Learn how Netflix uses microservices, parallel processing, VMAF quality metrics, and the Cosmos platform to process thousands of video titles at scale. Practical lessons for software developers. - Quick Answer: Netflix processes videos through a three-stage microservices pipeline: **Ingest** (receive and validate source files), **Processing** (split into chunks, encode in parallel across hundreds of EC2 instances using VMAF quality metrics), and **Distribution** (deliver via Open Connect CDN). Each title gets a custom encoding ladder optimized for its content complexity. ### Kubernetes Cheat Sheet: Essential kubectl Commands for Developers - URL: https://singhajit.com/kubernetes-cheat-sheet/ - Date: 2026-02-02 - Tags: kubernetes, devops, containers - Description: Complete Kubernetes cheat sheet with essential kubectl commands, debugging workflows, and practical examples. Learn pod management, deployments, services, and troubleshooting for container orchestration. - Quick Answer: Core debugging: `kubectl get pods`, `kubectl describe pod NAME`, `kubectl logs NAME`. Use `kubectl apply -f` with YAML (declarative) over `kubectl create` (imperative). **Pod** = container(s), **Deployment** = manages pods with replicas, **Service** = stable network endpoint. Always set resource limits. Labels connect everything. ### Dev Weekly: OpenAI Prism, Amazon Layoffs, Gemini Auto-Browse, MCP Apps Go Live - URL: https://singhajit.com/dev-weekly/2026/jan-26-feb-1/openai-prism-amazon-layoffs-gemini-chrome-mcp-apps/ - Date: 2026-02-01 - Tags: dev-weekly, tech-news, software-development-news - Description: OpenAI launches Prism, a free GPT-5.2 powered workspace for scientists. Amazon cuts thousands of corporate jobs. Google adds auto-browse to Chrome with Gemini 3. MCP Apps becomes official extension. OpenAI retires GPT-4o. Developer news Jan 26 - Feb 1, 2026. ### How OpenAI Scales PostgreSQL to 800 Million Users - URL: https://singhajit.com/how-openai-scales-postgresql/ - Date: 2026-02-01 - Tags: database, system-design - Description: Learn how OpenAI scales PostgreSQL to handle 800 million ChatGPT users. Deep dive into connection pooling with PgBouncer, read replicas, horizontal sharding, query optimization, and database architecture patterns used at massive scale. - Quick Answer: OpenAI scales PostgreSQL using: **PgBouncer** connection pooling (thousands of app instances share hundreds of DB connections), **read replicas** for read-heavy workloads, **horizontal sharding** by user_id across multiple instances, and aggressive query optimization with proper indexing. Monitor slow queries and connection limits proactively. ### System Design Cheat Sheet: Concepts Every Developer Should Know - URL: https://singhajit.com/system-design-cheat-sheet/ - Date: 2026-01-31 - Tags: system-design, architecture - Description: A practical system design cheat sheet covering scalability, load balancing, caching, database sharding, CAP theorem, and distributed systems patterns. Essential concepts for building systems that scale and preparing for system design interviews. - Quick Answer: Key concepts: **Horizontal scaling** > vertical. **CAP theorem** = pick 2 of consistency/availability/partition-tolerance. **Caching** with Cache-Aside pattern. **SQL** for transactions, **NoSQL** for scale. **Load balancer** distributes traffic. **Sharding** splits data across DBs. Design for 3-5x expected peak. Always start with requirements. ### How OAuth 2.0 Works: Authorization Flows, Tokens, and PKCE - URL: https://singhajit.com/oauth-2-explained/ - Date: 2026-01-29 - Tags: security, web-development - Description: Learn how OAuth 2.0 works. Understand authorization flows, access tokens, refresh tokens, and PKCE with practical code examples for web and mobile apps. - Quick Answer: OAuth 2.0 grants apps access to resources without sharing passwords. Use **Authorization Code + PKCE** (recommended for most apps). Access tokens are short-lived; refresh tokens get new ones. OAuth = authorization (what you can access). OpenID Connect adds authentication (who you are). Never use implicit flow (deprecated). ### How Google Docs Works Behind the Scenes - URL: https://singhajit.com/how-google-docs-works/ - Date: 2026-01-29 - Tags: system-design - Description: How does Google Docs let multiple people edit the same document at once? Learn the system design behind real-time collaboration, conflict resolution, and instant syncing across users. - Quick Answer: Google Docs uses **Operational Transform (OT)** for real-time collaboration. Each keystroke becomes an operation with position and content. When concurrent edits arrive, the server transforms them to account for each other's changes. WebSocket connections push updates within 50-200ms. Version history uses snapshots + operation logs. ### Skip List Data Structure: A Faster Alternative to Trees - URL: https://singhajit.com/data-structures/skip-list/ - Date: 2026-01-28 - Tags: data-structures, algorithms - Description: Learn how the skip list (skiplist) data structure works. Understand why Redis uses skip lists instead of red-black trees for sorted sets, and why LevelDB uses a skip list for its MemTable. Complete guide with implementation. - Quick Answer: Skip lists are multi-level linked lists with O(log n) search/insert/delete. Each node gets a **random height**. Higher levels act as express lanes for faster traversal. No complex rebalancing like red-black trees. Redis uses skip lists for sorted sets because they're simpler and equally fast. LevelDB uses them for MemTable. Great for concurrent access. ### Getting the Most Out of AI Coding Assistants - URL: https://singhajit.com/ai-coding-assistants-guide/ - Date: 2026-01-26 - Tags: AI, software-engineering - Description: Learn how to use AI coding assistants effectively. Practical tips for GitHub Copilot, Cursor, and Codeium. Best practices for prompt engineering, code review, and boosting developer productivity with AI pair programming. - Quick Answer: Best practices: **Plan first** (write requirements in markdown before coding), **keep prompts short and specific** (don't dump entire codebases), **review like a PR from a junior dev**, use **test-first workflow** (AI writes failing test, you review, AI makes it pass), and **document your codebase** (AI learns your patterns from docs/comments). ### Dev Weekly: Apple's Siri Chatbot, ClickHouse $15B, Turbopack Deep Dive, Grok Unblocked - URL: https://singhajit.com/dev-weekly/2026/jan-19-25/apple-siri-chatbot-clickhouse-turbopack-grok/ - Date: 2026-01-25 - Tags: dev-weekly, tech-news, software-development-news - Description: Apple confirms Gemini-powered Siri chatbot coming late 2026. ClickHouse valued at $15 billion. Turbopack incremental computation explained. Malaysia unblocks Grok AI. Google launches Me Meme. Amazon job cuts reported. Developer news Jan 19-25, 2026. ### How Meta Handles Millions of Serverless Function Calls Per Second - URL: https://singhajit.com/meta-xfaas-serverless-at-scale/ - Date: 2026-01-24 - Tags: system-design, distributed-systems - Description: Deep dive into Meta's XFaaS serverless platform. Learn how they handle 11.5 million function calls per second with 66% CPU utilization. Practical lessons on cold start elimination, load distribution, congestion control, and building serverless systems at scale. - Quick Answer: Meta's XFaaS handles **11.5M function calls/sec** across 100K+ servers with 66% CPU utilization. Key innovations: **universal workers** eliminate cold starts, load spreading across **time** (defer to off-peak) and **space** (route to other datacenters), and congestion control. Only used for non-user-facing functions due to variable latency. ### X Algorithm Explained: How the Open Source Recommendation System Works - URL: https://singhajit.com/system-design/x-twitter-for-you-algorithm/ - Date: 2026-01-22 - Tags: system-design, machine-learning, software-engineering - Description: The X algorithm explained — a detailed breakdown of how Twitter's open source recommendation algorithm works in 2026. Covers the full GitHub repository (xai-org/x-algorithm), the Grok-based transformer ranking model, Two-Tower retrieval, candidate pipeline architecture, scoring weights, and practical lessons for building recommendation systems at scale. - Quick Answer: X's open source recommendation algorithm (GitHub: xai-org/x-algorithm) works in three stages: **Candidate sourcing** (500M daily posts → ~1,500 candidates from follows + ML discovery), **Ranking** (Grok-based Phoenix transformer predicts engagement), **Filtering** (remove duplicates, blocked content, apply diversity). Written in Rust (62.9%) + Python (37.1%). Two-Tower model finds out-of-network content. ### How Does an LLM Generate Text? - URL: https://singhajit.com/how-llms-generate-text/ - Date: 2026-01-22 - Tags: AI, software-engineering - Description: Learn how large language models generate text step by step. Understand tokenization, transformer architecture, attention mechanism, and sampling strategies. A practical guide for software developers. - Quick Answer: LLMs generate text **one token at a time** in an autoregressive loop. Tokenization → Embedding → Transformer (attention mechanism) → Softmax probabilities → Sample next token → Repeat. **Temperature** controls randomness (low=focused, high=creative). **Top-p** limits sampling pool. LLMs predict likely tokens, not accurate ones - causing hallucinations. ### Complete Guide to Graph Data Structure: BFS, DFS, Adjacency List vs Matrix - URL: https://singhajit.com/data-structures/graph/ - Date: 2026-01-20 - Tags: data-structures, algorithms - Description: BFS and DFS run in O(V+E) with an adjacency list and O(V²) with an adjacency matrix. See why, with code, comparison tables, and 20+ graph algorithms. - Quick Answer: BFS and DFS both run in **O(V + E)** time with an adjacency list and **O(V²)** with an adjacency matrix. Adjacency list = O(V+E) space, best for sparse graphs. Adjacency matrix = O(V²) space, fast edge lookup. Use BFS for shortest unweighted paths; use DFS for cycle detection and topological sort. ### Dev Weekly: Microsoft Patch Tuesday Fixes 114 Flaws, Cloudflare Acquires Astro - URL: https://singhajit.com/dev-weekly/2026/jan-12-18/microsoft-patch-tuesday-lambdatest-testmu-github-restructure/ - Date: 2026-01-18 - Tags: dev-weekly, tech-news, software-development-news - Description: Microsoft Patch Tuesday fixes 114 vulnerabilities including zero-days. Cloudflare acquires Astro. LambdaTest rebrands to TestMu AI. GitHub restructures for AI. OpenAI acquires Torch. Developer news Jan 12-18, 2026. ### B-Tree Data Structure: How Databases Search Billions of Records - URL: https://singhajit.com/data-structures/b-tree/ - Date: 2026-01-18 - Tags: data-structures, algorithms - Description: Learn how the B-tree data structure powers database indexing. Covers B-tree vs B+ tree, insertion, deletion, and why PostgreSQL and MySQL use them. - Quick Answer: B-trees are self-balancing trees where nodes can have many children, minimizing tree height and disk access. A billion records = ~3-4 levels vs 30 in binary tree. **B+ trees** store data only in leaves and link them for fast range queries. PostgreSQL and MySQL use B+ trees for indexes. All operations O(log n). ### How HyperLogLog Works - URL: https://singhajit.com/data-structures/hyperloglog/ - Date: 2026-01-17 - Tags: data-structures - Description: Learn how HyperLogLog estimates unique counts in massive datasets using minimal memory. Used by Redis, Presto, and BigQuery for cardinality estimation. - Quick Answer: HyperLogLog estimates unique item count using only **12 KB** of memory regardless of dataset size. It tracks max leading zeros in hashed values across buckets. Standard error ~1-2%. Used by Redis (`PFADD`/`PFCOUNT`), BigQuery, Presto for COUNT DISTINCT. Sketches can be merged for distributed counting. ### How Count-Min Sketch Works - URL: https://singhajit.com/data-structures/count-min-sketch/ - Date: 2026-01-17 - Tags: data-structures - Description: Learn how Count-Min Sketch estimates item frequencies in data streams using minimal memory. Find heavy hitters and track counts without storing every item. - Quick Answer: Count-Min Sketch estimates item frequencies in streams using a 2D array of counters and multiple hash functions. Query returns the **minimum** of all hash positions. Never underestimates, may overestimate due to collisions. Perfect for finding heavy hitters (most frequent items). Sketches can be merged by adding counters. ### How Bloom Filters Work - URL: https://singhajit.com/data-structures/bloom-filter/ - Date: 2026-01-16 - Tags: data-structures - Description: What is a Bloom filter and how does it work? A simple guide to this space-efficient data structure used in databases, caches, and web browsers. - Quick Answer: Bloom filters answer 'is X in the set?' using a bit array and multiple hash functions. If it says **NO**, definitely not present. If **YES**, probably present (false positives possible). 1 billion items at 1% false positive rate = ~1.2 GB. Used by Cassandra, Bigtable, Chrome. Cannot delete items from standard Bloom filter. ### Local LLM Speed: RTX 3060, Qwen2 & Llama Benchmark Results - URL: https://singhajit.com/llm-inference-speed-comparison/ - Date: 2026-01-15 - Tags: AI, performance, benchmarks - Description: RTX 3060 12GB runs 14B models at 23 tok/s and 8B at 42 tok/s via llama.cpp. Benchmarks for RTX 4070, RTX 4090, Qwen2.5, and Llama 3.1 8B with cited sources. - Quick Answer: Real benchmarks: **RTX 3060 12GB runs 14B models at ~23 tokens/sec** with Q4 via llama.cpp (Hardware Corner). **8B models on RTX 3060** = 42 tok/s. **RTX 4070** = 52 tok/s for 8B, 33 tok/s for 14B. **RTX 4090** = 104 tok/s for 8B, 69 tok/s for 14B. llama.cpp is 3-10% faster than Ollama on NVIDIA GPUs. Q4_K_M quantization offers the best size/quality balance. ### How Snowflake IDs Work - URL: https://singhajit.com/snowflake-id-guide/ - Date: 2026-01-14 - Tags: system-design - Description: Learn how Snowflake IDs work, their 64-bit structure, and how to implement them in Java. Understand Discord's snowflake ID length, Twitter's timestamp bits, and why companies choose Snowflake over UUID for distributed systems. - Quick Answer: Snowflake IDs are 64-bit unique identifiers: **41-bit timestamp** + **10-bit machine ID** + **12-bit sequence**. Each server generates IDs independently without coordination. IDs are time-sortable, smaller than UUIDs (64 vs 128 bits), and efficient as database primary keys. Extract timestamp by right-shifting 22 bits and adding epoch. ### How to Run LLMs on Your Own Computer - URL: https://singhajit.com/running-llms-locally/ - Date: 2026-01-13 - Tags: AI, software-engineering - Description: Learn how to run large language models locally on your own hardware. This guide covers Ollama, llama.cpp, LM Studio, hardware requirements, quantization, and practical use cases for local LLM deployment. - Quick Answer: **Ollama** is the easiest way: `brew install ollama` then `ollama run llama3`. Hardware: 8GB RAM for 7B models, 16GB for 13B, 32GB+ for 70B. Use **Q4_K_M quantization** for best size/quality balance. Ollama provides an OpenAI-compatible API. Benefits: free, private, works offline. Alternative: LM Studio for GUI. ### How Database Indexing Works - URL: https://singhajit.com/database-indexing-explained/ - Date: 2026-01-13 - Tags: software-engineering, database - Description: Learn how database indexing works, from B-tree internals to practical query optimization. Understand clustered vs non-clustered indexes, composite indexes, covering indexes, and how to use EXPLAIN to analyze query performance. - Quick Answer: Indexes are B-tree structures giving O(log n) lookups instead of O(n) scans. **Clustered** = physical row order (one per table). **Composite** = multi-column (leftmost prefix rule). **Covering** = includes all needed columns. Use `EXPLAIN` to verify index usage. Avoid indexing: small tables, low-cardinality columns, write-heavy tables. ### Universal Commerce Protocol (UCP) Explained - URL: https://singhajit.com/universal-commerce-protocol-explained/ - Date: 2026-01-12 - Tags: AI, software-engineering - Description: Learn what Universal Commerce Protocol (UCP) is and how it enables AI agents to shop on behalf of users. Understand the architecture, core capabilities like checkout and order management, and how to integrate UCP into your ecommerce platform. - Quick Answer: UCP is Google's open standard for AI-driven shopping. Core capabilities: **Checkout** (cart, pricing, payments via AP2), **Identity Linking** (OAuth 2.0), **Order Management** (webhooks for status/shipping/returns). Partners: Shopify, Etsy, Wayfair, Target, Walmart. Works with MCP and A2A protocols. One integration works everywhere. ### Dev Weekly: GlassWorm Malware Steals macOS Dev Credentials, C# Wins Language of 2025, Postman Buys Fern - URL: https://singhajit.com/dev-weekly/2026/jan-5-11/postman-fern-glassworm-csharp-tiobe-vibe-coding/ - Date: 2026-01-11 - Tags: dev-weekly, tech-news, software-development-news - Description: GlassWorm malware targets macOS developers via malicious VS Code extensions. C# named TIOBE Language of the Year 2025. Postman acquires Fern. Senior dev salaries hit $235K as shortage worsens 40%. Weekly developer news Jan 5-11, 2026. ### Git Config: How to Set, Edit & Configure Git (with gitconfig Example) - URL: https://singhajit.com/git-config-guide/ - Date: 2026-01-11 - Tags: git, version-control, devops - Description: Complete git config guide with gitconfig example you can copy. Learn how to set git config options, edit git config file, configure git repo config, and use settings like feature.manyFiles true. Covers git setup config for user identity, aliases, editors, credentials, commit signing, and performance. - Quick Answer: To set git config: `git config --global key value` (e.g. `git config --global user.name "Your Name"`). To edit gitconfig file: `git config --global --edit` or manually at `~/.gitconfig`. Git config levels: system < global < local (local wins). Essential gitconfig options: `user.name`, `user.email`, `core.editor`, `pull.rebase=true`, `fetch.prune=true`, `feature.manyFiles=true`. Git repo config lives at `.git/config`. ### Git Cheat Sheet: Commands Every Developer Should Know - URL: https://singhajit.com/git-cheat-sheet/ - Date: 2026-01-11 - Tags: git, version-control, devops - Description: A practical Git cheat sheet with examples. Learn essential Git commands for branching, merging, undoing changes, and working with remote repositories. Includes common workflows and tips for solving everyday problems. - Quick Answer: Key commands: `git status` (check state), `git log --oneline` (history), `git stash` (save work temporarily), `git reset --soft HEAD~1` (undo commit, keep changes), `git reset --hard` (discard changes), `git rebase -i` (clean history). `fetch` downloads without merging; `pull` = fetch + merge. Never rebase shared commits. ### How to Build AI Agents That Actually Work - URL: https://singhajit.com/building-ai-agents/ - Date: 2026-01-10 - Tags: AI, software-engineering - Description: Learn how to build AI agents from scratch. Understand the ReAct loop, tool calling, memory patterns, and multi-agent systems with practical Python code examples. - Quick Answer: AI agents use the **ReAct loop**: Observe → Think → Act → Observe result → Repeat. Key components: **Tools** (functions the agent can call), **Memory** (working, short-term, long-term via vector DB), **Planning** (reasoning about next steps). Frameworks: LangChain, LangGraph, AutoGen, CrewAI. Or build from scratch with OpenAI function calling. ### Context Engineering Guide for AI Developers - URL: https://singhajit.com/context-engineering/ - Date: 2026-01-09 - Tags: AI, software-engineering - Description: Context engineering explained for developers. Learn to provide AI with the right information using RAG, memory management, and dynamic context loading. - Quick Answer: Context engineering = designing what information AI sees. Components: **system instructions** (behavior), **RAG** (retrieved docs), **memory** (conversation history), **tools** (capabilities), **user query**. Prompt engineering is a subset. The real skill is dynamically loading the right context within token limits. ### 23 Must-Know Gang of Four Design Patterns - URL: https://singhajit.com/gang-of-four-design-patterns/ - Date: 2026-01-08 - Tags: design-patterns - Description: Must-know Gang of Four patterns: Complete guide to all 23 design patterns every developer should know. Learn creational, structural, and behavioral patterns with practical examples, decision guides, and real world use cases. - Quick Answer: Gang of Four: 23 classic patterns in 3 categories. **Creational** (Singleton, Factory, Builder, Prototype, Abstract Factory), **Structural** (Adapter, Decorator, Facade, Proxy, Composite, Bridge, Flyweight), **Behavioral** (Strategy, Observer, Command, State, Template Method, Iterator, Mediator, Memento, Visitor, Chain of Responsibility, Interpreter). ### Dev Weekly: 2025 Year in Review - The Year AI Became Normal - URL: https://singhajit.com/dev-weekly/2025-year-in-review/ - Date: 2026-01-04 - Tags: dev-weekly, tech-news, software-development-news, 2025-year-in-review - Description: The definitive look back at 2025's biggest tech stories: Nvidia hits $5 trillion, AI transforms development, Python drops the GIL, major acquisitions reshape the industry, and what it all means for developers going into 2026. ### Majority Quorum in Distributed Systems Explained - URL: https://singhajit.com/distributed-systems/majority-quorum/ - Date: 2026-01-03 - Tags: distributed-systems - Description: Learn Majority Quorum: the consensus pattern behind Cassandra, etcd, and ZooKeeper. Master the W+R>N formula, fault tolerance, and split-brain prevention. - Quick Answer: Majority quorum requires **floor(N/2)+1** nodes to agree. For 5 nodes, quorum is 3. The **W+R>N** formula ensures consistency: if Write quorum + Read quorum > Total nodes, at least one read node has the latest write. Odd node counts (3, 5, 7) are preferred to prevent ties during network splits. ### 50+ Linux Commands Cheat Sheet: The Complete Developer Guide - URL: https://singhajit.com/linux-commands-cheat-sheet/ - Date: 2026-01-02 - Tags: linux, devops, command-line - Description: Master 50+ essential Linux commands with practical examples. Covers file management, process control, networking, permissions, grep, find, SSH, and system administration for developers. - Quick Answer: Essential Linux: `ls -la` (list), `cd` (navigate), `grep -rn "text" dir` (search), `find . -name "*.js"` (find files), `ps aux` (processes), `kill -9 PID` (stop), `chmod 755` (permissions), `df -h` (disk), `du -sh *` (sizes), `lsof -i :PORT` (port check), `ssh user@host` (remote). ### Regex Cheat Sheet: Patterns Every Developer Should Know - URL: https://singhajit.com/regex-cheat-sheet/ - Date: 2026-01-01 - Tags: programming, regex - Description: A practical regex cheat sheet with real examples. Learn regular expression syntax, character classes, quantifiers, lookaheads, and common patterns for email, URL, and phone validation. - Quick Answer: Core regex: `.` matches any char, `*` = 0+, `+` = 1+, `?` = 0 or 1, `\d` = digit, `\w` = word char, `\s` = whitespace, `^` = start, `$` = end. Use `()` for capture groups, `(?:)` for non-capturing. Lazy: `*?`, `+?`. Flags: `i` (case-insensitive), `g` (global), `m` (multiline). ### Dev Weekly: SoftBank's $22.5B OpenAI Bet, Disney + Sora (Dec 22–28) - URL: https://singhajit.com/dev-weekly/2025/dec-22-28/softbank-openai-disney-sora-tiktok/ - Date: 2025-12-28 - Tags: dev-weekly, tech-news, software-development-news - Description: SoftBank invests $22.5B in OpenAI. Disney licenses characters for Sora. TikTok US joint venture with Oracle. Developer news Dec 22-28, 2025. ### Caching Strategies Explained: The Complete Guide - URL: https://singhajit.com/caching-strategies-explained/ - Date: 2025-12-24 - Tags: software-engineering - Description: Complete guide to caching strategies and patterns. Learn cache-aside, read-through, write-through, write-behind patterns, Redis caching techniques, database caching strategies, and cache invalidation strategies with real-world examples. - Quick Answer: Five main strategies: **Cache-Aside** (app manages cache, most common), **Read-Through** (cache fetches on miss), **Write-Through** (sync to both), **Write-Behind** (cache first, async to DB), **Write-Around** (bypass cache on writes). LRU eviction for most cases. Start with Cache-Aside + Redis. Use Write-Through for critical consistency. ### CQRS Pattern: Splitting Read and Write Models - URL: https://singhajit.com/cqrs-pattern-guide/ - Date: 2025-12-23 - Tags: architecture, system-design - Description: Learn the CQRS pattern with practical examples. Understand when to use Command Query Responsibility Segregation, see real implementation code, and avoid common mistakes developers make. - Quick Answer: CQRS (Command Query Responsibility Segregation) uses separate models for reads and writes. Commands modify state; Queries retrieve it. Optimize each independently - denormalized read models for fast queries, normalized write models for consistency. Start with logical separation in one database; add physical separation only when needed. ### The Complete HTMX Guide: From Zero to Production - URL: https://singhajit.com/htmx-guide-modern-web-development/ - Date: 2025-12-22 - Tags: web-development - Description: HTMX tutorial and guide with practical examples. Learn HTMX from zero to production with code examples covering hx-get, hx-post, hx-swap, hx-trigger, and real-world patterns. Complete htmx guide for developers switching from React. - Quick Answer: HTMX extends HTML with attributes (`hx-get`, `hx-post`, `hx-target`, `hx-swap`) that make AJAX requests and swap HTML fragments without writing JavaScript. Server returns HTML, not JSON. Ideal for CRUD apps, admin panels, and teams with backend strength. 14KB gzipped, no build step. Use React/Vue for complex client-side state. ### Dev Weekly: Cursor Acquires Graphite, Coursera Merges with Udemy, Privacy Extensions Caught Selling AI Chats (Dec 15–21, 2025) - URL: https://singhajit.com/dev-weekly-cursor-graphite-coursera-udemy-dec-15-21/ - Date: 2025-12-21 - Tags: dev-weekly, tech-news, software-development-news - Description: Cursor acquires Graphite. Coursera and Udemy merge in $2.5B deal. Privacy extensions caught selling 8M users AI conversations. GitHub reverses self-hosted runner fees. Let's Encrypt shortens certificate lifetimes. US Genesis Mission with 24 tech companies. Google Gemini 3 Flash. OpenAI GPT-5.2 Codex. Developer news Dec 15-21, 2025. ### How Google Ads Supports 4.8 Billion Users with a SQL Database - URL: https://singhajit.com/how-google-ads-scales-with-spanner/ - Date: 2025-12-19 - Tags: system-design - Description: Deep dive into Google Spanner architecture. Learn how Google Ads handles 4.8 billion users with a globally distributed SQL database. Covers TrueTime, Paxos, automatic sharding, and practical lessons for building scalable systems. - Quick Answer: Google Spanner is a globally distributed SQL database with strong consistency. It uses **TrueTime** (atomic clocks + GPS in every datacenter) to order transactions globally and **Paxos** for consensus. Unlike traditional sharding, Spanner handles it automatically while providing full SQL with joins and ACID transactions across shards. ### Role of Queues in System Design - URL: https://singhajit.com/role-of-queues-in-system-design/ - Date: 2025-12-17 - Tags: system-design - Description: Deep dive into message queues in system design. Learn when and why to use queues, popular queue technologies like RabbitMQ, Kafka, and SQS, and real-world patterns from Uber, Slack, and Stripe. Practical guide with architecture diagrams and code examples. - Quick Answer: Message queues decouple services, absorb traffic spikes, and enable async processing. **Kafka**: high-throughput streaming with replay. **RabbitMQ**: complex routing and request-reply. **SQS**: simple AWS-native. Dead letter queues store failed messages for debugging. Pub/sub broadcasts to all subscribers; point-to-point delivers to one consumer. ### Dev Weekly: IBM Buys Confluent for $11B, Linux Foundation Launches Agentic AI Foundation, 30+ AI Coding Extensions Have Security Flaws (Dec 8–14, 2025) - URL: https://singhajit.com/dev-weekly-ibm-confluent-agentic-ai-foundation-dec-8-14/ - Date: 2025-12-14 - Tags: dev-weekly, tech-news, software-development-news - Description: IBM acquires Confluent for $11B. Linux Foundation launches Agentic AI Foundation with Anthropic MCP, OpenAI AGENTS.md, Block goose. 30+ AI coding extension vulnerabilities. Infragistics open sources Ignite UI. Developer news Dec 8-14, 2025. ### How Amazon S3 Stores 100 Trillion Objects Without Losing One - URL: https://singhajit.com/how-amazon-s3-works/ - Date: 2025-12-10 - Tags: system-design - Description: Deep dive into Amazon S3 architecture. Learn how S3 achieves 11 nines durability, handles massive scale, and why understanding it makes you a better developer. Includes practical examples, diagrams, and real-world insights. - Quick Answer: S3 achieves **11 nines durability** (99.999999999%) by storing objects redundantly across multiple AZs, checksumming data, and auto-repairing corruption. Since Dec 2020, S3 is **strongly consistent** - reads immediately reflect writes. Use multipart upload for files >100MB (parallel, resumable). Storage classes: Standard → Intelligent-Tiering → Glacier → Deep Archive. ### Dev Weekly: Anthropic Acquires Bun, Cloudflare Goes Down Again, Cursor Hits $2.3B (Dec 1–7, 2025) - URL: https://singhajit.com/dev-weekly-anthropic-bun-cloudflare-react-cve-dec-1-7/ - Date: 2025-12-07 - Tags: dev-weekly, tech-news, software-development-news - Description: Anthropic acquires Bun. Cloudflare outage hits 28% of traffic. React2Shell CVE-2025-55182. Cursor valued at $2.3B. OpenAI-Accenture $3B deal. Developer news Dec 1-7, 2025. ### Two-Phase Commit: The Protocol That Keeps Distributed Transactions Honest - URL: https://singhajit.com/distributed-systems/two-phase-commit/ - Date: 2025-12-06 - Tags: distributed-systems - Description: What is 2 phase commit? Complete guide to the two phase commit protocol (2PC) for distributed transactions. Learn how the 2 phase commit protocol coordinates atomicity across multiple databases, its phases, failure scenarios, and implementations in PostgreSQL, MySQL, and microservices. - Quick Answer: Two-Phase Commit (2PC) coordinates distributed transactions: **Phase 1 (Prepare)** - coordinator asks participants to vote YES/NO. **Phase 2 (Commit)** - if all vote YES, coordinator sends commit; otherwise abort. Ensures atomicity but is blocking - participants wait indefinitely if coordinator fails. Saga pattern is the non-blocking alternative. ### Cloudflare Outage December 2025: A Nil Value Exception That Lurked for Years - URL: https://singhajit.com/cloudflare-outage-december-2025/ - Date: 2025-12-06 - Tags: tech-news - Description: Complete technical analysis of the December 5, 2025 Cloudflare outage that affected 28% of global HTTP traffic. Deep dive into how a Lua nil value exception, dormant for years, was triggered by a killswitch applied to an execute action for the first time. Timeline, root cause analysis, and lessons for developers on type safety, feature flags, and deployment strategies. - Quick Answer: A Lua nil value exception dormant for years was triggered when a killswitch was applied to an 'execute' action for the first time. The code assumed action='execute' meant the execute field existed, but the killswitch made it nil. 28% of global HTTP traffic affected for 25 minutes. Rust's type system prevents this class of bug. ### Modular Monolith: The Architecture Most Teams Actually Need - URL: https://singhajit.com/modular-monolith-architecture/ - Date: 2025-12-04 - Tags: system-design - Description: Learn how modular monolith architecture combines the simplicity of monoliths with the organization of microservices. Understand module boundaries, communication patterns, and when to choose this architecture over microservices. - Quick Answer: A modular monolith is a single deployable app with **well-defined modules** that communicate through explicit interfaces, not by reaching into each other's internals. Combines monolith simplicity (single deploy, no network calls) with microservice organization (clear boundaries, team autonomy). Shopify and GitHub use this. Easy to extract modules to services later. ### The Complete Guide to Server-Sent Events (SSE) - URL: https://singhajit.com/server-sent-events-explained/ - Date: 2025-12-03 - Tags: system-design - Description: What is SSE? Server-Sent Events is a web standard for real-time server-to-client streaming over HTTP. Learn about the EventSource API, retry field default 3000 ms per MDN and the HTML specification, auto-reconnection behavior, Last-Event-ID, and when to choose SSE over WebSockets. - Quick Answer: SSE (Server-Sent Events) enables **server-to-client streaming** over a single HTTP connection using `text/event-stream`. Browser's EventSource API auto-reconnects (default 3000ms retry), sends Last-Event-ID for replay. One-way only (server→client). Use for dashboards, notifications, live scores. Simpler than WebSockets when bidirectional isn't needed. ### Long Polling Explained: Build Real-Time Apps Without WebSockets - URL: https://singhajit.com/long-polling-explained/ - Date: 2025-12-02 - Tags: system-design - Description: Learn how Long Polling enables real-time communication using plain HTTP. Understand the implementation, trade-offs, and when to choose Long Polling over WebSockets or Server-Sent Events. - Quick Answer: Long polling: client sends request, server **holds connection open** until data arrives or timeout, then client immediately reconnects. Works everywhere HTTP works, even through restrictive firewalls/proxies. Use when WebSockets are blocked, updates are infrequent (<1/sec), or on serverless. More overhead than WebSockets due to HTTP headers per request. ### How Stock Brokers Push 1 Million Price Updates Per Second to Your Screen - URL: https://singhajit.com/how-stock-brokers-handle-real-time-price-updates/ - Date: 2025-12-01 - Tags: system-design - Description: How stock brokers deliver millions of real-time price updates per second using WebSockets, Kafka, and ticker plants. Complete system design guide covering the fan-out problem, low-latency architecture, and real-time data distribution. - Quick Answer: Stock prices flow through: **Exchange** → **Ticker Plant** (normalizes formats) → **Kafka** (distributes) → **Broker Backend** → **WebSocket** → **Client**. The fan-out problem (1 update to millions of users) is solved by Kafka topics and connection-per-user WebSockets. Total latency: 300-500ms from exchange to screen. ### Dev Weekly: AWS's 900 Data Centers Revealed, GPT-5 in Copilot Studio, Meta Wants Google TPUs (Nov 24–30, 2025) - URL: https://singhajit.com/dev-weekly-aws-data-centers-gpt5-copilot-nov-24-30/ - Date: 2025-11-30 - Tags: dev-weekly, tech-news, software-development-news - Description: AWS reveals 900+ global data centers. Microsoft releases GPT-5 Chat in Copilot Studio. Meta negotiating to buy Google TPUs. Canon subsidiary hit by Oracle ransomware. Alibaba Qwen AI reaches 10M downloads. White House Genesis Mission announced. Developer news November 24-30, 2025 ### Dev Weekly: Cloudflare Outage, Jeff Bezos' $6.2B AI Startup, X Encrypted Chat (Nov 17–23, 2025) - URL: https://singhajit.com/dev-weekly-cloudflare-outage-google-antigravity-nov-17-23/ - Date: 2025-11-23 - Tags: dev-weekly, tech-news, software-development-news - Description: Cloudflare outage disrupts millions for 6 hours. Jeff Bezos launches Project Prometheus AI startup with $6.2B. X introduces encrypted chat. AMD challenges Nvidia AI dominance. Google Antigravity IDE launch. Developer news November 17-23, 2025 ### HOCON vs YAML vs TOML vs JSON: Complete Configuration Format Comparison - URL: https://singhajit.com/configuration-file-formats-comparison/ - Date: 2025-11-21 - Tags: software-engineering - Description: HOCON vs YAML vs TOML vs JSON comparison guide. Learn which configuration format to use with syntax examples, performance benchmarks, and real-world use cases from Docker, Kubernetes, Rust, and Akka. - Quick Answer: **JSON**: fastest parsing, best for APIs. **YAML**: DevOps standard (Docker, K8s), indentation-based. **TOML**: Rust/Python configs, explicit types, no ambiguity. **HOCON**: JVM apps (Akka, Play), supports variables `${var}` and file includes. All valid JSON is valid HOCON. Choose based on ecosystem, not features. ### Cloudflare Global Outage: How a Database Permission Change Broke the Internet - URL: https://singhajit.com/cloudflare-outage-november-2025/ - Date: 2025-11-19 - Tags: tech-news - Description: Complete technical analysis of the November 18, 2025 Cloudflare global outage that disrupted ChatGPT, X (Twitter), Spotify, Dropbox, Coinbase, and millions of websites. Timeline, root cause analysis, impact assessment, and critical lessons for developers on configuration management and system resilience. - Quick Answer: A database permission change caused duplicate entries in Cloudflare's Bot Management config, doubling the file size beyond a hard limit. The traffic routing software crashed globally, taking down ChatGPT, X, Spotify, and millions of sites for 6 hours. Lesson: validate configs in staging and set size limits with headroom. ### Dev Weekly: Microsoft's $10B Portugal AI Hub, .NET 10 LTS, TypeScript Takes #1 Spot (Nov 10–16, 2025) - URL: https://singhajit.com/dev-weekly-microsoft-portugal-dotnet10-typescript-nov-10-16/ - Date: 2025-11-16 - Tags: dev-weekly, tech-news, software-development-news - Description: Microsoft announces $10B AI hub in Portugal. .NET 10 LTS released with 30% faster startup times. TypeScript overtakes Python as GitHub's top language. Meta signs $3B cloud deal. React Native critical vulnerability affects 2M projects. Python 3.14 drops with free-threaded execution. Developer news November 10-16, 2025 ### Heartbeat: How Distributed Systems Know You're Still Alive - URL: https://singhajit.com/distributed-systems/heartbeat/ - Date: 2025-11-15 - Tags: distributed-systems - Description: Learn how heartbeat mechanisms detect failures in distributed systems. Master failure detection patterns with real-world examples from Kubernetes, Cassandra, HAProxy, and etcd. Complete guide covering push/pull patterns, gossip protocols, split brain prevention, and Phi Accrual detection. - Quick Answer: Heartbeats are periodic signals indicating a node is alive. **Push-based**: nodes send 'I'm alive' to a monitor. **Pull-based**: monitor queries nodes. Timeout = ~3x heartbeat interval. Quorum prevents split-brain (both sides thinking the other is dead). Kubernetes uses liveness probes (restart if dead) and readiness probes (stop traffic if unready). ### Your JSON is Costing You Thousands: Why TOON Might Save Your Budget - URL: https://singhajit.com/toon-vs-json-token-efficient-format/ - Date: 2025-11-14 - Tags: artificial-intelligence - Description: Learn how TOON (Token-Oriented Object Notation) reduces LLM token usage by 30-60% compared to JSON. Practical examples, benchmarks, and real-world use cases for cutting API costs while working with GPT, Claude, and other language models. - Quick Answer: TOON (Token-Oriented Object Notation) reduces LLM token usage by 30-60% by separating headers from values in arrays. Instead of repeating `{"name":"...","age":...}` for each item, TOON uses `name|age` header and pipe-separated rows. A 1000-user dataset can drop from 15,400 to 6,200 tokens. ### Dev Weekly: Microsoft's $9.7B AI Deal, Samsung-Nvidia AI Factory, Deepnote Goes Open Source (Nov 3–9, 2025) - URL: https://singhajit.com/dev-weekly-microsoft-ai-samsung-nvidia-deepnote-nov-3-9/ - Date: 2025-11-09 - Tags: dev-weekly, tech-news, software-development-news - Description: Microsoft signs $9.7B AI deal with Australia's IREN. Samsung and Nvidia building AI megafactory with 50,000 GPUs. Deepnote goes open source. Canva makes Affinity suite free. AI security concerns rising. Developer news November 3-9, 2025 ### Stop Blocking Your Paying Customers: Build a Smart Rate Limiter - URL: https://singhajit.com/dynamic-rate-limiter-system-design/ - Date: 2025-11-05 - Tags: system-design - Description: Learn how to design and implement a dynamic rate limiter that adapts to system load, user behavior, and traffic patterns. Real-world strategies from Stripe, Twitter, and Netflix. - Quick Answer: Rate limiters control request volume using **token bucket** (allows bursts, refills steadily) or **leaky bucket** (constant output rate). For distributed systems, use Redis with INCR+TTL or Lua scripts for atomicity. Return HTTP 429 with `X-RateLimit-*` headers and `Retry-After` to help clients back off properly. ### Dev Weekly: Nvidia Hits $5T, OpenAI's $1T IPO Plans, Aardvark Security (Oct 27–Nov 2, 2025) - URL: https://singhajit.com/dev-weekly-nvidia-5t-openai-ipo-aardvark-anthropic-oct-27-nov-2/ - Date: 2025-11-02 - Tags: dev-weekly, tech-news, software-development-news - Description: Nvidia hits $5 trillion market cap on Oct 29. OpenAI preparing for $1 trillion IPO and launches Aardvark security agent. Amazon cuts up to 30,000 jobs amid AI push. Anthropic opens Seoul office, Claude Haiku 4.5 on GitHub Copilot. Vercel AI SDK 6. Major security breaches. Developer news October-November 2025 ### Visitor Design Pattern - URL: https://singhajit.com/design-patterns/visitor/ - Date: 2025-11-01 - Tags: design-patterns, java - Description: Learn Visitor pattern in Java. Add operations to object structures without modifying classes. Uses double dispatch. ### Dev Weekly: AWS Outage, Anthropic's $10B+ Deal, ChatGPT Atlas Browser (Oct 20–26, 2025) - URL: https://singhajit.com/dev-weekly-aws-outage-anthropic-google-chatgpt-atlas-quantum/ - Date: 2025-10-26 - Tags: dev-weekly, tech-news, software-development-news - Description: Major AWS US-EAST-1 outage takes down Fortnite, Reddit, Roblox on Oct 20. Anthropic partners with Google Cloud for 1M TPUs in $10B+ deal, OpenAI launches ChatGPT Atlas browser, Google quantum breakthrough, Red Hat Developer Lightspeed, GitLab 18.5, Snyk Evo, Couchbase 8.0, developer news October 2025 ### How Shopify Powers 5 Million Stores Without Breaking a Sweat - URL: https://singhajit.com/shopify-system-design/ - Date: 2025-10-24 - Tags: system-design - Description: Deep dive into Shopify's system design and architecture. How they handle millions of merchants, billions in sales, and massive traffic spikes. Learn from their modular monolith, pod architecture, and scaling strategies. - Quick Answer: Shopify uses a **pod architecture** where stores are grouped into isolated pods, each with its own database cluster. They chose a **modular monolith** over microservices for simpler operations. MySQL sharded by shop_id. Black Friday handled via extensive caching, pre-scaling, load shedding, and queue-based order processing. ### WebTransport: The Protocol That Fixes What's Broken in WebSockets - URL: https://singhajit.com/how-webtransport-works/ - Date: 2025-10-22 - Tags: web-development, networking - Description: Understand WebTransport, the modern protocol built on HTTP/3 and QUIC that eliminates head-of-line blocking in WebSockets. Learn how multiplexed streams and flexible reliability models enable faster, more efficient real-time web communication for gaming, streaming, and IoT applications. - Quick Answer: WebTransport is built on HTTP/3 and QUIC, eliminating TCP's head-of-line blocking. Unlike WebSockets where one lost packet blocks everything, WebTransport uses **independent streams** - a lost packet in stream A doesn't block streams B and C. It also supports unreliable datagrams for real-time data like game positions. ### How DNS Works: The Complete Guide for Developers - URL: https://singhajit.com/how-dns-works-complete-guide/ - Date: 2025-10-21 - Tags: networking, system-design, tutorial - Description: Complete guide to how DNS works for developers. Learn DNS resolution step by step, caching layers, DNS record types (A, CNAME, MX, TXT), TTL, DNS performance optimization, and debugging with dig and nslookup. Includes diagrams and real-world examples. - Quick Answer: DNS translates domain names to IPs through a hierarchy: browser cache, OS cache, recursive resolver, root servers, TLD servers (.com), and authoritative nameserver. Records are cached by TTL. A records point to IPs directly; CNAMEs alias to other domains. Lower TTL before DNS changes for faster propagation. ### AWS US-East-1 Outage: How a Network Load Balancer Bug Took Down Half the Internet - URL: https://singhajit.com/aws-us-east-outage-october-2025/ - Date: 2025-10-21 - Tags: tech-news - Description: Complete analysis of the October 20, 2025 AWS US-East-1 outage that disrupted Netflix, Snapchat, Reddit, Robinhood, Fortnite, and major services worldwide. Timeline, root cause analysis, impact assessment, and critical lessons for developers on building resilient cloud architectures. - Quick Answer: A network load balancer bug incorrectly marked healthy servers as dead, causing DNS resolution failures for DynamoDB that cascaded to Lambda, API Gateway, and CloudWatch. The 12-hour outage affected Netflix, Snapchat, Robinhood, and millions of users. Lesson: deploy multi-region with external monitoring. ### Dev Weekly: OpenAI's Massive AMD Deal, App Store Launch & Developer Talent Crisis (Oct 13–19, 2025) - URL: https://singhajit.com/dev-weekly-openai-amd-deal-app-store-developer-talent-crisis/ - Date: 2025-10-19 - Tags: dev-weekly, tech-news, software-development-news - Description: OpenAI partners with AMD for 6 gigawatts of GPU compute challenging NVIDIA dominance, unveils ChatGPT Apps SDK at DevDay 2025 reaching $500B valuation, SoftBank buys ABB Robotics for $5.4B, UK faces developer shortage with aging workforce, Microsoft Agent Framework, Grafana Assistant, IBM-Anthropic partnership, Gartner predicts 90% AI adoption by 2028 ### Interpreter Design Pattern - URL: https://singhajit.com/design-patterns/interpreter/ - Date: 2025-10-18 - Tags: design-patterns, java - Description: Learn Interpreter pattern in Java. Evaluate domain-specific languages with abstract syntax trees. Build expression parsers. ### Postgres 18: The Release That Makes Databases Fast Again - URL: https://singhajit.com/postgres-18-features/ - Date: 2025-10-17 - Tags: postgres, database - Description: Postgre 18 brings async I/O with 3x performance gains, skip scan for multicolumn indexes, virtual generated columns, OAuth 2.0 authentication, temporal constraints, and UUIDv7 support. Learn what matters for your production database. - Quick Answer: PostgreSQL 18 brings **async I/O** (3x faster sequential scans), **skip scan** (use multicolumn indexes without leading column), virtual generated columns, OAuth 2.0 auth, temporal constraints, UUIDv7 support, and faster pg_upgrade. Major performance gains for I/O-bound workloads. ### Java 25 is Finally Here: The LTS Release That Changes Everything - URL: https://singhajit.com/java-25-lts-features/ - Date: 2025-10-15 - Tags: java - Description: Java 25 LTS is here with game-changing features: simplified main methods, flexible constructors, Scoped Values that replace ThreadLocal, compact object headers for memory savings, and built-in password hashing. Learn what matters for your daily work. - Quick Answer: Java 25 LTS brings: simplified `void main()` without class boilerplate, flexible constructors with code before `super()`, **Scoped Values** replacing ThreadLocal for virtual threads, compact object headers saving 8-12% memory, and built-in PBKDF2 password hashing. First LTS since Java 21. ### How Ticket Booking Systems Handle 50,000 People Fighting for One Seat - URL: https://singhajit.com/ticket-booking-system-design/ - Date: 2025-10-13 - Tags: system-design - Description: How ticket booking systems prevent double bookings, handle 50K concurrent users, and process payments. Inside BookMyShow and Ticketmaster architecture. - Quick Answer: Ticket systems prevent double-booking using **distributed locks** (Redis) with TTL. When you select a seat, a lock is acquired for ~5 minutes. Payment completes the booking; timeout releases the seat. Virtual queues control traffic, and database constraints provide final safety. The Taylor Swift crash happened when 14M users exceeded all capacity limits. ### Dev Weekly: Windows 10 Dies, Sora 2 Sparks Deepfake Fears, Nobel Goes to Quantum (Oct 6–12, 2025) - URL: https://singhajit.com/dev-weekly-windows-10-dies-sora-2-deepfakes-nobel-quantum/ - Date: 2025-10-12 - Tags: dev-weekly, tech-news, software-development-news - Description: Windows 10 reaches end of support October 14, OpenAI releases controversial Sora 2 video generator, OpenAI partners with AMD for 6 gigawatts of GPUs challenging NVIDIA, Nobel Prize in Physics honors quantum computing pioneers, Ubuntu replaces sudo with Rust, Microsoft patches critical zero-day, and 400 million PCs face retirement. ### Hash Collisions: The Hidden Performance Killer in Your Code - URL: https://singhajit.com/data-structures/hashtable-collisions/ - Date: 2025-10-06 - Tags: data-structures - Description: Deep dive into hash table collisions - understand why they happen, how they slow down your application, and practical techniques to handle them. Learn from real-world attacks, performance benchmarks, and production implementations. - Quick Answer: Hash collisions occur when different keys map to the same bucket. Handle them with **separate chaining** (linked lists per bucket) or **open addressing** (probe for next slot). Keep load factor below 0.75 and resize when exceeded. Modern languages use randomized hash seeds to prevent collision attacks. ### Dev Weekly: California Regulates AI Hiring, Claude Gets Developer Tools & EA's $55B Exit (Sep 29–Oct 5, 2025) - URL: https://singhajit.com/dev-weekly-california-ai-law-claude-developer-tools-ea-acquisition/ - Date: 2025-10-05 - Tags: dev-weekly, tech-news, software-development-news - Description: California enforces first major AI employment regulations, Anthropic releases Claude Sonnet 4.5 with developer tools, GitHub Copilot CLI enters public preview, Electronic Arts sells for $55 billion, AOL ends dial-up after 30 years, JUnit 6 released, plus major funding rounds and Google's Gemini for Home announcement. ### Memento Design Pattern - URL: https://singhajit.com/design-patterns/memento/ - Date: 2025-10-04 - Tags: design-patterns, java - Description: Learn Memento pattern in Java. Capture and restore object state for undo/redo without violating encapsulation. ### Why JWT Replaced Sessions: Building Auth That Scales - URL: https://singhajit.com/how-jwt-works/ - Date: 2025-10-02 - Tags: security - Description: Deep dive into JSON Web Tokens (JWT) - understand the structure, signature verification, security best practices, and common vulnerabilities. Learn how JWT enables stateless authentication and when to use (or avoid) it. - Quick Answer: JWT is a stateless token with three parts: **header** (algorithm), **payload** (claims like user ID), and **signature**. The server signs tokens at login; on each request, it verifies the signature without database lookup. Store in httpOnly cookies to prevent XSS. Use short expiration + refresh tokens for security. ### How Kafka Works: The Engine Behind Real-Time Data Pipelines - URL: https://singhajit.com/distributed-systems/how-kafka-works/ - Date: 2025-10-01 - Tags: distributed-systems - Description: Deep dive into Apache Kafka architecture - understand topics, partitions, consumer groups, and how Kafka achieves high throughput and fault tolerance. Learn from real-world examples and practical insights for building scalable data pipelines. - Quick Answer: Kafka is a distributed commit log, not a message queue. **Topics** are split into **partitions** distributed across brokers. Producers append messages; consumers read at their own pace using offsets. Unlike queues, Kafka retains messages for replay. Consumer groups enable parallel processing with automatic rebalancing on failures. ### Kubernetes Architecture: The Operating System for the Cloud - URL: https://singhajit.com/devops/kubernetes-architecture/ - Date: 2025-09-30 - Tags: system-design, devops - Description: Deep dive into Kubernetes architecture - understand how the control plane, worker nodes, and core components work together to orchestrate containers at scale. Learn from real-world examples and practical insights. - Quick Answer: Kubernetes has two parts: the **Control Plane** (API server, etcd, scheduler, controller manager) that makes decisions, and **Worker Nodes** (kubelet, kube-proxy, container runtime) that run workloads. etcd stores all cluster state, the scheduler assigns pods to nodes, and controllers ensure desired state matches reality. ### Dev Weekly: Nvidia's $100B AI Bet, Postgres 18 & Anthropic Copyright Battle (Sep 22–28, 2025) - URL: https://singhajit.com/dev-weekly-nvidia-100b-h1b-visa-anthropic-copyright-postgresql-18/ - Date: 2025-09-28 - Tags: dev-weekly, tech-news, software-development-news - Description: Nvidia makes the biggest AI investment in history, H-1B visa fees spike to $100K causing tech industry panic, federal judge rejects Anthropic's $1.5B copyright settlement, PostgreSQL 18 delivers 2-3x performance boost, Python holds top programming language spot, plus Meta's AR glasses and security updates. ### How Meta Achieves 99.99999999% Cache Consistency - URL: https://singhajit.com/meta-cache-consistency/ - Date: 2025-09-22 - Tags: system-design - Description: Deep dive into Meta's cache consistency architecture - how they handle billions of users with near-perfect cache consistency using TAO, memcache, and distributed invalidation strategies. Learn from their scaling challenges and architectural decisions. - Quick Answer: Meta achieves 99.99999999% cache consistency through **TAO** (graph cache for social data), global cache invalidation within milliseconds, version numbers to prevent stale writes, and lease mechanisms. When data changes, cached copies are invalidated everywhere before the write is acknowledged. ### Dev Weekly: Meta Connect Fails, Java 25 LTS, NPM Attack (Sep 15–21, 2025) - URL: https://singhajit.com/dev-weekly-meta-connect-fails-java-25-lts-npm-attack/ - Date: 2025-09-21 - Tags: dev-weekly, tech-news, software-development-news - Description: This week's top developer news: Meta's Connect 2025 live demo failures, Java 25 LTS goes live, new details on the NPM supply chain attack, plus Copilot and AWS updates, Kubernetes 1.34 rollouts, React 19 momentum, and more. ### Flyweight Design Pattern - URL: https://singhajit.com/design-patterns/flyweight/ - Date: 2025-09-20 - Tags: design-patterns, java - Description: Learn Flyweight pattern in Java. Share state to support large numbers of objects efficiently. Text editor and game examples. ### How Slack Built a System That Handles 10+ Billion Messages - URL: https://singhajit.com/slack-system-design/ - Date: 2025-09-19 - Tags: system-design - Description: Deep dive into Slack's system design and architecture - how they handle millions of users, billions of messages, and maintain real-time communication at scale. Learn from their scaling challenges, database design, and microservices architecture. - Quick Answer: Slack uses **workspace-based sharding** where each workspace gets its own database shard and RTM server. Messages flow through WebApp servers (PHP) for storage, then RTM servers (Java) broadcast via WebSocket to connected users. MySQL + Redis for storage, Elasticsearch for search, and tiered storage moves old messages to S3. ### Paxos: The Democracy of Distributed Systems - URL: https://singhajit.com/distributed-systems/paxos/ - Date: 2025-09-18 - Tags: distributed-systems - Description: Learn how Paxos algorithm achieves consensus in distributed systems. Complete guide with real-world examples, diagrams, and practical implementations covering Google Chubby, Cassandra, and etcd. - Quick Answer: Paxos achieves consensus through a two-phase voting process: **Prepare** (proposer asks acceptors for permission) and **Accept** (if majority agrees, proposer sends the value). Once a majority accepts, the value is chosen. Used by Google Chubby, Cassandra, and Spanner. Raft is a simpler alternative commonly used in newer systems. ### Dev Weekly: NPM Attack, AI Outage & VMware Exodus - URL: https://singhajit.com/dev-weekly-npm-attack-anthropic-outage-vmware-exodus/ - Date: 2025-09-13 - Tags: dev-weekly, tech-news, software-development-news - Description: Massive NPM supply chain attack affects 2+ billion weekly downloads, Anthropic outage cripples AI-dependent developers, VMware customers plan mass migration, Microsoft fixes Exchange Online outage, and more developer news from September 8-14, 2025 ### Write-Ahead Log: The Golden Rule of Durable Systems - URL: https://singhajit.com/distributed-systems/write-ahead-log/ - Date: 2025-09-10 - Tags: distributed-systems - Description: Learn how Write-Ahead Log (WAL) prevents data loss in distributed systems. Complete guide with real-world examples, code samples, and diagrams covering PostgreSQL, Kafka, and custom implementations. - Quick Answer: Write-Ahead Log (WAL) writes all changes to a log file **before** applying them to data. If the system crashes, the log is replayed to recover. PostgreSQL, MySQL, SQLite, MongoDB, and Kafka all use WAL. The key rule: no data modification happens until it's safely in the log on disk. ### Dev Weekly: Stripe Launches Tempo Blockchain, Google Fined $3.5B, Microsoft Open Sources BASIC - URL: https://singhajit.com/dev-weekly-stripe-tempo-blockchain-google-fine-microsoft-basic/ - Date: 2025-09-07 - Tags: dev-weekly, tech-news, software-development-news - Description: Stripe unveils Tempo payments blockchain with 100K+ TPS, EU fines Google $3.5B for ad tech abuse, Microsoft open sources 1978 BASIC code, Hollow Knight: Silksong crashes platforms, and essential developer news from September 1-7, 2025 ### Prototype Design Pattern - URL: https://singhajit.com/design-patterns/prototype/ - Date: 2025-09-06 - Tags: design-patterns, java - Description: Learn Prototype pattern in Java. Create objects by cloning existing ones. Implement deep copy and configuration templates. ### Distributed Counter System Design - URL: https://singhajit.com/distributed-counter-architecture-guide/ - Date: 2025-09-03 - Tags: system-design - Description: How to design a distributed counter for high-traffic systems. Complete system design guide covering sharded counters, sharded counter architecture, local aggregation, CRDTs, and production patterns with code examples. - Quick Answer: Design distributed counters using **sharded counters** (split count across multiple shards to avoid hotspots), **local aggregation** (batch updates locally before syncing), or **CRDTs** (conflict-free replicated data types). For high-traffic like social media likes, sharded counters with periodic aggregation offer the best write throughput. ### Dev Weekly: Nvidia's Mystery Customers, Meta Ditches Scale AI and Nx Supply chain attack - URL: https://singhajit.com/dev-weekly-nvidia-mystery-customers-meta-scale-ai-nx-security-incident/ - Date: 2025-08-31 - Tags: dev-weekly, tech-news, software-development-news - Description: Critical Nx build tool supply chain attack hits npm packages, Nvidia's revenue concentration concerns, WhatsApp zero-day patch, and essential developer security updates from August 25-31, 2025 ### How Stripe Prevents Double Payments With Idempotency Keys - URL: https://singhajit.com/how-stripe-prevents-double-payment/ - Date: 2025-08-29 - Tags: system-design - Description: Learn how Stripe prevents double payments using idempotency keys. Complete guide to stripe idempotency with code examples using tok_visa, database constraints, and retry logic. Prevent duplicate charges in your payment systems. - Quick Answer: Stripe prevents duplicate charges using **idempotency keys** - unique identifiers sent with each API request. If Stripe receives a request with a key it has seen before (within 24 hours), it returns the cached response instead of processing again. Generate a UUID for each payment attempt and retry with the same key on failures. ### GitHub Actions: CI/CD Automation Basics - URL: https://singhajit.com/github-actions-basics-cicd-automation/ - Date: 2025-08-27 - Tags: github-actions, ci-cd, devops - Description: Learn GitHub Actions fundamentals with practical examples. Master CI/CD automation, workflow triggers, and best practices for software developers. - Quick Answer: GitHub Actions automates CI/CD through YAML workflow files in `.github/workflows/`. Workflows trigger on events (push, PR, schedule), run jobs on virtual machines, and execute steps. Use `actions/checkout` to get code, matrix builds for multi-version testing, and secrets for sensitive data. Free tier includes 2,000 minutes/month. ### Dev Weekly: Coinbase Fires Engineers, GitHub Gets Agents Panel, and Intel Gets a Government Lifeline - URL: https://singhajit.com/dev-weekly-coinbase-fires-ai-github-agents-intel-bailout/ - Date: 2025-08-24 - Tags: dev-weekly, tech-news, software-development-news - Description: Coinbase CEO fires engineers over AI adoption, GitHub launches agents panel, US government takes 10% stake in Intel, and more developer news from August 18-24, 2025 ### 55 Million Requests Per Second: Inside Cloudflare's Magic - URL: https://singhajit.com/how-cloudflare-supports-55-million-requests-per-second/ - Date: 2025-08-20 - Tags: system-design - Description: Deep dive into Cloudflare's technical architecture - how 15 PostgreSQL clusters, ClickHouse, and Quicksilver work together to handle 55 million requests per second with millisecond latency. - Quick Answer: Cloudflare handles 55M RPS with only 15 PostgreSQL clusters through **PgBouncer** connection pooling, **bare metal servers** (no virtualization overhead), **Anycast routing** across 330+ data centers, and **HAProxy** load balancing. Users are automatically routed to the nearest datacenter, and connection pooling prevents database exhaustion. ### Dev Weekly: GPT-5 Arrives, GitHub CEO Departs, and AI Safety Gets Real - URL: https://singhajit.com/2025/08/17/dev-weekly-gpt5-github-ceo-ai-safety/ - Date: 2025-08-17 - Tags: dev-weekly, tech-news, software-development-news - Description: GPT-5 lands in GitHub Copilot, Thomas Dohmke steps down as GitHub CEO, Claude AI gets safety boundaries, and more developer news from August 11-17, 2025 ### Bridge Design Pattern - URL: https://singhajit.com/design-patterns/bridge/ - Date: 2025-08-16 - Tags: design-patterns, java - Description: Learn Bridge pattern in Java. Separate abstraction from implementation to avoid class explosion. Cross-platform examples. ### How Uber Finds Nearby Drivers at 1 Million Requests per Second - URL: https://singhajit.com/how-uber-finds-nearby-drivers-1-million-requests-per-second/ - Date: 2025-08-16 - Tags: system-design - Description: How does Uber find nearby drivers? Learn how Uber's system for finding you nearby drivers works at 1M+ requests per second using H3 hexagonal grids, geospatial indexing, and real-time matching. - Quick Answer: Uber uses **H3 hexagonal grid indexing** to find nearby drivers. Your GPS location is converted to a hexagon cell ID, then the system searches that cell and its neighbors for available drivers. This avoids checking every driver in the city, reducing search time from 10-15 seconds to under 3 seconds at 1M+ requests per second. ### Dev Weekly: GPT-5 Revolutionizes Development, Critical RubyGems Security Threat, and Python's Performance Leap - URL: https://singhajit.com/dev-weekly-gpt5-rubygems-security-python/ - Date: 2025-08-10 - Tags: dev-weekly, tech-news, software-development-news - Description: Essential weekly roundup covering OpenAI's GPT-5 breakthrough, RubyGems malware campaign security alert, Python 3.13.6 performance improvements, AWS Lambda billing changes, and JetBrains AI tool enhancements. ### The Complete Guide to k6 Load Testing - URL: https://singhajit.com/performance-testing-with-grafana-k6/ - Date: 2025-08-10 - Tags: performance-testing, testing - Description: Master k6 executors: ramping-arrival-rate, constant-arrival-rate, and ramping-vus explained with examples. Complete guide to k6 load testing scenarios, thresholds, and CI/CD. - Quick Answer: k6 offers three main executors: **ramping-vus** (gradually increase users), **constant-arrival-rate** (fixed RPS), and **ramping-arrival-rate** (gradually increase RPS). Use VU-based for user simulation, arrival-rate for guaranteed throughput testing. k6 automatically scales VUs to maintain target RPS when latency increases. ### How WhatsApp Scaled to Billions of Users with Just 50 Engineers - URL: https://singhajit.com/whatsapp-scaling-secrets/ - Date: 2025-08-07 - Tags: system-design - Description: Learn how WhatsApp handles 100 billion messages daily with a tiny team. Deep dive into Erlang, the actor model, Mnesia database, and the system design that powers 2 billion users. - Quick Answer: WhatsApp scaled to 2 billion users with 50 engineers by using **Erlang** (lightweight processes handling 2M+ connections per server), **Mnesia** (in-memory distributed database), and **FreeBSD** (superior networking). Each user gets a dedicated Erlang process, enabling direct process-to-process message delivery with minimal overhead. ### Prompt Engineering Basics for Software Developers - URL: https://singhajit.com/prompt-engineering-basics/ - Date: 2025-08-05 - Tags: prompt-engineering, artificial-intelligence, machine-learning - Description: Learn the fundamentals of prompt engineering and how it can enhance your interaction with AI models. Discover best practices, examples, and tips tailored for software developers. - Quick Answer: Effective prompt engineering requires being specific, providing examples, setting constraints, and iterating. For code generation, include the language, input/output specs, libraries to use, and example data. Avoid vague prompts and break complex tasks into focused sub-prompts. ### Git Command Line Basics: Essential Commands for Software Developers - URL: https://singhajit.com/git-command-line-basics/ - Date: 2025-08-04 - Tags: git, version-control - Description: Learn the basics of Git command-line usage with detailed explanations of essential commands. Discover why the command line is a powerful tool for developers. - Quick Answer: Essential Git commands: `git init` (create repo), `git clone` (copy remote), `git add` (stage), `git commit` (save), `git push/pull` (sync with remote), `git branch` (manage branches), `git merge` (combine). The command line offers speed, full feature access, and scripting capabilities over GUI tools. ### Taming Pipeline Chaos: How I Used GitLab APIs and GPT to Analyze Thousands of Failures and Boost Stability - URL: https://singhajit.com/analyzing-pipeline-failures-with-gitlab-and-gpt/ - Date: 2025-08-03 - Tags: ci-cd, openai, ai-application - Description: Learn how I leveraged GitLab APIs and GPT to analyze thousands of pipeline failures, identify patterns, and implement solutions that significantly improved stability. - Quick Answer: Use GitLab's REST APIs to fetch failed job logs programmatically, then feed them to GPT with a structured prompt to categorize failures by type (network, dependency, config). This automated analysis can reveal patterns like '40% of failures stem from external dependencies' and help prioritize fixes. ### Dev Weekly: AWS Goes AI-Native, Mistral Unleashes Voxtral & Layoffs Hit Hard - URL: https://singhajit.com/dev-weekly-ai-dlc-ignites-voxtral-speaks-mass-layoffs/ - Date: 2025-08-02 - Tags: dev-weekly, tech-news, software-development-news - Description: Weekly software development news covering AWS AI-DLC methodology, Mistral Voxtral open-source audio AI, Amazon SQS Fair Queues, Kubernetes v1.34, tech layoffs, and Windows 11 updates for developers. ### Iterator Design Pattern - URL: https://singhajit.com/design-patterns/iterator/ - Date: 2025-08-02 - Tags: design-patterns, java - Description: Learn Iterator pattern in Java. Traverse collections without exposing internals. Implement custom iterators with examples. ### How to Exclude a Single Module from `sbt test` in a Multi-Module Scala Project - URL: https://singhajit.com/exclude-single-module-sbt-test-scala/ - Date: 2025-08-01 - Tags: scala, sbt, testing, ci-cd - Description: Learn different approaches to exclude specific modules from sbt test execution in multi-module Scala projects, with code examples and CI-friendly solutions. - Quick Answer: Use `sbt 'set moduleName / Test / skip := true' test` to exclude a specific module from sbt test. This command-line approach requires no code changes and works perfectly in CI/CD pipelines. ### Dev Weekly: GitHub Goes Dark, Python Breaks Free, and Azure Hits 1M Pods - URL: https://singhajit.com/dev-weekly-github-goes-dark-python-breaks-free-azure-hits-1m-pods/ - Date: 2025-07-29 - Tags: dev-weekly, tech-news, software-development-news - Description: Weekly roundup: GitHub outage, Python 3.14 RC1, Azure pod scaling, security news, and more for software developers. ### Mediator Design Pattern - URL: https://singhajit.com/design-patterns/mediator/ - Date: 2025-07-19 - Tags: design-patterns, java - Description: Learn Mediator pattern in Java. Centralize complex communication between objects. Reduce coupling in chat rooms and UIs. ### Chain of Responsibility Design Pattern - URL: https://singhajit.com/design-patterns/chain-of-responsibility/ - Date: 2025-07-05 - Tags: design-patterns, java - Description: Learn Chain of Responsibility in Java. Pass requests through a chain of handlers. Used in middleware and validation. ### Composite Design Pattern - URL: https://singhajit.com/design-patterns/composite/ - Date: 2025-06-21 - Tags: design-patterns, java - Description: Learn Composite pattern in Java. Treat individual objects and compositions uniformly. Build tree structures like file systems. ### Abstract Factory Design Pattern - URL: https://singhajit.com/design-patterns/abstract-factory/ - Date: 2025-06-07 - Tags: design-patterns, java - Description: Learn Abstract Factory pattern in Java. Create families of related objects without specifying concrete classes. ### Template Method Design Pattern - URL: https://singhajit.com/design-patterns/template-method/ - Date: 2025-05-17 - Tags: design-patterns, java - Description: Learn Template Method pattern in Java. Define algorithm skeleton in base class, let subclasses override specific steps. ### State Design Pattern - URL: https://singhajit.com/design-patterns/state/ - Date: 2025-05-03 - Tags: design-patterns, java - Description: Learn State pattern in Java. Let objects change behavior based on internal state. Replace complex conditionals with state machines. ### Command Design Pattern - URL: https://singhajit.com/design-patterns/command/ - Date: 2025-04-19 - Tags: design-patterns, java - Description: Learn Command pattern in Java. Encapsulate requests as objects for undo/redo, queuing, and logging. Practical code examples. ### Proxy Design Pattern - URL: https://singhajit.com/design-patterns/proxy/ - Date: 2025-04-05 - Tags: design-patterns, java - Description: Learn Proxy pattern in Java. Control access to objects with virtual, protection, and remote proxies. Includes lazy loading examples. ### Facade Design Pattern - URL: https://singhajit.com/design-patterns/facade/ - Date: 2025-03-22 - Tags: design-patterns, java - Description: Learn Facade pattern in Java. Provide a simple interface to complex subsystems. Reduce coupling and create clean APIs. ### Adapter Design Pattern - URL: https://singhajit.com/design-patterns/adapter/ - Date: 2025-03-08 - Tags: design-patterns, java - Description: Learn Adapter pattern in Java. Make incompatible interfaces work together. Integrate legacy systems and third-party libraries. ### Builder Design Pattern - URL: https://singhajit.com/design-patterns/builder/ - Date: 2025-02-22 - Tags: design-patterns, java - Description: Learn Builder pattern in Java. Construct complex objects step by step with fluent interfaces. Handle optional parameters cleanly. ### Strategy Design Pattern - URL: https://singhajit.com/design-patterns/strategy/ - Date: 2025-02-08 - Tags: design-patterns, java - Description: Learn Strategy pattern in Java. Swap algorithms at runtime without changing client code. Includes payment and sorting examples. ### Factory Method Design Pattern - URL: https://singhajit.com/design-patterns/factory-method/ - Date: 2025-01-25 - Tags: design-patterns, java - Description: Learn Factory Method pattern in Java. Let subclasses decide which objects to create. Includes real-world examples and code. ### Singleton Design Pattern - URL: https://singhajit.com/design-patterns/singleton/ - Date: 2025-01-10 - Tags: design-patterns, java - Description: Learn Singleton pattern in Java. Ensure one instance with global access. Covers thread-safe implementation and common pitfalls. ### Github Actions for Android - URL: https://singhajit.com/android-ci-cd-using-github-actions/ - Date: 2021-02-07 - Tags: android, ci-cd, Github Actions - Description: Designing and setting up CI/CD for Android using Github Actions is simple. In this post we will implement entire CI/CD pipeline using Github Actions. ### Pairing matrix for agile teams - URL: https://singhajit.com/pairing-matrix-for-agile-teams/ - Date: 2021-01-05 - Tags: ruby, agile, rubygem - Description: This blog shows how you can automate the creation of pairing matrix for your agile team. The pairing matrix is created using pairing_matrix rubygem. ### How to integrate GraphQL with Sitecore using JSS - URL: https://singhajit.com/how-to-integrate-graphql-with-sitecore-using-jss/ - Date: 2020-10-29 - Tags: sitecore, graphql, jss - Description: Step by step instructions to integrate GraphQL with Sitecore using JSS ### How Flutter Works Under the Hood - URL: https://singhajit.com/flutter-under-the-hood/ - Date: 2020-09-20 - Tags: flutter, mobile-cross-platform, system-design - Description: How does Flutter work under the hood? A deep dive into Flutter architecture, the three-tree rendering system (Widget, Element, RenderObject), Impeller and Skia engines, Dart AOT and JIT compilation, hot reload, platform channels, and how Flutter renders UI at 60-120fps. Written for software developers. - Quick Answer: Flutter works through three layers: the Framework (Dart), the Engine (C++), and the Embedder (platform-specific). Your Dart code describes UI using widgets, but Flutter maintains three internal trees to make rendering fast. The Widget Tree holds your immutable UI descriptions. The Element Tree tracks state and decides what actually needs rebuilding. The RenderObject Tree handles layout and painting. The rendering engine (Impeller on iOS, Impeller or Skia on Android) draws every pixel directly to a canvas. Flutter does not use native UI components. In development, Dart runs on a JIT compiler for hot reload. In production, it compiles AOT to native ARM code. ### Monitoring individual queue in sidekiq - URL: https://singhajit.com/monitoring-individual-queue-in-sidekiq/ - Date: 2018-10-07 - Tags: rails, ruby, sidekiq, rubygem - Description: By default sidekiq does not allow you to monitor failures on individual queues. This article will help you monitor individual queue in sidekiq like a pro. ### Why aren't you using binstubs yet? - URL: https://singhajit.com/why-arent-you-using-binstubs-yet/ - Date: 2018-05-20 - Tags: rails, ruby - Description: Stop typing bundle exec before every command. Learn how Rails binstubs work and how to create custom binstubs for any gem to run executables in your bundle context. ### Offline Mode Of Android Apps - URL: https://singhajit.com/offline-mode-of-android-apps/ - Date: 2017-10-28 - Tags: android, presentation, conference - Description: This talk covers offline mode of android apps. In this talk I talked about, what it takes to build offline mode and how to design its architecture. ### My upcoming talk in DroidConUK - URL: https://singhajit.com/my-upcoming-talk-in-droidconuk/ - Date: 2017-09-11 - Tags: android, conference - Description: Sharing learnings on building offline mode in Android apps at DroidConUK, including design, challenges, and best practices. ### Integrating Sherlock with android apps - URL: https://singhajit.com/integrating-sherlock-with-android-apps-to-get-crash-reports/ - Date: 2017-04-26 - Tags: android - Description: Learn how to integrate Sherlock library for crash reporting in Android apps. Get instant crash notifications with full stack traces and device info that you can share via email or messaging apps. ### Why your android application needs awareness api - URL: https://singhajit.com/android-awareness-api/ - Date: 2017-02-02 - Tags: android - Description: Learn how to use Android Awareness API to make your app context-aware. Use Fence API for geofencing and Snapshot API to get user's current context including location, activity, weather, and nearby beacons. ### Java Custom Annotations: How They Work and How to Write Your Own - URL: https://singhajit.com/java-custom-annotations/ - Date: 2017-01-22 - Tags: java - Description: Learn Java custom annotations from scratch. Create an @interface, set @Retention and @Target, process annotations with reflection, and see how JUnit, Spring, and Jackson use the same idea. - Quick Answer: A **Java annotation** is metadata you attach to code with `@Name`. You declare a custom one with `@interface`, then tell the compiler where it may sit (`@Target`) and how long it lives (`@Retention`). **SOURCE** is dropped at compile time, **CLASS** is stored in the `.class` file but not visible at runtime, and **RUNTIME** can be read with reflection. Frameworks such as JUnit (`@Test`), Spring (`@Autowired`), and Jackson (`@JsonProperty`) are just custom annotations plus a processor. To write your own, define the annotation, mark the code, then either scan it at runtime with `isAnnotationPresent` or generate code at compile time with an annotation processor. ### Prevent push on red build with the help of gocd_pre_push - URL: https://singhajit.com/prevent-push-on-red-build-with-the-help-of-gocd-pre-push/ - Date: 2016-12-18 - Tags: ruby, rubygem, continuous integration, git, gocd - Description: gocd_pre_push helps agile teams to prevent pushing the changes in the central repo when the build is red by checking the status of concerned pipelines. ### Conway's Game Of Life - URL: https://singhajit.com/conways-game-of-life/ - Date: 2016-11-23 - Tags: javascript, game - Description: Learn about Conway's Game of Life, its rules, and how to implement it in JavaScript with code and demo. ### Notify when android device network status changes - URL: https://singhajit.com/notify-android-device-network-status-changes/ - Date: 2016-10-27 - Tags: android - Description: This article will show how you can listen network status and notify when android device network status changes using a snackbar notification. ### Observer Design Pattern - URL: https://singhajit.com/design-patterns/observer/ - Date: 2016-10-12 - Tags: java, design-patterns - Description: How observer design pattern works and what are the use cases. This article explains observer design pattern with the help of an example. ### Filter Design Pattern - URL: https://singhajit.com/design-patterns/filter/ - Date: 2016-10-11 - Tags: java, design-patterns - Description: How filter design pattern works and what are the use cases. This article explains filter design pattern with the help of an example. ### Ruby gem to fetch information from gocd as rich models - URL: https://singhajit.com/ruby-gem-to-fetch-information-gocd-rich-models/ - Date: 2016-10-02 - Tags: ruby, gocd, rubygem - Description: GoCD is a ruby gem to fetch information from gocd server as models. It will make the api calls on your behalf and get you the information that you want. ### Android Data Binding - URL: https://singhajit.com/android-data-binding/ - Date: 2016-06-06 - Tags: android - Description: Keep your android activities lean with Android Data Binding. Use ViewModels to render the data on UI and notify the UI when something changes in ViewModel ### Android Custom Animations - URL: https://singhajit.com/android-custom-animations/ - Date: 2016-04-16 - Tags: android - Description: Android Custom Animations will cover how to create custom animations using pure xml tags e.g "alpha", "scale", "translate" and "rotate". ### shell_session_update: command not found - URL: https://singhajit.com/shell_session_update-command-not-found/ - Date: 2016-04-15 - Tags: rvm, ruby, shell - Description: rvm issue: shell_session_update: command not found and its solutions. ### Android Draggable View: Complete Implementation Guide - URL: https://singhajit.com/android-draggable-view/ - Date: 2016-04-02 - Tags: android - Description: Complete guide to implementing draggable views in Android. Learn touch event handling, drag and drop framework, FrameLayout positioning, OnTouchListener implementation, and best practices for creating floating buttons and draggable UI elements. - Quick Answer: To make a view draggable in Android, use FrameLayout as the parent container and implement View.OnTouchListener. In ACTION_DOWN, capture the offset between view position and touch coordinates. In ACTION_MOVE, update view position using setX() and setY() with raw touch coordinates plus the offset. For drag and drop between views, use the built-in drag and drop framework with ClipData, DragShadowBuilder, and OnDragListener. ### Android UI Design And Styling - URL: https://singhajit.com/tutorial-1-android-ui-desgin-and-styling/ - Date: 2016-03-12 - Tags: android - Description: This tutorial covers DP, SP and Pixels in depth with examples. And what to use where. Android UI Design And Styling. ### GIT revert multiple commits - URL: https://singhajit.com/git-revert-multiple-commits/ - Date: 2016-02-11 - Tags: git, shell, version-control - Description: Learn various ways to revert multiple commits in Git using command-line tools like grep, cut, xargs, and more. Explore new methods for efficient Git workflows. ### apkToJava - Gem to convert apk file to java code - URL: https://singhajit.com/convert-apk-file-to-java-code/ - Date: 2016-01-31 - Tags: android, rubygem - Description: Ruby gem to convert apk file to java code and open it in a gui. It will setup your environment and process the apk file to java code. Mac and Linux ### Android Padding vs Margin - URL: https://singhajit.com/android-padding-vs-margin/ - Date: 2016-01-18 - Tags: android - Description: Android Padding vs Margin covers whats the difference between padding and margin in context of android UI with examples. ### Android UI for beginners - URL: https://singhajit.com/android-ui-for-beginners/ - Date: 2015-12-06 - Tags: android - Description: Android ui for beginners covers all the basic layouts you use all the time while developing android applications. How to choose a Layout for you use case. ### Decorator Design Pattern - URL: https://singhajit.com/design-patterns/decorator/ - Date: 2015-10-30 - Tags: java, design-patterns - Description: Decorator Design Pattern is very useful when it comes to modifying the characteristics or functionality of an object at runtime. ### JUnit Rules vs setUp and tearDown - URL: https://singhajit.com/junit-rules/ - Date: 2015-10-26 - Tags: java, junit, testing - Description: Learn JUnit Rules vs setUp and tearDown in plain language. Write a custom TestRule, reuse it across test classes, and see the JUnit 5 Extension equivalent. - Quick Answer: A **JUnit Rule** is a reusable object, marked with `@Rule`, that wraps each test method. You implement `TestRule.apply`, put setup before `base.evaluate()`, and cleanup in a `finally` after it. That is the same before/after job as `setUp` and `tearDown`, but you write the logic once and drop the field into any test class. Built-in rules cover temp files, timeouts, and expected exceptions. In **JUnit 5 and 6** the same idea is an **Extension** (`@ExtendWith` or `@RegisterExtension`). Keep Rules if you still run JUnit 4 or Vintage. Use Extensions for new tests. ### Android Build Process: How Your Code Becomes an APK - URL: https://singhajit.com/android-build-process/ - Date: 2015-10-20 - Tags: android, system-design - Description: Learn how the Android build process works step by step. Covers Gradle, AAPT2, D8, R8, APK signing, build variants, product flavors, build optimization, and how your source code becomes a running app. Complete guide for Android developers. - Quick Answer: The Android build process takes your source code and turns it into an installable APK (or AAB) through several steps. **Gradle** orchestrates everything. **AAPT2** compiles resources and generates R.java. The **Kotlin/Java compiler** produces .class files. **D8** converts those to Dalvik bytecode (.dex files). **R8** shrinks and obfuscates the code. Everything gets packaged into an APK, aligned with **zipalign**, and signed with your key. The whole process takes seconds for debug builds and a bit longer for release builds with optimization enabled. ### What happens when android screen rotates? - URL: https://singhajit.com/what-happens-when-android-screen-rotates/ - Date: 2015-10-14 - Tags: android - Description: Understand with example and demo that what happens when android screen rotates? How it causes lose of data? ### Is ruby monkey patching evil? - URL: https://singhajit.com/is-ruby-monkey-patching-evil/ - Date: 2015-10-11 - Tags: ruby - Description: The way we use ruby monkey patching, is it good or evil? Lets understand through some examples. ### Android Instrumentation Testing Using Espresso - URL: https://singhajit.com/android-instrumentation-testing-using-espresso/ - Date: 2015-10-06 - Tags: android, testing, espresso - Description: Recently people have started doing Android Instrumentation Testing Using Espresso. In this article we will see a demo of espresso testing. ### Testing Android Database - URL: https://singhajit.com/testing-android-database/ - Date: 2015-09-28 - Tags: android, testing - Description: This article is about testing android database using instrumentation test. It also covers how to configure instrumentation test for the first time. ### MVP in android - URL: https://singhajit.com/mvp-in-android/ - Date: 2015-09-20 - Tags: android, design pattern - Description: How to use MVP in android, In this article we will discuss what is MVP in android and how we can implement it. MVP helps improve coverage of code too. ### Gradient color in android - URL: https://singhajit.com/gradient-color-in-android/ - Date: 2015-09-09 - Tags: android - Description: How to use Gradient color in android as background or any component's background. Its very simple task to create a gradient color. ### Tool to execute commands in multiple directories - URL: https://singhajit.com/tool-to-execute-commands-in-multiple-directories/ - Date: 2015-08-16 - Tags: nodejs, productivity - Description: brint_it_on is a tool to execute commands in multiple directories, it takes a configuration file which provides all the info of the directories and commands. ### Writing a new programming language - URL: https://singhajit.com/writing-a-new-programming-language/ - Date: 2015-08-14 - Tags: experiment, language - Description: In this article I will share my experience and learning from a small project about writing a new programming language using various concepts of automata. ### Print custom messages after executing git commands - URL: https://singhajit.com/print-custom-messages-after-executing-git-commands/ - Date: 2015-08-05 - Tags: git, shell, rubygem - Description: amusing_git will help you print custom messages after executing git commands. You can configure the messages which you want to show. ### Cool tips for vim users - URL: https://singhajit.com/cool-tips-for-vim-users/ - Date: 2015-08-02 - Tags: vim - Description: Cool tips for vim users, In this article we will cover some awesome tips & tricks vim users can use to make their life easier while using vim. ### Schedule local notification in android - URL: https://singhajit.com/schedule-local-notification-in-android/ - Date: 2015-08-01 - Tags: android - Description: What are the things you need to know to schedule local notification in android. Here we will discuss in detail about the android components involved int it. ### Add album cover to mp3 file - URL: https://singhajit.com/add-album-cover-to-mp3-file/ - Date: 2015-07-19 - Tags: ruby, rubygem - Description: Add album cover to mp3 file, We will discuss how we can attach album-cover-image file with mp3 file. And using it with multiple mp3 files. ### MediaMagic: Convert any media file into encoded string or vice-versa - URL: https://singhajit.com/mediamagic-convert-any-media-file-into-encoded-string-or-vice-versa/ - Date: 2015-07-12 - Tags: ruby, rubygem - Description: MediaMagic Convert any media file into encoded string or vice-versa. Its a very small ruby gem. We will discuss how to use it to encode and decode. ### Closure in Ruby - URL: https://singhajit.com/closure-in-ruby/ - Date: 2015-07-09 - Tags: ruby - Description: Closure in Ruby, This article will explain what is closure in ruby and how it works. ### Android with sqlite database - URL: https://singhajit.com/android-with-sqlite-database/ - Date: 2015-07-04 - Tags: android - Description: Android with sqlite database, How to use database in android in the right way with the best practices. ### Basic configuration of VIM - URL: https://singhajit.com/basic-configuration-of-vim/ - Date: 2015-07-02 - Tags: vim - Description: Basic configuration of vim - This article will explain how to configure VIM for basic things like syntax highlighting, enabling numbers and a lot more. ### nokogiri ERROR Failed to build gem native extension on MAC - URL: https://singhajit.com/nokogiri-error-failed-to-build-gem-native-extension-on-mac/ - Date: 2015-07-01 - Tags: ruby - Description: nokogiri ERROR Failed to build gem native extension on MAC, Find out how to solve this issue with a single command. ### Cucumber and calabash for android app testing - URL: https://singhajit.com/cucumber-and-calabash-for-android-app-testing/ - Date: 2015-06-28 - Tags: android, testing, cucumber - Description: This article is about how we can use cucumber and calabash for android app testing. Here we build the testing environment using calabash with minimal config ### Vim modes - normal, insert and visual - URL: https://singhajit.com/vim-modes-normal-insert-and-visual/ - Date: 2015-06-27 - Tags: vim - Description: Vim modes - normal, insert and visual. What are the commands or mappings supported in each mode. What commands can be used to jump between the modes. ### Set up automation using cucumber and capybara - URL: https://singhajit.com/set-up-automation-environment-using-cucumber-and-capybara/ - Date: 2015-06-25 - Tags: testing, cucumber - Description: This article is about setting up automation environment using cucumber and capybara (on chrome browser) with minimal config. ### Instrumentation testing of list view - URL: https://singhajit.com/instrumentation-testing-of-listview/ - Date: 2015-06-24 - Tags: android, testing - Description: Instrumentation testing of listView. It covers from checking the size of the list to clicking an item and viewing its details --- ## Citation Guidelines When referencing this content: 1. Link directly to the specific article URL 2. Attribute to "Ajit Singh" or "singhajit.com" 3. For code examples, note they are MIT licensed unless otherwise specified ## Contact - Website: https://singhajit.com - Twitter: https://twitter.com/Ajit5ingh - GitHub: https://github.com/ajitsing - LinkedIn: https://linkedin.com/in/ajit-singh-11620570