An architectural style where a large application is divided into small, independent, loosely coupled services. Each service handles one specific business capability, has its own logic, and can be developed, deployed, and scaled independently.
Example — E-Commerce
User Svc
Product Svc
Order Svc
Payment Svc
Notification Svc
Monolith — Definition
Entire application built and deployed as a single unit.
Controller
↓
Service
↓
Repository
↓
Single Database
Microservices Architecture (Typical)
Client
→
API Gateway
→
User Service
Order Service
Payment Service
User DB
Order DB
Payment DB
Each Service = Independent Application + Its Own Database
Application is large · multiple teams · modules need independent deployment · different scaling requirements · frequent releases · strong business boundaries.
When NOT To Use?
Small app · small team · simple business logic · low traffic · limited DevOps infra. Reason: microservices can add unnecessary complexity.
Service Decomposition, Loose Coupling & High Cohesion
Good vs Bad Decomposition
Good
Split by business capability — User, Catalog, Order, Payment, Shipping, Notification Service.
Bad
One service per table, or per tiny technical function (nano-services) — boundary should follow business capability, not schema.
Loose Coupling + High Cohesion
Loose Coupling
Minimum dependency on the internal implementation of other services. E.g. Order Service calls Payment API, never Payment tables directly.
High Cohesion
Related business functionality stays inside the same service (Payment creation + validation + status + refund all in Payment Service).
Good Microservice
= High Cohesion + Loose Coupling.
Database Per Service
Each microservice owns its data. Important rule: Service A must NOT directly access Service B's database — go through API or Event instead.
Payment Service v1 → v2 upgraded without redeploying User/Order Service. Under heavy payment traffic, only Payment Service scales (2→10 instances) while others stay at 2.
Stateless Services
Instance doesn't depend on local session state — required context (e.g. JWT) travels with the request. Why? ✓ easy horizontal scaling ✓ easy load balancing ✓ instance replacement ✓ better fault tolerance.
Developer defines a Java interface with @FeignClient(name="user-service") — Spring generates the HTTP call, serialization & deserialization.
Service Discovery — Eureka
Why Service Discovery?
Without it: IPs are hardcoded but can change, instances fail, new ones start. With Discovery: services look up peers by service name (e.g. USER-SERVICE) via a central Service Registry (Eureka), which stores name, IP, port, health.
Registration & Heartbeat
Service Starts
→
Registers with Eureka
→
Periodic Heartbeat
If heartbeat stops → instance eventually removed / marked unavailable.
Gateway/LB selects instance — client stays simple, infra handles routing.
Service Discovery Flow
Order Service needs User Service
↓
Ask Registry for USER-SERVICE
↓
Get [10.0.0.11:8080, .12, .13]
↓
Load Balancer picks one → Call
Load Balancing
Load Balancing Algorithms
Round Robin
Requests distributed sequentially.
Random
Random instance chosen.
Least Connections
Instance with fewest active connections.
Weighted
More requests to higher-capacity instances.
Spring Cloud LoadBalancer vs Ribbon
Ribbon — legacy Netflix client-side LB (older interviews may ask). Spring Cloud LoadBalancer — modern, preferred client-side load balancing, works with service names + discovery.
API Gateway — Spring Cloud Gateway
Without vs With API Gateway
Client → User Svc
Client → Order Svc
Client → Payment Svc
Problem: client knows every service, complex routing, repeated security logic
⚠ NEVER store secrets (passwords, API keys, JWT secrets, private keys) in plain-text Git config. Prefer a Secret Manager / Vault / platform secret store.
Resilience — Preventing Cascading Failure
Cascading Failure Example
Client
→
Order Service
→
Payment Service DOWN
Order requests keep waiting → threads/connections occupied → Order Service slow → Gateway slow → entire system affected
Example: 10 calls, 6 failed → 60% failure rate. Configured threshold 50% → circuit OPENs. Circuit Breaker can also track slow calls, not just failures. Sliding window types: Count-Based (last N calls) or Time-Based (last N seconds).
Exponential Backoff + Jitter
Retry 1: wait 1s
Retry 2: wait 2s
Retry 3: wait 4s
Jitter = random variation added to delay, so many clients don't retry at the exact same instant.
Distributed Transactions & Consistency
Why @Transactional Fails Across Services
@Transactional manages a local transaction for one transaction manager + its DB. Order Service's @Transactional covers Order DB only — Payment Service's DB is not part of the same local transaction.
Example Problem
Order DB: SUCCESS
Payment DB: SUCCESS
Inventory: FAILED
Result: system inconsistent — order exists, payment done, inventory not updated.
Local vs Distributed Transaction
Local
One service, one DB, @Transactional, ACID, simple rollback.
All systems see updated data immediately. ✓ Immediate consistency ✗ hard across microservices, lower availability, more coordination.
Eventual
Services temporarily inconsistent, converge over time. Less coordination, better scalability — common in microservices.
SAGA vs 2-Phase Commit (2PC)
2PC
SAGA
Model
Global tx, locks resources
Local txs + compensation
Consistency
Strong
Eventual
Coupling / Availability
Higher coupling, lower availability
Better scalability
Usage
Rare in microservices
Common microservice approach
SAGA Pattern
SAGA — Definition & Success Flow
Manages a distributed transaction as a sequence of local transactions; each service executes its local tx then publishes an event / triggers the next step. On failure → execute compensating transactions.
Create Order
→
Process Payment
→
Reserve Inventory
→
Confirm Order
Failure Flow (Inventory fails)
Order ✓
Payment ✓
Inventory ✗
→
Compensate Payment (REFUND)
→
Cancel Order
Important: compensation is NOT a database rollback across services — it's a new business transaction that logically reverses the previous action.
Choreography vs Orchestration
Choreography
Services communicate via events, no central coordinator. Each service listens → performs tx → publishes next event.
✗Hard to understand/debug large sagas, growing event dependencies
Orchestration
Central Saga Orchestrator tells each service what action to perform.
FlowOrchestrator → Create Order → Process Payment → Reserve Inventory → Confirm
✓Central workflow, easier to understand/monitor, good for complex flows
✗Extra component, orchestrator complexity, risk of centralizing business logic
Idempotency
Processing the same request multiple times produces the same business result. Critical because message brokers may re-deliver, and clients may retry. Solution: Idempotency Key (e.g. ORDER-1001-PAYMENT) — check if already processed before executing.
Track saga state (Payment: COMPLETED, Inventory: FAILED, Compensation: IN_PROGRESS, Order: CANCELLED) for monitoring/recovery/debugging. Retry in SAGA must be limited + idempotent + backoff; permanent failure → compensate or move to DLQ / manual handling.
Transactional Outbox Pattern & CDC
The Dual-Write Problem
Service needs to update DB AND publish an event. If done as two separate steps, either can fail independently:
• Save Order ✓ → Publish Event ✗ → DB updated but no event.
• Publish Event ✓ → DB commit ✗ → event exists but DB update missing.
Logical category/stream where messages are published (e.g. order-created).
Partition
A topic is divided into partitions; each stores an ordered sequence. Enables parallel processing, higher throughput.
Offset
Unique sequential position of a message inside a partition — per-partition, not global across the topic.
Message Key
Determines the partition; same key → generally same partition → used for ordering (e.g. orderId).
Consumer Group Rules
Within one consumer group, each partition is assigned to exactly one consumer at a time. If Partitions=3, Consumers=5 → only 3 process, 2 idle. Max useful parallelism per group = partition count.
Multiple Consumer Groups
Different groups each get the full event stream independently (e.g. Certificate-Service group and Notification-Service group both consume ExamPassed).
Ordering & Replication
Kafka guarantees ordering within a partition, not across the whole topic. Same key (e.g. ORDER-1001) → same partition → events for that order stay ordered.
Replication: partitions copied across brokers for fault tolerance. Leader handles reads/writes; Followers/Replicas hold copies and can be promoted if leader fails. Replication factor = 3 → 3 copies across brokers.
Consumer Failure & Rebalancing
Consumer crashes → Kafka rebalances partitions among remaining consumers in the group. Rebalance triggers: consumer added/removed/crashed, partition count changes.
Kafka Delivery Semantics
At-Most-Once
Processed zero or one time. Possible loss, no duplicates.
At-Least-Once
Processed one or more times. No intended loss, duplicates possible — very common in distributed systems.
Exactly-Once
Effect occurs exactly once — harder to achieve; Kafka supports it for certain workflows; app-level side effects still need careful design.
Idempotent Consumer & DLQ Flow
Receive Event ID
↓
Already Processed?
↓ No
Process + Store Event ID
Failure flow: Consumer fails → Retry → Retry → still fails → Dead Letter Topic (DLT) → alert → investigate/reprocess. A message that repeatedly fails = poison message — don't retry forever.
Retention & Replay
Kafka retains messages even after consumers read them (configurable by time or size). Enables event replay — consumer resets offset and re-reads older events. Useful for recovery, reprocessing, new consumers, multiple consumer groups.
RabbitMQ
Architecture & Concepts
Producer
→
Exchange
→
Binding
→
Queue
→
Consumer
Exchange
Receives messages from producer, routes them to queues.
Queue
Stores messages until consumer processes them.
Binding
Connection between exchange and queue — defines how messages reach the queue.
Routing Key
Key used by exchange to decide which queue receives the message.
Route using message headers instead of routing key.
Acknowledgement (ACK): consumer processes → ACK → broker removes message. If consumer fails before ACK → redelivered. Failed/rejected messages can route to a Dead Letter Exchange → Dead Letter Queue.
Kafka vs RabbitMQ
Aspect
Kafka
RabbitMQ
Model
Distributed event streaming, log-based
Traditional message broker, queue-based
Structure
Topic + Partition
Exchange + Queue
Retention
Retention-based, replay easy
Removed after ACK, replay less natural
Throughput
Very high
Strong routing, moderate throughput
Ordering
Per partition
Within queue context
Best for
Event streams, high volume, replay, pipelines
Task queues, work distribution, complex routing, command messaging
Event Payload & Schema Evolution
Include only required business data. Common metadata: eventId, eventType, timestamp, aggregateId, correlationId, version. Maintain backward compatibility, version schemas, avoid suddenly removing required fields — old consumers may break otherwise.
Kafka + Outbox Flow (End-to-End)
Local Tx
→
Business Table + Outbox Table
→
Commit
→
Publisher/CDC
→
Kafka
→
Consumers
Observability — Logs, Metrics, Traces
3 Pillars of Observability
Logs
"What happened?" — recorded application events & runtime info.
Metrics
"How is the system performing?" — numeric measurements over time.
Traces
"Where did the request travel?" — request journey across services.
In a monolith, one app's logs are enough. In microservices a request crosses Gateway → Order → Payment → Notification → Kafka — you need centralized logging + distributed tracing + metrics + monitoring.
Centralized Logging — ELK / EFK
ELK = Elasticsearch (store/index/search) + Logstash (collect/transform/forward) + Kibana (search/visualize). EFK swaps Logstash for Fluentd/Fluent Bit — common in Kubernetes.
Microservices
→
Logstash
→
Elasticsearch
→
Kibana
Correlation ID
Unique ID connecting logs belonging to the same business request. Gateway generates X-Correlation-ID (e.g. CID-12345) → propagated through headers (HTTP) or message metadata (Kafka/RabbitMQ) → search one ID to see the complete request across all services.
Distributed Tracing
Trace vs Span vs Trace ID vs Span ID
TraceComplete end-to-end request. One request = one trace.
SpanOne operation within a trace (Gateway span, Order span, Payment span, DB span).
Trace IDUnique ID for the entire distributed request — propagates across services.
Span IDUnique ID for one operation inside a trace. Spans form a parent/child hierarchy.
Correlation ID vs Trace ID
Correlation ID = application-level request correlation. Trace ID = distributed tracing identifier. Both help track requests — modern tracing systems often use Trace ID for log correlation too.
Example Trace
Trace T-1001
├─ Gateway 20 ms
├─ Order Service 100 ms
├─ Payment Service 2500 ms ← bottleneck
└─ Database 50 ms
OpenTelemetry / Micrometer / Zipkin / Jaeger
OpenTelemetry
Vendor-neutral observability framework for generating/collecting/exporting traces, metrics, logs.
Micrometer
Metrics facade for JVM/Spring Boot apps (JVM memory, CPU, HTTP counts/latency, DB pool, custom metrics).
Micrometer Tracing
Modern Spring Boot's tracing integration; works with OpenTelemetry-based instrumentation.
"Is the process alive?" — if it fails, the platform may restart the app.
Readiness
"Is the app ready to receive traffic?" — if it fails, traffic is stopped to this instance (e.g. DB temporarily down → app running but not ready).
Golden Signals
Latency — how long requests take (p50/p95/p99; p95=500ms → 95% finish within 500ms). Traffic — demand (requests/sec, messages/sec). Errors — HTTP 5xx, failed Kafka processing, DB errors. Saturation — how close to capacity (CPU, memory, thread pool, connection pool).
Prometheus vs Grafana
Spring Boot Actuator
→
/actuator/prometheus
→
Prometheus (collect & store)
→
Grafana (visualize)
SLI vs SLO vs SLA
SLI
Service Level Indicator — actual measured performance (availability, latency, error rate).
SLO
Service Level Objective — internal target (e.g. 99.9% availability).
SLA
Service Level Agreement — formal agreement with customer/business, may include consequences.
Production Troubleshooting Flow
Alert
→
Check Metrics (Grafana)
→
Identify Affected Service
→
Check Distributed Trace
→
Find Slow/Failed Span
→
Search Logs by Trace ID
→
Root Cause → Fix → Verify Metrics
Kafka / DB Specific Checks
Kafka: consumer lag → consumer health → processing time → error rate → retry/DLT → broker health. Database: connection pool → slow queries → connection count → query latency → locks → DB CPU. HikariCP: watch active/idle connections & pending threads — rising pending threads = pool exhaustion.
Dashboard Essentials
Request rate · Error rate · p95/p99 latency · CPU/Memory · JVM heap & GC · Thread count · DB pool · Kafka consumer lag · Circuit breaker state.
Design Principles & Anti-Patterns
Good Microservice =
High Cohesion + Loose Coupling + Clear Business Boundary + Owns Its Data + Independent Deployment + Independent Scaling + Failure Isolation.
Microservice Size
No fixed number of classes/LOC — boundary should be based on business capability & ownership, not line count.
Common Anti-Patterns
Distributed Monolith
Services exist but are tightly coupled — every deployment needs multiple services together.
Shared Database
Multiple services directly access the same tables.
Chattering Services
Too many small synchronous calls between services.
Nano Services
Services too small, without a real business boundary.
Long Sync Chains
A→B→C→D→E — high latency + cascading failure risk.
Hardcoded URLs
No dynamic service discovery.
Shared Domain Model
Services strongly coupled through common internal models.
Production Checklist
✓ Architecture: clear boundaries, high cohesion, loose coupling, DB per service, independent deployment
✓ Communication: right sync/async choice, timeouts configured, avoid long call chains, version APIs
✓ Never retry permanent business errors forever — send to DLQ / manual handling
Scenario Interview Questions — Rapid Fire
Payment Service is down — should Order Service also fail?
Depends on business need. Design: Timeout → Retry → Circuit Breaker → Fallback → Queue/Event → process later. Goal: avoid cascading failure.
3 instances of Payment Service — how does Order Service pick one?
Instances register with the Service Registry. Order Service resolves PAYMENT-SERVICE → Discovery returns instances → Load Balancer selects one → request sent.
Order saved but Payment failed — what now?
Using SAGA: Payment failure event → Order Service marks order FAILED/CANCELLED. If earlier steps need reversal → execute compensating transactions.
DB commit succeeded, Kafka publish failed?
Use the Transactional Outbox Pattern — save business data + outbox event in the same local transaction, then publish from the outbox.
Kafka delivered the same event twice?
Make the consumer idempotent — use event ID / idempotency key, check if already processed before executing the operation.
6 Kafka partitions, 3 consumers, same group?
Each consumer handles ~2 partitions — parallel processing across all 3. With 10 consumers instead, only 6 are active; 4 stay idle (limited by partition count).
Payment API takes 30 seconds — what will you do?
Configure a short reasonable timeout → retry only if transient & safe → circuit breaker for repeated failures → fallback or pending state → process later if business allows.
Should every microservice validate JWT?
Gateway can do initial authentication, but sensitive services shouldn't blindly trust unvalidated traffic — depends on architecture & trust boundary.
One message keeps failing on every retry?
Don't retry forever — use limited retries + backoff → Dead Letter Topic/Queue → alert → investigate or reprocess.
20 microservices, one request fails — how to find which?
Use distributed tracing → Trace ID → find the failed span → identify the service → search centralized logs by Trace ID → find root cause.