The system should support millions of documents, concurrent ingestion and query workloads, incremental updates, document versioning, and reliable retrieval for LLM-powered applications.

1. Functional Requirements (FR)

1. Users should be able to upload documents through APIs or connect external sources such as Google Drive, S3, and internal knowledge bases.

2. The system should extract text, clean and chunk documents, and generate embeddings asynchronously.

3. Newly uploaded or updated documents should become searchable within seconds under normal operating conditions.

4. Users should be able to perform semantic and keyword-based searches and retrieve relevant document chunks.

5. The system should support document updates, deletion, versioning, and re-indexing without disrupting query serving.

2. Non-Functional Requirements (NFR)

1. Low indexing latency: Target a p95 time of less than 5 seconds from document ingestion to searchability for supported document sizes and normal workloads.

2. Low query latency: Target p95 retrieval latency below 200 ms, excluding LLM generation and external reranking latency.

3. Scalability: Support millions of documents and scale ingestion workers, embedding generation, indexing, and query serving independently.

4. High availability: Target 99.9% availability for search APIs, with replicated storage and graceful degradation during component failures.

5. Reliability and consistency: Prevent lost ingestion events, process retries safely, and ensure deleted or superseded document versions are not returned as valid results.

Note: These are proposed design targets, not guaranteed performance. Actual latency depends on document size, embedding model throughput, index configuration, and system load.

3. Capacity Estimation

Let’s assume the following workload for our initial design.

• Documents ingested: 10 million per day.

• Average document size: 100 KB.

• Average chunks per document: 20.

• Embedding dimensions: 768.

• Embedding representation: 32-bit floating point.

• Search requests: 10 million per day.

A. Document ingestion

Average document ingestion rate:

10,000,000 / 86,400 ≈ 116 documents/second.

Assuming a peak traffic multiplier of 5:

Peak ingestion rate ≈ 580 documents/second.

B. Embedding generation

Total chunks generated daily:

10 million × 20 = 200 million chunks/day.

Average embedding generation rate:

200 million / 86,400 ≈ 2,315 embeddings/second.

Peak embedding rate at 5× traffic:

Approximately 11,575 embeddings/second.

This is a significant workload. We need batched embedding generation, multiple workers, and potentially GPU-backed inference.

C. Vector storage

Each embedding occupies:

768 dimensions × 4 bytes = 3,072 bytes.

For 200 million chunks:

200 million × 3,072 bytes ≈ 614 GB.

This is the raw vector storage generated per day, excluding metadata, index structures, replication, and storage overhead.

At this rate, raw vector storage grows by approximately 18.4 TB per month.

Important design implication: We cannot rely on a single vector database instance. We need sharding, replication, index management, and potentially compression techniques such as product quantization.

D. Query traffic

10 million queries/day:

10 million / 86,400 ≈ 116 queries/second.

At 5× peak traffic:

Approximately 580 queries/second.

The retrieval system should scale independently of the ingestion pipeline because the two workloads have different resource requirements.

4. High-Level Architecture

Architecture diagram

                         DOCUMENT INGESTION
                                  |
                 +----------------+----------------+
                 |                                 |
           Upload API                         Source Connectors
                 |                           (S3, Drive, etc.)
                 +----------------+----------------+
                                  |
                          Ingestion Service
                                  |
                    +-------------+-------------+
                    |                           |
              Object Storage              Metadata DB
              (Original Files)          (Document Versions)
                    |
              Ingestion Event
                    |
                Kafka Topic
                    |
             Document Processing
                    |
           Text Extraction / OCR
                    |
             Cleaning and Parsing
                    |
               Chunking Service
                    |
             Chunk Generation
                    |
             Embedding Service
                    |
             Vector Index Writer
                    |
          +---------+----------+
          |                    |
     Vector Database       Keyword Index
       (ANN Search)        (BM25 / Lucene)
          |                    |
          +---------+----------+
                    |
             Search API Layer
                    |
             Query Processing
                    |
             Query Embedding
                    |
          +---------+----------+
          |                    |
     Vector Retrieval      Keyword Retrieval
          |                    |
          +---------+----------+
                    |
              Result Fusion
                    |
              Optional Reranker
                    |
             Top-K Relevant Chunks
                    |
             RAG / LLM Service
                    |
               Final Answer

Supporting components:

Redis:
- Cache frequent queries and document metadata.
- Store short-lived ingestion status.
- Cache query embeddings where appropriate.

Metadata Database:
- Store document ownership, versions, status, and indexing metadata.

Kafka:
- Buffer ingestion events.
- Decouple document processing, embedding generation, and indexing.

Object Storage:
- Store original documents and extracted document artifacts.

Observability:
- Track indexing latency, Kafka consumer lag, embedding failures,
  indexing failures, search latency, and document freshness.

5. Microservices and Responsibilities

1. API Gateway

Responsibilities:

• Authenticate users and validate tenant access.

• Apply rate limits and request-size limits.

• Route requests to ingestion and search services.

• Enforce tenant-level quotas.

2. Ingestion Service

Responsibilities:

• Accept document uploads and external-source events.

• Generate document IDs and ingestion job IDs.

• Store the original document in object storage.

• Create metadata records and publish ingestion events.

• Support idempotency to avoid duplicate ingestion jobs.

3. Document Processing Service

Responsibilities:

• Extract text from PDFs, DOCX files, HTML, and other supported formats.

• Run OCR for scanned documents where necessary.

• Clean extracted text and preserve useful metadata.

• Split documents into chunks suitable for retrieval.

• Record processing failures and support retries.

This service is CPU-intensive for many file formats, while OCR can require additional specialized compute.

4. Embedding Service

Responsibilities:

• Convert text chunks into vector embeddings.

• Batch multiple chunks into embedding requests.

• Support GPU-backed embedding models when appropriate.

• Apply backpressure when inference capacity is exhausted.

• Track the model version used to generate each embedding.

5. Indexing Service

Responsibilities:

• Write vectors and metadata to the vector database.

• Update the keyword index.

• Maintain document-to-chunk mappings.

• Handle retries and duplicate writes.

• Publish an indexing-completion event after the required indexes are ready.

6. Metadata Service

Responsibilities:

• Track document versions and ingestion states.

• Maintain tenant and access-control metadata.

• Track which chunks belong to each document version.

• Coordinate updates, deletions, and re-indexing.

7. Search Service

Responsibilities:

• Embed incoming search queries.

• Execute vector similarity search.

• Execute keyword retrieval through BM25.

• Apply metadata filters and access-control constraints.

• Merge results using a ranking strategy such as Reciprocal Rank Fusion (RRF).

• Return relevant chunks and their metadata.

8. Reranking Service

Responsibilities:

• Rerank the top candidate chunks using a cross-encoder or another ranking model.

• Improve relevance before sending context to the LLM.

• Operate within a configurable latency budget.

Reranking is optional for some workloads because it adds computational cost and latency.

9. RAG Orchestration Service

Responsibilities:

• Assemble retrieved chunks into context.

• Construct the prompt for the selected LLM.

• Generate the final answer.

• Return citations or references to source documents.

• Track retrieval quality, token usage, and generation latency.

6. API Design

We’ll define APIs for document ingestion, ingestion status, search, updates, and deletion.

API 1: Upload a document

POST /v1/documents

Request:

{
  "tenant_id": "tenant_123",
  "collection_id": "engineering_docs",
  "document_name": "kafka_architecture.pdf",
  "source_type": "upload",
  "content_type": "application/pdf",
  "content_url": "s3://documents/kafka_architecture.pdf",
  "metadata": {
    "department": "engineering",
    "visibility": "internal"
  }
}

Response:

{
  "document_id": "doc_123",
  "version": 1,
  "ingestion_job_id": "job_456",
  "status": "ACCEPTED"
}

Design considerations:

• The actual file can be uploaded through a presigned object-storage URL to avoid routing large payloads through application servers.

• The API creates a document record and an ingestion job.

• The document ID remains stable across versions, while each version represents a specific document state.

• The ingestion process is asynchronous; ACCEPTED does not mean the document is already searchable.

• Use an idempotency key to prevent duplicate ingestion when clients retry requests.

API 2: Check ingestion status

GET /v1/documents/{document_id}/status

Response:

{
  "document_id": "doc_123",
  "version": 1,
  "status": "INDEXING",
  "stages": {
    "upload": "COMPLETED",
    "extraction": "COMPLETED",
    "chunking": "COMPLETED",
    "embedding": "COMPLETED",
    "indexing": "IN_PROGRESS"
  }
}

Possible states:

ACCEPTED
PROCESSING
CHUNKING
EMBEDDING
INDEXING
SEARCHABLE
FAILED
DELETING
DELETED

The status API helps clients distinguish between successful ingestion and actual searchability.

API 3: Search documents

POST /v1/search

Request:

{
  "tenant_id": "tenant_123",
  "collection_id": "engineering_docs",
  "query": "How does Kafka handle consumer failures?",
  "top_k": 10,
  "filters": {
    "department": "engineering"
  },
  "search_mode": "HYBRID"
}

Response:

{
  "query_id": "query_789",
  "results": [
    {
      "document_id": "doc_123",
      "document_version": 1,
      "chunk_id": "chunk_101",
      "text": "Kafka consumers can recover from failures by...",
      "score": 0.92,
      "source": {
        "document_name": "kafka_architecture.pdf",
        "page": 12
      }
    }
  ]
}

Search modes:

• VECTOR: Semantic similarity search.

• KEYWORD: Traditional lexical search using BM25.

• HYBRID: Combine semantic and keyword results.

The service must enforce authorization independently of any tenant ID supplied in the request. Tenant identity should be derived from the authenticated principal and validated against the requested collection.

API 4: Update a document

PUT /v1/documents/{document_id}

Request:

{
  "content_url": "s3://documents/kafka_architecture_v2.pdf",
  "metadata": {
    "department": "engineering",
    "visibility": "internal"
  }
}

Response:

{
  "document_id": "doc_123",
  "version": 2,
  "ingestion_job_id": "job_457",
  "status": "ACCEPTED"
}

The new version is processed asynchronously. The system should track which version is active and prevent obsolete indexing jobs from replacing newer document content.

API 5: Delete a document

DELETE /v1/documents/{document_id}

Response:

{
  "document_id": "doc_123",
  "status": "DELETION_ACCEPTED"
}

Deletion should immediately mark the document as unavailable for new retrievals through an authoritative metadata or tombstone mechanism. Physical removal from vector and keyword indexes can happen asynchronously.

This distinction is important because distributed indexes may take time to propagate deletions.

7. Database Choices

We should use different storage systems for different access patterns rather than forcing everything into one database.

A. Object Storage — Original Documents

Choice: S3-compatible object storage.

Stores:

• Original documents.

• Extracted text artifacts where useful.

• Document-processing artifacts.

• Archived versions when retention policies require them.

Example object key:

tenant_123/
  engineering_docs/
    doc_123/
      version_1/original.pdf
      version_2/original.pdf

Why object storage?

• Low cost for large files.

• High durability.

• Easy integration with event-driven processing.

• Independent scaling of document storage and compute.

B. Metadata Database

Choice: PostgreSQL for the initial design; a horizontally scalable relational or key-value store may be appropriate at larger scale.

Stores:

• Document ownership and tenant association.

• Document version and lifecycle state.

• Ingestion job status.

• Chunk counts and indexing metadata.

• Active version and deletion state.

Example document record:

{
  "document_id": "doc_123",
  "tenant_id": "tenant_123",
  "collection_id": "engineering_docs",
  "current_version": 2,
  "status": "SEARCHABLE",
  "created_at": "2026-10-24T10:00:00Z",
  "updated_at": "2026-10-24T10:05:00Z"
}

Example version record:

{
  "document_id": "doc_123",
  "version": 2,
  "object_key": "tenant_123/engineering_docs/doc_123/version_2/original.pdf",
  "embedding_model": "embedding-model-v3",
  "chunk_count": 20,
  "status": "SEARCHABLE",
  "created_at": "2026-10-24T10:05:00Z"
}

C. Vector Database

Choice: A distributed vector database such as Milvus, Qdrant, or Pinecone.

Stores:

• Chunk embeddings.

• Chunk identifiers.

• Document and version references.

• Searchable metadata used for filtering.

Example vector record:

{
  "chunk_id": "chunk_101",
  "tenant_id": "tenant_123",
  "collection_id": "engineering_docs",
  "document_id": "doc_123",
  "document_version": 2,
  "chunk_index": 0,
  "embedding": [0.12, -0.04, 0.87],
  "metadata": {
    "department": "engineering",
    "page": 12
  }
}

The embedding array is abbreviated here for readability; the actual vector would contain all 768 dimensions in our capacity example.

Use an approximate nearest-neighbor index such as HNSW or IVF-based indexing depending on memory constraints, update patterns, and recall requirements.

D. Keyword Search Index

Choice: Elasticsearch or Apache Lucene-based infrastructure.

Stores:

• Extracted text.

• Chunk identifiers.

• Document and version references.

• Filterable metadata.

Why do we need keyword search alongside vectors?

Suppose a user asks:

Find documentation for error code KAFKA-1438.

Vector similarity alone may not reliably rank the exact identifier first. BM25-based retrieval is effective for exact terms, identifiers, and uncommon keywords.

Hybrid retrieval combines these strengths.

E. Kafka

Choice: Apache Kafka.

Example topics:

document-ingestion
document-extraction
document-chunking
embedding-generation
vector-indexing
keyword-indexing
document-deletion
document-dlq

Kafka provides buffering, replay, and decoupling between processing stages.

Use the document ID or a suitable document-and-version key for partitioning when ordering related events matters. Avoid assuming that Kafka guarantees exactly-once effects in external databases; consumers and index writers still need idempotent processing.

F. Redis

Use Redis for:

• Short-lived ingestion status caching.

• Query embedding caching.

• Carefully scoped result caching.

• Rate limiting.

Do not use Redis as the authoritative source of document lifecycle state.

8. Real-Time Ingestion Workflow

Let’s walk through what happens when a user uploads a new document.

Step 1: Accept the document

The client uploads a file and calls the ingestion API.

The ingestion service:

1. Authenticates the caller.

2. Validates collection permissions.

3. Stores the document in object storage.

4. Creates a metadata record.

5. Publishes an ingestion event to Kafka.

The API returns a job ID without waiting for the complete pipeline.

Step 2: Extract and normalize text

A processing worker consumes the event.

It extracts text from the file, performs OCR if necessary, normalizes the content, and preserves metadata such as page numbers and section headings.

Failed processing attempts are retried according to the error type and retry policy.

Step 3: Chunk the document

The chunking service divides the extracted text into smaller units.

For example:

Document: Kafka Architecture

Chunk 1:
Kafka topics and partitions...

Chunk 2:
Consumer groups and offset management...

Chunk 3:
Replication and leader election...

Chunk 4:
Consumer failure handling...

Chunk size should be chosen based on the embedding model, document structure, and retrieval quality.

Overlapping chunks can preserve context across boundaries, but excessive overlap increases storage and embedding costs.

Step 4: Generate embeddings

The embedding service processes chunks in batches.

Chunk Text
    |
    v
Embedding Model
    |
    v
768-Dimensional Vector

The service should:

• Batch requests to improve accelerator utilization.

• Bound queue sizes to avoid unbounded memory growth.

• Retry transient failures.

• Record the model version.

• Apply backpressure when the embedding service is overloaded.

Step 5: Update the indexes

The indexing service writes the chunk vectors and metadata to the vector database.

It also indexes the extracted text in the keyword search engine.

These are separate writes, so they are not automatically atomic.

The system must track indexing progress and only declare a document fully searchable when the required retrieval paths are ready, or when an explicitly supported partial-search policy allows it.

Step 6: Mark the document searchable

After the required indexing operations succeed, the system updates the authoritative document version status to SEARCHABLE.

The status endpoint can now report successful ingestion.

The important distinction is:

A document being stored does not mean it is searchable.

Searchability is achieved only after the required processing and indexing steps have completed.

9. Search Query Workflow

Consider this query:

How does Kafka recover when a consumer crashes?

The search flow is:

Client
  |
  v
API Gateway
  |
  v
Search Service
  |
  v
Query Embedding
  |
  +-------------------------+
  |                         |
  v                         v
Vector Search           BM25 Search
  |                         |
  +------------+------------+
               |
               v
          Result Fusion
               |
               v
           Reranking
               |
               v
       Top-K Relevant Chunks
               |
               v
          RAG Service
               |
               v
          LLM Generation
               |
               v
         Answer + Citations

Why hybrid search?

Vector search identifies semantically similar chunks even when the wording differs.

BM25 identifies exact terms and keyword matches.

We can combine rankings using Reciprocal Rank Fusion:

\[ RRF(d) = \sum_{r \in R} \frac{1}{k + rank_r(d)} \]

Where:

• \(d\) is a candidate document chunk.

• \(R\) is the set of retrieval result lists.

• \(rank_r(d)\) is the rank of that chunk in a result list.

• \(k\) is a smoothing constant.

The resulting candidates can be reranked before being passed to the LLM.

For latency-sensitive workloads, run vector and keyword retrieval in parallel rather than sequentially.

10. The Most Important Design Challenge: Searchability Within Seconds

The hardest part of this system is not merely generating embeddings. It is ensuring that newly generated embeddings become visible to queries quickly.

Consider the following timeline:

T+0 ms      Document uploaded
T+200 ms    Ingestion event published
T+800 ms    Text extraction completed
T+1200 ms   Chunking completed
T+2500 ms   Embeddings generated
T+3500 ms   Vector index updated
T+4200 ms   Keyword index updated
T+4500 ms   Document marked searchable

These are illustrative timings, not measured performance results.

The total indexing latency is approximately:

\[ T_{index} = T_{queue} + T_{extract} + T_{chunk} + T_{embed} + T_{vector} + T_{keyword} + T_{publish} \]

Some stages can run in parallel, so the critical-path latency may be lower than the sum of all stage durations.

To meet the five-second target, we need to monitor the entire pipeline rather than optimizing only embedding generation.

Optimization strategies

1. Minimize queueing delay

• Scale Kafka consumers according to queue lag.

• Keep sufficient embedding capacity for peak ingestion.

• Use bounded queues and admission control.

2. Batch embeddings carefully

• Larger batches improve throughput.

• Excessively large batches increase waiting time.

• Use adaptive batching based on traffic and latency objectives.

3. Optimize vector indexing

• Use a vector database that supports incremental inserts.

• Tune index parameters and write buffers.

• Avoid expensive full-index rebuilds for every update.

4. Optimize keyword indexing

• Use near-real-time index refreshes where supported.

• Balance refresh frequency against indexing overhead.

• Track refresh and visibility delays separately.

5. Measure freshness end to end

Record timestamps for upload, extraction, embedding completion, index visibility, and searchability.

This helps identify whether the bottleneck is Kafka lag, GPU capacity, vector indexing, or keyword-index refresh.

11. Document Updates and Versioning

Suppose version 1 of a document says:

Kafka retains messages for 7 days.

The document is updated, and version 2 says:

Kafka retains messages for 14 days.

We cannot allow stale chunks from version 1 to continue appearing as valid search results after version 2 becomes active.

Recommended strategy: Versioned indexing

1. Create version 2 without immediately deleting version 1.

2. Extract, chunk, and embed version 2.

3. Index version 2 in the vector and keyword indexes.

4. Verify the required indexes are ready.

5. Atomically update the authoritative active-version pointer.

6. Ensure queries exclude obsolete versions.

7. Delete version 1’s indexed chunks asynchronously after the transition is safe.

The authoritative active-version pointer can be stored in the metadata database, with suitable caching and invalidation.

Why not delete version 1 first?

If we delete version 1 before version 2 is searchable, the document temporarily disappears from search.

Versioned indexing enables a safer transition, although it requires additional storage and careful query filtering.

Important race condition

Suppose version 3 is submitted while version 2 is still processing.

If version 2 finishes after version 3, its worker must not overwrite the active version.

Use a monotonically increasing version number or fencing token and validate it when publishing indexing completion.

12. Handling Deletes and Stale Results

Deletion is more complicated than removing a metadata record.

A document’s chunks may exist in multiple places:

• Vector database.

• Keyword index.

• Redis caches.

• Replicas and index segments.

• Pending Kafka messages.

A robust deletion workflow should:

1. Mark the document or version as deleted in authoritative metadata.

2. Prevent retrieval from returning it immediately by enforcing a deletion or visibility check.

3. Publish a deletion event.

4. Remove the chunks from the vector database and keyword index.

5. Invalidate relevant caches.

6. Retry failed deletions and reconcile inconsistent states.

Key principle: Logical deletion should take effect before asynchronous physical cleanup completes.

This avoids relying on the vector database’s deletion propagation timing to enforce document visibility.

For strict access-control or deletion requirements, validate returned candidates against authoritative visibility state, or use a strongly consistent visibility mechanism. This can add latency, so it should be designed deliberately.

13. Consistency Model

We have three major data stores:

• Metadata database.

• Vector database.

• Keyword search index.

They do not share a single atomic transaction.

Therefore, the system should use an asynchronous, version-aware consistency model.

Recommended approach

Metadata database

• Authoritative source for document lifecycle and active version.

Vector and keyword indexes

• Eventually consistent projections of document content.

• Updated asynchronously through retryable indexing jobs.

Kafka

• Durable event transport for processing stages.

• Supports replay and recovery.

Outbox pattern

• If metadata changes and event publication must not be lost, write the metadata change and an outbox record in the same database transaction.

• A separate publisher sends outbox records to Kafka and marks them as published.

• Consumers must remain idempotent.

Example failure

The indexing service writes vectors successfully but crashes before recording indexing completion.

On retry, the service should upsert the same chunk IDs rather than create duplicates.

A stable identifier could be generated from:

tenant_id + document_id + version + chunk_index

The vector index writer can use this identifier to make retries safe.

14. Scaling Strategy

Document processing

Scale workers horizontally according to:

• Kafka consumer lag.

• Average processing time.

• CPU utilization.

• Memory pressure.

• OCR workload.

Embedding service

Scale according to:

• Pending chunks.

• Embeddings generated per second.

• GPU utilization.

• Batch size.

• End-to-end embedding latency.

GPU utilization alone is not sufficient. A GPU can be highly utilized while queueing delays violate freshness targets.

Vector database

Scale through:

• Sharding.

• Replication.

• Index partitioning.

• Appropriate ANN index selection.

• Vector compression where suitable.

For 10 billion vectors, the index’s memory footprint and metadata overhead become major design considerations.

Search service

Scale stateless search workers horizontally.

Use bounded concurrency and request timeouts to prevent slow downstream vector queries from exhausting the service.

Backpressure

If embedding generation falls behind:

Kafka lag increases
       |
       v
Embedding queue reaches threshold
       |
       v
Reduce ingestion admission rate
       |
       v
Scale workers or defer low-priority jobs
       |
       v
Protect query-serving capacity

Ingestion should not consume unlimited resources at the expense of interactive search.

15. Failure Scenarios

FailureHandling strategyDocument extraction failsRetry transient failures; send persistent failures to a DLQ.Embedding service becomes unavailableBuffer events in Kafka, apply backpressure, and resume after recovery.Vector database is unavailableRetry writes and preserve indexing state; keep affected versions unsearchable until the required index is ready.Keyword index is delayedTrack indexing completion separately and apply the configured searchability policy.Duplicate Kafka eventUse idempotent consumers and deterministic chunk IDs.Older version finishes after newer versionReject stale completion events using version checks or fencing tokens.Document is deleted while processingCheck deletion state before publishing the version as searchable.Redis becomes unavailableFall back to authoritative metadata and the underlying search systems, subject to capacity limits.Kafka consumer lag grows rapidlyScale consumers where possible, prioritize work, and apply admission control.A partial index update occursReconcile the vector and keyword indexes against authoritative metadata and retry missing operations.

16. Monitoring and Observability

We should track the following metrics.

Ingestion metrics

• Documents accepted per second.

• Document-processing failure rate.

• Extraction and chunking latency.

• Queue depth and Kafka consumer lag.

Embedding metrics

• Embeddings generated per second.

• Average batch size.

• GPU utilization and memory usage.

• Embedding failure rate and latency.

Indexing metrics

• Vector-index write latency.

• Keyword-index refresh latency.

• Time from upload to searchability.

• Number of partially indexed documents.

• Version conflicts and stale-index count.

Search metrics

• p50, p95, and p99 query latency.

• Vector and keyword retrieval latency.

• Cache hit ratio.

• Recall@K and ranking quality.

• Empty-result rate and stale-result rate.

RAG metrics

• Time to first token and total generation latency.

• Context relevance.

• Answer groundedness.

• Citation accuracy.

• Cost per query.

A particularly important metric is:

\[ \text{Indexing Freshness} = T_{\text{searchable}} - T_{\text{upload}} \]

Monitor its p50, p95, and p99 values. Averages alone can hide a backlog that affects a subset of documents.

17. Final Architecture Summary

The architecture can be summarized as follows:

INGESTION
API Gateway
    |
Ingestion Service
    |
Object Storage + Metadata DB
    |
Kafka
    |
Document Processing
    |
Chunking
    |
Embedding Service
    |
Vector Index + Keyword Index
    |
Document Marked Searchable


QUERY SERVING
Client
    |
API Gateway
    |
Search Service
    |
    +-------------------+
    |                   |
Vector Search       BM25 Search
    |                   |
    +---------+---------+
              |
         Result Fusion
              |
          Reranking
              |
        Relevant Chunks
              |
         RAG / LLM
              |
        Answer + Citations


LIFECYCLE MANAGEMENT
Metadata DB
    |
Versioning + Visibility
    |
Deletion Events
    |
Asynchronous Index Cleanup

Key design decisions

1. Kafka decouples ingestion from processing, allowing workers to scale independently.

2. Batched embedding generation improves throughput and accelerator utilization.

3. Hybrid search combines semantic relevance with exact keyword matching.

4. Versioned indexing enables safer updates without making existing content unavailable prematurely.

5. Authoritative metadata and idempotent consumers make retries, deletions, and recovery manageable.

6. Asynchronous index cleanup avoids requiring distributed transactions across the vector database and keyword index.

7. End-to-end freshness monitoring helps ensure that documents become searchable within the target time.

The central engineering challenge is balancing freshness, retrieval quality, and indexing cost while supporting continuous ingestion at scale.

C. CONSISTENCY, VERSIONING, AND DELETION

1. CONSISTENCY PROBLEM

Our RAG pipeline uses multiple independent components:

- PostgreSQL: Stores document metadata and lifecycle state.

- Kafka: Carries asynchronous processing events.

- Vector Database: Stores embeddings and chunk metadata.

- Keyword Index: Stores searchable text and metadata.

- Redis: Caches metadata and search results.

- Object Storage: Stores original documents.

These systems do not participate in one shared atomic transaction.

Therefore, we must handle:

- Partial failures.

- Duplicate events.

- Stale indexing jobs.

- Concurrent document updates.

- Delayed index visibility.

- Deletions that have not propagated to every system.

2. DOCUMENT VERSIONING

Suppose a document initially contains:

Document: payment-retry-policy

Version: 1

Content:

Failed payments should be retried 3 times.

A user updates the document:

Document: payment-retry-policy

Version: 2

Content:

Failed payments should be retried 5 times.

We must ensure that the old version does not appear as the current document after version 2 becomes active.

Recommended approach: Versioned indexing.

Step 1:

Create version 2 in PostgreSQL.

Step 2:

Publish an indexing event to Kafka.

Step 3:

Extract the document text and generate chunks.

Step 4:

Generate embeddings for version 2.

Step 5:

Write version 2 chunks to the vector database.

Step 6:

Write version 2 chunks to the keyword index.

Step 7:

Verify that all required indexes are ready.

Step 8:

Atomically update the active-version pointer in PostgreSQL.

Step 9:

Ensure search requests exclude obsolete versions.

Step 10:

Delete version 1 chunks asynchronously after the transition is safe.

Why not delete version 1 immediately?

If we delete version 1 before version 2 is searchable, the document temporarily disappears from search.

Versioned indexing allows us to prepare the new version before switching to it.

3. HANDLING CONCURRENT UPDATES

Consider this scenario:

Version 2 is submitted at 10:00:00.

Version 3 is submitted at 10:00:01.

However, version 3 finishes indexing before version 2.

Without proper safeguards, version 2 could finish later and incorrectly overwrite version 3.

Solution:

- Assign monotonically increasing version numbers.

- Store the active version in PostgreSQL.

- Include the version number in every indexing event.

- Validate the version before activating it.

- Reject stale completion events.

- Use conditional updates or compare-and-set operations.

Example:

Document ID: doc_123

Active version: 3

A worker finishes indexing version 2.

Before activating version 2, the worker checks the current version.

Since version 3 is already active, version 2 cannot replace it.

Important:

The active-version update must be conditional and atomic.

A worker must never activate an older version simply because its processing finished later.

4. HANDLING DUPLICATE KAFKA EVENTS

Suppose a Kafka consumer processes an indexing event.

1. The consumer writes a vector successfully.

2. The consumer crashes before committing its Kafka offset.

3. Kafka delivers the same event again.

4. The consumer attempts to write the vector again.

Without idempotency, duplicate chunks may be created.

Solution: Use deterministic chunk IDs.

Example:

chunk_id = hash(

tenant_id,

document_id,

document_version,

chunk_index

)

The vector database uses an upsert operation with this identifier.

The keyword index uses the same stable logical identifier for the chunk.

Repeated processing updates the existing logical record rather than creating a duplicate.

For model migrations or embedding regeneration, include the index generation or embedding model version in the indexing identity where necessary.

This makes retries safe without requiring distributed transactions.

5. TRANSACTIONAL OUTBOX PATTERN

Consider the following failure:

1. PostgreSQL successfully creates a document record.

2. The application crashes before publishing the Kafka event.

3. The document exists in PostgreSQL but is never processed.

We can prevent this failure using the transactional outbox pattern.

Workflow:

1. Begin a PostgreSQL transaction.

2. Insert the document metadata.

3. Insert an event into the outbox table.

4. Commit the transaction.

5. An outbox publisher reads unpublished events.

6. The publisher sends events to Kafka.

7. The publisher marks successfully published events.

Both the metadata record and outbox event are committed in the same database transaction.

If the publisher crashes after publishing an event but before marking it as published, it may publish the event again.

Therefore, consumers must remain idempotent.

The outbox pattern prevents a committed metadata change from silently losing its corresponding event.

6. DELETION WORKFLOW

Suppose a user deletes document doc_123.

The document may still exist in:

- PostgreSQL.

- The vector database.

- The keyword index.

- Redis caches.

- Kafka queues.

- Object storage.

Deleting the document from PostgreSQL alone is insufficient.

Recommended workflow:

Step 1:

Mark the document as deleted or unavailable in authoritative metadata.

Step 2:

Prevent new searches from returning the document.

Step 3:

Publish a deletion event to Kafka.

Step 4:

Delete the document’s chunks from the vector database.

Step 5:

Delete the document’s chunks from the keyword index.

Step 6:

Invalidate relevant Redis cache entries.

Step 7:

Ensure pending indexing jobs cannot reactivate the deleted document.

Step 8:

Delete the original document from object storage according to the retention policy.

Step 9:

Retry failed cleanup operations.

Step 10:

Periodically reconcile the indexes against authoritative metadata.

Key principle:

Logical deletion must take effect before asynchronous physical cleanup completes.

This prevents a deleted document from appearing in search merely because index cleanup is still running.

7. DELETION RACE CONDITIONS

Consider this sequence:

1. Version 2 begins embedding generation.

2. A user deletes the document.

3. The deletion event is processed.

4. The vector database removes existing chunks.

5. The delayed embedding worker finishes.

6. The worker writes new vectors for the deleted document.

The document could reappear in the index.

To prevent this:

- Track document lifecycle state in PostgreSQL.

- Assign a version or lifecycle generation to each indexing job.

- Check deletion state before activating a version.

- Reject writes from stale generations.

- Enforce visibility checks during retrieval.

- Run reconciliation to detect unexpected indexed records.

A deletion tombstone should remain available long enough to prevent delayed events from resurrecting the document.

Do not rely exclusively on the order in which Kafka consumers process events across different topics.

8. HANDLING STALE SEARCH RESULTS

Suppose the vector database returns a chunk from version 1 while version 2 is active.

Before returning the result, the system must verify that the chunk belongs to an active and authorized document version.

Possible strategies:

Strategy A: Filter by active version.

Maintain version metadata and apply the active-version constraint during retrieval when supported.

Strategy B: Validate candidates against authoritative metadata.

Retrieve candidate IDs, then verify document visibility and active-version state before returning results.

Strategy C: Use versioned visibility metadata.

Maintain a visibility mechanism that can reject obsolete versions without requiring an expensive database lookup for every candidate.

Recommended approach:

Use index-level filtering where possible, combined with an authoritative visibility mechanism for correctness-sensitive cases.

A stale candidate must never reach the LLM context if the document is deleted or unauthorized.

Caching must not bypass these checks.

9. CONSISTENCY BETWEEN VECTOR AND KEYWORD INDEXES

Suppose version 2 has been written to the vector database, but the keyword index has not finished updating.

A hybrid search request may return inconsistent results.

There are two approaches.

Approach A: Wait for both indexes.

Mark a document searchable only after both the vector and keyword indexes are ready.

Advantages:

- Simpler search semantics.

- Predictable behavior.

- Easier debugging.

Disadvantages:

- The slower index determines the time to searchability.

Approach B: Partial searchability.

Track the readiness of each index independently.

For example:

vector_index_status = READY

keyword_index_status = INDEXING

The search service can temporarily use the vector index while the keyword index catches up.

Advantages:

- Potentially faster availability of new content.

Disadvantages:

- Search results may be incomplete.

- The system needs explicit partial-readiness semantics.

Recommended initial design:

Wait for both indexes before marking a document fully searchable.

If the five-second freshness target cannot be met, consider a recent-write buffer or an explicitly defined partial-searchability policy.

10. CONSISTENCY MODEL

PostgreSQL:

- Authoritative source for document lifecycle state.

- Stores active version and deletion state.

Kafka:

- Durable asynchronous event transport.

- Supports retries and replay.

Vector Database:

- Stores searchable embeddings.

- Updated asynchronously.

Keyword Index:

- Stores searchable text.

- Updated asynchronously.

Redis:

- Stores disposable cached data.

- Never serves as the sole authority for document visibility.

Object Storage:

- Stores original document content.

The overall architecture uses eventual consistency for index updates, with explicit visibility controls for document versions and deletions.

This means the indexes may temporarily contain stale records, but the search service must not expose records that are no longer valid.

11. RECONCILIATION

Even with retries and idempotency, inconsistencies can occur.

Examples:

- A vector exists without corresponding metadata.

- A document is active in PostgreSQL but missing from the keyword index.

- A deleted document still has indexed chunks.

- A document version is marked searchable even though an index write failed.

Run a periodic reconciliation job.

Workflow:

1. Read document metadata and active versions.

2. Compare expected chunks against vector-index records.

3. Compare expected chunks against keyword-index records.

4. Identify missing or stale chunks.

5. Retry failed indexing operations.

6. Remove orphaned records where appropriate.

7. Report persistent inconsistencies.

Reconciliation should be idempotent and safe to run repeatedly.

For large datasets, use incremental reconciliation and partitioned scans instead of repeatedly scanning every record.

12. FINAL SYSTEM DESIGN SUMMARY

Kafka ingestion and backpressure:

- Kafka buffers traffic spikes.

- Independent consumer groups scale processing stages.

- Bounded queues prevent memory exhaustion.

- Backpressure protects downstream services.

- Retries and dead-letter queues isolate failures.

- Idempotent processing prevents duplicate logical records.

Vector indexing and hybrid search:

- HNSW supports fast approximate nearest-neighbor search.

- IVF-PQ reduces memory requirements for large collections.

- BM25 handles exact keywords and identifiers.

- Vector search handles semantic similarity.

- Reciprocal Rank Fusion combines retrieval rankings.

- Reranking improves the ordering of candidate chunks.

- Sharding and replication support scale and availability.

Consistency, versioning, and deletion:

- PostgreSQL maintains authoritative lifecycle state.

- Versioned indexing supports safer document updates.

- Conditional updates prevent stale workers from activating old versions.

- The transactional outbox prevents lost events after metadata commits.

- Idempotent consumers make retries safe.

- Tombstones and visibility checks prevent deleted documents from reappearing.

- Reconciliation repairs inconsistencies across indexes.

The central design principle is:

Use asynchronous processing for scalability, idempotency for reliable retries, versioning for safe updates, and authoritative visibility checks for correctness.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

Absolutely. Let’s design it as an interview-ready HLD, starting from requirements and APIs, then moving into architecture, storage, memory lifecycle, retrieval, conflict resolution, eviction, and personalization.

1. Functional Requirements

1. Store short-term memory

   • Maintain recent conversation/context for the active session.

   • Support fast reads/writes with low latency.

   • Automatically expire stale session context.

2. Extract long-term memories

   • Identify useful facts/preferences from conversations.

   • Example: “I prefer Go for backend development.”

   • Store them independently of a single conversation.

3. Support episodic memory

   • Store important past events/interactions.

   • Example: “User designed an AI Gateway on September 1.”

   • Preserve event time, context, and source conversation.

4. Support semantic memory

   • Store generalized knowledge/facts derived from multiple interactions.

   • Example: repeated conversations can establish that the user prefers system-design answers in a particular format.

5. Retrieve relevant memories

   • Given a new query, retrieve memories based on:

     • semantic similarity

     • recency

     • importance

     • user/session scope

     • memory type

6. Summarize and consolidate memories

   • Compress old conversations into summaries.

   • Merge multiple related memories.

   • Convert repeated episodic observations into semantic knowledge.

7. Update/delete memories

   • Allow memories to be corrected, deleted, or invalidated.

   • Support explicit user instructions such as “forget this.”

8. Resolve conflicting memories

   • Example:

     • Old: User prefers Java

     • New: User now prefers Go

   • Keep history while marking the latest valid preference.

9. Personalize responses

   • Provide the LLM with relevant long-term preferences, history, and context without sending the entire memory store.

2. Non-Functional Requirements

1. Low retrieval latency

   • p95 memory retrieval < 100 ms excluding LLM generation.

2. High availability

   • Target 99.99% availability for memory reads/writes.

3. Scalability

   • Support millions of users and billions of memories.

   • Horizontally scalable ingestion and retrieval.

4. Strong privacy/isolation

   • User memories must never leak across users/tenants.

   • Tenant/user ID must be part of authorization and storage partitioning.

5. Durability

   • Long-term, episodic, and semantic memories should survive service failures.

   • Target durable persistence with replication and backups.

6. Eventual consistency

   • Memory extraction and summarization can be asynchronous.

   • Recent conversation state should remain strongly consistent within a session.

7. Cost efficiency

   • Avoid sending unnecessary memories to LLMs.

   • Use tiered storage and summarization to reduce token/storage costs.

8. Explainability

   • Every memory should retain source/provenance information so we can answer:

   • “Why does the system think this about me?”

3. Memory Types

We can model memory as four logical layers:

                 Agent Memory
                      |
        +-------------+-------------+
        |             |             |
   Short-Term      Long-Term     Episodic
        |             |             |
   Conversation    Semantic      Past Events
      Context       Facts

Short-term memory

Recent context:

User:
"Let's design an Agent Memory System."

Assistant:
"Let's start with FR/NFR."

User:
"Add APIs too."

Usually stored with TTL.

Episodic memory

Specific events:

{
  "event": "User designed Agent Memory System",
  "timestamp": "2026-09-26",
  "context": "System design preparation"
}

Semantic memory

Generalized knowledge:

{
  "fact": "User prefers system design answers with FR, NFR,
            APIs, database choices and microservices architecture"
}

Long-term memory

Long-term memory is the persistent layer containing useful information, primarily semantic and important episodic memories.

4. High-Level Architecture

                         +------------------+
                         |   Agent / LLM    |
                         +--------+---------+
                                  |
                                  v
                       +---------------------+
                       |   Memory Gateway     |
                       +----------+----------+
                                  |
               +------------------+------------------+
               |                  |                  |
               v                  v                  v
        Short-Term Store    Memory Retrieval    Memory Write
               |                  |                  |
               v                  v                  v
             Redis          Query Processor     Event Queue
                                  |                  |
                                  v                  v
                          +---------------+    +-------------+
                          | Vector Search |    | Extraction  |
                          +-------+-------+    |   Worker    |
                                  |            +------+------+
                                  |                   |
                                  v                   v
                           +-------------+      +-------------+
                           | Semantic DB |      | Summarizer  |
                           +-------------+      +-------------+
                                  |
                                  v
                           +-------------+
                           | Memory DB   |
                           +-------------+

                         Supporting Services
                                  |
          +-----------+-----------+-----------+-----------+
          |           |           |           |           |
       Ranking     Conflict    Eviction   Personalization
       Service    Resolver     Service       Service

5. Microservices

1. Memory Gateway

Main entry point.

Responsibilities:

Authentication
Authorization
Read/write routing
Memory-type routing
Request validation

2. Short-Term Memory Service

Handles active conversation state.

Conversation
    |
    v
Short-Term Memory Service
    |
    v
Redis

Redis is suitable because we need:

• very low latency

• TTL

• session-oriented access

• high write throughput

3. Memory Extraction Service

Consumes conversation events asynchronously.

Conversation
     |
     v
Kafka
     |
     v
Memory Extraction Service
     |
     +----> Semantic Memory
     |
     +----> Episodic Memory

The extractor can use an LLM or smaller classification/extraction model.

Example:

"I hate dark mode. Always use light mode."

             |
             v

memory_type = PREFERENCE
key = theme
value = light
importance = 0.85

4. Summarization Service

Responsible for compression.

Conversation History
        |
        v
Summarizer
        |
        +----> Session Summary
        |
        +----> Episodic Summary
        |
        +----> Semantic Facts

5. Memory Retrieval Service

Responsible for finding memories relevant to the current request.

Query
 |
 v
Query Embedding
 |
 v
Vector Search
 |
 +---- semantic similarity
 |
 +---- recency
 |
 +---- importance
 |
 +---- user scope
 |
 +---- memory type
 |
 v
Ranking
 |
 v
Top-K memories

6. Conflict Resolution Service

Handles contradictory memories.

Example:

Memory 1:
User prefers Python

Memory 2:
User prefers Go

Resolver considers:

timestamp
confidence
source
explicitness
frequency
user correction

and produces:

Current:
User prefers Go

Previous:
User preferred Python

We should generally avoid physically deleting historical evidence unless requested; instead mark old memories as superseded.

7. Eviction/Consolidation Service

Responsible for:

TTL expiration
Low-value memory removal
Memory compression
Duplicate merging
Episodic -> semantic consolidation

6. APIs

Store short-term memory

POST /v1/memory/short-term

Request:

{
  "user_id": "u123",
  "session_id": "s456",
  "messages": [
    {
      "role": "user",
      "content": "Let's design an agent memory system"
    }
  ]
}

Response:

{
  "session_id": "s456",
  "status": "stored"
}

Get short-term memory

GET /v1/memory/short-term?user_id=u123&session_id=s456

Response:

{
  "session_id": "s456",
  "messages": [
    {
      "role": "user",
      "content": "Let's design an agent memory system"
    }
  ],
  "summary": "User is designing an agent memory system."
}

7. Store Long-Term Memory

POST /v1/memory/long-term

Request:

{
  "user_id": "u123",
  "type": "semantic",
  "key": "preferred_language",
  "value": "Go",
  "confidence": 0.94,
  "importance": 0.8,
  "source": {
    "conversation_id": "c123",
    "message_id": "m456"
  }
}

Response:

{
  "memory_id": "mem789",
  "status": "created"
}

8. Retrieve Memory

POST /v1/memory/search

Request:

{
  "user_id": "u123",
  "query": "What programming language should I use?",
  "memory_types": [
    "semantic",
    "episodic"
  ],
  "top_k": 10
}

Response:

{
  "memories": [
    {
      "memory_id": "mem789",
      "type": "semantic",
      "content": "User prefers Go for backend development",
      "score": 0.93
    },
    {
      "memory_id": "mem321",
      "type": "episodic",
      "content": "User recently discussed a Go backend system",
      "score": 0.81
    }
  ]
}

9. Delete Memory

DELETE /v1/memory/{memory_id}

Response:

{
  "memory_id": "mem789",
  "status": "deleted"
}

For privacy-sensitive systems, deletion should also trigger deletion from:

Primary DB
Vector DB
Caches
Search indexes
Derived summaries
Backups according to retention policy

10. Update Memory

PATCH /v1/memory/{memory_id}

Request:

{
  "value": "Java",
  "reason": "User explicitly changed preference"
}

Response:

{
  "memory_id": "mem789",
  "status": "updated",
  "version": 3
}

11. Database Choice

I’d use multiple stores rather than forcing everything into one database.

                    Memory System
                         |
        +----------------+----------------+
        |                |                |
       Redis          PostgreSQL       Vector DB
        |                |                |
   Short-term       Metadata/state     Embeddings

Redis

For:

Short-term memory
Session state
Hot memory cache
TTL

PostgreSQL / DynamoDB

For durable memory metadata:

memory_id
user_id
memory_type
content
importance
confidence
created_at
updated_at
version
status
source

For very large scale, DynamoDB/Cassandra can be used for horizontally scalable memory storage.

Vector Database

Stores:

embedding
memory_id
user_id
memory_type
metadata

Possible technologies:

Milvus
Qdrant
Weaviate
Pinecone
OpenSearch

The critical point is that vector search must always be filtered by:

tenant_id
+
user_id
+
memory permissions

before returning results.

12. Memory Schema

A logical memory object:

{
  "memory_id": "mem123",
  "user_id": "u123",
  "tenant_id": "t1",

  "type": "semantic",

  "content": "User prefers Go for backend development",

  "embedding": "...",

  "importance": 0.85,
  "confidence": 0.94,

  "created_at": "2026-09-26T10:00:00Z",
  "updated_at": "2026-09-26T10:30:00Z",

  "last_accessed_at": "2026-09-26T11:00:00Z",

  "access_count": 12,

  "status": "active",

  "version": 3,

  "supersedes": null,

  "source": {
    "conversation_id": "c123",
    "message_id": "m456"
  }
}

13. Memory Extraction Pipeline

This is one of the most important parts of the design.

User Message
     |
     v
Conversation Service
     |
     v
Kafka
     |
     v
Memory Extraction Worker
     |
     v
Candidate Memories
     |
     v
Importance + Confidence
     |
     v
Deduplication
     |
     v
Conflict Detection
     |
     v
Memory Store

Example:

User:
"I prefer Go for backend development and I'm preparing
for system design interviews."

Extractor generates:

[
  {
    "type": "semantic",
    "key": "preferred_backend_language",
    "value": "Go",
    "confidence": 0.95
  },
  {
    "type": "semantic",
    "key": "current_goal",
    "value": "system_design_interview_preparation",
    "confidence": 0.92
  }
]

14. Should Every Message Become a Memory?

No.

Otherwise memory becomes:

Huge
Noisy
Expensive
Hard to retrieve

We need a memory-worthiness classifier.

For each candidate:

Memory Score =
Importance
× Confidence
× Future Utility
× Persistence Probability

Example:

"What's the weather today?"
        -> Don't store

"I prefer Go for backend systems."
        -> Store

"I am designing a payment system today."
        -> Episodic

"I prefer system design answers with APIs and DB choices."
        -> Store as semantic preference

15. Retrieval Strategy

Pure vector similarity isn’t enough.

Suppose the query is:

“Design another system for me.”

Vector search might return hundreds of semantically similar memories.

Instead calculate:

FinalScore =
    α * SemanticSimilarity
  + β * Recency
  + γ * Importance
  + δ * AccessFrequency
  + ε * Confidence

For example:

semantic similarity = 0.82
importance           = 0.90
recency              = 0.70
confidence           = 0.95

Then rank memories.

16. Hybrid Retrieval

Use multiple retrieval strategies:

                   Query
                     |
        +------------+------------+
        |            |            |
     Vector       Keyword       Metadata
     Search       Search         Filter
        |            |            |
        +------------+------------+
                     |
                     v
                  Fusion
                     |
                     v
                  Reranker
                     |
                     v
                   Top-K

This handles queries where semantic similarity alone isn’t enough.

17. Memory Summarization

Suppose a conversation has:

500 messages
50,000 tokens

We shouldn’t continuously retrieve all of them.

Use hierarchical summarization:

Messages
   |
   v
Message chunks
   |
   v
Chunk summaries
   |
   v
Conversation summary
   |
   v
Long-term facts

Example:

500 messages

       ↓

10 chunk summaries

       ↓

1 conversation summary

       ↓

5 semantic memories

This dramatically reduces context size.

18. Episodic → Semantic Consolidation

This is an important distinction.

Suppose we have:

Episode 1:
User asked for Go system design.

Episode 2:
User asked for Go concurrency design.

Episode 3:
User chose Go for backend examples.

The system can infer:

Semantic Memory:
User frequently prefers Go for backend/system-design examples.

Pipeline:

Episodic Memories
       |
       v
Pattern Detection
       |
       v
LLM / Consolidation Model
       |
       v
Semantic Memory

19. Conflict Resolution

Suppose:

Memory A:
User prefers Java
confidence = 0.80
timestamp = Jan

Memory B:
User prefers Go
confidence = 0.95
timestamp = Sep

Resolution:

                    Conflict
                       |
        +--------------+--------------+
        |              |              |
     Recency        Confidence      Explicitness
        |              |              |
        +--------------+--------------+
                       |
                       v
                  Resolver
                       |
                       v
             Current Memory = Go

Important rule:

Explicit user statements should have higher priority than inferred memories.

Example:

"I no longer use Java. I use Go now."

should strongly supersede an older inferred preference.

We can maintain:

{
  "status": "superseded",
  "superseded_by": "mem456"
}

instead of deleting the old memory.

20. Eviction Policies

Different memory types need different policies.

Short-term

Use TTL:

Session memory -> 24 hours / 7 days

depending on product requirements.

Episodic

Use importance + recency:

Low importance + old + never accessed
                    ↓
                  Evict

Semantic

Semantic memories should be much harder to evict.

Evict when:

low confidence
contradicted
obsolete
never useful
user explicitly deletes it

21. Memory Lifecycle

The complete lifecycle becomes:

                Conversation
                     |
                     v
              Short-Term Memory
                     |
                     v
                   Kafka
                     |
                     v
              Memory Extraction
                     |
                     v
             Candidate Memories
                     |
          +----------+----------+
          |                     |
          v                     v
      Episodic              Semantic
          |                     |
          +----------+----------+
                     |
                     v
              Deduplication
                     |
                     v
             Conflict Resolution
                     |
                     v
                Persistence
                     |
                     v
                Embeddings
                     |
                     v
              Vector Retrieval
                     |
                     v
                 Ranking
                     |
                     v
               LLM Context

22. Personalization Layer

Before invoking the LLM:

User Query
    |
    v
Memory Retrieval
    |
    +---- Preferences
    +---- Relevant facts
    +---- Recent episodes
    +---- Current session
    |
    v
Personalization Context
    |
    v
LLM

For example:

User asks:
"Design a system for X."

Retrieved memories:

- User prefers Go
- User is preparing for system design interviews
- User prefers FR/NFR/API/DB/microservices format

The LLM can then produce the response using those preferences without the user repeating them.

23. Important Design Principle

Don’t put all memory into the LLM prompt.

Instead:

                 Memory Store
                      |
                 Retrieval
                      |
                 Relevance
                      |
                 Compression
                      |
                   Top-K
                      |
                      v
              Small Context
                      |
                      v
                     LLM

This controls:

• token cost

• latency

• irrelevant context

• privacy exposure

• hallucination risk

24. End-to-End Request Flow

For a new user request:

1. User sends request
          |
2. Gateway authenticates user
          |
3. Fetch short-term context from Redis
          |
4. Generate query embedding
          |
5. Search semantic + episodic memories
          |
6. Apply user/tenant filters
          |
7. Rank memories
          |
8. Compress selected memories
          |
9. Build LLM context
          |
10. Call LLM
          |
11. Return response
          |
12. Publish conversation event to Kafka
          |
13. Async extraction worker processes it
          |
14. New memories are created/updated

The key architectural decision is that memory extraction doesn’t block the user response.

                 User Request
                      |
                      v
                 Memory Read
                      |
                      v
                     LLM
                      |
                      v
                  Response
                      |
                      |
              asynchronous
                      |
                      v
                    Kafka
                      |
                      v
              Memory Processing

That keeps the online path fast while extraction, summarization, consolidation, and embedding generation happen asynchronously.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

Yes. Let’s start from scratch and structure it exactly like an AI Engineer / SDE-2 HLD interview answer.

The system should support:

• Short-term memory

• Long-term memory

• Episodic memory

• Semantic memory

• Memory extraction

• Memory retrieval

• Summarization

• Conflict resolution

• Memory eviction

• Personalization

Example:

Conversation 1:

User: I prefer Go for backend development.
Agent: Got it.

Conversation 2, several days later:

User: Suggest a backend project for me.

Agent:
Since you prefer Go and are interested in backend systems,
you could build a distributed rate limiter.

The agent needs to retrieve the relevant memory before generating the response.

2. Functional Requirements

1. Store the current conversation as short-term memory.

2. Extract important information from conversations and convert it into long-term memories.

3. Support different memory types:

   • Semantic

   • Episodic

   • Short-term

   • Long-term

4. Retrieve relevant memories based on the current user query.

5. Support semantic/embedding-based memory search.

6. Update an existing memory when the user provides newer information.

7. Detect and resolve conflicting memories.

8. Summarize and consolidate multiple related memories.

9. Automatically expire or evict low-value/stale memories.

10. Allow users to explicitly view, update, or delete their memories.

3. Non-Functional Requirements

1. Low latency

   • Memory retrieval should add less than ~100 ms to the normal request path.

2. High availability

   • Target 99.99% availability for memory retrieval.

3. Scalability

   • Support millions of users and billions of memories.

4. Durability

   • Long-term memories should survive node/service failures.

5. Consistency

   • Memory updates must be durable.

   • Vector indexes can be eventually consistent.

6. Security

   • Strict user and tenant isolation.

   • Encryption at rest and in transit.

7. Privacy

   • Memory must never leak between users/tenants.

8. Cost efficiency

   • Don’t store every conversation permanently.

   • Minimize embedding and LLM calls.

9. Fault tolerance

   • Failure of embedding/vector services shouldn’t bring down the main agent.

10. Observability

• Track retrieval latency, hit rate, memory quality, conflicts, failures, and storage usage.

4. Capacity Estimation

Let’s assume:

10M users
100 long-term memories/user

Therefore:

10M × 100
= 1 Billion memories

Assume each memory contains approximately:

1 KB metadata + text

Raw storage:

1B × 1 KB
≈ 1 TB

Now consider embeddings.

For a 1536-dimensional FP32 vector:

1536 × 4 bytes
= 6 KB/vector

Therefore:

1B × 6 KB
≈ 6 TB

This doesn’t include:

Vector index
Replication
Metadata
Backups
Storage overhead

So we are looking at multi-terabyte distributed storage.

5. High-Level Architecture

                         ┌──────────────┐
                         │    Client    │
                         └──────┬───────┘
                                │
                                ▼
                         ┌──────────────┐
                         │  AI Agent    │
                         └──────┬───────┘
                                │
                                ▼
                      ┌──────────────────┐
                      │  Memory Manager  │
                      └────────┬─────────┘
                               │
              ┌────────────────┼─────────────────┐
              │                │                 │
              ▼                ▼                 ▼
           Redis          Vector Database    PostgreSQL
        Short-term        Semantic Search    Source of Truth
           Memory
              │                │                 │
              └────────────────┼─────────────────┘
                               │
                               ▼
                       Context Builder
                               │
                               ▼
                              LLM
                               │
                               ▼
                            Response


              ASYNCHRONOUS MEMORY PIPELINE

                         Conversation
                              │
                              ▼
                       Kafka / Queue
                              │
               ┌──────────────┼──────────────┐
               ▼              ▼              ▼
          Extraction     Summarization   Embedding
            Worker          Worker         Worker
               │              │              │
               └──────────────┼──────────────┘
                              ▼
                       Memory Storage

6. Components

6.1 Memory Manager

This is the main component responsible for memory operations.

It decides:

Should I retrieve memory?
Should I write memory?
Which memories should be retrieved?
Which memories should be updated?

6.2 Short-Term Memory

Stores the current conversation.

Example:

User:
Design a Kafka system.

Agent:
Sure...

User:
Now explain partitioning.

The agent needs the previous messages to understand what “partitioning” refers to.

Redis is a good choice:

Redis
  │
  ├── conversation_id
  ├── recent messages
  └── TTL

Short-term memory can expire after:

hours / days

depending on the product.

7. Long-Term Memory

Important information gets promoted from short-term memory to long-term memory.

Example:

User:
I usually work with Go and DynamoDB.

Memory extraction produces:

{
  "type": "semantic",
  "content": "User works primarily with Go and DynamoDB",
  "importance": 0.9,
  "confidence": 0.95
}

This memory survives beyond the current conversation.

8. Episodic vs Semantic Memory

This distinction is important in an interview.

Episodic

Represents events that happened.

User discussed Kafka exactly-once semantics
during conversation conv_123.

Semantic

Represents general knowledge about the user.

User is interested in distributed systems.

So:

Episodic = What happened?

Semantic = What do we know?

9. Memory Write Flow

User Conversation
       │
       ▼
Short-Term Memory
       │
       ▼
Memory Extraction
       │
       ▼
┌─────────────────────┐
│ Is this worth       │
│ remembering?        │
└─────────┬───────────┘
          │
     ┌────┴────┐
     │         │
    No        Yes
     │         │
 Discard       ▼
          Importance
          + Confidence
               │
               ▼
          PostgreSQL
               │
               ▼
             Kafka
               │
               ▼
       Embedding Worker
               │
               ▼
          Vector DB

The important design decision is:

We should not store every message as long-term memory.

Otherwise the memory database becomes noisy and expensive.

10. Memory Extraction

We can use an LLM or a specialized extraction model.

Input:

Conversation:

User: I'm preparing for backend interviews.
User: I mainly use Go.
User: I don't have much experience with Kubernetes.

Output:

[
  {
    "type": "goal",
    "content": "User is preparing for backend interviews"
  },
  {
    "type": "preference",
    "content": "User primarily uses Go"
  },
  {
    "type": "knowledge_gap",
    "content": "User has limited Kubernetes experience"
  }
]

Then assign:

importance
confidence
TTL

11. Memory Retrieval

When a new query arrives:

User Query
    │
    ▼
Query Embedding
    │
    ▼
Vector DB
    │
    ▼
Top 20 memories
    │
    ▼
Metadata Filtering
    │
    ▼
Reranking
    │
    ▼
Top 3-5 memories
    │
    ▼
Context Builder
    │
    ▼
LLM

We shouldn’t send 100 memories to the LLM.

Only the most relevant memories should be included.

12. Retrieval Ranking

Vector similarity alone isn’t sufficient.

Use:

score =
    α × semantic_similarity
  + β × importance
  + γ × recency
  + δ × frequency
  + ε × confidence

For example:

Memory A

similarity = 0.91
importance = 0.95
recency = 0.80


Memory B

similarity = 0.95
importance = 0.30
recency = 0.20

Memory A may be more useful despite slightly lower semantic similarity.

13. Database Design

PostgreSQL

Acts as the source of truth.

Stores:

memory_id
user_id
tenant_id
type
content
importance
confidence
version
status
source
created_at
updated_at
last_accessed_at
expires_at

Example:

{
  "memory_id": "mem_123",
  "user_id": "user_456",
  "type": "semantic",
  "content": "User prefers Go",
  "importance": 0.9,
  "confidence": 0.95,
  "version": 3,
  "status": "active"
}

14. Vector Database

Stores:

memory_id
embedding
tenant_id
user_id
metadata

Used for:

semantic similarity search

Possible technologies:

Qdrant
Milvus
Weaviate
Pinecone
OpenSearch

The important architectural point:

PostgreSQL = Source of Truth

Vector DB = Search Index

If the vector database is lost, we can regenerate the vectors from PostgreSQL.

15. Redis

Use Redis for:

Current conversation
Session state
Recent memories
Frequently accessed memories
Retrieval cache

Example:

conversation:user_123
        ↓
recent 20 messages
        ↓
TTL = 24 hours

16. Kafka

Kafka is useful for asynchronous processing.

Events:

MemoryCreated
MemoryUpdated
MemoryDeleted
ConversationCompleted
EmbeddingRequired
MemoryConsolidationRequired

Example:

PostgreSQL
     │
     ▼
 MemoryCreated
     │
     ▼
   Kafka
     │
     ▼
Embedding Worker
     │
     ▼
Vector DB

This prevents embedding generation from blocking the user request.

17. Memory Update & Conflict Resolution

Suppose the system remembers:

v1:
User prefers Java

Later:

User:
I've switched to Go.

Create:

v2:
User prefers Go

Mark:

v1 → superseded
v2 → active

Instead of simply overwriting the old record, maintain versions.

18. Conflict Resolution

There are different types of conflicts.

Explicit correction

"I no longer use Java."

Strong signal → update memory.

Contextual difference

"I use Java at work."
"I use Go for personal projects."

Not a conflict.

Store:

Java → work
Go → personal

Uncertain information

If confidence is low:

confidence = 0.4

don’t immediately overwrite a high-confidence memory.

19. Memory Consolidation

Over time we might have:

"I like Go."

"I mostly use Go."

"Go is my preferred language."

"I use Go for backend."

These can be consolidated into:

User prefers Go for backend development.

Flow:

Related Memories
       │
       ▼
Consolidation Worker
       │
       ▼
Canonical Memory
       │
       ▼
Old Memories → Archived

This reduces storage and improves retrieval quality.

20. Memory Eviction

Some memories should disappear.

For example:

"I'm working on this temporary project this week."

Could have:

TTL = 7 days

While:

"I prefer Go."

could have no TTL.

Eviction can consider:

importance
recency
access frequency
confidence
TTL

Low-value stale memories are removed first.

21. API Design

Create Memory

POST /v1/memories

{
  "user_id": "user_123",
  "type": "semantic",
  "content": "User prefers Go",
  "importance": 0.9,
  "confidence": 0.95
}

Response:

{
  "memory_id": "mem_123",
  "status": "created"
}

Search Memory

POST /v1/memories/search

{
  "user_id": "user_123",
  "query": "What language does the user prefer?",
  "top_k": 5
}

Response:

{
  "memories": [
    {
      "memory_id": "mem_123",
      "type": "semantic",
      "content": "User prefers Go",
      "score": 0.94
    }
  ]
}

Update Memory

PUT /v1/memories/{memory_id}

Delete Memory

DELETE /v1/memories/{memory_id}

List User Memories

GET /v1/users/{user_id}/memories

22. Complete Read Path

                    User Query
                        │
                        ▼
                  ┌────────────┐
                  │ AI Agent   │
                  └─────┬──────┘
                        │
                        ▼
                 Memory Manager
                        │
                        ▼
                 Query Embedding
                        │
                        ▼
                   Vector DB
                        │
                   Top 20
                        │
                        ▼
                Metadata Filter
                        │
                        ▼
                   Reranker
                        │
                      Top 5
                        │
                        ▼
                Context Builder
                        │
                        ▼
                       LLM
                        │
                        ▼
                     Answer

23. Complete Write Path

                  Conversation
                       │
                       ▼
                Short-Term Memory
                       │
                       ▼
                Memory Extraction
                       │
                       ▼
              Importance/Confidence
                       │
                       ▼
                  PostgreSQL
                       │
                       ▼
                     Kafka
                       │
                       ▼
               Embedding Worker
                       │
                       ▼
                  Vector DB

24. Multi-Tenant Isolation

Every memory should contain:

tenant_id
user_id
application_id

Vector search should always apply these filters.

For example:

tenant_id = tenant_123
AND user_id = user_456

This is critical because a semantic search system must never return another user’s memory.

25. Failure Handling

Vector DB is down

Vector DB
    ↓
Unavailable
    ↓
Fallback to PostgreSQL

For important exact memories, PostgreSQL can still be queried.

Embedding Service is down

Don’t fail the user request.

Instead:

Memory Update
     ↓
Kafka
     ↓
Retry
     ↓
Embedding Worker
     ↓
Vector DB

Kafka is temporarily unavailable

Use a transactional/outbox pattern so memory events aren’t lost.

PostgreSQL
    │
    ├── Memory
    └── Outbox Event
             │
             ▼
           Kafka

26. Observability

Track:

memory_retrieval_latency
memory_search_hit_rate
memory_write_rate
memory_extraction_rate
memory_conflict_rate
memory_eviction_rate
memory_consolidation_rate

Also track AI-specific metrics:

retrieval_precision
retrieval_recall
memory_context_tokens
memory_token_savings
memory-induced hallucination rate

27. Final Architecture

                              CLIENT
                                │
                                ▼
                           ┌─────────┐
                           │ AI Agent│
                           └────┬────┘
                                │
                                ▼
                      ┌──────────────────┐
                      │ Memory Manager   │
                      └────────┬─────────┘
                               │
             ┌─────────────────┼─────────────────┐
             │                 │                 │
             ▼                 ▼                 ▼
          Redis            PostgreSQL        Vector DB
      Short-term          Source of Truth   Semantic Search
             │                 │                 │
             └─────────────────┼─────────────────┘
                               │
                               ▼
                       Context Builder
                               │
                               ▼
                              LLM
                               │
                               ▼
                            Response


                  ASYNC MEMORY PIPELINE
                               │
                               ▼
                             Kafka
                               │
             ┌─────────────────┼─────────────────┐
             ▼                 ▼                 ▼
        Extraction        Consolidation      Embedding
          Worker             Worker            Worker
             │                 │                 │
             └─────────────────┼─────────────────┘
                               │
                               ▼
                          Vector DB

Core architecture to remember:

Redis       → Short-term memory
PostgreSQL  → Durable source of truth
Vector DB   → Semantic retrieval
Kafka       → Async memory pipeline
LLM         → Extraction / summarization / reasoning

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

1. Functional Requirements

1. Unified LLM API

   • Provide a common API interface for all LLM providers.

   • Client should not need to know whether the request goes to OpenAI, Anthropic, Gemini, or an open-source model.

2. Intelligent Model Routing

   • Route requests based on:

     • Model capability

     • Latency

     • Cost

     • Provider availability

     • Context-window requirements

     • Tenant preferences

   • Example: route simple queries to a cheaper model and complex queries to a stronger model.

3. Fallback

   • If the selected provider/model fails, automatically route to another eligible model/provider.

   • Example:

OpenAI → Anthropic → Gemini → Self-hosted

1. Rate Limiting & Quotas

   • Support limits at:

     • Tenant/user level

     • API-key level

     • Provider level

     • Model level

   • Support both RPM and TPM limits.

2. Cost-Aware Scheduling

   • Track input/output tokens and model prices.

   • Prefer the cheapest model satisfying the request’s latency and quality requirements.

   • Enforce tenant-level budgets.

3. Retries

   • Retry transient failures such as:

     • HTTP 429

     • 500/502/503

     • Connection timeout

   • Use exponential backoff with jitter.

   • Avoid retrying permanent errors such as invalid API keys or malformed requests.

4. Health Monitoring

   • Continuously monitor:

     • Error rate

     • Latency

     • Token throughput

     • Rate-limit responses

     • Provider availability

   • Temporarily remove unhealthy models/providers from routing.

2. Non-Functional Requirements

1. High Availability

   • Gateway should have no single point of failure.

   • Target availability: 99.99%+.

2. Low Latency

   • Gateway overhead should ideally be <10–20 ms excluding LLM latency.

   • Support streaming responses.

3. Scalability

   • Horizontally scale gateway instances.

   • Support millions of requests/day and thousands of concurrent streaming requests.

4. Fault Tolerance

   • Provider failures should not bring down the gateway.

   • Support retries, circuit breakers, fallback and graceful degradation.

5. Security

   • Encrypt provider API keys.

   • Authentication and authorization for clients.

   • Tenant isolation.

   • Don’t expose provider credentials to clients.

6. Observability

   • Distributed tracing.

   • Metrics and structured logs.

   • Track latency, cost, tokens, errors and fallback frequency.

7. Cost Efficiency

   • Avoid unnecessary calls to expensive models.

   • Support configurable cost budgets and routing policies.

3. High-Level Architecture

                         ┌──────────────────┐
                         │     Clients      │
                         │ Apps / Services   │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │   API Gateway    │
                         │ Auth + Validation│
                         └────────┬─────────┘
                                  │
                                  ▼
                    ┌──────────────────────────┐
                    │       AI Gateway         │
                    │                          │
                    │ Request Manager           │
                    │ Model Router              │
                    │ Rate Limiter              │
                    │ Cost Manager              │
                    │ Retry / Circuit Breaker   │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────┴────────────┐
                    │                         │
                    ▼                         ▼
             ┌──────────────┐          ┌──────────────┐
             │ Model Router │          │ Health Engine│
             └──────┬───────┘          └──────────────┘
                    │
       ┌────────────┼─────────────┬─────────────┐
       ▼            ▼             ▼             ▼
 ┌──────────┐ ┌───────────┐ ┌──────────┐ ┌─────────────┐
 │ OpenAI   │ │ Anthropic │ │ Gemini   │ │ Open-source │
 │ Adapter  │ │ Adapter   │ │ Adapter  │ │ Adapter     │
 └────┬─────┘ └─────┬─────┘ └────┬─────┘ └──────┬──────┘
      │              │            │               │
      ▼              ▼            ▼               ▼
   OpenAI        Anthropic      Gemini       vLLM/TGI

4. Microservices Architecture

1. API Gateway

Responsibilities:

• Authentication

• API-key validation

• Request validation

• Tenant identification

• Request size limits

• TLS termination

Client
  ↓
API Gateway
  ↓
AI Gateway

2. Request/Inference Service

Responsible for:

• Normalizing requests

• Creating request IDs

• Managing streaming/non-streaming requests

• Sending request to Model Router

• Returning normalized response

3. Model Router

Core component.

It maintains a list of candidate models:

Request
   ↓
Model Router
   ↓
Candidate Models

GPT-X
Claude-X
Gemini-X
Llama-X

Routing score can be:

Score =
    capability_score
  - latency_penalty
  - cost_penalty
  - failure_penalty

Example:

GPT      cost = $$
Claude   cost = $
Gemini   cost = $
Llama    cost = very low

For a simple classification request:

Llama → selected

For complex reasoning:

Claude/GPT → selected

4. Provider Adapter Service

Each provider has a standardized adapter.

                 AI Gateway
                     |
        ┌────────────┼────────────┐
        ▼            ▼            ▼
    OpenAI        Anthropic     Gemini
    Adapter       Adapter       Adapter

Adapters convert the internal format into provider-specific APIs.

Internal:

{
  "model": "best",
  "messages": [],
  "temperature": 0.2
}

OpenAI adapter converts it into OpenAI’s format.

Anthropic adapter converts it into Anthropic’s format.

This prevents provider-specific logic from leaking into the core gateway.

5. Rate Limiter

Use Redis for distributed rate limiting.

Example:

Tenant A
   │
   ├── 100 RPM
   ├── 1M TPM
   └── $100/day

Tenant B
   │
   ├── 1000 RPM
   ├── 10M TPM
   └── $1000/day

Use Token Bucket or Leaky Bucket.

For LLMs, we need both:

RPM = Requests Per Minute
TPM = Tokens Per Minute

Example:

Request contains 10,000 tokens

Check:
RPM quota
+
TPM quota

If either exceeded → 429

6. Cost-Aware Scheduling

Maintain model pricing:

Model       Input       Output
--------------------------------
Model A     $5/M        $15/M
Model B     $1/M        $5/M
Model C     $0.2/M      $0.8/M

For every request:

estimated_cost =
input_tokens × input_price
+
estimated_output_tokens × output_price

Router can use:

Quality requirement
        +
Latency requirement
        +
Cost budget
        ↓
Candidate models
        ↓
Select cheapest suitable model

Example:

User asks:
"Translate this sentence"

Possible models:

GPT      $0.01
Claude   $0.008
Gemini   $0.004
Llama    $0.001

→ Llama selected

For complex reasoning:

Quality requirement = HIGH

Llama ❌
Gemini ⚠️
Claude ✅

→ Claude selected

7. Fallback Strategy

Use priority-based fallback.

Primary
   ↓
OpenAI GPT
   ↓ failure
Anthropic Claude
   ↓ failure
Gemini
   ↓ failure
Self-hosted Llama

But don’t fallback for every error.

Retryable

429
500
502
503
Timeout
Connection reset

Non-retryable

400
Invalid request
Invalid API key
Context too large
Unsupported model

8. Circuit Breaker

Suppose OpenAI starts returning:

70% requests → 503

Health service marks it unhealthy.

             OpenAI
               │
          Circuit OPEN
               │
               X
               
Requests → Anthropic

After a cooldown:

OPEN
 ↓
HALF-OPEN
 ↓
Send few requests
 ↓
Healthy → CLOSED

This prevents sending thousands of failing requests to a broken provider.

9. Retry Strategy

Use exponential backoff + jitter.

Attempt 1 → immediate
Attempt 2 → 100ms
Attempt 3 → 300ms
Attempt 4 → 700ms

With jitter:

delay = base × 2^attempt + random()

Also enforce:

max_retries = 2 or 3

Important: don’t retry blindly, especially for streaming requests or requests that may have already been processed.

Use an idempotency key where supported.

10. Health Monitoring

Create a Health Service.

Monitor every provider/model:

Provider
 ├── Availability
 ├── P50 latency
 ├── P95 latency
 ├── P99 latency
 ├── Error rate
 ├── 429 rate
 ├── Token throughput
 └── Cost

Example:

OpenAI GPT
Error Rate:    2%
P95:           1.2 sec
429 Rate:      0.5%
Status:        HEALTHY

Claude
Error Rate:    35%
P95:           5 sec
Status:        UNHEALTHY

Router then automatically reduces or stops traffic to Claude.

11. API Design

Generate Completion

POST /v1/chat/completions

Request:

{
  "model": "auto",
  "messages": [
    {
      "role": "user",
      "content": "Explain distributed systems"
    }
  ],
  "temperature": 0.2,
  "max_tokens": 1000,
  "stream": true
}

Response:

{
  "request_id": "req_123",
  "model": "claude-x",
  "provider": "anthropic",
  "usage": {
    "input_tokens": 100,
    "output_tokens": 500
  },
  "cost": 0.004
}

For streaming:

Client
  ↓
AI Gateway
  ↓
Provider

token → token → token → token
  ↑
stream back to client

Model Information

GET /v1/models

Response:

{
  "models": [
    {
      "id": "model-a",
      "provider": "openai",
      "context_window": 128000,
      "input_cost": 5,
      "output_cost": 15,
      "healthy": true
    }
  ]
}

Usage

GET /v1/usage

Response:

{
  "tenant_id": "tenant_123",
  "requests": 120000,
  "input_tokens": 50000000,
  "output_tokens": 10000000,
  "cost": 245.50
}

Routing Configuration

PUT /v1/routing/policy

Request:

{
  "strategy": "cost_optimized",
  "fallback": [
    "anthropic",
    "gemini",
    "self-hosted"
  ],
  "max_cost_per_request": 0.05
}

12. Database Choice

Use different databases for different workloads.

| Data                    | Database                 | Reason                          |
| ----------------------- | ------------------------ | ------------------------------- |
| Tenant/configuration    | PostgreSQL               | Strong consistency + relational |
| Model/provider metadata | PostgreSQL               | Configuration data              |
| Rate limiting           | Redis                    | Very low latency                |
| Cost/usage events       | Kafka + ClickHouse       | High-volume analytics           |
| Health metrics          | Prometheus               | Time-series metrics             |
| Logs                    | Elasticsearch/OpenSearch | Log search                      |
| Secrets                 | Vault/KMS                | Secure credential storage       |

13. Important DB Schemas

tenants

tenant_id PK
name
status
monthly_budget
created_at

models

model_id PK
provider
model_name
context_window
input_price
output_price
max_rpm
max_tpm
status

routing_policies

policy_id PK
tenant_id
strategy
primary_model
fallback_models
max_latency
max_cost

usage

request_id PK
tenant_id
model_id
provider
input_tokens
output_tokens
latency
cost
status
created_at

14. Kafka Usage

Kafka can be used for asynchronous events:

LLM Request
     │
     ▼
AI Gateway
     │
     ├──────────────→ Provider
     │
     └──────────────→ Kafka
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
         Cost Service  Analytics  Monitoring

Events:

LLM_REQUEST_COMPLETED
LLM_REQUEST_FAILED
MODEL_FALLBACK
PROVIDER_RATE_LIMITED
MODEL_HEALTH_CHANGED

This keeps the request path fast.

15. Redis Usage

Redis is used for:

1. Rate limiting
2. Provider health state
3. Circuit breaker state
4. Model routing cache
5. Tenant quotas
6. Short-lived request metadata

Example:

rate_limit:{tenant_id}
quota:{tenant_id}
health:{provider}:{model}
circuit:{provider}

16. Complete Request Flow

Client
  │
  ▼
API Gateway
  │
  ├── Authentication
  ├── Authorization
  └── Request validation
  │
  ▼
Rate Limiter
  │
  ├── RPM check
  └── TPM check
  │
  ▼
Request Service
  │
  ▼
Model Router
  │
  ├── Cost
  ├── Quality
  ├── Latency
  ├── Health
  └── Tenant policy
  │
  ▼
Provider Adapter
  │
  ▼
OpenAI / Anthropic / Gemini / vLLM
  │
  ▼
Response
  │
  ├── Token usage
  ├── Cost calculation
  ├── Latency
  └── Provider status
  │
  ├────────→ Kafka → Analytics
  │
  └────────→ Client

17. Key Interview Design Decisions

Why adapters?

To isolate provider-specific APIs and allow adding a new provider without modifying routing logic.

Why Redis for rate limiting?

Rate limiting is on the hot path and requires low-latency atomic operations across multiple gateway instances.

Why Kafka for usage?

Usage events can be extremely high volume. Kafka decouples request processing from analytics, billing and monitoring.

Why PostgreSQL?

Tenant, model and routing configurations require transactions and strong consistency.

Why Prometheus?

Health and latency metrics are naturally time-series data and need efficient aggregation.

How do we avoid one provider outage affecting the entire system?

Health checks
+
Circuit breaker
+
Retries
+
Fallback
+
Multi-provider routing

How do we minimize cost?

Cost-aware router
+
Model capability classification
+
Tenant budgets
+
Token estimation
+
Usage tracking

The core idea is to make the gateway provider-agnostic: clients talk to one API, while the gateway dynamically decides which model, which provider, and whether to retry/fallback, based on cost + quality + latency + health + tenant policy.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

1. Requirements

Functional

• Store and search 10B+ embeddings with configurable dimensions such as 768/1536.

• Support real-time insert, update and delete of vectors.

• Support approximate nearest-neighbor search using HNSW / IVF-PQ.

• Support metadata filtering such as tenant_id, category, timestamp, etc.

• Support top-K similarity search with optional filters.

• Provide replication, backup and disaster recovery.

Non-functional

• Search latency: <100 ms p95 for normal workloads.

• High availability: 99.99%+.

• Horizontally scalable by adding nodes.

• No data loss for acknowledged writes; target RPO ≈ 0 with synchronous/replicated WAL depending on deployment.

• Support tenant isolation and predictable performance under noisy-neighbor workloads.

2. High-Level Architecture

                        ┌──────────────────┐
                        │   Client / Apps  │
                        └────────┬─────────┘
                                 │
                                 ▼
                       ┌───────────────────┐
                       │   API Gateway     │
                       │ Auth / Rate Limit │
                       └─────────┬─────────┘
                                 │
                 ┌───────────────┴────────────────┐
                 ▼                                ▼
        ┌─────────────────┐              ┌─────────────────┐
        │ Query Coordinator│              │  Write Service  │
        └────────┬────────┘              └────────┬────────┘
                 │                                │
                 ▼                                ▼
        ┌─────────────────┐              ┌─────────────────┐
        │ Query Planner   │              │ WAL / Log       │
        │ + Filter Engine │              │ Kafka           │
        └────────┬────────┘              └────────┬────────┘
                 │                                │
                 ▼                                ▼
        ┌──────────────────────────────────────────────────┐
        │              Shard Router / Metadata             │
        └───────────────┬──────────────────────────────────┘
                        │
             ┌──────────┼──────────┐
             ▼          ▼          ▼
          Shard 1    Shard 2     Shard N
          ┌─────┐    ┌─────┐     ┌─────┐
          │Index│    │Index│     │Index│
          │HNSW │    │IVF-PQ│    │HNSW │
          ├─────┤    ├─────┤     ├─────┤
          │Vector│   │Vector│    │Vector│
          │Store │   │Store │    │Store │
          ├─────┤    ├─────┤     ├─────┤
          │Meta  │    │Meta  │    │Meta  │
          └─────┘    └─────┘     └─────┘
             │          │           │
             └──────────┼───────────┘
                        ▼
                Object Storage
               Backups / Snapshots

3. Data Model

A vector record can look like:

Vector {
    vector_id
    tenant_id
    embedding
    dimension
    metadata
    created_at
    updated_at
    version
    deleted
}

Example metadata:

{
  "tenant_id": "tenant_123",
  "category": "electronics",
  "language": "en",
  "timestamp": 1755000000
}

Separate vector and metadata storage is useful because metadata filtering and vector similarity have very different access patterns.

4. Sharding

At 10B vectors, a single machine is impossible.

Assume:

10B vectors
1536 dimensions
FP32 = 4 bytes

Raw vector storage:

10B × 1536 × 4
≈ 61.4 TB

And this excludes:

• HNSW index

• metadata

• replication

• WAL

• operating-system overhead

• temporary indexing memory

With 3× replication, raw vector data alone becomes roughly:

61.4 TB × 3 ≈ 184 TB

Therefore we horizontally shard the dataset.

Sharding strategy

Use consistent hashing / hash-based partitioning:

hash(tenant_id + vector_id) → shard

For very large tenants, use:

hash(vector_id) → shard

rather than putting an entire tenant on one shard.

A logical partition could be:

Partition P1 → Shard 17
Partition P2 → Shard 42
Partition P3 → Shard 83
...

Use virtual shards so that physical nodes can be added or removed without massive data movement.

5. Query Flow

For:

Top K similar vectors
where category = "phone"

the flow is:

Client
  ↓
API Gateway
  ↓
Query Coordinator
  ↓
Metadata / Filter Planner
  ↓
Determine relevant shards
  ↓
Parallel ANN search
  ↓
Shard 1 → top K
Shard 2 → top K
Shard 3 → top K
...
  ↓
Merge candidates
  ↓
Optional exact distance calculation
  ↓
Global Top K
  ↓
Client

If searching 100 shards and requesting K=10, each shard may return:

local_top_k = 50

Then the coordinator merges:

100 × 50 = 5000 candidates

and performs exact distance computation on those candidates.

This gives better recall than directly returning each shard’s top 10.

6. Indexing — HNSW

HNSW creates a graph of vectors.

        A -------- B
       /            \
      C ---- D ------ E
       \             /
        F -----------G

Each vector is connected to nearby vectors.

Search starts from an entry point and traverses the graph toward increasingly similar vectors.

Advantages

• Excellent recall.

• Very low query latency.

• Good for relatively dynamic workloads.

Disadvantages

• High memory consumption.

• Expensive updates.

• Large graph at 10B vectors.

Therefore, we generally don’t maintain one gigantic HNSW graph.

Instead:

Shard
 ├── Segment 1 → HNSW
 ├── Segment 2 → HNSW
 ├── Segment 3 → HNSW
 └── ...

7. Indexing — IVF-PQ

For extremely large datasets, IVF-PQ is attractive because it substantially reduces memory/storage requirements.

IVF

First cluster vectors:

             Vector Space

        C1        C2

              C3

    C4                 C5

During search:

Query
  ↓
Find nearest clusters
  ↓
Search only those clusters

Instead of searching 10B vectors, we may search only vectors belonging to the closest few clusters.

PQ

Product Quantization compresses vectors into compact codes.

For example:

1536-dimensional vector
        ↓
Split into sub-vectors
        ↓
Quantize each sub-vector
        ↓
Compact code

This significantly reduces memory requirements.

HNSW vs IVF-PQ

| Feature   | HNSW             | IVF-PQ           |
| --------- | ---------------- | ---------------- |
| Recall    | Very high        | High, tunable    |
| Latency   | Very low         | Low              |
| Memory    | High             | Much lower       |
| Updates   | Easier           | More complex     |
| 10B scale | Expensive        | More suitable    |
| Best use  | Hot/dynamic data | Massive datasets |

A practical architecture can use HNSW for recent/hot segments and IVF-PQ for large historical segments.

8. Real-Time Updates

This is one of the hardest parts.

Building a huge ANN index after every write is impossible.

Use a mutable + immutable segment architecture.

                    Incoming writes
                          │
                          ▼
                     WAL / Kafka
                          │
                          ▼
                  Mutable Memtable
                          │
                 ┌────────┴────────┐
                 ▼                 ▼
             HNSW index       Metadata index
                 │
                 ▼
           Periodic flush
                 │
                 ▼
          Immutable Segment
                 │
                 ▼
         Background compaction

Write path

Client
  ↓
Write API
  ↓
WAL
  ↓
Replicas
  ↓
Memtable
  ↓
Realtime HNSW

Once a segment reaches a threshold:

Memtable → Immutable segment

Background workers then build/optimize IVF-PQ indexes.

This gives low write latency without continuously rebuilding the entire index.

9. Updates and Deletes

Never physically modify a huge immutable segment immediately.

For an update:

vector_id = 123
version = 7

write:

version = 8

and mark version 7 obsolete.

For delete:

tombstone(vector_id)

During query:

candidate
   ↓
metadata/version check
   ↓
if deleted/old → discard

Later, compaction removes obsolete vectors and tombstones.

This is similar to an LSM-tree style architecture.

10. Metadata Storage

Do not store large metadata blobs inside the ANN index.

Use a separate metadata store:

Vector ID
   │
   ├── ANN index
   │
   └── Metadata Store
          ├── tenant_id
          ├── category
          ├── timestamp
          ├── language
          └── attributes

Depending on requirements:

• RocksDB → local metadata/index state.

• PostgreSQL → transactional metadata with moderate scale.

• Cassandra/ScyllaDB → massive distributed metadata.

• Object storage → immutable historical metadata/segments.

For 10B-scale infrastructure, I would favor a distributed KV store for metadata that needs horizontal scaling.

11. Filtering

Filtering is tricky because:

similarity search
+
metadata filtering

must be performed efficiently.

Example:

Find 20 nearest vectors
WHERE
    tenant_id = 123
    AND category = "car"
    AND language = "en"

Approach 1 — Pre-filter

First identify matching IDs:

Metadata Index
      ↓
1M matching vectors
      ↓
ANN search

Good when filters are highly selective.

Approach 2 — Post-filter

ANN search
   ↓
1000 candidates
   ↓
metadata filter
   ↓
20 results

Good when filter selectivity is low.

Better approach

Use an adaptive planner:

Estimate filter selectivity
        ↓
Highly selective?
   /           \
 Yes            No
 ↓               ↓
Pre-filter     ANN first

For frequently used fields, maintain bitmap/inverted indexes.

12. Replication

Each shard has multiple replicas:

Shard 17

Replica A → Node 1
Replica B → Node 8
Replica C → Node 21

Use 3 replicas across different failure domains.

        Shard Leader
        /     |     \
       /      |      \
 Replica 1  Replica 2  Replica 3
 Zone A      Zone B      Zone C

Writes

Client
  ↓
Leader
  ↓
WAL
  ↓
Replicas
  ↓
Quorum ACK

For stronger durability:

W + R > N

For example:

N = 3
W = 2
R = 2

A leader failure triggers leader election.

13. Read Scaling

ANN queries are generally read-heavy.

Therefore:

               Query Coordinator
                 /    |    \
                /     |     \
             Replica Replica Replica

Queries can be distributed across replicas.

Use:

• replica-aware routing

• load-based routing

• cache-aware routing

• tenant-aware routing

Avoid sending every query to the leader.

14. Hot/Cold Architecture

10B embeddings may not have uniform access patterns.

Use:

HOT
 ↓
RAM + HNSW
 ↓
Warm
 ↓
SSD + IVF
 ↓
COLD
 ↓
Object Storage + IVF-PQ

For example:

Recent 100M vectors → HNSW
Older 9.9B vectors → IVF-PQ

Search both and merge results.

This significantly reduces RAM requirements.

15. Disaster Recovery

Use multiple layers.

WAL

Every mutation goes into durable WAL:

Write
 ↓
WAL
 ↓
Vector Store

Kafka/object storage can retain the mutation stream.

Snapshots

Periodically create:

Shard snapshot
+
ANN index
+
Metadata
+
WAL checkpoint

Store snapshots in durable object storage.

Example:

Object Storage

bucket/
 ├── shard-001/
 │    ├── snapshot-100
 │    ├── snapshot-200
 │    └── WAL-offset
 ├── shard-002/
 └── ...

Recovery

Latest snapshot
      ↓
Restore
      ↓
Replay WAL
      ↓
Rebuild missing indexes
      ↓
Replica synchronization
      ↓
Shard becomes healthy

For regional disaster recovery:

Region A
  ↓ async replication
Region B

For critical workloads, keep a warm standby cluster.

16. Failure Handling

Node failure

Node dies
   ↓
Shard coordinator detects failure
   ↓
Route traffic to replica
   ↓
Create replacement replica
   ↓
Stream missing data

Replica failure

Rebuild from:

healthy replica
+
WAL

Corrupted index

Don’t treat the ANN index as the source of truth.

The source of truth should be:

WAL + durable vector segments

The ANN index is a derived structure.

Therefore:

Corrupted HNSW
      ↓
Delete index
      ↓
Rebuild from vector segments

17. Microservices

API Gateway
   │
   ├── Auth / Rate Limiting
   │
   ▼
Query Service
   │
   ├── Query Planner
   ├── Filter Engine
   └── Shard Router
          │
          ▼
      Vector Nodes
       ├── ANN Engine
       ├── Vector Store
       ├── Metadata Store
       └── WAL Consumer

Write Service
   │
   ├── Validation
   ├── WAL Producer
   └── Partition Router

Index Service
   ├── HNSW Builder
   ├── IVF Builder
   ├── PQ Compressor
   └── Compaction

Metadata Service
   └── Distributed KV Store

Snapshot Service
   ├── Checkpoint
   └── Object Storage

Cluster Manager
   ├── Membership
   ├── Leader Election
   ├── Rebalancing
   └── Health Checks

18. Important APIs

Insert

POST /v1/vectors

{
  "id": "vec_123",
  "tenant_id": "tenant_1",
  "embedding": [0.12, 0.43, "..."],
  "metadata": {
    "category": "car",
    "language": "en"
  }
}

Response:

{
  "id": "vec_123",
  "version": 7,
  "status": "accepted"
}

Search

POST /v1/search

{
  "tenant_id": "tenant_1",
  "query_vector": [0.12, 0.43, "..."],
  "top_k": 20,
  "filter": {
    "category": "car",
    "language": "en"
  }
}

Response:

{
  "results": [
    {
      "id": "vec_892",
      "score": 0.94,
      "metadata": {
        "category": "car"
      }
    }
  ]
}

Delete

DELETE /v1/vectors/{id}

Response:

{
  "id": "vec_123",
  "status": "deleted"
}

19. Database / Storage Choices

Vector data
     → SSD/Object Storage

ANN index
     → Local NVMe SSD + RAM

Metadata
     → Cassandra/ScyllaDB or distributed KV

WAL
     → Kafka

Snapshots
     → S3/GCS/Object Storage

Cluster metadata
     → etcd

The important architectural principle is:

Don’t make the ANN index your source of truth.

Treat vectors + WAL + durable segments as the source of truth and make HNSW/IVF-PQ rebuildable derived indexes.

20. Final Architecture

                         Clients
                            │
                            ▼
                     ┌─────────────┐
                     │ API Gateway │
                     └──────┬──────┘
                            │
                 ┌──────────┴──────────┐
                 ▼                     ▼
          ┌─────────────┐       ┌─────────────┐
          │ Query Svc   │       │ Write Svc   │
          └──────┬──────┘       └──────┬──────┘
                 │                     │
                 ▼                     ▼
          ┌─────────────┐          ┌────────┐
          │Query Planner│          │  WAL   │
          └──────┬──────┘          │ Kafka  │
                 │                 └───┬────┘
                 │                     │
                 └──────────┬──────────┘
                            ▼
                    ┌───────────────┐
                    │ Shard Router  │
                    └───────┬───────┘
                            │
          ┌─────────────────┼─────────────────┐
          ▼                 ▼                 ▼
       Shard 1           Shard 2           Shard N
     ┌─────────┐       ┌─────────┐       ┌─────────┐
     │ HNSW    │       │ IVF-PQ  │       │ HNSW    │
     │ Index   │       │ Index   │       │ Index   │
     ├─────────┤       ├─────────┤       ├─────────┤
     │Vectors  │       │Vectors  │       │Vectors  │
     ├─────────┤       ├─────────┤       ├─────────┤
     │Metadata │       │Metadata │       │Metadata │
     └────┬────┘       └────┬────┘       └────┬────┘
          │                 │                 │
          └─────────────────┼─────────────────┘
                            ▼
                     Object Storage
                    Snapshots / Backup

Key design decisions

1. Virtual sharding enables horizontal scaling to 10B+ vectors.

2. HNSW is used for hot/recent data where low latency and high recall matter.

3. IVF-PQ is used for massive cold datasets to reduce memory/storage.

4. Mutable HNSW + immutable segments enables real-time updates without rebuilding huge indexes.

5. Metadata indexes + adaptive filtering handle vector + structured queries efficiently.

6. 3-way replication across AZs provides high availability.

7. WAL + snapshots + replay provides disaster recovery.

8. ANN indexes are derived data, allowing them to be rebuilt after corruption or failure.

9. Query fan-out + local Top-K + global merge enables distributed ANN search.

10. Hot/warm/cold tiers keep the frequently accessed vectors in RAM while cheaply storing billions of older vectors.

If you’re looking for paid 1:1 mentorship with a strong focus on Low-Level Design (LLD)—especially Multithreading and Concurrency—along with High-Level Design (HLD), Data Structures & Algorithms (DSA), System Design, LLM/Generative AI systems, and research paper discussions, feel free to reach out.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

1. Functional Requirements — 5 points

1. Task Creation

   • User should be able to submit a high-level task.

   • Example: “Build a Python service that analyzes customer transactions.”

2. Task Planning

   • Planner Agent decomposes the task into smaller executable subtasks.

   • Creates dependencies between subtasks.

3. Multi-Agent Collaboration

   • Route subtasks to appropriate agents such as Researcher, Coder, Reviewer, etc.

   • Agents should be able to consume outputs generated by other agents.

4. Workflow Execution

   • Execute independent tasks in parallel and dependent tasks sequentially.

   • Support retries, timeouts, cancellation, and failure handling.

5. Monitoring & Results

   • Track task/agent status, execution history, token usage, latency, failures, and final output.

   • User should be able to query workflow progress and retrieve results.

2. Non-Functional Requirements — 5 points

1. Scalability

   • Support thousands/millions of workflows with dynamically scalable agent workers.

2. Availability

   • Target 99.9%+ availability.

   • Agent/workflow failures should not cause loss of workflow state.

3. Low Latency

   • Task scheduling/routing should ideally add <50–100 ms overhead.

   • Independent agent tasks should execute concurrently.

4. Reliability

   • At-least-once task execution with idempotent workers.

   • Retry transient failures using exponential backoff.

5. Observability & Security

   • Distributed tracing, metrics, logs, token/cost tracking.

   • Tenant-level isolation and authorization for shared memory.

3. High-Level Architecture

                         ┌──────────────────┐
                         │      Client      │
                         │ Web / API / SDK  │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │    API Gateway   │
                         └────────┬─────────┘
                                  │
                                  ▼
                    ┌─────────────────────────┐
                    │   Workflow Service      │
                    │                         │
                    │ Create / Pause / Resume │
                    └───────────┬─────────────┘
                                │
                                ▼
                    ┌─────────────────────────┐
                    │   Planner Agent         │
                    │                         │
                    │ Task → DAG of subtasks │
                    └───────────┬─────────────┘
                                │
                                ▼
                  ┌──────────────────────────────┐
                  │      Task Scheduler          │
                  │                              │
                  │ DAG + dependency management  │
                  └────────────┬─────────────────┘
                               │
                  ┌────────────┼─────────────┐
                  ▼            ▼             ▼
             ┌────────┐   ┌────────┐   ┌────────┐
             │Research│   │ Coder  │   │Reviewer│
             │ Agent  │   │ Agent  │   │ Agent  │
             └───┬────┘   └───┬────┘   └───┬────┘
                 │            │             │
                 └────────────┼─────────────┘
                              ▼
                    ┌────────────────────┐
                    │   Shared Memory    │
                    │                    │
                    │ Context / Artifacts│
                    │ Agent Outputs      │
                    └────────────────────┘

                              │
                              ▼
                    ┌────────────────────┐
                    │   Event Bus        │
                    │ Kafka / SQS        │
                    └────────────────────┘

                              │
                              ▼
                    ┌────────────────────┐
                    │ Observability      │
                    │ Metrics / Logs /   │
                    │ Traces / Cost      │
                    └────────────────────┘

4. Core Workflow

Suppose the user asks:

“Build a REST API for analyzing transactions.”

Planner creates:

                Root Task
                   │
          ┌────────┴────────┐
          ▼                 ▼
   Research APIs       Research DB
          │                 │
          └────────┬────────┘
                   ▼
              Write Code
                   │
                   ▼
             Run Tests
                   │
                   ▼
                Review
                   │
             ┌─────┴─────┐
             │           │
           PASS         FAIL
             │           │
             ▼           ▼
           Done       Fix Code
                         │
                         └──────→ Review

The workflow is represented as a DAG.

Each node contains:

Task {
    task_id
    workflow_id
    agent_type
    input
    dependencies[]
    status
    retry_count
    timeout
}

5. Task Scheduling

The Task Scheduler maintains the workflow DAG.

Task lifecycle

PENDING
   ↓
READY
   ↓
RUNNING
   ↓
SUCCESS

Failure:

RUNNING
   ↓
FAILED
   ↓
RETRY
   ↓
READY

After maximum retries:

FAILED → DEAD_LETTER

Scheduling algorithm

For every completed task:

1. Mark task SUCCESS
2. Find dependent tasks
3. Check whether all dependencies are SUCCESS
4. If yes → mark dependent task READY
5. Push task to appropriate agent queue

Independent tasks can therefore execute concurrently.

6. Task Routing

Use an Agent Registry.

Agent Registry

researcher → researcher_queue
coder      → coder_queue
reviewer   → reviewer_queue
planner    → planner_queue

Scheduler determines:

task.agent_type = CODER

and publishes:

Kafka:
    topic = agent.coder.tasks

Coder workers consume from this topic.

For dynamic routing, maintain agent metadata:

Agent {
    agent_id
    type
    capabilities[]
    current_load
    max_concurrency
    health
}

Example:

"Search latest Kafka documentation"
            ↓
Researcher
            ↓
Researcher-3

Routing can consider:

• capability

• current load

• priority

• model availability

• cost

• region

7. Shared Memory

Agents shouldn’t directly depend on each other’s local memory.

Use a centralized Shared Memory Service.

                     Shared Memory
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
    Task Context       Artifacts        Agent Outputs
        │                 │                 │
    Redis/Postgres    Object Storage    Vector DB

Types of memory

Short-term memory

Current workflow context.

Redis

Durable state

Workflow/task state.

PostgreSQL

Large artifacts

Code, documents, datasets.

S3 / Object Storage

Semantic memory

Research findings, previous knowledge.

Vector DB

Agents access memory using:

GET /workflows/{workflow_id}/context

and:

POST /workflows/{workflow_id}/memory

8. Retry Strategy

Not every failure should be retried.

Retryable

LLM timeout
Network failure
Rate limit
Temporary DB failure
Temporary service unavailable

Non-retryable

Invalid prompt
Invalid tool arguments
Permission denied
Invalid workflow

Use exponential backoff:

retry 1 → 1 sec
retry 2 → 2 sec
retry 3 → 4 sec
retry 4 → 8 sec

with jitter.

Each task has:

max_retries = 3

After that:

Dead Letter Queue

9. Deadlock Prevention

Since workflows are DAG-based, the system should reject cyclic dependencies.

Example invalid workflow:

A → B
B → C
C → A

When creating the DAG, perform cycle detection using:

DFS / Topological Sort

If:

number_of_processed_nodes != total_nodes

then a cycle exists.

Return:

400 INVALID_WORKFLOW

Runtime deadlock prevention

Also use:

• Task timeout

• Workflow timeout

• Lease expiration

• Heartbeats

• Maximum dependency depth

• Maximum task execution time

• Cancellation propagation

Example:

Worker takes task
      ↓
Lease = 5 minutes
      ↓
Worker sends heartbeat
      ↓
Worker crashes
      ↓
Lease expires
      ↓
Scheduler requeues task

10. APIs

Create Workflow

POST /v1/workflows

Request

{
  "task": "Build a REST API for analyzing customer transactions",
  "priority": "HIGH",
  "max_retries": 3,
  "metadata": {
    "language": "python"
  }
}

Response

{
  "workflow_id": "wf_12345",
  "status": "PLANNING",
  "created_at": "2026-08-10T17:00:00Z"
}

Get Workflow Status

GET /v1/workflows/{workflow_id}

Response

{
  "workflow_id": "wf_12345",
  "status": "RUNNING",
  "progress": 65,
  "tasks": [
    {
      "task_id": "t1",
      "agent": "researcher",
      "status": "SUCCESS"
    },
    {
      "task_id": "t2",
      "agent": "coder",
      "status": "RUNNING"
    },
    {
      "task_id": "t3",
      "agent": "reviewer",
      "status": "PENDING"
    }
  ]
}

Get Workflow Result

GET /v1/workflows/{workflow_id}/result

Response

{
  "workflow_id": "wf_12345",
  "status": "COMPLETED",
  "result": {
    "code_artifact": "s3://bucket/code.zip",
    "review": "Approved",
    "summary": "REST API implemented successfully"
  }
}

Cancel Workflow

POST /v1/workflows/{workflow_id}/cancel

Response

{
  "workflow_id": "wf_12345",
  "status": "CANCELLED"
}

Get Shared Memory

GET /v1/workflows/{workflow_id}/memory

Response

{
  "workflow_id": "wf_12345",
  "memory": [
    {
      "task_id": "t1",
      "type": "RESEARCH_RESULT",
      "content": "PostgreSQL is selected as the primary database"
    },
    {
      "task_id": "t2",
      "type": "CODE_ARTIFACT",
      "uri": "s3://bucket/code.zip"
    }
  ]
}

11. Microservices Architecture

1. API Gateway

Responsibilities:

Authentication
Authorization
Rate limiting
Request validation
Routing

2. Workflow Service

Responsible for:

Create workflow
Pause/resume
Cancel workflow
Workflow state

3. Planner Service

Converts:

User Goal
   ↓
Task DAG

Example:

Research → Code → Test → Review

4. Scheduler Service

Responsible for:

DAG execution
Dependency resolution
Task prioritization
Retries
Timeouts
Task leases

5. Agent Router

Routes tasks based on:

Agent type
Capability
Load
Cost
Model
Availability

6. Agent Workers

Separate worker pools:

Researcher Workers
Coder Workers
Reviewer Workers
Planner Workers

Each worker can independently scale.

7. Memory Service

Provides:

Read context
Write context
Semantic search
Artifact references
Versioning

8. Artifact Service

Stores:

Source code
Documents
Research reports
Images
Large outputs

Object storage such as S3 is appropriate.

9. Event Bus

Kafka/SQS handles:

Task created
Task completed
Task failed
Retry task
Workflow completed
Agent events

10. Observability Service

Collect:

Metrics
Logs
Distributed traces
LLM token usage
LLM cost
Agent latency
Task retry count
Workflow success rate

12. Database Choice

| Data                | Database                 | Why                                    |
| ------------------- | ------------------------ | -------------------------------------- |
| Workflow/Task state | PostgreSQL               | Transactions + relational DAG metadata |
| Hot workflow state  | Redis                    | Low-latency reads/writes               |
| Agent queues/events | Kafka                    | Durable asynchronous processing        |
| Large artifacts     | S3/Object Storage        | Cheap, scalable blob storage           |
| Semantic memory     | Vector DB                | Similarity search                      |
| Logs                | ClickHouse/Elasticsearch | High-volume analytics/search           |

PostgreSQL Schema

workflows
---------
workflow_id PK
tenant_id
status
priority
input
created_at
updated_at


tasks
-----
task_id PK
workflow_id FK
agent_type
status
input
output
retry_count
max_retries
timeout
created_at
updated_at


task_dependencies
-----------------
task_id
depends_on_task_id
PRIMARY KEY(task_id, depends_on_task_id)


agent_executions
----------------
execution_id PK
task_id
agent_id
started_at
completed_at
status
error
token_usage
cost

13. Important Design Decision: DB + Kafka

Don’t use Kafka as the source of truth for workflow state.

Use:

PostgreSQL = Source of Truth
Kafka      = Event/Task Transport
Redis      = Cache

For example:

Task completed
     │
     ├── UPDATE PostgreSQL
     │
     └── Publish TaskCompleted event → Kafka

For reliable DB → Kafka publishing, use the Transactional Outbox Pattern:

                    PostgreSQL
                 ┌─────────────┐
                 │ task update │
                 │ outbox event│
                 └──────┬──────┘
                        │
                  Outbox Worker
                        │
                        ▼
                      Kafka

This prevents the situation where the DB update succeeds but the Kafka event is lost.

14. End-to-End Flow

User
 │
 ▼
API Gateway
 │
 ▼
Workflow Service
 │
 ▼
Planner Agent
 │
 │ Creates DAG
 ▼
Task Scheduler
 │
 ├──────────────┐
 ▼              ▼
Researcher     Researcher
 │              │
 └──────┬───────┘
        ▼
      Coder
        │
        ▼
      Tests
        │
        ▼
     Reviewer
        │
    ┌───┴───┐
    │       │
  PASS     FAIL
    │       │
    ▼       ▼
  Done    Coder
            │
            └────→ Reviewer

Every agent interaction is recorded in:

PostgreSQL → workflow/task state
Redis      → hot context
Kafka      → events/tasks
S3        → artifacts
Vector DB  → semantic memory

And monitored through:

OpenTelemetry
      ↓
Metrics + Logs + Traces
      ↓
Grafana / Alerting

Key interview point: The heart of this system is a persistent DAG-based workflow engine + asynchronous task queues + shared memory layer. The agents themselves remain stateless workers, which makes them independently scalable and allows the scheduler to handle retries, failures, timeouts, and recovery centrally.

Multiple applications (chatbots, support assistants, coding assistants, search copilots, etc.) send repeated or similar prompts to LLM providers. Since LLM inference is expensive and introduces latency, we need a Distributed Prompt Cache that:

• Reduces LLM API costs

• Reduces response latency

• Supports multiple applications and tenants

• Handles exact and semantic prompt matching

• Preserves privacy and security

• Scales horizontally

2. Functional Requirements

Core Features

• Cache LLM responses for repeated prompts

• Support exact-match lookup

• Support semantic similarity lookup

• Configurable TTL policies

• Cache invalidation support

• Multi-tenant isolation

• Cache warming/pre-population

• Distributed deployment

Non-Functional Requirements

• High availability (99.9%+)

• Low latency (<20 ms cache lookup)

• Horizontally scalable

• Secure tenant isolation

• Cost-efficient storage

3. High-Level Architecture

                +----------------+
                | Applications   |
                | (Multi-tenant) |
                +-------+--------+
                        |
                        v
           +-------------------------+
           | Prompt Cache Gateway    |
           +-----------+-------------+
                       |
         +-------------+--------------+
         |                            |
         v                            v

+----------------+        +---------------------+
| Exact Cache    |        | Semantic Cache      |
| (Redis)        |        | (Vector DB)         |
+--------+-------+        +----------+----------+
         |                           |
         +------------+--------------+
                      |
                      v
            +------------------+
            | Cache Decision   |
            | Service          |
            +--------+---------+
                     |
          Cache Miss |
                     v
             +---------------+
             | LLM Provider  |
             | (GPT/Claude)  |
             +-------+-------+
                     |
                     v
             +---------------+
             | Cache Writer  |
             +---------------+

4. Request Flow

Exact Match Hit

User Request
      |
      v
Generate Cache Key
      |
      v
Redis Lookup
      |
      +---- HIT ----> Return Response

Cache Miss

User Request
      |
      v
Exact Match Lookup
      |
      +---- MISS
               |
               v
       Semantic Search
               |
         Similarity > Threshold?
             /        \
           Yes        No
            |          |
            v          v
      Return Cached   Call LLM
       Response         |
                         v
                  Store in Cache

5. Cache Key Design

The cache key must uniquely identify prompts that should produce identical responses.

Key Structure

hash(
    tenant_id +
    model_name +
    model_version +
    system_prompt +
    user_prompt +
    temperature +
    top_p
)

Example

tenantA:gpt-4o:v2:
SHA256(normalized_prompt)

Why Include These Fields?

| Field         | Reason                                     |
| ------------- | ------------------------------------------ |
| Tenant ID     | Prevent data leakage                       |
| Model Name    | Different models produce different outputs |
| Model Version | Responses change after upgrades            |
| Temperature   | Affects output randomness                  |
| System Prompt | Changes behavior                           |
| User Prompt   | Main query                                 |

6. Prompt Normalization

Before generating cache keys:

Transformations

" What is Kubernetes? "
"What is Kubernetes?"

Normalize:

• Trim whitespace

• Lowercase (optional)

• Remove repeated spaces

• Canonicalize formatting

Example:

"What  is Kubernetes?"

↓

"what is kubernetes?"

This improves cache hit rates.

7. Semantic Similarity Matching

Exact matching misses logically equivalent prompts.

Examples

Prompt A:

What is Kubernetes?

Prompt B:

Explain Kubernetes.

Both should reuse the same response.

Semantic Cache Architecture

Step 1: Generate Embedding

Prompt
   |
Embedding Model
   |
Vector (1536 dimensions)

Step 2: Store in Vector DB

Possible technologies:

• Pinecone

• Weaviate

• Milvus

• pgvector

• OpenSearch Vector Search

Store:

{
  "tenant":"A",
  "embedding":[...],
  "response":"...",
  "ttl":"..."
}

Lookup

New Prompt
      |
Generate Embedding
      |
Vector Search
      |
Top K Results
      |
Similarity Score

Decision Rule

similarity >= 0.92

→ Cache Hit

Else

→ LLM Call

Similarity Metrics

• Cosine similarity

• Euclidean distance

• Dot product

Cosine similarity is most common.

8. Cache Storage Design

L1 Cache

Redis

Purpose:

• Ultra-fast exact lookup

• Millisecond latency

Stores:

key -> response

L2 Cache

Vector Database

Purpose:

• Semantic retrieval

Stores:

embedding -> response

9. TTL Strategy

Not all prompts age equally.

Dynamic TTL

| Prompt Type  | TTL     |
| ------------ | ------- |
| Math         | 30 days |
| Coding Help  | 7 days  |
| News         | 1 hour  |
| Weather      | 15 mins |
| Stock Prices | 5 mins  |

TTL Formula

TTL =
baseTTL *
accessFrequencyMultiplier *
confidenceMultiplier

Frequently reused prompts stay longer.

10. Cache Invalidation

One of the hardest problems.

Trigger 1: Model Upgrade

GPT-4o-v1
      ↓
GPT-4o-v2

Invalidate entries tied to old version.

Trigger 2: Prompt Template Change

System Prompt Changed

Invalidate matching namespace.

Trigger 3: Knowledge Refresh

Example:

Company policy updated

Invalidate related cache entries.

Trigger 4: Manual Purge

Admin API:

DELETE /cache/{tenant}
DELETE /cache/key/{id}

11. Privacy Isolation

Critical in enterprise environments.

Tenant Isolation

Every cache entry contains:

tenant_id

Lookup only within same tenant.

tenantA ≠ tenantB

No cross-tenant reuse.

Namespace Isolation

tenantA/app1
tenantA/app2
tenantB/app1

Separate cache namespaces.

Encryption

At Rest:

• AES-256

In Transit:

• TLS

Sensitive Data Protection

Before caching:

Prompt
   |
PII Detector
   |
Sensitive?

If sensitive:

Do Not Cache

Examples:

• Credit card numbers

• SSN

• Passwords

• Medical records

12. Cache Warming

Pre-populate cache before traffic arrives.

Method 1: Historical Analysis

Use top prompts from logs.

Top 10,000 prompts
      |
Generate Responses
      |
Populate Cache

Method 2: Product Launch Warmup

Example:

New HR Bot

Preload:

• Leave policy

• Benefits

• Payroll FAQ

Method 3: Scheduled Refresh

Nightly job:

Refresh frequently used entries

before expiry.

13. Scalability Design

Redis Cluster

Redis Shard 1
Redis Shard 2
Redis Shard 3

Partition by:

hash(cache_key)

Vector DB Cluster

Embedding Space
      |
Partition/Sharding
      |
Multiple Nodes

Supports billions of vectors.

14. APIs

Lookup

POST /cache/lookup

Request:

{
  "tenant":"A",
  "prompt":"What is Kubernetes?"
}

Response:

{
  "hit": true,
  "response":"..."
}

Insert

POST /cache/store

Invalidate

DELETE /cache/key/{id}

15. Metrics & Monitoring

Track:

Cache Metrics

Exact Hit Rate
Semantic Hit Rate
Miss Rate

Cost Metrics

Tokens Saved
LLM Calls Avoided
Dollar Savings

Performance Metrics

P95 Latency
P99 Latency
Redis Latency
Vector Search Latency

16. Design Decisions Summary

| Component         | Choice                                 |
| ----------------- | -------------------------------------- |
| L1 Cache          | Redis Cluster                          |
| L2 Semantic Cache | Vector DB (Milvus/Pinecone/pgvector)   |
| Cache Key         | Hash(tenant + model + prompt config)   |
| Similarity Metric | Cosine Similarity                      |
| Threshold         | 0.90–0.95                              |
| TTL               | Dynamic by content type                |
| Invalidation      | Versioning + Event-driven purge        |
| Privacy           | Tenant namespaces + PII filtering      |
| Warming           | Historical prompts + scheduled refresh |

Final Design

A two-level distributed prompt cache consists of:

1. L1 Redis exact-match cache for sub-millisecond retrieval.

2. L2 Vector semantic cache for similar prompts.

3. Tenant-aware cache keys for privacy.

4. Dynamic TTL and invalidation policies to maintain freshness.

5. Cache warming pipelines to maximize hit rates.

6. Horizontal scaling via Redis Cluster and distributed vector databases.

This architecture can typically reduce 30–70% of LLM calls, significantly lowering inference costs and improving response latency across multiple applications.

If you’re looking for paid 1:1 mentorship with a strong focus on Low-Level Design (LLD)—especially Multithreading and Concurrency—along with High-Level Design (HLD), Data Structures & Algorithms (DSA), System Design, LLM/Generative AI systems, and research paper discussions, feel free to reach out.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

Functional Requirements (FR)

• Users should be able to interact with the AI agent using voice in real time.

• System should convert speech to text (ASR), generate responses using an LLM, and convert responses back to speech (TTS).

• System should support full-duplex conversations where users can interrupt the AI while it is speaking (barge-in).

• System should maintain conversational context and session history across multiple turns.

• System should support multilingual conversations and voice personalization.

Non-Functional Requirements (NFR)

• Support up to 10 million concurrent users globally.

• End-to-end voice response latency should be less than 300 ms.

• Achieve 99.99% availability with no single point of failure.

• Scale horizontally across regions with automatic failover.

• Optimize GPU utilization to maximize throughput and minimize inference costs.

Capacity Estimation

Assumptions:

• 10M concurrent users.

• Average audio packet every 100 ms.

• Average audio chunk size = 3 KB.

Incoming traffic:

10M × 10 chunks/sec
=
100M audio chunks/sec

Bandwidth:

100M × 3 KB
=
300 GB/sec

This requires a globally distributed architecture with edge processing.

Core APIs

Start Session

Request

POST /v1/session/start

{
  "userId": "u123",
  "language": "en",
  "voice": "female_1"
}

Response

{
  "sessionId": "sess_789",
  "websocketUrl": "wss://voice.company.com/session/sess_789"
}

Streaming Audio API

WebSocket

wss://voice.company.com/session/{sessionId}

Client continuously sends audio chunks.

{
  "sequence": 1,
  "audioChunk": "<binary>"
}

Session End API

POST /v1/session/end

{
  "sessionId": "sess_789"
}

Health API

GET /v1/health

High-Level Components / Microservices

• API Gateway

• WebSocket Gateway

• Session Management Service

• Streaming ASR Service

• Conversation Orchestrator

• Context/Memory Service

• LLM Inference Service

• TTS Service

• Barge-In Detection Service

• GPU Scheduler

• Model Serving Platform

• Redis Session Cache

• Kafka/Pulsar Event Bus

• Monitoring & Analytics Service

Database Schema

Session Table

CREATE TABLE sessions (
    session_id VARCHAR(128) PRIMARY KEY,
    user_id VARCHAR(128),
    status VARCHAR(32),
    started_at TIMESTAMP,
    last_activity TIMESTAMP
);

Conversation History

CREATE TABLE conversations (
    id BIGSERIAL PRIMARY KEY,
    session_id VARCHAR(128),
    role VARCHAR(16),
    message TEXT,
    created_at TIMESTAMP
);

User Voice Preferences

CREATE TABLE voice_profiles (
    user_id VARCHAR(128) PRIMARY KEY,
    language VARCHAR(32),
    preferred_voice VARCHAR(64)
);

Microservices Interaction

Step 1: User Starts Conversation

User opens application.

Request reaches API Gateway.

API Gateway forwards request to Session Service.

Session Service:

• Creates session.

• Allocates region.

• Stores metadata in Redis.

Returns WebSocket endpoint.

Step 2: User Starts Speaking

Audio stream arrives through WebSocket Gateway.

Gateway forwards chunks to Streaming ASR Service.

Example:

User:
"Book me a flight to London"

Audio is streamed continuously.

Step 3: Streaming ASR

Instead of waiting for complete speech:

Traditional ASR:

User speaks
     ↓
Speech ends
     ↓
ASR starts

High latency.

Streaming ASR:

Audio Chunk 1
Audio Chunk 2
Audio Chunk 3

ASR continuously emits partial transcripts.

Example:

"Book"
"Book me"
"Book me a flight"
"Book me a flight to London"

This saves hundreds of milliseconds.

Step 4: Conversation Orchestrator

Receives partial transcript.

Maintains conversation state.

Fetches:

• User profile

• Conversation memory

• Previous messages

Builds prompt.

Step 5: LLM Inference Service

Prompt sent to LLM.

Example:

User:
Book me a flight to London

Context:
User prefers economy seats.

LLM begins generating tokens immediately.

Streaming output:

Sure
Sure, I
Sure, I can
Sure, I can help

No need to wait for complete answer.

Step 6: TTS Service

As soon as first tokens arrive:

Sure,

TTS starts synthesis.

Produces audio stream immediately.

Pipeline:

LLM Tokens
    ↓
TTS
    ↓
Audio Chunks
    ↓
User

This overlap is critical for achieving <300 ms latency.

End-to-End Streaming Pipeline

User Speech
      ↓
Streaming ASR
      ↓
Partial Transcript
      ↓
LLM Streaming
      ↓
Partial Tokens
      ↓
Streaming TTS
      ↓
Audio Response

All stages run simultaneously.

Barge-In (Interruption Handling)

Problem

AI is speaking:

AI:
"Sure, I can help you book..."

User interrupts:

"No, I meant Paris."

System must stop speaking immediately.

Barge-In Flow

User audio detected.

Barge-In Service receives voice activity event.

User Voice Detected
       ↓
Stop Current TTS
       ↓
Cancel Current LLM Generation
       ↓
Start New ASR Stream

Conversation becomes:

AI:
Sure I can help...

User:
No, I meant Paris.

AI:
Understood. You want Paris.

Session Management

Session Service maintains:

• Session ID

• User preferences

• Current conversation state

• Active model

• Current TTS stream

• Current ASR stream

Hot session state stored in Redis.

Session
      ↓
Redis

Access latency:

< 5 ms

Long-term history stored in distributed database.

Autoscaling Strategy

ASR Autoscaling

Scale based on:

Active Audio Streams

Metric:

Streams per GPU

LLM Autoscaling

Scale based on:

Waiting Requests
Token Throughput
GPU Queue Length

TTS Autoscaling

Scale based on:

Concurrent Audio Synthesis Requests

Kubernetes HPA scales services independently.

GPU Utilization Optimization

Running one request per GPU wastes resources.

Bad:

GPU
 └─ Request A

Utilization:

15%

Use Continuous Batching.

Good:

GPU
 ├─ Req A
 ├─ Req B
 ├─ Req C
 ├─ Req D
 └─ Req E

Utilization:

80-95%

GPU Scheduler

Responsibilities:

• Assign requests to GPUs.

• Perform dynamic batching.

• Route to least loaded GPU.

• Handle model placement.

• Manage KV cache memory.

Latency Breakdown (<300 ms Target)

| Component       | Latency |
| --------------- | ------- |
| Network         | 20 ms   |
| Streaming ASR   | 50 ms   |
| Prompt Building | 10 ms   |
| LLM First Token | 80 ms   |
| Streaming TTS   | 80 ms   |
| Buffering       | 20 ms   |

Total:

≈ 260 ms

High-Level Architecture

Client
   |
WebSocket Gateway
   |
Session Service
   |
Conversation Orchestrator
   |
   +----------------+
   |                |
Streaming ASR   Memory Service
   |                |
   +-------+--------+
           |
           v
      LLM Service
           |
           v
       TTS Service
           |
           v
      Audio Stream
           |
         Client

Key Design Decisions

• Streaming ASR instead of batch ASR.

• Streaming token generation from LLM.

• Streaming TTS synthesis.

• Redis-backed session state.

• Immediate barge-in cancellation.

• Dynamic GPU batching.

• Regional deployment with edge WebSocket gateways.

• Stateless services with Kubernetes autoscaling.

• Event-driven communication via Kafka/Pulsar.

• Continuous GPU utilization monitoring and scheduling.

• Users should be able to submit natural language questions to the RAG system.

• System should retrieve the most relevant documents/chunks from the knowledge base.

• System should generate answers using retrieved context and an LLM.

• System should cache query embeddings to avoid repeated embedding generation.

• System should cache retrieval results to avoid repeated vector searches.

• System should cache final LLM responses for repeated or semantically similar queries.

• System should support semantic cache lookup based on embedding similarity.

• System should allow ingestion, update, and deletion of knowledge base documents.

• System should invalidate stale cache entries when underlying documents change.

• System should provide source citations/metadata used for answer generation.

• System should support multi-tenant isolation (different organizations/knowledge bases).

• System should expose cache hit/miss metrics for monitoring and analytics.

Non-Functional Requirements (NFR)

• Support 100M+ queries per day.

• Handle peak traffic of 10K–50K QPS.

• Achieve P95 query latency below 500 ms for cached requests.

• Achieve P99 latency below 2 seconds for uncached requests.

• Maintain 99.99% service availability.

• Scale horizontally without downtime.

• Ensure fault tolerance with no single point of failure.

• Support eventual consistency for distributed cache invalidation.

• Minimize LLM and vector search costs through high cache hit ratios.

• Maintain cache freshness after KB updates.

• Support multi-region deployments with local cache access.

• Ensure tenant-level security and data isolation.

• Enable observability through metrics, logs, and distributed tracing.

• Support rolling deployments and zero-downtime upgrades.

• Provide disaster recovery and backup mechanisms.

Core APIs

Query API

POST /v1/query

Request

{
  "query": "What is GPT-5?",
  "tenantId": "org123",
  "userId": "user456"
}

Response

{
  "answer": "GPT-5 is...",
  "sources": ["doc1", "doc2"],
  "cacheHit": "response_cache",
  "latencyMs": 45
}

Document Ingestion API

POST /v1/documents

Request

{
  "documentId": "doc123",
  "content": "document content",
  "metadata": {
    "category": "AI"
  }
}

Response

{
  "status": "success"
}

Document Update API

PUT /v1/documents/{documentId}

Request

{
  "content": "updated content",
  "version": 5
}

Response

{
  "status": "updated"
}

Cache Invalidation API

POST /v1/cache/invalidate

Request

{
  "documentId": "doc123"
}

Response

{
  "status": "invalidated"
}

Database Schema

Documents

CREATE TABLE documents (
    document_id VARCHAR(64) PRIMARY KEY,
    content TEXT,
    version BIGINT,
    metadata JSONB,
    updated_at TIMESTAMP
);

Embeddings

CREATE TABLE embeddings (
    embedding_id BIGSERIAL PRIMARY KEY,
    document_id VARCHAR(64),
    embedding VECTOR(1536),
    version BIGINT
);

Response Cache Metadata

CREATE TABLE response_cache (
    cache_key VARCHAR(255) PRIMARY KEY,
    answer TEXT,
    kb_version BIGINT,
    created_at TIMESTAMP,
    expires_at TIMESTAMP
);

Retrieval Cache Metadata

CREATE TABLE retrieval_cache (
    query_hash VARCHAR(255) PRIMARY KEY,
    document_ids JSONB,
    kb_version BIGINT,
    created_at TIMESTAMP
);

Microservices

• API Gateway

• Query Orchestrator Service

• Embedding Service

• Embedding Cache Service

• Retrieval Cache Service

• Vector Search Service

• Response Cache Service

• LLM Service

• Knowledge Base Service

• Cache Invalidation Service

• Metadata Service

• Monitoring & Analytics Service

• Kafka/Pulsar Event Bus

High-Level Request Flow

Client
   |
API Gateway
   |
Query Orchestrator
   |
   +--> Response Cache
   |        |
   |        +--> Hit --> Return Answer
   |
   +--> Embedding Cache
   |        |
   |        +--> Miss --> Embedding Service
   |
   +--> Retrieval Cache
   |        |
   |        +--> Miss --> Vector Search
   |
   +--> LLM Service
   |
Return Answer

Microservices Interaction (Request Flow)

1. Client → API Gateway

• User sends a query such as “What is GPT-5?”

• API Gateway performs authentication, rate limiting, request validation, and routing.

• Request is forwarded to the Query Orchestrator Service.

2. Query Orchestrator → Response Cache Service

• Query Orchestrator first checks whether the final answer is already available in the Response Cache.

• Cache lookup can happen in two ways:

  • Exact match lookup using query hash.

  • Semantic lookup using query embedding similarity.

• If a valid cache entry exists, the response is returned immediately.

• This is the fastest path and avoids all downstream processing.

Example

"What is GPT-5?"

already exists in cache.

Response is returned in ~10-50 ms.

3. Query Orchestrator → Embedding Cache Service

If response cache misses:

• System needs query embedding.

• Query Orchestrator checks Embedding Cache.

Cache Hit

• Cached embedding vector is returned.

Cache Miss

• Request goes to Embedding Service.

• Embedding model generates vector.

• Generated vector is stored in Embedding Cache for future use.

Example:

"What is GPT-5?"

↓

[0.12, 0.45, 0.98 ...]

4. Query Orchestrator → Retrieval Cache Service

Once query embedding is available:

• System checks Retrieval Cache.

• Retrieval Cache stores previously retrieved document IDs for similar queries.

Cache Hit

Query Embedding
       ↓
[Doc1, Doc7, Doc9]

Documents are returned directly.

Cache Miss

System performs vector search.

5. Query Orchestrator → Vector Search Service

• Vector Search Service receives query embedding.

• Searches Vector Database using Approximate Nearest Neighbor (ANN) algorithms such as HNSW or IVF.

• Retrieves Top-K relevant chunks/documents.

Example:

Query:
"What is GPT-5?"

Retrieved:

Doc12
Doc19
Doc42

Results are stored in Retrieval Cache.

6. Query Orchestrator → Knowledge Base Service

• Query Orchestrator requests full content for retrieved document IDs.

• KB Service fetches chunk text and metadata.

Example:

Doc12 → GPT-5 architecture
Doc19 → GPT-5 capabilities
Doc42 → GPT-5 release information

These chunks form the context.

7. Query Orchestrator → LLM Service

• Query and retrieved context are sent to the LLM Service.

• LLM Service constructs prompt.

Example:

System Prompt

Retrieved Context:
[Doc12]
[Doc19]
[Doc42]

User Query:
What is GPT-5?

• LLM generates final answer.

8. Query Orchestrator → Response Cache Service

Before returning answer:

• Generated response is stored in Response Cache.

• Metadata is stored:

  • Query hash

  • Embedding hash

  • Retrieved document IDs

  • KB version

  • TTL

This allows future requests to avoid LLM generation.

9. Query Orchestrator → Client

Final response is returned.

{
  "answer": "GPT-5 is...",
  "sources": ["Doc12", "Doc19", "Doc42"],
  "cacheHit": false
}

Document Update Flow

When knowledge base content changes:

1. Admin → KB Service

Document update request arrives.

Doc19 Updated

2. KB Service → Vector Search Service

• New document embedding is generated.

• Old vector is replaced in Vector DB.

3. KB Service → Kafka/Event Bus

Publish event:

DocumentUpdated
{
   docId: 19,
   version: 22
}

4. Event Bus → Cache Invalidation Service

Cache Invalidation Service consumes update event.

5. Cache Invalidation Service → Retrieval Cache

Invalidate retrieval entries containing Doc19.

QueryA → [Doc19, Doc22]
QueryB → [Doc7, Doc19]

Both entries removed.

6. Cache Invalidation Service → Response Cache

Invalidate responses generated using Doc19.

ResponseX
ResponseY
ResponseZ

All removed.

7. Fresh Requests Use New Data

Future queries:

Query
  ↓
Vector Search
  ↓
Updated Doc19
  ↓
LLM

Users receive fresh answers.

End-to-End Summary

User Query
    ↓
API Gateway
    ↓
Query Orchestrator
    ↓
Response Cache ?
    ↓ miss
Embedding Cache ?
    ↓ miss
Embedding Service
    ↓
Retrieval Cache ?
    ↓ miss
Vector Search
    ↓
Knowledge Base
    ↓
LLM Service
    ↓
Response Cache Update
    ↓
User

The Query Orchestrator Service is the central coordinator. All other services (cache, embedding, vector search, KB, LLM) are specialized services that it calls in sequence to answer the query efficiently.

1. Schedule device backups.

2. Trigger automatic backups as per schedule.

3. View backup status and history.

4. Restore device data from selected backup.

5. Send backup/restore success or failure notifications.

NFR:

1. High availability of backup and restore services.

2. Secure data storage and transmission through encryption.

3. Scalable to support multiple devices concurrently.

4. Minimal impact on device performance during backup.

5. Maintain backup data integrity and reliability.

API’s with Request and Response

1. Create Backup Schedule

API: POST /api/v1/backup/schedules

Request

{
  "deviceId": "DEV001",
  "frequency": "DAILY",
  "time": "02:00"
}

Response

{
  "scheduleId": "SCH001",
  "status": "CREATED",
  "message": "Backup schedule created successfully"
}

2. Trigger Backup

API: POST /api/v1/backups

Request

{
  "deviceId": "DEV001",
  "backupType": "FULL"
}

Response

{
  "backupId": "BKP001",
  "status": "IN_PROGRESS",
  "message": "Backup initiated successfully"
}

3. Get Backup Status

API: GET /api/v1/backups/{backupId}

Request

{
  "backupId": "BKP001"
}

Response

{
  "backupId": "BKP001",
  "deviceId": "DEV001",
  "status": "COMPLETED",
  "backupTime": "2026-07-08T02:00:00Z"
}

4. Get Available Backups for Restore

API: GET /api/v1/devices/{deviceId}/backups

Request

{
  "deviceId": "DEV001"
}

Response

{
  "backups": [
    {
      "backupId": "BKP001",
      "createdAt": "2026-07-08T02:00:00Z"
    },
    {
      "backupId": "BKP002",
      "createdAt": "2026-07-07T02:00:00Z"
    }
  ]
}

5. Restore Backup

API: POST /api/v1/restores

Request

{
  "deviceId": "DEV001",
  "backupId": "BKP001"
}

Response

{
  "restoreId": "RST001",
  "status": "IN_PROGRESS",
  "message": "Restore initiated successfully"
}

6. Get Restore Status

API: GET /api/v1/restores/{restoreId}

Request

{
  "restoreId": "RST001"
}

Response

{
  "restoreId": "RST001",
  "deviceId": "DEV001",
  "status": "COMPLETED",
  "restoredAt": "2026-07-08T05:00:00Z"
}

Databases and Schema

Database Choice

• PostgreSQL – Backup schedules, backup metadata, restore jobs, notifications, audit logs

• Redis – Caching and job scheduling

• Object Storage (S3 compatible) – Backup file storage

• Elasticsearch/OpenSearch (Optional) – Log search and analytics

Tables

Device

• device_id (PK, UUID)

• device_name (VARCHAR)

• device_type (VARCHAR)

• owner_id (VARCHAR)

• status (VARCHAR)

• created_at (TIMESTAMP)

Backup_Schedule

• schedule_id (PK, UUID)

• device_id (FK, UUID)

• frequency (VARCHAR)

• backup_time (TIME)

• is_active (BOOLEAN)

• created_at (TIMESTAMP)

• updated_at (TIMESTAMP)

Backup_Job

• backup_id (PK, UUID)

• device_id (FK, UUID)

• schedule_id (FK, UUID)

• backup_type (VARCHAR)

• status (VARCHAR)

• file_size (BIGINT)

• storage_path (VARCHAR)

• started_at (TIMESTAMP)

• completed_at (TIMESTAMP)

Backup_Version

• version_id (PK, UUID)

• backup_id (FK, UUID)

• version_number (INT)

• checksum (VARCHAR)

• retention_days (INT)

• created_at (TIMESTAMP)

Restore_Job

• restore_id (PK, UUID)

• device_id (FK, UUID)

• backup_id (FK, UUID)

• status (VARCHAR)

• initiated_by (VARCHAR)

• started_at (TIMESTAMP)

• completed_at (TIMESTAMP)

Notification

• notification_id (PK, UUID)

• device_id (FK, UUID)

• event_type (VARCHAR)

• recipient (VARCHAR)

• status (VARCHAR)

• sent_at (TIMESTAMP)

Audit_Log

• audit_id (PK, UUID)

• entity_type (VARCHAR)

• entity_id (UUID)

• action (VARCHAR)

• performed_by (VARCHAR)

• timestamp (TIMESTAMP)

• remarks (TEXT)

Relationships

• Device → Backup_Schedule (1:N)

• Device → Backup_Job (1:N)

• Backup_Job → Backup_Version (1:N)

• Device → Restore_Job (1:N)

• Backup_Job → Restore_Job (1:N)

• Device → Notification (1:N)

• All entities → Audit_Log (1:N)

Microservices Architecture – Device Backup Scheduler & Restore

1. API Gateway

   • Single entry point for all client requests

   • Authentication, authorization, routing, rate limiting

2. Device Management Service

   • Manages device registration and device metadata

   • Provides device details to backup and restore services

3. Backup Scheduler Service

   • Creates and manages backup schedules

   • Triggers backup jobs based on configured frequency

4. Backup Service

   • Executes backup operations

   • Stores backup metadata and uploads backup files to object storage

5. Restore Service

   • Retrieves backup data

   • Initiates and monitors restore operations

6. Notification Service

   • Sends email/SMS/in-app notifications for backup and restore events

7. Audit & Logging Service

   • Records user actions and system events

   • Provides audit trail for compliance and troubleshooting

8. Storage Service

   • Manages backup file storage and retrieval from object storage

9. Auth Service

   • User authentication and authorization

   • Issues and validates JWT/OAuth tokens

Service Interaction Flow

Backup Scheduling

1. Client → API Gateway

2. API Gateway → Auth Service (token validation)

3. API Gateway → Backup Scheduler Service

4. Backup Scheduler Service → Device Management Service (validate device)

5. Backup Scheduler Service → PostgreSQL (save schedule)

Backup Execution

1. Scheduler triggers Backup Service

2. Backup Service → Device Management Service (device details)

3. Backup Service → Storage Service (store backup file)

4. Backup Service → PostgreSQL (save metadata)

5. Backup Service → Notification Service (backup status)

6. Notification Service → User

Restore Execution

1. Client → API Gateway

2. API Gateway → Restore Service

3. Restore Service → Backup Service (backup metadata)

4. Restore Service → Storage Service (retrieve backup file)

5. Restore Service → Device Management Service (device validation)

6. Restore Service → PostgreSQL (update restore status)

7. Restore Service → Notification Service (restore status)

Audit Flow

1. Every service publishes events

2. Audit & Logging Service consumes events

3. Audit records stored in Audit_Log table

Communication Pattern

• Synchronous: REST APIs (API Gateway ↔ Services)

• Asynchronous: Kafka/RabbitMQ events for Backup Trigger, Restore Trigger, Notifications, Audit Logging

• Database per service pattern

• Object Storage for backup files

• Redis for caching and scheduled job execution

High-Level Flow
Client → API Gateway → Auth Service → Backup Scheduler Service → Backup Service → Storage Service → Restore Service → Notification Service → Audit Service → Databases/Object Storage/Event Bus.

1. Deploy containers on available compute nodes using Docker/OCI compliant images.

2. Migrate running containers across nodes or clusters while preserving application state and network connectivity.

3. Manage container lifecycle operations including start, stop, restart, update, and scaling.

4. Create and manage virtual networks, service discovery, and cross-network communication between containers.

5. Continuously monitor container and node health, collect metrics, detect failures, and trigger automatic recovery actions.

Non-Functional Requirements

1. Ensure high availability with a target uptime of at least 99.99%.

2. Scale to support up to 100,000 nodes and 1 million running containers.

3. Achieve container deployment and scheduling latency of less than 5 seconds.

4. Provide fault tolerance such that node, network, or service failures do not cause application downtime.

5. Enforce enterprise-grade security through RBAC, TLS-encrypted communication, image signing, verification, and audit logging.

Core APIs

Deploy Container

POST /v1/containers

Request

{
  "image":"nginx:latest",
  "cpu":"2",
  "memory":"4GB",
  "network_id":"net-123",
  "replicas":3
}

Response

{
  "container_id":"ctr-123",
  "status":"DEPLOYING"
}

Get Container

GET /v1/containers/{containerId}

Response

{
  "container_id":"ctr-123",
  "status":"RUNNING",
  "node_id":"node-45",
  "ip":"10.0.1.20"
}

Migrate Container

POST /v1/containers/{containerId}/migrate

Request

{
  "target_node":"node-99"
}

Response

{
  "migration_id":"mig-111",
  "status":"IN_PROGRESS"
}

Scale Container

POST /v1/containers/{containerId}/scale

Request

{
  "replicas":20
}

Response

{
  "status":"SCALING"
}

Create Network

POST /v1/networks

Request

{
  "name":"payment-network",
  "cidr":"10.10.0.0/16"
}

Response

{
  "network_id":"net-123"
}

Database Choice

| Component          | DB            |
| ------------------ | ------------- |
| Container Metadata | PostgreSQL    |
| Cluster State      | etcd          |
| Metrics            | Cassandra     |
| Logs               | Elasticsearch |
| Cache              | Redis         |

Database Schemas

Container

{
  "container_id":"ctr-123",
  "image":"nginx:latest",
  "status":"RUNNING",
  "node_id":"node-45",
  "network_id":"net-123",
  "cpu":"2",
  "memory":"4GB",
  "created_at":"timestamp"
}

Node

{
  "node_id":"node-45",
  "hostname":"worker-1",
  "ip":"192.168.1.10",
  "cpu_total":"64",
  "cpu_available":"20",
  "memory_total":"256GB",
  "memory_available":"120GB",
  "status":"HEALTHY"
}

Cluster

{
  "cluster_id":"cluster-1",
  "name":"us-east-cluster",
  "region":"us-east",
  "status":"ACTIVE"
}

Network

{
  "network_id":"net-123",
  "name":"payment-network",
  "cidr":"10.10.0.0/16",
  "gateway":"10.10.0.1"
}

Migration Job

{
  "migration_id":"mig-111",
  "container_id":"ctr-123",
  "source_node":"node-45",
  "target_node":"node-99",
  "status":"IN_PROGRESS",
  "created_at":"timestamp"
}

High Level Components

                    +------------------+
                    |      User        |
                    +---------+--------+
                              |
                              v
                    +------------------+
                    |    API Gateway   |
                    +---------+--------+
                              |
          -----------------------------------------
          |              |            |           |
          v              v            v           v

+----------------+ +--------------+ +---------------+
| Deployment     | | Migration    | | Network       |
| Service        | | Service      | | Service       |
+-------+--------+ +------+-------+ +-------+-------+
        |                 |                 |
        |                 |                 |
        v                 v                 v

+--------------------------------------------------+
|              Scheduler Service                   |
+--------------------------------------------------+
                        |
                        v

+--------------------------------------------------+
|              Cluster Manager                     |
+--------------------------------------------------+
                        |
        -----------------------------------
        |                |               |
        v                v               v

+------------+ +-------------+ +-------------+
| Node Agent | | Node Agent  | | Node Agent  |
| Node-1     | | Node-2      | | Node-N      |
+------------+ +-------------+ +-------------+

Container Deployment Flow

User
 |
 | Deploy Container
 v

API Gateway
 |
 v

Deployment Service
 |
 v

Scheduler
 |
 | Find best node
 v

Cluster Manager
 |
 v

Node Agent
 |
 | Pull Image
 | Create Container
 v

Container Runtime
 |
 v

Container Running

Container Migration Flow

User
 |
 | Migrate Container
 v

Migration Service
 |
 v

Scheduler
 |
 | Select Target Node
 v

Checkpoint Service
 |
 | Save Container State
 v

Target Node Agent
 |
 | Restore State
 v

Network Service
 |
 | Update Routing
 v

Container Running on New Node

Microservices

1. API Gateway

• Authentication
  

• Rate limiting
  

• Routing
  

2. Deployment Service

• Container deployment
  

• Scaling requests
  

3. Scheduler Service

• Node selection
  

• Resource allocation
  

4. Migration Service

• Container relocation
  

• State transfer
  

5. Network Service

• Overlay networking
  

• Service discovery
  

6. Cluster Manager

• Cluster state management
  

• Node registration
  

7. Health Monitoring Service

• Metrics collection
  

• Alerting
  

8. Auto-Healing Service

• Restart failed containers
  

• Reschedule workloads
  

9. Image Registry Service

• Image storage
  

• Image versioning
  

10. Audit & Security Service

• RBAC
  

• Secrets management
  

• Compliance logs
  

Microservice Interaction

                 +----------------+
                 | API Gateway    |
                 +-------+--------+
                         |
     ------------------------------------------------
     |              |              |                |
     v              v              v                v

Deployment   Migration      Network         Monitoring
 Service      Service       Service           Service
     |              |             |                |
     ------------------------------------------------
                         |
                         v

                  Scheduler
                         |
                         v

                 Cluster Manager
                         |
         ---------------------------------
         |               |               |
         v               v               v

      Node Agent     Node Agent      Node Agent
         |               |               |
         ---------------------------------
                         |
                         v

                Container Runtime
             (containerd / Docker)

Bottlenecks & Scaling

Scheduler Bottleneck

• Use sharded schedulers.
  

• Leader election via etcd.
  

Metrics Explosion

• Store metrics in Cassandra.
  

• Aggregate before persistence.
  

Container Migration

• Use incremental checkpoints.
  

• Transfer only changed memory pages.
  

Multi-Region Networking

• Overlay network (VXLAN/WireGuard).
  

• Global service discovery.
  

Cluster State

• Store in etcd with Raft consensus.
  

• Multiple replicas for HA.
  

This design is very close to how modern orchestration platforms such as Kubernetes, Nomad, and Docker Swarm manage large-scale container deployments across clusters and networks.

📩 Contact: programmingappliedai@gmail.com

Functional Requirements (FR)

1. Asset Tracking

• System should track assets (packages, vehicles, devices, orders) in real-time.

• Support GPS coordinate updates.

2. Location History

• Store and retrieve complete location history of tracked entities.

• Allow querying historical movement.

3. Geofence Management

• Create geofences around specific locations.

• Trigger events when tracked entities enter or exit geofences.

4. Alerts & Notifications

• Generate alerts for:

  • Geofence breach

  • Device offline

  • Route deviation

  • Low battery

5. Tracking Dashboard

• Display current location, status, speed, route, and historical trails.

• Support filtering and search.

Non-Functional Requirements (NFR)

1. High Availability

• 99.99% uptime.

2. Scalability

• Support 10M+ tracked devices.

• Support 100K location updates/sec.

3. Low Latency

• Location updates visible within 2-3 seconds.

4. Durability

• No loss of tracking events.

5. Security

• OAuth/JWT authentication.

• Encrypted data in transit and at rest.

APIs

1. Register Tracking Device

Request

POST /api/v1/devices

{
  "deviceId": "DEV123",
  "assetType": "VEHICLE",
  "assetId": "CAR123"
}

Response

{
  "deviceId": "DEV123",
  "status": "REGISTERED"
}

2. Update Location

Request

POST /api/v1/location/update

{
  "deviceId": "DEV123",
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45,
  "timestamp": 1745000000
}

Response

{
  "status": "SUCCESS"
}

3. Get Current Location

Request

GET /api/v1/location/current/{deviceId}

Response

{
  "deviceId": "DEV123",
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45,
  "timestamp": 1745000000
}

4. Get Location History

Request

GET /api/v1/location/history

{
  "deviceId": "DEV123",
  "startTime": 1745000000,
  "endTime": 1746000000
}

Response

{
  "locations": [
    {
      "latitude": 12.9716,
      "longitude": 77.5946,
      "timestamp": 1745000000
    }
  ]
}

5. Create Geofence

Request

POST /api/v1/geofence

{
  "geofenceId": "GF123",
  "name": "Warehouse",
  "centerLatitude": 12.9716,
  "centerLongitude": 77.5946,
  "radiusMeters": 500
}

Response

{
  "status": "CREATED"
}

6. Get Alerts

Request

GET /api/v1/alerts/{deviceId}

Response

{
  "alerts": [
    {
      "alertType": "GEOFENCE_EXIT",
      "timestamp": 1745000000
    }
  ]
}

Database Design

Device Table

{
  "deviceId": "DEV123",
  "assetId": "CAR123",
  "assetType": "VEHICLE",
  "status": "ACTIVE",
  "registeredAt": "2026-01-01T10:00:00Z"
}

Partition Key:

deviceId

Database:

PostgreSQL

Current Location Table

{
  "deviceId": "DEV123",
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45,
  "lastUpdatedAt": "2026-01-01T10:10:00Z"
}

Partition Key:

deviceId

Database:

Redis

Location History Table

{
  "deviceId": "DEV123",
  "timestamp": 1745000000,
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45
}

Partition Key:

deviceId

Sort Key:

timestamp

Database:

Cassandra

Geofence Table

{
  "geofenceId": "GF123",
  "name": "Warehouse",
  "centerLatitude": 12.9716,
  "centerLongitude": 77.5946,
  "radiusMeters": 500,
  "createdAt": "2026-01-01T10:00:00Z"
}

Database:

PostgreSQL

Alert Table

{
  "alertId": "ALT123",
  "deviceId": "DEV123",
  "alertType": "GEOFENCE_EXIT",
  "message": "Vehicle left warehouse zone",
  "createdAt": "2026-01-01T10:15:00Z"
}

Database:

PostgreSQL

High Level Architecture

                +----------------+
                | Mobile Device  |
                | GPS Tracker    |
                +--------+-------+
                         |
                         v
                +----------------+
                | API Gateway    |
                +--------+-------+
                         |
                         v
                +----------------+
                | Tracking       |
                | Service        |
                +--------+-------+
                         |
          +--------------+--------------+
          |                             |
          v                             v

 +----------------+         +-------------------+
 | Redis Current  |         | Kafka             |
 | Location Cache |         | Location Stream   |
 +----------------+         +---------+---------+
                                      |
                                      |
             +------------------------+------------------+
             |                                           |
             v                                           v

   +-------------------+                     +------------------+
   | Location History  |                     | Geofence Engine  |
   | Service           |                     | Service          |
   +--------+----------+                     +--------+---------+
            |                                         |
            v                                         v

      +------------+                          +--------------+
      | Cassandra  |                          | Alert Service|
      +------------+                          +------+-------+
                                                      |
                                                      v
                                            +------------------+
                                            | Notification     |
                                            | Service          |
                                            +------------------+

Microservices

1. Device Service

Responsibilities:

• Device registration
  

• Device lifecycle management
  

• Device metadata storage
  

Database:

PostgreSQL

2. Tracking Service

Responsibilities:

• Receive GPS updates
  

• Validate payload
  

• Publish events to Kafka
  

Database:

Redis

3. Location History Service

Responsibilities:

• Persist location updates
  

• Historical queries
  

Database:

Cassandra

4. Geofence Service

Responsibilities:

• Geofence creation
  

• Entry/exit detection
  

Database:

PostgreSQL

5. Alert Service

Responsibilities:

• Generate alerts
  

• Store alert history
  

Database:

PostgreSQL

6. Notification Service

Responsibilities:

• SMS notifications
  

• Email notifications
  

• Push notifications
  

Database:

None (Stateless)

End-to-End Flow

1. GPS Device sends location update.

2. API Gateway forwards request to Tracking Service.

3. Tracking Service:
   - Validates request
   - Updates Redis current location
   - Publishes LocationEvent to Kafka

4. Location History Service consumes event.
   - Stores event in Cassandra

5. Geofence Service consumes event.
   - Checks geofence rules

6. If breach detected:
   - Generate Alert Event

7. Alert Service stores alert.

8. Notification Service sends SMS/Email/Push.

9. Dashboard fetches:
   - Current location from Redis
   - Historical route from Cassandra
   - Alerts from PostgreSQL

Scale Estimation

10 Million Devices

Average Update Frequency = 30 seconds

Writes/sec
= 10,000,000 / 30
≈ 333,000 writes/sec

Daily Location Records
≈ 28 Billion records/day

Storage Choice:
- Redis -> Current State
- Kafka -> Streaming
- Cassandra -> Time Series History
- PostgreSQL -> Metadata & Alerts

This design is suitable for a logistics, fleet-management, package-tracking, or IoT tracking platform operating at internet scale.

📩 Contact: programmingappliedai@gmail.com

Example:

Search Query: "red"
Expected Output: List of all file IDs/names containing the word "red".

Traffic:

• ~10 lakh requests/day (~11.5 requests/sec average)

• Assume:

  • Exact keyword matching only

  • No typo tolerance

  • Words contain only a-z

  • Files can be added/updated periodically

Functional Requirements (FR)

FR1. File Ingestion

System should accept and process files for indexing.

FR2. Word Extraction & Tokenization

System should parse files and extract normalized words (a-z only).

FR3. Inverted Index Creation

System should maintain an inverted index mapping:

word → list of file_ids

Example:

red → [file1, file8, file20]

FR4. Search API

User should be able to search for a word and retrieve all matching files.

FR5. Incremental Updates

System should support:

• New file addition
  

• File modification
  

• File deletion
  
  without rebuilding the complete index.
  

Non-Functional Requirements (NFR)

NFR1. Low Latency

Search response time should be under 100ms.

NFR2. High Availability

Search service should remain available during node failures.

NFR3. Scalability

System should scale horizontally as:

• Number of files increases
  

• Query traffic increases
  

NFR4. Durability

Indexed metadata and file mappings should not be lost.

NFR5. Efficient Storage

Index storage should be optimized using compression and deduplication.

High Level Architecture

                +------------------+
                |   Client/User    |
                +--------+---------+
                         |
                         v
               +---------+----------+
               |   Search API Layer |
               +---------+----------+
                         |
              +----------+-----------+
              |                      |
              v                      v
     +--------+-------+      +-------+--------+
     |   Query Cache  |      | Search Service |
     |    (Redis)     |      +-------+--------+
     +----------------+              |
                                     v
                          +----------+----------+
                          |   Inverted Index    |
                          |  (ElasticSearch /  |
                          |    Lucene Index)   |
                          +----------+----------+
                                     |
                  +------------------+------------------+
                  |                                     |
                  v                                     v
        +---------+---------+             +-------------+----------+
        | Index Builder     |             | Metadata DB            |
        | Tokenizer Engine  |             | (File Metadata Store)  |
        +---------+---------+             +------------------------+
                  |
                  v
         +--------+--------+
         | Distributed File |
         | Storage (S3/HDFS)|
         +------------------+

Core Idea — Inverted Index

Traditional search over 1 million files is expensive:

For every query:
   Scan all files

Complexity:

O(number_of_files)

Instead, build an inverted index.

Inverted Index Structure

{
   "red"  : [1, 7, 10, 55],
   "blue" : [2, 3, 10],
   "car"  : [5, 6]
}

Search becomes:

Lookup(word)

Complexity:

O(1) or O(logN)

depending on storage engine.

Components

1. File Storage Service

Stores original files.

Possible storage:

• HDFS
  

• S3
  

• Blob storage
  

Each file gets:

{
  "file_id": "f123",
  "path": "/docs/a.txt",
  "uploaded_at": 1712345678
}

2. File Ingestion Service

Responsible for:

• Accepting files
  

• Sending indexing jobs to queue
  

Flow:

Upload File
    ↓
Generate file_id
    ↓
Push indexing job to Kafka/SQS

3. Tokenizer / Parser Service

Reads file contents and extracts words.

Steps:

• Lowercase conversion
  

• Remove punctuation
  

• Split into words
  

Example:

"The Red Car"

becomes:

["the", "red", "car"]

4. Index Builder

Builds inverted index.

Example:

file1 -> ["red", "car"]

Index:
red -> file1
car -> file1

5. Search Service

Handles user queries.

Flow:

1. Receive query "red"
   

2. Check Redis cache
   

3. Lookup inverted index
   

4. Return matching files
   

Database Design

A. File Metadata Table

{
  "file_id": "string",
  "file_name": "string",
  "storage_path": "string",
  "uploaded_at": "timestamp",
  "version": "int",
  "status": "ACTIVE | DELETED"
}

Database:

• PostgreSQL
  
  or
  

• MySQL
  

B. Inverted Index Storage

Logical Schema

{
  "word": "red",
  "document_frequency": 1200,
  "posting_list": [
    "file1",
    "file8",
    "file20"
  ]
}

Preferred technologies:

• ElasticSearch
  

• Apache Lucene
  

• OpenSearch
  

Search Flow

User Searches "red"
        |
        v
Search API
        |
        v
Redis Cache Check
   |          |
Hit          Miss
 |             |
Return      Query Index
                 |
                 v
        Get Posting List
                 |
                 v
         Fetch Metadata
                 |
                 v
            Return Files

File Indexing Flow

File Upload
    |
    v
Ingestion Service
    |
    v
Message Queue (Kafka)
    |
    v
Indexer Worker
    |
    v
Tokenizer
    |
    v
Update Inverted Index

Scaling Strategy

Read Scaling

Search traffic is read-heavy.

Use:

• Multiple search replicas
  

• Load balancer
  

• Redis cache
  

Index Partitioning (Sharding)

Partition words alphabetically:

a-f  → shard1
g-l  → shard2
m-r  → shard3
s-z  → shard4

or use:

• Consistent hashing
  

Caching Strategy

Popular searches cached in Redis.

Example:

key   = "red"
value = [file1, file2, file8]
TTL   = 10 mins

Reduces repeated index lookups.

Estimation

Requests

10 lakh/day
≈ 11.5 req/sec average
Peak ≈ 100-200 req/sec

Very manageable with:

• Redis
  

• ElasticSearch cluster
  

Fault Tolerance

Replication

Maintain:

• Replica index shards
  

• Multi-node search cluster
  

Queue Durability

Kafka persists indexing jobs.

If indexer crashes:

• Job replay possible
  

Optimizations

1. Compression

Store posting lists using:

• Delta encoding
  

• Bitmaps
  

2. Skip Lists

Improve search traversal for large posting lists.

3. Batch Indexing

Instead of updating index per file:

• Batch updates every few seconds.
  

Bottlenecks

Hot Keywords

Words like:

the, is, and

can have huge posting lists.

Solution:

• Stop-word filtering
  

• Compressed bitmaps
  

Technologies

| Component     | Technology            |
| ------------- | --------------------- |
| API Layer     | Go / Java Spring Boot |
| Queue         | Kafka                 |
| Cache         | Redis                 |
| Search Engine | ElasticSearch         |
| Metadata DB   | PostgreSQL            |
| Storage       | S3 / HDFS             |

Final Design Summary

The system uses an inverted index to efficiently map:

word → files

Core benefits:

• Fast lookup
  

• Horizontal scalability
  

• Incremental indexing
  

• Low latency search
  

The architecture separates:

• File storage
  

• Index building
  

• Search serving
  

allowing the system to scale independently for ingestion and querying.

📩 Contact: programmingappliedai@gmail.com

Functional Requirements (FR)

1. Penalty/Tax Rule Management
   System should allow admins to define and manage dynamic rules for penalties/taxes based on:

   • Minimum rides completed

   • Cancellation rate

   • Average rider/driver ratings

   • Inactive days

   • Peak-hour participation

   • Fraud signals

2. Periodic Evaluation Engine
   System should periodically evaluate riders/drivers (daily/weekly/monthly) against configured rules and determine applicable penalties or taxes.

3. Penalty Calculation & Ledger Management
   System should calculate penalty amounts accurately and maintain a financial ledger/history for every deduction, adjustment, refund, or waiver.

4. Notification & Transparency
   Users should receive notifications explaining:

   • Why the penalty was applied

   • Rule violated

   • Amount deducted

   • Appeal process

   • Deadline for compliance

5. Appeal & Dispute Handling
   Riders/drivers should be able to raise disputes against penalties, upload proofs, and track appeal status through the platform.

Non-Functional Requirements (NFR)

1. Scalability
   System should support millions of riders/drivers and evaluate penalties for large datasets efficiently during monthly batch processing.

2. High Availability
   Critical services like rule evaluation, deductions, and notifications should maintain high availability (99.9%+) to avoid financial inconsistencies.

3. Consistency & Accuracy
   Financial calculations and deductions must be strongly consistent to prevent duplicate or incorrect penalties.

4. Extensibility
   System should support adding new penalty rules and policy types without requiring major architectural changes.

5. Auditability & Compliance
   Every rule evaluation, deduction, override, and admin action should be fully auditable for regulatory and legal compliance.

APIs for Penalty/Tax Enforcement System

1. Create Penalty Rule

API

POST /api/v1/rules

Request

{
  "ruleName": "Minimum Monthly Ride Rule",
  "entityType": "DRIVER",
  "condition": {
    "metric": "MONTHLY_RIDES",
    "operator": "LESS_THAN",
    "value": 100
  },
  "penalty": {
    "type": "FIXED",
    "amount": 500
  },
  "evaluationFrequency": "MONTHLY",
  "isActive": true
}

Response

{
  "ruleId": "RULE_12345",
  "status": "CREATED",
  "createdAt": "2026-05-26T10:00:00Z"
}

2. Evaluate User Eligibility for Penalty

API

POST /api/v1/evaluations/run

Request

{
  "entityType": "DRIVER",
  "entityId": "DRIVER_789",
  "evaluationPeriod": "2026-05"
}

Response

{
  "entityId": "DRIVER_789",
  "evaluationPeriod": "2026-05",
  "violations": [
    {
      "ruleId": "RULE_12345",
      "metricValue": 72,
      "expectedValue": 100,
      "penaltyAmount": 500
    }
  ],
  "totalPenalty": 500,
  "status": "PENALTY_APPLIED"
}

3. Fetch Penalty History

API

GET /api/v1/penalties/{entityId}

Response

{
  "entityId": "DRIVER_789",
  "penalties": [
    {
      "penaltyId": "PEN_1001",
      "ruleId": "RULE_12345",
      "amount": 500,
      "status": "DEDUCTED",
      "createdAt": "2026-05-26T12:00:00Z"
    },
    {
      "penaltyId": "PEN_1002",
      "ruleId": "RULE_99999",
      "amount": 200,
      "status": "WAIVED",
      "createdAt": "2026-04-26T12:00:00Z"
    }
  ]
}

4. Deduct Penalty from Wallet

API

POST /api/v1/wallets/deduct

Request

{
  "entityId": "DRIVER_789",
  "penaltyId": "PEN_1001",
  "amount": 500,
  "currency": "INR"
}

Response

{
  "transactionId": "TXN_56789",
  "walletBalance": 3200,
  "status": "SUCCESS"
}

5. Raise Penalty Dispute

API

POST /api/v1/disputes

Request

{
  "penaltyId": "PEN_1001",
  "entityId": "DRIVER_789",
  "reason": "Ride count calculation incorrect",
  "attachments": [
    "s3://proofs/screenshot1.png"
  ]
}

Response

{
  "disputeId": "DISP_4567",
  "status": "UNDER_REVIEW",
  "createdAt": "2026-05-26T13:00:00Z"
}

6. Get Evaluation Summary Dashboard

API

GET /api/v1/reports/summary?period=2026-05

Response

{
  "period": "2026-05",
  "totalDriversEvaluated": 1000000,
  "totalPenaltiesApplied": 125000,
  "totalAmountCollected": 45000000,
  "topViolationReasons": [
    {
      "rule": "MIN_MONTHLY_RIDES",
      "count": 80000
    },
    {
      "rule": "LOW_RATING",
      "count": 45000
    }
  ]
}

Databases and Schema

```json
{
  "users": {
    "user_id": "BIGINT",
    "user_type": "ENUM(DRIVER, RIDER)",
    "name": "VARCHAR",
    "avg_rating": "DECIMAL(2,1)",
    "total_rides": "BIGINT",
    "wallet_balance": "DECIMAL",
    "status": "ENUM(ACTIVE, BLOCKED)",
    "created_at": "TIMESTAMP"
  },

  "penalty_rules": {
    "rule_id": "BIGINT",
    "rule_name": "VARCHAR",
    "entity_type": "ENUM(DRIVER, RIDER)",
    "metric": "VARCHAR",
    "operator": "ENUM(LESS_THAN, GREATER_THAN)",
    "threshold_value": "DECIMAL",
    "penalty_type": "ENUM(FIXED, PERCENTAGE)",
    "penalty_amount": "DECIMAL",
    "evaluation_frequency": "ENUM(DAILY, WEEKLY, MONTHLY)",
    "is_active": "BOOLEAN",
    "created_by": "BIGINT",
    "created_at": "TIMESTAMP"
  },

  "user_metrics": {
    "metric_id": "BIGINT",
    "user_id": "BIGINT",
    "month": "VARCHAR",
    "completed_rides": "INT",
    "cancelled_rides": "INT",
    "avg_rating": "DECIMAL(2,1)",
    "online_hours": "INT",
    "last_updated_at": "TIMESTAMP"
  },

  "penalties": {
    "penalty_id": "BIGINT",
    "user_id": "BIGINT",
    "rule_id": "BIGINT",
    "metric_value": "DECIMAL",
    "expected_value": "DECIMAL",
    "penalty_amount": "DECIMAL",
    "status": "ENUM(APPLIED, DEDUCTED, WAIVED)",
    "evaluation_period": "VARCHAR",
    "created_at": "TIMESTAMP"
  },

  "wallet_transactions": {
    "transaction_id": "BIGINT",
    "user_id": "BIGINT",
    "penalty_id": "BIGINT",
    "transaction_type": "ENUM(DEBIT, CREDIT)",
    "amount": "DECIMAL",
    "balance_after_txn": "DECIMAL",
    "transaction_status": "ENUM(SUCCESS, FAILED)",
    "created_at": "TIMESTAMP"
  },

  "disputes": {
    "dispute_id": "BIGINT",
    "penalty_id": "BIGINT",
    "user_id": "BIGINT",
    "reason": "TEXT",
    "attachment_url": "VARCHAR",
    "status": "ENUM(UNDER_REVIEW, APPROVED, REJECTED)",
    "resolved_by": "BIGINT",
    "resolution_notes": "TEXT",
    "created_at": "TIMESTAMP",
    "resolved_at": "TIMESTAMP"
  },

  "audit_logs": {
    "audit_id": "BIGINT",
    "entity_type": "VARCHAR",
    "entity_id": "BIGINT",
    "action": "VARCHAR",
    "performed_by": "BIGINT",
    "old_value": "JSON",
    "new_value": "JSON",
    "created_at": "TIMESTAMP"
  }
}
```

HLD

```text
========================================================
HIGH LEVEL DESIGN (HLD)
========================================================

                    +----------------------+
                    |      Admin Panel     |
                    +----------+-----------+
                               |
                               v
                    +----------------------+
                    |   Rule Management    |
                    |      Service         |
                    +----------+-----------+
                               |
                               v
                    +----------------------+
                    |   Penalty Rules DB   |
                    +----------------------+



+------------+       +----------------------+       +------------------+
| Ride       | ----> |  Event Ingestion     | ----> | Kafka/Event Bus  |
| Service    |       |  Service             |       +--------+---------+
+------------+       +----------------------+                |
                                                             |
+------------+                                               |
| Rating     | ----------------------------------------------+
| Service    |                                               |
+------------+                                               |
                                                             v

                                                +----------------------+
                                                | Metrics Aggregator   |
                                                | Service              |
                                                +----------+-----------+
                                                           |
                                                           v
                                                +----------------------+
                                                | User Metrics DB      |
                                                +----------+-----------+
                                                           |
                                                           v

                                                +----------------------+
                                                | Penalty Evaluation   |
                                                | Engine               |
                                                +----------+-----------+
                                                           |
                                      +--------------------+------------------+
                                      |                                       |
                                      v                                       v

                        +--------------------------+         +------------------------+
                        | Penalty Ledger Service   |         | Notification Service   |
                        +------------+-------------+         +------------+-----------+
                                     |                                     |
                                     v                                     v

                        +--------------------------+         +------------------------+
                        | Penalty DB               |         | Email/SMS/Push         |
                        +------------+-------------+         +------------------------+
                                     |
                                     v

                        +--------------------------+
                        | Wallet Service           |
                        +------------+-------------+
                                     |
                                     v

                        +--------------------------+
                        | Wallet Transactions DB   |
                        +--------------------------+



========================================================
FLOW DIAGRAM - MONTHLY PENALTY EVALUATION
========================================================

1. Ride Completed
        |
        v
2. Ride Service emits event to Kafka
        |
        v
3. Metrics Aggregator consumes events
        |
        v
4. Update monthly metrics in User Metrics DB
        |
        v
5. Monthly Scheduler triggers Evaluation Engine
        |
        v
6. Evaluation Engine fetches:
       - Active Rules
       - User Metrics
        |
        v
7. Rule Validation
       Example:
       completed_rides < 100
        |
        +---- NO ----> No action
        |
        +---- YES ---> Generate penalty
                             |
                             v
8. Store penalty in Penalty DB
                             |
                             v
9. Call Wallet Service
                             |
                             v
10. Deduct amount from wallet
                             |
                             v
11. Store wallet transaction
                             |
                             v
12. Send notification to user



========================================================
FLOW DIAGRAM - DISPUTE HANDLING
========================================================

1. User raises dispute
        |
        v
2. Dispute Service stores dispute
        |
        v
3. Admin reviews dispute
        |
        +---- APPROVED ----+
        |                  |
        |                  v
        |          Refund penalty
        |                  |
        |                  v
        |          Update wallet balance
        |
        +---- REJECTED ----+
                           |
                           v
                    Keep penalty active
                           |
                           v
                    Notify user



========================================================
CORE MICROSERVICES
========================================================

1. Rule Management Service
   - Create/update/delete rules
   - Rule versioning

2. Metrics Aggregator Service
   - Consume ride/rating events
   - Aggregate monthly metrics

3. Penalty Evaluation Engine
   - Execute rule evaluation jobs
   - Generate penalties

4. Penalty Ledger Service
   - Maintain immutable penalty history

5. Wallet Service
   - Deduct/refund money
   - Ensure transactional consistency

6. Notification Service
   - SMS/Push/Email alerts

7. Dispute Service
   - Dispute lifecycle management

8. Audit Service
   - Maintain compliance logs



========================================================
SCALABILITY CONSIDERATIONS
========================================================

1. Kafka partitions by user_id

2. Batch evaluation jobs partitioned by region/user buckets

3. Redis cache for active rules

4. Read replicas for analytics queries

5. Monthly partitioned tables for penalties and transactions

6. Idempotent wallet deduction APIs

7. Retry queues + DLQ for failed deductions
```

========================================================
MICROSERVICES AND INTERACTIONS
========================================================

1. API GATEWAY
--------------------------------------------------------
Responsibilities:
- Entry point for all clients
- Authentication & authorization
- Rate limiting
- Request routing

Interacts With:
- Rule Management Service
- Penalty Service
- Dispute Service
- Wallet Service
- Notification Service



2. RULE MANAGEMENT SERVICE
--------------------------------------------------------
Responsibilities:
- Create/update/delete penalty rules
- Enable/disable rules
- Rule versioning
- Rule validation

Database:
- penalty_rules DB

Publishes Events:
- RULE_CREATED
- RULE_UPDATED
- RULE_DELETED

Interacts With:
- Admin Panel
- Penalty Evaluation Engine
- Redis Cache



3. EVENT INGESTION SERVICE
--------------------------------------------------------
Responsibilities:
- Consume ride/rating/cancellation events
- Standardize incoming events
- Push events to Kafka

Consumes From:
- Ride Service
- Rating Service

Publishes To:
- Kafka Topics



4. METRICS AGGREGATOR SERVICE
--------------------------------------------------------
Responsibilities:
- Consume Kafka events
- Aggregate monthly metrics
- Compute:
   - completed rides
   - cancellation %
   - avg ratings
   - active hours

Database:
- user_metrics DB

Consumes:
- ride_completed events
- rating_updated events
- cancellation events

Publishes:
- METRICS_UPDATED



5. PENALTY EVALUATION ENGINE
--------------------------------------------------------
Responsibilities:
- Run periodic evaluation jobs
- Fetch active rules
- Evaluate user metrics
- Generate penalties

Consumes:
- METRICS_UPDATED
- Scheduler triggers

Reads From:
- penalty_rules DB
- user_metrics DB

Publishes:
- PENALTY_CREATED

Interacts With:
- Penalty Ledger Service
- Wallet Service
- Notification Service



6. PENALTY LEDGER SERVICE
--------------------------------------------------------
Responsibilities:
- Store immutable penalty history
- Maintain audit trail
- Penalty state management

Database:
- penalties DB

Consumes:
- PENALTY_CREATED

Publishes:
- PENALTY_APPLIED
- PENALTY_WAIVED

Interacts With:
- Wallet Service
- Dispute Service
- Audit Service



7. WALLET SERVICE
--------------------------------------------------------
Responsibilities:
- Deduct penalty amount
- Refund amount on dispute approval
- Maintain financial consistency
- Handle retries/idempotency

Database:
- wallet_transactions DB

Consumes:
- PENALTY_APPLIED
- REFUND_REQUESTED

Publishes:
- PAYMENT_SUCCESS
- PAYMENT_FAILED

Interacts With:
- Penalty Service
- Notification Service



8. NOTIFICATION SERVICE
--------------------------------------------------------
Responsibilities:
- Send push/email/SMS notifications
- Notify about:
   - penalties
   - refunds
   - disputes
   - warnings

Consumes:
- PENALTY_CREATED
- PAYMENT_SUCCESS
- DISPUTE_RESOLVED

External Integrations:
- Email provider
- SMS gateway
- Push notification provider



9. DISPUTE SERVICE
--------------------------------------------------------
Responsibilities:
- Create/manage disputes
- Upload proof documents
- Track dispute lifecycle

Database:
- disputes DB

Consumes:
- User dispute requests

Publishes:
- DISPUTE_CREATED
- DISPUTE_APPROVED
- DISPUTE_REJECTED

Interacts With:
- Penalty Ledger Service
- Wallet Service
- Notification Service



10. AUDIT SERVICE
--------------------------------------------------------
Responsibilities:
- Track all admin/system actions
- Maintain compliance logs
- Store immutable audit records

Database:
- audit_logs DB

Consumes:
- RULE_UPDATED
- PENALTY_APPLIED
- DISPUTE_RESOLVED
- WALLET_DEDUCTED



========================================================
SERVICE INTERACTION FLOW
========================================================

Ride Service
     |
     v
Event Ingestion Service
     |
     v
Kafka/Event Bus
     |
     v
Metrics Aggregator Service
     |
     v
User Metrics DB
     |
     v
Penalty Evaluation Engine
     |
     +--------------------+
     |                    |
     v                    v
Penalty Ledger      Notification Service
Service
     |
     v
Wallet Service
     |
     v
Wallet Transactions DB
     |
     v
Notification Service



========================================================
ASYNC COMMUNICATION
========================================================

Kafka Topics:
- ride_completed
- ride_cancelled
- rating_updated
- metrics_updated
- penalty_created
- penalty_applied
- payment_success
- dispute_created
- dispute_resolved



========================================================
WHY MICROSERVICE ARCHITECTURE?
========================================================

1. Independent scaling of evaluation engine

2. Wallet service isolated for strong consistency

3. Rule engine can evolve independently

4. Async event-driven processing reduces coupling

5. Easier fault isolation

6. Supports millions of monthly evaluations

📩 Contact: programmingappliedai@gmail.com

Example:

• Detect high CPU spikes on routers

• Predict link failures

• Detect packet drops or abnormal latency

• Restart interfaces automatically

• Shift traffic to healthy routes

2. Functional Requirements (FR)

1. The system should continuously collect router telemetry such as CPU utilization, memory usage, packet drops, interface statistics, link status, temperature, and logs/SNMP traps at configurable intervals.

2. The system should detect failures and anomalies like router downtime, link degradation, interface flapping, traffic anomalies, high latency, and packet loss within a few seconds.

3. The system should automatically execute preventive or corrective actions such as restarting interfaces, switching routes, triggering failover, throttling traffic, or rebooting unhealthy devices based on predefined policies.

4. The system should generate alerts and incidents through channels like email, Slack, PagerDuty, and SMS for critical network issues and remediation events.

5. The system should store historical telemetry and fault data to support trend analysis, root cause analysis, capacity planning, and ML/rule-based failure prediction.

3. Non-Functional Requirements (NFR)

NFR1. High Availability

Monitoring system should remain operational even during node failures.

Target: 99.99% availability.

NFR2. Low Detection Latency

Faults should be detected within:

• 2–5 seconds for critical failures

• <1 second for heartbeat failures

NFR3. Massive Scalability

Support:

• Millions of telemetry events/sec

• 100K+ routers

• Multi-region deployments

NFR4. Fault Tolerance

No data loss during:

• Broker failures

• Service crashes

• Network partitions

NFR5. Extensibility

System should support:

• New device types

• New telemetry protocols

• Custom remediation plugins

without major redesign.

4. High Level Architecture

                 +----------------+
                 |   Admin UI     |
                 +--------+-------+
                          |
                          v
                 +----------------+
                 | API Gateway    |
                 +--------+-------+
                          |
      ------------------------------------------------
      |              |               |               |
      v              v               v               v

+------------+ +-------------+ +-------------+ +--------------+
| Device Mgmt| | Alert Svc   | | Policy Svc  | | Analytics Svc|
+------------+ +-------------+ +-------------+ +--------------+

                          |
                          v

                +------------------+
                | Kafka / Pulsar   |
                +--------+---------+
                         |
     ---------------------------------------------------
     |                 |               |               |
     v                 v               v               v

+------------+ +--------------+ +-------------+ +--------------+
| Collectors | | Detection Svc| | ML Engine   | | Remediation  |
+------------+ +--------------+ +-------------+ +--------------+

                         |
                         v

                 +---------------+
                 | Routers       |
                 +---------------+

5. Core Components

A. Telemetry Collectors

Responsible for:

• SNMP polling
  

• Streaming telemetry
  

• Syslog ingestion
  

• NetFlow/sFlow collection
  

Protocols:

• SNMP
  

• gRPC telemetry
  

• NETCONF
  

• SSH
  

• Syslog
  

B. Event Streaming Layer

Use:

• Apache Kafka
  
  or
  

• Apache Pulsar
  

Purpose:

• Buffer telemetry
  

• Decouple services
  

• Enable replay
  

• Handle burst traffic
  

Topics:

telemetry.cpu
telemetry.memory
telemetry.interface
alerts
remediation.tasks

C. Detection Engine

Performs:

• Threshold detection
  

• Pattern detection
  

• Correlation
  

• Stateful analysis
  

Examples:

• CPU > 90% for 5 mins
  

• Packet loss > 20%
  

• Interface flapping > 5 times/min
  

Techniques:

• Rule engine
  

• CEP (Complex Event Processing)
  

D. ML Prediction Engine

Predicts:

• Link failure probability
  

• Hardware degradation
  

• Congestion patterns
  

Models:

• Time-series forecasting
  

• Isolation Forest
  

• LSTM
  

• ARIMA
  

E. Remediation Service

Executes automated actions:

• Route failover
  

• Interface reset
  

• Restart BGP sessions
  

• Traffic rerouting
  

Must support:

• Rollback
  

• Audit logs
  

• Approval workflows
  

F. Alerting Service

Integrations:

• PagerDuty
  

• Slack
  

• Email/SMS systems
  

Supports:

• Deduplication
  

• Escalation
  

• Suppression
  

6. APIs

A. Register Router API

Request

POST /api/v1/routers

{
  "routerId": "RTR-1001",
  "hostname": "blr-edge-1",
  "ip": "10.0.0.5",
  "location": "Bangalore-DC",
  "vendor": "Cisco",
  "model": "ASR-9000"
}

Response

{
  "status": "REGISTERED",
  "routerId": "RTR-1001"
}

B. Push Telemetry API

POST /api/v1/telemetry

Request

{
  "routerId": "RTR-1001",
  "timestamp": 1710001231,
  "metrics": {
    "cpu": 91,
    "memory": 72,
    "packetLoss": 15,
    "latency": 240
  }
}

Response

{
  "status": "INGESTED"
}

C. Create Remediation Policy

POST /api/v1/policies

Request

{
  "policyName": "HighCPURecovery",
  "condition": "cpu > 90 for 5m",
  "action": "restart_interface",
  "severity": "HIGH"
}

Response

{
  "policyId": "POL-101",
  "status": "ACTIVE"
}

D. Get Active Alerts

GET /api/v1/alerts?severity=CRITICAL

Response

[
  {
    "alertId": "ALT-1001",
    "routerId": "RTR-1001",
    "message": "Packet loss exceeded threshold",
    "severity": "CRITICAL",
    "timestamp": 1710009999
  }
]

E. Execute Manual Remediation

POST /api/v1/remediation/execute

Request

{
  "routerId": "RTR-1001",
  "action": "restart_bgp"
}

Response

{
  "executionId": "REM-1009",
  "status": "IN_PROGRESS"
}

7. Database Design

A. Router Metadata Table

CREATE TABLE routers (
    router_id VARCHAR(50) PRIMARY KEY,
    hostname VARCHAR(255),
    ip_address VARCHAR(50),
    vendor VARCHAR(50),
    model VARCHAR(50),
    location VARCHAR(255),
    status VARCHAR(20),
    created_at TIMESTAMP
);

B. Telemetry Table (Time-Series DB)

CREATE TABLE telemetry_metrics (
    router_id VARCHAR(50),
    metric_name VARCHAR(50),
    metric_value DOUBLE,
    ts TIMESTAMP,
    PRIMARY KEY(router_id, ts, metric_name)
);

Recommended DB:

• InfluxDB
  

• OpenTSDB
  

C. Alert Table

CREATE TABLE alerts (
    alert_id VARCHAR(50) PRIMARY KEY,
    router_id VARCHAR(50),
    severity VARCHAR(20),
    message TEXT,
    status VARCHAR(20),
    created_at TIMESTAMP
);

D. Policy Table

CREATE TABLE remediation_policies (
    policy_id VARCHAR(50) PRIMARY KEY,
    policy_name VARCHAR(255),
    condition_expression TEXT,
    action VARCHAR(100),
    severity VARCHAR(20),
    enabled BOOLEAN
);

E. Remediation Execution Table

CREATE TABLE remediation_execution (
    execution_id VARCHAR(50) PRIMARY KEY,
    router_id VARCHAR(50),
    action VARCHAR(100),
    status VARCHAR(50),
    started_at TIMESTAMP,
    completed_at TIMESTAMP,
    rollback_status VARCHAR(50)
);

8. Storage Choices

| Data Type         | Database             |
| ----------------- | -------------------- |
| Router Metadata   | PostgreSQL           |
| Telemetry Metrics | InfluxDB / Cassandra |
| Logs              | Elasticsearch        |
| Alerts            | PostgreSQL           |
| Event Streaming   | Kafka                |
| ML Features       | S3/HDFS              |

9. Microservices

| Service                     | Responsibility                |
| --------------------------- | ----------------------------- |
| API Gateway                 | Authentication, routing       |
| Device Service              | Router metadata management    |
| Telemetry Ingestion Service | Accept telemetry              |
| Detection Engine            | Fault detection               |
| Policy Engine               | Evaluate remediation policies |
| Remediation Service         | Execute preventive actions    |
| Alert Service               | Notifications                 |
| Analytics Service           | Dashboards and trends         |
| ML Prediction Service       | Predict future failures       |

10. Microservices Interaction Flow

Scenario:

CPU spike detected on router.

1. Router sends telemetry
2. Collector publishes to Kafka
3. Detection Engine consumes telemetry
4. Rule matched: CPU > 90%
5. Alert generated
6. Policy Engine checks remediation rules
7. Remediation Service restarts interface
8. Result published back to Kafka
9. Alert Service sends PagerDuty alert
10. Analytics Service updates dashboard

11. Sequence Flow Diagram

Router
   |
   | Telemetry
   v
Collector Service
   |
   | Publish Event
   v
Kafka
   |
   | Consume
   v
Detection Engine
   |
   | Fault Detected
   v
Policy Engine
   |
   | Execute Action
   v
Remediation Service
   |
   | SSH/NETCONF
   v
Router

Detection Engine
   |
   | Create Alert
   v
Alert Service
   |
   | Notify
   v
PagerDuty / Slack / Email

12. Scaling Strategy

Horizontal Scaling

• Multiple collectors
  

• Partitioned Kafka topics
  

• Stateless detection workers
  

Kafka Partitioning

Partition by:

hash(routerId)

Ensures:

• Ordering per router
  

• Balanced processing
  

Telemetry Optimization

Use:

• Compression
  

• Batch ingestion
  

• Sampling
  

• Aggregation windows
  

13. Reliability & Fault Tolerance

Retry Mechanisms

• Exponential backoff
  

• Dead-letter queues
  

Exactly Once Processing

Use:

• Kafka transactions
  

• Idempotent consumers
  

Multi-Region Deployment

Deploy:

• Active-active clusters
  

• Geo-replication
  

14. Security Considerations

• mTLS between routers and collectors
  

• RBAC for remediation APIs
  

• Encrypted telemetry
  

• Audit logs for all actions
  

• Secret management using:
  

  • HashiCorp Vault
    

15. Bottlenecks & Optimizations

| Bottleneck            | Optimization        |
| --------------------- | ------------------- |
| High telemetry volume | Kafka batching      |
| Detection lag         | Stream processing   |
| Alert storms          | Alert deduplication |
| Expensive queries     | Pre-aggregation     |
| Large storage         | Tiered retention    |

16. Interview Deep Dive Questions

1. How will you detect interface flapping efficiently?
   

2. Why choose Kafka over RabbitMQ?
   

3. How would you avoid alert storms?
   

4. How will rollback work during failed remediation?
   

5. How do you guarantee ordering per router?
   

6. How would ML-based prediction integrate?
   

7. How would the system behave during Kafka outages?
   

8. How do you prevent false positives?
   

9. How would you support millions of routers?
   

10. How would you design multi-region failover?
    

Consider this code:

class Counter {
    int count = 0;

    void increment() {
        count++;
    }
}

public class Test {
    public static void main(String[] args) throws Exception {
        Counter counter = new Counter();

        Thread t1 = new Thread(() -> {
            for (int i = 0; i < 100000; i++) {
                counter.increment();
            }
        });

        Thread t2 = new Thread(() -> {
            for (int i = 0; i < 100000; i++) {
                counter.increment();
            }
        });

        t1.start();
        t2.start();

        t1.join();
        t2.join();

        System.out.println(counter.count);
    }
}

At first glance, the answer should obviously be:

200000

Two threads.

Each increments 100000 times.

So why does the output often become:

173421
189302
196774

and almost never exactly 200000?

Let’s break it down.

The Real Problem: count++ Is NOT Atomic

Most developers assume:

count++;

is a single operation.

It is not.

Under the hood, it becomes three separate CPU operations:

1. Read count from memory
2. Increment locally
3. Write updated value back

Equivalent pseudo-operations:

temp = count;
temp = temp + 1;
count = temp;

Now imagine two threads executing simultaneously.

The Race Condition

Suppose:

count = 5

Now both threads execute count++.

Thread Interleaving

StepThread 1Thread 2count1Read 552Read 553Increment to 654Increment to 655Write 666Write 66

Expected result:

7

Actual result:

6

One increment is lost.

This is called a:

⚠️ Lost Update Problem

And because this happens thousands of times across both threads, the final result becomes unpredictable.

Why Does This Happen?

Because:

• Threads share the same heap memory
  

• CPU scheduling is nondeterministic
  

• Context switching can occur anytime
  

• count++ is not thread-safe
  

This entire category of bugs falls under:

⚠️ Race Conditions

A race condition occurs when:

Multiple threads access and modify shared state concurrently without proper synchronization.

How to Fix It

There are multiple ways.

Each has different performance and scalability tradeoffs.

1. Using synchronized

class Counter {
    int count = 0;

    synchronized void increment() {
        count++;
    }
}

Now only one thread can execute increment() at a time.

How It Works

Java uses an intrinsic monitor lock.

Before entering:

increment()

thread acquires the monitor.

Other threads wait.

Pros

✔ Easy to understand

✔ Guarantees correctness

✔ Built into JVM

Cons

❌ Blocking

❌ Context switching overhead

❌ Poor scalability under heavy contention

2. Using ReentrantLock

import java.util.concurrent.locks.ReentrantLock;

class Counter {
    int count = 0;

    ReentrantLock lock = new ReentrantLock();

    void increment() {
        lock.lock();

        try {
            count++;
        } finally {
            lock.unlock();
        }
    }
}

Why Use It?

More flexible than synchronized.

Supports:

• Fair locking
  

• Try lock
  

• Interruptible lock acquisition
  

• Timed waits
  

Pros

✔ More control

✔ Better advanced concurrency handling

Cons

❌ Manual unlock required

❌ Still blocking

3. Using AtomicInteger (Best for Counters)

import java.util.concurrent.atomic.AtomicInteger;

class Counter {
    AtomicInteger count = new AtomicInteger();

    void increment() {
        count.incrementAndGet();
    }
}

Why This Is Better

AtomicInteger uses:

⚡ Compare-And-Swap (CAS)

instead of locking.

What CAS Does

CPU instruction:

if current_value == expected_value
    update
else
    retry

No thread blocking.

No monitor locking.

Extremely efficient.

Internally

while(true) {
   int existing = value;
   int next = existing + 1;

   if(CAS(existing, next))
       break;
}

Why Modern Systems Prefer Atomic Operations

High-performance systems like:

• Kafka
  

• Netty
  

• Aerospike
  

• Cassandra
  

• Redis internals
  

heavily rely on:

• Lock-free algorithms
  

• CAS operations
  

• Atomic primitives
  

because locks become bottlenecks at scale.

Performance Comparison

ApproachThread SafeBlockingScalablePlain int❌❌❌synchronized✅✅MediumReentrantLock✅✅MediumAtomicInteger✅❌High

But AtomicInteger Is Not Always Enough

For extremely high contention systems:

100+ threads
millions of increments/sec

even CAS retries become expensive.

That’s why Java introduced:

⚡ LongAdder

Used internally in:

• ConcurrentHashMap
  

• Metrics systems
  

• High throughput counters
  

It reduces contention using striped counters.

Real-World Backend Engineering Insight

This tiny example explains why distributed systems are hard.

Now imagine:

• Millions of concurrent users
  

• Multiple JVMs
  

• Distributed databases
  

• Replication
  

• Network retries
  

The exact same “lost update” problem appears everywhere:

• Inventory systems
  

• Banking transactions
  

• Payment systems
  

• Distributed counters
  

• Leaderboards
  

Concurrency bugs scale from a single integer → to entire distributed architectures.

Final Takeaway

The biggest lesson is:

Concurrency problems are rarely visible in code syntax.

The bug is hidden inside:

count++;

What looks like one operation is actually multiple CPU-level operations racing against each other.

Understanding this deeply is foundational for:

• Multithreading
  

• JVM internals
  

• High-performance backend engineering
  

• Distributed systems design
  

Because at scale:

Correctness is harder than performance.

#Java #Concurrency #Multithreading #BackendEngineering #DistributedSystems #JVM #SystemDesign

📩 Contact: programmingappliedai@gmail.com

1. Problem Framing

Design a Podcast Tracking Service where users can:

• Follow podcasts

• Track listening progress

• Get updates on new episodes

• Resume playback across devices

✅ 2. Functional Requirements (FR)

1. User can search & discover podcasts

2. User can follow / unfollow podcasts

3. User can view episodes of a podcast

4. User can mark episode as played / in-progress

5. Track listening progress (timestamp-based)

6. Resume playback from last position

7. Notify users on new episode release

8. Show user feed (new episodes from followed podcasts)

9. Support multi-device sync

10. Maintain listening history

⚙️ 3. Non-Functional Requirements (NFR)

1. High write throughput (progress updates are frequent)

2. Low latency reads (resume playback must be instant)

3. High availability (user data must not be lost)

4. Scalability (millions of users, podcasts)

5. Eventual consistency acceptable (for feeds/notifications)

6. Durability of progress tracking

7. Idempotent APIs (progress updates retry-safe)

🧱 4. Core Entities

Key Entities:

• User

• Podcast

• Episode

• UserSubscription

• ListeningProgress

• ListeningHistory

🗄️ 5. Database Design

🔹 1. User

user_id (PK)
name
email
created_at

🔹 2. Podcast

podcast_id (PK)
title
description
publisher
category
created_at

🔹 3. Episode

episode_id (PK)
podcast_id (FK)
title
duration
publish_time
audio_url

🔹 4. UserSubscription

user_id (PK, FK)
podcast_id (PK, FK)
subscribed_at

👉 Composite PK for fast lookup

🔹 5. ListeningProgress (HOT PATH)

user_id (PK)
episode_id (PK)
progress_seconds
last_updated
status (IN_PROGRESS / COMPLETED)

👉 Stored in NoSQL (Cassandra/DynamoDB)

Reason: high write frequency

🔹 6. ListeningHistory

user_id
episode_id
completed_at

👉 Append-only → good for analytics

🗃️ 6. Database Choices

| Use Case               | DB                            |
| ---------------------- | ----------------------------- |
| User, Podcast metadata | MySQL / PostgreSQL            |
| Episodes               | MySQL (or Elastic for search) |
| Listening Progress     | Cassandra / DynamoDB          |
| Feed / Notifications   | Redis + Kafka                 |
| Search                 | Elasticsearch                 |

🔌 7. API Design

🔹 1. Follow Podcast

POST /users/{userId}/follow

Request:
{
  “podcastId”: “p123”
}

🔹 2. Get User Subscriptions

GET /users/{userId}/subscriptions

🔹 3. Get Episodes of Podcast

GET /podcasts/{podcastId}/episodes

🔹 4. Update Listening Progress (CRITICAL API)

POST /progress

Request:
{
  “userId”: “u1”,
  “episodeId”: “e1”,
  “progressSeconds”: 120
}

👉 Must be:

• Idempotent
  

• High throughput
  

🔹 5. Get Resume Playback

GET /progress?userId=u1&episodeId=e1

🔹 6. Get User Feed

GET /users/{userId}/feed

🧩 8. High-Level Architecture

                ┌──────────────┐
                │ API Gateway  │
                └──────┬───────┘
                       │
        ┌──────────────┼──────────────┐
        │              │              │
 User Service   Podcast Service   Progress Service
        │              │              │
        │              │              │
   MySQL DB       MySQL DB       Cassandra
                                      │
                                  Kafka
                                      │
                              Notification Service
                                      │
                                   Redis