Req/Res Pros Elegant and Sinple Stateless Scaleable
Cons Bad for muktple sevrs High Couling Cahing, Cicnbut Breaking
- Redis is an in-memory data structure store.
- It's commonly used as a message broker and caching solution.
- Enables fast read and write operations.
- Can persist dataset by storing it on disk for durability.
- Redis offers two main persistence options:
- AOF (Append-Only File):
- Logs every write operation received by the server.
- Appends each operation to a file.
- This log can be replayed during startup to reconstruct the dataset.
- RDB (Redis DataBase):
- Performs point-in-time snapshots of the dataset at specific intervals.
- Creates a compact single-file representation of the entire Redis dataset.
- This file can be used to restore the dataset in case of failure or restart.
- AOF (Append-Only File):
- Asynchronous Communication: Enables producers to send messages to a queue without immediate processing, enhancing system resilience.
- Message Persistence: Messages are stored until consumed, ensuring reliable delivery even during failures.
- Load Leveling: Balances system load by buffering requests during peak periods and throttling consumption during low traffic.
- Guaranteed Delivery: Provides features like acknowledgments and retries for reliable message delivery.
- Message Ordering: Some queues maintain message order to ensure sequential processing.
- Use Cases: Commonly used for task distribution, event-driven architectures, job scheduling, and reliable communication in distributed systems.
- Example: Leetcode's Submission Evaluation
- Leetcode evaluates user submissions using a separate evaluation service or worker nodes.
- Instead of processing submissions directly on its primary backend, Leetcode delegates this task to evaluation workers.
- When a user submits a solution, Leetcode pushes the evaluation task to a message queue.
- Evaluation workers pick up tasks from the queue when they are ready to process them.
- This asynchronous processing allows Leetcode to handle evaluation tasks efficiently without impacting the responsiveness of its primary backend.
- Publish-Subscribe Model: Publishers send messages to topics without knowledge of subscribers, who express interest in specific topics.
- Loose Coupling: Promotes independence between publishers and subscribers, enabling scalability and flexibility.
- Scalability: Handles large numbers of publishers and subscribers efficiently.
- Real-Time Communication: Well-suited for broadcasting updates and coordinating events in distributed systems.
- Event-Driven Architecture: Key component for reacting to events and messages independently.
- Use Cases: Widely used in real-time analytics, chat applications, IoT platforms, and distributed system architectures.
- Example: 2-Player Chess Game
- Consider a 2-player online chess game with potential high traffic and the need for server scaling.
- In a particular game, each player may be connected to different servers for load balancing purposes.
- When one player makes a move, their server needs to notify the opponent's server about the move.
- Without knowing the recipient server in advance, the player's server publishes the game event (e.g., game ID) to a pub-sub model.
- The pub-sub model determines which server is subscribed to that game ID and immediately notifies it about the move.
- The recipient server then notifies the opponent about the move, ensuring real-time gameplay without server-specific dependencies. Pros
- Scales between muliple services
- Great for Microservices
- Loose Coulping Cons
- Message delivery issue
- Compexity
- Cache or store data in memory.
- Maintain state as per application requirements.
- Not completely reliant on databases.
- Stickiness is required to ensure users remain connected to the same server throughout their session.
- Stickiness is often maintained by routers or load balancers through health checks and server assignments.
- Do not store data in memory.
- Do not require sticky sessions.
- Users belonging to the same group or session may be connected to different servers.
- Rely on message brokers like message queues and pub-sub models like Redis or Kafka for inter-server communication.
Kafka is a distributed streaming platform designed for real-time data processing. It provides a pub-sub messaging model where producers publish messages to topics, and consumers subscribe to topics to receive messages.
- Brokers: Kafka cluster nodes responsible for storing and managing partitions.
- Topics: Logical channels for publishing and subscribing to messages.
- Partitions: Segments of topics that enable parallel processing and data distribution.
- Producers: Applications that publish messages to Kafka topics.
- Consumers: Applications that subscribe to topics and consume messages.
- Consumer Groups: Groups of consumers that work together to consume messages from topics.
- ZooKeeper: Coordination service used for managing Kafka cluster metadata and leader election.
- Kafka is optimized for high throughput and low latency, making it suitable for real-time data processing and event-driven architectures.
- It achieves high throughput by efficiently batching and compressing messages, minimizing network overhead, and leveraging disk-based storage for sequential writes.
- Kafka's design prioritizes low-latency message delivery, enabling near-real-time data processing and stream processing applications.
- Kafka is horizontally scalable, allowing clusters to handle large volumes of data and high throughput.
- Partitions are distributed across multiple brokers for scalability.
- Replication of partitions ensures fault tolerance and data durability.
- Producers write messages to partitions using partitioning strategies like round-robin or key-based partitioning.
- Messages within partitions are ordered, but across partitions, order is not guaranteed.
- Consumers fetch messages from partitions using pull-based mechanisms and maintain their own offsets.
- Kafka employs consumer groups to ensure each partition is consumed by exactly one consumer in the same group.
- The number of partitions determines the maximum parallelism of a topic, not the number of consumers.
- Purpose: Increase consumption rate or achieve higher throughput for a specific topic.
- Mechanism: Each consumer in the group consumes from a different partition.
- Benefits:
- Parallel message processing.
- Faster consumption.
- Purpose: Allow multiple consumers from the different groups to consume the same partition.
- Mechanism: Each consumer group has its own set of consumers.
- Benefits:
- Independent message consumption.
- Flexibility for different processing needs.