Skip to content

Commit b8f51f0

Browse files
sanurahsanurah
committed
Initial commit backpressure
temp
1 parent d823283 commit b8f51f0

File tree

5 files changed

+331
-0
lines changed

5 files changed

+331
-0
lines changed

Diff for: ‎backpressure/README.md

+303
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,303 @@
1+
---
2+
title: "Backpressure Pattern in Java: controlling data streams from producers to consumer inorder to prevent overwhelming the consumer"
3+
shortTitle: Backpressure
4+
description: "Explore the Backpressure design pattern in Java with detailed examples. Learn how it helps by preventing system overload, ensuring stability and optimal performance by matching data flow to the consumer’s processing capability."
5+
category: Concurrency
6+
language: en
7+
tag:
8+
- Decoupling
9+
- Event-driven
10+
- Reactive
11+
---
12+
13+
## Intent of the Backpressure Design Pattern
14+
15+
The Backpressure Design Pattern is a strategy used in software systems (especially in data streaming, reactive programming, and distributed systems)
16+
to handle situations where a fast producer overwhelms a slow consumer. The intent is to prevent system instability, resource exhaustion, or crashes by managing the flow of data between components.
17+
18+
## Detailed Explanation of Backpressure Pattern with Real-World Examples
19+
20+
### Real-world examples
21+
22+
#### 1. Real-Time Data Streaming (Reactive Systems)
23+
- **Stock Market Data**
24+
- High-frequency trading systems generate millions of price updates per second, but analytics engines can't process them all in real time.
25+
- Backpressure mechanisms (e.g., in RxJava, Akka, Reactor) throttle or drop excess data to avoid overwhelming downstream systems.
26+
- **IoT Sensor DataQ**
27+
- Thousands of IoT devices (e.g., smart factories, wearables) send continuous telemetry, but cloud processing has limited capacity.
28+
- Reactive frameworks apply backpressure to buffer, drop, or slow down data emission.
29+
30+
#### 2. Message Queues (Kafka, RabbitMQ)
31+
- **E-Commerce Order Processing**
32+
- During flash sales (e.g., Black Friday), order requests spike, but payment and inventory systems can’t keep up.
33+
- Message queues like Kafka and RabbitMQ use, Limited queue sizes to drop or reject messages when full or Consumer acknowledgments to slow producers if consumers lag.
34+
- **Log Aggregation**
35+
- Microservices generate massive logs, but centralized logging (E.g.: ELK Stack) can’t ingest them all at once.
36+
- Kafka applies backpressure by pausing producers when consumers are slow.
37+
38+
#### 3. Stream Processing (Apache Flink, Spark)
39+
- **Social Media Trends (Twitter, TikTok)**
40+
- Viral posts create sudden spikes in data, but trend analysis is computationally expensive.
41+
- Backpressure in Spark Streaming prioritizes recent data and discards older, less relevant updates.
42+
- **Fraud Detection in Banking**
43+
- Millions of transactions flow in, but fraud detection models take time to analyze each one.
44+
- slow down ingestion if processing lags (Throttling), save progress to recover from backpressure-induced delays (Checkpointing).
45+
46+
### In plain words
47+
48+
The Backpressure design pattern is a flow control mechanism that prevents overwhelming a system by regulating data production based on the consumer’s processing capacity.
49+
50+
### Wikipedia says
51+
52+
Back pressure (or backpressure) is the term for a resistance to the desired flow of fluid through pipes. Obstructions or tight bends create backpressure via friction loss and pressure drop.
53+
54+
In distributed systems in particular event-driven architecture, back pressure is a technique to regulate flow of data, ensuring that components do not become overwhelmed.
55+
56+
### Architectural Diagram
57+
![backpressure](./etc/backpressure.png)
58+
59+
## Programmatic Example of Backpressure Pattern in Java
60+
61+
First we need to identify the Event on which we need the pub-sub methods to trigger.
62+
For example:
63+
64+
- Sending alerts based on the weather events such as earthquakes, floods and tornadoes
65+
- Sending alerts based on the temperature
66+
- Sending an email to different customer support emails when a support ticket is created.
67+
68+
The Message class below will hold the content of the message we need to pass between the publisher and the subscribers.
69+
70+
```java
71+
public record Message(Object content) {
72+
}
73+
74+
```
75+
76+
The Topic class will have the topic **name** based on the event
77+
78+
- Weather events TopicName WEATHER
79+
- Weather events TopicName TEMPERATURE
80+
- Support ticket created TopicName CUSTOMER_SUPPORT
81+
- Any other custom topic depending on use case
82+
- Also, the Topic contains a list of subscribers that will listen to that topic
83+
84+
We can add or remove subscribers from the subscription to the topic
85+
86+
```java
87+
public class Topic {
88+
89+
private final TopicName name;
90+
private final Set<Subscriber> subscribers = new CopyOnWriteArraySet<>();
91+
//...//
92+
}
93+
```
94+
95+
Then we can create the publisher. The publisher class has a set of topics.
96+
97+
- Each new topic has to be registered in the publisher.
98+
- Publish method will publish the _Message_ to the corresponding _Topic_.
99+
100+
```java
101+
public class PublisherImpl implements Publisher {
102+
103+
private static final Logger logger = LoggerFactory.getLogger(PublisherImpl.class);
104+
private final Set<Topic> topics = new HashSet<>();
105+
106+
@Override
107+
public void registerTopic(Topic topic) {
108+
topics.add(topic);
109+
}
110+
111+
@Override
112+
public void publish(Topic topic, Message message) {
113+
if (!topics.contains(topic)) {
114+
logger.error("This topic is not registered: {}", topic.getName());
115+
return;
116+
}
117+
topic.publish(message);
118+
}
119+
}
120+
```
121+
122+
Finally, we can Subscribers to the Topics we want to listen to.
123+
124+
- For WEATHER topic we will create _WeatherSubscriber_
125+
- _WeatherSubscriber_ can also subscribe to TEMPERATURE topic
126+
- For CUSTOMER_SUPPORT topic we will create _CustomerSupportSubscribe_
127+
- Also to demonstrate the async behavior we will create a _DelayedWeatherSubscriber_ who has a 0.2 sec processing deplay
128+
129+
All classes will have a _onMessage_ method which will take a Message input.
130+
131+
- On message method will verify the content of the message is as expected
132+
- After content is verified it will perform the operation based on the message
133+
- _WeatherSubscriber_ will send a weather or temperature alert based on the _Message_
134+
- _CustomerSupportSubscribe_will send an email based on the _Message_
135+
- _DelayedWeatherSubscriber_ will send a weather alert based on the _Message_ after a delay
136+
137+
```java
138+
public interface Subscriber {
139+
void onMessage(Message message);
140+
}
141+
```
142+
143+
And here is the invocation of the publisher and subscribers.
144+
145+
```java
146+
public static void main(String[] args) throws InterruptedException {
147+
148+
final String topicWeather = "WEATHER";
149+
final String topicTemperature = "TEMPERATURE";
150+
final String topicCustomerSupport = "CUSTOMER_SUPPORT";
151+
152+
// 1. create the publisher.
153+
Publisher publisher = new PublisherImpl();
154+
155+
// 2. define the topics and register on publisher
156+
Topic weatherTopic = new Topic(topicWeather);
157+
publisher.registerTopic(weatherTopic);
158+
159+
Topic temperatureTopic = new Topic(topicTemperature);
160+
publisher.registerTopic(temperatureTopic);
161+
162+
Topic supportTopic = new Topic(topicCustomerSupport);
163+
publisher.registerTopic(supportTopic);
164+
165+
// 3. Create the subscribers and subscribe to the relevant topics
166+
// weatherSub1 will subscribe to two topics WEATHER and TEMPERATURE.
167+
Subscriber weatherSub1 = new WeatherSubscriber();
168+
weatherTopic.addSubscriber(weatherSub1);
169+
temperatureTopic.addSubscriber(weatherSub1);
170+
171+
// weatherSub2 will subscribe to WEATHER topic
172+
Subscriber weatherSub2 = new WeatherSubscriber();
173+
weatherTopic.addSubscriber(weatherSub2);
174+
175+
// delayedWeatherSub will subscribe to WEATHER topic
176+
// NOTE :: DelayedWeatherSubscriber has a 0.2 sec delay of processing message.
177+
Subscriber delayedWeatherSub = new DelayedWeatherSubscriber();
178+
weatherTopic.addSubscriber(delayedWeatherSub);
179+
180+
// subscribe the customer support subscribers to the CUSTOMER_SUPPORT topic.
181+
Subscriber supportSub1 = new CustomerSupportSubscriber();
182+
supportTopic.addSubscriber(supportSub1);
183+
Subscriber supportSub2 = new CustomerSupportSubscriber();
184+
supportTopic.addSubscriber(supportSub2);
185+
186+
// 4. publish message from each topic
187+
publisher.publish(weatherTopic, new Message("earthquake"));
188+
publisher.publish(temperatureTopic, new Message("23C"));
189+
publisher.publish(supportTopic, new Message("[email protected]"));
190+
191+
// 5. unregister subscriber from TEMPERATURE topic
192+
temperatureTopic.removeSubscriber(weatherSub1);
193+
194+
// 6. publish message under TEMPERATURE topic
195+
publisher.publish(temperatureTopic, new Message("0C"));
196+
197+
/*
198+
* Finally, we wait for the subscribers to consume messages to check the output.
199+
* The output can change on each run, depending on how long the execution on each
200+
* subscriber would take
201+
* Expected behavior:
202+
* - weatherSub1 will consume earthquake and 23C
203+
* - weatherSub2 will consume earthquake
204+
* - delayedWeatherSub will take longer and consume earthquake
205+
* - supportSub1, supportSub2 will consume [email protected]
206+
* - the message 0C will not be consumed because weatherSub1 unsubscribed from TEMPERATURE topic
207+
*/
208+
TimeUnit.SECONDS.sleep(2);
209+
}
210+
```
211+
212+
Program output:
213+
214+
Note that the order of output could change everytime you run the program.
215+
The subscribers could take different time to consume the message.
216+
217+
```
218+
14:01:45.599 [ForkJoinPool.commonPool-worker-6] INFO com.iluwatar.publish.subscribe.subscriber.CustomerSupportSubscriber -- Customer Support Subscriber: 1416331388 sent the email to: [email protected]
219+
14:01:45.599 [ForkJoinPool.commonPool-worker-4] INFO com.iluwatar.publish.subscribe.subscriber.WeatherSubscriber -- Weather Subscriber: 1949521124 issued message: 23C
220+
14:01:45.599 [ForkJoinPool.commonPool-worker-2] INFO com.iluwatar.publish.subscribe.subscriber.WeatherSubscriber -- Weather Subscriber: 60629172 issued message: earthquake
221+
14:01:45.599 [ForkJoinPool.commonPool-worker-5] INFO com.iluwatar.publish.subscribe.subscriber.CustomerSupportSubscriber -- Customer Support Subscriber: 1807508804 sent the email to: [email protected]
222+
14:01:45.599 [ForkJoinPool.commonPool-worker-1] INFO com.iluwatar.publish.subscribe.subscriber.WeatherSubscriber -- Weather Subscriber: 1949521124 issued message: earthquake
223+
14:01:47.600 [ForkJoinPool.commonPool-worker-3] INFO com.iluwatar.publish.subscribe.subscriber.DelayedWeatherSubscriber -- Delayed Weather Subscriber: 2085808749 issued message: earthquake
224+
```
225+
226+
## When to Use the Backpressure Pattern
227+
228+
- Event-Driven Systems
229+
- Use Pub/Sub when your system relies on events (e.g., user registration, payment completion).
230+
- Example: After a user registers, send a welcome email and log the action simultaneously.
231+
232+
- Asynchronous Communication
233+
- When tasks can be performed without waiting for immediate responses.
234+
- Example: In an e-commerce app, notify the warehouse and the user after a successful order.
235+
236+
- Decoupling Components
237+
- Ideal for systems where producers and consumers should not depend on each other.
238+
- Example: A logging service listens for logs from multiple microservices.
239+
240+
- Scaling Systems
241+
- Useful when you need to scale services without changing the core application logic.
242+
- Example: Broadcasting messages to thousands of clients (chat applications, IoT).
243+
244+
- Broadcasting Notifications
245+
- When a message should be delivered to multiple receivers.
246+
- Example: Sending promotional offers to multiple user devices.
247+
248+
- Microservices Communication
249+
- Allow independent services to communicate without direct coupling.
250+
- Example: An order service publishes an event, and both the billing and shipping services process it.
251+
252+
## When to avoid the Backpressure Pattern
253+
254+
- Simple applications where direct calls suffice.
255+
- Strong consistency requirements (e.g., banking transactions).
256+
- Low-latency synchronous communication needed.
257+
258+
## Benefits and Trade-offs of Backpressure Pattern
259+
260+
### Benefits:
261+
262+
- Decoupling
263+
- Publishers and subscribers are independent of each other.
264+
- Publishers don’t need to know who the subscribers are, and vice versa.
265+
- Changes in one component don’t affect the other.
266+
- Scalability
267+
- New subscribers can be added without modifying publishers.
268+
- Supports distributed systems where multiple services consume the same events.
269+
- Dynamic Subscription
270+
- Subscribers can subscribe/unsubscribe at runtime.
271+
- Enables flexible event-driven architectures.
272+
- Asynchronous Communication
273+
- Publishers and subscribers operate independently, improving performance.
274+
- Useful for background processing (e.g., notifications, logging).
275+
- Broadcast Communication
276+
- A single event can be consumed by multiple subscribers.
277+
- Useful for fan-out scenarios (e.g., notifications, analytics).
278+
- Resilience & Fault Tolerance
279+
- If a subscriber fails, others can still process messages.
280+
- Message brokers (e.g., Kafka, RabbitMQ) can retry or persist undelivered messages.
281+
282+
### Trade-offs:
283+
284+
- Complexity in Debugging
285+
- Since publishers and subscribers are decoupled, tracing event flow can be difficult.
286+
- Requires proper logging and monitoring tools.
287+
- Message Ordering & Consistency
288+
- Ensuring message order across subscribers can be challenging (e.g., Kafka vs. RabbitMQ).
289+
- Some systems may process events out of order.
290+
- Potential Latency
291+
- Asynchronous processing introduces delays compared to direct calls.
292+
- Not ideal for real-time synchronous requirements.
293+
294+
## Related Java Design Patterns
295+
296+
* [Observer Pattern](https://github.com/sanurah/java-design-patterns/blob/master/observer/): Both involve a producer (subject/publisher) notifying consumers (observers/subscribers). Observer is synchronous & tightly coupled (observers know the subject). Pub-Sub is asynchronous & decoupled (via a message broker).
297+
* [Mediator Pattern](https://github.com/sanurah/java-design-patterns/blob/master/mediator/): A mediator centralizes communication between components (like a message broker in Pub-Sub). Mediator focuses on reducing direct dependencies between objects. Pub-Sub focuses on broadcasting events to unknown subscribers.
298+
299+
## References and Credits
300+
301+
* [Apache Kafka – Pub-Sub Model](https://kafka.apache.org/documentation/#design_pubsub)
302+
* [Microsoft – Backpressure Pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber)
303+
* [Martin Fowler – Event-Driven Architecture](https://martinfowler.com/articles/201701-event-driven.html)

Diff for: ‎backpressure/etc/backpressure.png

18 KB
Loading

Diff for: ‎backpressure/pom.xml

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<modelVersion>4.0.0</modelVersion>
6+
<parent>
7+
<groupId>com.iluwatar</groupId>
8+
<artifactId>java-design-patterns</artifactId>
9+
<version>1.26.0-SNAPSHOT</version>
10+
</parent>
11+
12+
<artifactId>backpressure</artifactId>
13+
14+
<properties>
15+
<maven.compiler.source>21</maven.compiler.source>
16+
<maven.compiler.target>21</maven.compiler.target>
17+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
18+
</properties>
19+
20+
</project>

Diff for: ‎backpressure/src/main/java/com/iluwatar/App.java

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
package com.iluwatar;
2+
3+
public class App {
4+
public static void main(String[] args) {
5+
System.out.println("Hello, World!");
6+
}
7+
}

Diff for: ‎pom.xml

+1
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,7 @@
234234
<module>version-number</module>
235235
<module>virtual-proxy</module>
236236
<module>visitor</module>
237+
<module>backpressure</module>
237238
</modules>
238239
<repositories>
239240
<repository>

0 commit comments

Comments
 (0)