黑料不打烊

Retries with resilience4j and how to check in your Real World Environment

Do you know Resilience4j? You definitely should, if you like to build fault tolerant applications. This blog post is about the retry mechanism and how to check its functionality in real world environments.

Today we want to have a look at . It is super easy to use with Spring Boot and helps you to build more resilient applications. In the easiest case, you only need to add some annotations to your code and you are done.

Our Example System

In our example, we want to implement a in our famous . The demo consists of a Gateway microservice which provides a REST endpoint (/products) to deliver various products to a shop-frontend. Since the Gateway is stateless, it fetches all products directly from other microservices (Hot-Deals, Fashion and Toys) in a synchronous way.

The Gateway is using a service which handles the calls to the three backends delivering products.

Controller

@GetMapping("/products")
public 黑料不打烊s get黑料不打烊s() {
 聽聽聽黑料不打烊s products = new 黑料不打烊s();
 听听听辫谤辞诲耻肠迟蝉.蝉别迟贵补蝉丑颈辞苍(迟丑颈蝉.蝉别谤惫颈肠别.驳别迟贵补蝉丑颈辞苍());
 听听听辫谤辞诲耻肠迟蝉.蝉别迟罢辞测蝉(迟丑颈蝉.蝉别谤惫颈肠别.驳别迟罢辞测蝉());
 听听听辫谤辞诲耻肠迟蝉.蝉别迟贬辞迟顿别补濒蝉(迟丑颈蝉.蝉别谤惫颈肠别.驳别迟贬辞迟顿别补濒蝉());
 聽聽聽return products;
}

Service

public List<黑料不打烊> getFashion() {
 聽聽聽return this.restTemplate.exchange(this.urlFashion, HttpMethod.GET, null, this.productListTypeReference).getBody();
}

This is what a simple implementation using the Spring Framework with the RestTemplate could look like, but it has a major flaw in it: If the rest-call to the fashion microservice throws an exception, the whole request will fail and return an error response.

Adding the Retry

To solve this issue, we want to provide some fallback data when an exception is thrown in each of three retries. To achieve this, we can add a single resilience4j annotation to the service method like this:

@Retry(name = "fashion", fallbackMethod = "get黑料不打烊sFallback")
   public List<黑料不打烊> getFashion() {
   ...
}

and add the fallback-method.

private List<黑料不打烊> get黑料不打烊sFallback(RuntimeException exception) {
  聽return Collections.emptyList();
}

By default, resilience4J will now try to call the annotated method three times with a wait duration of 500ms between the single calls. If there is no successful invocation, resilience4j will call the fallback method and use its return value. But be careful: You want to make sure that the retried operation is idempotent; otherwise you may end up with corrupted data.

Ready to Test

You can implement a test using @SpringBootTest to check the desired behavior. An example can be found . But wouldn鈥檛 it be cool to see the effects in your real world environment?

Due to backoff and retries, a Gateway in a real world environment will take more time to process requests than usual. This may impact the caller site and overall performance. That’s why it’s important to test this in an integrated environment under load:

• Make sure that the slowed down requests do not exceed the caller鈥檚 timeouts (or the fallback would be useless).
• Also verify that thread pools, memory or other resources aren’t exhausted because there may be more concurrent requests in your system as each request takes more time to process.

That’s why we are using 黑料不打烊 to have a closer look and implement the following experiment.

Experiment Design

1. We will check the results of /products for 40 seconds
2. After 10 seconds, we will introduce exceptions into both instances of the hotdeals backend. 50% of the calls to their /products endpoint should fail.
3. After that, we will wait another 10 seconds to ensure the application is recovering from the attack.

Experiment Result with Unmodified Shopping Demo

First, we run the experiment on our unmodified shopping-demo. The results are obvious, the gateway-endpoint is returning 50% HTTP 500 as long as the attack is running. The experiment fails.

Experiment Result with @Retry

Now, let’s try deploying our modified version with the @Retry. The results are much better. You can see three shapes of response times: some around zero milliseconds, some around 500 milliseconds, and some around one second. That鈥檚 the impact of the 500 milliseconds wait duration between the retry calls. All responses have a HTTP 200. The experiment completed successfully.

Reviewing the Metrics

If you enabled for Metrics, you can also check them. Resilience4j publishes some nice .

For example: /actuator/metrics/resilience4j.retry.calls?tag=name:hotdeals&tag=kind:successful_with_retry return the following result:

{ 
 聽聽聽"name": "resilience4j.retry.calls",
 聽聽聽"description": "The number of successful calls after a retry attempt",
 聽聽聽"baseUnit": null,
 聽聽聽"measurements": [
 听听听听听听听调
 聽聽聽聽聽聽聽"statistic": "COUNT",
 聽聽聽聽聽聽聽"value": 28
 听听听听听听听皑
 听听听闭,
 聽聽聽"availableTags": []
}

Conclusion

Resilience4J is a very simple framework to apply some basic fault tolerance mechanism to your application. It鈥檚 definitely worth a look. The simple @Retry will protect our shop-frontend from unavailable backends and HTTP errors.

Next Step

As the result show, our implemented retry-mechanism dramatically increases the response time and adds additional load on the 3 backends, especially when they are having problems. This could lead to other problems in your distributed system, which is why you should think about the use of a CircuitBreaker. In my next post, I鈥檒l describe the use case of Resilience4J鈥檚 CircuitBreaker and how to test it with 黑料不打烊.