This article describes an easy way to retry a failed HTTP client request in GO language. It presents an example of a retryable HTTP Client-Server Communication.

Introduction

HTTP (Hypertext Transfer Protocol) is a widely used application layer protocol, based on a request-response communication model between client and server. Even though HTTP requests use the reliable TCP (Transmission Control Protocol), they can still fail with 4xx or 5xx error codes. These failures can be due to reasons like a bad request format, authentication or authorization errors, network issues, or server problems.

The good news is that in some cases, HTTP clients can recover from failures by retrying the request. However, it's crucial to determine when exactly to retry a failed request, as retries can sometimes make things worse. Deciding which errors to retry is beyond the scope of this document, but the most common retryable HTTP errors include Request Timeout (error code 408) and Internal Server Error (error code 500).

HTTP Client-Server in GO

HTTP client-server communication in Go is facilitated through the net/http package, which provides required tools for creating web servers and making HTTP requests.

Learn how to set up your development environment for Go language programming by following the guide provided here.

Download source code

https://github.com/azam-akram/dev-toolkit-go/tree/master/utils-go/http-utils-go

Retryable HTTP Client:

I used the GO Resty HTTP client (https://github.com/go-resty/resty), which offers a very easy-to-use and configurable retry mechanism.

package main

import (
	"fmt"
	"net/http"
	"time"

	"github.com/go-resty/resty/v2"
)

const (
	RETRY_COUNT                 = 3
	RETRY_MIN_WAIT_TIME_SECONDS = 5
	RETRY_MAX_WAIT_TIME_SECONDS = 15
)

func main() {
	client := resty.New().
		SetRetryCount(RETRY_COUNT).
		SetRetryWaitTime(RETRY_MIN_WAIT_TIME_SECONDS * time.Second).
		SetRetryMaxWaitTime(RETRY_MAX_WAIT_TIME_SECONDS * time.Second).
		AddRetryCondition(
			func(r *resty.Response, err error) bool {
				return r.StatusCode() == http.StatusRequestTimeout ||
					r.StatusCode() >= http.StatusInternalServerError
			},
		)

	resp, err := client.R().Get("http://localhost:8989/")
	if err != nil {
		fmt.Println("Error occurred while making request:", err)
		return
	}

	fmt.Println("Status:", resp.Status())
	fmt.Println("Response Body:", resp.String())
}

Let’s take a look at the retry configuration parameters in the above HTTP client.
RETRY_COUNT: Maximum retry attempts the client makes in case of an error response.
RETRY_MIN_WAIT_TIME_SECONDS: Minimum wait time (in seconds) between two consecutive retries.
RETRY_MAX_WAIT_TIME_SECONDS : Maximum wait time (in seconds) between two consecutive retries.

Another important thing is to configure which errors we want to trigger retry requests. In the above example, the client attempts a retry in case it receives either Request Timeout (error code 408) or Server Internal Error (error code 500) from the server.

Resty client supports quite seamless retry request.

When we run the above code (provided HTTP server is also up and running), the client gives up after RETRY_COUNT of attempts, and prints,

Status:  500 Internal Server Error
Response:  Something went wrong

HTTP Server:

To demo HTTP request retry, we write our own HTTP server, which returns fake error response up to a specific (MAX_COUNTER = 4) number of times.

package main

import (
	"encoding/json"
	"fmt"
	"net/http"
)

const (
	LISTENING_PORT = 8989
	MAX_COUNTER    = 4
)

type Profile struct {
	Name    string   `json:"name"`
	Hobbies []string `json:"hobbies"`
}

var (
	counter = 0
	handler Handler
)

type Handler interface {
	Handle(w http.ResponseWriter, r *http.Request)
}

type HTTPServer struct{}

func NewHTTPServer() Handler {
	if handler == nil {
		handler = &HTTPServer{}
	}
	return handler
}

func (h HTTPServer) Handle(w http.ResponseWriter, r *http.Request) {
	fmt.Printf("HTTPServer::Handler counter = %d\n", counter)

	if counter < MAX_COUNTER {
		http.Error(w, "Something went wrong", http.StatusInternalServerError)
		counter++
		return
	}

	fmt.Printf("HTTPServer: recovered from problem after %d failed attempts\n", counter)

	profile := &Profile{
		Name:    "User",
		Hobbies: []string{"Sports", "Walk"},
	}

	js, err := json.Marshal(profile)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Type", "application/json")
	w.Write(js)
}

func main() {
	fmt.Printf("Server is listening at port %d\n", LISTENING_PORT)

	h := NewHTTPServer()
	http.HandleFunc("/", h.Handle)

	if err := http.ListenAndServe(fmt.Sprintf(":%d", LISTENING_PORT), nil); err != nil {
		fmt.Println("Error starting server:", err)
	}
}

This is console logs of the HTTP server in case if client gives up after a fixed number of retries,

Server is listening at port 8989
HTTPServer::Handler counter =  0
HTTPServer::Handler counter =  1
HTTPServer::Handler counter =  2
HTTPServer::Handler counter =  3

A trick to recover HTTP server from “internal server error”:

We can make a slight change in HTTP server to set a value of MAX_COUNTER smaller than HTTP client RETRY_COUNT, which means the client continues to retry until the server recovers from “internal server error”.

In the server code set,

const MAX_COUNTER = 2

Just to remind you that HTTP client has configured,

RETRY_COUNT = 3

This will allow the server to move out of if-condition,

if COUNTER < MAX_COUNTER {
..
}
fmt.Printf("HTTPServer: recovered from problem after %d failed attempts", COUNTER)
..

And return a successful response consisting of a “user profile” JSON payload.

Server console prints:

Server is listening at port 8989
HTTPServer::Handler counter =  0
HTTPServer::Handler counter =  1
HTTPServer::Handler counter =  2
HTTPServer: recovered from problem after 2 failed attempts

Client console prints:

Status:  200 OK
Response:  {"Name":"User","Hobbies":["Sports","walk"]}

Final words:

Go Resty client makes it super easy to embed a retry mechanism in an HTTP client. However, we should be aware of which error to retry requests.