[Main] [Projects] [Book Recs] [Talks] [Hire]

How I Write HTTP Clients

I have written a lot of HTTP clients that talk to 3rd party APIs over the years and there have been some patterns that have emerged and so I thought I would share them in hopes they help others. The major concepts I want to highlight are:

  • Composing Client Options
  • JSON Encoding/Decoding to Go Types
  • HTTP Request Signing
  • Convenience Methods
  • Binary Readers

To illustrate these concepts I am going to pick through the Infogram Go client I created specifically for this article since one did not exist.

Composing Client Options

Most HTTP clients share 3 things in common with each other.

  1. They use http.Client to make HTTP calls
  2. They talk to a base endpoint URL of some kind
  3. They require some sort of credentials so the API can authenticate requests

Starting with the Client struct we can define what is needed:

type Client struct {
	HTTPClient *http.Client
	Endpoint   string
	APIKey     string
	APISecret  string
}

This is a pretty common starting point for all HTTP clients and this also allows the user to supply their own http.Client and endpoint based on their needs.

httpClient := http.Client{
	Timeout: 10 * time.Second,
}

client := infogram.Client(
	HTTPClient: &httpClient,
	Endpoint: "https://example.com/infogram",
	APIKey: "api-key",
	APISecret: "api-secret",
)

For convenience, I use sync.Once to ensure the first time to Do() is called there is a configured HTTPClient and Endpoint set to avoid nil pointer exceptions and missing endpoints.

Another benefit of this approach is making the package easily testable without needing to make HTTP calls over the internet to the Infogram API. I can create a new http.Client from a httptest.Server.

mockInfogramAPIServer := httptest.NewServer(...)
defer mockInfogramAPIServer.Close()

client := infogram.Client(
	HTTPClient: mockInfogramAPIServer.Client(),
	Endpoint: mockInfogramAPIServer.URL,
	APIKey: "api-key",
	APISecret: "api-secret",
)

JSON Encoding/Decoding to Go Types

JSON encoding/decoding is relatively straightforward if you can utilize struct tags, but some times there are other Go types you want to support that do not support JSON encoding/decoding via struct tags out of the box. However, we can implement the JSONMarshaler and JSONUnmarshaler interfaces to support those types. In the case of Infograms API there are some URLs that get returned in the JSON and instead of just keeping them as string types I would like them to be url.URL types.

// Theme defines the type returned by the Infogram API
type Theme struct {
	Id        int
	Title     string
	Thumbnail *url.URL
}

// MarshalJSON implements json.Marshaler
func (t *Theme) MarshalJSON() ([]byte, error) {
	data := make(map[string]interface{})

	data["id"] = t.Id
	data["title"] = t.Title
	data["thumbnail_url"] = t.Thumbnail.String()

	return json.Marshal(data)
}

// UnmarshalJSON implements json.Unmarshaler
func (t *Theme) UnmarshalJSON(bytes []byte) error {
	data := make(map[string]interface{})

	if val, found := data["id"]; found {
		v, ok := val.(int)
		if !ok {
			return errors.New("id needs to be an int")
		}
		t.Id = v
	}
	if val, found := data["title"]; found {
		v, ok := val.(string)
		if !ok {
			return errors.New("title needs to be an string")
		}
		t.Title = v
	}
	if val, found := data["thumbnail_url"]; found {
		v, ok := val.(string)
		if !ok {
			return errors.New("thumbnail_url needs to be an string")
		}
		var err error
		t.Thumbnail, err = url.Parse(v)
		if err != nil {
			return errors.New("thumbnail_url needs to be a parsable URL")
		}
	}

	return nil
}

The Infographic type is similar, but I omitted it for brevity.

HTTP Request Signing

One of the most common ways to authorize HTTP requests to an API is to specify a token of some kind that the API provides to you. This token is set in one of two places.

  1. Query Parameter
  2. Authorization HTTP Header

Query Parameter

Merely attaching a predefined query parameter to every call that the API specifies is a simple way to authorize requests.

http://exmaple.com/api/v1?api_key=<token>

The Go way:

req, _ := http.NewRequest(http.MethodGet, "http://exmaple.com/api/v1?api_key=<token>", nil)

Authorization HTTP Header

The other, more common, method is setting the Authorization HTTP Header with the same token.

GET /api/v1 HTTP/1.1
Host: example.com
Authorization: Bearer <token>

The Go way:

req, _ := http.NewRequest(http.MethodGet, "http://exmaple.com/api/v1", nil)
req.Header.Add("Authorization", "Bearer <token>")

While these methods work just fine, there is another method you might come across called request signing. I specifically like to implement this step in its own method to make building and performing requests more composable if the user wants to manipulate more of the http.Request itself.

The specific rules for request signing depends on the API, but Infogram lists their implmentation to be recreated.

// SignRequest adds the `api_key` and `api_sig` query parameter in accordance with https://developers.infogr.am/rest/request-signing.html
func (c *Client) SignRequest(req *http.Request) error {
	var data url.Values

	switch req.Method {
	case http.MethodGet, http.MethodDelete:
		data = req.URL.Query()
	default:
		req.ParseForm()
		data = req.Form
	}

	query.Set("api_key", c.apiKey)

	var sig bytes.Buffer
	sig.WriteString(req.Method)
	sig.WriteByte('&')
	sig.WriteString(req.URL.EscapedPath())
	sig.WriteByte('&')

	...

	h := hmac.New(sha1.New, []byte(c.APISecret))
	h.Write(sig.Bytes())
	signature := h.Sum(nil)

	data.Add("api_sig", base64.StdEncoding.EncodeToString(signature))

	switch req.Method {
	case http.MethodGet, http.MethodDelete:
		req.URL.RawQuery = data.Encode()
	default:
		req.Form = data
	}

	return nil
}

The result of calling this adds a api_sig query parameter with the signature of the request data signed by the API secret provided by Infogram.

Convenience Methods

Now all that is left is the API calls themselves and for that we need to create methods to invoke them. What I like to do is to keep these as small as possible and move as much of the boilerplate logic into their own functions.

The main endpoints we need for Infogram follow the same pattern.

// Infographics fetches the list of infographics
func (c *Client) Infographics() ([]Infographic, error) {
    // 1. Create Infographics request
    // 2. Sign the request
    // 3. Send the request
    // 4. Decode the response and handle API errors
    // 5. Return the resulting Go type
}

// Infographics fetches a single infographic by identification number
func (c *Client) Infographic(id string) (*Infographic, error) {...}

// Infographics fetches a available themes to use for infographics
func (c *Client) Themes() ([]Theme, error) {...}

Instead of putting most of the duplicated logic into each function we can create more generic function to handle them like we did with SignRequest() and just call it with the appropriate parameters.

func (c *Client) Do(req *http.Request, v interface{}) (*http.Response, error) {
	req.RequestURI = ""

	res, err := c.HTTPClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer res.Body.Close()

	if res.StatusCode > 299 {
		_, err := io.ReadAll(res.Body)
		if err != nil {
			return nil, err
		}

		return res, fmt.Errorf("http error code: %d", res.StatusCode)
	}

	if v != nil {
		if w, ok := v.(io.Writer); ok {
			io.Copy(w, res.Body)
		} else {
			decErr := json.NewDecoder(res.Body).Decode(v)
			if decErr == io.EOF {
				decErr = nil // ignore EOF errors caused by empty response body
			}
			if decErr != nil {
				return nil, decErr
			}
		}
	}

	return res, nil
}

Do() now handle most of the heavy lifting of preparing requests, sending them, and decoding the API responses. With these functions our convenience methods for each endpoint becomes small and easily repeatable for adding additional endpoints. This also allows to ensure good test coverage over each of the three methods used inside since the convenience methods just call them in a specific order. Notice it also returns the raw http.Response to the caller as well in case that is needed for any reason. I did not want to limit the caller's ability to inspect that response.

// Infographics fetches the list of infographics
func (c *Client) Infographics() ([]Infographic, error) {
	req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/%s/%s", c.Endpoint, "infographics", id), nil)
	if err != nil {
		return nil, fmt.Errorf("new infographic request: %w", err)
	}

	err = c.SignRequest(req)
	if err != nil {
		return nil, err
	}

	var infographic Infographic
	_, err = c.Do(req, &infographic)
	if err != nil {
		return nil, fmt.Errorf("performing infographics request: %w", err)
	}

	return &infographic, nil
}

Binary Readers

Infogram offers more than just JSON responses to their API. When interacting with the /infographics endpoint specifically you can request to have a PDF, PNG, or HTML version of the infographic returned to you. We can create an io.Reader for each one of these formats for the Infographic type to make this downloading possible.

type Infographic struct {...}

func (i *Infographic) reader(client *Client, format string) (io.Reader, error) {
	req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/%s/%s?format=%s", client.Endpoint, "infographics", i.Id, format), nil)
	if err != nil {
		return nil, fmt.Errorf("new infographic PDF reader request: %w", err)
	}

	err = client.SignRequest(req)
	if err != nil {
		return nil, err
	}

	res, err := client.HTTPClient.Do(req)
	if err != nil {
		return nil, err
	}

	return res.Body, nil
}

// PDFReader returns an io.Reader of the Infographic in PDF format
func (i *Infographic) PDFReader(client *Client) (io.Reader, error) {
	return i.reader(client, "pdf")
}

// PNGReader returns an io.Reader of the Infographic in PNG format
func (i *Infographic) PNGReader(client *Client) (io.Reader, error) {
	return i.reader(client, "png")
}

// HTMLReader returns an io.Reader of the Infographic in HTML format
func (i *Infographic) HTMLReader(client *Client) (io.Reader, error) {
	return i.reader(client, "html")
}

Since these functions are flexible and composable I can do this and still use the SignRequest() after than and then perform a standard http.Client.Do() call and just return the http.Response.Body back to the call to complete the reading of the response to a file, buffer, or anything else that accepts a io.Reader.

// get infographic 100 from the API
ig100 _ := client.Infographic(100)

// get the PDF from the API as an io.Reader
ig100reader, _ := ig100.PDFReader(client)

// create a local file for the PDF infographic
pdf, _ := os.Create("infographic-100.pdf")

// copy the results from the API to the local file
io.Copy(pdf, ig100reader)

Conclusion

Now I have a complete and composable Infogram API client in Go with a clear surface area.

client.go

// Client is used to interact with the Infogram API
type Client struct {
	HTTPClient *http.Client
	Endpoint   string
	APIKey     string
	APISecret  string
}

// Do performs the *http.Request and decodes the http.Response.Body into v and return the *http.Response. If v is an io.Writer it will copy the body to the writer.
func (c *Client) Do(req *http.Request, v interface{}) (*http.Response, error) {...}

// SignRequest adds the `api_key` and `api_sig` query parameter in accordance with https://developers.infogr.am/rest/request-signing.html
func (c *Client) SignRequest(req *http.Request) error {...}

// Infographics fetches the list of infographics
func (c *Client) Infographics() ([]Infographic, error) {...}

// Infographics fetches a single infographic by identification number
func (c *Client) Infographic(id int) (*Infographic, error) {...}

// UserInfographics fetches the list of infographics for the user's identification number
func (c *Client) UserInfographics(id string) ([]Infographic, error) {...}

// Infographics fetches a available themes to use for infographics
func (c *Client) Themes() ([]Theme, error) {...}

types.go

// Infographic defines the type returned by the Infogram API
type Infographic struct {...}

// PDFReader returns an io.Reader of the Infographic in PDF format
func (i *Infographic) PDFReader(client *Client) (io.Reader, error) {...}

// PNGReader returns an io.Reader of the Infographic in PNG format
func (i *Infographic) PNGReader(client *Client) (io.Reader, error) {...}

// HTMLReader returns an io.Reader of the Infographic in HTML format
func (i *Infographic) HTMLReader(client *Client) (io.Reader, error) {...}

// MarshalJSON implements json.Marshaler
func (i *Infographic) MarshalJSON() ([]byte, error) {...}

// UnmarshalJSON implements json.Unarshaler
func (i *Infographic) UnmarshalJSON(bytes []byte) error {...}

// Theme defines the type returned by the Infogram API
type Theme struct {...}

// MarshalJSON implements json.Marshaler
func (t *Theme) MarshalJSON() ([]byte, error) {...}

// UnmarshalJSON implements json.Unmarshaler
func (t *Theme) UnmarshalJSON(bytes []byte) error {...}

I've had really good luck with designing HTTP clients in Go this way and I hope this helps others as well. You can find the full implementation of the Infogram API Go client on GitHub. I plan to keep iterating on the client more and more so if there is something you see missing or there is a problem feel free to file an issue or submit a change.