[Main] [Book Recs] [Talks] [Sponsor]

Auto-Generating an OpenAPI Specification for gRPC and gRPC Gateway

When building new services, gRPC can be very handy to not only define those services, but also generate a lot of the boilerplate code for server implementations and client libraries. We get the added benefit of a more efficient wire protocol, Protocol Buffers, because it's in binary and a lot smaller than JSON. However, sometime we still need to support HTTP and some REST-ful behavior of our services. It would be nice to still be able to leverage all the benefits of using gRPC and Protocol Buffers without sacrificing or needing to write a lot of wrapper/proxy code to expose those services over HTTP. Taking this even further, it would be even better to leverage auto-generated documentation of our HTTP endpoints with the OpenAPI Specification and allow developers to interact with it with a hosted version of the Swagger UI.

Fortunately, the gRPC ecosystem has plugins to support this setup, and requires minimal code to wire everything together.

Defining a gRPC Service

Here we define a simple gRPC CRUD ExampleService. For the sake of brevity, we'll just use google.protobuf.Empty messages for the requests and responses. Typically, we would define message CreateRequest {...} and message CreateResponse {...} types with the appropriate fields.

protos/service.proto

syntax = "proto3";

package protos;

import "google/protobuf/empty.proto";

option go_package = "github.com/blainsmith/grpc-gateway-openapi-example/gen/proto/go/protos;protos";

service ExampleService {
  rpc Create(google.protobuf.Empty) returns (google.protobuf.Empty) {}

  rpc Read(google.protobuf.Empty) returns (google.protobuf.Empty) {}

  rpc Update(google.protobuf.Empty) returns (google.protobuf.Empty) {}

  rpc Delete(google.protobuf.Empty) returns (google.protobuf.Empty) {}
}

Generating and Implementing the Server in Go

Now that we have our service defined, we can generate the Go server interface and client using the Protocol Buffer compiler and the appropriate plugins. We'll be using buf to do this work, which uses protoc under the hood. We'll create two configuration files for buf so it knows what to generate.

This is a basic buf configuration that defines our service name and the dependencies we need to build our protos/service.proto file. We need to define buf.build/googleapis/googleapis as a dependency in order to use google.protobuf.Empty as our request and response messages.

buf.yaml

version: v1
name: buf.build/blainsmith/grpc-gateway-openapi-example
deps:
  - buf.build/googleapis/googleapis
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

The second configuration file tells buf how to generate our Go code and where to put the output files.

buf.gen.yaml

version: v1
plugins:
  - name: go
    out: gen/protos/go
    opt: paths=source_relative
  - name: go-grpc
    out: gen/protos/go
    opt:
      - paths=source_relative

After running buf build and buf generate we end up with 2 new files. The first file contains our generate wire Protocol Buffer messages, and the second contains the gRPC server and client.

gen/protos/go/protos/service.pb.go

// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// 	protoc-gen-go v1.25.0
// 	protoc        (unknown)
// source: protos/service.proto

package protos

...

gen/protos/go/protos/service_grpc.pb.go

// Code generated by protoc-gen-go-grpc. DO NOT EDIT.

package protos

...

type ExampleServiceClient interface {
	Create(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
	Read(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
	Update(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
	Delete(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
}

...

type ExampleServiceServer interface {
	Create(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
	Read(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
	Update(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
	Delete(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
	mustEmbedUnimplementedExampleServiceServer()
}

...

Now we can implement the server in our own Go file.

Implementing the gRPC Server

We now need to define the logic that implements protos.ExampleServiceServer, and to do that, we just make our own file and import the generated protos package. We create our source file, define our own struct, and implement the CRUD functions that are required by protos.ExampleServiceServer.

service/example.go

package service

import (
	"context"

	"github.com/blainsmith/grpc-gateway-openapi-example/gen/protos/go/protos"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
	"google.golang.org/protobuf/types/known/emptypb"
)

type ExampleService struct {
	*protos.UnimplementedExampleServiceServer
}

func (ExampleService) Create(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
	return nil, status.Errorf(codes.Unimplemented, "method Create not implemented")
}
func (ExampleService) Read(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
	return nil, status.Errorf(codes.Unimplemented, "method Read not implemented")
}
func (ExampleService) Update(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
	return nil, status.Errorf(codes.Unimplemented, "method Update not implemented")
}
func (ExampleService) Delete(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
	return nil, status.Errorf(codes.Unimplemented, "method Delete not implemented")
}

Starting the gRPC Server

Now that we have an implemented service, we need to start it by listening on a port to expose it.

cmd/example.go

package main

func main() {
    lis, err := net.Listen("tcp", ":9090")
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	
    grpcServer := grpc.NewServer()
	protos.RegisterExampleServiceServer(grpcServer, &service.ExampleService{})
    
    err = grpcServer.Serve(lis)
    if err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

Now we have our gRPC server running so clients can connect and call our CRUD functions.

Annotate the gRPC Service with HTTP Routes

Now comes the interesting part of exposing an HTTP server to handle REST calls to the same ExampleService implementation we defined in service/example.go without changing that code.

We start by modifying our .proto file an annotate the RPC calls to include optional HTTP routes to be attached to.

syntax = "proto3";

package protos;

import "google/api/annotations.proto";
import "google/protobuf/empty.proto";

option go_package = "github.com/blainsmith/grpc-gateway-openapi-example/gen/proto/go/protos;protos";

service ExampleService {
  rpc Create(google.protobuf.Empty) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      post: "/create"
    };
  }

  rpc Read(google.protobuf.Empty) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      get: "/read"
    };
  }

  rpc Update(google.protobuf.Empty) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      put: "/update"
    };
  }

  rpc Delete(google.protobuf.Empty) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      delete: "/delete"
    };
  }
}

Generate the gRPC Gateway

Now that we have our service annotated with routes, we can use another gRPC plugin to generate a new server which will accept HTTP calls and proxy them to gRPC calls. We modify our buf.gen.yaml file to include a new plugin.

buf.gen.yaml

version: v1
plugins:
  - name: go
    out: gen/protos/go
    opt: paths=source_relative
  - name: go-grpc
    out: gen/protos/go
    opt:
      - paths=source_relative
  - name: grpc-gateway
    out: gen/protos/go
    opt:
      - paths=source_relative

Now we can run buf generate again, which will output another file.

gen/protos/go/protos/service.pb.gw.go

// Code generated by protoc-gen-grpc-gateway. DO NOT EDIT.
// source: protos/service.proto

/*
Package protos is a reverse proxy.

It translates gRPC into RESTful JSON APIs.
*/
package protos

...

// RegisterExampleServiceHandlerServer registers the http handlers for service ExampleService to "mux".
// UnaryRPC     :call ExampleServiceServer directly.
// StreamingRPC :currently unsupported pending https://github.com/grpc/grpc-go/issues/906.
// Note that using this registration option will cause many gRPC library features to stop working. Consider using RegisterExampleServiceHandlerFromEndpoint instead.
func RegisterExampleServiceHandlerServer(ctx context.Context, mux *runtime.ServeMux, server ExampleServiceServer) error {
    ...
}

...

Starting the HTTP Server

We have to add two things to our main() function now to connect the HTTP server to the gRPC server. The first thing is we need a gRPC client, which our HTTP server will use to talk to the gRPC server.

  1. The HTTP Server accepts incoming requests
  2. It converts the request to a gRPC call
  3. It uses the gRPC Client that has an open connection to the gRPC Server
  4. the gRPC Server invokes the ExampleService function
  5. The result is returned to the gRPC Client
  6. The HTTP Server transforms the result to a response back to the caller
          ┌────────────────────────┐           ┌────────────────────────┐
 Request  │          HTTP          │           │          gRPC          │
──────────►         Server         │           │         Server         │
          │                        │           │                        │
          │                        │           │                        │
          │           ┌──────────┐ │           │                        │
          │           │   gRPC   ◄─┼───────────►   ┌────────────────┐   │
 Response │           │  Client  │ │           │   │ ExampleService │   │
◄─────────┤           └──────────┘ │           │   └────────────────┘   │
          │                        │           │                        │
          └────────────────────────┘           └────────────────────────┘

cmd/example.go

ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()

// start the gRPC server
lis, err := net.Listen("tcp", ":9090")
if err != nil {
    log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
protos.RegisterExampleServiceServer(grpcServer, &service.ExampleService{})
go grpcServer.Serve(lis)

// dial the gRPC server above to make a client connection
conn, err := grpc.Dial(":9090", grpc.WithInsecure())
if err != nil {
    log.Fatalf("fail to dial: %v", err)
}
defer conn.Close()

// create an HTTP router using the client connection above
// and register it with the service client
rmux := runtime.NewServeMux()
client := protos.NewExampleServiceClient(conn)
err = protos.RegisterExampleServiceHandlerClient(ctx, rmux, client)
if err != nil {
    log.Fatal(err)
}

// create a standard HTTP router
mux := http.NewServeMux()

// mount the gRPC HTTP gateway to the root
mux.Handle("/", rmux)

// start a standard HTTP server with the router
err = http.ListenAndServe(":8080", mux)
if err != nil {
    log.Fatal(err)
}

Now we have a service listening on 2 different ports. The gRPC server port is :9090 and the HTTP server port is :8080. The routes we have defined in our .proto file can be hit with Postman, cURL, or your favorite REST API tool.

The problem now is what do these HTTP request and responses look like in JSON? How do we know what request body to create, and what do the responses contain? This is where another plugin and the Swagger UI comes in to play.

Generating the OpenAPI Specification

The last few steps are to pull in another gRPC plugin to generate the OpenAPI Specification JSON file, and then expose that file and the Swagger UI in our HTTP server. Back in the buf.gen.yaml file we add in the last plugin for openapiv2.

buf.gen.yaml

version: v1
plugins:
  - name: go
    out: gen/protos/go
    opt: paths=source_relative
  - name: go-grpc
    out: gen/protos/go
    opt:
      - paths=source_relative
  - name: grpc-gateway
    out: gen/protos/go
    opt:
      - paths=source_relative
  - name: openapiv2
    out: gen

Running buf generate again will output a JSON file that contains the OpenAPI Specification of our annotated gRPC service.

gen/protos/service.swagger.json

{
  "swagger": "2.0",
  "info": {
    "title": "protos/protos/service.proto",
    "version": "version not set"
  },
  "tags": [
    {
      "name": "ExampleService"
    }
  ],
  "consumes": [
    "application/json"
  ],
  "produces": [
    "application/json"
  ],
  "paths": {
    "/create": {
      "post": {
        "operationId": "ExampleService_Create",
        "responses": {
          "200": {
            "description": "A successful response.",
            "schema": {
              "type": "object",
              "properties": {}
            }
          },
          "default": {
            "description": "An unexpected error response.",
            "schema": {
              "$ref": "#/definitions/rpcStatus"
            }
          }
        },
        "tags": [
          "ExampleService"
        ]
      }
    },
...

Now we can use this and render the Swagger UI for it, similar to https://generator.swagger.io, but for our generated specification file. To do that, we need to grab the entire dist folder from https://github.com/swagger-api/swagger-ui/tree/master/dist and add it to our repository as swagger-ui.

Next, we will modify our HTTP server to have 2 new routes. One for the specification file, and the other for the Swagger UI.

cmd/example.go

...
// mount the gRPC HTTP gateway to the root
mux.Handle("/", rmux)

// mount a path to expose the generated OpenAPI specification on disk
mux.HandleFunc("/swagger-ui/swagger.json", func(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "./gen/protos/service.swagger.json")
})

// mount the Swagger UI that uses the OpenAPI specification path above
mux.Handle("/swagger-ui/", http.StripPrefix("/swagger-ui/", http.FileServer(http.Dir("./swagger-ui"))))

// start a standard HTTP server with the router
err = http.ListenAndServe("localhost:8080", mux)
if err != nil {
    log.Fatal(err)
}
...

Lastly, we will change the Swagger initializer url to load the swagger.json path we added to our HTTP server.

swagger-ui/swagger-initializer.js

...

window.ui = SwaggerUIBundle({
    url: "swagger.json", // this should be
    dom_id: '#swagger-ui',

...

Play the API with the Swagger UI

Now that everything is wired up, all we need to do is start our service again and then open up http://localhost:8080/swagger-ui in a browser, and we get an interactive Swagger UI to play with.

gRPC Gateway Swagger UI

Closing Thoughts

Now that you have the foundation set up, you can expand upon this to build a full fledged gRPC service and continually re-generate the OpenAPI Specification to support HTTP calls, all without compromising or leaving the comfort of your service.proto definition. This stays as the source of truth as you implement the server itself in Go as normal gRPC function calls.

You can find the complete example at https://git.sr.ht/~blainsmith/grpc-gateway-openapi-example and if you have any questions feel free to file an issue or contact me any way you're comfortable with.