Auto-Generating an OpenAPI Specification for gRPC and gRPC Gateway
When building new services, gRPC can be very handy to not only define those services, but also generate a lot of the boilerplate code for server implementations and client libraries. We get the added benefit of a more efficient wire protocol, Protocol Buffers, because it's in binary and a lot smaller than JSON. However, sometime we still need to support HTTP and some REST-ful behavior of our services. It would be nice to still be able to leverage all the benefits of using gRPC and Protocol Buffers without sacrificing or needing to write a lot of wrapper/proxy code to expose those services over HTTP. Taking this even further, it would be even better to leverage auto-generated documentation of our HTTP endpoints with the OpenAPI Specification and allow developers to interact with it with a hosted version of the Swagger UI.
Fortunately, the gRPC ecosystem has plugins to support this setup, and requires minimal code to wire everything together.
Defining a gRPC Service
Here we define a simple gRPC CRUD ExampleService
. For the sake of brevity, we'll just use google.protobuf.Empty
messages for the requests and responses. Typically, we would define message CreateRequest {...}
and message CreateResponse {...}
types with the appropriate fields.
protos/service.proto
syntax = "proto3";
package protos;
import "google/protobuf/empty.proto";
option go_package = "github.com/blainsmith/grpc-gateway-openapi-example/gen/proto/go/protos;protos";
service ExampleService {
rpc Create(google.protobuf.Empty) returns (google.protobuf.Empty) {}
rpc Read(google.protobuf.Empty) returns (google.protobuf.Empty) {}
rpc Update(google.protobuf.Empty) returns (google.protobuf.Empty) {}
rpc Delete(google.protobuf.Empty) returns (google.protobuf.Empty) {}
}
Generating and Implementing the Server in Go
Now that we have our service defined, we can generate the Go server interface and client using the Protocol Buffer compiler and the appropriate plugins. We'll be using buf
to do this work, which uses protoc
under the hood. We'll create two configuration files for buf
so it knows what to generate.
This is a basic buf
configuration that defines our service name and the dependencies we need to build our protos/service.proto
file. We need to define buf.build/googleapis/googleapis
as a dependency in order to use google.protobuf.Empty
as our request and response messages.
buf.yaml
version: v1
name: buf.build/blainsmith/grpc-gateway-openapi-example
deps:
- buf.build/googleapis/googleapis
breaking:
use:
- FILE
lint:
use:
- DEFAULT
The second configuration file tells buf
how to generate our Go code and where to put the output files.
buf.gen.yaml
version: v1
plugins:
- name: go
out: gen/protos/go
opt: paths=source_relative
- name: go-grpc
out: gen/protos/go
opt:
- paths=source_relative
After running buf build
and buf generate
we end up with 2 new files. The first file contains our generate wire Protocol Buffer messages, and the second contains the gRPC server and client.
gen/protos/go/protos/service.pb.go
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.25.0
// protoc (unknown)
// source: protos/service.proto
package protos
...
gen/protos/go/protos/service_grpc.pb.go
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
package protos
...
type ExampleServiceClient interface {
Create(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
Read(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
Update(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
Delete(ctx context.Context, in *emptypb.Empty, opts ...grpc.CallOption) (*emptypb.Empty, error)
}
...
type ExampleServiceServer interface {
Create(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
Read(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
Update(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
Delete(context.Context, *emptypb.Empty) (*emptypb.Empty, error)
mustEmbedUnimplementedExampleServiceServer()
}
...
Now we can implement the server in our own Go file.
Implementing the gRPC Server
We now need to define the logic that implements protos.ExampleServiceServer
, and to do that, we just make our own file and import the generated protos
package. We create our source file, define our own struct
, and implement the CRUD functions that are required by protos.ExampleServiceServer
.
service/example.go
package service
import (
"context"
"github.com/blainsmith/grpc-gateway-openapi-example/gen/protos/go/protos"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/emptypb"
)
type ExampleService struct {
*protos.UnimplementedExampleServiceServer
}
func (ExampleService) Create(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
return nil, status.Errorf(codes.Unimplemented, "method Create not implemented")
}
func (ExampleService) Read(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
return nil, status.Errorf(codes.Unimplemented, "method Read not implemented")
}
func (ExampleService) Update(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
return nil, status.Errorf(codes.Unimplemented, "method Update not implemented")
}
func (ExampleService) Delete(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
return nil, status.Errorf(codes.Unimplemented, "method Delete not implemented")
}
Starting the gRPC Server
Now that we have an implemented service, we need to start it by listening on a port to expose it.
cmd/example.go
package main
func main() {
lis, err := net.Listen("tcp", ":9090")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
protos.RegisterExampleServiceServer(grpcServer, &service.ExampleService{})
err = grpcServer.Serve(lis)
if err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
Now we have our gRPC server running so clients can connect and call our CRUD functions.
Annotate the gRPC Service with HTTP Routes
Now comes the interesting part of exposing an HTTP server to handle REST calls to the same ExampleService
implementation we defined in service/example.go
without changing that code.
We start by modifying our .proto
file an annotate the RPC calls to include optional HTTP routes to be attached to.
syntax = "proto3";
package protos;
import "google/api/annotations.proto";
import "google/protobuf/empty.proto";
option go_package = "github.com/blainsmith/grpc-gateway-openapi-example/gen/proto/go/protos;protos";
service ExampleService {
rpc Create(google.protobuf.Empty) returns (google.protobuf.Empty) {
option (google.api.http) = {
post: "/create"
};
}
rpc Read(google.protobuf.Empty) returns (google.protobuf.Empty) {
option (google.api.http) = {
get: "/read"
};
}
rpc Update(google.protobuf.Empty) returns (google.protobuf.Empty) {
option (google.api.http) = {
put: "/update"
};
}
rpc Delete(google.protobuf.Empty) returns (google.protobuf.Empty) {
option (google.api.http) = {
delete: "/delete"
};
}
}
Generate the gRPC Gateway
Now that we have our service annotated with routes, we can use another gRPC plugin to generate a new server which will accept HTTP calls and proxy them to gRPC calls. We modify our buf.gen.yaml
file to include a new plugin.
buf.gen.yaml
version: v1
plugins:
- name: go
out: gen/protos/go
opt: paths=source_relative
- name: go-grpc
out: gen/protos/go
opt:
- paths=source_relative
- name: grpc-gateway
out: gen/protos/go
opt:
- paths=source_relative
Now we can run buf generate
again, which will output another file.
gen/protos/go/protos/service.pb.gw.go
// Code generated by protoc-gen-grpc-gateway. DO NOT EDIT.
// source: protos/service.proto
/*
Package protos is a reverse proxy.
It translates gRPC into RESTful JSON APIs.
*/
package protos
...
// RegisterExampleServiceHandlerServer registers the http handlers for service ExampleService to "mux".
// UnaryRPC :call ExampleServiceServer directly.
// StreamingRPC :currently unsupported pending https://github.com/grpc/grpc-go/issues/906.
// Note that using this registration option will cause many gRPC library features to stop working. Consider using RegisterExampleServiceHandlerFromEndpoint instead.
func RegisterExampleServiceHandlerServer(ctx context.Context, mux *runtime.ServeMux, server ExampleServiceServer) error {
...
}
...
Starting the HTTP Server
We have to add two things to our main()
function now to connect the HTTP server to the gRPC server. The first thing is we need a gRPC client, which our HTTP server will use to talk to the gRPC server.
- The HTTP Server accepts incoming requests
- It converts the request to a gRPC call
- It uses the gRPC Client that has an open connection to the gRPC Server
- the gRPC Server invokes the ExampleService function
- The result is returned to the gRPC Client
- The HTTP Server transforms the result to a response back to the caller
┌────────────────────────┐ ┌────────────────────────┐
Request │ HTTP │ │ gRPC │
──────────► Server │ │ Server │
│ │ │ │
│ │ │ │
│ ┌──────────┐ │ │ │
│ │ gRPC ◄─┼───────────► ┌────────────────┐ │
Response │ │ Client │ │ │ │ ExampleService │ │
◄─────────┤ └──────────┘ │ │ └────────────────┘ │
│ │ │ │
└────────────────────────┘ └────────────────────────┘
cmd/example.go
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// start the gRPC server
lis, err := net.Listen("tcp", ":9090")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
protos.RegisterExampleServiceServer(grpcServer, &service.ExampleService{})
go grpcServer.Serve(lis)
// dial the gRPC server above to make a client connection
conn, err := grpc.Dial(":9090", grpc.WithInsecure())
if err != nil {
log.Fatalf("fail to dial: %v", err)
}
defer conn.Close()
// create an HTTP router using the client connection above
// and register it with the service client
rmux := runtime.NewServeMux()
client := protos.NewExampleServiceClient(conn)
err = protos.RegisterExampleServiceHandlerClient(ctx, rmux, client)
if err != nil {
log.Fatal(err)
}
// create a standard HTTP router
mux := http.NewServeMux()
// mount the gRPC HTTP gateway to the root
mux.Handle("/", rmux)
// start a standard HTTP server with the router
err = http.ListenAndServe(":8080", mux)
if err != nil {
log.Fatal(err)
}
Now we have a service listening on 2 different ports. The gRPC server port is :9090
and the HTTP server port is :8080
. The routes we have defined in our .proto
file can be hit with Postman, cURL, or your favorite REST API tool.
The problem now is what do these HTTP request and responses look like in JSON? How do we know what request body to create, and what do the responses contain? This is where another plugin and the Swagger UI comes in to play.
Generating the OpenAPI Specification
The last few steps are to pull in another gRPC plugin to generate the OpenAPI Specification JSON file, and then expose that file and the Swagger UI in our HTTP server. Back in the buf.gen.yaml
file we add in the last plugin for openapiv2
.
buf.gen.yaml
version: v1
plugins:
- name: go
out: gen/protos/go
opt: paths=source_relative
- name: go-grpc
out: gen/protos/go
opt:
- paths=source_relative
- name: grpc-gateway
out: gen/protos/go
opt:
- paths=source_relative
- name: openapiv2
out: gen
Running buf generate
again will output a JSON file that contains the OpenAPI Specification of our annotated gRPC service.
gen/protos/service.swagger.json
{
"swagger": "2.0",
"info": {
"title": "protos/protos/service.proto",
"version": "version not set"
},
"tags": [
{
"name": "ExampleService"
}
],
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"paths": {
"/create": {
"post": {
"operationId": "ExampleService_Create",
"responses": {
"200": {
"description": "A successful response.",
"schema": {
"type": "object",
"properties": {}
}
},
"default": {
"description": "An unexpected error response.",
"schema": {
"$ref": "#/definitions/rpcStatus"
}
}
},
"tags": [
"ExampleService"
]
}
},
...
Now we can use this and render the Swagger UI for it, similar to https://generator.swagger.io, but for our generated specification file. To do that, we need to grab the entire dist
folder from https://github.com/swagger-api/swagger-ui/tree/master/dist and add it to our repository as swagger-ui
.
Next, we will modify our HTTP server to have 2 new routes. One for the specification file, and the other for the Swagger UI.
cmd/example.go
...
// mount the gRPC HTTP gateway to the root
mux.Handle("/", rmux)
// mount a path to expose the generated OpenAPI specification on disk
mux.HandleFunc("/swagger-ui/swagger.json", func(w http.ResponseWriter, r *http.Request) {
http.ServeFile(w, r, "./gen/protos/service.swagger.json")
})
// mount the Swagger UI that uses the OpenAPI specification path above
mux.Handle("/swagger-ui/", http.StripPrefix("/swagger-ui/", http.FileServer(http.Dir("./swagger-ui"))))
// start a standard HTTP server with the router
err = http.ListenAndServe("localhost:8080", mux)
if err != nil {
log.Fatal(err)
}
...
Lastly, we will change the Swagger initializer url
to load the swagger.json
path we added to our HTTP server.
swagger-ui/swagger-initializer.js
...
window.ui = SwaggerUIBundle({
url: "swagger.json", // this should be
dom_id: '#swagger-ui',
...
Play the API with the Swagger UI
Now that everything is wired up, all we need to do is start our service again and then open up http://localhost:8080/swagger-ui in a browser, and we get an interactive Swagger UI to play with.
Closing Thoughts
Now that you have the foundation set up, you can expand upon this to build a full fledged gRPC service and continually re-generate the OpenAPI Specification to support HTTP calls, all without compromising or leaving the comfort of your service.proto
definition. This stays as the source of truth as you implement the server itself in Go as normal gRPC function calls.
You can find the complete example at https://github.com/blainsmith/grpc-gateway-openapi-example and if you have any questions feel free to file an issue or contact me any way you're comfortable with.