~

Migrate to buf from prototool

Why? Because prototool is outdated, and can not run on M1 mac.

We need 3 files:

• build.go: need to install protoc-gen-* binaries with pin version in go.mod
• buf.yaml
• buf.gen.yaml

FYI, I use:

• grpc-ecosystem/grpc-gateway v1.16.0
• bufbuild/protoc-gen-validate
• kei2100/protoc-gen-marshal-zap

build.go:

//go:build tools
// +build tools

import (
  _ "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway"
  _ "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger"
  _ "github.com/kei2100/protoc-gen-marshal-zap/plugin/protoc-gen-marshal-zap"
)

buf.yaml

version: v1
deps:
  - buf.build/envoyproxy/protoc-gen-validate:6607b10f00ed4a3d98f906807131c44a
  - buf.build/kei2100/protoc-gen-marshal-zap:081f499bbca4486784773e060c1c1418
  - buf.build/haunt98/googleapis:b38d93f7ade94a698adff9576474ae7c
  - buf.build/haunt98/grpc-gateway:ecf4f0f58aa8496f8a76ed303c6e06c7
breaking:
  use:
    - PACKAGE
lint:
  use:
    - DEFAULT

buf.gen.yaml:

version: v1
plugins:
  - plugin: buf.build/grpc/go:v1.3.0
    out: pkg
  - plugin: buf.build/protocolbuffers/go:v1.31.0
    out: pkg
  - plugin: buf.build/bufbuild/validate-go:v1.0.2
    out: pkg
    opt:
      - lang=go
  - plugin: marshal-zap
    out: pkg
  - plugin: grpc-gateway
    out: pkg
    opt:
      - logtostderr=true
  - plugin: swagger
    out: .
    opt:
      - logtostderr=true

Update Makefile:

gen:
  go install github.com/kei2100/protoc-gen-marshal-zap/plugin/protoc-gen-marshal-zap
  go install github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
  go install github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger
  go install github.com/bufbuild/buf/cmd/buf@latest
  buf mod update
  buf format -w
  buf generate

Run make gen to have fun of course.

If using bufbuild/protoc-gen-validate, kei2100/protoc-gen-marshal-zap, better make a raw copy of proto file for other services to integrate:

raw:
    mkdir -p ./raw
    cp ./api.proto ./raw/
    sed -i "" -e "s/import \"validate\/validate\.proto\";//g" ./raw/api.proto
    sed -i "" -e "s/\[(validate\.rules)\.string.min_len = 1\]//g" ./raw/api.proto
    sed -i "" -e "s/import \"marshal-zap\.proto\";//g" ./raw/api.proto
    sed -i "" -e "s/\[(marshal_zap\.mask) = true]//g" ./raw/api.proto

FAQ

Remember bufbuild/protoc-gen-validate, kei2100/protoc-gen-marshal-zap, grpc-ecosystem/grpc-gateway is optional, so feel free to delete if you don't use theme.

If use vendor:

• Replace buf generate with buf generate --exclude-path vendor.
• Replace buf format -w with buf format -w --exclude-path vendor.

If you use grpc-gateway:

• Replace import "third_party/googleapis/google/api/annotations.proto"; with import "google/api/annotations.proto";
• Delete security_definitions, security, in option (grpc.gateway.protoc_gen_swagger.options.openapiv2_swagger).

The last step is delete prototool.yaml.

If you are not migrate but start from scratch:

• Add buf lint to make sure your proto is good.
• Add buf breaking --against "https://your-grpc-repo-goes-here.git" to make sure each time you update proto, you don't break backward compatibility.

Tips

Some experience I got after writing proto files for a living:

• Ignore DRY (Do not Repeat Yourself) when handling proto, don't split proto into many files. Trust me, it saves you from wasting time to debug how to import Go after generated. Because proto import and Go import is 2 different things. If someone already have split proto files, you should use sed to fix the damn things.

Thanks

• uber/prototool
• bufbuild/buf