First, we need to do some setup — we check out the repository (actions/checkout) and install the latest Buf CLI (bufbuild/buf-setup-action). After that, we are ready to call the lint action (bufbuild/buf-lint-action) that ensures our proto files follow the defined style guide.
After the lint is successful, we execute an action ensuring the new schema is backwards compatible with the old one. We achieve this by first fetching the main branch and executing the breaking action (bufbuild/buf-breaking-action) against the current content of the main branch.
If the validate
job succeeds and the action is being executed on a commit to the main branch, then we trigger the job push
.
You’ll notice this job also starts with the checkout and Buf setup actions, followed by the push action (bufbuild/buf-push-action) that takes a secret token to authenticate with the Buf Schema Registry and pushes the new Protobuf definitions.
These Github Actions result in a workflow that doesn’t rely on the developer having their local environment set up correctly, as the CI/CD is the single place where all Protobuf files are validated. Additionally, we don’t need to share secrets between developers, the CI/CD takes care of pushing schemas to the registry.
Schema Registry
We use the Buf Schema Registry to host the Protobuf definitions and get a UI for our docs. The registry also tracks old versions of the same schema file so anyone referencing an older version can keep using it or update to the new version using buf mod update
.
Remote Code Generation
Pushing our Protobuf definitions to the Buf Schema Registry opens up the possibility of using remote code generation. The registry will take care of generating the Go code for us and expose it as a go module, ready to be imported. This feature allows us to entirely skip the manual compiling step and simply import the compiled code as a dependency.
For instance, to fetch the latest Conduit connector protocol code we can invoke this command:
go get go.buf.build/protocolbuffers/go/conduitio/conduit-connector-protocol
Every time we update the Protobuf definitions and push them to the registry, the code will be remotely generated and ready to be used in any dependent code.
Local Development
Our workflow heavily leans on hosted services like Github Actions and the Buf Schema Registry, so the natural question is how can we do local development? The answer are go mod replace directives.
To switch to locally generated Protobuf code we follow the following steps:
buf generate
— executing this in theproto
folder will compile the proto files and generate Go code locally in the folderinternal
go mod init github.com/conduitio/conduit-connector-protocol/internal
— executing this in folderinternal
will initialize a (temporary) Go module in the newly generated Go codego mod edit -replace go.buf.build/library/go-grpc/conduitio/conduit-connector-protocol=./internal
— executing this at the root of the repository will replace any references to the remotely generated code with the locally generated code (similarly we can do this for other repositories that depend on remotely generated code)
Conclusion
Buf is a great tool that allows us to streamline the management of our Protobuf files, ensures we follow code guidelines and don’t unknowingly introduce breaking changes. It solves these problems in an elegant way and enhances the developer experience.
You know what else enhances the developer experience? Conduit! We’re still very much in the early stages and rely on the feedback of our community to steer the project in the right direction. Try it out… if you like it join the discussion and show us some love!