A short overview of protocol buffers

This post is a slightly modified version of slides for a gophers meetup in Frankfurt. They served primarily as a way to support the talk but are published here for reference.

History

• Internal release 2001 [wiki]
• Public release 2008 [wiki]

What

Protocol buffers are Google’s efficient way to communicate over a network.

They are a foundation for gRPC [grpc] an RPC framework that makes building services rather easy.

Protocol buffers are a good fit when used in conjunction with a service based architecture.

Similar projects include

https://thrift.apache.org/ Apache Thrift

https://github.com/Microsoft/bond Microsoft Bond

Why

• Architecture of Google’s services [sre]

.image google-app-request.jpg

Project is over 15 years old and assumed to be very mature.

How

• Binary serialization (no way to detect parameter names, etc.)
• Specification and code generation
• Cross platform

field types and position

No meta information - fields are transmitted in a specific order from definition. Own type system. Together encoded as the key to the value.

For an integer variable at position 1:

Wire type is 0

Field index is 1

Step               Representation
Assign index 1                  1
Shift by 3                   1000
Or with wire type 0          1000

Key is 0x08

https://developers.google.com/protocol-buffers/docs/encoding#structure Message structure

field types and position

For an integer variable at position 2

Wire type is 0

Field index is 2

Step               Representation
Assign index 2                 10
Shift by 3                  10000
Or with wire type 0         10000

Key is 0x10

For a string variable at position 1:

Wire type is 2

Field index is 1

Step               Representation
Assign index 1                  1
Shift by 3                   1000
Or with wire type 2          1010

Key is 0x0A

variable sized fields

Only the bytes required to transfer a number is used. Sample encoding:

0000011 0010000 ( 400 )
0010000 0000011 ( reversed )
10010000 00000011 ( add most significant bit )
144 3 ( decimal representation )
90 03 ( hex representation ) [ index 1: 08 90 03 ]

0000111 011100 001001 1000010 ( 3900610 )
1000010 001001 011100 0000111 ( reversed )
11000010 1001001 1011100 00000111 ( add most significant bit )
194 73 92 7 ( decimal representation )
C2 49 5C 07 ( hex representation ) [ index 1: 08 C2 49 5C 07 ]

0000001 0010110 ( 150 )
0010110 0000001 ( reversed )
10010110 00000001 ( add most significant bit )
150 1 ( decimal representation )
96 01 ( hex representation ) [ index 1: 08 96 01 ]

Javascript

Release notes for 3.0.0-beta-2 state support for Javascript:

Added proto3 support for JavaScript. The runtime is written in pure
JavaScript and works in browsers and in Node.js. To generate JavaScript
code for your proto, invoke protoc with "--js_out". See js/README.md
for more build instructions. [300-beta]

The current release (3.3.0) has no beta flag for Javascript

Further reading

https://developers.google.com/protocol-buffers/ Google Developer

https://github.com/google/protobuf Github

https://landing.google.com/sre/book.html Site Reliability Engineering, [sre]

https://en.wikipedia.org/wiki/Protocol_Buffers#Language_support Wikipedia [wiki]

http://www.grpc.io/ grpc/grpc.io [grpc]

https://github.com/google/protobuf/releases/tag/v3.0.0-beta-2 Beta Release [300-beta]