Interact With Protobuf Endpoints in Human-Readable Text Formats Using protoCURL

Do you have difficulties debugging Protocol-Buffers-based HTTP REST endpoints? Since Protobuf uses binary payloads, we face the problem, that we cannot easily write or read our Protobuf payloads with curl directly on the terminal. Ideally, we would like to to use curl with Protobuf just like we use curl with JSON or XML with classic text-based HTTP REST endpoints. ¹

To this problem, we present protoCURL - cURL for Protobuf: The command line tool to quickly and easily write requests in human-readable text formats on the command line against Protocol Buffer over HTTP endpoints and view the output in a text-based format as well.

This can become handy when debugging Protobuf-based HTTP endpoints when they are used instead of JSON.²

The tool was created by myself (GollyTicker) with initial sponsorship from QAware, because of the need for debugging Protobuf REST APIs in our projects.

An Example with protoCURL

With protoCURL a request can be as simple as this:³

protocurl -I test/proto -i ..HappyDayRequest -o ..HappyDayResponse \
  -u http://localhost:8080/happy-day/verify \
  -d 'includeReason: true'

The command line arguments have this meaning:

• -I test/proto points to proto files in the filesystem
• -i ..HappyDayRequest and -o ..HappyDayResponse tell which message types are used for the input and expected for the output
  • The .. instructs protocurl to automatically infer the (unique) full package path of the message types
• -u <url> is the HTTP REST endpoint url to send the request to
• -d 'includeReason: true' describes the input data in the Protobuf Text Format

The Protobuf endpoint uses the proto file happyday.proto - which consists of these (condensed) request and response messages:

package happyday;

message HappyDayRequest {
  google.protobuf.Timestamp date = 1;
  bool includeReason = 2;
}
message HappyDayResponse {
  bool isHappyDay = 1;
  string reason = 2;
  string formattedDate = 3;
}

The server will tell us whether the UTC day of a specific timestamp is a happy day or not. We simply say, that everyday, except for Wednesdays, is a happy day.⁴

Concretely, given a HappyDayRequest,

• the server converts the date to UTC and sets the boolean isHappyDay if and only if the weekday of the date is Wednesday.
• If includeReason is set, then a reason for the isHappyDay boolean is given as a string.
• The server will also format the date to UTC as formattedDate.

The above protoCURL command will produce this output:

=========================== Request Text     =========================== >>>
includeReason: true
=========================== Response Text    =========================== <<<
isHappyDay: true
reason: "Thursday is a Happy Day! ⭐"
formattedDate: "Thu, 01 Jan 1970 00:00:00 GMT"

We can now easily see the request and response in the Protobuf Text Format without needing to use protoc or a full-blown programming language to manually (de-) serialize the content. Since we didn’t provide a date, the Protobuf server implicitly assumes all of the timestamp.proto fields to be zero - which corresponds to epoch time 0.

Explanation

In the background, protocurl essentially does the following:

1. encode the textual Protobuf message to a binary request payload (using a bundled protoc)
2. send the binary payload in a POST request to the HTTP REST endpoint (via curl, if possible) and receive the binary response payload
3. decode the binary response payload back to text and display it

More Examples

As a second example, let’s see, what happens, when we set the date to the first Wednesday in 2023:

protocurl -I test/proto -i ..HappyDayRequest -o ..HappyDayResponse \
  -u http://localhost:8080/happy-day/verify \
  -d 'includeReason: true, date: { seconds: 1672790400 }'

Note, the syntax for the text format above.

This will produce:

=========================== Request Text     =========================== >>>
date: {
  seconds: 1672790400
}
includeReason: true
=========================== Response Text    =========================== <<<
reason: "Tough luck on Wednesday... 😕"
formattedDate: "Wed, 04 Jan 2023 00:00:00 GMT">

Furthermore, we can also use JSON as the text format:

protocurl -I test/proto -i ..HappyDayRequest -o ..HappyDayResponse \
  -u http://localhost:8080/happy-day/verify \
  -d "{ \"date\": \"2023-01-01T00:00:00Z\", \"includeReason\": true }"

The JSON format is automatically detected and also used for the output:

=========================== Request JSON     =========================== >>>
{"date":"2023-01-01T00:00:00Z","includeReason":true}
=========================== Response JSON    =========================== <<<
{"isHappyDay":true,"reason":"Sunday is a Happy Day! ⭐","formattedDate":"Sun, 01 Jan 2023 00:00:00 GMT"}

Now you can send requests like the examples above