Using MQTT and Google’s Protocol Buffers to implement efficient RPCs

Introduction

Over the decades there were many attempts — some more successful than others — to implement RPCs (Remote Procedure Calls) in a programming language and network agnostic way.

SOAP (Simple Object Access Protocol), REST (Representational State Transfer), CORBA (Common Object Request Broker Architecture), JSON (JavaScript Object Notation), to name a few, are some protocols and architectures created with that purpose in mind.

In this article, I will describe our team experience using a combination of two relatively recent technologies to accomplish an efficient implementation of RPCs, derived from work developed by my colleague Julie Vogelman.

MQTT and Protobufs

MQTT

MQTT was authored by Andy Stanford-Clark (IBM) and Arlen Nipper (Cirrus Link, then Eurotech) in 1999. It was standardized in 2013 by OASIS.

MQTT defines a lightweight publish/subscribe protocol designed for resource-constrained devices and slow network links.

MQTT uses a broker that allows devices to publish data octets to topics, as well as subscribe to topics and receive data published to them.

Topics are strings that look like a path to a folder or file:

/this/is/an/example/of/a/topic

and are not predefined; a device simply publishes to a topic and any device that knows it can subscribe to receive data from it.

MQTT also allows devices to subscribe using wildcards. There are 2 wildcard symbols: ‘+’ and ‘#’

So if a device publishes to topics /this/is/an/example/of/a/topic and /this/is/an/example/of/another/topic, a subscriber can receive messages from both topics in 2 ways:

1. Subscribe to /this/is/an/example/of/+/topic
2. Subscribe to /this/is/an/example/of/#

Another feature of MQTT brokers that made them attractive for us is that they support both UDP and TCP, as well as one-way and two-way TLS.

In order to use MQTT developers can take advantage of open source libraries like Paho, which already support C and Java with Lua, Python, C++ and JavaScript on the way; and free, open source brokers like Eclipse mosquitto.

Protocol Buffers

Google initially developed protocol buffers — protobufs for short — in 2001 for internal use; it was made public in 2008.

Protobuf specifies a language-neutral mechanism to serialize data in a simple structure that is efficient for transmission between devices.

Its use is relatively simple: the developer defines a file with .proto with the data structure definition, like this:

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
  required string callbacktopic = 4;
}

Then use language generators provided by Google to create the serialization code for your programming language of choice. Currently Google supports C++, C#, Dart, Go, Java and Python.

Putting it all together

Here is a simple example that illustrates how it all gets glued together. The topic will look like the classpath for a Java class but it’s just our choice.

For this example, besides the Person message above we will define the Result message:

message Result {
    required int resultcode = 1;
}

Client on a device in C++

1. Subscribes to topic /com/myclient/MainClass/setPersonResult
2. Defines the Person message in the messages.proto file
3. Defines the Result message in messages.proto file
4. Generates the C++ code to serialize/deserialize the messages

Server application in Java

1. Defines the same Person message as the client
2. Defines the same Result message as the client
3. Generates the Java code to serialize/deserialize the messages
4. Subscribes to topic /com/myserver/MainClass/+, so it can receive method calls for all methods in this class

Flow

1. Client invokes the generated methods with the appropriate parameters; in particular, callbacktopic set to “/com/myclient/MainClass/setPersonResult”
2. Client publishes the serialized buffer to the topic /com/myserver/MainClass/setPerson
3. Server receives the data, deserializes and processes it by invoking method MainClass.setPerson (in this case we took advantage of Java’s Reflection and got the class and method straight from the topic)
4. Once processed, server invokes the generated methods to serialize the result and publishes to /com/myclient/MainClass/setPersonResult
5. Client receives the result, invokes the methods to deserialize it and calls the appropriate method to handle the result; in this case, because the client is in C++, we relied on a hash table to map topics to method pointers.

This schema scaled up well to all the API calls; the broker hides the internals of the server app, which can be protected behind VPN, firewalls or containers (we used Kubernetes and Docker to isolate the server app by the way).

Also the broker can handle thousands of messages per second, so no problem there.

It works really well, however…

However…

The same features that make it so flexible and efficient also make it very vulnerable. Here are some issues:

1. What if several instances of the client use the same topic for the callback?

Well, it is a problem because they will all receive the responses sent to all clients. Not good, each client must receive the result for its own method call.

The solution is to add a unique identifier to the string so the callback topic is unique to the client. Something like:

/com/myclient/Th8858Hhjhkjhcvb&uAA34/MainClass/setPersonResult

2. What if an attacker subscribes to MQTT for all topics, using topic ‘#’?

This is a big problem. If the MQTT interface is exposed publicly — via URL or public IP address — then an attacker can easily eavesdrop all traffic between all clients and the server. A HUGE no-no.

We envisioned 4 possible solutions for this problem:

1. The client and server negotiate an encryption key — for example, using Diffie-Hellman — and use it to encrypt the content before sending it to the topic; anyone eavesdropping will see the content but will not be able to read it
2. Whitelisting IP addresses. Only whitelisted IP addresses could connect to the MQTT broker. Works but it is cumbersome to maintain.
3. Use VPN. Access to the MQTT interface requires VPN so only clients with the credentials can connect. Works well if access is restricted to a client app or if users are required to register for using the app; not so much for an open, public API
4. Use a broker implementation that offers Access Control. Normally these are commercial paid brokers, not free.

So as you can see, these solutions work but are not great. So, in conclusion…

Conclusion

Using MQTT and Protobufs for implementing RPCs works really well. It is efficient, easy to implement, fast and reliable.

But security is a concern; it is not a solution for every RPC application. For public APIs there are other approaches that are more secure and therefore preferred.

If however your use case is restricted to communication between your client application and your server, then it is a solution that worked well for us and you may want to check it out.