Aniruddha Deb

Message Bus

4 min read

Spec

You need to write a message bus. A message bus has a set of publishers and a set of subscribers, along with some control flow to create/delete streams. Publishers publish data to streams and subscribers get messages pushed to them from these streams.

The message bus has the following protocol in protobuf:

syntax = "proto3";

enum CommandType {
	UNKNOWN = 0;
	PUB_TOPICS = 1;
	SUB_TOPICS = 2;
	GET_TOPICS = 3;
	GET_ALL_TOPICS = 4;
	UNSUB_TOPICS = 2;
}

message Ack {}

enum ErrorCode {
	UNKNOWN_ERROR = 0;
	BAD_PROTO_VERSION = 1;
	BAD_PEER_VERSION = 2;
	RESERVED_TOPIC_NAME = 3;
	TOPIC_DOES_NOT_EXIST = 4;
}

message Error {
	ErrorCode code;
	string description;
}

message Handshake {
	uint32 proto_version = 1;
	string client_version = 2;
	string peer_name = 3;
}

message Shakehand {
	uint32 proto_version = 1;
	string server_version = 2;
	optional uint64 client_id = 3;
	optional Error error = 4;
}

message Metric {
	string name = 1;
	double value = 2;
}

message Metrics {
	repeated Metric all;
}

message PubData {
	Topic topic = 1;
	bytes data = 2;
}

message SubData {
	Topic topic = 1;
	int64 update_num = 2;
	int64 timestamp = 3;
	bytes data = 4;
}

message Topics {
	repeated Topic topics
}

message Command {
	CommandType cmd;
	oneof {
		Topics topics;
	}
}

message Response {
	oneof payload {
		Ack ack;
		SubData data;
		Error err;
		Shakehand shakehand;
	}
}

message Request {
	oneof payload {
		Command command;
		Handshake handshake;
		PubData data;
	}
}

The message flow is as follows:

  1. Clients connect to the server using a handshake, with their proto version and peer version. The ok field, if set, will be ignored
    1. Server validates the result, and responds with either an Ack or an Error. If there’s an error, client must disconnect.
  2. Clients after getting a handshake ack can send commands or data.
    1. For Commands:
      1. PUB_TOPICS: send a list of topics this client will publish. The ids will be set by the server, so the messages must contain only names. Topics without name are ignored. If any topic does not exist, it’s created. A list of topics successfully created with their IDs is returned as a response. In case a reserved/protected topic name is used, an error with code 3 (RESERVED_TOPIC_NAME) is returned.
      2. SUB_TOPICS: subscribes to a list of topics. The topics in the request must have either a name or an ID.
      3. GET_TOPICS/GET_ALL_TOPICS: Gets the fully qualified topic for a list of partially specified topics.
      4. GET_METRICS: Pushes all metrics out at once to the requesting client.
    2. For Data:
      1. Sending data to a topic that does not exist will return an error
      2. (?) Sending data to a topic that the client has not declared themselves as a publisher on?
      3. All received data across all clients must have IDs that are monotonically increasing

As a server, the list of metrics you have to publish on the _metrics flow every second are:

  1. num_clients - number of connected clients
  2. num_msgs - number of messages
    1. num_msgs.sent
    2. num_msgs.received
  3. e2e_time - end to end time from publish reaching server to last subscriber receiving the update, across all messages in the last second
    1. e2e_time.min
    2. e2e_time.max
    3. e2e_time.mean
  4. fanout - number of clients subscribed to any flow
    1. fanout.max
    2. fanout.min
    3. fanout.mean
  5. fanin - number of clients publishing to any flow
    1. fanin.max
    2. fanin.min
    3. fanin.mean
  6. drops - number of published messages that did not reach clients

Metrics are encoded into the data of the topic, and the proto defined above can be used to decode them.

The server must support atleast two transports: TCP and UDP. Try implementing unix sockets or shared memory if you’re up for it. This is to make sure the transport is flexible and can be swapped out as needed.

C++ - Test Driven Development

Questions during dev:

  1. Where should you keep protos? - keeping them with the source files is ok. See here
  2. I don’t know how to write a CMakeLists from scratch ;-;
  3. How tightly should proto types be coupled with code? Just use them at the serialization boundary or have them be a part of the code itself?
    1. Have a mapping/serialization layer at the boundary - keeping it in code is more flexible
  4. What type to use for raw, movable data? std::string?

Graph View