kBytes: Ideas & Computer Things

Understanding Linux Message Queues

Posted on | 2168 words | ~11mins

Introduction

Message queues are message-passing mechanisms that facilitate the implementation of Inter-Process Communication in modern Operating Systems. As message queues are local-only, messages cannot be exchanged over a network, only between processes running under the same address space. Linux, being POSIX-compliant, implements two variants of message queues — System V and POSIX. System V Message Queue API is older, dating back to the original System V Unix in 1983. POSIX, on the other hand, is a more modern reinterpretation of the Message Queue, and supports features such as message notification, out-of-the-box thread-safety, and has a more Linux/BSD-compliant file-descriptor-based queue identifier. However, POSIX queues are often unavailable in multiple systems, while System V is almost guaranteed to be available.

This post will focus on the POSIX message queue. The reasoning is to push towards a POSIX-based message queue adoption, which has been growing in recent times, and to levarage my larger experience managing POSIX message queues as compared to System V queues.

The POSIX Message Queue

POSIX message queues have an inherent concept of message priority. The queue is implemented using a red-black tree to self-balance (improving performance) and to maintain relative priority between messages inserted in the queue. A message of higher priority will be placed in the same tree, but will be read earlier by the reader process as compored to lower priority messages. The POSIX message queue on Linux supports the following main methods:

  • int mq_open(const char *name, int oflag, mode_t mode, struct mq_attr *attr): Attach to a queue specified by the char pointer name. The name must start with a “/” and cannot be larger than 255 (unless changed at a system level). The mode of operation of the mq_open is defined by the flags in oflag. For example, a new queue can be created by using the O_CREAT control flag and O_EXCL to fail if the queue already exists. To attach to the queue in a read-only manner, a process can use the O_RDONLY control flag, and to read and write, the process must use the O_RDWR control flag. Notice the control flag is not the same as permission, just like the mode in which a process opens a file is not defining the permission of that file, but the expected mode of operation on that file. The mode attribute defines the capacity of operation of the opened queue. If the owner decides that all other processes that attach to this pipe can only read from it, then this is where such information would be made. Finally, the attr pointer is used by the queue to maintain management/control information. Return the local identifier of the message queue, just like a file handler.

  • int mq_send(mqd_t mqdes, const char msg_ptr[.msg_len], size_t msg_len, unsigned int msg_prio): Send a message stored in the msg_ptr buffer of size msg_len with priority msg_prio to the queue identified by mqdes (the file handler returned by mq_open). Independently of priority, all messages sent are placed in the tree data structure in the order they are received. At any given time, the message queue cannot have more than mq_maxmsg, defined in the message queue attributes (default of 10). This value can be extended. If the queue is full, mq_send blocks until there is enough space. If a queue is opened with the O_NONBLOCK oflag, then this method fails immediately, returning -1.

  • int mq_receive(mqd_t mqdes, const char msg_ptr[.msg_len], size_t msg_len, unsigned int msg_prio): Receives a message of size msg_len from queue mqdes, places it into the buffer msg_ptr, and places the priority of the message received in the msg_prio variable, if the pointer is not null. In a non-blocking queue (O_NONBLOCK oflag), a call to an empty message queue returns -1.

  • int mq_unlink(const char *name): Unlink the queue identified by name from the current process. Notice once a queue is unlinked, the queue is marked for removal, but is not removed immediately as other processes may still be referring to that queue. Once all processes unlink the queue, it is then properly removed from the system.

Other methods exist but are not covered in this post as the ones presented above are enough to build simple applications using the message queue. One can now understand the overall flow of POSIX-based message queues. A message queue must first be created by a server process, another process then attaches to the message queue (both creation and attachment are handled by the mq_open method), both processes communicate in a bi-directional manner using mq_send and mq_receive, and once communication is over, processes can use mq_unlink to dettach from the message queue and allow the Operating System to free up resources. Now, let us implement the concept of message queues in a sample client-server application.

Example: Client-Server Model

To bring the concept of message queues closer to reality, let us explore the idea of a local client-server system. This system, as the name implies, is composed of a client (an application that initiates communication with the server), and a server (an application that replies to requests from clients). The key to the usage of message queues is the word “communication.” If the communication is to always be carried in the local address space (same machine), then it makes sense for two applications to also share the same message queue, as a message queue will naturally incur way less overhead than network-based communication (e.g. sockets). The server must first setup a publicly known message queue and listen on that message. The code below performs precisely this operation:

#include <mqueue.h>
#include <stdio.h>

typedef struct message {
  int number;
} message_t;

int main() {
  // Message queue attributes (management info)
  struct mq_attr attr;
  attr.mq_flags = 0;
  attr.mq_maxmsg = 10;
  attr.mq_msgsize = sizeof(message_t);
  attr.mq_curmsgs = 0;

  // Create a queue with "SERVER_NAME" as the key, read-write permission
  // for the server process, allow only the server to read and write,
  // while other processes can only write.
  int queue = mq_open("SERVER_NAME", O_RDWR | O_CREAT | O_EXCL, 622, &attr);

  message_t buffer;
  while (1) {
    // Receive sizeof(message_t) bits from the message queue. Passing
    // a null pointer to the mq_receive method just skips the storage
    // of the priority of the message. This is a blocking request, so
    // the call will block here until a message is received.
    int byte_count = mq_receive(queue, (char*)&buffer, sizeof(message_t), (void*)0);
    printf("Received a number from the client: %d\n", buffer.number);
  }

  // You might want to consider a special routine to unlink the server
  // queue on server termination, in order to free up the named queue
  // and release resources back to the system.
}

The code for the server is quite simple to understand. We first initialize the queue with a publicly known name of “SERVER_NAME.” In an infinite loop, we keep on receiving messages from the message queue. Since the message queue is blocking, the while loop get stuck at the mq_receive system call until a message actually comes through. All messages obey the same structure, that is the struct message_t. Every call mq_receive will read no more and no less than sizeof(message_t). Now, for the client:

#include <mqueue.h>
#include <stdio.h>

typedef struct message {
  int number;
} message_t;

int main() {
  // Attach to the queue with "SERVER_NAME" as the key. As defined by the
  // server, this queue is write-only, therefore the client should attach
  // with no more than write-only capabilities
  int queue = mq_open("SERVER_NAME", O_WRONLY);

  // Send a message to the server with priority 0. The message is simply
  // the struct message_t with 10 as the number attribute.
  message_t message = { 10 };
  mq_send(queue, (char*)&message, sizeof(message_t), 0);
}

The client is even simpler than the server. It attaches to the already created server, called “SERVER_NAME,” and sends a simple message to the server before terminating. Notice that message queues are bidirectional, so the server could reply to the client’s message through the same channel, as long as the message queue is read and write for both the server and client. Also notice that the client does not unlink the channel. This is a deliberate choice as unlinking the message queue would mark that message queue for deletion, which in turn would stop any other clients from attaching to the same channel unless it is recreated.

You may have noticed an issue: This is not a multi-client model. First of all, the queue we created does not respect the bidirectional nature of a server-client model of communication. Secondly, even if it did, a single client could listen to another client’s messages. If you are operating in an insecure channel, that is not acceptable. In the next section we extend the example to support isolated bidirectional communication.

Extension: Multi-Client Model

Unfortunately, message queues do not have an inherent concept of isolation. If a message queue is created in the Linux file system, and that message queue is publicly available to the system (no permission flags stop programs outside the message queue owner’s group from reading or writing). The goal of a message queue is to just transport message from a producer (the server in our model) and a consumer (the client in our model). If there are hundreds of clients, the message queue is not worried about who is allowed to read and who is not (if the permission is wide-open to processes outside the owner’s group, which is the majority of application in a micro-kernel, for example). So the question remains, is there a way to provide some sort of artificial isolation in a multi-client scenario?

The answer is yes, but it requires a rather large modification to the model of the application. Instead of just sending and receiving messages through a unique channel without handshake, the client and server will first agree on an isolated channel that they can communicate without eavesdropping. To do so, the server and client must first handshake, exchange a unique channel identifier, and proceed on communicating through that channel. Now, the server can use Linux poll or select functions to listen of all the channels and at once, the handshake message queue and the per-client queues. Once it receives a message from the handshake message queue, it attaches to that client’s message queue. If it receives a message from a client message queue, it replies to that client’s request through the private message queue.

However, this has a potential drawback: malicious processes can exhaust the available number of message queues in the system. If a process abuses the message queue system, the server has no way to know all message queues were opened by the same process, and in turn this might create a nightmare scenario for system administrators. Secondly, client’s may also not unlink their own queues, which forces the server to have watchdogs to close queues that have not been used for a certain amount of time. Finally, client’s can flood the server with requests. The server can have a rate limiter to block the number of requests a client can make on a given amount of time, but if the client is able to open an infinite amount of channels, it is easy to see how this can lead to degraded performance. To ensure trust between server and client, the server must have a signing utility to validate a process’ ability to use the service. Kernel-level Access Control List and message queue number and rate limiting support could also lead to improved security in message queues. Linux for example does support limits to how many queues a process can have open at a time, but it does not provide ACL management for POSIX-based message queues.

Conclusion

Message queues are a powerful feature offered by modern operating systems. They are faster than sockets due to the low overhead of message passing, provide out-of-the-box priority ordering among messages, are thread safe by nature (POSIX only), and enfore the size, shape and integrity of messages in the same address space. If two applications need to communicate in the same machine (and address space), then it makes sense to consider the benefits of message queue and to evaluate if it fits the expected workflow of the system in consideration.

However, message queues can also be frustrating to implement properly, with security being one of its hardest features to get right. Permission management must be flawless for message queues to be fully isolated, and the architecture of a multi-client model requires lengthy setup before communication be done is a safe manner. It is also hard to avoid abuse from malicious process, as message queues do not have inherent Kernel-level mechanisms to trace back a message to a single client, so they might no be the best option in untrusted systems. Message queues are also not portable most of the times, as they require explicit Kernel support for message passing.

The decision to implement message queues or not must be done on a system-by-system basis, taking into consideration the environment, performance, efficiency, and safety requirements before opting for message queues, pipes, or sockets for message passing and inter-process communication.