The top 15 WebSphere MQ best practices

http://www.ibm.com/developerworks/websphere/library/techarticles/0807_hsieh/0807_hsieh.html

Introduction

IBM® WebSphere® MQ (hereafter called MQ) is IBM’s message-oriented middleware product. It enables independent and potentially non-concurrent applications on a heterogeneous system to communicate with one other. It is supported on more than 80 different platforms and environments.

One of the strengths of MQ is its ability to be highly configurable and customizable for diverse customer environments and data transmission needs. However, this strength can open the door for poorly configured systems that don’t support future expansion, changing development standards and protocols, and robust security. This article describes the most common best practices in designing, building, running, and maintaining MQ solutions in order to achieve the full benefits of MQ. Keep in mind that not all recommendations are appropriate for all situations, and that they are offered as guidelines, not hard-and-fast rules. Large, complex enterprises may often have good reasons for deviating from some of the recommendations. This article was written based on MQ V6 and assumes you have a basic knowledge of MQ.

MQ enables application designers to construct time-independent architectures. Its versatility, flexibility, and wide range of implementations and environments make it difficult to describe procedures to cover all cases. This list of best practices should enable architects, system administrators, and application developers to avoid common mistakes and make optimal use of MQ in their solutions. The list is organized according to the four solution phases of design, build, run, and maintain.

Design

Use short names for queue managers and MQ objects

Sometimes just naming a queue is a challenge. In other instances, you’ve created so many queues and other objects that your naming convention no longer works or is not as descriptive as you’d like. In general, names of MQ objects can have up to 48 characters, except for channel names, which can be no more than 20 characters. You can use upper and lower case A-Z, numeric 0-9, underscore (_), period (.), and two special characters: forward slash (/) and percent (%). If either of these two special characters are used, the names must be enclosed in double quotes.

In addition to general rules, here are some recommended guidelines for naming queue managers and MQ objects:

• Use full UPPERCASE for all objects, including the queue manager, to prevent portability issues in heterogeneous environments.
• Queue manager names should be unique in the MQ network and reflect the location, function, and environment of the queue manager (dev, test, etc.). Channel names should reflect their function and origin/destination queue manager connection flow; use FROMQUEUEMANAGERNAME.TOQUEUEMANAGERNAME for all sender and receiver channels and TO.QUEUEMANAGERNAME for clustered channels. If you have more than one channel connected to the same manager, extend the convention with different priority or protocol in the name but stay within the 20 characters limit. Another common best practice is to use SOURCEQUEUEMANAGERNAME.TO.DESTQUEUEMANAGERNAME
• Keep queue manager names short, no more than 8 characters.
• Queues should not have the word QUEUE in their name (since it’s implied), nor should they have the topology in the name (local, remote, alias, and so on).
• Applications should have a high-level qualifier consisting of a few characters, followed by some delimiter (like a dot), so that the names sort easily and the purpose of the queue is obvious (such as APPXYZ.SERVICE1.REQUEST).
• Do not use spaces in any names of MQ objects, including system host names.
• For MQ authorizations, even though the maximum length of group names and user IDs is 20 characters, keep them under 12 characters to prevent complications.
• Although the forward slash (/) and percent (%) special characters are allowed, avoid them because they can cause cross-platform difficulties.

Always assign a dead letter queue (DLQ) to the queue manager

The DLQ is a local queue that is also referred as the undelivered-message queue. It’s a good practice to create and assign one for each queue manager and use it to catch messages that are not sent due to network or destination issues. If you do not define a DLQ, errors in the application programs may cause channels to shut down. If this happens, not only will the queue stop receiving messages, but it may effect the normal operations of MQ — for example, administrative commands might not be received and executed. Under restrictive application environments where exact processing of messages in order needs to be assured, a DLQ can be omitted to force a shutdown of the processing. It is recommended that applications be designed to avoid this scenario.

In addition, the DLQ should be monitored as messages arriving in that queue are likely an error. DLQ’s should be defined, available, and sized for the largest messages the system is expected to handle. In more robust environments, set up a dead-letter handler as a trigger so that the dead messages are automatically re-tried without intervention. If the retry fails, then custom DLQ rules should be used to forward or remove the messages as needed. MQ provides a sample DLQ handler (runmqdlq/amqsdlq) for your learning and experimentation.

If you have not associated a DLQ to an existing queue manager, use the MQSC ALTER command to add the DLQ to the queue-manager object.

Use standards like JMS whenever possible

Today’s dynamic business needs are driving IT departments to implement standards-based computing. JMS is a J2EE based standard for messaging, and provides a standard API for applications to use when performing enterprise messaging with application portability. Although JMS is a vendor-independent Java messaging API, its implementation may differ between JMS providers, and it is important to understand the variations. Using standards such as JMS in your solution will facilitate application portability and reduce dependency on vendor-specific API’s. This will in turn ease integration challenges, and reduce vendor tie-in. SOAP/JMS messaging is increasingly becoming a Web services supported platform, which will help customers in their SOA-based implementations.

MQ is one of the more popular JMS providers to J2EE applications in the market today. MQ JMS implementations can interoperate with other MQ programs without the need for bridges. The MQ JMS implementation supports both point-to-point and publish/subscribe messaging models. MQ is supplied with an integrated broker in MQ V5.3 Fix Pack 8 or later. With standardized JMS APIs, you can upgrade to other JMS brokers, such as WebSphere Message Broker, without redevelopment of the applications or interfaces.

Build with performance in mind

Although MQ was built for high transaction volumes, poor architecture and application design can compromise its ability to process messages as fast and effectively as possible. To mitigate potential performance issues, here are several recommendations:

• Message size and length can affect the performance of the application that processes the message, and the network time of data transmission. Send only essential data in the message.
• Use persistent messages for critical or essential data only. Persistent messages are logged to disk and can reduce the performance of your application.
• Retrieving messages from a queue by message or correlation identifiers will reduce application performance. It causes the queue manager to search all messages in the queue until it finds the desired message. If applications have high-performance requirements, applications should be designed to process messages sequentially.
• The MaxMsgLength parameter stores the value for the maximum size of a message allowed on the queue. The 4 MB default can be changed to better align with your application processing needs, which will have the benefit of using system resources in the most efficient manner.
• Ensure that messaging applications are designed to work in parallel with each other and with multiple instances of applications. The queue manager executes one service request within a queue at a given time to maintain integrity. Avoid programs that use numerous MQPUT calls in a sync point without committing them. Affected queues can fill up with messages that are currently inaccessible while other applications or tasks might be waiting to get these messages.
• When applications have intermittent message transmission needs, use the MQPUT1 call to put only one message on the queue. For higher volume applications, where multiple messages are being put, the traditional usage use MQOPEN call followed by a series of MQPUT calls and an MQCLOSE call is more appropriate.
• Keep connections and queues open if you are going to reuse them instead of repeatedly opening and closing, connecting and disconnecting.
• The maximum number of threads an application can run on a system can affect the performance of the solution, especially on Windows®.
• Configure channels with a disconnect interval so that they can go inactive when there is no activity on the channel after a period of time. This will reduce overhead and help improve overall performance.
• MQ performance is commonly bound by disk I/O writes. Ensure that the storage team is involved with disk layouts to ensure the fastest reliable disk writes possible.

Avoid location assumptions and fixed queue names in programs

Avoiding the use of hard-coded MQ objects and location-names in your applications will provide for significant architectural flexibility, portability, and deployment flexibility. Deployment flexibility will be enhanced by leveraging alias queues and/or model queues along with WebSphere environment variables. Adhering to these rules enables applications to be deployed on different environments (unreliable network, mixed operating system, communication protocol, and mixed language environments) without changing the application.

When feasible, cluster your MQ servers

If your infrastructure can support it and your budget allows it, hardware and software clustering provide a great way to increase the resiliency, scalability, and performance of your MQ solutions. Here are some recommendations when considering MQ clustering:

• MQ queue managers do not restart automatically after crashing, and in general, neither MQ clustering nor hardware-based clustering like HACMP provide this behavior by default. If this functionality is a requirement, hardware-based clustering solutions and system automation tooling can usually be configured to provide this behavior.
• Applications that have message affinities may lead to cluster workload management routines that are complicated and less than optimal. Removing message affinities in your application can increase you application availability and scalability options.
• Unless you have specific message affinities within your application, do not use the MQOO_BIND_ON_OPEN option on an MQOPEN call to force messages to be sent to a specific destination. If the destination queue manager is not available, the messages are not routed to another queue manager in the cluster.
• Do not use generic names such as CONNAME for your channel. You need to specify the host name or IP address of the full repository queue manager.
• With appropriate permissions, you can PUT messages with a destination of any queue in the cluster; however you can only GET messages from a local instance of a cluster queue.
• You must choose at least one and preferably two queue managers in the cluster to serve as the full repository. Adding more than two full repositories often degrades overall performance, because the cluster will need to send additional traffic and spend more time maintaining all of the repositories.
• When deciding which servers will host the full repositories, select servers that are highly reliable, well-managed, and have a static IP.
• Use hub and spoke or bus models for maximum flexibility and performance, especially in large environments.

Design infrastructures with fewer queue managers with more queues

For architectural as well as performance reasons, it is usually better to create one queue manager with 100 queues as opposed to 100 queue managers with one queue apiece. Where it makes sense, try to limit the number of queue managers in an MQ environment. A single queue manager per server can usually fulfill the needs of all of the queues and applications on that server. While the queue manager can fulfill multiple roles, the segregation of responsibility should occur at the queue level, preferably with a function identifier in the queue name.

Build

Capture completion and reason codes in all application MQI calls

Design your applications to take advantage of MQ completion and reason codes returned from MQI calls, which enables an application to determine if the message arrived safely and was processed correctly, or if there was a problem with the delivery of the message or processing. When you need to call IBM Support, reason codes along with your problem description help speed problem determination and resolution.

Tips when writing development code for MQI calls:

• MQ methods will throw an exception whenever the completion code or reason code resulting from an MQ call is not zero.
• When using API’s other than MQI, be sure to capture the linked exception. In the case of the MQ implementation of JMS, MQ raises a java MQException that contains the completion code, reason code, and details of the exception. Do not return just the JMSexception, as it will not contain the necessary completion and reason codes.

Code applications to continually process messages

When writing an MQ application, consider pulling messages off the queue as soon as possible and deciding whether to process them immediately or to send them to a failure queue. Do not design the application code that requires messages to be cleared (such as messages that are not in the anticipated format) or needs administrator intervention prior to the application being able to process additional messages off of the queue.

Close and disconnect connections properly

Code applications to properly close and disconnect their connections prior to disconnecting or shutting down. Failure to do so, especially when client connections are in use, will result in hung connections, which will increase resource consumption and may continue until the maximum connections are reached, which will block new connections.

Be aware of the different features of various MQ clients

Not all MQ clients provide the same functionality. When deciding which MQ client to use, keep in mind the following points:

• Understand the connection options, such as client binding, server binding, managed client binding modes, and the limitations of the client for C, JMS, Java, or .NET applications. For example, the .Net MQ client does not have support SSL channel encryption, XA transactions, and channel compression. Another example is that the Java MQ client supports only the TCP/IP transport and does not read any of the MQ environment variables at startup. In this situation, all environment variables including the connection information are stored in the MQEnvironment class in the qm.ini file.
• Network and firewall problems may cause MQ connections to fail, and may be misdiagnosed as MQ problems.
• Assign connecting applications their own unique channel definitions, so that administrators can easily determine which application is connecting, as well as implement additional security segmentation options.
• In production systems, the SYSTEM.DEF.SVRCONN channel should have a non-privileged user added to the MCAUSER field, and potentially be placed in a stopped status. Applications should not be configured to use this channel, and should instead be assigned their own distinct channel. Besides being a best practice not to utilize system objects, this channel is a known security hazard.
• When you set a value for the MCAUSER attribute, enclose the value in single quotes to avoid issues with case sensitivity on distributed platforms.

Run

Use security best practices

On distributed systems, the mqm group provides administrative access to all MQ resources on a system. Therefore, limit membership in the mqm group.

Security in MQ can be divided into two broad categories: MQ consumers and MQ administrators. Consumers normally take the form of the IDs used to execute applications that connect to MQ. Administrators are users who need to interactively query or alter MQ configurations via the included tooling (such as runmqsc). In some environments, the application-account IDs (consumers) are sometimes placed in the mqm group for simplicity. In these situations, these IDs should be locked down so that they cannot be used for interactive login, since any user who gains access to these IDs will have full administrative control over MQ. An alternative is to use the WMQ OAM setmqaut facility to specify detailed API-level permissions for each resource or MQ object (such as queues). This technique gives much more granular control and is preferred when feasible.

Regarding administrative authority, where security and auditing are a concern, third-party security or configuration tools such as the freely available SupportPac MS0E should be used for accessing and altering MQ configurations. In absence of these controls, only members of the MQ administration team should belong to the mqm group. Other users should be given lesser levels of authority, which might be granted via OAM-level setmqaut commands or through the use of MQ SupportPacs such as MS0E.

Restrict remote administrative authority using SYSTEM.ADMIN.SVRCONN channels on production-level (restricted-access) boxes

In MQ V5.3 or earlier, the Windows-based MQ Explorer administrative tool connects to queue managers using the SYSTEM.ADMIN.SVRCONN channel. Facilities did not exist to further secure that channel, causing security hazards for restricted-access systems since anyone with knowledge of the queue manager can connect with full administrative access and no audit trail when using the Explorer. However, this channel is not created by default and must be added manually. In MQ V6.0 or later, any client channel can be used for administrative access, so consider additional measures such as SSL when remote access is enabled on production systems.

Do not use the sample get/put utilities for production purposes

MQ comes with sample programs that illustrate how to get and put messages. These samples come in both compiled and uncompiled forms. Do not use these programs verbatim as production-grade applications to get or put messages, since they have buffer limits and may not have the robustness or functionality needed by production-grade applications.

Use runmqlsr in lieu of inetd listener

In MQ V5.3 or later, the runmqlsr network listener program allows for multi-threaded connections. The advantage of using a multi-threaded process as opposed to initiating a new process per connection via amqcrsta is decreased system resource usage, as well as eliminating potential administrative outages. If you have systems with large numbers of connections, such as an integration hub or ESB, and existing listeners running via the UNIX-based inetd listening service (in combination with the amqcrsta program), consider migrating those listeners to runmqlsr.

Place the listener port in the qm.ini file in the TCP stanza (or in the Windows Registry) so that ports are well associated with queue managers. Then, when the listeners are started, no command-line port number needs to be referenced. See MQ documentation for stanza details.

Use SupportPacs to extend MQ functionality

MQ offers many free, downloadable plug-ins called MQ SupportPacs to extend functionality:

• MS03: savequeue manager — Provides code that exports all MQ object settings from runmqsc in a format that can later be reused to import (and recreate) queue-manager configurations. Depending on your environment, it can be vital for backup and recovery.
• MS0E: runmqadm — Provides an administrative wrapper that offers runmqsc and administrative-level access to users who are not members of the mqm group, enabling you to grant privileges in a much-more-granular fashion.
• MA01: q utility — Lets you browse messages and move then between queues.
• MO03: qload utility — Lets you move messages to and from files, for transport to different systems or for later reuse.
• MS81: MQIPT — An additional product that works in conjunction with MQ to tunnel connections through firewalls via SSL or HTTP(S). It can be used to aggregate connections from multiple queue managers across a single transport and port.
• MC91: High availability — Provides scripts and instructions for configuring MQ on highly-available UNIX systems such as HACMP and Sun Cluster.

Maintain

When not using third-party configuration tools, automate all configuration and changes via MQSC scripts

Make a practice out of writing any changes to queue managers into runmqsc commands in the form of a script. Any commands that can be typed into a runmqsc editor can also be written to a file for later execution. The script can be validated prior to execution using the runmqsc -v < filename switch. At execution time, output from the scripts can easily be captured to a log file for later review. This practice will help avoid costly typos and speed up execution of changes at execution time.

Use the appropriate logging mechanism

A common misconception in the MQ community is that linear logging is more appropriate for production-level systems. In addition to the circular logging benefits, linear logging only incrementally provides object-recovery abilities, and it comes with the cost of regular maintenance to prevent file systems from filling. If the MS03 SupportPac is set up to run regularly, a copy of objects can be replayed from those scripts, minimizing the need for linear logging. Circular-logging queue managers require less maintenance and have less risk of crashing due to using up disk space.

Configure automated maintenance of queue managers

Consider creating scripts to automate the regular maintenance of queue managers. Maintenance should include a regular savequeue manager (MS03), security setting backup (amqoamd), linear-log cleanup (if applicable), and backup of other important settings such as .ini files. Also, consider reviewing the contents of the FDC error directory for any FDC files that are old and should be removed or are new and should be examined since they indicate critical errors in the MQ system.

Many third-party products can help provide this functionality. Consideration these vendor products as well as the freely available SupportPacs before writing and maintaining your own scripts, which will require maintenance with each new release of MQ.

Schedule and apply fix packs on a regular basis

Many calls to IBM Support occur because of problems that have already been fixed in an IBM fix pack. Therefore we encourage administrators to plan, schedule, and apply MQ maintenance on a regular basis.