In today's application architectures, distribution and communication between applications and services are core concepts. To benefit from distribution, you must keep a few basic principles in mind; otherwise, you may easily encounter performance and scalability issues. These problems don't often appear during the development phase, but when you conduct load testing or move to production, you might realize that the software architecture you chose doesn't meet performance and scalability requirements. In this article, we focus on some key points to remember when building distributed applications.
Distributed applications require interaction between them. The scope ranges from simple point-to-point interactions on large-scale cluster architectures to dynamic service-oriented or service-based architectures. Communication across system boundaries is also critical for enhancing the scalability and availability of software systems. Today, software architecture has made distribution a core and essential concept. The Java platform plays a central role because its features, along with good API and product support, enable these characteristics. Use cases range from system integration on standard software like SAP to internal or external service integration. SOA provides such methods to make services and applications flexible and reusable, allowing rapid responses to new market demands. Additionally, trends such as using grid computing, virtual machines, and multi-core blade servers lead to an increasing number of clustered applications. This is mainly driven by the pursuit of high scalability and high availability. Moreover, the trend of cloud computing indicates that distributed platforms will become more popular in the future. Additionally, systems are becoming more dynamic in their flexibility, such as adding application nodes at runtime. These trends also lead to increasingly complex system structures, making it harder for developers to understand how service calls are implemented in products. This complexity and lack of understanding can easily lead to increased resource consumption (CPU, memory, network) and reduced performance.
The Demon Behind the Mask
Today, remote technologies make it easier to implement distributed applications. The details of underlying communication and the infrastructure on the server and client sides are transparent to developers. Nowadays, exposing a Java class as a service sometimes only requires adding a simple annotation to the class. Services can also be accessed easily through proxies generated by tools, as shown in the figure below. However, this is just the tip of the iceberg.
Figure 1. Upper-level architecture of remote protocols
The core blocks of the remote stack are object serialization and formatting of transmission. Typically, application developers don’t need to know these details. However, this is where many performance issues arise. Inefficient serialization means transmitting unnecessary data over the network. Displaying complex objects and large amounts of data leads to high CPU and memory usage during serialization and deserialization. The underlying infrastructure and its configuration significantly impact the application’s performance. On the client side, the main factors are connection management and the underlying threading model. Guidelines for using connections in distributed applications resemble those for database connections. Establishing a connection takes time, depending on the protocol. For example, establishing an HTTPS connection incurs more overhead than a simple TCP/IP connection. Connections are also important system resources, so connection pooling is crucial. Proper configuration is key here because incorrect configurations do more harm than good. The threading model involves how requests are handled. It’s important whether requests are processed synchronously or asynchronously. Synchronous communication blocks a process until a response is received. In asynchronous communication, a callback is invoked when a response is received, allowing the thread to be used for other transactions. On the server side, the number of available worker threads defines the maximum number of service requests that can be processed in parallel. The network itself is also an important component of distributed applications. Networks are more critical in limiting scalability than affecting performance, which is often overlooked during development because no actual network is called.
The Beauty of Remote Calls Lies In...
There are many options available, and Java offers numerous possibilities and technologies for implementing distributed applications. The choice of remote technology has a significant impact on the application's architecture, performance, and scalability. The oldest yet most widely used remote protocol is RMI (as shown in the figure below).
Figure 2. RMI Architecture
RMI is a standard protocol for J2EE applications. As its name suggests, it was designed to call methods on objects located on remote Java virtual hosts. Objects are exposed on the server side, enabling clients to invoke these objects through proxies. The same server-side objects are used by multiple threads. Thread pools are managed by the RMI infrastructure. Communication is handled via TCP/IP and uses JRMP or IIOP/GIOP (CORBA protocol)-based protocols specific to RMI. Application servers also provide their own proprietary protocols to optimize performance. Just as server-side references need to be managed, the RMI infrastructure provides garbage collection to manage references. This distributed garbage collector (DGC) itself uses the RMI protocol to manage the lifecycle of server-side objects. Besides being powerful on the client and server sides, RMI has other implementations.
RMI only supports synchronous communication, the drawbacks of which have been discussed above. Additionally, it cannot provide low-level caching for data-driven services because it is based on a binary protocol. Developers and system architects can change the configuration parameters of the infrastructure to optimize performance. JMS is the second most used protocol on the J2EE platform, as shown in the figure below.
Figure 3. JMS Architecture Diagram
Unlike RMI, JMS is an asynchronous protocol. Communication is queue-based, allowing listeners to react to messages. JMS is not a standard remote call protocol, but it still facilitates service-to-service interaction. Many ESB implementations in SOA use JMS-based middleware to pass information between services. Since JMS is asynchronous, typical synchronization issues can be avoided. In many systems, the key to high scalability lies in releasing resources (like threads) quickly. In many cases, asynchronous processing is the only appropriate method. JMS offers various transport formats. XML is the most common message format, but binary formats are also possible. Designing the message structure is an important part of application architecture because it directly impacts the application's performance and scalability.
SOAP-based Web Services (as shown in the figure below) and related WS-* standards have become increasingly important in the Java enterprise application domain.
Figure 4. Synchronous and Asynchronous SOAP Architecture
SOAP was designed to replace CORBA and initially received strong industry support. Thanks to the joint efforts of WS-I, different platforms can connect relatively easily. SOAP is an XML-based RPC protocol, so it is often associated with bandwidth wastage.
Article Address: [Eden Network] http://www.edenw.com/tech/devdeloper/java/2010-07-20/4817.html
An increasing number of REST-based services are replacing SOAP. REST services in Java are described in JSR 311 and are designed based on HTTP-supported basic operations. However, REST is not an RPC protocol but rather resource-oriented, designed for accessing and manipulating (web) resources. Both protocols support synchronous communication, which is required by the underlying HTTP protocol. WS-Addressing extends the SOAP protocol, so it also allows the implementation of asynchronous services. The biggest advantage of REST is its ability to cache easily through HTTP proxies. REST relies on using the underlying HTTP protocol, regardless of the mechanism used.
Possible Mistakes
Many potential issues can arise in distributed applications, as shown in the figure below:
Figure 5. Causes of Issues in Distributed Applications
On the client side, the main problem lies in poor interaction design—too many service calls or choosing the wrong communication mode. Long-running synchronous transactions can easily cause performance problems. In the communication layer, high network loads caused by large amounts of data and excessive service calls are the main issue. On the server side, inappropriate service interface design and improper use of serialization strategies lead to performance and scalability problems. Let's examine these issues in detail.
Causes of Issues in Distributed Applications
The correct choice of communication protocol mainly depends on the overall architecture of the system and underlying requirements. If you work in an environment where mainframes, Java, and .NET components interact, using SOAP won't work. In a pure Java environment, using RMI on JRMP remains the best-performing and most scalable solution, offering out-of-the-box programming support. In many SOA implementations, SOA and Web Service-based implementations are synonymous. Therefore, there are increasingly more use cases of pure Java applications using SOAP as an RPC protocol, even though adopting such a method has no advantages. Surveys show that using SOAP frequently still makes sense compared to RMI-JRMP. Besides the standard protocols described, some other XML-based and binary protocols are used in some applications. Hessian performs well, and there are implementations in other programming languages. For example, using Spring to expose POJOs to remote calls makes switching between different protocols without changing the implementation relatively easy. Spring supports RMI, HTTP, Hessian, Burlap, JAX-RPC, JAX-WS, and JMS.
Anti-pattern: Chatty Applications
A core principle in building distributed applications is to minimize remote calls. This means reducing the cost of data serialization, the cost of establishing connections, and additional network loads. Furthermore, the consumption of CPU, memory, and network resources limits scalability. Therefore, designing interfaces for remote applications in a way that ensures the minimum necessary number of service interactions is very important. Especially for applications initially built locally and then encountering excessive service interactions for scalability reasons. These issues often appear during load testing or production but show no problems during local development and testing. Appropriate performance management methods can avoid these problems by analyzing remote behavior during development. The figure below shows an example of analyzing a remote application's behavior using dynaTrace.
Based on this analysis, interfaces can be redesigned, and application logic can be rethought to reduce the number of remote calls. Possible methods include combining the logic of several methods into one or using data containers around objects in several call requests. The location of specific data can also help reduce remote calls because the data is available where needed. Particularly when reading data, using caching can significantly improve performance and scalability. It is important to consider the distribution of services and possible communications early in software design when they become requirements or are likely to become requirements.
Anti-pattern: Large Message Formats
When calling remote services, it usually means that data will be transmitted over different protocols. Like XML transmitted over the SOAP protocol or binary data transmitted over the RMI protocol. Most technologies transmit either the data of objects or the objects themselves. Serialization usually occurs in the underlying remote implementation. The cost of serialization corresponds to the size of the transmitted objects. In practice, we find that serialization costs account for up to 98%. How does this happen? An authorization service interface needs a user object for authorization. This user object not only includes the username and password but also many attributes and references to data for other use cases. Standard SOAP serialization creates data messages of thousands of bytes. These data must be parsed and mapped to the user object structure by the service, leading to high CPU and memory consumption. The solution is clear. The interface should be refactored to require only the user ID and password. Thus, besides choosing the right remote technology, designing the content of the messages is crucial for building applications with good performance and scalability. Often, designing well-structured general objects brings high-performance returns.
Anti-pattern: Distributed Deployment
Distributed Java enterprise applications can result in an application being split into multiple services and deployment units deployed on some application servers. Distribution means that a new deployment package for one component does not require redeploying other components. Another possibility is that heavily used services can be deployed on independent hardware or deployed multiple times. For complex applications with many deployment units, understanding service interactions becomes increasingly difficult. This can lead to two services that interact frequently being deployed on different hardware, resulting in many remote call situations. In large-scale applications, analyzing the frequency of interactions and data size consistency with the deployment structure is important. Often, moving from distributed deployment to local deployment can greatly improve performance without sacrificing flexibility and scalability. Especially for stateless services, deploying them on different nodes enhances their locality.
Conclusion
From these anti-patterns, it is evident that considering scalability in the early stages of application design is crucial. It is a key driver of application architecture. Improving performance and scalability later in the process is much more challenging in most or all cases. Detailed analysis of the application product to identify the frequency of remote calls or large data volumes and optimizing system consistency is indispensable. If you encounter similar or different problems, please let me know so I can expand my anti-pattern records.
Article Address: [Eden Network] http://www.edenw.com/tech/devdeloper/java/2010-07-20/4817.html