Sample chapter from:
Web Services Implementation Guide

Authors:
Brian Travis and Mae Ozkan

About the book:
Getting value from stuff you don't own is a compelling idea. Just think, someone else has created something that has value, and has made it available to you. You can now leverage its value in ways that makes all parties richer. That's the promise of web services. You need to know about web services. The advent of standards-based web services marks a new era of system development. First mainframe, then client-server, the Web, now web services makes it possible to leverage every legacy system of the past and do so much more with your information assets.

This book will show you what a web service is, and how you can align your systems to take advantage of this new concept in system integration. This book covers the concepts and technologies of web services.

Web Services Implementation Guide is a multiple-volume set. Volume 1 shows you how to align your thinking and systems to get the most of web services. Volume 2 is a technical treatise for developers. Volume 3 discusses real-life implementations of web services solutions.

ISBN:
0964960230

In this chapter:

Invoking Services
- Method-based
- Message-based
Accessing Services
Coupling
Routes to Access: The Integration Matrix
Tightly Coupled, Method-based Systems
Tightly Coupled, Message-based Systems
Loosely Coupled, Method-based Systems
Loosely Coupled, Message-based Systems
Mutual Ignorance
Loosely Coupled Systems and XML
Integration Problem Statement
Hub Model With Adapters
The Broker
SOAP in Application-to-Application Integration
So, What's It All Mean?

Chapter 9 Integration Models

The integration models we have today are either loosely coupled or tightly coupled. They are message-based or method-based. They are synchronous or asynchronous. Putting these things together in an intelligent way is essential if you are going to archive the promise of web services.

In this chapter, we will discuss all of these models, and show which ones are used by web services and why.

Invoking Services

Objects can be invoked using methods or messages.

Method-based

A method-based invocation is one where the name of some method is known, and the method is invoked by its name, passing whatever parameters are needed by the method. Most object-oriented systems are method-based, including COM and CORBA. Systems that want to access these objects need to know what the methods are, and how they are exposed.

One way objects expose their methods is through an interface definition language (IDL). By interrogating an object's IDL, a system will know its method names, parameter names, and datatypes of the parameters, and the return values.

Message-based

A message-based system is one that relies on messages to invoke an object. These messages are self-describing, and can be routed to the appropriate object via some intelligent message-transport system that can read the message and determine its end point. In a message-based system, a client that wants to access an object must know only what data that object needs. As a result, implementation details of the object are not necessarily known to the client. The data in the message interface is the only common knowledge. An example of a message-based system is a document queuing system.

Accessing Services

Objects can be accessed synchronously or asynchronously.

Synchronous

A synchronous system requires that a response come back right away when a request is made. Synchronous calls are usually made in high-performance systems that cannot wait for an answer.

An example of a synchronous request is a method call of a CORBA object. When a method is invoked, the calling program usually waits for some response. The program will lock up until the response comes back. For synchronous calls, it is good programming practice to set some kind of timeout, so that if the call does not come back in a certain time, a supervisor program can unlock the calling routine and provide a graceful exit.

Organizations need to communicate synchronously in order to find answers to important questions that require decisions in real time. These real-time synchronous communications usually work using a request/response model. For example, when a travel agent is looking for a plane seat, she needs an answer right away. When you are at the supermarket checking out using your credit card, the credit card machine calls the bank and gets approval in real time.

Asynchronous

An asynchronous system is one that will issue some kind of a request and wait patiently for a response. A system that uses asynchronous calls will usually set some kind of event handler to listen for a response, then it will make the request and continue on. When the listener receives the event indicating that the response has been received, it will inform the application that the data is ready.

Asynchronous systems are more efficient, because they do not spend time waiting for a response. This multi-threaded approach is popular because of its versatility, but there are some problems, also.

Programming for asynchronous systems is more difficult than for synchronous systems because of the management of the listeners. Also, it is possible to have more than one asynchronous thread happening simultaneously. If this happens, it is possible that responses will come back out of order and need to be sorted by the application.

Synchronous vs. Asynchronous

To get an idea of asynchronous versus synchronous communication, think of the difference between ordering a custom part for your monster truck through the mail versus talking to someone on the phone.

To order by mail, you need to specify every detail of the part, including methods for payment and shipment. When the receiving organization gets your letter, they will process it and return a response to you.
To order by phone, you might say "I want to buy a part". The operator says, "What kind of part?" You say, "A five ton gas filled shock absorber." Clerk: "We have most colors in stock. What color would you like?" You: "It needs to be blood red." Clerk: "That's funny, but we had a run on those parts just yesterday, and they are on back order. Would you like a different color?" You: "Yes, how about the screaming yellow one. Is that in stock?", ...and so on.

The first is an example of an asynchronous architecture using a self-describing message. It was necessary that the message sent to the parts vendor contained all of the information required to make the transaction work.

The second example is a synchronous communication system where a dialog is made in real time. When you are on the phone with someone from the call center trying to get some help, the questions you ask get answered in real time. This is more efficient than getting support by mail, but it requires more resources from the call center operator.

Synchronous communication is better at dealing with changes in real time, but it is expensive. In the telephone example, the organization needs to keep telephone operators ready to take calls, whether they come in or not. During peak times, callers might receive a busy signal or need to listen to elevator music until a clerk is ready. On the other hand, asynchronous messaging can be done in leisure time, creating a system where peak loads do not have much of an effect on processing time.

The trick with asynchronous messaging is seen here. If there is a problem with the order, the asynchronous message should have information about alternatives if the desired item is not in stock. This is where most problems arise in asynchronous messaging systems, and it is an area where you should take the time to anticipate all conceivable problems before they arise.

A synchronous communications architecture usually requires that two systems be somewhat compatible in order to communicate. In system terms, a synchronous call might be a method invocation of a CORBA object, which returns useful data or a result code. This tight integration is fine as long as all applications can speak the same interface protocols. However, if there are two systems that do not speak the same interface protocols, there is a requirement for bridges or other code that provides interpretation between the incompatible systems. These bridges are usually pretty unreliable and are a common point of failure.

Coupling

A system can be tightly coupled or loosely coupled.

Tightly Coupled

A tightly coupled system is one where the relationship between two or more providers requires them to communicate with each other directly. A tightly coupled system requires all parties to understand many implementation details about each other. Tightly coupled client components can create dependencies on a specific hardware platform, a specific operating system, a specific interface model, a specific programming language, and a specific match between versions of client and server components.

An example of a tightly coupled system is Microsoft's COM Transaction Integrator (COMTI) component. COMTI can integrate with legacy mainframe applications by pretending to be a mainframe terminal. The COMTI object can be used to fill in a screen full of data, send it to a mainframe application, then scrape the response screen coming from the mainframe. A COMTI application is designed for a particular mainframe program user interface. Any changes in the program or the user interface requires changes to the COMTI application.

In the tightly coupled model, then, the client needs to have intimate knowledge of the implementation details of the server program.

Tightly coupled systems, therefore, can make some assumptions about trust. That is, they can assume that the intimate knowledge the client and server have of each other requires and demands that they trust each other.

Loosely Coupled

Two systems are considered loosely coupled if the only mandate imposed on both systems is to understand the operations published by a service and the parameters required to invoke that operation.

An example of a loosely coupled system is the topic-based publish-and-subscribe model used by knowledge management systems.

This loosely coupled architecture must take into account the fact that the server might not trust the client. This complicates the system, but is probably a good design goal, anyway.

Tightly Coupled vs. Loosely Coupled

A tightly coupled system is more efficient than a loosely coupled system because the calls invoked can be made directly rather than indirectly or through some intermediary. For this reason, tightly coupled systems are used when execution speed is an important factor.

However, a tightly coupled system is difficult to maintain, because changes in either side usually require the other side to adapt immediately. For example, if the order of the parameters is changed in an object call, or the datatypes are modified, the calling system needs to be updated to reflect those changes.

On the other hand, implementation details in a loosely coupled system can be changed, as long as those changes do not affect the functionality of the subsystems that are called.

Consider the real life example described above in the section on asynchronous messaging of the telephone call to the parts operator. This scenario requires that both sides first of all are using the same type of telephone equipment. This is not really a problem these days, because just about any telephone in the world can make a physical call to any other. Once connected, however, both parties must speak the same human language. If one party is speaking French and the other understands only German, a compatibility problem arises.We need a bridge in the form of a human interpreter to make the connection. This could result in inaccurate translations and miscommunication between the parties.

Using asynchronous message-based architectures eliminates the need for systems to be tightly linked to each other. Loosely coupled architectures have the advantage that messages are passed, instead of method invocations. These messages provide a degree of independence between the sending system and the receiving system. Systems that use messaging are called loosely coupled architectures.

The Java Remote Method Invocation (RMI) architecture is natively a tightly coupled procedure. Likewise, when a Visual Basic client calls COM object, it expects the COM object to adhere to a tightly defined specification.

Routes to Access: The Integration Matrix

In real systems, you will have a combination of message-based/method-based, synchronous/ asynchronous, loosely coupled/tightly coupled interactions. In fact, these concepts are usually tied together to create reliable or efficient systems.

For example, most method-based systems are tightly coupled and probably synchronous. A message-based system is usually, by definition, a loosely coupled system. Asynchronous calls can be made with messages or methods, depending on what you need to do.

An asynchronous messaging architecture allows systems to communicate by sending self describing messages back and forth. These messages contain enough information to make some processing happen. The transmitting system is obliged to create a message that contains enough information to convey its desire, and a receiving system is obliged to read the message and do something useful.

The receiving system is also able to reject the message if there is a problem. It is good manners, however, to tell the transmitting system what the problem is.

What if we want a synchronous loosely coupled system? We have technologies to accomplish loosely coupled asynchronous systems. However, some scenarios require synchronous transactions. Business is usually asynchronous because of reliability and scalability reasons. However, some business functions need to be synchronous, such as some queries for inventory, schedule changes, credit card transaction approvals or disapprovals, etc.

The table in Table 4, The Integration Matrix shows a matrix of coupling versus invocation.

Tightly Coupled, Method-based Systems

A method call requires the knowledge of the object names, support of object types, and a request made in the object's language. As a result, the method calls, such as Remote Method Invocation (RMI) and Remote Procedure Call (RPC), are tightly coupled, method-based integration methods.

COMTI, COM/DCOM, CORBA are also examples of tightly coupled, method-based systems. The integration is designed and architected for a certain interface, form, and language. We cannot take already existing tightly coupled interfaces and use them for other pieces of applications. The client and the server both need to speak the same language and usually need to be members of the same technology.

Tightly Coupled, Message-based Systems

A tightly coupled message-based system is MAPI. The Mail API is a message-based architecture because members communicate with other members by passing well-defined messages to each other. However, the messages they send are tightly coupled to the API, such that they are bound to that application and cannot be easily used by any other.

However, in most cases a tightly coupled, message-based integration model is usually an indication of bad architecture. Such a system has the disadvantage of being message-based (complexity in creating and parsing messages), but does not have the advantages of tight coupling (fast, synchronous access).

Loosely Coupled, Method-based Systems

SOAP supports synchronous calls by definition. It is natively designed for remote procedure calls outside of the organization. Since method calls instantiate the objects outside the organization, it is not built on a trusted architecture. SOAP provides a structured message definition, which enables loosely coupled architecture even for method invocations which are synchronous.

SOAP has two personalities, which are reflected here. SOAP is inherently a message- based architecture. However, when SOAP is invoking methods on remove procedures, it feels more like a method-based architecture. In that case, we will put SOAP for RPC in this category. See The Two Personalities of SOAP on page 150 for more on the two personalities of SOAP.

Loosely Coupled, Message-based Systems

SOAP fits here, too. SOAP can be used to send documents that describe themselves. These self-describing messages can be processed by any system that understands them. One category of such a system is a queuing mechanism.

Message queues such as Microsoft's MSMQ and IBM�s MQ Series offer asynchronous integration in a reliable manner, meaning they ensure one-time guaranteed delivery with error-handling and error-recovery mechanisms.

In addition to message queue applications, SOAP is message-based and loosely coupled by its definition. Consider two programmers, one that is writing client applications in C on AIX, and the other who writes COBOL programs on a mainframe computer server. They use IBM MQ Series queuing system to communicate.

These applications exchange business documents or record layouts. Since the client is unaware of the server's applications, languages, and error handling mechanisms it is loosely coupled system.

If a mainframe application and an AIX application are integrated using MQ Series, than the programmers on the mainframe side only need to know how to program in mainframe applications like COBOL and how to use MQ Series to exchange messages. Only the structure of these messages and schemas need to be known to both sides. Likewise, on the AIX side, programmers need to know only how to program their native applications and how to use MQ Series to exchange messages.

Therefore, programmers on both sides know a standard application and their native platform and language. The integration architecture is a loosely coupled and asynchronous. Consider a scenario where there is a COM object that we need to access. The first thought is to write a Visual Basic client, since it speaks the same language as the COM object and has the same type definitions. The client and server live in the same world. It is the same scenario for the Java programs calling CORBA objects.

However, imagine a loosely coupled system where a program, whether it is a Visual Basic client or a Java application, sends a message to the server object. In response to that message, the server object's behavior will be exactly the same regardless of who the requestor is and what native language it is written in.

In our case the server, our COM object, is unaware whether a Java client or a Visual Basic client is the requestor calling to invoke a method. There is no difference between a Visual Basic and a Java program calling the COM object, because the data that COM object needs for a method invocation has been sent using a standard language and format that COM object understands.

As a result, this remote procedure call, which was natively a tightly coupled system, becomes a message-based, loosely coupled system. This scenario is described in Chapter 7, Simple Object Access Protocol (SOAP) on page 139.

The client requesting an answer is unaware of the server's implementation, language, alphabet, and native format. When the answer comes back to the Java or Visual Basic client, the answer is not in a COM definition. It is in a standard message that will work for Java, COM, COBOL, and whatever else can create and use these messages.

Mutual Ignorance

As a result, there is mutual ignorance between the client and the server. Loosely coupled systems ease integration problems. Yesterday, a COM object was accessed by a Visual Basic application. Today, a Java application can call the object. Tomorrow, a Java server application can expose its data to a Visual Basic application.

As a result, any object can feed its data to any other object in a message-based, loosely coupled way synchronously and asynchronously, whatever the business and applications require, as long as the applications support the same standard message- based interface, speaking the standard language.

This standard language is SOAP. SOAP is embraced by the industry and solution providers that support Java, COBOL, COM, and many others. As a result, these stated heterogeneous technologies now support SOAP as a protocol in addition to their native supports.

This utopia is the basis of web services.

Loosely Coupled Systems and XML

In loosely coupled systems, the communication is usually accomplished by messages. Let's think about a scenario where a brand company is outsourcing some of its production to a manufacturing company that has spare capacity. The brand owner company's production system is updated in real time when the formula of their product changes, or when their inventories change. These changes need to be transmitted to the manufacturer.

Production needs to be updated synchronously. However, because the manufacturer is outside of the brand company's firewalls, how will the production get updated in real time as requirements change?

If the applications are outside of the company, tightly coupled systems do not work. No company will give the details of its implementation so that the client will access the server that resides behind the company firewalls with all of its valuable information. So how do we create a system that can get information, in real time, between enterprises?

Answer: SOAP

As a result, we need the advantages of message-based systems in addition to the advantages of synchronous systems. SOAP defines a standard structured message that is exchanged between partners. The good news is that SOAP enables synchronous as well as asynchronous communication opportunities in loosely coupled way, without having all parties concerned with each others' implementation details.

If the tools and knowledge for building SOAP messages and structures was expensive, such as they are in the EDI world, SOAP would be adopted only by the largest players. However, XML syntax, and the tools for building vocabularies using schemas, are cheap and easy to use. Translation from one XML vocabulary to another is also cheap using XSLT.

Therefore, XML syntax, along with other standards that are built on top of it, are important in loosely coupled environments. The standards and tools provide easy plumbing and integration in a loosely coupled manner. That is why XML standardization and loosely coupled applications go hand-in-hand. The structured and standardized message formats enable the communication and integration between different applications and objects inside and across the firewalls.

SOAP in EAI and Middleware Applications

As the SOAP standard became understood, providers of Enterprise Application Integration (EAI) solutions announced that they planned to support SOAP in their application- to-application integration solutions.

Why? Because SOAP is becoming a standard protocol just as FTP, queuing, SMTP, and many others that we now take for granted.

The reason why we call SOAP a protocol is because it is just a foundation for other solutions. These solutions are called extensions, and are built on top of SOAP. These SOAP extensions will solve communication and integration problems in a standard way. SOAP extensions are being planned to solve problems such as reliable messaging, security, transaction handling, error handling, message routing, and orchestration. These extensions are covered in Chapter 11, Turning a Protocol Into a Framework on page 251.

Natively supporting XML and SOAP standards will enable middleware technologies to take advantage of many new opportunities. An e-commerce infrastructure is being built by consortiums with substantial industry support from not only application providers but also manufacturers, distributors, and many vertical enterprises. SOAP is not only for cross-enterprise application and document integration.

SOAP standards can be used internally as well. An organization will describe an internal services that will be delivered to other internal service partners, such as departments or other application groups. In SOAP in Application-to-Application Integration on page 204, we discuss how you can use SOAP to integrate internal applications.

Business Decision Making in Real Time

Today's early SOAP implementations have been mainly document exchange, such as BizTalk or ebXML. However, SOAP was originally a solution to solve application access problems, which are structured, loosely coupled, synchronous procedure calls. When an organization moves to synchronous, real-time transactions, it has new opportunities in messaging and partner relationships.

As a result, an organization can thoroughly integrate and collaborate with customers, trading partners, and applications that do not necessarily reside in their organization. For example, for credit card processing, an organization needs the data which is stored in a bank's system. When a purchase happens, the organization needs to reach this valuable information and give a decision in real time.

If an organization can reach a bank's credit information, transactions happen easier, faster, and cheaper, and they get their money sooner. Money today is always better than money tomorrow.

Another example is found it search engines. Google now offers a SOAP API for searching the Internet. An organization can use this API and include search capabilities in their applications. An organization that does not have and will never want to build a search engine themselves now have a service that they can use simply using the SOAP API that Google offers.

Reaching the utopia of "getting value from data that we do not own" is a compelling concept. SOAP provides new business opportunities in this respect.

As the software industry adopts a standard protocol and syntax, such as SOAP and XML, it becomes easier to integrate everything. The integration problems that are caused heterogeneous applications are gone.

As SOAP becomes a standard protocol that applications support, integration will not be a problem. As a result, the data will not be held hostage in a proprietary format any more. Data will flow through the applications and services, and applications will have the ability to use any data no matter what format data natively resides in.

Integration problems will be solved not only in internal application integration and behind the firewall but also across the firewalls with partners.

Integration Problem Statement

In Chapter 3, WhereWe've Been, WhereWe're Headed on page 39 we told the story of Jean and David. They are clerks that work in two organizations that are doing business. David is a clerk in the supplier organization. He is the one who carries data from one party to another in the organization, whether that party be a machine or a person. David is the one who is responsible for the business flow and he is the one who integrates computer applications, people, and other resources inside and outside the organization.

Human Integration

In our scenario, David talked to his counterpart, Jean, on the telephone. During this conversation, data flowed through speech. The two people understand each other because they are speaking the same language, and using the same business terminologies and nomenclature.

When David talks to his director, he knows what the director is talking about by understanding her speech. When they are communicating with e-mail, he understands what is written in e-mail. Again, because they speak the same language, use the same terminologies, and have the same human speech and recognition capabilities.

When David enters data into a computer that has a user interface, he understands it because it is designed with humans and their capabilities in mind. David knows nothing about the internal processes of the computer application. However, he faces an interface that he understands and he is trained in using it because it is familiar with what he has seen all of his life.

Until now, we have not had a problem integrating humans with humans or applications with humans. We have a standard in communication, which is language.

We know about humans and we design our systems for them. As a result, data flows around, and humans and applications integrate just fine. Human capabilities are keys to system integration.

However, when organizations deploy different applications in-house and they want fast and automated business processes that do not involve human integration, how will they do it?

Application-to-Application Integration

Organizations have heterogeneous systems. They have applications and data that belong to different computational generation eras, written using different programming languages, which employ different vocabularies and syntax rules, data types, and many other incompatibilities.

Because these systems cannot integrate with each other, they need humans to free the data.Without human interaction the data will be held hostage. Organizations have very important and valuable data. It takes significant resources to maintain that data. How can they leverage their data in an automated manner? How can they get rid of the need for human interactions that carry data from one system to another? We have discussed "JESSIE" and what troubles she causes when she is the central integration solution in Chapter 2, Integration Today on page 29.

We need to integrate our businesses, our applications, and our business processes in order to pass valuable data and as a result do commerce with one another. Only then will we be able to achieve the promise of web services.

The Integration Problem

The problem is to integrate applications or, by extension, trading partners.We define integration as applications or trading partners passing data from one to another and consuming it without human interaction.

The application integration problem has been with us since the day organizations started using two computers. Traditionally, getting systems to talk to each other required building some kind of connector that could translate between two different systems.

An organization would need to build some kind of integration code, and expose a connection point in order to pass and use each other's data. This is called "point-to-point" integration and is illustrated in Figure 101, Point-to-Point Integration.

In a point-to-point architecture, integrating systems requires custom code for each interface. When any changes happen in either application A or B, The interface programs must be updated and changed. Therefore, the integration solution using point-to-point architecture is hard to manage in the long term.

Application integration becomes increasingly difficult as new applications are added to the environment. For every new system added, it is necessary to create integration code exposed by connection point interfaces for each existing system.

For example, in a point-to-point integration architecture, consider five different applications. Each application speaks and understands its proprietary language, has proprietary data types, and uses different protocols to communicate. If we integrate Application A and Application B, we need to add a connection point and codes on both sides so they can communicate. The code on application A will be speaking the language of A and B. When application A wants data from B, this code translates the request to B's format and data type, with B's object or method names, and transfers it to B in a protocol that B supports.

When application B needs data from application A, then the code in B will be converting B's request to A's language, with A's specific implementation details. As a result, for a two-way communication between application A and B, programmers need to write applications that enable integration that sits on both sides.

This integration code is usually called an "adapter". Just like an adapter for an electrical plug, they convert the existing standard into another standard that a specific application can consume.

The number of adapters needed grows exponentially. There is a formula that describes this, it is N2-N. So if there are five applications, and all of them require interactions with each other, the calculation is 52 - 5 = 20.

This is commonly called the "N-squared problem", and is common where there is a requirement for a many-to-many interaction. An example with five applications is shown in Figure 102, The N-squared Problem.

Problems With Point-to-Point Integration

Applications can be integrated now. However, any changes in applications break the integrity of the connections. They need to be changed and updated. Changes in one application interface means changing N adapter applications.

Business Partner Integration

This same scenario is true when we are trying to integrate N external business partners, such as customers or suppliers, in a point-to-point manner. Trading partners all have different frameworks and applications that handle e-commerce in the best way for each one. The problem becomes integrating not just applications but entire organizations that sit outside of your system. It is an even a harder problem, because their business practices require us to consider another set of integration points.

In order to integrate N trading partners, programmers need to write N2 - N different applications that support N frameworks at each interface. These applications will be updated every time some trading partner changes the framework that they use. If we add another trading partner into the picture, we need to integrate all existing trading partners with the new one.

Hub Model With Adapters

Now that we've seen the "N-squared" problem, what is the answer? What kind of architecture can solve this exponential integration problem?

One of the models organizations use in addition to point-to-point integration is called the "hub" model. An organization will use a system that sits in the middle of the systems that need to be integrated to decrease the number of interface points. This central system is called a "hub", because it sits in the center, and looks like a wheel with spokes when the connections are made. See Figure 103, The Hub Model.

When the number of interface points decreases, the integration problem becomes a linear problem. The adapters that organizations need to add to their applications for interfacing drops to N. And when a new application is added to the system, then only one adapter is needed for its integration to all the existing applications. Now, instead of requestor application communicating with the respondent, the requestor communicates with hub application, which then communicates with the respondent application.

When application A wants data from application B, A sends the request to hub application using the hub application's language and protocol. The hub translates the request and sends it to application B, which takes the request and converts to its own format.

The adapter on A understands only its language and the hub application's language. In the same manner, the adapter on B understands only its language and the hub's language.

The Broker

Using a hub is the system equivalent of humans using a broker or some other kind of middleman. Let's say we are trying to buy a house.We could drive around town looking for signs posted on people's front lawns indicating their desire to sell, and then contact each owner and ask about prices and terms. When we find a house we like, we could do negotiating directly with the seller and take care of everything ourselves. What a hassle. And what a danger. We do not know all of the laws having to do with the transfer of real property. We do not know everything about title searches and inspection reports and mortgages and deeds and and and...

Instead, we use a broker to do much of the work for us. Let's call her Meg. Meg first meets with us to understand what we want from the house. She wants to know the price range we are interested in, the locations, and other factors like the size, number of bedrooms and bathrooms, and whether it has room for our Elvis- on-velvet collection.

All of the brokers have gotten together and created a network of information where they share all of the properties that they have. Meg search searches this set of listings, looking for properties that match our specifications.

Then she runs us around in her car so we can pick out the perfect house. But that's just the start. Now, we negotiate with the seller, not directly, but through Meg. Meg does not want the buyer and seller ever talking to each other because problems might arise. Apparently buying a house is quite an emotional time, and sometimes people say the wrong things. But all is well, because Meg is taking care of the interface. She is the hub.

Meg deals with the inspector, the various government agencies, the banks, the insurance people, and everything else. If we have any question whatsoever, we can just call Meg, and she will get the answer.

Adding a middleman into any business process has its advantages and disadvantages. The advantages of middlemen are pretty nice:

She takes care of details in communicating with the house owners.
She is a one-stop information center for us.
She is an expert in house nomenclature and the local market.
She knows details of the transaction, or has experts available that can cover every part of it.

However, Meg has some disadvantages, as well:

She is expensive.
She can lose details when communicating with the seller or the other parties.
She has her own agendas, which might conflict with ours. For example, our agenda is to get the lowest cost, while hers is to make most profit out of the deal.

The hub model of integration also has its advantages and disadvantages, which are much like the ones in the human world:

The main advantage of the hub model is that it decreases the number of interface points. As a result, the adapters we need decreases to a linear proportion.
Like its human counterpart, the hub can be a specialist in all things having to do with a particular type of transaction.
The main disadvantage of using a hub model is the cost. By using an intermediary, the transaction executes slower than in a point-to-point integration model.
Also, by putting a hub application in the middle, all applications in the environment need to learn how to speak the hub application's language and protocols. The integration standard becomes some other application's proprietary language and protocol. Application A, B, C, D and E all need hub-standard adapters.

Using a hub-based integration architecture does not solve all of an organization's problems. An organization will usually deploy more than one hub application because of the limited environments and functions hub applications support. For example, some hub applications are good for integrating applications using message queues as a protocol. Other hub applications offer protocols and specialize in adapters for a certain number of platforms.

By deploying a hub-based integration architecture, organizations accept the need to modify their applications to be understandable to the hub application's proprietary language and protocol. Other applications do not know anything about the hub application except at its interface points. Programmers need to write adapters� or organizations need to buy adapters�that enable existing applications to communicate with the hub application and through the hub application with other applications that require their data to operate.

We will see where the next generation of interoperability is, but first, let's take a look at integrating internal applications with SOAP.

SOAP in Application-to-Application Integration

We have discussed how SOAP works in cross-enterprise integration in Chapter 7, Simple Object Access Protocol (SOAP) on page 139.We used HTTP and port 80 in order to exchange SOAP request and response messages as shown in Figure 104, SOAP Over HTTP.

The SOAP client's job is to form a request document using an XML vocabulary and to send it to SOAP server using HTTP port 80. The SOAP server's job is to read the SOAP message and translate the body of SOAP message into a language that the remote object speaks and pass it that information. Once the remote object gets the document and reads the content, it does what it has been doing all its life and returns an answer to the SOAP server. The SOAP server takes the answer, converts it to a standard response language and sends it back as a SOAP document using HTTP port 80.

Now we would like to change the scenario. Instead of integrating objects across the firewall, let's integrate objects inside the same organization.

SOAP over HTTP is not necessarily scalable for application-to-application communication. HTTP requires a certain amount of overhead that is required for the untrusted Web, but is overkill for internal use, and not fast enough.

The organization probably has an internal network that will connect applications such as SOAP client and SOAP server messaging. It would make sense, then, to use the internal network connection, using a high-performance protocol such as TCP, instead of HTTP.

In this scenario, The SOAP client is sending the SOAP request document over TCP, or any internal network. This is shown in Figure 105, SOAP Over Internal Network.

However, in real life business environments, client applications do not invoke objects by themselves, send requests every once in a while to the local server, get answers and throw them into the trash. Usually a requestor object that is in need of data that the responder object can provide will be sending a message to the responder local object.

Therefore, let's attach an object to the SOAP client. This is shown in Figure 106, Object is The Client. Now we have a local object instantiating the SOAP client. The SOAP client understands the local object's language and translates it into the standard language and packages the request as a SOAP document; and sends it over TCP.

Now, the local object can get information from the other local object using SOAP over an internal network.

Instead of referring to the objects "local objects", we would like to give them names. They can be a Java program and a COBOL program that have been encapsulated as objects. This is shown in Figure 107, Objects Now Have Names.

The Java object sends a SOAP request document to the COBOL object. The COBOL object runs its application and returns an answer. The answer is sent to the Java object as a SOAP response document.

Java does not speak COBOL�s language and COBOL does not speak Java's language. They do not have the same type definitions, the same method call syntaxes, or the same data structures. However, the SOAP server that sits on COBOL program can speak COBOL and SOAP, The SOAP client that sits on the Java object can speak Java and SOAP. Therefore, the integration between two different worlds is enabled by a SOAP server and client that act as translators.

Now, what if the COBOL program needs to get data from the Java program? The scenario we have discussed above can be reversed by reversing SOAP client and server programs, as shown in Figure 108, Client and Server Reversed.

Now, the SOAP client program is on COBOL�s side, and SOAP Server is on Java's side. As a result of this change, COBOL program can access the Java object. The COBOL program is sending a SOAP request document with its SOAP client program. The Java object is getting the request and invoking the methods, and returning an answer back to the COBOL program as a SOAP response document. The SOAP client document gets the SOAP response and feeds it into COBOL program, as a screen input or a COBOL record layout.

What if we needed two-directional information flows? What if some COBOL programs and functions needed data that the Java object has, and some Java functions needed data that COBOL has? Then we need SOAP client and SOAP server on both sides.

As a result, in Figure 109, Everyone is a Server and a Client, SOAP clients and SOAP servers reside on both sides.

This enables two-way messaging. Java objects and COBOL objects can communicate with each other and access each other's data with SOAP clients and SOAP servers that act as translators.

The SOAP client and SOAP server programs form a "SOAP implementation" when they are combined together. From now on, we will refer to the SOAP client and SOAP server programs together as a SOAP implementation. This is shown in Figure 110, SOAP Implementation.

SOAP implementations work as plumbing for the objects. This evolution is the same as the evolution from separate programs to COM objects. Before COM, the objects and applications in the Microsoft world were separate, they could not access each other's information. Data was imprisoned.

However, with COM, objects that are developed on the Microsoft platform have plumbing that helps the objects integrate with each other using that plumbing. This plumbing was meant to act as we have described here. That is, allowing system architects and programmers to integrate different applications without the need for adapters. However, they enable data flowing from one object to another only as long as everyone they need to access is COM-compliant. In contrast to that, SOAP implementations act as standardized plumbing, which enables heterogeneous applications to talk each other. In addition to that, SOAP implementations enable objects of different types to integrate with each other using open standards that are built on top of XML.

These objects can be from completely different eras, architectures, and locations. Their data can be useful to each other's application and process now because SOAP implementations have integrated the applications and data, as shown in Figure 111, Objects Communicating Directly.

In our scenario, we are referring to the internal network as TCP. However it can be any protocol that SOAP supports or will support, such as User Datagram Protocol (UDP), or IBM MQ Series. For example, IBM�s SOAP implementation in theirWeb Services Toolkit has native IBM MQ-Series support. The communication cloud can be named with whatever communication protocol Java and COBOL use in their organization, as shown in Figure 112, Any Network or Transport Technology.

What will happen when a different object that speaks a completely different language is added to the business? in Integration Problem Statement on page 197, we mentioned the "N-squared problem". What kind of changes do we need to make in our web service-enabled integration model in order to embrace COM object and its valuable data?

All we need to do to communicate with each other is to use SOAP implementations as our plumbing. The COM object needs to speak the standard language in the organization with the help of a SOAP implementation around the COM object.

As we see in Figure 113, New Object, when we add a SOAP implementation that can speak COM�s language as well as the standard language, the integration problem is solved. A SOAP implementation that speaks COM and SOAP is Microsoft's SOAP Toolkit. As are most SOAP implementations, Microsoft's SOAP Toolkit is free. As a result, the objects of different types, and programs built during different eras, can communicate with the help of SOAP implementations that act as plumbing.

This is a loosely coupled system. The COM object does not know what kind of object is accessing it. It behaves exactly the same whether it is a Visual Basic client or a CORBA or a COBOL object accessing the COM object. CORBA and COBOL with the SOAP implementation behave exactly the same.

Services Point-of-View

Web services are not just for external communications. By using web services, organizations need to focus on how they can make their organization a service provider. They need to adopt a services point-of-view.

Any application or group of applications, departments, or organizations that deliver a service can be exposed as a web service. Considering this point in the application- to-application integration scenario, the heterogeneous objects or programs or groups of these objects or programs integrates using SOAP implementation. They communicate with each other. For example, in the scenario we just discussed, a COM object can be an ordering service, the Java program can be a shipping service and COBOL object can be an inventory service.

As a matter of fact, SOAP in application-to-application integration is not scalable enough to be deployed for now. Open text messages are bigger than binary files to transfer. SOAP messages have overhead due to envelope definition. We do not have fast enough networks to integrate applications using SOAP.

We do not think SOAP will replace the integration middleware solutions we have today in the near future. However, support for SOAP in the middleware is a fact. Many middleware technologies support SOAP in internal integration.

In addition to that, is be conceivable and possible to integrate web services in an organization using SOAP implementations. You could call this "services-to-services" integration. Integration of services will happen using SOAP implementations whether it is integrating services inside or outside the organization.

Each of these objects defines a service in the organization. Therefore, integration of these services is enabled by SOAP implementation as shown in Figure 114, SOAP Implementations as Web Services.

Beyond the Hub: A Standard Infrastructure

In Hub ModelWith Adapters on page 201, we covered the Hub model for integration. This model has some benefits over point-to-point integration because it reduces the number of adapters required. But what if we were to get rid of adapters altogether?

As we stated in the beginning of this chapter, we have an integration problem because applications do not use the same protocols, speak the same syntax, and adhere to the same frameworks to pass parameters and exchange data.

If applications were all written using a single format, syntax, and protocol, we would not have a problem.

Would not it be nice if there was a standard syntax and protocol that all application vendors support? If this was the case, then all applications can communicate with each other without the need for adapters. Applications will support the one agreed syntax and agreed protocols out of the box, just like they support FTP, HTTP, without adding extra code.

Now let's consider integrating web services.We have already discussed that web services are business processes exposed by an interface that supports XML and standards that are built on top of it, such as SOAP and WSDL. Each web service supports XML syntax and the SOAP envelope protocol. A description of the package is expressed in a standard language called WSDL.

Due to standards, web services will not have integration problems. Every web service can communicate because its syntax, definitions, and protocols are all standardized by definition.

Standardization in interfacing with the outside world eliminates the integration problem. Integrating N different web services requires no adapters at all. This is because they already speak using the standard alphabet (XML), using a standard protocol (SOAP), and a standard description language (WSDL).

Web services integration is shown in Figure 115, Integrating Systems with SOAP. Notice that there is no hub and no adapters. When web service A needs data that web service B has:

Web service A reads the contract of web service B, which is B's WSDL file.
Web service A composes a request in XML syntax, using B's language that it learned from the WSDL file.
Web service A sends the request to B using SOAP as a protocol.
Web service B gets the request from A, processes it according to its business process definition, that is described by a workflow, and then sends an answer back to web service A.
The language of the answer is also described in B's WSDL file.
Response from web service B arrives to A using SOAP protocol. Web service A reads the response and uses the data to service its business processes.

Web services can reside inside or outside of an organization's firewalls. Whether they are inside or outside the organization, they are defined and expressed using the same standards. They all support the standardized alphabet, vocabulary definition, protocols, and frameworks by definition.

In conclusion, a web service supports loosely coupled, message-based integration using XML, SOAP and WSDL. And because of this fact, there are no integration problems for integrating web services.

We will discuss the integration of web services inside the organization in the following section.

Web Services Inside of an Organization

We discuss the use of web services outside of company's firewalls in Chapter 3, Where We've Been, WhereWe're Headed on page 39. Now, let's take another scenario. In that scenario, the company that provided spare parts, ToyCarParts, has a web service for ordering. The ordering process has a business flow that is made up of many applications. The business process was initiated by a purchase order that arrives at the order processing department.

We discuss the importance of integration, workflow and automation in web services among the entities that make up a business process in Table 2, I.T. Architecture Grid.

Inside the organization there are business processes that can be described as services. These services are not necessarily directly consumed by outsiders, but they are consumed by applications, or business processes inside the organization.

As we see the workflow for ordering in Figure 116, Process Workflow, checking the customer account is a web service exposed for internal use.

When a business process is analyzed and formed as a web service, it can be modularized and reused in the organization. Once the process of checking a customer account is a service, then the details of this business process do not need to be defined every time it is needed. It will be defined in one place and reused by many.

After a production decision is given in the workflow by a human decision- maker, the production system needs to be notified. For an ordering service workflow, the details of the production business processes are out of its scope. However, if the production system is not a self-contained service, the ordering process needs to define and handle errors, problems, or other exceptions coming from the production business process.

In the same manner, if we define our production system, integration, workflow, and automation in a production web service definition, then we can reuse it for many other services. Production will be a web service that is for internal use only because the data needs to be private. The outcomes of the production web service can be a purchase order acknowledgment or other notifications if things go wrong or the delivery date changes.

Integration

The ordering service is made up of many applications that integrate with each other following a certain workflow. The production and credit approval web services work the same way.

The ordering web service calls the credit approval and production web services. As a result, the integration problem within the organization between the ordering web service, the production web service, and the credit approval web service becomes a problem of simply integrating several web services.We can also add shipment and accounting web services, which will be needed for the goods to be delivered and invoice to be sent. This is illustrated in Figure 117, Object as Web Service.

As a result, once organizations define web services to be delivered inside and outside of their organizations, they integrate these services with the same standard implementations, XML and SOAP. The integration is not a problem as we have discussed in integration problem Integration Problem Statement on page 197.

Therefore, vendor organizations leverage already existing integration architecture, while taking this further and integrating business processes within the organization by standards, such as XML and SOAP.

In conclusion, data is no longer imprisoned in applications or objects, but flowing through services inside and outside of organizations. The architecture and technologies used for internal or external web services are the same. This is another advantage to leverage companies' know-how. Once organizations build the web services integration architecture, the same technologies are used inside and outside the organization.

So, What's It All Mean?

The integration architectures we have today are very limited.

Integration can be synchronous. If it is synchronous, then probably we are using a proprietary direct object call. This option is tightly coupled and scalable. However, there are management issues as we have discussed in Integration Problem Statement on page 197.

If we want to deploy an asynchronous integration option, then we can do it with a message-based loosely coupled system such as a queuing application. The integration model using SOAP can be synchronous or asynchronous, but because its message-based architecture, it enables loosely coupled integration in all cases. Therefore, SOAP gives us a solution to integrate heterogeneous objects, and objects in different physical boxes in a loosely coupled manner both in synchronous or asynchronous models.

SOAP is an accepted standard by the leaders of application and solution developers industry such as Microsoft, IBM, Sun Microsystems, BEA and many other organizations including open source initiatives.

Organizations have been successfully using methods to send messages to trading partners without knowing anything about their business implementations for years using paper documents. By deploying SOAP over HTTP, an organization can make DCOM or CORBA calls over the Internet in the same way, without knowing about the implementation details of the target organization. We can also use these technologies to do business document exchange.

SOAP is another alternative to existing protocols for sending business documents in a loosely coupled manner. SOAP also allows organizations to treat trading partners' services and offerings as a part of their in house applications by its message- based method invocation.

Even if web services become common in communicating with trading partners and leveraging their information, it is important to understand that a web service is not about only trading partner relationships. It can be an internal department that is delivering a service, or an application or even an object.

The new technologies built on top of XML enable services, whether in-house or external, to communicate easily. These technologies ensure that the building blocks for application communication are easy to implement and extensible.

The new technologies give opportunities for integration in document and method-based models. As a result, the web and any integration protocols will change how data lives its life. Data can flow among heterogeneous systems no matter where the application sits, behind or across the firewall, or on what platform the application lives, and what language the application speaks. This is enabled by high adoption of XML technologies and the existence of implementations of these technologies in every platform, for every application, from Microsoft to Java, from COBOL to enterprise integration applications.