Top 10 pitfalls of P2P integration to avoid in the cloud

This guest post comes courtesy of Ross Mason, CTO and founder of MuleSoft. Disclosure: MuleSoft is a sponsor of BriefingDirect podcasts.

By Ross Mason

While integration isn’t necessarily a new problem, the unique challenges of integrating in the cloud require a new approach. Many enterprises, however, are still using point-to-point (P2P) solutions to address their cloud integration needs.

In order to tackle cloud integration successfully, we need to move beyond P2P integration and avoid repeating the same mistakes. To aid in that effort, here is a list (in no particular order) of the top 10 pitfalls of P2P integration to avoid repeating in the cloud:

1. Building vs. buying: If you have developers with integration experience in your IT department, you can have them build a custom P2P integration in house, rather than buy a packaged solution. Building your own integration, however, typically means that you will also have to manage and maintain a codebase that isn’t central to your business and is difficult to change.

2. Quickfire integrations: Let’s say you need to integrate two systems quickly and hire a developer to work on the project over a couple of days. You notice an improvement in business efficiency and see an opportunity to integrate additional systems. You hire the same developer and expect the same quickfire integrations, but the complexity of the project has increased exponentially. The takeaway? It’s always a good idea to approach integration systematically and establish a plan up front, rather than integrate your systems in an ad hoc P2P fashion.

3. Embedding integrations in your application: Although it might be tempting to embed P2P integrations in your web application, you should be cautious about this approach. It may be fine for really simple integrations, but over time, your integration logic becomes scattered in different web apps. Instead, you should think of integration as a separate tier of your application architecture and centralize this logic.

One of the consistently recurring mistakes of doing quick P2P integrations is assuming that things will not break.

4. Creating dependencies between applications: When you integrate applications in a P2P fashion, you create a dependency between them. For example, let’s say you’ve integrated App A and App B. When App A is modified or updated, you will need to change the integration that connects it to App B. You also need to re-test the integration to make sure it works properly. If you add App C to the mix, your workload can increase exponentially.

5. Assuming everything always works: One of the consistently recurring mistakes of doing quick P2P integrations is assuming that things will not break. The reality is that integrations don’t always work as planned. As you integrate systems, you need to design for errors and establish a course of action for troubleshooting different kinds of errors. Error handling is particularly troublesome when integrating software-as-a-service (SaaS) applications, because you have limited visibility and control over the changes that SaaS vendors make to them.

Test each integration

6. It worked yesterday: Just because P2P integration worked for one project does not mean it will work for another. The key is to test each integration you build. Unfortunately, P2P integrations are often built and deployed quickly without sufficient planning or proper testing, increasing the chances for errors. Although it can be difficult and does require a decent amount of effort, testing integrations is absolutely critical.

7. Using independent consultants: Many companies are not staffed with developers who have enough integration expertise and hire consultants to resolve their integration issues. The problem with this approach is that you often have limited visibility into whatever the consultant delivers. If you need to make changes, you typically need to work with the same consultant, which is not always possible.

8. Creating single points of failure: As your P2P integration architecture grows in size and complexity, its chances of becoming a single point of failure in your entire network increase as well. Minimizing the potential for single points of failure should be a priority when it comes to integration, but the lack of decoupling in a P2P approach makes it hard to eliminate bottlenecks in your system.

Quick P2P integrations are relatively manageable when you have 2 or 3 systems to connect, but when you start adding other systems, your architecture quickly becomes a complicated mess.

9. Black-box solutions: Custom-built P2P solutions are usually black box in nature. In other words, they lack reporting capabilities that tell you what is happening between systems. This makes it very hard to debug problems, measure performance, or find out if things are working properly.

10. Creating a monster: Quick P2P integrations are relatively manageable when you have 2 or 3 systems to connect, but when you start adding other systems, your architecture quickly becomes a complicated mess. And because no two P2P integrations are exactly the same, managing your integrations becomes a major pain. If you invest in doing some design work up front, however, this will save you from having to throw away a tangled P2P architecture and starting from scratch to find a new solution under pressure. If you have a well thought out design and a simple architecture, you can reduce the management burdens and costs associated with integration.

Ross Masson is the CTO and Founder of MuleSoft. He founded the open source Mule project in 2003. Frustrated by integration "donkey work," he started the Mule project to bring a modern approach, one of assembly, rather than repetitive coding, to developers worldwide. Now, with the MuleSoft team, Ross is taking these founding principles of dead-simple integration to the cloud with Mule iON, an integration platform as a service (iPaaS).

You may also be interested in:

• MuleSoft takes full-service integration to the cloud with ION iPaaS ESB platform
• Don't use an ESB unless you absolutely, positively needs one, Mule CTO warns
• MuleSoft announces availability of enterprise-class ESB management console
• Cloud and SaaS force a rethinking of integration and middleware as services for services
  

MuleSoft takes full-service integration to the cloud with iON iPaaS ESB platform

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Read a full transcript or download a copy. Join the iON beta program. Sponsor: MuleSoft.

Enterprise application integration (EAI) as a function is moving out of the enterprise and into the cloud. So called integration platform as a service (iPaaS) has popped up on the edge of the enterprise. But true cloud integration as a neutral, full service, and entirely cloud-based offering has been mostly only a vision.

Yet, if businesses need to change rapidly as the cloud era unfolds, to gain and use new partners and new services, then new and flexible integration capabilities across and between extended applications and services are essential.

The older point-to-point methods of IT integration, even for internal business processes, are slow, brittle, costly, complex and hard to manage. Into this opportunity for a new breed of cloud integration services steps MuleSoft, a market leading, open-source enterprise service bus (ESB) provider, which aims to create a true cloud integration platform called Mule iON. [Disclosure: MuleSoft is a sponsor of BriefingsDirect podcasts.]

MuleSoft proposes nothing short of an iPaaS service that spans software as a service (SaaS) to legacy, SaaS to SaaS, and cloud to cloud integration. In other words, all of the above, when it comes to integrations outside of the enterprise.

BriefingsDirect recently learned more about MuleSoft iON, how it works and its pending general availability in the summer of 2011. There's also the potential for an expanding iON marketplace that will provide integration patterns as shared cloud applications, with the likelihood of spawning constellations of associated business to business ecosystems.

Explaining the reality for a full-service cloud-based integration platform solution are two executives from MuleSoft, Ross Mason, Chief Technology Officer and Founder, and Ali Sadat, the Vice President of Mule iON at MuleSoft. They are interviewed by Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: It strikes me that the number of integrations that need to be supported are further and further toward the edge -- and then ultimately outside the organization.

Like it or not, any company of any size has some if not most of its data now outside of the firewall.

Mason: We describe it internally as the center of the enterprise gravity is shifting. The web is the most powerful computing resource we’ve had in the information age, and it’s starting to drag the data away from the enterprise outside into the platform itself. What this means for enterprises is, like it or not, any company of any size, has some if not most of its data now outside of the firewall.

I'm not talking about the Fortune 2000. They still have 95 percent of their data behind the firewall, but they’re also changing. But, for all of the enterprises and for forward-thinking CIOs, this is a very big and important difference in the way that you run your IT infrastructure and data management and security and everything else.

It turns a lot of things on its head. The firewall is constructed to keep everything within. What’s happening is the rest of the world is innovating at a faster speed and we need to take advantage of that inside enterprises in order to compete and win in our respective businesses.

There are a number of drivers in the marketplace pushing us toward integration as a service and particularly iPaaS. First of all, if we look back 15 years, integration became a focal point for enterprises, because applications were siloing their data in their own databases and for business to be more effective, they have to get that data out of those silos and into a more operational context, where they could do extended business processes, etc.

What we're seeing with cloud, and in particular the new wave of SaaS applications, is that we're doing a lot of the same mistakes for the same behaviors that we did 10 years ago in the enterprise. Every new SaaS application becomes a new data silo in the cloud and it’s creating an integration challenge for anyone that has the data across multiple SaaS providers.

New computing models

And it's not just SaaS. The adoption of SaaS is one key thing, but also the adoption of cloud and hybrid computing models means that our data also no longer lives behind the firewall. Couple that with the drivers around mobile computing that are enabling our workforce and consumers, when they are on the go, again, outside of the firewall.

Add the next social media networks and you have a wealth of new information about your employees, customers, and consumers, available through things like LinkedIn and Facebook. You've also got the big data explosion. The rise of things like Hadoop for managing unstructured data has meant that we end up pushing more data outside of our firewalls to third party services that help us understand our data better.

There are four key drivers: the adoption of SaaS applications; the movements by using more cloud and hybrid models; mobile is driving a need to have data outside of the enterprise; and then social media and also big data together are redefining where we find and how we read our information.

Gardner: It also appears that there will be a reinforcing effect here. The more that enterprises use cloud services, the more they’ll need to integrate. The more they integrate, the more capable and powerful the cloud services, and so on and so on. I guess we could anticipate a fairly rapid uptake in the need for these external integrations.

Mason: We think we might be a bit early in carving out the iPaaS market, but the response we're hearing, even from our largest organizers, is that most have lots of needs around cloud integration, even if it's just to help homogenize departmental applications. We’ve been blown away at MuleSoft at the demand for iON already. [Join the iON beta program.]

New open enterprises

The open-source model is absolutely critical, and the reason is that one of the biggest concerns for anyone adopting technology is who am I getting into bed with? If I buy from Amazon, ultimately, I'm getting into there with Amazon and their whole computing model, and it’s not an easy thing to get out.

With integration, it’s even more of a concern for people. We’ve looked through the vendor lock-in of the 1990s and 2000s, and people are a little bit gun-shy from the experiences they had with the product vendors like Atria and IBM and Oracle.

When they start thinking about IaaS or the cloud, then having a platform that’s open and freely available and that they can migrate off or on to and manage themselves is extremely important. Open source, and particularly MuleSoft and the Mule ESB, provides that platform.

Gardner: Ali, how do you see iPaaS process enablement happening?

Sadat: It’s a pretty interesting problem that comes up. The patterns and the integrations that you need to do now are getting, in a sense, much more complex, and it’s definitely a challenge for a lot of folks to deal with it.

We’re talking not only to cloud-to-cloud or enterprise-to-enterprise, but now extending it beyond the enterprise to the various cloud and the direction of data can flow either from the enterprise to the cloud or from the cloud to the enterprise. The problems are getting a little more challenging to solve.

The other thing that we’re seeing out there is that a lot of different application programming interfaces (APIs) are popping up -- more and more every day. There are all kinds of different technologies either being exposed to traditional web services or REST-based web services.

We’re seeing quite a few APIs. By some accounts, we're in the thousands or tens of thousands right now. In terms of APIs, they're going to be exposed out there for folks who are trying to figure out and how to integrate. [Join the iON beta program.]

Gardner: What do you propose for that?

Hybrid world

Sadat: It’s something a hybrid world, and I think the answer to that is a hybrid model, but it needs to be very seamless from the IT perspective.

If I want to do a real-time integration between Salesforce and an SAP, how do I enable that? If I poke holes from my firewall that’s going to definitely expose all kinds of security breaches that my network security folks are not going to like that. So how do I enable that? This is where iON comes into play.

We’re going to sit there in a cloud, open up a secure public channel where Salesforce can post events to iON, and then via a secured connection back to the enterprise, we can deliver that directly to SAP. We can do on the reverse side too. This is something that the traditional TIBCOs and WedMethods of of the world weren’t designed to solve and they weren’t even thinking about this problem when they designed and developed that application. [Join the iON beta program.]

The difference between integration running on-premise or in the cloud shouldn't matter as much, and the tooling should be the same. So, it should be able to support both a cloud-based management, and also be able to manage and drive us in the enterprise, and set up on-premise tools.

One of the things you’ll see about iON is a lot of familiar components. If you have been a Mule user or Mule ESB user, you will see that at the heart of iON itself. What we're providing now is the ability to be able to deploy your solutions, your integration applications to a cloud and be able to manage it there, but we also are going to give you the capability to be able to integrate back into the enterprise.

Gardner: Why not just use what Salesforce provides you and let that be the integration point? Why would you separate the integration cloud capability?

Sadat: Integration, as a whole, is much better served by neutral party than just going by any one of the application vendors. You can certainly write custom code to do it, and then people have been doing it, but they've seen over and over that that doesn’t necessarily work.

Having a neutral platform that speaks to APIs on both sides is very important. You’re not going to find Salesforce, for example, adopting SAP APIs, and vice versa. So, having that neutral platform is very important. Then, having that platform and being able to carry out all the kinds of different integration patterns that you need is also important.

We do understand the on-premise side of it. We understand the cloud side of the problem. We're in a unique position to bring those two together.

Having that platform and being able to carry out all the kinds of different integration patterns that you need is also important.

Gardner: Ross, please define for me what you consider the top requirements for this unique new neutral standalone integration cloud?

Mason: I'll start with the must-haves on the PaaS itself. In my mind the whole point of working with a PaaS is not just to do integration, but it’s for a provider, such as MuleSoft, to take all the headache and hard work out of the architecture as well.

For me, a true PaaS would allow a customer to buy a service level agreement (SLA) for the integration applications. That means we are not thinking about CPUs, about architecture, or I/O or memory usage, and just defining the kind of characteristics they want from their application. That would be my Holy Grail of why a PaaS is so much better?

For integration, you need that, plus you need deep expertise in that integration itself. Ali just mentioned that people do a lot of their own point to points and SaaS providers do their own point integrations as well.

We spend a lot of money in the enterprise to integrate applications. You do want a specialist there, and you want someone who is independent and will adopt any API that makes sense for the enterprise in a neutral way.

We’re never going to be pushing our own customer relationship management (CRM) application. We're not going to be pushing our own enterprise resource planning (ERP). So, we’re a very good choice for being able to pull data from whichever application you're using. Neutrality is very important.

Hugely important

Finally, going back to the open-source thing again, open source is hugely important, because I want to know that if I build an integration on a Switzerland platform, I can still take that away and run it behind my firewall and still get support. Or, I just want to take it away and run it and manage it myself.

With iON, that’s the promise. You’ll be able to take these integration apps and the integration flows that you build, and run them anywhere. We're trying to be very transparent on how you can use the platform and how you can migrate on as well as off. That’s very important. [Join the iON beta program.]

Gardner: You’ve come out on May 23 with the announcement about iON and describing what you intend.

Sadat: That’s correct. We started with our private beta, which is coming to an end. As you mentioned, we’re now releasing a public beta. Pretty much anybody can come in, sign up, and get going in a true cloud fashion. [Join the iON beta program.]

We're allowing ourselves a couple months before the general availability to take in feedback during the beta release. We’re going to be actively working with the beta community members to use the product and tell us what they think and what they'd like changed.

One of the other things we’re doing soon after the general availability is releasing a series of iON applications that we'll be building and releasing. These will be both things that we’ll offer as ways to monetize certain integrations, but also as reference applications for partners and developers to look at, be able to mimic, and then be able to build their own applications on top of it.

Gardner: What is it they are going to get?

Sadat: At the core of it, they get Mule. That’s pretty essential, and there’s a whole lot of reasons why they do that. They get a whole series of connectors and various transports they can use. One of the things that they do get with iON is the whole concept of this virtual execution environment sitting in the cloud. They don’t have to worry about downloading and installing Mule ESB. It’s automatically provided. We'll scale it out, monitor it, and provide all that capability in the cloud for them.

Once they’re ready, they push it out to iON, and they execute it. They can then manage and monitor all the various flows that are going through the process.

They just need to focus on their application, the integration problems that they want to solve, and use our newly released Mule Studio to orchestrate these integrations in a graphical environment. Once they’re ready, they push it out to iON, and they execute it. They can then manage and monitor all the various flows that are going through the process.

The platform itself will have a pretty simple pricing model. It’s going to be composed of couple of different dimensions. As you need to scale out your application, you can run more of these units of work. You'll be able to handle the volume and throughout that you need, but we are also going to be tracking events. So this is, in Mule terminology, equivalent to a transaction. Platform users will be able to buy those in select quantities and then be able to get charged for any overage that they have.

Also, partners and ISVs today don’t have a whole lot of choices in terms of being able to build and embed OEM services in a cloud fashion into various applications or technologies that they are building. So, iON is going to provide that platform for them.

Embeddable platform

One of the key things of the platform itself is that it is very embeddable. Everything is going to be exposed as a series of APIs. SIs and SaaS providers can easily embed that in their own application and even put their own UI on top of it, so that underneath it it says iON, but on top, it’s their own look and feel, seamlessly integrated into their own applications and solutions. This is going to be a huge part of iON.

Gardner: Looking at the future how does the mobile trend in particular affect the need for a neutral third-party integration capability?

Mason: Mobile consumers are consuming data, basically. The mobile application model has changed, because now you get data from the server and you render on the device itself. That’s pretty different from the way we’ve been building applications up till fairly recently.

What that means is that you need to have that data available as a service somewhere for those applications to pick it up. An iPaaS is a perfect way of doing that, because it becomes the focal point where it can bring data in, combine it in different ways, publish it, scrub it, and push it out to any type of consumer. It's not just mobile, but it’s also point-of-sale devices, the browser, and other applications consuming that data.

Mobile is one piece, because it must have an API to grab the data from, but it’s not the only piece. There are lots of other embedded devices in cars, medical equipment, and everything else.

If you think about that web, it needs to talk to a centralized location, which is no one enterprise. The enterprise needs to be able to share its data with integration outside of its own firewall in order to create these applications.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Read a full transcript or download a copy. Join the iON beta program. Sponsor: MuleSoft.

You may also be interested in:

• MuleSource Takes Aim at SOA Governance, Launches Subscription-Based ESB
• MuleSource Fires up MuleForge Collaboration Site: Apache Tuscany Team Announces 0.99 SCA Release
• Don't Use an ESB Unless You Absolutely, Positively Need One

Don’t use an ESB unless you absolutely, positively need one, Mule CTO warns

“To ESB or not to ESB,” that is the question Ross Mason, MuleSource CTO, raises in a his blog this week.

It would be heresy among marketers at many vendors, but the MuleSource CTO is actively discouraging architects and developers from using an enterprise service bus (ESB), including his company’s open-source version, unless they are sure they really need one.

Misuse of ESBs leads to overly complex architectures that can be more difficult to remedy than a straightforward Web services-based architecture that omits the ESB in early versions of an enterprise application, Mason argued in a phone conversation about his blog.

“There are two main mistakes I see most of the time,” he told BriefingsDirect. “There’s not enough of an integration requirement or there’s not enough use of the ESB features to warrant it.”

You don’t need an ESB if your project involves two applications, or if you are only using one type of protocol, he explains.

“If I’m only using HTTP or Web services, I’m not going to get a lot of value from an ESB as opposed to using a simpler Web services framework,” Mason said. “Web services frameworks are very good at handling HTTP and SOAP. By putting in an ESB, you’re adding an extra layer of complexity that’s not required for that job.”

Architects and developers using an ESB in these cases are probably engaging in "resume-driven development (RDD)." If anybody asks you if you’ve deployed an ESB in an application you’ve worked on you can say, yes. And then you can hope the hiring manager doesn’t ask if the application really required the technology.

Another mistake, Mason cites, is using an ESB and thinking that you are future-proofing an application that doesn’t need it now, but might someday.

“You’ll Never Need It (YNNI), that acronym has been around awhile for a reason,” Mason says. “That’s another killer problem. If you select an ESB because you think you might need it, you definitely don’t have an architecture that lays out how you’re going to use an ESB because you haven’t given it that much thought. That’s a red flag. You could be bringing in technology just for the sake of it.”

Adding his two-cents to the “Is service-oriented architecture (SOA) dead” debate, the MuleSource CTO says such over-architecting is one of the things that contributes to the problems being encountered by IT in SOA that has given the acronym a bad name. “Architecture is hard enough without adding unnecessary complexity,” he said. “You need to keep it as simple as possible.”

Ironically, adding an ESB because you might need it someday can lead to future problems that might be avoided if you left it out to begin with and then added it in later, Mason said.

“The price of architecting today and re-architecting later is going to be a lot less than architecting badly the first time,” he explained. “If you have a stable architecture, you can augment it later with an ESB, which is going to be easier than trying to plug in an ESB where it’s not going to be needed at that time.”

While the conversation focused on the pitfalls of using an ESB where you don’t need one, the MuleSource CTO naturally believes there are architectures where the ESB makes sense. To begin with, you need to be working on a project where you have three or more applications that need to talk to each other, he explained.

“If you’ve got three applications that have to talk to each other, you’ve actually got six integration points, one for each service, and then it goes up exponentially,” Mason said.

The ESB technology is also needed where the protocols go beyond HTTP. “You should consider an ESB when you start using Java Message Service (JMS), representational state transfer (REST), or any of the other protocols out there,” Mason said. “When communications start getting more complicated is when an ESB shows its true value.”

BriefingsDirect contributor Rich Seeley provided research and editorial assistance on this post. He can be reached at RichSeeley@aol.com.