SOA and compute clouds point to rethinking data entirely: roles and permissions, not rows and tables

There's nothing like a conversation to bring out insights and new ideas. Tony Baer and I were chatting on this very pearl of productivity last week, that an open roundtable analyst call always allowed us to move the needle forward in terms of thought leadership in ways that solo writing and one-think cannot.

And then there's Twitter, which is sort of a blend of a roundtable chat, solo writing, one-on-one and blogging -- all to and with a refined, social graph-defined audience. And it was in this act of Twitter-thinking this morning that I slid into a new realization, new for me anyway. It has to do with data, and the need for multi-permeable access to data, across organizational boundaries, stored in many places, that both protects valued and proprietary data while breaking processes out of the corporate IT straight jacket.

I'm seeking better understanding of how cloud (public and private)-base webby apps can and should be a big part of SOA (especially greenfield services), and may even become the driver for rapid use and adoption of SOA. I'm also fascinated by PaaS, SaaS, IaaS, AWS, GAE, and the services ecology development from both small and large providers. And I know that social networking and new media will play into big business in a big way.

These issues have huge impact on many vendors, from Salesforce.com, Amazon, Microsoft, IBM, Google, Yahoo!, Oracle, SAP, and HP, on down to a burgeoning class of startup-class of cloud capabilities supporters. Notice how I lumped together older-style enterprise IT vendors and "consumery" services providers?

The hang-up on understanding how these two worlds -- public clouds and enterprise IT (soon private clouds) -- come together hinges back to who/what handles, owns, manages and offers secure access to (or not) the data.

It's clear that applications and logic can become services, atomized or aggregated. User interfaces can be delivered as a service, as parts of apps or as entire desktops. Same for mobile access.

Of the various levels of abstraction of what goes into IT-based activities, almost all can be deconstructed and more productively delivered as thick, thin, or mixed (software plus services) webby apps. Agile business processes are the new coin of the realm. The days of standalone, PC-based apps are over. Off the wire services will make up more and more of what we will soon colloquially refer to as productivity applications.

But then there's the data, still strapped into a definition that associates it to applications, even as applications as we know them are evolving dramatically. After talking with Dave Linthicum, now CEO of StikeIron, on his vision ... and Kirill Sheynkman, president and CEO of Elastra (as well as many others in the Enterprise 2.0 space) ... I'm now convinced it's time for some radical re-thinking on data, and what it is that it should actually be related to and associated with.

Perhaps it's time to fully divorce data from applications, and wed it all instead to people and groups, guided by roles and permissions, and therefore no longer co-located with applications or even enterprises. House it where it can be used easiest, and stored and protected cheapest. It's heresy today perhaps, but more of the data that matters will be in the cloud anyway.

We always think of data as tied to an application, or in a managed store (often times distributed) that applications read-write to. The store can be analyzed, protected, backed-up, consolidated and cleaned up, messed up again, poked and prodded. We still think of data belonging to some store, and by association, to a department or corporation or IT sys admin.

Data is controlled and managed as centrally as possible, except that it's always scattered and inconsistent. A battle rages in every enterprise as it tries to manage and control its data. It's a losing battle that is costing more and more of the IT spend pie each quarter. And there are many good reasons for this battle -- except that it's holding us all back.

It's now clear that the current mentality of data and its place is holding us back in unacceptably unproductive ways. In this day and age, you will never control your data at the margins. Those margins used to be PCs and departmental servers. Now they are becoming clouds, social networks, free web email apps, Twitter. The data that impacts and drives your company is a complex system not unlike the weather, or quantum-level particles. Try and grab and control that rainbow, if you will. Probability rules, not exactitude.

CXOs can define the happenings of their enterprise by the audits, create the ledgers, and fill up the financials data warehouses. But so much more is going on beyond the glass rooms. The best that CXOs can hope for is to approximate the state of most actual enterprise data, and even that leaves out what is happening in the social media domains, where the innovation, customer feedback and process insights are often occurring.

All those hard drives, all those iPods, all those memory sticks. All the metadata of what your workers do via the web -- out in the murky clouds -- it is all out of your reach. This is already the case, and it will not be a genie you can back in the bottle.

And the choices for retrenchment? Close off your workers from access to web search? Deny them access to your suppliers' portals? Erect firewalls that separate your customers, clients, and prospects from your own sales, marketing, and fulfillment providers? YOU CAN'T MOVE AT INTERNET TIME WITHOUT BEING ON THE INTERNET.

In other words, the ways we treat data today is unnatural. And pretending otherwise is unsustainable. There is a huge productivity opportunity for those that can re-think data, from concept to execution, that can exploit the gathering clouds. This requires radical rethinking, I'm pretty sure, though I can't say I know what the new data landscape will look like.

It will take a social capital-level ecology of complex systems in a pattern of ongoing churn for the answers to arise and quickly become outdated. There are deep and fundamental disruptions under way in media and software now that will force the hand on data. If you don't use what's free from the web and clouds, and your competition does, then what?

So what comes next? There's this murky middle muddle now between public and private clouds, SOA, independently located data, master data, and tools and development. If Google App Engine, Amazon Web Services and the near-certain follow-ons from Microsoft/Yahoo, HP, Oracle, IBM, and perhaps a handful of other major players have even a minor impact -- data conceptually is deeply changed. The cloud providers will give way the tools, single sign-on ID management, runtime, storage, and they will probably let you keep the data that you think is your data.

What enterprises will no longer have is the control of the market data from the public clouds about the users, the groups, their behaviors, attention, their demographics, buying patterns -- all the service on-ramps and off-ramps to your enterprise's revenue growth. Private clouds may not alone be enough to reach mass or long-tail audiences. Will your database of users, your email blast list, your CRM data be as good as Google's, or Microsoft's? Can you sell and market your goods and services without using online ads? Who will sell them to you?

If you're thinking of data as associated to internal applications, as the read-write store for ERP activities, or as the list of your sales leads and prospects in CRM -- you may want to think again. Your data and the clouds' data will need to work together, and perhaps those that bite the bullet and leverage the public cloud's to the hilt will have a huge advantage over those that do not.

Take a hard look at the diagram on this blog by Dan Farber, called "Google's Vision." Salesforce.com and Google think of data quite a bit differently than you do.

And just as you are in cozy partnership (like you have a choice) with your ERP or SOA vendors, enterprises and businesses of almost all sizes will be in partnership soon with the clouds. The shared and protected data alike will be scattered about, too complex for closed marts and masters. The cloud that can manage data in way that allows both user-level and process-level access, with granular permissioning -- and allows CXOs to feel good about it all -- gets the gold ring. The cloud business is a 50-year business.

How should your data be treated in this new cloud era?

Google App Engine creation process live on Twitter

I'm watching Dion Hinchcliffe and a small group of other observers and developers create a Google App Engine (GAE) application live via Twitter.

We're on the cutting edge of using social media and near-real time collaboration tools (free) to learn and use (GAE) for free, and then blog on the process (also free). The price is obviously right, and the ease and transparency of sharing and witnessing are just about friction-free.

As Dion points out (and Dan Farber makes note), there are trade-offs between GAE and Amazon Web Services. And there are concerns to be evaluated and vetted over the application lifecycle remaining in the Google cloud, as Garett Rogers makes note.

But the process I'm witnessing here on Twitter is nothing short of breathtaking for its rapid, agile and productive online team approach (we are located all over) to web app development. Other Google services could be used, too, like Groups. And, of course, developers are well acquainted with other forms of collaboration such as CollabNet.

If even for minor apps, services, or for prototyping development of subsets of large projects, this is all very compelling. I'm fascinated by how developers will use GAE within existing projects and processes. GAE will not be used in isolation, I suspect, but will be a powerful tool in the WOA quiver. And that may also prompt more use of GAE as the end-all, be-all for more an more apps.

I know a lot of people use Amazon as a test bed for their apps. Google App Engine will be very attractive for that too.

But what Google can soon bring to the table is an ability to put these apps and services in front of a ton of other developers and huge potential audiences of end users and consumers. Google has clout of scale, metadata and reach that Amazon does not.

Like others, I hope that Google adds more tools to Python on GAE, like Ruby. I also hope they find a way to port parts or all of the apps off of GAE. Perhaps for a cost, you could choose to not only deploy via the Google cloud, but perhaps get the basic script and code for extraction and use elsewhere, or for mixed-purpose development.

Bungee Labs has that option, that the developers' own the code IP and can take the apps elsewhere. [Disclosure: Bungee Labs is a sponsor of BriefingsDirect podcasts.]

SearchSOA.com names Nexaweb's Enterprise Web Suite as 'Product of the Year'

Nexaweb Technologies, the application platform provider, just keeps piling on the accolades. Its latest achievement is the Product of the Year award for its Enterprise Web Suite from SearchSOA.com.

The gold-status award for the Burlington, Mass. Company, came in the RIA /Composite Application Assembly category, and was based on innovation, performance, ease of integration into environment, ease of use and manageability, functionality, and value. [Disclosure: Nexaweb is a sponsor of BriefingsDirect podcasts. I have also been a reviewing judge for SearchSOA.com product rankings.]

In January, Nexaweb received the Editors' Choice Award from CMP's Intelligent Enterprise.

Just over a month ago, I blogged on Nexaweb, and its role in helping enterprises modernize without pain, confusion or excessive cost.

For Nexaweb, the end game for enterprises is flexible composite workflows, and so the newest offerings are more than tools and platform, there’s a professional services component, to take the best practices and solutions knowledge to market as well. The process includes applications assets capture and re-factoring (sort of like IT resources forensics), re-composition, deployment and then proper maintenance. In the bargain, you can gain an enhanced platform, increased automation, and services orientation.

The goal is to harvest all those stored procedures, but target them to newer architectures — from Struts to Spring — and move from client/server to Enterprise 2.0, is a leap-frog of sorts. The re-use of logic then allows those assets to be applied to model-driven architectures and the larger datacenter transformation values.

Nexaweb Advance pairs Nexaweb’s Enterprise Web Suite with automated code generation tools and professional services to deliver a model-driven architecture approach to the transformation of legacy PowerBuilder, ColdFusion, C++, VisualBasic, and Oracle Forms applications . . .

Nexaweb says that their Enterprise Web Suite allows companies to extend and modernize legacy applications, while leveraging their investments in J2EE and services-oriented architecture (SOA) platforms. The company claims that organizations using their offerings have reduced application development, deployment, and maintenance costs by an average of 50 percent.

Gangsta cloud wars could pivot on the traffic-driving power of Google and Microsoft/Yahoo

As Phil Wainewright points out, the platform as a service (PaaS) wars are ramping up with the arrival of Google App Engine. And while there will be preferences around Python or Ruby, stack or ala carte -- there's one ingredient that Google and Microsoft/Yahoo (sure to follow suit) can bring to the table that sets them apart, makes them the kingmakers.

And that is the APIs that will expose the newbie apps to the huge respective audiences already gorging themselves on the cloud's burgeoning offerings -- from search to news to blogs to social networks to videos to maps to weather reports. More services makes it a richer cloud experience, so come on down and sign in, please!

Google and Microsoft/Yahoo are not just functional clouds, they are the long tail channels to the vast online audiences, and they can link the right apps to the right users very well, massively. And they can automate the marriage between a service and a customer by scaling it up to nation-wide exposure, or scaling it down to an audience of one. The more users, the more audience for the apps. The more apps, the more users. And so on.

So you build an application on their cloud and they can bring you massive traffic almost overnight. They can allow for single sign-on to your apps from many millions of users already logged in, already identified by their metadata preferences and attention gestures. The web is great, but the integrated cloud is even better.

The clouds can give you, dear developer/entrepreneur, a tasty share of the revenues chugging out of their respective advertising platform engines. All you creative types, just build out the compelling content, services, apps and media, and together -- you and your cloud provider -- can make beautiful business together.

You shouldn't have to worry about attracting an audience and advertisers. The cloud will do all of that for you. (No need for media companies, either.) You just need to invest your creativity, your business acumen, your attention. The cloud is here for you.

Yes, the PaaS wars will soon munge into the metadata wars. Here's how it works: Developers and media creatives build and post innovative web apps, content and services using the cloud tools, publishing platforms and cloud runtime -- all for free. You can mash up other great services from the cloud ecology to make those apps even better. A favor's a favor.

Then, the cloud will match up your services with an audience that the algorithms know wants and loves these services and content. We already see this with blogs, video, podcasts, social discussions, etc.

It's a great vision except there's also a potential downside: You gotta choose your cloud, your Big Brother. Just like a protection racket, it might turn out. Easy to join, not so easy to get out. One big Family.

And the Family, or cloud entity, can make instant stars of an application or service. The cloud with the best metadata can combine users and content/services. The cloud with the most users and the best data about them wins. It's the new muscle. This is why Microsoft must attain Yahoo, and why Yahoo is right to hold out for more money.

I'm not saying abuse is assured. I'm only saying that the handful of clouds will be hugely powerful. Perhaps more powerful than any mainstream media ever was. And they may not do any evil, may remain a great partner and provide the garden that sprouts a million dazzling businesses and a creative nirvana.

But on the other hand, these goombahs at the Cloud, Inc. may want to make you an offer you can't refuse. But you gotta join da gang, see, and stay close to the home, boys. You know, for protection, from the other Cloud, the other goombahs.

Stick with us, kid, and we'll make you rich, they will say. Don't go to the other side, they are closed and proprietary. See?

Yes, the gangsta cloud wars is what we're coming up on. And you're only going to have a handful of gangs to choose from. And you may well make the choice for the rest of the life of your applications.

As SOA hype turns five, IBM turns to Smart SOA Social Network to bind communities of users

The global WebSphere ecology has trooped to Las Vegas this week for the IBM Impact 2008 conference, with a kick-off rally of sorts in the MGM Grand arena this morning.

I saw Fleetwood Mac here at a Comdex, must have been 12 years ago. IBM has sure done all it can to make this into a "SOA rocks" party. Opening tune from an actual marching band was "Tusk." No frisbees, no doobies, though.

This is a big, big crowd (6000) nonetheless, and clear-minded. Software infrastructure and middleware for the Global 2000 is a huge and fast-growing business, and IBM has had a good time at the trough. But nothing stays the same in software for long, so we need to look for the next chapter.

IBM may not be the fastest mover in the market, but when IBM does move, so does the market. IBM can define software's enterprise direction like no other company. HP may be the biggest IT company be some measures, and Microsoft gets around, but no company is more deeply and broadly embedded in the world's major corporations than IBM.

The goal for IBM has been clear for several years, they want the strategic relationships with the big boys, they want to be Barney in all the best boardrooms: I love you, and you love me.

So let's talk business transformation, deep verticals expertise, and decades-in-the-making interwoven ecologies of products, services, suppliers, partners, and technical experts -- from all corners of the landscape.

On stage first we have (8: 39 am PT) the CIO of Harley-Davidson, Jim Haney. He didn't say much, but nice bike. Now the opening video.

Next some classical music from attractive young ladies, circus performers from the ceiling ... it's Vegas.

"Experience" is the theme from Sandy Carter, IBM's SOA honcho. "Smart SOA" comes in, something relevant to the event, but then ... cut to Drew Carey, now of The Price is Right fame. We'll get a few laughs. Let me try ... How much would you pay for that Z Series running Linux instances to support 347 business services? How about 16 services? Maybe 8?

"I don't know what we're selling here, honestly," said Carey. That drew applause. "Don't buy their stupid SOA, buy our smart SOA." Even more laughter. (8:56 am PT)

Six comediens on stage now, several recognizable. This must have cost big bucks. Smart SOA meets over-budget marketing.

Now up, Robert LeBlanc, IBM's GM Global Consulting and SOA. They did some surveys and found that "change" is the biggest challenge for enterprise executives. How to conduct ongoing business model innovation, how to go global integration, manage talent, and gain "business transparency." These are what concern CXOs, says LeBlanc, adding that SOA is an enabler for the answers. (9:14 am PT)

LeBlanc: "Service oriented businesses" do the "key agility indicators" better. You can better beat the competition using SOA. There are multiple onramps to service orientation. Align business and IT. All this will be supported by industry frameworks and solutions, from IBM. We must do a better job of managing risk.

IBM uses SOA too, eats its own dogfood, and helps them run the global elements, better manage mergers and acquisitions, and manage double-digit growth in BRIC countries. Can your IT handle this need for agility and "transforamitive change"? End LeBlanc.

Harley-Davidson CIO Haney is back. (9:23 am PT). "Service orientation is not about the technology," ... it's about the process and bringing the customer into the process. He describes a Harley social networking app that motorcycle riders can map out their rides. By focusing on process integration, they went to ride experiences, needing events, maps, gasoline stops, etc.

Sounds like a web app mashup to me. End to end process integration allows the company to focus out to the consumer, dealer, supplier, rider. "You create an experience that's richer," says Haney. They show the app, again it looks more like WOA than SOA.

Now Steve Mills, SVP of Software at IBM. Theme remains "Smart SOA" and where they are going next. (9:36 am PT) Gives history of software and SOA. Again, it's about business and not technology, he says. Not much new here, says I.

Business managers should focus on services orientation, while the IT folks focus on SOA. Time to rethink applications, as representations of end-to-end processes. And we still struggle with integrations across applications.

"Smart SOA" (which IBM has trade marked) means "robustness" with added horizontal responsiveness and reach, says Mill. A federated ESB helps a lot in moving the value to extended enterprise activities. In 2008, we are now into the phase of how to scale SOA, using the high-performance ESB (like IBM's ESB).

Mills acknowledges that SOA is difficult to understand, that it's about automating business. SOA helps deliver better services an better returns, he says. End Mills. No news, really. (9:55 am PT)

More comedy routines. So IBM has 6000 people in a room and basically repeats the SOA mantra of past years. Checked the wires, no news there either, at least so far. (10:07 am PT)

Tom Rosamilia, GM IBM WebSphere Software is now up. It's about the business, as long as the technology works, he says. Makes pitch for ESB as core to all the BM products and services. No mention of open source alternatives, which are quite popular, for ESBs.

Rosamilia mentions that WebSphere as an app server brand is now 10 years old. And IBM MQ Series messaging software is 15 years old. And CISCS is 40 years old. Wow.

BPM enables b SOA. "You can do BPM without SOA, but I wouldn't recommend it," says Rosamilia.

IBM's news these days, apparently, are more about such acquisitions as Cognos and Telelogic, both of which are quite large for IBM.

Ah, at least, something new: "WebSphere Business Events" ... combinations of events that can be viewed and analyzed. Not much detail there yet. Business events to be big push for IBM this year, he says. (10:25 am PT) End Rosamilia.

Thomas Liese, strategic project executive for AMB Generali of Italy, now up, presumable for a case study. In a customer service project in insurance, they sough standardization, governance, reusability, automation, and input and output management. Says he saved 50% in total costs for the customer service function by using SOA and ESB.

IBM SOA honcho Sandy Carter is back. She says focus is in customers and best practices. She says a Wintergreen Research study shows IBM has improved its SOA marketshare (whatever that is) is up by 11 percent to a total of 64 percent SOA market share.

The Smart SOA Social Network is announced to connect the customers and allow them to share best practices, news, advancements. This expands communities set up last year for SOA developers and architects. They get 120,000 developers and engineers visiting the IBM SOA portal per month, says Carter.

So the SOA Social Network connects the various communities, sort of like a federated social network effect. The network will be role-based in terms of managing and linking among the groups. This reminds me of OpenSocial, but for building out SOA communities, whether they are on Facebook, Twitter, Second Life, or what have you. This may be large defensive against losing control or access to the community discussions. And it may not be open.

An online exchange built on IBM Lotus Connections will do the binding among and between the current SOA communities, says Carter. Furthermore, a "SOA Jam" will be created for SOA brainstorming on next big thoughts on SOA.

Lotus Connections is not exactly taking over the social networking space, so it will be an uphill road for IBM to carve out its own social networking effect on social networking. This seems very much like when large vendors created their own content portals around such initiatives as SOA, but now with their own social networks.

IBM wants to create the uber SOA and/or enterprise software community. You can use your existing social networks, but the effect may well be that users migrate to the IBM SOA exchange social network. IBM no doubt wants a 64% market share on the communities too.

End of show, 11:00 am PT.