Monday, May 7, 2012

Proper security and protection measures enable rapid cloud adoption, say HP experts on discussion panel

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: HP.

It now falls to CIOs to not only rapidly adapt to cloud computing, but to find the ways to protect their employees and customers as they adopt cloud models – even as security threats grow.

This is a serious -- but not insurmountable challenge.

Cloud computing has clearly sparked the imagination of business leaders, who see it as a powerful new way to be innovative and gain first-mover advantages -- with or without traditional IT's consent.

This simply now means that the center of gravity for IT services is shifting toward the enterprise’s boundaries – moving increasingly outside their firewalls. And so how can companies have it both ways -- exploit cloud's promise but also provide enough security to make the risks acceptable? How can organizations retain rigor and control while pursuing cloud benefits?

In a special BriefingsDirect sponsored HP Expert Chat discussion on how to define and obtain improved security, I recently moderated an in-depth session with Tari Schreider, HP Chief Architect of HP Technology Consulting and IT Assurance Practice. Tari is a Distinguished Technologist with 30 years of IT and cyber security experience, and he has designed, built, and managed some of the world’s largest information protection programs.

In our discussion, you’ll see the latest recommendations for how to enable and protect the many cloud models being considered by companies the world over.
If you understand the security risk, gain a detailed understanding of your own infrastructure, security can move from an inhibitor of cloud adoption to an enabler.

As part of our chat, we're also joined by three other HP experts, Lois Boliek, World Wide Manager in the HP IT Assurance Program; Jan De Clercq, World Wide IT Solution Architect in the HP IT Assurance Program; and Luis Buezo, HP IT Assurance Program Lead for EMEA. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]

If you understand the security risk, gain a detailed understanding of your own infrastructure, and follow proven reference architectures and methods, security can move from an inhibitor of cloud adoption to an enabler.

Here are some excerpts from our discussion on how to make the move:

Schreider: It's always a pleasure to be able to chat about some of the technology issues of the day, and certainly cloud computing protection is the topic that’s top of mind for many of our customers.

I want to begin talking about the four immutable laws of cloud security. For those of you who have been involved in information security over time, you understand that there is a certain level of immutability that is incumbent within security. These are things that will always be, things that will never change, and it is a state of being.

When we started working on building clouds at HP a few years ago, we were also required to apply data protection and security controls around those platforms we built. We understood that the same immutable laws that apply to security, business continuity, and disaster recovery extended into the cloud world.

First is an understanding that if your data is hosted in the cloud, you no longer directly control its privacy and protection. You're going to have to give up a bit of control, in order to achieve the agility, performance, and cost savings that a cloud ecosystem provides you.

The next immutable law is that when your data is burst into the cloud, you no longer directly control where the data resides or is processed.

One of the benefits of cloud-based computing is that you don’t have to have all of the resources at any one particular time. In order to control your costs, you want to have an infrastructure that supports you for daily business operations, but there are ebbs and flows to that. This is the whole purpose of cloud bursting. For those of you who are familiar with grid-based computing, the models are principally the same.

Different locations

Rather than your data being in one or maybe a secondary location, it could actually be in 5, 10, or maybe 30 different locations, because of bursting, and also be under the jurisdiction of many different rules and regulations, something that we're going to talk about in just a little bit.

The next immutable law is that if your security controls are not contractually committed to, then you may not have any legal standing in terms of the control over your data or your assets. You may feel that you have the most comprehensive security policy that is rigorously reviewed by your legal department, but if that is not ensconced in the terminology of the agreement with a service provider, then you don’t have the standing that you may have thought you had.

The last immutable law is that if you don’t extend your current security policies and controls in the cloud computing platform, you're more than likely going to be compromised.

You want to resist trying to create two entirely separate, disparate security programs and policy manuals. Cloud-based computing is an attribute on the Internet. Your data and your assets are the same. It’s where they reside and how they're being accessed where there is a big change. We strongly recommend that you build that into your existing information security program.

Gardner: Tari, these are clearly some significant building blocks in moving towards cloud activities, but as we think about that, what are the top security threats from your perspective? What should we be most concerned about?
The reason to move to cloud is for making data and assets available anywhere, anytime.


Schreider: Dana, we have the opportunity to work with many of our customers who, from time to time, experience breaches of security. As you might imagine, HP, a very large organization, has literally hundreds of thousands of customers around the world. This provides us with a unique vantage point to be able to study the morphology of cloud computing platform, security, outages, and security events.

One of the things that we also do is take the pulse of our customer base. We want to know what’s keeping them up at night. What are the things that they're most concerned with? Generally, we find that there is a gap between what actually happens and what people believe could happen.

I want to share with you something that we feel is particularly poignant, because it is a direct interlock between what we're seeing actually happening in the industry and also what keeps our clients up late at night.

First and foremost, there's the ensured continuity of the cloud-computing platform. The reason to move to cloud is for making data and assets available anywhere, anytime, and also being able to have people from around the world accept that data and be able to solve business needs.

If the cloud computing platform is not continuously available, then the business justification as to why you went there in the first place is significantly mooted.

Loss of GRC control

N
ext is the loss of span of governance, risk management, and compliance (GRC) control. In today’s environment, we can build an imperfect program and we can have a GRC management program with dominion over our assets and our information within our own environment.

Unfortunately, when we start extending this out into a cloud ecosystem, whether private, public, or hybrid, we don’t necessarily have the same span of control that we have had before. This requires some delicate orchestration between multiple parties to ensure that you have the right governance controls in place.

The next is data privacy. Much has been written on data privacy and protection across the cloud ecosystem. Today, you may have a data privacy program that’s designed to address the security and privacy laws of your specific country or your particular state that you might reside in.

However, when you're moving into a cloud environment, that data can now be moved or burst anywhere in the world, which means that you could be violating data-privacy laws in another country unwittingly. This is something that clients want to make sure that they address, so it does not come back in terms of fines or regulatory penalties.

Mobility access is the key to the enablement of the power of the cloud. It could be a bring-your-own-device (BYOD) scenario, or it could be devices that are corporately managed. Basically you want to provide the data and put it in the hands of the people.
You have to make sure that you have an incident-response plan that recognizes the roles and responsibilities between owner and custodian.


Whether they're out on an oil platform and they need access to data, or whether it’s the sales force that need access to Salesforce.com data on BlackBerrys, the fact remains that the data in the cloud has to land on those mobile devices, and security is an integral part.

You may be the owner of the data, but there are many custodians of the data in a cloud ecosystem. You have to make sure that you have an incident-response plan that recognizes the roles and responsibilities between owner and custodian.

Gardner: Tari, the notion of getting control over your cloud activities is important, but a lot of people get caught up in the devil in the details. We know that cloud regulations and laws change from region to region, country to country, and in many cases, even within companies themselves. What is your advice, when we start to look at these detailed issues and all of the variables in the cloud?

Schreider: Dana, that is a central preoccupation of law firms, courts, and regulatory bodies today. What tenets of law apply to data that resides in the cloud? I want to talk about a couple of areas that we think are the most crucial, when putting together a program to secure data from a privacy perspective.

Just as you have to have order in the courts, you have to have order in the clouds. First and foremost, and I alluded to this earlier, is that the terms and conditions of the cloud computing services are really what adjudicates the rights, roles, and responsibilities between a data owner and a data custodian.

Choice of law

However, within that is the concept of choice of law. This means that, wherever the breach of security occurs, the courts can actually go to the choice of the law, which means whatever is the law of the land where the data resides, in order to determine who is at fault and at breach of security.

This is also true for data privacy. If your data resides in your home location, is that the choice of law by which you follow the data privacy standards? Or if your data is burst, how long does this have to be in that other jurisdiction before it is covered by that choice of law? In either case, it is a particularly tricky situation to ensure that you understand what rules and regulations apply to you.

The next one is transporter data flow triggers. This is an interesting concept, because when your data moves, if you do a data-flow analysis for a cloud ecosystem, you'll find that the data can actually go across various borders, going from jurisdiction to jurisdiction.

The data may be created in one jurisdiction. It may be sent to another jurisdiction for processing and analysis, and then may be sent to another location for storage, for intermediate use, and yet a fourth location for backup, and then possibly a fifth location for a recovery site.

This is not an atypical example. You could have five triggering events across five different borders. So you have to understand the legal obligations in multiple jurisdictions.
The onus is predominantly placed on the owner of the data for the integrity of the data. The CSP basically wants no direct responsibility for maintaining the integrity of that data.


The next one is reasonable security, which is, under the law, what would a prudent person do? What is reasonable under the choice of law for that particular country? When you're putting together your own private cloud, in which you may have a federated client base, this ostensibly makes you a cloud service provider (CSP).

Or, in an environment where you are using several CSPs, what are the data integrity disclaimers? The onus is predominantly placed on the owner of the data for the integrity of the data, and after careful crafting of terms and conditions, the CSP basically wants no direct responsibility for maintaining the integrity of that data.

When we talk about who owns the data, there is an interesting concept, and there are a few test cases that are coursing their way through various courts. It’s called the Berne Convention.

In the late 1990s, there were a number of countries that got together and said, "Information is flowing all over the place. We understand copyright protection for works of art and for songs and those types of things, but let’s take it a step further."

In the context of a cloud, could not the employees of an organization be considered authors, and could not the data they produce be considered work? Therefore wouldn’t it be covered by the Berne Convention, and therefore be covered under standard international copyright laws. This is also something that’s interesting.

Modify policies

The reason that I bring this to your attention is that it is this kind of analysis that you should do with your own legal counsel to make sure that you understand the full scope of what’s required and modify your existing security policies.

The last point is around electronic evidence and eDiscovery. This is interesting. In some cases it can be a dual-edged sword. If I have custody of the data, then it is open under the rules of discovery. They can actually request that I produce that information.

However, if I don’t directly have control of that data, then I don’t have the right, or I don’t have the obligation, to turn it over under eDiscovery. So you have to understand what rules and regulations apply where the data is, and that, in some cases, it could actually work to your advantage.
Join the next Expert Chat presentation on May 15 on support automation best practices.
Different risk profiles

Y
our risk profile may be different, if you are the custodian, versus the risk profile if you're the owner of the data. This is something that you can very easily put together and present to your executives. It allows you to model the safeguards and controls to protect the cloud ecosystem.

Gardner: We certainly know that there is a great deal of opportunity for cloud models, but unfortunately, there is also significant down side, when things don’t go well. You're exposed. You're branded in front of people. Social media allows people to share issues when they arise. What can we learn from the unfortunate public issues that have cropped up in the past few years that allows us to take steps to prevent that from happening to us?

Schreider: These are all public events. We've all read about these events over the last 16-18 months, and some of them have occurred within just the last 30 days or so. This is not to admonish anybody, but basically to applaud these companies that have come forward in the interest of security. They've shared their postmortem of what worked and what didn’t work.

What goes up can certainly come down. Regardless of the amount of investment that one can put into protecting their cloud computing environment, nobody is immune, whether it’s a significant and pervasive hacking attempt against an organization, where sensitive data is exfiltrated, or whether it is a service-oriented cloud platform that has an outage that prevents people from being able to board a plane.

When an outage happens in your cloud computing environment, it definitely has a reverberation effect. It’s almost a digital quake, because it can affect people from around the world.
You want to make sure that you have a secure system development lifecycle methodology to ensure that the application is secure and has been tested for all conventional threats and vulnerabilities.


One of the things that I mentioned before is that we're very fortunate that we have that opportunity to look at disaster events and breaches of security and study what worked and what didn’t.

I've put together a little model that would reanalyze the storm damage. if you look at the types of major events that have occurred. I've looked at the control construct that would exist, or should exist, in a private cloud and the control construct that should exist in a public cloud, and of course in a hybrid cloud. It's the convergence of the two, and we would be able to mix and match those.

If you have a situation where you have an external threat that infiltrates an application, hacks into it, compromises an application, in a private cloud environment, you want to make sure that you have a secure system development lifecycle methodology to ensure that the application is secure and has been tested for all conventional threats and vulnerabilities.

In a public cloud environment, you normally don’t have that same avenue available to you. So you want to make sure that you either have presented to you, or on behalf of the service provider, have a web-application security review, external threat and vulnerability test.

In a cloud environment, where you are dealing in the situation of grouping many different customers and users together, you have to have a basis to be able to segregate data and operation, so that one of that doesn’t affect everybody.
The level of investment that you make in protecting your cloud environment should be commensurate with the value of the assets that are being burst or hosted in that cloud environment.


Protection through layers

We're a big believer in, whether it's cloud or just in security, having an information technology architecture that's defined by layers. What is the business rationale for the cloud and what are we trying to protect? How should it work together functionally? Technically, what types of products and services will we use, and then how will it all be implemented?

We also have a suite of products that we can bring to our cloud computing environment to ensure that we're securing and providing governance, securing applications, and then also trying to detect breaches of security. I've talked about our reference architecture.

Something that's also unique is our P5 Model, where basically we look at the cloud computing controls and we have an abstraction of five characteristics that should be true to ensure that they are deployed correctly.

As I mentioned before, we're either a principal member, contributing member, or founding member of virtually every cloud security standards organization that's out there. Once again, we can't do it by ourselves, and that's why we have strategic partners with VMwares and the Symantecs of the world.

Gardner: There's a question here about key challenges regarding data lifecycle specifically. How do you view that? What are some of the issues about secure data, even across the data lifecycle?

Key challenges

Luis Buezo: Based on CSA recommendations, we're not only talking about data security related to confidentiality, integrity, and availability, but there are other key challenges in the cloud like location of the data to guarantee that the geographical locations are permitted by regulations.

There's data permanence, in order to guarantee that data is effectively removed, for example, when moving from one CSP to a new one, or data backup and recovery schemes. Don't assume that cloud-based data is backed up by default.

There are also data discovery capabilities to ensure that all data requested by authorities can be retrieved.

Another example is data aggregation on inference issues. This will be implemented to prevent revealing protected information. So there are many issues with having data lifecycle management.

Gardner: Our next question is about being cloud ready for dealing with confidential company data, how do you come down on that?

Jan De Clercq: HP's vision on that is that we think that many cloud service today are not always ready for letting organizations store their confidential or important data. That's why we recommend to organizations, before they consider moving data into the cloud, to always do a very good risk assessment.

They should make sure that they clearly understand the value of their data, but also understand the risks that can occur to that data in the cloud provider’s environment. Then, based on those three things, they can determine whether they should move their data into the cloud.

We also recommend that consumers get clear insights from the CSP on exactly where their organization's data is stored and processed, and where travels inside the network environment of the job provider.

As a consumer you need to get a complete view on what's done with your data and how the CSP is protecting them.

Gardner: Okay, Jan, what are essential data protection security controls that they should look for from their provider?

Clercq: It’s important that you have security controls in place that protect the entire data lifecycle. By data lifecycle we mean from the moment that the data is created to the moment that the data is destroyed.

Data creation

W
hen data is created it’s important that you have a data classification solution in place and that you apply proper access controls to the data. When the data is stored, you need confidentiality, integrity, and availability protection mechanisms in place. Then, you need to look at things like encryption tools, and information rights management tools.

When the data is in use, it’s important that you have proper access control in place,so that you can make sure that only authorized people can access the data. When the data is shared, or when it’s sent to another environment, it’s important that you have things like information rights management or data loss prevention solutions in place.

When the data is archived, it’s important that it is archived in a secured way, meaning that you have proper confidentiality, integrity, and availability protection.

When the data is destroyed, it’s important, as a consumer, that you make sure that the data is really destroyed on the storage systems of your CSP. That’s why you need to look at things like crypto-shredding and other data destruction tools.

Gardner: Tari, how does cloud computing change my risk profile? It's a general subject, but do you really reduce or lose risk control when you start doing cloud?
When the data is destroyed, it’s important, as a consumer, that you make sure that the data is really destroyed on the storage systems of your CSP.


Schreider: An interesting question to be sure, because in some cases, your risk profile could be vastly improved. In other cases, it could be significantly diminished. If you find yourself no longer in a position to be able to invest in a hardened data center, it may be more prudent for you to move your data to a CSP that is already classified as a data-carrier grade, Tier 1 infrastructure, where they have the ability to invest the tens of millions of dollars for a hardened facility that you wouldn’t normally be able to invest yourself.

On the other hand, you may have a scenario where you're using smaller CSPs that don’t necessarily have that same level of rigor. We always recommend, from a strategic perspective when you are looking at application deployment, you consider its risk profile and where best to place that application and how it affects your overall threat posture.

Gardner: Lois, how can HP help clients get started, as they determine how and when to implement cloud?

Lois Boliek: We offer a full lifecycle of cloud-related services and we can help clients get started on their transition to the cloud, no matter where they are in that process.

We have the Cloud Discovery Workshop. That’s where we can help customers in a very interactive work session on all aspects of considerations of the cloud, and it will result in a high-level strategy and a roadmap for helping to move forward.

Business/IT alignment

We also offer the Hybrid Delivery Strategy Services. That’s where we drill down into all the necessary components that you need to gain business and IT alignment, and it also results in a well-defined cloud service delivery model.

We also have some fast-start services. One of those is the CloudStart service, where we come in with a pre-integrated architecture to help speed up the deployment of the production-ready private cloud, and we can do that in less than 30 days.

We also offer a Cloud System Enablement service, and in this we can help fast track setting up the initial cloud service catalog development, metering, and reporting.

Gardner: Does HP have the services to implement protection in the cloud?

Boliek: We believe in building security into the cloud environment from the beginning through our architectures and our services. We offer something called HP Cloud Protection Program, and what we have done is extended the cloud service offerings that I've just mentioned by addressing the cloud security threats and vulnerabilities.
We always recommend that you consider its risk profile and where best to place that application and how it affects your overall threat posture.


We've also integrated a defense in depth approach to cloud infrastructure. We address the people, process, policies, products improved, and the P5 Model that Tari covered, and this is just to help to address confidently and securely build out the hybrid cloud environment.

We have service modules that are available, such as the Cloud Protection Workshop. This is for deep-dive discussions on all the security aspects of cloud, and it results in a high-level cloud security strategy and next steps.

We offer the Cloud Protection Roadmap Service, where we can define the specific control recommendations, also based on our P5 Model, and a roadmap that is very customized and specific to our clients’ risk and compliance requirements.

We have a Foundation Service that is also like a fast start, specific to implementing the pre-integrated, hardened cloud infrastructure, and we mitigate the most common cloud security threats and vulnerabilities.

Then, for customers who require very specific custom security, we can do custom design and implementation. All these services are based on the Cloud Reference Architecture that Jan and Tari mentioned earlier, as well as extensive research that we do ahead of time, before coming out with customers with our Cloud Protection Research & Development Center.

Gardner: Tari, what should users do to determine how good their service provider is when it comes to these security issues?

Incumbent on us

Schreider: I wish we did have a rating system, but unfortunately, it's still incumbent upon us to determine the veracity of the claims of security and continuity of the CSPs.

However, there are actually a number of accepted methods to gauge whether one's CSP is secure. Many organizations have had what's referred to as an attestation. Formally, most people are familiar with SAS 70, which is now SSAE 16, or you can have an ISO 27000.

Basically, you have an independent attestation body, typically an auditing firm, that will come in and test the operational efficiency and design of your security program to ensure that whatever you have declared as your control schema, maybe ISO, NIST, CSA, is properly deployed.

However, there is a fairly significant caveat here. These attestations can also be very narrowly scoped, and many of the CSPs will only attach it to a very narrow portion of their infrastructure, maybe not their entire facility, and maybe not even the application that you're a customer of.

Also, we found that CSPs many application-as-service providers don’t even own their own data centers. They're actually provided elsewhere, and there also may be some support mechanisms in place. In some cases, you may have to evaluate three attestations just to have a sense of security that you have the right controls in place, or the CSP does.
Join the next Expert Chat presentation on May 15 on support automation best practices.
Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: HP.

You may also be interested in:

Thursday, May 3, 2012

Ariba Network helps Cox Enterprises manage procurement across six different ERP systems

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: Ariba.

The latest BriefingsDirect podcast, from the 2012 Ariba LIVE Conference in Las Vegas, explores the latest in cloud-based collaborative commerce with Cox Enterprises, a $15 billion communications, media, and automotive services company.

We'll learn how Cox, through the Ariba Network, manages multiple ERP systems for an improved eProcurement strategy, and has moved toward more efficient indirect spend efforts to improve ongoing operations and drive future growth across more than 50,000 employees.

To hear more about how they have done this, Interarbor Solutuons Principal Analyst Dana Gardner interviews Brooke Krenn, the Senior Manager of Procurement Systems for Cox Enterprises, based in Atlanta. [Disclosure: Ariba is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:
Gardner: A lot of organizations either have organically developed multiple systems for different groups or, for merger and acquisition reasons, have different ERPs. How has that been a challenge, when it comes to procurement?

Krenn: We have six separate ERP systems spanning major subsidiaries, including Cox Communications, Manheim, Cox Media Group, and AutoTrader.com. Cox is a very interesting company in that our business units are very diverse and very unique. Across four divisions and our holding company we have those six ERP systems.

So with that, obviously, there are a lot of challenges. There's not a lot of common ground, when it comes to purchasing. Across those six ERP systems we needed some way to drive consistency, as we focused on really capitalizing on our indirect spend across all the business units.

Procurement systems team

My team is the Procurement Systems Team. We fall under supply chain in Cox Enterprises. I have a team of three, and we manage our eProcurement platform, with which we do about $50 million year-end POs, and average about 1,500 POs a month. We also manage our P-Card program, which is about $130 million a year in spend, and also our fuel card program, which is about $50 million a year.

Historically, our spend, specifically the indirect spend, has been all over the place. We haven’t had a lot of visibility into that spend and haven’t had a consistent manner in which we purchased.

Ariba was one of the top contenders, simply because of the user experience was most important to us, and also how quickly we could implement it.



We had an eProcurement solution for about 10 years. We were on that software for a decade, and it was just very dated. It wasn't supported very well. We knew it was time to make that change. Where we were in the economy, everyone was looking at the most logical places to save time and money and to become more efficient. Obviously, procurement was one of those areas where we could do very quickly.

We knew the first step was replacing the software that we did have. Immediately, Ariba was one of the top contenders, as we looked for a new solution, simply because of the user experience was most important to us, and also how quickly we could implement it.

Gardner: So you’re going from an on-premises software installed affair to now more of a software-as-a-service (SaaS) and cloud affair. Was that something that was difficult or something you were looking forward to?

Krenn: Moving to the cloud in an on-demand solution was great for us. Having the on-premises software in the past, any time there was an upgrade or an update, we had to be sure IT knew about it and we scheduled the time on a night or a weekend. We had to call on resources internally within the company. So it was very exciting for us to move to an on-demand solution and all of the technology that was available with that.

A great change

For the users, it's been a great change, because now they consistently know there's one place to go. When they need to order office supplies, when they need to order something for their break room, when they need to order business cards, they know where to go. In all of our divisions and all of our locations, employees want to do the right thing. They want to purchase the right way. A lot of times they're just not sure of what to do.

So with this implementation of a new tool, we were able to really drive them in the right direction, and it was an easy solution for them. It was easy for us to implement, and it's been very easy for our end users and our employees to adopt.

Gardner: Has that, in fact, translated into other metrics of success that you could describe for us?

With this implementation of a new tool, we were able to really drive them in the right direction, and it was an easy solution for them.



Krenn: Probably one of the biggest wins for us has been just driving compliance against our contracts. We’re able to see very easily now when a location or a business unit within one of the divisions is purchasing off-contract or when they're not utilizing one of our preferred or negotiated suppliers. That's probably been the biggest win for us.

We have the visibility now to see very quickly within our P2P tool and also within our spend management tool to see where this spend is taking place and able to reach out directly to those locations or to those employees that are purchasing off-contract. Obviously, the more purchasing power we have, the more spend we are driving to these contracts, the better our pricing is going to be going forward.

Unconventional

We went about implementing our new P2P solution a bit unconventionally, you could say. About 98 percent of our transactions are actually on a supplier card -- a P-Card model, which has just been tremendously successful for us. With that, we didn't have to integrate directly into our six separate ERPs because our payment method is with that supplier card.

Ease of implementation was one of the biggest wins. Also with that is the ease of use for the end user. There's no reconciliation for them at the end of the month. We’re taking care of all of that GL coding information, all of the approvals, upfront. The supplier card model, again, has been great on the end user side as well as on the AP reconciliation side.
Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: Ariba.

You may also be interested in:

Monday, April 30, 2012

Another vote for the Apache Hadoop stack

This guest post comes courtesy of Tony Baer's OnStrategies blog. Tony is senior analyst at Ovum.

By Tony Baer

As we’ve noted previously, the measure of success of an open source stack is the degree to which the target remains intact. That either comes as part of a captive open source project, where a vendor unilaterally open sources their code (typically hosting the project) to promote adoption, or a community model where a neutral industry body hosts the project and gains support from a diverse cross section of vendors and advanced developers. In that case, the goal is getting the formal standard to also become the de facto standard.

The most successful open source projects are those that represent commodity software – otherwise, why would vendors choose not to compete with software that anybody can freely license or consume? That’s been the secret behind the success of Linux, where there has been general agreement on where the kernel ends, and as a result, a healthy market of products that run atop (and license) Linux. For community open source projects, vendors obviously have to agree on where the line between commodity and unique value-add begins.

And so we’ve discussed that the fruition of Hadoop will require some informal agreement as to exactly what components make Hadoop, Hadoop. For a while, the question appeared in doubt, as one of the obvious pillars – the file system – was being contested with proprietary alternatives like MapR and IBM’s GPFS.

Retrenching

What’s interesting is that the two primary commercial providers that signed on for the proprietary files systems – IBM and EMC (via partnership with MapR) – have retrenched. They still offer the proprietary file system systems in question, but both now also offer purer Apache versions. IBM made the announcement today, buried below the fold after its announced intention to acquire data federation search player, Vivisimo. The announcement had a bit of a grudging aspect to it – unlike Oracle, which has a full OEM agreement with Cloudera, IBM is simply stating that it will certify Cloudera’s Hadoop as one of the approved distributions for InfoSphere BigInsights – there’s no exchange of money or other skin in the game. If IBM also gets demand for the Hortonworks distro (or if it wants to keep Cloudera in its place), it’ll also likely add Hortonworks to the approved list.

Against this background is a technology that is a moving target. The primary drawback – that there was no redundancy or failover with the HDFS NameNode (which acts as a file directory) – has been addressed with the latest versions of Hadoop. The other – which provides POSIX compliance so Hadoop can be accessed through the NFS standard) – is only necessary for very high, transactional-like (OK, not ACID) performance which so far has not been an issue. If you want that kind of performance, Hadoop’s HBase offers more promise.

What’s interesting is that the two primary commercial providers that signed on for the proprietary files systems have retrenched.

But just as the market has passed judgment on what comprises the Hadoop “kernel” (using some Linuxspeak), that doesn’t rule out differences in implementation. Teradata Aster and Sybase IQ are promoting their analytics data stores as swappable, more refined replacements for HBase (Hadoop’s column store), while upstarts like Hadapt are proposing to hang SQL data nodes onto HDFS.

When it comes to Hadoop, you gotta reverse the old maxim: The more things stay the same, the more things are actually changing.

This guest post comes courtesy of Tony Baer's OnStrategies blog. Tony is senior analyst at Ovum.

You may also be interested in:

Friday, April 27, 2012

Fast data hits the big data fast lane

This guest post comes courtesy of Tony Baer's OnStrategies blog. Tony is senior analyst at Ovum.

By Tony Baer

Of the 3 “V’s” of Big Data – volume, variety, velocity (we’d add “Value” as the 4th V) – velocity
has been the unsung ‘V.’ With the spotlight on Hadoop, the popular image of Big Data is large petabyte data stores of unstructured data (which are the first two V’s). While Big Data has been thought of as large stores of data at rest, it can also be about data in motion.

“Fast Data” refers to processes that require lower latencies than would otherwise be possible with optimized disk-based storage. Fast Data is not a single technology, but a spectrum of approaches that process data that might or might not be stored. It could encompass event processing, in-memory databases, or hybrid data stores that optimize cache with disk.

Fast Data is nothing new, but because of the cost of memory, was traditionally restricted to a handful of extremely high-value use cases. For instance:
  • Wall Street firms routinely analyze live market feeds, and in many cases, run sophisticated complex event processing (CEP) programs on event streams (often in real time) to make operational decisions.
  • Telcos have handled such data in optimizing network operations while leading logistics firms have used CEP to optimize their transport networks.

    While Big Data has been thought of as large stores of data at rest, it can also be about data in motion.

  • In-memory databases, used as a faster alternative to disk, have similarly been around for well over a decade, having been employed for program stock trading, telecommunications equipment, airline schedulers, and large destination online retail (e.g., Amazon).
Hybrid in-memory and disk have also become commonplace, especially amongst data warehousing systems (e.g., Teradata, Kognitio), and more recently among the emergent class of advanced SQL analytic platforms (e.g., Greenplum, Teradata Aster, IBM Netezza, HP Vertica, ParAccel) that employ smart caching in conjunction with a number of other bells and whistles to juice SQL performance and scaling (e.g., flatter indexes, extensive use of various data compression schemes, columnar table structures, etc.).

Many of these systems are in turn packaged as appliances that come with specially tuned, high-performance backplanes and direct attached disk.

Finally, caching is hardly unknown to the database world. Hot spots of data that are frequently accessed are often placed in cache, as are snapshots of database configurations that are often stored to support restore processes, and so on.

So what’s changed?


The usual factors: the same data explosion that created the urgency for Big Data is also generating demand for making the data instantly actionable. Bandwidth, commodity hardware and, of course, declining memory prices, are further forcing the issue: Fast Data is no longer limited to specialized, premium use cases for enterprises with infinite budgets.

Not surprisingly, pure in-memory databases are now going mainstream: Oracle and SAP are choosing in-memory as one of the next places where they are establishing competitive stakes: SAP HANA vs. Oracle Exalytics.

Both Oracle and SAP for now are targeting analytic processing, including OLAP (by raising the size limits on OLAP cubes) and more complex, multi-stage analytic problems that traditionally would have required batch runs (such as multivariate pricing) or would not have been run at all (too complex, too much delay).

Not surprisingly, pure in-memory databases are now going mainstream.



More to the point, SAP is counting on HANA as a major pillar of its stretch goal to become the #2 database player by 2015, which means expanding HANA’s target to include next generation enterprise transactional applications with embedded analytics.

Potential use cases for Fast Data could encompass:
  • A homeland security agency monitoring the borders requiring the ability to parse, decipher, and act on complex occurrences in real time to prevent suspicious people from entering the country
  • Capital markets trading firms requiring real-time analytics and sophisticated event processing to conduct algorithmic or high-frequency trades
  • Entities managing smart infrastructure which must digest torrents of sensory data to make real-time decisions that optimize use of transportation or public utility infrastructure
  • B2B consumer products firms monitoring social networks may require real-time response to understand sudden swings in customer sentiment
For such organizations, Fast Data is no longer a luxury, but a necessity.

More specialized use cases are similarly emerging now that the core in-memory technology is becoming more affordable. YarcData, a startup from venerable HPC player Cray Computer, is targeting graph data, which represents data with many-to-many relationships. Graph computing is extremely process-intensive, and as such, has traditionally been run in batch when involving Internet-size sets of data. YarcData adopts a classic hybrid approach that pipelines computations in memory, but persisting data to disk. YarcData is the tip of the iceberg – we expect to see more specialized applications that utilize hybrid caching that combine speed with scale.

Memory’s not the new disk


T
he movement – or tiering – of data to faster or slower media is also nothing new. What is new is that data in memory may no longer be such a transient thing, and if memory is relied upon for in situ processing of data in motion or rapid processing of data at rest, memory cannot simply be treated as the new disk. Excluding specialized forms of memory such as ROM, by nature anything that’s solid state is volatile: there goes your power… and there goes your data.

Not surprisingly, in-memory systems such as HANA still replicate to disk to reduce volatility. For conventional disk data stores that increasingly leverage memory, Storage Switzerland’s George Crump makes the case that caching practices must become smarter to avoid misses (where data gets mistakenly swapped out).

There are also balance of system considerations: memory may be fast, but is its processing speed well matched with processor?



There are also balance of system considerations: memory may be fast, but is its processing speed well matched with processor? Maybe solid state overcomes I/O issues associated with disk, but may still be vulnerable to coupling issues if processors get bottlenecked or MapReduce jobs are not optimized.

Declining memory process are putting Fast Data on the fast lane to mainstream. But as the technology is now becoming affordable, we’re still early in the learning curve for how to design for it.

This guest post comes courtesy of Tony Baer's OnStrategies blog. Tony is senior analyst at Ovum.

You may also be interested in:

Thursday, April 26, 2012

Case study: Strategic approach to disaster recovery and data lifecycle management pays off for Australia's SAI Global

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: VMware.

The latest BriefingsDirect case study discussion focuses on how business standards and compliance services provider SAI Global is benefiting from a strategic view of IT enabled disaster recovery (DR).

Learn here how SAI Global has brought advanced backup and DR best practices into play for its users and customers. Examine too how this has not only provided business continuity assurance, but it has also provided beneficial data lifecycle management and virtualization efficiency improvement.

Mark Iveli, IT System Engineer at SAI Global, based in Sydney, Australia, details on how standardizing DR has helped improve many aspects of SAI Global’s business reliability. The discussion is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: VMware is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:
Iveli: When we started to get into DR, we handled it from an IT point of view and it was very much like an iceberg. We looked at the technology and said, "This is what we need from a technology point of view." As we started to get further into the journey, we realized that there was so much more that we were overlooking.

We were working with the businesses to go through what they had, what they didn’t have, what we needed from them to make sure that we could deliver what they needed. Then we started to realize it was a bigger project.

The initiative for DR started about 18 months ago with our board, and it was a directive to improve the way we had been doing things. That meant a complete review of our processes and documentation.

We had a number of business units that all had different strategies for their disaster recovery, and different timings and mechanisms to report on it.

Through the use of VMware Site Recovery Manager (SRM) in the DR project, we've been able to centralize all of the DR processes, provide consistent reporting, and be able to schedule these business units to do all of their testing in parallel with each other.

So we can make a DR session, so to speak, within the business and just run through the process for them and give them their reports at the end of it.

We've installed SRM 4.1 and our installation was handled by an outsource company, VCPro. They were engaged with us to do the installation and help us get the design right from a technical point of view.

Trying to make it a daily operational activity is where the biggest challenge is, because the implementation was done in a project methodology.



Trying to make it a daily operational activity is where the biggest challenge is, because the implementation was done in a project methodology. Handing it across to the operational teams to make it a daily operation, or a daily task, is where we're seeing some challenges.

I'm a systems engineer with SAI Global, and I've been with the company for three years. When the DR project started to gather some momentum, I asked to be a significant part of the project. I got the nod and was seconded to the DR project team because of my knowledge of VMware.

That’s what my role is now -- keeping the SRM environment tuned and in line with what the business needs. That’s where we're at with SRM.

Complete review

The first 12 months of this journey so far has been all around cleaning up, getting our documentation up to spec, making sure that every business unit understood and was able to articulate their environments well. Then, we brought all that together so that we could say what’s the technology that’s going to encapsulate all of these processes and documentation to deliver what the business needs, which is our recovery point objective (RPO) and for our recovery time objective (RTO).

SAI Global is an umbrella company. We have three to four main areas of interest. The first one, which we're probably most well-known for, is our Five Ticks brand, and that’s the ASIS standards. The publication, the collection, the customization to your business is all done through our publishing section of the business.

That then flows into an assurance side of the business, which goes out and does auditing, training, and certification against the standards that we sell.

We continue to buy new companies, and part of the acquisition trail that we have been on has been to buy some compliance businesses. That’s where we provide governance risk and compliance services through the use of Board Manager, GRC Manager, Cintellate, and in the U.S., Integrity 360.

Finally, last year, we acquired a company that deals solely in property settlement, and they're quite a significant section of the business that deals a lot with banks and convincing firms in handling property settlements.

So we're a little bit diverse. All three of those business sections have their own IT requirements.

Gardner: Like many businesses, your brand is super important. The trust associated with your performance is something you will take seriously. So DR, backup and recovery, business continuity, are top-line issues for you.

Because of what we do, especially around the property settlement and interactions with the banks, DR is critical for us.



Is there anything about what you've been doing as a company that you think makes DR specifically important for you?

Iveli: From SAI Global’s point of view, because of what we do, especially around the property settlement and interactions with the banks, DR is critical for us.

Our publishing business feels that their website needs to be available five nines. When we showed them what DR is capable of doing, they really jumped on board and supported it. They put DR as high importance for them.

As far as businesses go, everyone needs to be planning for this. I read an article recently where something like 85 percent of businesses in the Asia-Pacific region don’t have a proper DR strategy in place. With the events that have happened here in Australia recently with the floods, and when you look at the New Zealand earthquakes and that sort of stuff, you wonder where the businesses are putting DR and how much importance they've got on it. It’s probably only going to take a significant event before they change their minds.

Gardner: I was intrigued, Mark, when you said what DR is capable of doing. Do you feel that there is a misperception, perhaps an under-appreciation of what DR is?

Process in place

Iveli: The larger DR whole was just that these business units had a process in place, but it was an older process and a lot of the process was designed around a physical environment.

With SAI Global being almost 100 percent virtual, moving them into a virtual space opened their minds up to what was possible. So when we can sit down with the business units and say, "We're going to do this DR test," they ask if it will impact production. No, it won’t. How is it happening? "Well, we are going to do this, this, and this in the background. And you will actually have access to your application the way it is today, it’s just going to be isolated and fenced off."

They say, "This is what we've been waiting for." We can actually do this sort of stuff. They're starting to see and ask, "Can we use this to test the next version of the applications and can we test this to kind of map out our upgrade path?"

We're starting to move now into a slightly different world, but it has been the catalyst of DR that’s enabled them to start thinking in these new ways, which they weren’t able to do before.

Gardner: So being able to completely switch over and recover with very little interruption in terms of the testing, with very little downtime or loss, the opportunity then is to say, "What else can we do with this capability?"

It has been the catalyst of DR that’s enabled them to start thinking in these new ways, which they weren’t able to do before.



Iveli: Absolutely. With this new process, we've taken the approach of baby steps, and we're just looking to get some operational maturity into the environment first, before we start to push the boundaries and do things like disaster avoidance.

Having the ability to just bring these environments across in a state that’s identical to production is eye-opening for them. Where the business wants to take it is the next challenge, and that’s probably how do we take our DR plan to version 2.0.

We need to start to work with the likes of VMware and ask what our options are now. We have this in place, people are liking it, but they want to take it into a more highly available solution. What do we do next? Use vCloud Director? Do we need to get our sites in an active/active pairing?

However, whatever the next technology step is for us, that’s where the business are now starting to think ahead. That’s nice from an alignment point of view.

Gardner: Those DR maturation approaches put you in a position to further leverage virtualization. Is there sort of a virtuous adoption pattern, when you combine modern DR with widespread virtualization?

Iveli: Because all of a sudden, your machines are just a file on a data store somewhere, now you can move these things around. As the physical technologies continue to advance -- the speed of our networks, the speed of the storage environments, metro clustering, long haul replication -- these technologies are allowing businesses to think outside of the box and look at ways in which they can provide faster recovery, higher availability, more elastic environments.

You're not pinned down to just one data center in Sydney. You could have a data center in Sydney and a data center in New Zealand, for instance, and we can keep both of those sites online and in sync. That’s couple of years down the track for our business, but that’s a possibility somehow through the use of more virtualization technology.

Gardner: Any advice for those listening in who are beginning their journey? For those folks that are recognizing the risks and seeing these larger benefits, these more strategic benefits, how would you encourage them to begin their journey, what advice might you offer?

Iveli: The advice would be to get hired guns in. With DR, you're not going to be able to do everything yourself. So spend a little bit more money and make sure that you get some consultants in like VCPro. Without these guys, we probably would have struggled a little bit just making sure that our design was right. These guys ensured that we had best practice in our designs.

Before you get into DR, do your homework. Make sure that your production environment is pristine. Clean it up. Make sure that you don’t have anything in there that’s wasting your resources.

Come around with a strong business case for DR. Make sure that you've got everybody on board and you have the support of the business.

Make sure that your production environment is pristine. Clean it up. Make sure that you don’t have anything in there that’s wasting your resources.



When you get into DR, make sure that you secure dedicated resources for it. Don't just rely on people coming in and out of the project. Make sure that you can lead people to the resource and you make sure that they are fully engaged in the design aspects and the implementation aspects.

And as you progress with DR, incorporate it as early as you can into your everyday IT operation. We're seeing that, because we held it back from our operations, just handing it over and having them manage the hardware and the ESX and the logical layers, the environment, they were struggling just to get their head around it and what was what, where should this go, where should that go.

And once it’s in place, celebrate. It can be a long haul. It can be quite a trying time. So when you finally get it done, make sure that you celebrate it.

Gardner: And perhaps a higher degree of peace of mind that goes with that.

Iveli: Well, you'll find out when you get through it, how much easier this is making your life, how much better you can sleep at night.
Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: VMware.

You may also be interested in: