91d437f0fb56cfb5b64de81b60bd6738

IBM announced new offering in Cloud http://www.ibm.com/software/data/infosphere/hadoop/

The comments from CNet:

• IBM chooses Hadoop to analyze big data
• IBM BigSheets to preserve fleeting Web data

tags: linux bi opensource hadoop cloud big apache infosphere data biginsights

I discussed the subject with one of colleagues who is a Cloud Computing architect. I’m logging what my comment here

=-=-

To continue our previous talk is a huge discussion. I’d recommend you could start to get involved in http://twitter.com and follow

http://twitter.com/mashable
http://twitter.com/j3ffyang -> me
http://twitter.com/OpenIBM -> I own this too as of being admin for IBM Open Source Global community

When you hit any of above, you can see the bytes come from http://aws.amazon.com actually, which is an IaaS with no doubt. Twitter is a kind of social collaboration network -> an application running over Amazon Web Services (AWS). Twitter doesn’t own any hardware resources. Whenever Twitter needs computing capability, it goes to AWS and AWS fulfills its request… on demand. Even though Twitter sometimes out of service due to its overload.

@ IaaS, AWS not only gives power of hardware (CPU, memory, disk and network…), but also provides plugin / API to connect Twitter with Hadoop… and Simple Queue Service (SQS), and Simple Database (SDB).

You shouldn’t be surprised @ Twitter’s power to gather thousand of thousand developers around it, if keeping our eyes open. (Some reasons of the motivation of social collaboration in term of Web2.0). See these:

http://tweetwheel.com
http://ftags.com
http://tweetvalue.com/
http://www.tweetizen.com/
http://twittersnooze.com/
http://mrtweet.net/home/j3ffyang
http://tweettrail.com/search/hadoop
…
…

This list is almost endless. They’re all independent of twitter.com. The above are all SaaS, plugable into Twitter.com where provides API -> http://apiwiki.twitter.com/ and http://twitter.com/downloads – @ PaaS

This is an ecosystem.

Building a private open source’d Cloud Computing environment isn’t that difficult as you think. Get the code from http://eucalyptus.cs.ucsb.edu and it took me about 3~ 4 days @ my spare time to complete the installation and configuration. This environment shares code with Amazon Elastic Compute Cloud (EC2), which means your Amazon Machine Image (AMI) can run on top of this.

Look @ some screenshots

Cloud is a huge topic and every vendor/ people has its own definition as each has its own goal/ interest. I give you my personal comment

I’m not expert of storage. But in term of Cloud, storage doesn’t bring much difference than other elements, like SaaS, PaaS, Network as a Service… In my earlier speeches @ software group and lab, I was asked what Cloud is. My answer in one word is Scalability, if in two words, they’re Dynamic Scalability.

Let’s look at Apache Hadoop, which makes EC2, S3, SDB, CloudFront in Amazon. The most shining part is Hadoop enables clustering commodity servers – low end, high end, *nix or Windows, 1 CPU or 16 CPUs, Pentium I or IV, memory low… all doesn’t matter. They all can become a part of Cloud that provides storage and computing capability. Even though there is still performance bugs in current release (bring-up a Hadoop cluster of 1,500 machines takes about 1 hour, when its primary NameNode is down and restarting…), its release now is 0.18. Not only does it bring marketing hype in Cloud, but also real business to cloud service provider, such as Amazon Web Service, FlexiScale, ParaScale, RackSpace and a lot.

Does IBM have the similar product or solution? IBM focuses on high- end and enterprise! IBM may miss some. I’m engaged with one telco customer in China, who has only commodity servers, without having WebSphere, Tivoli, and DB2. They expect IBM build a solution of internet hosting service. From my point of view, Amazon Machine Image (AMI) closely fits their requirement/ environment in RackSpace’s business model.

Private Storage Cloud is a trend that I cannot deny. I think AMI- like and RackSpace- like cloud models work in Private Cloud as well, with enhancement of APIs to IBM branded products.

Back to Amazon, on top of Apache Hadoop, what else uniqueness does Amazon have? Amazon developed tons of friendly APIs that allows easy access to their Cloud architecture.

I don’t have much in- depth view on Sun’s storage. But WikiMedia 3 days ago announced to choose Sun’s Cloud Computing service to enrich its multimedia experience… http://www.sun.com/aboutsun/pr/2008-11/sunflash.20081120.1.xml . It’s a kind of signal…

EMC releases Atmos > http://www.emc.com/products/detail/software/atmos.htm that is IBM’s strong competitor.

Microsoft has too ironic image that limits its offering/ service on operating system oriented even it is eager to expand its product line of Cloud Computing by Windows Live and Hyper-V. Microsoft is too OS dependent that doesn’t encourage the choice/ adoption in cloud perspective.

The entire article comes from
http://www.on-demandenterprise.com/…Telcos_Will_Dominate_Enterprise_Cloud_Computing

I’m capturing some key bullets from it with my personal highlight in bold and italic… It’s worth to take time to read this through.

A cloud definition:
it will be helpful to define what a cloud service is. I define it as a CLOUD: Common, Location-independent, Online Utility provisioned on-Demand. Common (i.e., shared) in that it multiplexes demand from multiple customers and applications into a common pool of resources. Location-independent, because it shouldn’t matter where you are or where the service is. Online, in the sense that it is accessible over a network, as well as “not down.” A utility because it provides value and offers usage-sensitive pricing. And on-demand in that the ability to provision capacity or service should be as fast as possible to meet variable demand requirements, enhancing business agility and providing capacity at the lowest total cost.

Under this definition, not only can computing be cloud-based, but so can be storage, security, audio conferencing, video conferencing, Web conferencing, messaging, collaboration, software as a service and so forth. In fact, cloud services have been around since well before today’s latest networked IT architectures and business models. Hotel chains are cloud services: they time- and space-division multiplex guests traveling as individuals and in groups, on vacation or business, into dynamically allocated units of capacity (rooms). They are location-independent, in that no matter what city you are in, you are likely to find a service node (a local hotel from the chain). They are online, accessible over wide-area highways and local-area hallways. They also are utilities (pay per room per night). And they are available on-demand (although reservations are recommended during peak season).
…

In fact, such providers have 10 major strategic advantages in this market:

1. Enterprise sales capability — Telcos have a long history of selling to enterprises as well as consumers. For example, AT&T had annual revenues of $119 billion in 2007 — more than either IBM or HP — and roughly half of those revenues come from businesses. Unlike their consumer or start-up counterparts, enterprise CIOs do not want to go online to initiate and manage a relationship. They want dedicated account teams collaborating closely with them and their teams for the long term, in many cases with a permanent on-site presence. Some might argue that there is a major business model transformation underway. After all, who needs an enterprise sales force when employees can just use their credit card to provision services?This is unlikely to happen in the enterprise for three reasons. First, most enterprises have tight controls on purchasing that extend to $10 worth of business cards, much less buying online computing and storage capacity. Second, no corporate information security officer is likely to appreciate the idea of tens of thousands of employees purchasing cloud services and placing proprietary corporate data willy-nilly across providers and platforms. Third, enterprise IT shops already have experienced the chaos and hidden costs associated with loss of control of applications, desktop images, and foundation architecture in departmental computing and rich desktop environments, and thus are not likely to support a model of individual purchases of cloud capacity and services. If the enterprise wants to avail itself of the benefits of the cloud, credit card purchasing is not the way to go.
2. Lifecycle service and support — It’s not just sales, but also after-sales service and support, including: lifecycle management teams ensuring successful service delivery 24/7; advanced tooling for service monitoring and management; portals for network and application performance, usage monitoring and configuration and provisioning changes; and even e-bonding between enterprise systems and service provider systems.
3. Reliable operations at scale — Rather than offering services that still remain in “Preview Release” or permanent “Beta” purgatory after many years to avoid any implied service reliability or feature stability commitments, service providers go through a comprehensive suite of pre-launch interoperability, certification, and scalability engineering and testing. In fact, telcos are used to engineering services for four or five nines of availability, even as they scale up to tens of millions of customers. This reliability at scale is in telcos’ DNA and service culture, as well as in regulatory requirements. Imagine a trauma victim calling 911 and getting a pre-recorded message saying, “Your call did not go through — but, hey, we’re still in beta.” It isn’t clear that a new economy culture of random innovation is compatible with a culture of continuous delivery of the same service to tens of millions of customers day after day.
4. SLAs with financial penalties – Not only won’t enterprises accept “Well, after all, it’s still in beta” as an excuse for service outages, they demand meaningful SLAs (service level agreements) with clear metrics for evaluating achievement of those SLAs, backed up by monitoring and management systems, and financial penalties such as credits or refunds if service levels aren’t met. A “free” or low-cost service with questionable delivery quality is about as attractive to a CIO as an offer of free neurosurgery from someone who just skimmed a blog on how to do it in three easy steps.
5. Full enterprise solutions portfolio — Cloud computing services don’t exist in a vacuum; many other services may be procured in conjunction with them, either due to technical architecture requirements or due to contracting benefits, such as discounts for total spend. Related services such as network access and transport, MPLS VPNs for backhauling to the enterprise datacenter, application management, global load balancing, asymmetric Web acceleration, network-based firewalls and other network-based security services, content delivery, Voice over IP, Video over IP, managed messaging, Web conferencing and remote access can offer synergies when combined with cloud computing and storage.
6. Integrated hosting and network services — This has real benefits in terms of cost and performance. It generates cost advantages in a number of ways. First, having hosting facilities on net — that is, in the same locations as core network backbone switching and routing facilities — eliminates expenses associated with building additional access facilities to reach a third-party datacenter. Integrated providers also can access network facilities at cost, rather than at market prices. And larger providers should be able to achieve more compelling economies of scale. Having hosting facilities on net also means better performance by reducing router hops and associated physical propagation delays.
7. Vendor independence — Service providers tend to be software and hardware vendor-agnostic. The reason for this is that their broad customer bases have wide ranges of requirements and preferences, and service providers are strategically intent on reaching as wide a market as possible. Consequently, lock-in to a specific storage, server, operating system, hypervisor, middleware, database or application vendor would be self-defeating by limiting market penetration. This contrasts with some of the existing players, who mostly seem to have at least some proprietary elements to their platforms.
8. Global footprint — It’s not news that today’s enterprises have gone global. Whether it’s a global base of employees, customers, supply chain partners, offshore contact centers or skill base for innovation, reach and footprint are critical. Large, integrated global service providers have the capability to provide services locally and consistently virtually anywhere in the world to support today’s increasingly interactive applications with proximate infrastructures that reduce response time — and with the sales and support resources to directly engage with regional or local leadership, or corporate executives headquartered anywhere from Shanghai to Dubai, Bangalore to Brussels, or Sydney to Sao Paulo.
9. Financial stability and market commitment — In today’s tumultuous economic environment, enterprises are more focused than ever on the financial stability, brand and business viability of service providers providing key parts of their infrastructures. Commitment to hosting and cloud computing as part of their provider’s core business is important, as opposed to cloud services being a potentially temporary excursion from different core businesses such as online retailing or advertising. Over the last few years, high and rising stock prices have permitted some new economy players substantial flexibility in capital investments, but recent drops of fifty or sixty percent may slow such adventurism for the foreseeable future.
10. Technologies are easier to replicate than relationships and operations — Don’t the famously highly paid developers at the new economy companies have an edge in creating new technologies such as automated provisioning that enable cloud services to rapidly scale up and down? If they do — which is arguable — it isn’t sustainable. Such technologies have been around for years from companies as small as BladeLogic and as large as IBM (e.g., Tivoli Provisioning Manager), with variations such as VMware’s vCenter and VMotion fitting into the mix. For every highly paid developer at an online bookseller, there is a highly motivated developer at a start-up or large global software firm, developing software tools for others, like integrated service providers, to incorporate into their tooling and management platforms. Even Animoto, the poster child for non-consumer use of cloud computing services, leveraged a third party, RightScale, to manage dynamic allocation of these services. Service providers also can choose best-in-class capabilities and focus on integration. Much harder to replicate are global networks that have been built for literally hundreds of billions of dollars of investment, and the experienced skill base, long-term enterprise customer relationships, management tools, support organizations, service culture, and local access and regulatory relationships that enable services to be delivered successfully at scale.

Source: http://www.rpath.com/corp/cloud-adoption-model

The cloud computing adoption approach

The cloud computing adoption approach

• Level 1: Virtualization. The first level of cloud adoption employs hypervisor-based infrastructure and application virtualization technologies for seamless portability of applications and shared server infrastructure.
• Level 2: Cloud Experimentation. Virtualization is taken to a cloud model, either internally or externally, based on controlled and bounded deployments utilizing the Amazon Elastic Compute Cloud (EC2) for compute capacity and as the reference architecture.
• Level 3: Cloud Foundations. Governance, controls, procedures, policies, and best practices begin to form around the development and deployment of cloud applications. Initially, level 3 efforts will focus on internal, non-mission critical applications.
• Level 4: Cloud Advancement. Governance foundations allow organizations to scale up the volume of cloud applications through broad-based deployments in the cloud
• Level 5: Cloud Actualization. Dynamic workload balancing across multiple utility clouds. Applications are distributed based on cloud capacity, cost, proximity to user and other criteria.

View a PDF of the basics of the model.

Source: http://www.ibm.com/developerworks/linux/./l-cloud-computing

Author: M. Tim Jones, Consultant Engineer, Emulex Corp.

Click the source URL to view the original article. I brief some key points with diagrams

Figure 1. Cloud Computing migrates resources within internet

Figure 2. Virtualization and resource use

Figure 3. The layers of Cloud Computing

Figure 4. Cloud Computing landscape

Linux and open source in the Cloud

Software-as-a-Service

SaaS is the ability to access software over the Internet as a service. An early approach to SaaS was the Application Service Provider (ASP). ASPs provide subscriptions to software that is hosted or delivered over the Internet. The ASP delivers the software and charges fees based on its use. In this way, you don’t purchase the software but simply lease it on an as-needed basis.

Example SaaS An interesting example of traditional versus SaaS applications is the application life cycle management tool from SoftwarePlanner.com. This company offers their tool using the traditional model, where customers host the application suite within their enterprise, or as SaaS, where customers host the application suite and make it available over the Internet.

Another perspective on SaaS is the use of software over the Internet that executes remotely. This software can be in the form of services used by a local application (defined as Web services) or a remote application observed through a Web browser. One example of a remote application service is Google Apps, which provides several enterprise applications through a standard Web browser. Remotely executing applications commonly rely on an application server to expose needed services. An application server is a software framework that exposes APIs for software services (such as transaction management or database access). Examples include Red Hat JBoss Application Server, Apache Geronimo, and IBM® WebSphere® Application Server. Many other application servers exist, and an extensive list is included in Resources.

Another recent example of SaaS is Google’s Chrome browser. The browser is an ideal environment as a new desktop through which applications can be delivered (either locally or remotely) in addition to the traditional Web browsing experience. (For more information, see Resources.)

Platform-as-a-Service

PaaS can be described as an entire virtualized platform that includes one or more servers (virtualized over the set of physical servers), operating systems, and specific applications (such as Apache and MySQL for Web-based applications). In some cases, these platforms can be predefined and selected; in others, you can provide a VM image that contains all the necessary user-specific applications.

One interesting example of a PaaS is Google App Engine. App Engine is a service that allows you to deploy your Web applications on Google’s very scalable architecture. App Engine provides you with a sandbox for your Python application that can be referenced over the Internet (and additional languages will be supported in the future). App Engine provides Python APIs for persistently storing and managing data (using the Google Query Language, or GQL) in addition to support for authenticating users, manipulating images, and sending e-mail. The sandbox in which the Web application runs restricts access to the underlying operating system. Although App Engine limits the functionality available to your application, it supports the construction of useful Web services. Check out Resources for more information.

Note: Deploying applications in App Engine is free within certain bandwidth and storage constraints. To build production Web sites with App Engine, usage fees are assessed.

Another example of a PaaS is 10gen, which is both a cloud platform and a downloadable open source package for creating your own private cloud. A software stack similar to App Engine, 10gen provides similar functionality to App Engine—with certain differences. With 10gen, you can develop applications in Python as well as the JavaScript and Ruby programming languages. The platform also uses the sandbox concept to isolate applications and provide a reliable environment over a large number of computers (built, of course, on Linux) using their own application server.

Infrastructure-as-a-Service

IaaS is the delivery of computer infrastructure as a service. This layer differs from PaaS in that the virtual hardware is provided without a software stack. Instead, the consumer provides a VM image that is invoked on one or more virtualized servers. IaaS is the rawest form of computing as a service (outside of access to the physical infrastructure). The most well-known commercial IaaS provider is Amazon Elastic Compute Cloud (EC2). In EC2, you can specify a particular VM (operating system and application set), and then deploy your applications on it or provide your own VM image to execute on the servers. You’re then billed simply for compute time, storage, and network bandwidth.

The Eucalyptus project (Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems) is an open source implementation of Amazon EC2 that is interface-compatible with the commercial service. Like EC2, Eucalyptus relies on Linux with Xen for operating system virtualization. Eucalyptus was developed at the University of California, Santa Barbara, for the purpose of cloud computing research. You can download it from the university’s Web site (see Resources), or you can experiment with it via the Eucalyptus Public Cloud with certain restrictions.

Another EC2 style of IaaS is the Enomalism cloud computing platform. Enomalism is an open source project that provides a cloud computing framework with functionality similar to EC2. Enomalism is based on Linux, with support for both Xen and the Kernel Virtual Machine (KVM). But unlike other pure IaaS solutions, Enomalism provides a software stack based on the TurboGears Web application framework and Python.