8 - 9 March 2016

Brussels, Belgium

04/03/2015

Towards Common Metrics for SLAs

Bob Bohn
Focus Area

When considering the life cycle of cloud services, the procurement tends not to be a straightforward exercise. First, one needs to develop the technical and business requirements of the service. With requirements in hand, selecting services can only begin after comparing them in a reliable and repeatable manner. By employing a common vocabulary for cloud computing, communication between customer and provider is able to be more precise and meaningful. The comparisons themselves are based on the properties of the cloud services and the metrics to be used should relate the properties back to the original set of requirements. This information is used in a Service Level Agreement (SLA) between the provider and customer. The provider will fulfill the Service Level Objectives (SLOs) of the customer at a certain measured level as stated in the SLA. The customer can use their metrics to monitor the performance of the provider and react or make decisions on the service based on the monitoring results. Using a standardized set of metrics, templates or method for determining cloud service metrics makes it easier and quicker to define SLAs and SLOs, and to compare them with others.

Market sector targets

The target audience for this consists of cloud customers, cloud providers, cloud auditors and those involved in the procurement of cloud services at all levels. The importance for metrics that can be used in cloud computing cannot be understated. Developing metrics that are reliable, repeatable and measureable are timely considering the continued growth in cloud computing and market forces. Ultimately, these metrics will result in cloud computing being bought & sold in a confident and trustworthy manner that will add to additional growth. Reliable & trusted cloud metrics give a cloud provider additional marketing and business tools which allow them to set themselves apart from the competition.

Addressing key concerns impeding the mainstream adoption of the cloud: privacy, security, trust

According to the NIST definition1 - Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This definition is a good starting point for those who are interested in understanding what cloud computing is. However, this model is not enough to purchase or procure clouds services for an enterprise. In October 2014, NIST published a two-volume USG Cloud Computing Technology Roadmap (NIST SP 500-293) which identifies the top level requirements in security, interoperability and portability for a quick, secure and successful migration to the cloud2. Three principle requirements that are important for this discussion are:

Requirement 3: Technical specifications to enable development of consistent, high-quality Service-Level Agreements

Requirement 4: Clearly and consistently categorized cloud services

Requirement 10: Defined and implemented cloud service metrics

When considered together, these requirements point out a clear pathway for customers and procurement officials to embark on purchasing cloud services.

To undertake an assessment of available cloud services requires a type of normalization of the services in which they can be compared against each other. This comparison depends upon two things:

  • An ability to discuss the concepts in cloud computing using a common vocabulary.
  • An ability to decide upon and develop relevant service metrics that reflect the organizational requirements.
Related to “safe and fair contracts”: cloud law, service level agreements (SLAs)

The use of a common vocabulary to understand and communicate the concepts that underpin cloud computing is a crucial aspect in order to make comparisons. In addition, this same vocabulary gives a customer the necessary tools to discuss a framework for metrics with the provider, the capabilities of the services and language that can be used in contractual agreements like the Service Level Agreement (SLAs). For example, the simple terms “availability” and “response time” may hold different meanings amongst cloud providers because they start with their specific definitions. One provider may describe availability as being 99.5% available for use 24x7 whereas another provider may define it as 99.5% available for use from 9am-5pm. These are distinctly different and a customer should be alert. In many situations, the availability to the customer does not take into account service downtime or service maintenance. Using a consistent method to compare the cloud services satisfies Requirement 4.

The development of reliable metrics for cloud services rely on translating your business and technical requirements into a form that can be measured in a repeatable fashion. For example, a possible metric for an email service might be related to its ability to filter spam, detect viruses or malware. If one is considering purchasing for a large organization, the time it takes to update mail servers or the number of times a message will be resent in case it is deemed undeliverable could be important. NIST has recently released a draft document entitled Cloud Service Metrics Description (NIST SP 500-307) which discusses a method for describing cloud service metrics. Employing a method to develop cloud service metrics is a useful exercise and helps to fulfill Requirement 10. The figure (Figure 1) below shows how a customer uses metrics in order to make a decision when selecting a provider. 


Figure 1: Using Cloud Service Metrics to Decide between Providers A and B.

When used together, the customer has some important tools to assist in the procurement process. By employing the common vocabulary and a description of the metrics, one can express these technical specifications in the SLA. This document will be used in the managing and monitoring of the services and their levels from the cloud provider. The following figure (Figure 2) gives insight into how the customer uses metrics with SLA as a verification method. 

Figure 2: Verification and Monitoring of Cloud Services using Metrics and SLAs.

With results in hand on the service levels through the monitoring process, the customer has the power to make additional decisions such as purchasing more services, deciding on less or to take action on a service that does not meet its agreed upon level.

Is there a common vision and goal for international dialogue on the cloud, IoT or big data?

There is a common vision and goal for international dialogue on cloud computing. There have been many advances in cloud computing standards. The most relevant are the joint standards produced by ISO and the ITU-T. In August 2014, ISO standards 17788 (Cloud Vocabulary) and 17789 (Cloud Computing Reference Architecture) were released. In addition, ISO is also working standards for SLAs – 19086. This is a 4 part standard that will focus on concepts, metrics, requirements and security.

Links and references

1. NIST Definition of Cloud Computing, NIST SP 800-145 http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf

 

2. US Government Cloud Computing Technology Roadmap Volume I: High-Priority Requirements to Further USG Agency Cloud Computing Adoption; and Volume II: Useful Information for Cloud Adopters, NIST SP 500-293

http://dx.doi.org/10.6028/NIST.SP.500-293

 

3. Cloud Service Metrics Description (draft), NIST SP 500-307 http://www.nist.gov/itl/cloud/upload/RATAX-CloudServiceMetricsDescription-DRAFT-20141111.pdf

Authors:

Robert B. Bohn, Ph.D., NIST Cloud Computing Program, NIST, Gaithersburg, MD