January 22, 2017

Hybrid Cloud and your applications lifecycle: 7 lessons learned


Hybrid Cloud is a must nowadays, I will not spend a word to convince you (you’d not be reading this post if you didn’t believe it). This is the story of a real project.

This post provides more context about the story I summarized at Just 1 step to deploy your applications in the cloud(s).
The structure of the post is:
  • Motivation
  • Use Cases
  • Time
  • Software Stack
  • Benefit of the architecture we implemented
  • Lessons Learned (the most important part)




Motivation for hybrid cloud, and most of the work in my customers' projects, include the following areas:
- Cost control (there is a strong debate: some swear it’s cheaper, others have discovered hidden costs: e.g. network traffic in production, after they made a business case just on the cost of VM provisioning).
- Governance model (IT must find a way to maintain control over resources usage, design patterns, compliance and security when application developers chose private cloud or public cloud).
- Mature technical solution: architecture and technology (there are many good products and system integrators in the market)

But, once you have made a decision, what will you run in the hybrid cloud?
Will your applications be spread across the boundary of your datacenter (one tier inside, other tiers outside)?
Or can we say that it is rather a multi-cloud deployment, where you have a number of resource pools that you can use as a target for deployments?

This project was made by a large corporation, to test how a hybrid cloud can be built and operated and to verify the impact on their current organization.
It is not a full production environment, it’s a pilot project that demonstrated on a small scale how easily you can build a software defined fully automated data center, including both resource pools from your local data center(s) and from public cloud providers.

The solution is expected to be cost effective, of course, but the greater benefits come from business agility and consistent governance.


Use Cases:

The evaluation was focused on 3 main use cases, all requiring that end users order the deployment of a complex software stack from a service catalog: the target for the deployment can be either the private cloud or the public cloud, or a combination of the two. These are the areas where the implementation demonstrated the value of the multi-cloud solution:
  • Business Intelligence (self-deployment of R Studio and additional tools)
  • ETL (self-deployment of a common software for ETL that data scientists would use in autonomy)
  • High Performance Computing (HPC) on OpenStack, with the integration of a DevOps pipeline.

Subject matter experts were provided from different lines of business in the company to support the implementation activities and evaluate the result. 
The use cases represent some frequent activities that the company needs in their usual business, especially in R&D. Improving efficiency and quality in the associated processes will have an impact on the overall business outcome. Applications were selected for the self service catalog that are deployed frequently (every week) and whose installation process takes time (some man days, accounting for both infrastructure and software setup), delaying business objectives.

Time:

All the activities in the project were delivered in time (six weeks), including the setup of the hardware and software systems for the hybrid cloud, the implementation of the 3 main use cases and some additional use cases, the functional tests and the stress tests. This is a demonstration that a proper selection of the technology and a good organization of the project allow for immediate return.
Challenges like setup of the remote access to the lab for remote experts, constraints in the networking and security configuration in the lab, some missing information about the process to install the applications (essential to build the model for the automation) slowed down the implementation. See Lessons learned.

Software Stack:

This is a complete end to end solution: its adoption will happen with a phased approach, starting from the components that grant an easy and immediate impact on the most critical business requirements and adopting some non-functional components later to complete the architecture. The extension from private cloud (based on any combination of VMware, other hypervisors and OpenStack) to a hybrid cloud (integrating AWS, Azure and more) was very quick (it is just a matter of configuration and definition of the governance model). Checkmarks in the picture show what we realized in the short timeframe of the project. The rest is part of a phased plan. The blue boxes show the components provided by Cisco.


a full solution for the hybrid cloud

The fundamental component in this architecture is Cisco CloudCenter (CCC), that has 2 main roles: 
- providing an orchestration solution that offers users the possibility to self-deploy complex software stacks from blueprints offered in a catalog, 
- brokering cloud resources from both private and public clouds (in the project we integrated VMware, OpenStack and AWS, but more clouds are supported).
CloudCenter manages the lifecycle of software applications in the cloud (at a level of abstraction where the underlying physical infrastructure does not matter).
The OpenStack use cases for HPC are supported by a Cisco Validated Design named UCSO: it includes a reference architecture for running the Red Hat OSP8 distribution on a certified hardware platform made of Cisco UCS servers and Nexus 9000 switches. The setup process and the operations are defined by the official deployment guide and Cisco's technical support assumes responsibility on the entire stack, including the Red Hat software.
The management of the entire DC infrastructure from a single orchestration platform was made possible by Cisco UCSD (UCS Director): a single dashboard and workflow engine to manage servers, network and storage, both physical and virtual. The status, the performances and the remaining capacity of all the systems were monitored with Cisco UCSPM (UCS Performance Manager).


Benefits of the architecture we implemented

The implementation of the multi-cloud solution demonstrated the major benefits that a hybrid cloud delivers.
  • A consistent architecture based on software (and eventually hardware) components that integrate easily and satisfy all the business and technical requirements.
  • All components in the architecture are loosely coupled and their integration is based on standard protocols and documented open API. As a consequence, every component can be replaced by an alternative solution (from a different vendor, from the open source, from a custom build) with no fear of vendor lock in.
  • The adoption of a hybrid cloud solution can happen gradually, starting with a core implementation with the most critical components (e.g. CCC, ACI and UCSO), adding more features as a second step (infrastructure automation and monitoring) with UCSD and UCSPM, eventually a unified service catalog and ITSM portal later.

Lessons Learned:

  1. use cases
  2. network topology
  3. security and trust
  4. reusable work (repositories and services)
  5. engage SME and business owners
  6. document
  7. refine (iterations, devops)

Use Cases 
The selection of the use cases is important. You need a quick return to demonstrate the value of the hybrid cloud: the adoption of the hybrid model should address immediate business needs, that the end users can appreciate, rather than be driven just by an industry trend. 
IT projects should not start because a new technology is very smart, but because the outcome makes the business easier and more productive.
Always engage your end users in the planning phase and avoid academic use cases that have a limited appeal on the decision makers. In this project we were lucky because the preparation was done by the steering committee very well in advance.
Once the models for the automation were ready, we could test any combination of the deployment for the application tiers: everything in the private cloud, everything in the public cloud, or the front end deployed on one side and the back end on the other side. The benchmarking capabilities of the product (CCC) allowed to compare the price/performances ratio of the different options based on vSphere, Openstack and AWS - specifically for each application, with tailored reporting.
 
Network Topology
A hybrid environment connects - by definition - areas that were designed separately (your datacenter and the public cloud). They have security policies and configurations that are not meant to work together, and this makes it difficult. Before you start the setup, dedicate the right time to collect all the requirements and to design the connectivity properly. 
We had some issues with the network proxies and the firewalls because of the protocols and ports that we needed to open to allow a proper integration of the Cloud Management Platform (running on premise) with the orchestration engine (with one instance running in each cloud region used in the project, to leverage the local API exposed by the cloud provider and to manage the lifecycle of the applications in the cloud). 

communication among the components of Cisco CloudCenter

Another important requirement is to have a unique repository for all the artifacts, the blueprints and the installation packages for the applications: it should be reachable from all the target clouds that you plan to use, regardless its location (it can be either in the private or in the public cloud, but all the servers you deploy will access it to stand up a new instance of the application). 
The same applies to any public repository that is used in the setup of the applications (both commercial software and open source components, e.g. packages installed using yum).
See also CCC Components Overview for more detail.

Security and Trust
It's important that a good level of trust is established between the architects building the hybrid cloud and the operations team, especially the security guys. Special rules and new policies need to be setup to allow the new platform to work, it's impossible to keep the same old governance model that addresses a single end user identity. 
Sometimes I feel like I'm living - again - the same conflict that I had with Database Administrators, when I tried to configure JDBC database connection pools in the first Java application servers in the 80's. The system should be trusted, and a delegation of the decisions (authentication, authorization and audit) accepted.

Reusable work (repositories and services)
When you model a software application to automate its deployment, you should identify any building block that can potentially be reused in a different model. If you create a reusable (parametric) deliverable and save it individually in a common repository, next time you'll have the work ready to be reused.
This applies to architectural building blocks like database servers, web servers, load balancers, firewalls, distributed caches, etc. 
If they have been created as separate services, instead of just being a part of a monolithic model, they will appear in your designer's palette everytime you model a similar application and you can drag and drop them in the topology. We did that in the project and we saved a lot of time in the implementation. 

Engage SME and business owners
It is important that subject matter experts (SME) collaborate at the definition of blueprints and the build of the automation model. Even though documentation exists for the deployment of the application, you should work together. 
The user knows all the requirements, he knows how to verify and troubleshoot, he has encountered all the setup issues already.
I've learned that the best way to document the setup process for an application, so that you can use it as a reference for the automation, is to ask the SME to install it in front of you in a clean environment where the application was never run, and record a video of the process. It's faster than writing documentation, more complete and reliable. We did that using the desktop sharing feature in Webex and we recorded the sessions.  

Document
While you do the work, keep track of all the steps. Take (maybe informal) notes, but mostly take a lot of screen shots to document what you did. You can keep them on a wiki or on a shared folder, they will help a lot when you have time to create the formal documentation of the project. If you need to troubleshoot, eventually involving other people, this information will be unvaluable.
Of course, versioning and taking snapshots of all deliverables also helps in case you need to go back for whatever reason. 

Refine (iterations, devops)
Create the implementation for a minimum viable product (MVP) as soon as you can. Get the product (i.e. the entire self service catalog, or just the implementation of a single application blueprint) to early customers as soon as possible, to get their feedback before you go too deep in the implementation.
Applied to a hybrid cloud scenario, this will help to evaluate:
- quality of the service you are building, including documentation
- how much the users need it and use it in the real world
- performances of the distributed environment and any bottleneck (network, computing, configuration)
- security implications 
You will have all the time to make it perfect, through iterations that improve the implementation, collect feedback, allow for tuning the design and the configuration. No need to work in a hurry and make mistakes, while you keep your users waiting for the final "perfect" product but they don't see any progress.

October 12, 2016

Just 1 step to deploy your applications in the cloud(s)


As described in my previous post about Terraform, the deployment environment for a new application can be created "on demand" by configuring physical and virtual resources.
Good open source products allow to describe the desired state and to automate the setup of a target infrastructure.
They can also deploy your software application and configure it properly.

But is some use cases this is not enough.
You might want to offer your users - depending on their needs and their skills - a visual catalog in a web portal.
You might want to apply a governance model based on policies, use different clouds as possible targets for the deployment, offer a easy way to manage the life cycle of the deployment (start, stop, scale up/down, terminate) and get reports on usage of the resources.

If this is the case, there are good solutions available.
One of these is Cisco Cloud Center, a powerful tool that offers two main use cases: 
  • modeling the deployment of a software stack (creating a template or blueprint for common deployments) and 
  • brokering cloud services (different resource pools available from a single catalog).

A easy to consume (and manage) self service catalog
A easy to consume (and manage) self service catalog


Open Source or commercial products?


In the same project where I used Terraform to deploy Apache on Openstack, I also used Cisco Cloud Center to deploy a portal application on Openstack.
But at the same time, I offered the possibility to target the same deployment to a public cloud (AWS in this particular case) or to the private cloud (choosing between Openstack and vmware in this particular case). No duplication of the effort was needed, because the model you creat is not referred to a specific cloud as a target. It will be matched, when a user orders it, with one of the cloud avaliable for him or for his project.
So I was able to show the difference between a free, open source solution (Terraform) and a commercial product (Cloud Center) in a similar scenario.

The second option addresses different needs of the organization and offers a richer solution.
It’s up to you to evaluate which one fits your requirements better. 


Modeling, policies and multitenancy


One of the differences is that Cloud Center offers a graphical editor to model the topology and the dependencies among all the building blocks of your deployment.
You have a library of services (software applications from a repository, physical and virtual services like load balancers and firewalls).
Services can be dragged and dropped in the editor, then you set their properties and dependencies. 
The architecture of the application you're modeling can be based on a single server or a number of servers with different roles.
If the application architecture has multiple tiers, every tier gets its own attributes and policies: as an example, you can set the minimum and maximum number of instances in a cluster of web servers (or application servers or database servers). 
Autoscaling policies will tell the orchestrator to increase or decrease the number of servers based on metrics like consumption of cpu or memory, inbound/outbound traffic, etc.
Everytime the cluster changes, the orchestrator will modify the configuration of load balancers and firewalls accordingly: no manual intervention is needed.
Models are saved in the catalog and offered to users in a multitenant organization: every tenant is given a portion of resources (target cloud environments) and services (models available in the catalog to deploy applications) that the tenant administrator can offer to his own users and groups... and sub tenants. Every tenant cannot see other tenants' stuff.

a graphical editor to model the deployment of your applications
A graphical editor to model blueprints for application deployment in hybrid cloud

Dashbooard and Reporting


Every user has a dashboard that shows the consolidated information about all the applications he has deployed (or the other users in the same tenant), and can manage the lifecycle of all the deployments.

Of course the administrator of the system sees the global view including all the assets.
Active VM per cloud and per application are shown in the dashboard, as well as associated costs.



a unified dashboard for all your deployments in all the clouds
Cloud Center's Dashboard


A powerful reporting features allows to filter deployments and costs by user or group, application, environment and cloud.
Data can also be exported in different formats, to be consumed by humans and other systems.


powerful reporting allows for governance, showback and chargeback
Unified reporting



Architecture

The architecture of the Cloud Center product is based on two Virtual Machines: the Manager (CCM) and the Orchestrator (CCO).
The Manager is the engine where policies and application models are defined, and where the user portal runs. The Orchestrator lives within each of the target clouds (indeed, there is one CCO in each cloud region), receives commands from the Manager and executes them locally using the API of the cloud platform.
Cisco provides orchestrator images that are specialized for every cloud supported by Cloud Center.  So you have a single place to manage all your cloud resources, and a single model to maintain: you don't need a model, or a workflow, or a script for every target cloud where the syntax of that specific API is used. You create a single model, that is completely decoupled from the target of the deployment: this reduces the amount of work (a single model instead of many) and makes the maintenance of the model easier and more consistent (you don't have to evolve many models for the same application).

 
One manager drives a separate orchestrater for every cloud region you have access to
Cisco Cloud Center architecture


Comparison


Two solutions for the same use case, one for free and one at a cost?
Indeed they address different requirements: as described above, Cloud Center is for enterprise organizations that need to rationalize their usage of cloud resources. It is used by the corporate IT to provide flexibiliy and agility to their developers (within a governance model), to standardize the architecture of their projects based on blueprints (including what products, what versions, what setup configuration they prefer) and to get reports on consumption.

Service providers can use Cloud Center to broker third parties' resources, offering a single catalog to their customers. The hierarchical multi tenant organization and the sophisticated cost models that can be offered make it simple.

I suggest you to consider it if you are using, or plan to use, two or more cloud providers (counting also your private cloud or your virtualized data center). You will see an immediate benefit in terms of compliance and efficiency.

References




September 19, 2016

Infrastructure as Code with Terraform and Cisco Metapod


Recently I worked with a customer to explore the concept of Infrastructure as Code.
They like open source solutions for the automation of the infrastructure and for managing the software applications life cycle.
To reach the first objective their goal is a private cloud based on Openstack, while they will use Ansible and Terraform to manage the environments for different projects.

Managing the Infrastructure as Code means that the definition of the infrastructure is maintained in text files, that could be stored in a version control system like you do with the source code of the application.
If you do that, the same lifecycle applies to both the infrastructure and the application: creation of staging or production environments, automated testing, etc.
Using blueprints helps to improve the quality of the final result of the project and grants compliance with policies and eventual legal obligations.
Benefits include speed, cost savings (avoiding a static allocation of pre-provisioned resources) and risk reduction (removing errors and security violations).
Terraform is one of the best open source tools to manage your Infrastructure as Code: it’s easy to install, learn and use (one hour).
You could start from tutorials and free examples available on Internet.

Here is an example of full automation (we'll try to get a little better result):



As a first step, to make the usage of Openstack easier on a large scale, we discussed the value of a managed service.
If the IT organization could just focus their effort on the development and operations of the business applications, instead of running the infrastructure, they would create more value for the internal customers (company's lines of business).
So I proposed the adoption of Cisco Metapod, that is Openstack as a managed service (delegation of all the tough administrative and operational work to a specialized 3rd party, while you just use the Openstack user interface and API enjoying a SLA of 99.99% uptime). 
I have described the advantage of adopting Openstack as a managed service in this post: Why don't you try Openstack (without getting your hands dirty)? 
Services offered by Cisco Metapod around Openstack
Services offered by Cisco Metapod

We created a lab where Openstack abstracts the resources from the physical and virtual infrastructure (etherogeneous servers, network and storage) and the configuration of different environments is managed by Terraform, so that you can create, destroy, restore and update a complex system in few minutes.


Free to use Openstack for your apps, instead of managing it: focus on your real business

With Terraform you can describe the architecture in a declarative form (in a manifest file).
You simply describe what you need (the desired state), not how the different components (devices and software) must be configured with all their parameters and their specific syntax.
The goal of Terraform is to match the current state of the system with the desired state.


Desired State vs Current State


Terraform is used to create, manage, and manipulate infrastructure resources. Examples of resources include physical machines, VMs, network switches, containers, etc. Almost any infrastructure noun can be represented as a resource in Terraform. Terraform is agnostic to the underlying platforms by supporting providers. A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Atlas, DNSimple, CloudFlare). 

Infrastructure as Code
Infrastructure as Code




Desired State

In my lab I reused a good example that I found on GitHub at https://github.com/berendt/terraform-configurations 
It contains all the resources you need to deploy a new Devstack instance (a all-in-one instance of Openstack, useful for developers) including the needed networks, public addresses, firewall rules on a target cloud platform. That, incidentally, is a Openstack instance (so we are deploying Openstack on Openstack).

Here is the content of the main.tf file used by Terraform: it references variables with the format ${variable_name}, including the output from actions on other resources. Dependencies between resources are managed automatically by Terraform. A separate file can contain the predefined values for your variables (like the references to the Openstack lab in my example).

If you are not interested in the content of this file (I guess it applies to 70% of my readers) you can skip it and go to next picture... there is also a good recorded demo down there   :-)

 

main.tf (the manifest file where Terraform ready the desired state of all the resources):

provider "openstack" {
  user_name  = "${var.user_name}"
  tenant_name = "${var.tenant_name}"
  password  = "${var.password}"
  auth_url  = "${var.auth_url}"
}

resource "openstack_networking_network_v2" "terraform" {
  name = "terraform"
  region = "${var.region}"
  admin_state_up = "true"
}

resource "openstack_compute_keypair_v2" "terraform" {
  name = "SSH keypair for Terraform instances"
  region = "${var.region}"
  public_key = "${file("${var.ssh_key_file}.pub")}"
}

resource "openstack_networking_subnet_v2" "terraform" {
  name = "terraform"
  region = "${var.region}"
  network_id = "${openstack_networking_network_v2.terraform.id}"
  cidr = "192.168.50.0/24"
  ip_version = 4
  enable_dhcp = "true"
  dns_nameservers = ["208.67.222.222","208.67.220.220"]
}

resource "openstack_networking_router_v2" "terraform" {
  name = "terraform"
  region = "${var.region}"
  admin_state_up = "true"
  external_gateway = "${var.external_gateway}"
}

resource "openstack_networking_router_interface_v2" "terraform" {
  region = "${var.region}"
  router_id = "${openstack_networking_router_v2.terraform.id}"
  subnet_id = "${openstack_networking_subnet_v2.terraform.id}"
}

resource "openstack_compute_secgroup_v2" "terraform" {
  name = "terraform"
  region = "${var.region}"
  description = "Security group for the Terraform instances"
  rule {
    from_port = 1
    to_port = 65535
    ip_protocol = "tcp"
    cidr = "0.0.0.0/0"
  }
  rule {
    from_port = 1
    to_port = 65535
    ip_protocol = "udp"
    cidr = "0.0.0.0/0"
  }
  rule {
    ip_protocol = "icmp"
    from_port = "-1"
    to_port = "-1"
    cidr = "0.0.0.0/0"
  }
}

resource "openstack_compute_instance_v2" "terraform" {
  name = "terraform"
  region = "${var.region}"
  image_name = "${var.image}"
  flavor_name = "${var.flavor}"
  key_pair = "${openstack_compute_keypair_v2.terraform.name}"
  security_groups = [ "${openstack_compute_secgroup_v2.terraform.name}" ]
  floating_ip = "184.94.252.189"

  network {
    uuid = "${openstack_networking_network_v2.terraform.id}"
  }

  provisioner "remote-exec" {
    script = "deploy.sh"
    connection {
      user = "${var.ssh_user_name}"
      key_file = "${var.ssh_key_file}"
    }
  }
}


To make it simple, for this blog post I replaced the part that deploys Devstack with a simpler setup of a web server (Apache).

deploy.sh (Terraform will copy and execute it on the remote instance as soon as it is created):

#!/bin/bash

# author: Joe Topjian (@jtopjian)
# source: https://gist.github.com/jtopjian/4ffc82bfcbbcc78d07e4

sudo apt-get update
sudo apt-get install -y -f apache2


The goal is to demonstrate how easy it is to create a new software environment on a Cisco Metapod Openstack target from scratch and run it. 
The following pictures show the Metapod console before and after running the "terraform apply” command on my computer.

This is before I run the command:

The Openstack console from Cisco Metapod: view of the tenant networks
The Openstack console from Cisco Metapod

And this is the expected result (network and server infrastructure created, apache installed):

Terraform has created all the required resources in Openstack
Resources created in Openstack

Next video (the most important part of this post) is a recorded demonstration of the creation of the new Apache server: you can see the launch of the “terraform apply” command that, after reading the manifest file, creates a network, a subnet, a router with two interfaces, a floating ip and a instance on Openstack. Then the Apache web server is downloaded and installed in the new instance.
The Metapod console is left in the background and you see the Openstack objects pop up as they are created.
Finally the home page of the new web server is tested.

 


 


Conclusion

It is very easy to get rid of the delays, the misunderstandings and the inefficiency of many current IT organizations.
If you standardize the process that developers follow to obtain the environment for a new project - in all the phases of the life cycle - you can enable a faster go to market for new business initiatives making your customers happy.
It would be a first step towards DevOps (more is required, mostly in changing the culture of both developers and people in operations).

Infrastructure as code is a brilliant way to create the needed infrastructure on demand (and release it when no longer needed), to maintain it based on blueprints and manage the definition of the infrastructure with the same tools you use for the application source code: a text editor (or your preferred IDE), a version control system, an automation tool.

If you have a IaaS platform like Openstack, provisioning of both virtual and physical resources is made easy.
If you do a further step forward with a managed service, someone will grant that your Openstack is correctly configured for production, up to date and in perfect health. You enjoy all the benefits, without the hassle of setting it up and operating it daily.




References:
Configuration to run acceptance tests for Terraform/OpenStack - https://github.com/berendt/terraform-configurations
Why don't you try Openstack (without getting your hands dirty)? - http://lucarelandini.blogspot.it/2016/06/why-dont-you-try-openstack.html