How We’re Managing Private Cloud Infrastructure

A lot of the hard parts of vRealize Automation is just deciding on what technology stack(s) to implement first and how best to support automated builds and deploys.  VMWare likes to make a big deal about Infrastructure as a Service (IaaS) and Software Defined Data Center (SDDC) – but they don’t actually include with vRA all you need for SDDC, and IaaS is only marginally useful without an application deployment framework.

Many of you who know me in person know that while I’m a big technology geek and I love getting my hands dirty with the details, there’s also a side of me that’s very much strategy focussed and always thinking about the big picture; how will the software I’m writing or working with today be used in a year, in three years, in five years?  I thought it’d be fun (and useful) to take a spin through some of the strategy decisions we’ve made or are considering in regards to our private cloud infrastructure.

Goals For Private Cloud

It sounds like a truism, but you can’t develop an effective strategy on how to get from where you are to where you’re going unless you know where you’re going.  The first step of making anything better is agreeing on what definition of better you’re using.  Our team started by talking about what our end goals for Private Cloud were, and we arrived at the following loose list.  We may or may not deliver on every item below, time will tell, but in a perfect world, at Inmar, this is what Private Cloud looks like:

  • Support Continuous Integration / Continuous Deployment through ephemeral deploy
    • “The code is the app” – nothing other than application deploy required for functionality
    • All persistance in database
    • All configuration through automation
  • Decrease time to deliver
    • Current builds take days/weeks – most of that time waiting in a support queue
    • Basic OS in minutes
    • Application stack in hours
    • No approval workflows to slow down delivery of dev or QA resources
  • Centralized Management
    • As few management tools / interfaces as necessary
    • The bulk of our time in one tool (this has turned out to be two: vRO for Infrastructure and Chef for Applications)
  • Interoperability with public cloud technologies where it makes sense
  • Support all the platforms and runtimes that developers here use
    • Future proof by considering all possible runtimes as potential needs
    • Windows, Linux, Mac at a minimum
    • OS/400 where it makes sense
  • Decrease cost of running VMs
  • Manage all aspects of VM lifecycle automatically

Application Stack Provisioning

We discovered very quickly in our private cloud automation that we would need a consistent way to provision application stacks.  This is where we start mapping strategies to tactics.  We had some pretty common requirements for this framework, most of them directly supporting some of the overall goals (above):

  • Support a wide variety of platforms
  • Use similar mechanisms across platforms
  • Central management (preferably on-premise)
  • Low or no cost per VM
  • Interoperable across the cloud technologies currently in use
    • AWS
    • Azure

We looked at a variety of solutions but pretty quickly honed in on three possibilities; Chef, Puppet, and Powershell DSC.  We’re still keeping an eye on Powershell DSC but knew it wasn’t going to be the entire answer for us for two reasons; first off, it doesn’t actually provide a management and provisioning framework for the DSC scripts you create, meaning we’d still have to build something on top of it… and secondly because it doesn’t support Linux.

That left us with Chef and Puppet.  Honestly, they’re both good systems and they both have pros and cons… the bottom line for us really boiled down to three points:

  • With our ephemeral deploy end goal we needed initial provisioning more than we needed to reconfigure servers on the fly.  Both systems support both but the support for re-configuring was stronger in Puppet
  • We could install and manage as many VMs as needed with Open Source Chef on-premise
  • Our existing AWS infrastructure uses Chef

So, ultimately, we chose Chef.  FYI, a lot of our Windows recipes still drop down to Powershell for things that are difficult or impossible in Chef, even with the Windows LWRP, and that’s why we’re keeping an eye on Powershell DSC – our model is to use Powershell DSC for anything not natively doable in Chef, but use Chef Cookbooks to store and provision out the actual DSC script, the piece I noted Powershell DSC as missing above.

Phase One: vCAC 6.0

This is actually my second in a series of posts about what we’re doing different and better now that vRA 6.2 with full web service REST support is out.  Showing you the end goal here won’t communicate quite how much we’re benefiting, so I thought we’d start with our 6.0 picture:


Some notes about the above:

  • Notice we’re not doing a lot of work in vCAC / vCO – it’s job is just to provide the basic infrastructure
  • Chef actually configures and customizes the VM into an application stack server for us
  • TFS deploys code to the VM
    • We customize the build parameters when we queue it, and that’s good enough for most everybody
    • For ones that need more customization, we write configuration specific details to an XML file
    • T4 picks up the details at compile time and transforms code files before they are compiled
  • Notice the out of band arrow “Configuration Process Specific to Application” from an End User to the Cloud Services Team?  Very critical to our success, this is one of the ways we’re closing the automation loop – by inviting feedback from our user community and partnering with them to make our solution more valuable to them
  • There is no place for AppD in the above diagram
    • Its a steaming pile
      • Didn’t support our complicated AD configuration when we tried it in November
      • This is the kind of value-add a vendor only gets one try on; we cannot now afford to go back and re-evaluate it, even if our sales guy were to come to our door tomorrow and say he guarantees it’ll work and we’ll get value out of it
      • He’s not saying that
    • We can use the Web Service to build multiple-machine deployments
    • All of the ways we configure and deploy to a server that are VM-specific also include Advanced Services that ask for the VM name as part of their parameters (ex: we can ask for a TFS deploy as part of the server build, or as a separate advanced service.  The AS requests VM Name as well as the parameters you would specify on the server build)
  • We’re talking to the vCO web service, not vCAC here

Phase Two: vRA 6.2

So what does having the web service on the vRA side of things do for us?  The biggest thing is we’ve moved our single point of contact out of vRO and into vRA.  On the surface this seems like a small change, but under the covers it means we can guarantee a consistent experience to our users regardless of whether they use the UI or API to communicate with us, and whatever level of experience with automation technologies, they can get the exact same end result as they would see on the Web UI.



That’s it for today!  Post a comment, what are you using to close the automation loop here?  Got some crazy puppet infrastructure?  Doing everything with bash/batch scripts stored in vRO and scp/xcopy?  Bite the bullet and go full on Powershell DSC?  I and my readers would love to know!

As always, follow @merlinjim on twitter to be notified when I write new articles!


I've been automating everything I can get my hands on since I was a wee lad, these days its mostly Office, UC4, or VMWare - but I have a strong interest in AI, microfluidics, and 3D printing when I'm not slaving for "da man"

Tagged with: , , , ,
Posted in vRA
3 comments on “How We’re Managing Private Cloud Infrastructure
  1. […] I’m sharing our Chef / vRO pattern – my previous article on our strategy and why we’re using Chef got a lot of traffic, so now that I’ve wet […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Head Shot of Jim

Jim McCracken

Enter your email address to follow this blog and receive notifications of new posts by email.

Follow me on Twitter
Past Posts
Automate The Cloud pages
%d bloggers like this: