As Enterprises integrate DevOps into more of their development lifecycles they start to bump up against some of the practicalities of managing data. A major tenet of DevOps is being able to ship code quicker to give you that edge against your competitors. It may be fast to write code and a continuous integration pipeline and continuous deployment capability allows you to take that new code, test it and push it out to production in an automated and repeatable fashion.
DevOps and Data
Data however is often one of the speed bumps that causes all this fancy CI/CD to slow to a crawl. If your developers need to test their small change against a large chunk of data you need to somehow have access to this data. Creating copies of databases or files is usually slow and inefficient, a time consuming process that negates most of the speedy DevOps cleverness you’ve done for your code writing.
I’ve worked on numerous projects where a robocopy/rsync was run weekly over the weekend to refresh the 100s of GBs UAT and DEV environment from production data, taking in effect three copies of production. This could only run at the weekend due to the size of the transfer and the impact on the underlying storage and network. One solution had to have the database down during the copy which meant the production one couldn’t even be used for a few hours over the weekend while the copy happened. Put that in your DevOps pipeline and smoke it!
Some storage vendors are able to work around some of the speed problem by being able to mount snapshots but Actifio has a very interesting, slick and more comprehensive solution. Actifio presented at a recent Tech Field Day 11 event.
The DevOps capabilities of Actifio are part of a far bigger solution which they call Copy Data Virtualisation. I previewed the solution in my pre-event post: Tech Field Day 11 Preview: Actifio
Basically you can create multiple copies of data very quickly without creating as many physical copies of the data. These copies can be used for multiple things, backups, analytics, compliance, forensics, DR, migrations etc. as well as DevOps.
CloudPhysics is a SaaS based solution for sucking up all your on-premises vSphere metadata into its own data lake and performing any number of analytics crunching on it.
The Cloud Physics offering is built upon a system of displaying cards where you can correlate configuration and/or performance information to show you for example datastore utilisation or iSCSI LUNs.
One of the interesting aspects of CloudPhysics is how they can actively monitor the bloggosphere to crowd-source knowledge to help its customers. There are a whole bunch of built in cards which customers can use to report on their environments but something I didn’t realise was that CloudPhysics can also monitor blogs for issues plaguing vSphere environments. If the investigation involves gathering data from your vSphere deployment, CloudPhysics likely has that data already.
At its recent Tech Field Day 11 presentation, CloudPhysics showed how information from fellow delegate Andreas Lesslhumer’s blog which was about tracking down whether a vSphere Changed Block Tracking (CBT) bug which breaks backups affected you. CloudPhysics was able to code the information Andreas wrote about into a new card which customers could then use to report on their own infrastructure, so much easier than writing the code to gather the information yourself.
This could be even more important if you are not even aware of the bug. CloudPhysics or even any user can scan the VMware Knowledge Base as well as many other blogs and write a card to tell you for example that with the exact version of vSphere you are running on some or all of your hosts whether an issue affects you. Of course this wouldn’t apply to you if you were continually scanning all the official and community sites for all bugs reported and able to report on them! Thought you weren’t, well CloudPhysics may have your back.
I would have loved to have had this a few years ago when I had spent ages correlating vSphere versions with HP/Broadcom/Emulex Nic card drivers and firmware to track down the too many issues that plagued the HP Virtual Connect blade chassis networking at the time. I wrote a PowerCLI script which invoked Putty and SSH to connect to each ESXi host to gather the firmware version so I could check the support matrix, it was time consuming and cumbersome. CloudPhysics would have made this so much easier. I could have used the Developer Edition to create my own cards so much quicker and then this could have been made available to others by publishing it to the Card Store.