Revisting ​​Essential

Back in July 2017, I wrote about my 10 essential iOS applications. I thought now that we’re reaching the end of 2018, it might be a good time to revisit that list.

As I mentioned at the time:

I find it helpful to mix things up from time to time, even going as far as doing a reset of my app icon layouts periodically to reshuffle the deck chairs and throw out any old cruft hiding in corners. One of my favorite activities is to delete apps that don’t get used anymore, or used enough to take up my attention.

This time I took the approach of installing only the apps that I know I need on a daily basis, and then filling in the rest as the need for them arise.

A few weeks ago, I took a similar approach but this time with a far more aggressive regimen. I uninstalled nearly every third-party application from my phone. Then I started to analyze the feature/function of every app and determine if the app itself provided something more than just a wrapper around an already functional mobile web site.

I find that having fewer things installed on my devices brings me some joy.

Shopping, banking, social media, travel, news, food, weather, shipping. Almost app categories were fair game. About the only group that was mostly safe were apps that controlled the various smart devices around my house: Nest, Hue, eero, Rachio, Lutron, myQ, etc. 

From there, it was about finding the apps that were the truly essential apps in my workflow:

  • I recently switched to Outlook as my primary email/calendar application. This means I can displace the stock Mail and Calendar apps, as well as remove Fantastical, which was on my 2017 list. Currently, though, I’ve been experimenting with having only my work email in Outlook, and my personal email in the stock app, just for workload isolation. I can’t decide if there is more of less mental friction in keeping them together or keeping them separated.
  • I also have been using Microsoft To-Do as a reminders replacement, mostly because of the Tasks integrations with Outlook on Mac. (However, I’ve been bad recently at actually doing the things in here.) I’ve been comingling work and personal tasks in here. This has replaced Things for the time being.
  • 1Password is simply irreplaceable. You’ll pry it from my cold, dead hands.
  • Then there is Overcast for podcasts. I’ve experimented with alternatives in the last few months from the stock Podcasts app, Pocket Casts, and Castro, and always come home to Overcast for the basic reason that podcasts just sound better in there.
  • And of course, Tweetbot for Twitter. I just can’t quit you.
  • Shortcuts has replaced Workflow after Apple bought them and built much of it into iOS 12.
  • I keep the ads and other trackers away in Safari with Better.
  • I have Parcel setup to automatically track Amazon, FedEx, UPS, and USPS shipments, of which there are many this time of year. (Seriously, the entrance of my house looks like a loading dock right now.)
  • Zoom is a requirement for work.
  • As is Slack.
  • I’d be locked out of both of those without my RSA soft-token.
  • And I love the ability to upload receipts with Concur.
  • While authenticating through Workspace One.
  • AT&T Call Protect has become my new junk filter for phone calls. This replaced Nomorobo from the 2017 list, which is still a fantastic app, but AT&T’s app is network integrated.
  • Finally, despite my new love for Nespresso, I still have a requirement for Starbucks on the go and like to have my order ready as I walk in the door.

From there I started a review with the assumption that I could avoid anything else. Despite quitting Facebook a couple years ago, I’m still on Instagram for close friends and family. I tried for weeks to limit myself to using the web app, in an effort to avoid another app install, but also having it try to entice me to spend more time in it with a dedicated shortcut on my home screen. After trying to limit my usage, I gave in and reinstalled it because it was just too damn hard not to.

I had a similar experience with LinkedIn. The issue there was more around the usability of the website on a mobile device. It was pretty terrible. I’d like to keep this uninstalled but I occasionally end up dropping it back on and then off again. I’ve uninstalled it again recently.

My primary bank has mobile check deposit, and I have family members who for some reason continue to write me checks despite showing the multitude of better ways to transfer money around. So it was a given for reinstall because the only thing worse than writing a check is having to physically go into a bank. So 2007.

I had Microsoft OneDrive installed for accessing work files from my phone, but realized I never used it outside of the native integration of the Outlook app. So, I deleted it. This may return if I find some other reason I was using it.

Target came back, despite my original wipe of shopping applications because of its store card being integrated with the app. The only thing I like more than being able to delete an application from my phone is taking a physical card out of my wallet.

A few other apps that were considered essential in 2017, I’ve since abandoned. Carrot Weather is great, and probably one of the best third-party weather apps on iOS, but I find the stock app to be good enough most of the time. Pcalc is another great app, but I don’t find myself needing to calculate anything so complicated at the stock app can’t get the job done. Cloak was on the list last time and has since been sold and rebranded as encrypt.me. Like many things that get sold, it just doesn’t feel like it has the same level of love and care as the original owners, and so it’s been cut.


L1TF

VMware has published new security advisories, knowledge base articles, updates and tools in response to newly disclosed speculative-execution vulnerabilities on Intel CPUs — collectively as “L1 Terminal Fault” — that can occur on Intel processors made from 2009 to 2018.

I’m going to outline our response to this issue, and make an attempt to summarize this complex event as best as I can. I would highly suggest reading through the linked articles as they’ll be more extensive and evolving.

Because this is complex, and evolving, to properly respond to these issues, consider KB55636 as the centralized source of truth from VMware.

Like the previously known Meltdown, Rogue System Register Read, and “Lazy FP state restore” vulnerabilities, the “L1 Terminal Fault” vulnerability can be exploited when affected Intel microprocessors speculate beyond an unpermitted data access.

L1TF – VMM (CVE-2018-3646 | VMSA-2018-0020)

This is the specific L1TF issue that affects the vSphere/ESXi hypervisor. It has two known attack vectors, both of which need to be mitigated. The first attack vector is mitigated through patches for both vCenter and ESXi.

The second attack vector is mitigated by enabling a new advanced configuration option (hyperthreadingMitigation) included in the updates. However, this advanced configuration option may have a performance impact so we have not be enabled it by default. This will limit operational risk by giving you time to analyze the effects prior to enabling.

There are new updates to both vCenter and ESXi that deliver the mitigation to L1TF:

  • vCenter 6.7.0d, 6.5u2c, 6.0u3h, and 5.5u3j
  • ESXi670-20180840x, ESXi650-20180840x, ESXi600-20180840x, and ESXi550-20180840x

There are also new versions of VMware Workstation (14.1.3) and Fusion (10.1.3) which address this issue.

L1TF – OS (CVE-2018-3620)

This is a local privilege escalation which requires base operating system updates for mitigation. Patches are pending for affected VMware appliances. Make sure you contact your operating system vendor(s) (Microsoft, Oracle, Red Hat, etc) for mitigation instructions in guest virtual machines as well.

L1TF – SGX (CVE-2018-3615)

This does not affect VMware products.

VCIX

I’m pleased to announce that yesterday I passed the VCAP6.5-DCD exam, thus earning the VMware Certified Implementation Engineer – Data Center Virtulization “milestone” after elevating the VCAP5-DCA exam that I earned back in 2014.

The DCD exam has been on my list of things to do since not long after I did the DCA. My first attempt was during the beta cycle for the 6.0 exam. The results for that exam took so long to be returned, and after shifting in job roles since then, I’d not had an oppertunity to sit for it until now. The 6.5 version of the exam differs from the 6.0 in that there are no longer the “Visio” style questions, which I think were problematic for the exam from the beginning. There are 60 questions consisting of multiple-choice, drag-n-drop, and multi-select questions, with 140 minutes to complete the exam. I was able to complete the exam in just under 90 minutes, and I didn’t feel like I was rushing.

In terms of advice I can pass on to others who are interested in taking this exam, make sure that you understand:

  • AMPRS (Availability, Manageability, Performance, Recoverability and Security)
  • RCAR (Requirement, Constraint, Assumption and Risk)
  • The difference between Functional and Non-Functional requirements

If you are hands on with vSphere 6.5, especially working with vCenter HA, PSC/SSO and cluster design, you should have all of the bases covered. I have been removed from much of that in the day-to-day for the last year or two, so that was probably the more challenging part of the exam for me. I think if I’d done more to read up on differences between 5.x/6.0 and 6.5, I’d have come back with a better score. But, pass is pass.

Shine

Once upon a time there was a meeting of minds,
The sun and the moon made a deal with the sky,
One would take the morning and the other the night,
Together they would blanket the world with light,
But the moon had a shadow, he felt like a liar,
The sun was the only one who carried the fire,
The sun saw this, she kept on glowing,
Bound to the moon, never saying, “you owe me”
She said “I’ll shine on you.” Jason Mraz

Who will you shine on today?

VMware TAM

I have accepted a job with VMware, as a Technical Account Manager (TAM).

To say I’m excited about this would be a gross understatement. VMware has been the company I’ve spent the majority of my technical focus on up to this point, and since announcing this change on Twitter last week I’ve been thrilled with the replies like “I’ve been here 4 years and it’s an amazing place to work.” During the interview process, one of the current TAM’s told me point blank: “This is the best job I’ve had in my career.” All of this has maintained a level of anticipation about this career change that I’ve not had for any other.

It’s not as if this is a surprise because I interact with so many great people on a regular basis who work for VMware, who seem to genuinely love the work they’re doing. But it’s been refreshing to get the same messages from people I’d never even met before. 

The last year has been a rebuilding year, for me. In early 2017, I left my role as an data center engineer at a Value Added Reseller, to go back into a customer role. I had been working as a consultant for nearly six years, but prior to that I spent seven years on the customer side. So now I was back working 9-5, at the same desk. It was tough because I loved consulting, and I literally couldn’t wait to get back, but for various reasons I needed the transition. The role I took was intentionally outside my comfort zone, to force myself to do something different and pickup new skills. It was challenging in ways both expected and unexpected. The team I was working on has some great people, and it has been a fun to work with them, even if all the while I knew this wasn’t the place I wanted to stay at for very long.

This year in transition was a change that I needed, being a customer was a place to lay low, reset, and figure out my future and my priorities. There was no travel and no on-call, not even an expectation to even have email on my phone, let alone respond after hours to it.

But now I’m back, and ready to get to work doing what I love, for the company that I’ve spent the last decade focusing on, in the company of all the great people who’ve helped me get to this point.

The value of certification — For the love of the blueberry shirt

Occasionally I’ll wear my “blueberry” VMware certification shirt to work. Some people in the community love these shirts, some people don’t. I, do.

Blue also happens to be my favorite color.

Occasionally someone I work with in my /current/ workplace will comment on it. Before the last year, it was a bit of personal marketing while working as a VAR engineer. When I’d show up on site maybe there was a bit of “you can trust me because hey look it says right here I’m not some rando off the street.” In my current role, it’s not always obvious that I’m engaged in the VMware ecosystem. Since the shirt is, very blue, it gathers comments that range from “oh I didn’t know you were a…” to genuine curiosity of “what does that mean?”

Occasionally though, someone makes the less than flattering comment: “you know no one here cares about certifications, right?”

My usual response? “I do.”

In the moment I might get a little defensive and mention the number of hours required to sit for multiple VCAP exams, the underlying VCP exams, between training classes, time spent doing self guided learning or the process and stress of the actual exam.

The cost of the training, both in currency and time, is sometimes carried by the owner or sometimes their employer. I’ve been fortunate enough in my recent career to have had an employer that would make those investments on our behalf. It wasn’t always that way. Despite being deeply engaged with VMware products since 2007, it took until 2011 to obtain my first VCP. The financial hit for the required class was too much for me to take on at the time.

That VCP was my first industry certification of any kind.

I’m acutely aware that certification doesn’t mean you’re an expert, or that there are plenty of folks running around with certificates for things they have no practical experience with. That’s one reason why I’m such an advocate, and so proud of obtaining two practical/administration VCAP certificates. You can’t just memorize a test dump to walk in and regurgitate against multiple choice questions. You have to demonstrate your competency in a -slow- live environment.

So it’s fine that “nobody” in your organization cares about certifications. They have a value, if sometimes only to the holder.

In the wake of the last comment I got at work, I ordered two new blueberry VCAP shirts. My old one was getting a little rough looking. They’ll come in handy, especially in my next role.

In re, doorbell tweets

I received a lot of feedback from my tweet about ditching a new Ring for Nest Hello.

Rather than tweetstorm it up, I’ll try and summarize it all here as to why I’m switching.

Most of it boils down to already owning a fair amount of devices in the Nest ecosystem (2nd-gen Thermostat, 3 Dropcams, 3 Protects) and wanting to stay in that. All my smart home gear is split between HomeKit and Nest. Since Ring doesn’t play in either of those ecosystems, it’s yet another platform to manage, and especially since Nest and HomeKit have zero integration without flaky hacks (Homebridge) adding a third platform that talks to neither, was already a step backwards.

I’ve toyed with the idea of replacing the Nest equipment, over time, but if I do it’ll have to be into HomeKit compatible devices. Ecobee has a great thermostat alternative, but as it is, Nest makes some of the best cameras, and there’s not an alternative to the Protect that I’m interested in right now. There is a First Alert competitor that looks interesting but I’ve had bad experiences with false alarms from standard First Alert detectors recently, so my trust in them is broken.

I was already planning to buy the Hello after we moved into our new house last month, but when Costco ran a promotion on the Ring 2 that included a year of monitoring and an extra Chime at a significant discount, I couldn’t resist trying it out.

The Ring is functionally fine. One of my biggest gripes however is the recording isn’t always on, and when it is triggered by an event it’s for a limited period of time. Most recently I noticed this when our new neighbors came to the door to introduce themselves. Being bad at remembering names, I went back to the video tape only to find it cut off after about 20 seconds. I still don’t know their names.

Since it’s not always on, and it’s in sunlight most late afternoons, after activating the first few seconds are over-exposed and worthless.

The benefits of Nest for me do come at a price. The cost of the Ring 2 package was about $50 less than the Hello, and the monitoring for Hello would run another $60 for the year.

One other consideration is that the Hello just looks nicer, in my opinion. The Ring isn’t ugly, it’s just kind of meh looking, to me. I admit to also having a sour impression to the quality of the hardware, having already swapped out faulty Ring for family members, as well as having some come DOA.

My essential 10 iOS apps

Two weeks ago, after regretfully trying to use the iOS 11 developer betas on my primary devices, I was forced by general instability to roll back to iOS 10.

Unfortunately, there’s no great way to do this without doing a restore and fresh install. I had a backup from iOS 10 that I’d taken prior to jumping on the beta train, but it was old now. This process is further complicated by the way Apple Watch activity and health data is really maintained on the phone, not the watch itself.

The result was I ended up fresh installing iOS 10.3.3 (beta 6) on my iPhone 7 and iPad Pro 10.5”, as well as doing a factory reset of my Watch. It also meant losing a couple years worth of workout data, awards and streaks. But such is beta life. It did give me an opportunity to reassess what gets installed on these devices. I find it helpful to mix things up from time to time, even going as far as doing a reset of my app icon layouts periodically to reshuffle the deck chairs and throw out any old cruft hiding in corners. One of my favorite activities is to delete apps that don’t get used anymore, or used enough to take up my attention.

This time I took the approach of installing only the apps that I know I need on a daily basis, and then filling in the rest as the need for them arise.

For the most part, these are the apps that if I couldn’t put anything else on my phone, I’d be able to make due. These are the apps that I interact with daily or that provide an essential service or workflow.

Continue reading “My essential 10 iOS apps”

Will work for reasonable salary + benefits

The process of looking for a new job is stressful. If you already have one, you’re a bit like a secret agent, sneaking around town trying to complete the mission of getting someone new to agree to sign your paychecks, without the old boss finding out. If you don’t have a job, it’s even more stressful, as you wait around and watch your bank accounts dwindle, with nothing to replenish it.

I knew by March of this year that I was ready to move on from my now previous employer. I’ve never really had a difficult time finding a job when I decided to commit to the process. I don’t think this time was any different in that respect, but it was interesting. I was very lucky and excited to accept the position that I had the most interest in of all those I looked at during the entire process.

My process was around the same time that my friend @davemhenry was in the midst of his #HireDaveNow campaign on Twitter. It was kind of fun to watch Dave advertise himself, while I was lurking in the shadows, although I’m sure it was super stressful for him at the time. It would have been refreshing to be able to shout “I’m available” to the world.

Someone eventually hired Dave.

Continue reading “Will work for reasonable salary + benefits”

An end and a beginning

This morning I gave two weeks notice to my current employer, a Kansas City based VAR, where I have been a senior data center engineer for the last six years.

I’ve enjoyed many aspects of my current role; becoming certified in new technologies, learning new skills, and solving problems for customers. I’ve had the pleasure to work with a lot of talented people within the organization and within our partners… and of course, with our customers.

Looking through my documentation folders, it appears I’ve worked with at least 242 different customers on technology implementations. Some of these have been single day, one and done type customers. They needed a VNX, so I stood it up for them, and I never talked to them again. But really, many of these have been customers that I’ve had the pleasure of serving as a trusted advisor, where I can not only help guide them through infrastructure changes, but also build relationships. I will absolutely miss working with them on a daily basis.

There have also been other countless service tickets, some in the wee-hours of the morning, where I’ve helped people recover data or reassemble failed infrastructure. Some of those sleepless nights I might miss a little less.

I have been both lucky and challenged to travel a lot in my current role. Growing up and living in the Kansas City area my entire life, it was fun to be able to go to Boston, Seattle, San Fransisco, Austin, Atlanta, D.C., etc., for projects and training. Even the less glamorus places like western Illinois, northern Arkansas, eastern Oregon, or southern Tennesse could be fun for a while. While the travel schedule was not as aggressive as some in our industry endure, it was starting to became more than I wanted keep up with. Having two young children, and wanting to be always be present in their activities has been getting harder and harder.

I also reached a point a couple of years ago where I wanted to go a different direction in my career. I like to brag to people that I was a system administrator at age 13 and I became a consultant at 27, but I’ve always basically been the guy pushing the buttons and turning the screws to make things work.

My new role will be back on the customer side, but this time in a much larger enterprise than anything else I’ve really worked in before, and now my role will be a more strategic, architecture focused role. I’ll be working within the company business units to standarize systems, define technical requirements for projects, and act as a liason between the development, business and operations teams. I won’t be abandoning my experience as a virtualization, storage, and core infrastructure guy, I will be leveraging it to also get out of that comfort zone. I will be able to really focus on being the trusted advisor, within the organization, and less on pushing boxes into racks.

It will be a major change for me, and a new type of challenge, but it is one that I’m excited to be making.

Dell EMC Elect

I’m going to start this by saying something that might seem strange for a post like this, but is no surprise to my closest friends: The last two months, and especially the last two weeks, have been very stressful and mentally draining. Without getting into the details of it all, I will simply say that the biggest contributing factor, or at least the medium that has facilitated the stress, has been social media.

I decided to temporarily set my Twitter account private for a few days last week, something I’d never done in nine years on the service. The only thing I learned from that, is that having a private Twitter account sucks. Over the last few months, I’ve unfollowed and set mute filters for topics that generated more noise than signal. I’ve tried to step back and get some perspective on what’s going on in the world right now.

Continue reading “Dell EMC Elect”

Looking back at Neowin

Most of the people who know and interact with me professionally, or on social media know me as “vmstan” — and if you asked most of those people they’d tell you I only pay attention to two things when it comes to technology: VMware and Apple.

They’d be mostly right.

But there was a time before that, where I was “Marshalus” — and if you asked most of the people who knew what he paid attention to it was one thing: Microsoft. Specifically, covering Microsoft at Neowin.

That’d have been mostly right, too.

Continue reading “Looking back at Neowin”

Just enough Windows

I’ve not been a true “Windows user” on a daily basis since the glorious afternoon my first MacBook Pro arrived in 2011. That didn’t exactly mean I quit using Windows on that day, but over time I’ve continued to slim down my actual needs of the Windows desktop operating system to the point where now I keep a Windows VM around for “just enough” of the things I need from it.

Windows 10 is a huge advancement over Windows 7, which is where I left off as a PC user and over these last six years Microsoft has learned a lot from Windows 8.x being such a mess. But Windows 10 is an OS intended for use on everything from 4” smartphones to watercooled gaming rigs with multiple 27” 4K displays.

In this guide I’ve focused on simple methods of stripping out a lot of the things that don’t apply to virtual machine usage, and some of the cruft that is really only useful for someone running it on a daily driver. Typically I can reduce the idle memory and disk footprint by about 25% without loss in necessary functionality.

These instructions are not all specific to VMware Fusion, but some are. This also isn’t designed to be the “ultimate guide” in Windows 10 performance, space savings, or anything else. It’s a quick and clean way to do most of those things but not all encompassing. I think it’s easy for some of those types of optimization guides to focus on getting Windows to the point where it’s so lacking it’s almost unusable or starts breaking core functions.

This is a “light” optimization for my usage. It could it yours as well, if you have similar needs like running a small collection of utility type applications, such as a couple of EMC product deployment tools, or the old VMware client.

Continue reading “Just enough Windows”

Apple Leather Case for iPhone 7

I have an on-again, off-again relationship with iPhone cases. I put them on. I take them off. I generally don’t like cases. I’ve only broken my iPhone one time and that was when my 6 Plus came out of my pocket attached to my hand, unintentionally, on a sticky day. My iPhone 5 and 6 were rarely in cases, and had minimal wear and tear. I’m usually pretty careful. I also buy AppleCare+ on them, even though I’m lucky enough to rarely need it.

Continue reading “Apple Leather Case for iPhone 7”

This is a post about my pants

I have some strict requirements around work pants. My wife hates the “I can see your socks while you’re standing up” hipster look, so they have to be full length. Honestly it’d be a great look since I’m 6’4” but as a result I’m at a 36” inseam. I’m also currently 220lbs, which results in a 36” waist. I could probably lose some weight, but it’s not happening today.

I also have a job that’s requires me to dress nicely to meet a customer in the morning, but be willing to crawl under raised floors and chuck 50# boxes around later that afternoon, without a change of clothes. Expensive slacks will get destroyed. Wearing jeans everyday is frowned upon. I also don’t want to deal with getting pants tailored.

Between size, cost, looks and durability, I’ve found one pair of pants that consistently meet all my requirements.

Continue reading “This is a post about my pants”

Security from obscurity

A couple of years ago, one of our network security architects at work told me that I was in the wrong business. Storage, virtualization, data centers, it’s all going to the cloud. I’d soon be out of a job. 

I barely knew the guy. At first I politely laughed when he said it, but then realized he was serious. Not really a great way to make new friends at work. The irony of the situation was that he tracked me down on one of the few times I was in the office, and approached me to help him lay out some of the VMware requirements for a Trend Micro Deep Security implementation. 

It wasn’t more than a few months later, that he didn’t work for my employer anymore … not by his choice … and I’m still there, two years later, still billable most of the week. 

I don’t even remember his name. 

But, he’s wasn’t wrong, just a jerk. It’s not as if he was delivering some sort of life changing message, that I’d never heard before. It’s one I hear repeated very often on social media, in conference presentations, etc, and in the wake of this Amazon re:Invent conference last week, I’m hearing it a lot. 

It’s undeniable that a big part of my job is chucking boxes of rust and silicon into racks, stringing copper and fiber optics around, and making it all sing together in unison. I kind of enjoy it.

It’s also undeniable that things are changing.

Migrate to VCSA

Last night I did my first customer migration from a Windows based vCenter to the VMware vCenter Server Appliance (VCSA) using the new 6.0 U2M utility.

The customer was previously running vCenter 5.1 GA on a Windows Server 2008 R2 based physical HP host. In order to migrate to the VCSA, we first had to do two in place upgrades of vCenter from 5.1 GA to 5.1 U3, then again from 5.1 U3 to 5.5 U3d. After that, onto the VCSA migration.

Given the length of time the system was running on 5.1 GA code (ouch) and the amount of step upgrades required just to get things cleaned up, there was some cause for nervousness.

I admit, even though I’d read up on it, tested it in a lab, and heard other success stories … I still expected my first try to be kind of a mess.

But, it was not. The entire migration process took around 30 minutes, and was nearly flawless.

I had more issues with the upgrade from 5.1 to 5.5 than anything else during this process. Somewhere during that 5.5 upgrade the main vCenter component quit communicating with the SSO and inventory service. There were no errors presented during the upgrade, but it resulted in not being able to login at all through the C# client, and numerous errors after eventually logging in as [email protected] to the Web Client.

I tried to run through the KB2093876 workarounds, but was not successful. I ended up needing to uninstall the vCenter Server component, remove the Microsoft ADAM feature from the server, and then reinstall vCenter connected to the previous SQL database. Success.

Given those issues, I was nervous about the migration running into further issues, mostly from the old vCenter.

But again, it worked as advertised.

After the migration I did notice the customer’s domain authentication wasn’t working using the integrated Active Directory computer account. After adjuting the identity provider to use LDAP, it worked fine. I’ve had this happen randomly enough on fresh VCSA installs to think its something to do with the customer environment, but I was under the wire to get things back up and felt there was no shame in LDAP.

I’ve done too many new deployments of the VCSA since 5.x to count, and at this point was already pretty well convinced there was no reason for most of my customers to deploy new Windows based vCenters. I’d also done a fair bit of forklift upgrades with old vCenters where we ditch everything to deploy a new VCSA, which isn’t elegant, but it works if for my smaller customers that still don’t yet have anything like View, vRA, SRM, integrated backups/replication, etc.

Now I’m confident that any existing vCenter can be successfully migrated.

Windows vCenters, physical and virtual: I’m coming for you.

Crashing ESXi with Cisco RAID controllers

Recently I had two VMware Horizon View proof of concept setups for work, where we designed an all in one Cisco UCS C240 M4 box, full of local SSD and spindles, in various RAID sets. This lets the customer kick the tires on View in a small setup to see if its a good fit for their environment, but on something more substantial than cribbing resources from the production environment.

  • 5x 300GB 10K SAS RAID 5 for Infrastructure VMs (vCenter, View Broker/Composer, etc)
  • 10x 300GB 10K SAS RAID 10 for VM View Linked Clones
  • 6x 240GB SSD RAID 5 for View Replicas
  • 1x hot spare for each drive type
  • VMware ESXi 6.0 U2 is installed on a FlexFlash SD pair

After getting all the basics configured, we had a single View connection broker, with another View Composer VM on a local SQL Express 2012 instance for the database. Both were version 7.0.2. At the first site the VM base image we attempted to deploy was an optimized Windows 7 x64 instance.

But under any sort of load during a deployment of more than a handful of desktops, the entire box would come to a total stop. In some cases the only way to restore any functionality was to pull the power and restart the infrastructure VMs, one by one. Of course, once the broker and composer instances were connected, they’d attempt to create more desktops and the cycle would continue. In an attempt to isolate the issue, we tried various versions of the VMware Tools, a new Windows 7 x86 image, and I even duplicated the behavior by building a nearly identical View 6.2.3 environment, within the same box.

After digging through the esxtop data as clones were being created, I could see KAVG/Latency across all RAID sets jumps to as high as 6000ms right before all disk activity on the system eventually stops.

It didn’t matter what configuration I tried, it was present with a fresh install of ESXi 6.0 U2, and after applying the latest host patches. It was present on the out of box UCS firmware of 2.0(10), and with the stock RAID drivers from the Cisco ISO. It was present after updating the firmware, and the drivers. It also happened regardless of if the RAID controller write back cache was enabled/disabled for the various groups.

Cisco is very particular about making ESXi drivers for their components match their UCS compatibility matrix, so before I decided to give TAC a call, I made sure (again) that everything matched exactly. TAC ended up reviewing the same logs, to determine if this was a hardware issue, and while they made a couple of suggestions for adjustments, they were not successful in diagnosing a root cause. Yet, they insisted based on what they were seeing that it was not a hardware issue.

With this particular customer, we were also impacted by a variety of issues relating to the health of the DNS and Active Directory environment. With that in mind, we decided to focus on fixing the other environmental issues and in the meantime, not overload the UCS box until a deeper analysis could be done.

Try Try Again

A day or so into the second setup at another customer, and I encountered the exact same issues. This time with a Windows 10 x64 image, and View 7.0.2. The same crazy latency numbers under any amount of significant load, until the entire box stopped responding.

The physical configuration differed slightly in that we were integrating the C-Series UCS into the customers fabric interconnects, so the firmware and driver versions were even more different than the first host which was a standalone configuration connected to the customer’s network. After digging into it again with a fresh brain, and more perspective, I found the cause.

I started looking through the RAID controller driver details again. In both cases, VMware uses the LSI_MR3 driver as the default driver for the Cisco 12G RAID (Avago) controller in ESXi 6.0 U2. In both environments I verified that we were running the suggested driver versions based on the Cisco UCS compatibility matrix, and we were. So I started digging at this controller and wondered what VMware suggests for VSAN (keeping in mind we aren’t running VSAN at either site) and sure enough, they DO NOT suggest using the LSI_MR3 driver, but instead list the “legacy” MEGARAID_SAS driver as their recommendation, for the exact same controller.

After applying the alternative driver, I’ve not been able to break the systems.

What is odd, is that this appears to be related specifically to Cisco’s version of the controllers.

This week I did a similar host setup (although not for View) using a bunch of local SSD/SAS drives in a Dell PowerEdge 730xd, with their 12G PERC H730 RAID cards (which from what I can see appear to be rebranded versions of the same controller) and VMware’s compatibility matrix has the LSI_MR3 drivers listed.

I left those drivers enabled, and the customer ran a series of agressive PostgreSQL benchmarks against the SSD sets, with impressive results, and no issues from the host.

So, long story short, if you’re using local RAID sets for anything other than some basic boot volumes that don’t need any serious I/O, with the Cisco 12G RAID controller, you don’t want to use the Cisco recommended drivers.

Installation instructions

  • Download the new driver (for ESXi 6.0 U2)
  • Extract the .vib file from the driver bundle and copy it to a datastore on the host
  • Enable SSH on the host and connect to it via your terminal application of choice
  • Apply the driver from the SSH session and disable the old one.
  • Reboot the host
  • Reconnect via SSH, and run core adapter list command to verify it’s active

This should verify that your RAID controller (typically either vmhba0 or vmhba1 is now using the megaraid_sas driver. If the “UID” is listed as “Unknown” in this readout, it’s normal.

Stronger Together

I wasn’t going to get sucked into the 2016 election, then Donald Trump ran for President I’ve struggled with what to write about this election for months. I’m not shy about my opinions on social media but when it came down to going long form about making my arguments in this election, I’ve written and deleted more than can be known.

I’ve always been deeper into politics than probably any of my friends and family. It started when I would watch Dana Carvy do his iconic impressions of both Bush and Perot on SNL during the 1992 election.

My brother and father are sports junkies. Politics is my drug of choice.

During the 2008 and 2012 campaigns, I was active on social media as a vocal supporter of Obama. It was hard not to be. I think he’ll go down as one of the most transformational figures of my lifetime.

But this year, it’s different. In past elections, I’ve said not great things about GOP candidates. My statements in hindsight, never expected Donald Trump to enter politics.

Mitt Romney is by all accounts an honorable man, and would have kept this country safe. I would have been frustrated but not ashamed before the world, and before my children, that he was our representative.

Now is the election of 2016, and there’s a chance that Donald Trump, could be our next President.

I repeat: there is a chance that this racist, fascist, sexist, disgusting excuse for a man, could be the next President of the United States.

I have two young boys. They know who we support in this election. My oldest son was more upset than I was when someone stole the Clinton/Kaine yard sign from our yard. In a year when it’s not popular to advertise that you endorse either candidate, it’s even more important to take a stand.

And, so, I built a bigger sign.

But in our house we teach our children to treat each other with love and respect. We teach our children to stand up the ideas and the rights of themselves, and for others.

As a straight, white, college-educated, male, in Kansas, I could easily sit back and hide. I could leave the signs down. I could be a registered independant. I could save my money. I could stay quiet.

I won’t.

I support Hillary Clinton for President.

And so should you.

So, vote. For our republic, for the rights and the protection of everyone. Vote. Against bigotry, against hate, against someone who can’t see that America is already great.

Vote.