Kernel Panic

I’ve been thinking for over two years about how to talk about this, how to talk about the day I couldn’t process anymore. The day where my bullshit buffers overflowed. The day of my internal kernel panic. When starved for resources, my brain finally threw up a PSOD, and quit.

vmstan 1.0 build 11091983
PCPU 0 locked up. Failed to ack.
frame=4C lbs=E1 age=21
VMK uptime: 9:15:33:45.014
No coredump target configured

If there were some predictive analytics running on me back then, it would have been evident that this was coming for a long time. Unbeknownst to anyone else, I knew that I couldn’t keep up and that I couldn’t go on forever in my degraded state. The warning signs were all there, but no one was reading the logs.

The root cause analysis would tell you: lousy code embedded on possibly flawed hardware. No permanent fix, but mitigations are available.

I have depression.

My crash that day was in part because I was pretending there wasn’t a problem. For years I would think to myself “I should go to a therapist,” but you’d never catch me say the words out loud, to anyone. It was the result of repressing emotions for years, the consequence of not wanting to admit to anyone else that I could have a problem.

I was an anxious child. Even as far back as eight years old, I remember spending a lot of time in a state of worry. Thinking about if my family had enough money or fearing what would happen at school. It was around this time that I started to get bullied — teased continuously about how skinny I was, or about how I didn’t have an issue hanging out with girls on the playground. I had a “friend” who lived up the street, who’d do things like sucker-punch me in the stomach. I was, from an early age, an attractive target for people who wanted to make themselves feel better at my expense.

Getting to junior high school, around age 12, was a chance to reset. While the peer bullies didn’t completely disappear, I found a group of like-minded friends with a common interest. I immediately got involved with our school’s nascent computer club, and we did at the time what seemed crazy for a bunch of 12-year-olds in 1996 — we made a website for our school that won a grant from Microsoft for $20,000.

The grant enabled our club to purchase a 16-system Windows NT lab for our school. The members of the club built and ran the lab, and for nearly three years this was the real focus of my school day. The librarians would pull me out of class anytime there was an issue, of which there were many.

I was 13 and trying to manage NT security policies without any training, without Google to fall back on. What could go wrong?

My final year there, the advisors put an incredible amount of stress on me to keep the lab running. One of them was working on getting more technology-related grants for the school and district. They had to show off our lab, my lab, as a way they were using the new technology to benefit the educational process in the school. Their efforts were complicated when the lab didn’t work.

If I couldn’t do the job, they were going to let someone else handle it, or worse, give it to the school district IT department to manage. I took an immense amount of personal pride in this project, and while in hindsight I shouldn’t have been the only one responsible, at the time I felt a lot of ownership for it.

It all came to a head when I walked into a club meeting, to a sign posted by the adult sponsor that said: “fix the lab, or else.” — In front of the entire club I went on an angry tirade, but not in front of the sponsors who were, in that moment, hiding somewhere far away. Whatever the issues with the lab were, whatever their reasoning, it didn’t necessitate the behavior that was targeted at me by an adult. The damage to my psyche was there by the time my mom finally found out what was going on and intervened, they backed off.

I’d had my first moment of IT burnout.

I spent my high school years in the jazz and marching bands. I became the editor of my high school newspaper. Instead of the club, I’d organize LAN parties for my friends, and I had a side project that involved running a dedicated server for web hosting, with around 30 clients.

After working in retail while sporadically attending college, I landed my first real IT role working as a network analyst for a private university. I had supportive management who in hindsight let me do all kinds of crazy things. We didn’t have the money to hire professional services, so we implemented our ESX clusters in 2007. Storage, backups, security, data center networking, campus wireless, we did it all.

I got to touch everything in the environment, and sometimes do things that would blow it up.

It was there that I would spearhead a virtual desktop project that ended up gaining notoriety, with case studies and awards. I was busy but incredibly proud of the work that I was doing. I found my job stressful but very satisfying. There were some personalities, like any place, but it was fulfilling. When I have dreams about going back to work at an old employer, this is the place. However, the pay was lacking and I was newly married and planning on starting a family.

In 2010, I started working for a software company which at the time was “the company” to work for in the area … and I was pretty miserable for the first few months.

In almost every way, the new company was a great place to be with more money and better benefits. It was a goal of mine to work for them for a long time. I intentionally bought my first house in proximity to their building and within a year had a job there as a systems administrator. The company had a lot of great people working for them, and it was a very relaxed workplace with lots of new technology coming in the door every month. We even had a regulation-sized dodgeball court down the hall from the datacenter. There was a slide between floors. And yet, I was uncomfortable.

As strange as this sounds, I missed my VDI environment.

Eventually, I started finding fulfillment in the new projects that I was assigned. I needed to show my new coworkers that I was worthy of sitting alongside them. I’m probably always harder on myself than others are of me, in this respect.

Very quickly, I started building some great relationships with my coworkers there, but my time in the role would be short-lived. The company was acquired, and it became apparent that the new owners were not interested in being in the IT business. Outsourcing firms did the hands-on administration in their corporate environment, and as a result, all our jobs were now in jeopardy.

Nothing bonds a small group of coworkers together like knowing upper management is out to get you. My direct manager did his best to shield us from this, and to set us up in positions that wouldn’t be as impacted, but he could only do so much and in the end, for his sanity, left to take an architecture role with a VAR. He was a big part of that team being successful, and we all wanted to follow him, and continue working with each other, so by the end of 2011, one-by-one we all got engineering roles at the same VAR.

“Value Added Reseller”

Now, if you’ve never worked for a VAR, I would suggest listening to the Datanauts podcast from April of last year, to help get an understanding of what VAR-life is. Being a post-sales engineer, you’re an essential part of the “value” the customer is purchasing. Initially, it took a lot of work to adjust to this life. At first, I thought it was because I missed having ownership of my infrastructure. For the first couple of years before walking into a client I’d never dealt with before, I’d sit in my car in a state of severe anxiety. Mostly afraid that customer would be the one to unmask me as a massive fraud. After repeatedly not being discovered, I decided my secret was safe.

But in hindsight I was never happy. I might have acted that way to customers and coworkers that I interacted with on projects, but inside there was something else growing.

I never felt like I belonged as an employee of the company. Other than the people I’d come into the job with from the previous employer, in many ways I felt out of place among my coworkers. Not to dwell on the details, but my standards for professionalism were not always in lock step with the rest of the organization.

After a few years of toiling through grunt work, there were a few key technologies that I had worked myself up to “expert” status on, so customers, sales teams, and project managers would want me to work those implementations regardless of where in the midwest they were located. I started traveling too much, but in some ways, it was the intended result of my goal to become so relied on that I was the best or only choice, for my job security. So, I didn’t complain.

I rarely complained to management about anything there; I was more adept at just doing the work. I was making pretty good money and concerned that I couldn’t do better anywhere else, but most of all, I was subject to a restrictive non-compete agreement. I was already getting a lot of flak from my family for changing jobs as often as I did, despite knowing that it was pretty standard for people in this industry.

My father, in contrast, has had two jobs in 40 years, and if he wasn’t laid off from the first in the early 1990s, he might still be there.

After three years, I thought about leaving. After four, I looked around but got turned down in the final round for what at the time would have been my “dream job” — Even after a few months from being turned down, when that company got acquired, and it was apparent that the “dream” would have been a nightmare, I still felt resentment. I’d never been turned down after making it through that many interviews, and I didn’t want to do it again. So, I turned down multiple opportunities to interview for positions with VMware shortly after that.

My customers were continually satisfied with my work, and I enjoyed the challenges presented in doing it. My boss put me in for substantial raises and bonuses each year, and so I collected an ever-increasing paycheck. I kept thinking it’d all get better soon. Despite all that, while I was still miserable, I stayed silent.

I thrived in darkness.

Meanwhile, the state of the world seems to be mirroring the darkness that I was feeling inside. By the fall of 2016, I’m a secretly broken person floating from day to day, hoping I’d wake up and eventually all the crap inside will go away and feel “normal” again … but it never does; it only gets worse. I’m now spending too much time on social media being angry at everything, and like swimming in the pink slime in Ghostbusters, I’m just feeding off the negative energy of everyone else.

I began feeling paranoid about the world around me. It would start the moment that I woke up in the morning, and last until the night came as I sat in my children’s bedrooms, worried that someone was going to break into the house, and take them away. I always thought about all the terrible things that could happen to my family or me. I’d think about the danger lurking behind every corner, about all the terrible people in the world. When I would be out of town for work, as I was frequently, I’d be mostly unable to sleep.

I’m thankful that I’ve never really had much of a taste for alcohol, but I’d eat terrible meals while traveling and continue to gain weight. My back hurt and I’d have headaches all the time, something that I’ve had to deal with ever since I was a teenager.

But of course, I told this to no one. I’d have my yearly physical and during the mental health screening, lie-lie-lie through the whole thing. I knew what the doctor wanted to hear because I’d been telling the same lies my entire life. If the doctor couldn’t find it, I wasn’t going to volunteer the information.

It’s not as if I thought that by being honest in my answers, they’d send me away to an asylum. It was that everyone would discover my weakness. It was something else that someone could direct against me, or maybe worse when everyone else found out they’d feel sorry for me: “Poor Michael, he’s not tough enough.”

I’d be in the system as having a problem. It didn’t matter that these doctors were the same people who would prescribe me antibiotics for my chronic sinus infections, and that never seemed to bother me — now we’re talking about being marked as having a mental health issue.

Then at age 33, I was bullied, again.

I knew it was going to be a problem almost as soon as it started, and I remember the moment it began. Sitting in a Zaxby’s restaurant in Hopkinsville, Kentucky, just checking Twitter — as I all too frequently do — I’d just flown in on a Sunday afternoon for a week-long project. I’d never been to Zaxby’s before, so I was looking forward to trying something new, but instead, this was the lunch when everything went from bad to awful. Now the harassment started, and later that night is when my Twitter impersonator first appeared.

When you’re worried about people in the world coming after you, it doesn’t help matters when they do.

I fought back the harassment by effectively doing nothing, hoping they’d give up by not getting a reaction out of me, but there it was every time I’d open Twitter. I’d see some shitty comment about me or something about my wife. Maybe something about the things that I was passionate about or someone from the community being targeted, with my name and likeness. When I thought it would help, I’d play whack-a-mole with Twitter to get accounts closed for impersonation or abuse, but they’d just come back.

The harassment went on for a little over a month, and then finally, around 3 pm on a Friday, I’d come home from a particularly complicated customer site and opened Twitter to find another shitty comment. Another fucking tweet. At that moment, I crashed.

I make analogies to a server crashing, because in hindsight that’s what felt like was happening. I couldn’t think, I just started crying, and I nearly threw up. My stomach had been in knots for weeks up to this point, but it felt like everything was coming undone. My body was still moving, but for some time I wasn’t in control. I remember very clearly everything that day leading up to that moment, but very little during the event. What wasn’t de-staged from the cache when the controller died, is gone.

I didn’t have to tell my wife. It was impossible for her not to know; she was there when the crash happened and immediately went into action to get me help. I knew something like this was coming, so I’d been trying to reach out to a therapist in the days prior, but could never get our schedules synchronized. By chance, the therapist called me that afternoon in the middle of my panic, and I let it all go to her over the phone, having never met with her previously. She could tell just by the tone of my voice, my breathing, that I needed immediate care. Off to the emergency room, I went.

For the first night in a long while, I felt a little better.

I got sent home with a temporary prescription for Xanax, and a directive to talk to my doctor. The strange thing is that while I felt better because I finally let my secret out, I spent the weekend an absolute wreck. The flood gates of emotion had opened, and I had a backlog of questions to address. I also knew that there was no going back and that I would be forced to tackle even more issues and make some significant changes in my life.

For a while, what transpired in the aftermath of that night, left me feeling betrayed by some people that I thought I could trust, and feeling helpless. Like so many times in life before, I was told by those who had the power to stop what was going on, that there was nothing they were willing to do, that this was my problem, and that I was to blame. My pain was always my fault. It’s never the responsibility of the people who were inflicting it.

I don’t know what exactly motivates people to be so cruel to others. It’s unfortunate, but in the end, we are all flawed individuals. Even the bullies are coping. It doesn’t matter if they’re a kid in school or adults who act like children. It could be a teacher who is supposed to be looking out for the kids, and not their self-serving interests. I might cope with my issues by becoming withdrawn from others. They might deal by lashing out, by letting the demons inside them come out and feed the beast hiding in others.

I quickly began looking for a new job, far away from the life I was leading. No travel. Regular hours. I wasn’t even sure if I wanted to stay in the same realm of IT. These events are why I spent a year at a customer in a role that I was utterly dispassionate about; it was a place to hide and regroup. Even though I was physically in a different situation, I was still a mess of a human being inside. If you worked with me there, you don’t even know the real me. It was a period of my life where I’m not even sure I knew who the real me was. I’d wonder that every day, and it still crosses my mind. I told almost no one there about what had happened — no one on my team, or my boss. Truthfully, I was ashamed.

My brain needed to heal.

I now take medication for something I acknowledge is both an internal and an environmental condition. I have gone to therapy to discuss my issues with bullies, and other factors that contribute to my anxiety and depression. Every one of us is a collection of biology and experiences that got us to this point in our lives. Great athletes can be physically superior to others, but neglect the gray-matter between their ears, too.

I try to listen to my body and recognize both the internal and the external factors that help me get me centered and calm my brain. I have more routines that help me regulate how I keep myself organized. When I go to the doctor, I tell her what is wrong. The awareness that I have now about my self makes this much more manageable. But this is a daily, hourly, practice. Even in doing all of these things though, this isn’t something you just beat; you can have things under control and still have struggles.

Even in moments of achievement, I can be prone to feeling depressed. I don’t have to have a bully or be in a bad situation. Brains are funny like that. When VMware hired me in April of last year, I felt as if I’d reached a highpoint in my career and celebrated the realization of a decade long goal. Almost instantly after signing my offer, I started to worry.

Imposter syndrome began to kick in.

What would happen to me if this didn’t go well? How soon would it be before everyone I’d respected up to this point found out that I was a massive fraud? Where would I work, what would I do, how would I feed my family!? I was having heartburn, trouble sleeping, and always worried that it would all come crashing down. The dark side of my brain was back, trying to sabotage me again.

But you know what helped this? Doing the work, having a positive and supportive management team, but mostly just talking to my teammates and others about their struggles. To quit being afraid of “getting caught” and freely admit what I know and what I don’t. Realizing that this is the most fabulous job I’ve ever had, in the company of some of the smartest and kindest people I’ve ever met. A year later I wake up and get dressed and wonder how this happened. Sharing my thoughts, and opening up to let them do the same. We all had the same fears. We were not alone.

Every single person you meet is dealing with their devil inside. You don’t have to be to the point where you’re contemplating the end of your life. Your best friend can appear to you to have things rightly figured out, yet be suffering in total silence, never showing any outward sign of the struggles they’re going through. Nothing can be “wrong,” and yet they can be in pain. The jerk in line at the coffee shop may always be a jerk, or you possibly caught them on a bad day. Your coworker can be fighting to keep his kids while his ex-wife tries to move them across the country, and every extra hour at work is just a distraction to keep his mind off the fight.

This story has sat in my drafts folder for over a year. What started as a rambling journal that was never planned to see the light of day, has been evolving as I get more perspective on my recovery from that life. As I’ve contemplated my journey, I noticed something else happening. People I knew and respected were now becoming vocal about their similar issues. Whereas previously I felt as if I was the only one, I now see where I am one of many.

Eric Lee, who I’d known both as a co-worker, client and community leader for many years, posted his story. The parallels were striking, especially in the overlap of where we’d worked and what we’d done in our careers. The next day I took him out to lunch and shared my story. I would watch his presentations about what he calls “IT Burnout” at VMUG and other settings. I’d contemplate how I needed to tell my story, but I was still scared.

More and more people would start to tell their stories, and I’d privately share mine with more people. Aaron Buley started sharing his struggles on his YouTube channel. He posted a video called “All Humans Struggle – Depression” which calls it for what it is. Cody De Arkland talks about carrying baggage, Erik Shanks talks about the dark side of stress, and Al Rasheed quoted Paul Simon. Even the CTO of Citrix has a story to tell.

I call them out as examples from the Greater Virtualization Community, but it’s not just them. Dwayne “The Rock” Johnson has talked about his struggles. Jon “Mad Man” Hamm has as well. Here are two people who play tough guys on TV, yet struggle in real life. We see these people and realize they’re actors playing a part, but it’s not far off from the persona that we share with the public and the one inside of each of us.

Then there are “real people” like Jason Kander. Someone who has served in Missouri state office then ran for the U.S. Senate and later led a nationwide voting rights organization. He was in the process of running for mayor of Kansas City when he abruptly brought his campaign to a halt to declare that, he too, was broken — struggling with depression as a result of PTSD from his time as an Army intelligence officer in Afghanistan.

It is in those moments that I remember; we all have struggles.

And I’m done being quiet about mine.


Excess Battery

I recently picked up an iPhone XS Smart Battery Case, for my iPhone X. I’ve previously owned the 6S battery case for my iPhone 6, and the iPhone 7 case. Let me first say that even though Apple has designed the case for the XS, it worked on the X running iOS 12.2.2 without any effort.

Smart Battery Case for iPhone XS

The two cosmetic issues of not being designed specifically for the X is that because the camera housing on the XS is slightly taller than the X, the cutout on the back of the case leaves a little extra space that wouldn’t be seen when using an XS. This is … fine, I know it’s there, but unless you make a habit of putting your camera six inches away from your target, no one else would ever see it.

The other issue is that the speaker grills on the XS are not symmetrical, but it is on the X, so the XS case doesn’t have cutouts that match that at the bottom. However, from what I understand, the additional holes on the X grill serve no purpose. They’re just there for aesthetics, so there is no loss of functionality. On the XS there is an antenna line on the exterior of the case, so Apple did away with the faux-holes entirely.

Comparing the new case to previous iterations, they’re functionally very similar. Some reviews have mentioned that there is no passive antenna array on the new cases, as there was with the 6S and 7 version. I did not rip the silicone off to verify this. According to Rene Riche, the XS case runs at a higher voltage that yields a total of 10.1 Wh of energy compared to 7.13 Wh for the iPhone 6/6s case and 8.98 Wh for the iPhone 7 case. Otherwise, it features the same Lightning pass-through that allows you to charge both the phone and the case, or hook up to a car stereo or iTunes without any issues. When using a “fast charging” setup such as an 18W charging adapter and USB-C to Lightning cable, the case and phone are able to leverage the additional power delivery.

iPhone 7 Smart Battery Case

I have mixed feelings about the new design of the bottom of the case, compared to previous battery cases. I really like that there isn’t a chin that extends off the phone anymore, although on the previous designs this provided a channel to redirect the sound to the front of the phone, which was great for watching videos and not having to cup the end of the phone with your hand to “boost” the sound towards your head. The more obvious solution to this is use headphones and not expose your audio with the world like an animal, but sometimes you just want to share.

The chin on previous versions houses the charging circuitry and controller for the battery. The lack of chin means these electronics have relocated to the back of the case. I think the new design looks better than just having a straight up hump in the middle of the case, although it means it’s not as easy to hold one-handed from the bottom. The hump in previous designs gave you a place to rest your pinky. (Yes, I do the pinky support the bottom grip thing which I’m sure is bad for my hand.) It’s harder for me to single hand type with the case on, but that could be like any form factor change, requiring some time to adapt.

Back in my day, the 4″ iPhone 5 felt “huge,” and now we have people palming the iPhone XS Max like it’s no big deal.

Usually, I’m caseless or use the Apple Leather Case which doesn’t have any coverage of the bottom of the phone. Because of this, I’m aggressive with my swipe gestures up from the bottom of the phone, but that is annoying with the addition of a case lip. If I actively adjust my swipe it works and doesn’t catch, so I have to train myself to do that.

iPhone X Leather Case

Unlike previous iterations of the Smart Battery Case, this one has Qi charging integrated, and it is fantastic. I have about nine Qi chargers in my house now, so if Qi weren’t a supported option, I wouldn’t have even bothered to test the case. The Qi charging target seems smaller for the battery case than on the X without the case. I’ve become pretty accustomed to throwing my phone on a charger and getting it placed where it would juice up without much effort.

Many times I placed the case’d phone on a wireless charger it required adjustment to get it in the exact right spot. The Qi target is at the top of the battery bulge, so it ends up being the middle of the phone, but based on the layout one could expect it to be in the middle of the bulge.

The case came out of the box with a minimal charge, and I didn’t focus on giving a dedicated charge to the case that day (it can be charged wired or wireless, independent of the phone) so I think it was at around 7% when I went to bed. When I woke up 7 hours later, the phone and case were at 100% charge on both. After a morning banging away on it, I was at 87% on the case, 100% on the phone. As for heat while charging or in usage, I’ve not noticed anything hotter than I would on a naked phone.

One thing I would like to see is some more “Smart” around when the battery is engaged in the iOS battery status. Using the case means always being at 100% for the phone until the case is depleted, and it would be nice to see some statistics around the charge status of the case in this view.

One other concern: I did perceive a minor drop in signal strength when using the phone, for both LTE and Wi-fi. Putting the phone in Field Test Mode, I was able to see changes to “RSRP” and “SINR” values on cellular, with the case on and off in nearly identical conditions. I didn’t do this testing extensively or average out across different sites and times. I also didn’t test this with an XS, so while it is possible this is due to the case not being designed for the X, it could also just be expected for this case. I’m also not a wireless engineer, so I don’t truly understand what all the values mean and what all the variables are.

Wi-fi signal indicators also could be decreased when the case was on, in a corporate environment. I did not test throughput or any other values. I don’t want to speculate or say that my results are typical, so I’ll let someone who understands this more dig into it.

Overall, I think it’s a solid product, but it’s not for me. Sadly this is the same conclusion that I’ve come with previous iterations. I just don’t like a lot of extra bulk on my phone. My wife has gotten way more mileage out of me owning those previous case iterations than I ever did, using them 24/7 for many months during the release cycles. The extra battery helps offset a Facebook app that sucks it up like a Dyson does dirt on my floor. However she recently switched to an iPhone XR, and the battery life has been pretty stellar, to where the case may not be necessary.

With the aforementioned abundance of Qi chargers in my home, a general dislike of all iPhone cases, and an Anker battery with 72.36 Wh, featuring USB-C PD in my daily bag, I don’t think it’s worth it for me to have the extra bulk on the phone all the time.

Revisting ​​Essential

Back in July 2017, I wrote about my 10 essential iOS applications. I thought now that we’re reaching the end of 2018, it might be a good time to revisit that list.

As I mentioned at the time:

I find it helpful to mix things up from time to time, even going as far as doing a reset of my app icon layouts periodically to reshuffle the deck chairs and throw out any old cruft hiding in corners. One of my favorite activities is to delete apps that don’t get used anymore, or used enough to take up my attention.

This time I took the approach of installing only the apps that I know I need on a daily basis, and then filling in the rest as the need for them arise.

A few weeks ago, I took a similar approach but this time with a far more aggressive regimen. I uninstalled nearly every third-party application from my phone. Then I started to analyze the feature/function of every app and determine if the app itself provided something more than just a wrapper around an already functional mobile web site.

I find that having fewer things installed on my devices brings me some joy.

Shopping, banking, social media, travel, news, food, weather, shipping. Almost app categories were fair game. About the only group that was mostly safe were apps that controlled the various smart devices around my house: Nest, Hue, eero, Rachio, Lutron, myQ, etc. 

From there, it was about finding the apps that were the truly essential apps in my workflow:

  • I recently switched to Outlook as my primary email/calendar application. This means I can displace the stock Mail and Calendar apps, as well as remove Fantastical, which was on my 2017 list. Currently, though, I’ve been experimenting with having only my work email in Outlook, and my personal email in the stock app, just for workload isolation. I can’t decide if there is more of less mental friction in keeping them together or keeping them separated.
  • I also have been using Microsoft To-Do as a reminders replacement, mostly because of the Tasks integrations with Outlook on Mac. (However, I’ve been bad recently at actually doing the things in here.) I’ve been comingling work and personal tasks in here. This has replaced Things for the time being.
  • 1Password is simply irreplaceable. You’ll pry it from my cold, dead hands.
  • Then there is Overcast for podcasts. I’ve experimented with alternatives in the last few months from the stock Podcasts app, Pocket Casts, and Castro, and always come home to Overcast for the basic reason that podcasts just sound better in there.
  • And of course, Tweetbot for Twitter. I just can’t quit you.
  • Shortcuts has replaced Workflow after Apple bought them and built much of it into iOS 12.
  • I keep the ads and other trackers away in Safari with Better.
  • I have Parcel setup to automatically track Amazon, FedEx, UPS, and USPS shipments, of which there are many this time of year. (Seriously, the entrance of my house looks like a loading dock right now.)
  • Zoom is a requirement for work.
  • As is Slack.
  • I’d be locked out of both of those without my RSA soft-token.
  • And I love the ability to upload receipts with Concur.
  • While authenticating through Workspace One.
  • AT&T Call Protect has become my new junk filter for phone calls. This replaced Nomorobo from the 2017 list, which is still a fantastic app, but AT&T’s app is network integrated.
  • Finally, despite my new love for Nespresso, I still have a requirement for Starbucks on the go and like to have my order ready as I walk in the door.

From there I started a review with the assumption that I could avoid anything else. Despite quitting Facebook a couple years ago, I’m still on Instagram for close friends and family. I tried for weeks to limit myself to using the web app, in an effort to avoid another app install, but also having it try to entice me to spend more time in it with a dedicated shortcut on my home screen. After trying to limit my usage, I gave in and reinstalled it because it was just too damn hard not to.

I had a similar experience with LinkedIn. The issue there was more around the usability of the website on a mobile device. It was pretty terrible. I’d like to keep this uninstalled but I occasionally end up dropping it back on and then off again. I’ve uninstalled it again recently.

My primary bank has mobile check deposit, and I have family members who for some reason continue to write me checks despite showing the multitude of better ways to transfer money around. So it was a given for reinstall because the only thing worse than writing a check is having to physically go into a bank. So 2007.

I had Microsoft OneDrive installed for accessing work files from my phone, but realized I never used it outside of the native integration of the Outlook app. So, I deleted it. This may return if I find some other reason I was using it.

Target came back, despite my original wipe of shopping applications because of its store card being integrated with the app. The only thing I like more than being able to delete an application from my phone is taking a physical card out of my wallet.

A few other apps that were considered essential in 2017, I’ve since abandoned. Carrot Weather is great, and probably one of the best third-party weather apps on iOS, but I find the stock app to be good enough most of the time. Pcalc is another great app, but I don’t find myself needing to calculate anything so complicated at the stock app can’t get the job done. Cloak was on the list last time and has since been sold and rebranded as Like many things that get sold, it just doesn’t feel like it has the same level of love and care as the original owners, and so it’s been cut.


VMware has published new security advisories, knowledge base articles, updates and tools in response to newly disclosed speculative-execution vulnerabilities on Intel CPUs — collectively as “L1 Terminal Fault” — that can occur on Intel processors made from 2009 to 2018.

I’m going to outline our response to this issue, and make an attempt to summarize this complex event as best as I can. I would highly suggest reading through the linked articles as they’ll be more extensive and evolving.

Because this is complex, and evolving, to properly respond to these issues, consider KB55636 as the centralized source of truth from VMware.

Like the previously known Meltdown, Rogue System Register Read, and “Lazy FP state restore” vulnerabilities, the “L1 Terminal Fault” vulnerability can be exploited when affected Intel microprocessors speculate beyond an unpermitted data access.

L1TF – VMM (CVE-2018-3646VMSA-2018-0020)

This is the specific L1TF issue that affects the vSphere/ESXi hypervisor. It has two known attack vectors, both of which need to be mitigated. The first attack vector is mitigated through patches for both vCenter and ESXi.

The second attack vector is mitigated by enabling a new advanced configuration option (hyperthreadingMitigation) included in the updates. However, this advanced configuration option may have a performance impact so we have not be enabled it by default. This will limit operational risk by giving you time to analyze the effects prior to enabling.

There are new updates to both vCenter and ESXi that deliver the mitigation to L1TF:

  • vCenter 6.7.0d, 6.5u2c, 6.0u3h, and 5.5u3j
  • ESXi670-20180840x, ESXi650-20180840x, ESXi600-20180840x, and ESXi550-20180840x

There are also new versions of VMware Workstation (14.1.3) and Fusion (10.1.3) which address this issue.

L1TF – OS (CVE-2018-3620)

This is a local privilege escalation which requires base operating system updates for mitigation. Patches are pending for affected VMware appliances. Make sure you contact your operating system vendor(s) (Microsoft, Oracle, Red Hat, etc) for mitigation instructions in guest virtual machines as well.

L1TF – SGX (CVE-2018-3615)

This does not affect VMware products.


I’m pleased to announce that yesterday I passed the VCAP6.5-DCD exam, thus earning the VMware Certified Implementation Engineer – Data Center Virtulization “milestone” after elevating the VCAP5-DCA exam that I earned back in 2014.

The DCD exam has been on my list of things to do since not long after I did the DCA. My first attempt was during the beta cycle for the 6.0 exam. The results for that exam took so long to be returned, and after shifting in job roles since then, I’d not had an oppertunity to sit for it until now. The 6.5 version of the exam differs from the 6.0 in that there are no longer the “Visio” style questions, which I think were problematic for the exam from the beginning. There are 60 questions consisting of multiple-choice, drag-n-drop, and multi-select questions, with 140 minutes to complete the exam. I was able to complete the exam in just under 90 minutes, and I didn’t feel like I was rushing.

In terms of advice I can pass on to others who are interested in taking this exam, make sure that you understand:

  • AMPRS (Availability, Manageability, Performance, Recoverability and Security)
  • RCAR (Requirement, Constraint, Assumption and Risk)
  • The difference between Functional and Non-Functional requirements

If you are hands on with vSphere 6.5, especially working with vCenter HA, PSC/SSO and cluster design, you should have all of the bases covered. I have been removed from much of that in the day-to-day for the last year or two, so that was probably the more challenging part of the exam for me. I think if I’d done more to read up on differences between 5.x/6.0 and 6.5, I’d have come back with a better score. But, pass is pass.


Once upon a time there was a meeting of minds,
The sun and the moon made a deal with the sky,
One would take the morning and the other the night,
Together they would blanket the world with light,
But the moon had a shadow, he felt like a liar,
The sun was the only one who carried the fire,
The sun saw this, she kept on glowing,
Bound to the moon, never saying, “you owe me”
She said “I’ll shine on you.” Jason Mraz

Who will you shine on today?

VMware TAM

I have accepted a job with VMware, as a Technical Account Manager (TAM).

To say I’m excited about this would be a gross understatement. VMware has been the company I’ve spent the majority of my technical focus on up to this point, and since announcing this change on Twitter last week I’ve been thrilled with the replies like “I’ve been here 4 years and it’s an amazing place to work.” During the interview process, one of the current TAM’s told me point blank: “This is the best job I’ve had in my career.” All of this has maintained a level of anticipation about this career change that I’ve not had for any other.

It’s not as if this is a surprise because I interact with so many great people on a regular basis who work for VMware, who seem to genuinely love the work they’re doing. But it’s been refreshing to get the same messages from people I’d never even met before. 

The last year has been a rebuilding year, for me. In early 2017, I left my role as an data center engineer at a Value Added Reseller, to go back into a customer role. I had been working as a consultant for nearly six years, but prior to that I spent seven years on the customer side. So now I was back working 9-5, at the same desk. It was tough because I loved consulting, and I literally couldn’t wait to get back, but for various reasons I needed the transition. The role I took was intentionally outside my comfort zone, to force myself to do something different and pickup new skills. It was challenging in ways both expected and unexpected. The team I was working on has some great people, and it has been a fun to work with them, even if all the while I knew this wasn’t the place I wanted to stay at for very long.

This year in transition was a change that I needed, being a customer was a place to lay low, reset, and figure out my future and my priorities. There was no travel and no on-call, not even an expectation to even have email on my phone, let alone respond after hours to it.

But now I’m back, and ready to get to work doing what I love, for the company that I’ve spent the last decade focusing on, in the company of all the great people who’ve helped me get to this point.

Blueberry Lover

Occasionally I’ll wear my “blueberry” VMware certification shirt to work. Some people in the community love these shirts, some people don’t. I, do.

Blue also happens to be my favorite color.

Occasionally someone I work with in my /current/ workplace will comment on it. Before the last year, it was a bit of personal marketing while working as a VAR engineer. When I’d show up on site maybe there was a bit of “you can trust me because hey look it says right here I’m not some rando off the street.” In my current role, it’s not always obvious that I’m engaged in the VMware ecosystem. Since the shirt is, very blue, it gathers comments that range from “oh I didn’t know you were a…” to genuine curiosity of “what does that mean?”

Occasionally though, someone makes the less than flattering comment: “you know no one here cares about certifications, right?”

My usual response? “I do.”

In the moment I might get a little defensive and mention the number of hours required to sit for multiple VCAP exams, the underlying VCP exams, between training classes, time spent doing self guided learning or the process and stress of the actual exam.

The cost of the training, both in currency and time, is sometimes carried by the owner or sometimes their employer. I’ve been fortunate enough in my recent career to have had an employer that would make those investments on our behalf. It wasn’t always that way. Despite being deeply engaged with VMware products since 2007, it took until 2011 to obtain my first VCP. The financial hit for the required class was too much for me to take on at the time.

That VCP was my first industry certification of any kind.

I’m acutely aware that certification doesn’t mean you’re an expert, or that there are plenty of folks running around with certificates for things they have no practical experience with. That’s one reason why I’m such an advocate, and so proud of obtaining two practical/administration VCAP certificates. You can’t just memorize a test dump to walk in and regurgitate against multiple choice questions. You have to demonstrate your competency in a -slow- live environment.

So it’s fine that “nobody” in your organization cares about certifications. They have a value, if sometimes only to the holder.

In the wake of the last comment I got at work, I ordered two new blueberry VCAP shirts. My old one was getting a little rough looking. They’ll come in handy, especially in my next role.

In Re, Doorbell Tweets

I received a lot of feedback from my tweet about ditching a new Ring for Nest Hello.

Rather than tweetstorm it up, I’ll try and summarize it all here as to why I’m switching.

Most of it boils down to already owning a fair amount of devices in the Nest ecosystem (2nd-gen Thermostat, 3 Dropcams, 3 Protects) and wanting to stay in that. All my smart home gear is split between HomeKit and Nest. Since Ring doesn’t play in either of those ecosystems, it’s yet another platform to manage, and especially since Nest and HomeKit have zero integration without flaky hacks (Homebridge) adding a third platform that talks to neither, was already a step backwards.

I’ve toyed with the idea of replacing the Nest equipment, over time, but if I do it’ll have to be into HomeKit compatible devices. Ecobee has a great thermostat alternative, but as it is, Nest makes some of the best cameras, and there’s not an alternative to the Protect that I’m interested in right now. There is a First Alert competitor that looks interesting but I’ve had bad experiences with false alarms from standard First Alert detectors recently, so my trust in them is broken.

I was already planning to buy the Hello after we moved into our new house last month, but when Costco ran a promotion on the Ring 2 that included a year of monitoring and an extra Chime at a significant discount, I couldn’t resist trying it out.

The Ring is functionally fine. One of my biggest gripes however is the recording isn’t always on, and when it is triggered by an event it’s for a limited period of time. Most recently I noticed this when our new neighbors came to the door to introduce themselves. Being bad at remembering names, I went back to the video tape only to find it cut off after about 20 seconds. I still don’t know their names.

Since it’s not always on, and it’s in sunlight most late afternoons, after activating the first few seconds are over-exposed and worthless.

The benefits of Nest for me do come at a price. The cost of the Ring 2 package was about $50 less than the Hello, and the monitoring for Hello would run another $60 for the year.

One other consideration is that the Hello just looks nicer, in my opinion. The Ring isn’t ugly, it’s just kind of meh looking, to me. I admit to also having a sour impression to the quality of the hardware, having already swapped out faulty Ring for family members, as well as having some come DOA.

Define Essential

Two weeks ago, after regretfully trying to use the iOS 11 developer betas on my primary devices, I was forced by general instability to roll back to iOS 10.

Unfortunately, there’s no great way to do this without doing a restore and fresh install. I had a backup from iOS 10 that I’d taken prior to jumping on the beta train, but it was old now. This process is further complicated by the way Apple Watch activity and health data is really maintained on the phone, not the watch itself.

The result was I ended up fresh installing iOS 10.3.3 (beta 6) on my iPhone 7 and iPad Pro 10.5”, as well as doing a factory reset of my Watch. It also meant losing a couple years worth of workout data, awards and streaks. But such is beta life. It did give me an opportunity to reassess what gets installed on these devices. I find it helpful to mix things up from time to time, even going as far as doing a reset of my app icon layouts periodically to reshuffle the deck chairs and throw out any old cruft hiding in corners. One of my favorite activities is to delete apps that don’t get used anymore, or used enough to take up my attention.

This time I took the approach of installing only the apps that I know I need on a daily basis, and then filling in the rest as the need for them arise.

For the most part, these are the apps that if I couldn’t put anything else on my phone, I’d be able to make due. These are the apps that I interact with daily or that provide an essential service or workflow.

Continue reading Define Essential

Will work for reasonable salary + benefits

The process of looking for a new job is stressful. If you already have one, you’re a bit like a secret agent, sneaking around town trying to complete the mission of getting someone new to agree to sign your paychecks, without the old boss finding out. If you don’t have a job, it’s even more stressful, as you wait around and watch your bank accounts dwindle, with nothing to replenish it.

I knew by March of this year that I was ready to move on from my now previous employer. I’ve never really had a difficult time finding a job when I decided to commit to the process. I don’t think this time was any different in that respect, but it was interesting. I was very lucky and excited to accept the position that I had the most interest in of all those I looked at during the entire process.

My process was around the same time that my friend @davemhenry was in the midst of his #HireDaveNow campaign on Twitter. It was kind of fun to watch Dave advertise himself, while I was lurking in the shadows, although I’m sure it was super stressful for him at the time. It would have been refreshing to be able to shout “I’m available” to the world.

Someone eventually hired Dave.

Continue reading Will work for reasonable salary + benefits

An end and a beginning

This morning I gave two weeks notice to my current employer, a Kansas City based VAR, where I have been a senior data center engineer for the last six years.

I’ve enjoyed many aspects of my current role; becoming certified in new technologies, learning new skills, and solving problems for customers. I’ve had the pleasure to work with a lot of talented people within the organization and within our partners… and of course, with our customers.

Looking through my documentation folders, it appears I’ve worked with at least 242 different customers on technology implementations. Some of these have been single day, one and done type customers. They needed a VNX, so I stood it up for them, and I never talked to them again. But really, many of these have been customers that I’ve had the pleasure of serving as a trusted advisor, where I can not only help guide them through infrastructure changes, but also build relationships. I will absolutely miss working with them on a daily basis.

There have also been other countless service tickets, some in the wee-hours of the morning, where I’ve helped people recover data or reassemble failed infrastructure. Some of those sleepless nights I might miss a little less.

I have been both lucky and challenged to travel a lot in my current role. Growing up and living in the Kansas City area my entire life, it was fun to be able to go to Boston, Seattle, San Fransisco, Austin, Atlanta, D.C., etc., for projects and training. Even the less glamorus places like western Illinois, northern Arkansas, eastern Oregon, or southern Tennesse could be fun for a while. While the travel schedule was not as aggressive as some in our industry endure, it was starting to became more than I wanted keep up with. Having two young children, and wanting to be always be present in their activities has been getting harder and harder.

I also reached a point a couple of years ago where I wanted to go a different direction in my career. I like to brag to people that I was a system administrator at age 13 and I became a consultant at 27, but I’ve always basically been the guy pushing the buttons and turning the screws to make things work.

My new role will be back on the customer side, but this time in a much larger enterprise than anything else I’ve really worked in before, and now my role will be a more strategic, architecture focused role. I’ll be working within the company business units to standarize systems, define technical requirements for projects, and act as a liason between the development, business and operations teams. I won’t be abandoning my experience as a virtualization, storage, and core infrastructure guy, I will be leveraging it to also get out of that comfort zone. I will be able to really focus on being the trusted advisor, within the organization, and less on pushing boxes into racks.

It will be a major change for me, and a new type of challenge, but it is one that I’m excited to be making.

Dell EMC Elect

I’m going to start this by saying something that might seem strange for a post like this, but is no surprise to my closest friends: The last two months, and especially the last two weeks, have been very stressful and mentally draining. Without getting into the details of it all, I will simply say that the biggest contributing factor, or at least the medium that has facilitated the stress, has been social media.

I decided to temporarily set my Twitter account private for a few days last week, something I’d never done in nine years on the service. The only thing I learned from that, is that having a private Twitter account sucks. Over the last few months, I’ve unfollowed and set mute filters for topics that generated more noise than signal. I’ve tried to step back and get some perspective on what’s going on in the world right now.

Continue reading Dell EMC Elect

Looking back at Neowin

Most of the people who know and interact with me professionally, or on social media know me as “vmstan” — and if you asked most of those people they’d tell you I only pay attention to two things when it comes to technology: VMware and Apple.

They’d be mostly right.

But there was a time before that, where I was “Marshalus” — and if you asked most of the people who knew what he paid attention to it was one thing: Microsoft. Specifically, covering Microsoft at Neowin.

That’d have been mostly right, too.

Continue reading Looking back at Neowin

Just enough Windows

I’ve not been a true “Windows user” on a daily basis since the glorious afternoon my first MacBook Pro arrived in 2011. That didn’t exactly mean I quit using Windows on that day, but over time I’ve continued to slim down my actual needs of the Windows desktop operating system to the point where now I keep a Windows VM around for “just enough” of the things I need from it.

Windows 10 is a huge advancement over Windows 7, which is where I left off as a PC user and over these last six years Microsoft has learned a lot from Windows 8.x being such a mess. But Windows 10 is an OS intended for use on everything from 4” smartphones to watercooled gaming rigs with multiple 27” 4K displays.

In this guide I’ve focused on simple methods of stripping out a lot of the things that don’t apply to virtual machine usage, and some of the cruft that is really only useful for someone running it on a daily driver. Typically I can reduce the idle memory and disk footprint by about 25% without loss in necessary functionality.

These instructions are not all specific to VMware Fusion, but some are. This also isn’t designed to be the “ultimate guide” in Windows 10 performance, space savings, or anything else. It’s a quick and clean way to do most of those things but not all encompassing. I think it’s easy for some of those types of optimization guides to focus on getting Windows to the point where it’s so lacking it’s almost unusable or starts breaking core functions.

This is a “light” optimization for my usage. It could it yours as well, if you have similar needs like running a small collection of utility type applications, such as a couple of EMC product deployment tools, or the old VMware client.

Continue reading Just enough Windows

Apple Leather Case for iPhone 7

I have an on-again, off-again relationship with iPhone cases. I put them on. I take them off. I generally don’t like cases. I’ve only broken my iPhone one time and that was when my 6 Plus came out of my pocket attached to my hand, unintentionally, on a sticky day. My iPhone 5 and 6 were rarely in cases, and had minimal wear and tear. I’m usually pretty careful. I also buy AppleCare+ on them, even though I’m lucky enough to rarely need it.

Continue reading Apple Leather Case for iPhone 7

This is a post about my pants

I have some strict requirements around work pants. My wife hates the “I can see your socks while you’re standing up” hipster look, so they have to be full length. Honestly it’d be a great look since I’m 6’4” but as a result I’m at a 36” inseam. I’m also currently 220lbs, which results in a 36” waist. I could probably lose some weight, but it’s not happening today.

I also have a job that’s requires me to dress nicely to meet a customer in the morning, but be willing to crawl under raised floors and chuck 50# boxes around later that afternoon, without a change of clothes. Expensive slacks will get destroyed. Wearing jeans everyday is frowned upon. I also don’t want to deal with getting pants tailored.

Between size, cost, looks and durability, I’ve found one pair of pants that consistently meet all my requirements.

Continue reading This is a post about my pants

Security from obscurity

A couple of years ago, one of our network security architects at work told me that I was in the wrong business. Storage, virtualization, data centers, it’s all going to the cloud. I’d soon be out of a job. 

I barely knew the guy. At first I politely laughed when he said it, but then realized he was serious. Not really a great way to make new friends at work. The irony of the situation was that he tracked me down on one of the few times I was in the office, and approached me to help him lay out some of the VMware requirements for a Trend Micro Deep Security implementation. 

It wasn’t more than a few months later, that he didn’t work for my employer anymore … not by his choice … and I’m still there, two years later, still billable most of the week. 

I don’t even remember his name. 

But, he’s wasn’t wrong, just a jerk. It’s not as if he was delivering some sort of life changing message, that I’d never heard before. It’s one I hear repeated very often on social media, in conference presentations, etc, and in the wake of this Amazon re:Invent conference last week, I’m hearing it a lot. 

It’s undeniable that a big part of my job is chucking boxes of rust and silicon into racks, stringing copper and fiber optics around, and making it all sing together in unison. I kind of enjoy it.

It’s also undeniable that things are changing.

Migrate to VCSA

Last night I did my first customer migration from a Windows based vCenter to the VMware vCenter Server Appliance (VCSA) using the new 6.0 U2M utility.

The customer was previously running vCenter 5.1 GA on a Windows Server 2008 R2 based physical HP host. In order to migrate to the VCSA, we first had to do two in place upgrades of vCenter from 5.1 GA to 5.1 U3, then again from 5.1 U3 to 5.5 U3d. After that, onto the VCSA migration.

Given the length of time the system was running on 5.1 GA code (ouch) and the amount of step upgrades required just to get things cleaned up, there was some cause for nervousness.

I admit, even though I’d read up on it, tested it in a lab, and heard other success stories … I still expected my first try to be kind of a mess.

But, it was not. The entire migration process took around 30 minutes, and was nearly flawless.

I had more issues with the upgrade from 5.1 to 5.5 than anything else during this process. Somewhere during that 5.5 upgrade the main vCenter component quit communicating with the SSO and inventory service. There were no errors presented during the upgrade, but it resulted in not being able to login at all through the C# client, and numerous errors after eventually logging in as [email protected] to the Web Client.

I tried to run through the KB2093876 workarounds, but was not successful. I ended up needing to uninstall the vCenter Server component, remove the Microsoft ADAM feature from the server, and then reinstall vCenter connected to the previous SQL database. Success.

Given those issues, I was nervous about the migration running into further issues, mostly from the old vCenter.

But again, it worked as advertised.

After the migration I did notice the customer’s domain authentication wasn’t working using the integrated Active Directory computer account. After adjuting the identity provider to use LDAP, it worked fine. I’ve had this happen randomly enough on fresh VCSA installs to think its something to do with the customer environment, but I was under the wire to get things back up and felt there was no shame in LDAP.

I’ve done too many new deployments of the VCSA since 5.x to count, and at this point was already pretty well convinced there was no reason for most of my customers to deploy new Windows based vCenters. I’d also done a fair bit of forklift upgrades with old vCenters where we ditch everything to deploy a new VCSA, which isn’t elegant, but it works if for my smaller customers that still don’t yet have anything like View, vRA, SRM, integrated backups/replication, etc.

Now I’m confident that any existing vCenter can be successfully migrated.

Windows vCenters, physical and virtual: I’m coming for you.