My Blog Had The 18-Hour Flu: Thoughts on Downtime and Management Pressure
So I think things are back to normal after a somewhat stressful day of wondering why my blog “broke.” Unlike some web/blog outages and problems, it wasn't due to anything I did or tried to change. It just broke. Settings that worked fine up until Thursday morning no longer worked. It wasn't a general Blogger outage (the first thing I checked), it was my site's settings.
Downtime, pressure, scowls, and flashbacks
I had some flashbacks to my time working at the GM Livonia Engine plant, circa 1995. We would have machining centers go down (for various reasons, including not doing enough planned maintenance) and managers would respond by coming out to the gemba to stand and glare, pressuring employees and maintenance folks to work faster. We always joked that nothing ever got fixed by a superintendent or plant manager standing there
As the length of downtime progressed, arms would be crossed, brows would furrow, and scowls would grow. Oh yeah, don't forget the yelling. At times yesterday, I felt like I was standing there, scowling at myself (as both superintendent and maintenance tech for the blog). As they say, hope is not a strategy, and scowling or pity do not get machines back up and running. I tried to get out of that mode and mood ASAP.
Thinking back to GM again, I had the idea back in 1995 that, if machines got fixed faster with the plant superintendent standing there (the guy who claimed that you had to “beat down on people until they did exactly as they're told“), then I figured our plant should buy a bunch of life-sized cardboard cutouts that could be strategically placed around the plant. If you kept them far enough away, you might not know if it was “real Bob” or “cardboard Bob” watching you work.
It wasn't a very collaborative or “Lean culture” environment, at least not until we got a new plant manager with a NUMMI background.
My Blog Downtime
Many of you might want to quit reading here if you don't care about blog tech stuff. My blog and hosting setup is a bit unusual maybe — it's hosted for free on the Blogger service (owned by Google), but my domain (leanblog.org) is hosted (a paid service) through godaddy.com. There are some settings on godaddy.com that point your browser from leanblog.org to the material stored on the Blogger servers. That was all working fine… until yesterday.
I did some internet searching and found some reports that folks with my setup were impacted by the rollout of a new Google feature called “Sites.” It's a correlation, but also seems like a causation. I could blame Google for what some have called a faulty design and faulty integration of the services (a shared databased…. ah “synergy”…) Some other Blogger blogs were down for the same reason (including this one that had the same 404 error as me).
I'll look in the mirror and think about how I could have avoided the problem. It's a bit loose use of the term, but I could have “5S-ed” my site set up a while back. Months back, I had tried setting up the “Google Apps” platform for my domain, leanblog.org. I was never able to get it working, so the setup was just, well, sitting there. If I had never partially set up Google Apps, I wouldn't have had the problems that occurred when they added the “Sites” feature. Not sure how I could have (or should have) anticipated that. Live and learn. Note to self: Don't ever leave half-installed features hanging out there… I shut down all of the Google Apps features and that helped get things back up and running.
My other (and final) reflection point was that, in my haste to “fix” things, I rushed into one “solution” that broke something else. The tech instructions from Blogger are very general and not helpful enough and godaddy.com's tech support couldn't help. I'll give a big thanks to the website “Blogbloke” and his instructions that got things back up and running.
I hope that's it for the downtime… thanks for your patience and we'll get back to real Lean topics soon.