Ever since I started podcasting in 2006, I've been using a company called “Hipcast” to be the “hosting” company for the podcast audio files and feeds. For those of you not familiar with podcasting, that's the website where I upload podcast audio files… and they then serve those files when requested by services like Apple Podcasts or Spotify.
This has been an aggravating week or so, since Hipcast has been having a lot of technical issues going back to last Tuesday or so. Their website has been down, the podcast feeds (RSS feeds) have been broken. People haven't been able to listen to my podcasts unless they had already downloaded the episodes through their app.
One of my regular listeners pulled the virtual “andon cord” and emailed me to let me know there was a problem. I wasn't thrilled because I had just published an excellent episode with Joy Furnival, and I was disappointed that the “launch” of that was now messed up.
Hipcast was up, then down. They posted messages about things being fixed, and then it all broke again.
I felt frustrated and powerless. All I could do is wait… but I'm impatient. I pictured me and their other customers looking like this (except I was working from home and not in a suit).
People posted things like this:
Questions like that are hard to answer, but remind me of…
Scars from General Motors
My impatient waiting reminded me of working at General Motors in 1995… under the first plant manager, or the one who was very much the traditional style of GM leader. This was before the NUMMI-trained plant manager came in to start fixing the culture.
I was the industrial engineer for an engine block machining line. The actual value-added work of cutting metal was done by machines. Those machines were run and monitored by UAW employees and received support from a process engineer, whose role it was to be in the details of the machines, working with the vendors, etc. while I was focused more on the overall flow and management of the area — the bigger picture vs. the details.
Often, one of those giant expensive machines would go down for some reason, meaning the flow of the engine block line was interrupted. Sometimes, an expert would have to be flown in from Germany.
Sometimes, the downtime was very much due to bad management decision making.
A UAW worker (and maybe the engineer) would say, “It's time to stop and change one of the cutting tools, because this is the planned frequency.”
A senior management person would basically say, “No, you can't stop the machine to change the tools… because we're behind on production and I'm getting chewed out, so we can't afford to stop now. Those tools have a longer lifespan anyway, they're overengineered.”
Sometimes, that would end up biting them in the rear. Instead of taking a short amount of time for a tool change (was it under an hour? I forget), there would be some catastrophic failure that would really mess up the machine. That made the “behind on production” problem even worse.
I used to joke that you could tell how long a machine had been down, based on which level of management was standing there, arms crossed, looking mad (and not being able to do anything about the situation but wait). The longer the downtime, the higher the level. Leaders would constantly demand an ETA for the fix.
If the machine had been down for a few shifts or days, the plant superintendent (who looked a bit like the guy from Caddyshack) would really start yelling at people:
Of course, Bob would yell and scream instead of (at least outwardly) reflecting on the decision that he had made that led to this problem. I used to joke that we could have just bought a cardboard cutout of Bob… put him far enough away and people might not have known the difference… the message of “work faster” would have been the same?
I'm sure similar things might happen in software companies, I don't know. Management makes a bad decision, then something bad happens. Management gets mad and pressures the people who know how to fix it… and those people resent having to work 24/7 to resolve a problem that could have been avoided (the machine going down). The glares and the yelling… not helpful.
Taking Ownership of my Situation
But, now back to the podcast host situation. I firmly believe that a company doesn't get to blame a supplier or a vendor for bad performance or bad behavior. Leaders shouldn't blame employees for bad performance when leaders are the ones more responsible for the system (and if you're hiring bad employees… why are you doing that??)
Unlike the GM managers, I couldn't directly pressure anybody to fix the situation with my podcast hosting any faster. I could post a whiny tweet or a complaint on their Facebook page… but that wasn't really helping.
It's one of those situations where them stopping to give a status update or to respond to some “when will it be fixed?” message then ends up SLOWING DOWN the efforts, at least a little bit.
I was happy with Hipcast for almost 13 years. I'm not happy about the downtime. I'm not happy about the lack of communication on their part. But, I chose the supplier. This is on me.
Earlier this year, I was thinking, “Why is Hipcast never on anybody's lists of top podcast hosts? Did I make a bad choice? Should I move to one of the “Top 10” podcast hosts or somebody who is always recommended by others?
But, I wasn't proactive. I thought about switching, but I didn't take action. I wondered about their low $9.99 monthly price… appreciating that it had never gone up over time (then again, hardware and bandwidth costs in this domain have only fallen). I wondered about their business model… what happened if Hipcast went out of business?
Well, I wish I had acted on my “what if something went wrong?” hunch.
I've spent a lot of time over the last five days researching other podcast hosting companies and working to do what I can to get things up and running at other companies.
And, I say “companies” because I've decided, for now, to not put all of my podcast eggs in the same basket. I have six different podcasts now and I should have recognized the risk of them all being at the same company. I mean, it had been convenient and cost effective to have them all in the same account.
I've gotten my flagship podcast “Lean Blog Interviews” back up and running on a company that is considered one of the top tier hosting companies. I'll pay about $29 a month for that.
I also put the new “Lean Whiskey” podcast there for the initial launch after planning to use Hipcast. That created additional work for me, but assured a smooth launch last Friday.
I put the “Practicing Lean” podcast on a different platform, one that will be free since the podcast hast fewer listeners. And, now that the full audiobook is published there, I probably won't be adding new episodes.
Part of my thought process was to experiment and try out different companies. I could read reviews and look at the pros and cons… but the only way to know for sure was to try them. I tried a few podcast hosts that I ended up not going with for one reason or another. One had great features, but seemed like too small of a company. Others just didn't seem to work as well (or it was my preference).
The rest of my podcasts
are on a relatively new company called Anchor.fm. I was skeptical about them being new (and being free) but they were acquired earlier this year by Spotify, which gives me hope that they will be a major player in the podcast space.
I know there is some risk there… but I don't have all of my digital audio eggs in the same server basket. I did also import Lean Blog Interviews into Anchor.fm. I haven't decided yet if that's an additional backup or if I'll use that service going forward.
Status Update for Listeners
So, if you're a listener, here's where I think things stand.
All of the “RSS feeds” should be restored now. If you've been a subscriber, you shouldn't have to take any action. Please let me know if you have problems with podcast episodes on your smart phone or other podcast devices.
One known problem still is with “embedded players” for various episodes. Those single-episode players (which are installed individually on each podcast episode blog post) are from Hipcast and those currently aren't working. Moving the RSS feed and podcast hosting doesn't fix those players.
The process for updating the RSS feeds in the Google Play Podcasts and Spotify services is not self-serve as it is for Apple, so it's taking longer to get feeds resolved there, compared to Apple Podcasts and Stitcher.
Update: Hipcast says, as of Monday morning, that their service is fully restored. I think the embedded players are working again.
A customer asked about “root cause analysis” and the CEO is going to post something, they claim:
With Hipcast getting things back and running, I'll probably leave most of these embedded players there (on pages like this)… continuing to pay $9.99 a month vs. putting in the time (or paying someone) to copy-and-paste code for the new hosts' players is something I have to consider. I apologize for the ones that aren't working.
But, recent episodes (like the one with Joy Furnival) have the new host's player installed so that works. Thanks for your patience… or if you've been impatient with me being impatient with Hipcast… at least I think I have things on a better path (or better paths) now.
What do you think? Please scroll down (or click) to post a comment. Or please share the post with your thoughts on LinkedIn – and follow me or connect with me there.
Did you like this post? Make sure you don't miss a post or podcast — Subscribe to get notified about posts via email daily or weekly.
Check out my latest book, The Mistakes That Make Us: Cultivating a Culture of Learning and Innovation:
I use hipcast and I can’t even get the website to come up at all today. No one is responding. I’m super frustrated.
Carrie, I feel your pain. The website loads for me… but after all of the downtime, I lost faith in their ability to permanently fix the problem. And the lack of communication made things worse. Good luck to you!
I have my podcasts moved, but the 301 redirects from Hipcast are not working yet.
Things are working well today… the Hipcast streaming players aren’t working and their website has been done. My podcast redirects are still not working.
I’m glad I’ve moved the RSS feeds to other companies, at least.
Just to play devil’s advocate, didn’t you say that you haven’t had any problem in the past 13 years? So might this week’s downtime just be noise and part of the expected common cause variation within a predominantly in-control process?
Right. This seems like a “special cause.” But that doesn’t mean that I can be confident that they will get things fixed. It’s been over a week and there are still problems with their site and their service. Can I guarantee that a new host won’t also have problems? No. But, I think going with a bigger company (Hipcast might have fewer than 10 employees, if LinkedIn is accurate) reduces the risk. Hipcast doesn’t list the name of their CEO or other leaders on their website. It makes me worry that it’s one guy in a garage.
My biggest beef is not the downtime… it’s more about the poor communication and the lack of attention to us as customers. That rubs me the wrong way.
At least I’ve learned how to switch hosts, so that process will be easier if I need to do that in the future.
Sorry, you’re right, special cause variation within a basically in control process is what I had meant to say. My basic thought was that theoretically the proper response to special cause variation is to prevent that specific event from happening again rather than changing the entire process. In any case, I’m glad it’s all worked out!
You’re right, but the only way I can get things up and running is to change hosts. Hipcast should be figuring out how to prevent this from happening again.
Blog post from the CEO of Hipcast… +1 for the correct use of the word “humbling.”
A Humbling Experience.
Hipcast ceased operations at the end of 2022… I’ve been happy with Podbean as a podcast host since I switched in 2019.