Warning: file_get_contents(/home/www/settings/mirror_forum_db_enable_sql): failed to open stream: No such file or directory in /var/www/html/content/Forum/functions.php on line 8
Someone suggested a postmortem report on the last set of incidents. That's a great idea. I'll be doing a blog post soon. Maybe I can weave all of the details into some grand philosophical treatise. Love me a grand philosophical treatise.
We love you! You're a real star. As in a Marvel Universe!
You've made sooo many lives more beautiful by what you've been doing!
Thank you, thank you, thank you!
Because.. life is often short and fleeting, as we know.
Someone suggested a postmortem report on the last set of incidents. That's a great idea. I'll be doing a blog post soon. Maybe I can weave all of the details into some grand philosophical treatise. Love me a grand philosophical treatise.
But, in the meantime, I've given this a lot of thought and these things seem true:
1. Doing things in an orderly, methodical fashion is a lot easier when you're dealing with proven, documented systems & tools rather than inventing everything as you go.
2. Some aspects of my personal psychology really get in our way.
Here's an example: the normal workflow when developing or updating software is to configure and test everything in a 'sandbox' or development environment, and to only deploy to production (public-facing) servers when you're sure everything is working correctly.
That requires separate servers just for testing and development. In the early days of RP luxuries like that were out of the question. I felt lucky to have even one dedicated database server, and buying or leasing another one would take money I'd rather spend it on expanding our stream relay network. So much sexier, y'know.
So I developed what they call a "cowboy coding" approach: trying out new features and fixing bugs on the same servers — and database data — that you depend on to deliver our streams, apps, & website. Frantically typing raw SQL commands and writing nasty BASH scripts as everything melts down because of a semicolon. A FREAKIN' SEMICOLON! The stress of moments like that — so many moments like that — has probably taken a year or ten off my lifespan. Especially when it comes to semicolons. The least useful bit of punctuation in the mongrel tongue being force-fed to the entire world, but the whole web would grind to a halt without 'em.
The hopefully-final straw is that, for a variety of reasons, this time around I was more aware than ever of how difficult my working style makes things for others. Cowboy coders are generally a solo act, and RP is much more of a rock band at this point. When a band is in the groove, getting people off their asses and onto the floor, the last thing they need is for the keyboard player to announce between songs that he'd just converted to the Church of A432, and that they needed to stop, ignore their audience, and retune all of their instruments to match.
The audience, all jacked and sweaty just moments ago, is left standing around, bored and disappointed — and the keyboard player's assurances that the disruption is all in service of a glorious greater good don't really help.
So, apologies to all of you — and my bandmates — for the disruptions. We'll be setting up that development server. And I'll be using it.
3. Something else that I — and several others involved — bring to the table is a kind of irrational optimism that leads us to assume that whatever we're doing will happen smoothly & quickly, without any problems. That leads one to ignore the need to deeply research potential problems in advance, and to treat a superficial understanding of something as good enough.
CarPlay integration in the new iOS app is a good case in point. There were three of us making the key decisions about the app, and none of us had ever actually used CarPlay. We assumed that we knew enough to accurately assess things and make the right choice. Obviously, we were wrong. If we'd taken the time to do our homework we would have known that before we'd disrupted our relationship with thousands of loyal listeners. Like you.
I really appreciate you taking the time to write this â truly. Itâs obvious you care about Radio Paradise, and we donât take that lightly.
First, I want to be clear: there was no plan to âgo offline.â This wasnât a scheduled maintenance window where we knowingly shut down the stream. It was backend work intended to strengthen the infrastructure, and unfortunately it triggered unexpected failures. Thatâs on us â but it wasnât a deliberate decision to allow dead air and there was no way to plan for something we had no idea would trigger such issues.
Youâre absolutely right that dead air is painful. William knows that better than anyone. We all do. The stream itself is sacred to us. The moment something goes down, it becomes all-hands-on-deck until itâs restored.
One thing that might not be visible from the outside is how lean we actually are. While there are people who contribute in various roles, our core technical infrastructure is maintained by essentially two engineers and one of those engineers is also the boss and main DJ. They are supporting a global, 24/7, streaming service that operates across web, mobile apps, CarPlay, Android Auto, smart speakers, and multiple audio formats â all without corporate backing, venture capital, or a broadcast network behind us.
We really are a mom-and-pop operation serving a massive worldwide audience. Thatâs part of the beauty â and part of the fragility.
The house analogy wasnât meant to imply neglect or known instability. It was meant to acknowledge that after 26 years of continuous evolution, systems accumulate complexity. Modernizing that complexity is necessary to keep RP viable long-term. Sometimes that means carefully touching foundational pieces. And sometimes, despite testing, something behaves differently in the wild than it does in staging.
Could we build more redundancy? Of course. We are actively working toward greater resilience. But that takes time and resources, and we grow those carefully and deliberately.
There were no ignored red flags. There was no cavalier decision-making. There was careful long hours of work that had unintended consequences â and a team that responded as quickly as possible.
As for revenue impact â yes, outages matter. Weâre fully aware of that. No one here treats this casually. This station is our livelihood and our lifeâs work.
Weâre constantly balancing:
⢠Stability
⢠Innovation
⢠Limited staffing
⢠Financial sustainability
⢠And a global audience that expects perfection
Itâs not corporate radio. Itâs not iHeart. Itâs not Spotify. Itâs a small, fiercely committed team trying to keep human-curated radio alive in a very automated world.
Your feedback is heard. And your expectations come from a place of wanting RP to thrive â which we share.
Weâre not perfect. But we are deeply committed.
And weâre still here.
Thanks for the reply. And thanks so much for all the hard work. I really appreciate, I shudder to think what my life would be like without the stream that you hole sacred. I also hold it sacred.
I'm just asking that the next time something happens, come into the thread and post something, anything. Let us know that you know. Radio silence here about what's going on is not good. It doesn't have to be a long screed, just a quick note, we know that there's an issue, we're working on it and we'll be back up asap. That would have been so much better than just silence from the team.
I really appreciate you taking the time to write this â truly. Itâs obvious you care about Radio Paradise, and we donât take that lightly.
First, I want to be clear: there was no plan to âgo offline.â This wasnât a scheduled maintenance window where we knowingly shut down the stream. It was backend work intended to strengthen the infrastructure, and unfortunately it triggered unexpected failures. Thatâs on us â but it wasnât a deliberate decision to allow dead air and there was no way to plan for something we had no idea would trigger such issues.
Youâre absolutely right that dead air is painful. William knows that better than anyone. We all do. The stream itself is sacred to us. The moment something goes down, it becomes all-hands-on-deck until itâs restored.
One thing that might not be visible from the outside is how lean we actually are. While there are people who contribute in various roles, our core technical infrastructure is maintained by essentially two engineers and one of those engineers is also the boss and main DJ. They are supporting a global, 24/7, streaming service that operates across web, mobile apps, CarPlay, Android Auto, smart speakers, and multiple audio formats â all without corporate backing, venture capital, or a broadcast network behind us.
We really are a mom-and-pop operation serving a massive worldwide audience. Thatâs part of the beauty â and part of the fragility.
The house analogy wasnât meant to imply neglect or known instability. It was meant to acknowledge that after 26 years of continuous evolution, systems accumulate complexity. Modernizing that complexity is necessary to keep RP viable long-term. Sometimes that means carefully touching foundational pieces. And sometimes, despite testing, something behaves differently in the wild than it does in staging.
Could we build more redundancy? Of course. We are actively working toward greater resilience. But that takes time and resources, and we grow those carefully and deliberately.
There were no ignored red flags. There was no cavalier decision-making. There was careful long hours of work that had unintended consequences â and a team that responded as quickly as possible.
As for revenue impact â yes, outages matter. Weâre fully aware of that. No one here treats this casually. This station is our livelihood and our lifeâs work.
Weâre constantly balancing:
⢠Stability
⢠Innovation
⢠Limited staffing
⢠Financial sustainability
⢠And a global audience that expects perfection
Itâs not corporate radio. Itâs not iHeart. Itâs not Spotify. Itâs a small, fiercely committed team trying to keep human-curated radio alive in a very automated world.
Your feedback is heard. And your expectations come from a place of wanting RP to thrive â which we share.
Weâre not perfect. But we are deeply committed.
Knock it off Randy. Alanna very thoughtfully responded to his thoughts and they had a wonderful exchange of information. Then you inject yourself where you are not needed. You are the dead horse. Just go away.
Anal, much? Be happy the music is back and just let it go. No need to beat a dead horse.
Knock it off Randy. Alanna very thoughtfully responded to his thoughts and they had a wonderful exchange of information. Then you inject yourself where you are not needed. You are the dead horse. Just go away.
Thank you for understanding the spirit in which I wrote my comments, and for responding to the concerns about the outage. I also wish to express regret for my initial characterization of the maintenance as cavalier or careless, and I owe you and William an apology.
It sounds like RP didn't really expect things to go sideways. Shit happens. We all get it. Especially those of us who have full time careers in IT, operations, software development, etc.
In that context, I think it would be terrific and truly enlightening if William and Jarred compiled an "incident post-mortem" report.
There are many examples of post-mortems published by larger institutions like Reddit, Facebook, Microsoft, or Cloudflare.
In brief, they are a chronological narrative that begin with a summary the planned changes, and go on to include: highlights of work-in-progress and completed tasks; the moment when the system broke down in an unplanned or unexpected way; what work was done to restore service; how long it took to restore service; and performance & reliability data as the system started to come back online. Incident post-mortems often conclude with a "root cause analysis" that narrows down and explains the exact reason for the unexpected outage.
As always, "Thanks for Listening."
Anal, much? Be happy the music is back and just let it go. No need to beat a dead horse.
I really appreciate you taking the time to write this â truly. Itâs obvious you care about Radio Paradise, and we donât take that lightly.
First, I want to be clear: there was no plan to âgo offline.â This wasnât a scheduled maintenance window where we knowingly shut down the stream. It was backend work intended to strengthen the infrastructure, and unfortunately it triggered unexpected failures. Thatâs on us â but it wasnât a deliberate decision to allow dead air and there was no way to plan for something we had no idea would trigger such issues.
Youâre absolutely right that dead air is painful. William knows that better than anyone. We all do. The stream itself is sacred to us. The moment something goes down, it becomes all-hands-on-deck until itâs restored.
One thing that might not be visible from the outside is how lean we actually are. While there are people who contribute in various roles, our core technical infrastructure is maintained by essentially two engineers and one of those engineers is also the boss and main DJ. They are supporting a global, 24/7, streaming service that operates across web, mobile apps, CarPlay, Android Auto, smart speakers, and multiple audio formats â all without corporate backing, venture capital, or a broadcast network behind us.
We really are a mom-and-pop operation serving a massive worldwide audience. Thatâs part of the beauty â and part of the fragility.
The house analogy wasnât meant to imply neglect or known instability. It was meant to acknowledge that after 26 years of continuous evolution, systems accumulate complexity. Modernizing that complexity is necessary to keep RP viable long-term. Sometimes that means carefully touching foundational pieces. And sometimes, despite testing, something behaves differently in the wild than it does in staging.
Could we build more redundancy? Of course. We are actively working toward greater resilience. But that takes time and resources, and we grow those carefully and deliberately.
There were no ignored red flags. There was no cavalier decision-making. There was careful long hours of work that had unintended consequences â and a team that responded as quickly as possible.
As for revenue impact â yes, outages matter. Weâre fully aware of that. No one here treats this casually. This station is our livelihood and our lifeâs work.
Weâre constantly balancing:
⢠Stability
⢠Innovation
⢠Limited staffing
⢠Financial sustainability
⢠And a global audience that expects perfection
Itâs not corporate radio. Itâs not iHeart. Itâs not Spotify. Itâs a small, fiercely committed team trying to keep human-curated radio alive in a very automated world.
Your feedback is heard. And your expectations come from a place of wanting RP to thrive â which we share.
Weâre not perfect. But we are deeply committed.
And weâre still here.
Thank you for understanding the spirit in which I wrote my comments, and for responding to the concerns about the outage. I also wish to express regret for my initial characterization of the maintenance as cavalier or careless, and I owe you and William an apology.
It sounds like RP didn't really expect things to go sideways. Shit happens. We all get it. Especially those of us who have full time careers in IT, operations, software development, etc.
In that context, I think it would be terrific and truly enlightening if William and Jarred compiled an "incident post-mortem" report.
There are many examples of post-mortems published by larger institutions like Reddit, Facebook, Microsoft, or Cloudflare.
In brief, they are a chronological narrative that begin with a summary the planned changes, and go on to include: highlights of work-in-progress and completed tasks; the moment when the system broke down in an unplanned or unexpected way; what work was done to restore service; how long it took to restore service; and performance & reliability data as the system started to come back online. Incident post-mortems often conclude with a "root cause analysis" that narrows down and explains the exact reason for the unexpected outage.
I really appreciate you taking the time to write this â truly. Itâs obvious you care about Radio Paradise, and we donât take that lightly.
First, I want to be clear: there was no plan to âgo offline.â This wasnât a scheduled maintenance window where we knowingly shut down the stream. It was backend work intended to strengthen the infrastructure, and unfortunately it triggered unexpected failures. Thatâs on us â but it wasnât a deliberate decision to allow dead air and there was no way to plan for something we had no idea would trigger such issues.
Youâre absolutely right that dead air is painful. William knows that better than anyone. We all do. The stream itself is sacred to us. The moment something goes down, it becomes all-hands-on-deck until itâs restored.
One thing that might not be visible from the outside is how lean we actually are. While there are people who contribute in various roles, our core technical infrastructure is maintained by essentially two engineers and one of those engineers is also the boss and main DJ. They are supporting a global, 24/7, streaming service that operates across web, mobile apps, CarPlay, Android Auto, smart speakers, and multiple audio formats â all without corporate backing, venture capital, or a broadcast network behind us.
We really are a mom-and-pop operation serving a massive worldwide audience. Thatâs part of the beauty â and part of the fragility.
The house analogy wasnât meant to imply neglect or known instability. It was meant to acknowledge that after 26 years of continuous evolution, systems accumulate complexity. Modernizing that complexity is necessary to keep RP viable long-term. Sometimes that means carefully touching foundational pieces. And sometimes, despite testing, something behaves differently in the wild than it does in staging.
Could we build more redundancy? Of course. We are actively working toward greater resilience. But that takes time and resources, and we grow those carefully and deliberately.
There were no ignored red flags. There was no cavalier decision-making. There was careful long hours of work that had unintended consequences â and a team that responded as quickly as possible.
As for revenue impact â yes, outages matter. Weâre fully aware of that. No one here treats this casually. This station is our livelihood and our lifeâs work.
Weâre constantly balancing:
⢠Stability
⢠Innovation
⢠Limited staffing
⢠Financial sustainability
⢠And a global audience that expects perfection
Itâs not corporate radio. Itâs not iHeart. Itâs not Spotify. Itâs a small, fiercely committed team trying to keep human-curated radio alive in a very automated world.
Your feedback is heard. And your expectations come from a place of wanting RP to thrive â which we share.
Weâre not perfect. But we are deeply committed.
Here's the thing though: William is a former radio DJ. He knows exactly how bad "dead air" is. You get a massive drop off in listeners, and it takes a while to recover. That translates into lost revenue.
When a commercial radio stations had to do planned maintenance to backend infrastructure, such as repairing or replacing a rack of audio equipment, they plan ahead with an alternate broadcast to avoid complete radio silence during the actual maintenance window going on behind the scenes. typically this would entail playing a long tape with pre-recorded music.
Alanna is framing this outage as cleaning up a house that's been lived in for 26 years. This definitely implies that the problems were known, and that some kind of maintenance window could've been created.. Jarred could have altered the API to play the same music on a loop for 24 hours along with a recurring announcement recorded by William or Alanna or Josh that certain functions would be unavailable (downloads, comments, voting, etc).
That none of these contingencies were planned or prepared, and that the maintenance went ahead means that there is a critical lack of planning and redundancy happening at Radio Paradise.
I'm a career IT professional; if I made a change like this without announcing it, I'd be seriously reprimanded. Possibly fired.
William is playing a bumper on the main mix where he touts that donations from loyal listeners have been able to find employment for 12 staff members who never have to look for another "real job" because working at Radio Paradise is amazing.
I know it's easy for me to critique the station from here... but I've actually been to the station at Eureka. On the surface, it looks to be a very well-oiled machine and a squeaky-clean operation. And it is abundantly clear that all the staff love the station and want it to keep going forever. Yet... out of those 12 people, did anyone raise ANY alarms or red flags?
This organization is too mature to be making "aw shucks" mistakes. The day long dead-air could translate into lost donation revenue.
I really appreciate you taking the time to write this â truly. Itâs obvious you care about Radio Paradise, and we donât take that lightly.
First, I want to be clear: there was no plan to âgo offline.â This wasnât a scheduled maintenance window where we knowingly shut down the stream. It was backend work intended to strengthen the infrastructure, and unfortunately it triggered unexpected failures. Thatâs on us â but it wasnât a deliberate decision to allow dead air and there was no way to plan for something we had no idea would trigger such issues.
Youâre absolutely right that dead air is painful. William knows that better than anyone. We all do. The stream itself is sacred to us. The moment something goes down, it becomes all-hands-on-deck until itâs restored.
One thing that might not be visible from the outside is how lean we actually are. While there are people who contribute in various roles, our core technical infrastructure is maintained by essentially two engineers and one of those engineers is also the boss and main DJ. They are supporting a global, 24/7, streaming service that operates across web, mobile apps, CarPlay, Android Auto, smart speakers, and multiple audio formats â all without corporate backing, venture capital, or a broadcast network behind us.
We really are a mom-and-pop operation serving a massive worldwide audience. Thatâs part of the beauty â and part of the fragility.
The house analogy wasnât meant to imply neglect or known instability. It was meant to acknowledge that after 26 years of continuous evolution, systems accumulate complexity. Modernizing that complexity is necessary to keep RP viable long-term. Sometimes that means carefully touching foundational pieces. And sometimes, despite testing, something behaves differently in the wild than it does in staging.
Could we build more redundancy? Of course. We are actively working toward greater resilience. But that takes time and resources, and we grow those carefully and deliberately.
There were no ignored red flags. There was no cavalier decision-making. There was careful long hours of work that had unintended consequences â and a team that responded as quickly as possible.
As for revenue impact â yes, outages matter. Weâre fully aware of that. No one here treats this casually. This station is our livelihood and our lifeâs work.
Weâre constantly balancing:
⢠Stability
⢠Innovation
⢠Limited staffing
⢠Financial sustainability
⢠And a global audience that expects perfection
Itâs not corporate radio. Itâs not iHeart. Itâs not Spotify. Itâs a small, fiercely committed team trying to keep human-curated radio alive in a very automated world.
Your feedback is heard. And your expectations come from a place of wanting RP to thrive â which we share.
Weâre not perfect. But we are deeply committed.
Weirdness!
The Beyond 128k AAC stream played 3 of Josh's channel jingles in a row, and after that, it was completely out of sync with the What's Playing list and the web player.
This went on for about an hour until I started writing this message to report it, and guess what⦠right before I was about to hit the Submit button, again 3 jingles were played and suddenly the stream was back in sync with the website!
RP is not a non profit organization. It is a listener supported for profit operation funded by voluntary contributions. People seem to have taken RP for granted based upon many of the comments I have been reading over the past weeks. That it has survived for as long as it has is a tribute to William's vision, skills and dedication to undertake and succeed in a way never before accomplished in "radio" history.
Be grateful, patient and understanding when dealing with your personal inconveniences. These disruptions are in no way planned or desired by anyone that I know of. There is no model for this operation. It is the prototype and hopefully always will be. Having been here all this time I could not even imagined where we would be looking back 25 years ago when I first tuned in on my Windows Me computer listening on winmx. And then wondering how long it would last. Fortunately Alanna has risen to the occasion to carry the torch forward.
Pioneers risk everything to undertake their journey. Just think where internet radio would be without B & R's vision, dedication and perserverence. These are growing pains so be grateful that RP is still growing and not rotting on the vine resting on its laurels. I am.
.
People who donate to nonprofit organizations expect them to be good stewards with their funds and do everything they can to keep the organization healthy and strong.
RP is not a non profit organization. It is a listener supported for profit operation funded by voluntary contributions. People seem to have taken RP for granted based upon many of the comments I have been reading over the past weeks. That it has survived for as long as it has is a tribute to William's vision, skills and dedication to undertake and succeed in a way never before accomplished in "radio" history.
Be grateful, patient and understanding when dealing with your personal inconveniences. These disruptions are in no way planned or desired by anyone that I know of. There is no model for this operation. It is the prototype and hopefully always will be. Having been here all this time I could not even imagined where we would be looking back 25 years ago when I first tuned in on my Windows Me computer listening on winmx. And then wondering how long it would last. Fortunately Alanna has risen to the occasion to carry the torch forward.
Pioneers risk everything to undertake their journey. Just think where internet radio would be without B & R's vision, dedication and perserverence. These are growing pains so be grateful that RP is still growing and not rotting on the vine resting on its laurels. I am.
.
Here's the thing though: William is a former radio DJ. He knows exactly how bad "dead air" is. You get a massive drop off in listeners, and it takes a while to recover. That translates into lost revenue.
When a commercial radio stations had to do planned maintenance to backend infrastructure, such as repairing or replacing a rack of audio equipment, they plan ahead with an alternate broadcast to avoid complete radio silence during the actual maintenance window going on behind the scenes. typically this would entail playing a long tape with pre-recorded music.
Alanna is framing this outage as cleaning up a house that's been lived in for 26 years. This definitely implies that the problems were known, and that some kind of maintenance window could've been created.. Jarred could have altered the API to play the same music on a loop for 24 hours along with a recurring announcement recorded by William or Alanna or Josh that certain functions would be unavailable (downloads, comments, voting, etc).
That none of these contingencies were planned or prepared, and that the maintenance went ahead means that there is a critical lack of planning and redundancy happening at Radio Paradise.
I'm a career IT professional; if I made a change like this without announcing it, I'd be seriously reprimanded. Possibly fired.
William is playing a bumper on the main mix where he touts that donations from loyal listeners have been able to find employment for 12 staff members who never have to look for another "real job" because working at Radio Paradise is amazing.
I know it's easy for me to critique the station from here... but I've actually been to the station at Eureka. On the surface, it looks to be a very well-oiled machine and a squeaky-clean operation. And it is abundantly clear that all the staff love the station and want it to keep going forever. Yet... out of those 12 people, did anyone raise ANY alarms or red flags?
This organization is too mature to be making "aw shucks" mistakes. The day long dead-air could translate into lost donation revenue.
My thoughts exactly. As a user and code contributor to the Lyrion Music Server open source project, I am very disappointed in the seemingly haphazard way that the RP API has repeatedly been broken, fixed, broken again, changed, and otherwise bounced around for the past month or so. This is not housecleaning. It is uncontrolled chaos. Many Lyrion users, including myself, are becoming very frustrated and I have no doubt that many are just moving on to other more stable, if less excellently curated, streaming sources. Please try to do better. We are all rooting for you.
I think that an in-depth "post-mortem" report would go a long way to helping the community understand what caused the outage, and gain a deeper appreciation of the complexity of RP, and whatever was being cleaned up under the hood.
In case anyone's resting on their laurels thinking it's all good now, I'm still getting disruptions. It happened on both the iOS app and Lyrion Music Server. Here's the error that the RP plugin for LMS just threw:
Plugins::RadioParadise::ProtocolHandler::new (29) We seem to be in a redirection loop for url: https://audio-geo.radioparadise.com/dj/4/10014873.flac
Edit: I will say that things are slooowly getting better, so it's at least going in the right direction. Trying not to sound like a whiny little bitch.
