Hosting a 24/7 livestream with a terminal

Introduction

I'm a huge fan of videogames, and one of my favorite titles of this year is Mewgenics, a turn based strategy game about… you guessed it, cats. The game comes with a fake radio station (called WMEW 99.9) that plays all the music tracks of the game plus some hilarious fake commercials and segments. You can listen to it while playing the game or (secret trick) by adding -radiomode true in the launch options.

On Youtube you can find videos of 6, 8 or even 10 hours of the radio playing, but it seems like no one ever considered hosting a 24/7 livestream. No one except me! I’ve never tried doing something similar, so it seemed like a fun experiment and a good learning opportunity. In this article I will explain how I managed to do that, as well as all the optimizations to try to keep operational costs at a minimum. Let’s go!

You can find the source code of this project here

The plan

First of all was figuring out where to stream. I immediately excluded any device in my house because my internet is... well...

As we’ll see, this is going to be more than enough to host the stream, but still, I don’t like the idea of keeping a laptop always turned on in my house anyway.

So cloud it is then. I decided to host an Ubuntu virtual machine in Azure since I opened an account recently and I have the 200$ credit that Microsoft gives the first month. However to make things challenging I decided to go with a relatively small machine with 2vCPU and 1 GB of ram. Believe it or not, in the end these specs will turn to be way too overkill :)

From here the options I had in front of me were two:

1. install a GUI library, RDP into the machine, install Steam, launch the game in radio mode and stream it
2. Extract the audio files from the game, recreate the radio and stream it.

Option 1 is what I would do with my home laptop, but on the cloud it’s pretty expensive: the specs of the VM need to be *decent* in order to run at the same time Steam, Mewgenics and the software for the stream. Let’s be honest, 1 GB of RAM probably isn’t enough…

From a cost optimization perspective, option 2 is much more promising. Plus, this opens up the possibility of customizing the radio, and think about it, we’re gonna host a livestream directly from a terminal. It might just be me, but I find that really cool.

Diving into the game assets

First of all, let’s look at the game directory. As you can see, it is surprisingly simple:

The game has been made using a custom in-house game engine, and that gigantic .gpak file contains all the assets (now I understand why the game freezes for 25 seconds everytime I boot it up lol). Luckily for us, there’s a tool on github called GPAK-Extractor, which does exactly what we need. After diving into the game assets for a while, I quickly stumbled across this “radio” directory, which contains all the .ogg audio files we need. Easy!

After this, I used Gemini to quickly build a python script that takes care of randomizing the order of the tracks to be played, as well as a simple webserver to help me fine-tune all the parameters. I won’t delve into the details of this, instead it is now time to figure out how to handle the streaming part.

No GUI? No problem!

When it comes to streaming I’ve always used OBS Studio, but since we’re on a terminal that’s obviously not an option. So I started looking for alternatives that do not require a graphical interface, and that’s when I discovered a tool called ffmpeg. Just like OBS, we need to provide a video and an audio source (with the -i option). To keep things simple (and as we’ll see later, cheap), for the video we can just use a simple jpg image. But for the audio, we have a problem: when streaming, Youtube requires an uninterrupted, real-time stream of data. But running a new ffmpeg process every time we change a track means breaking that connection, potentially ending the stream. How do we solve this?

First we need to understand how ffmpeg works behind the scenes. I’m simplifying a bit, but there’s essentially two steps:

1. Decoding: the .ogg file gets converted into a format called Raw PCM
2. Encoding: the decoded bytes are then converted into a format called AAC which is ultimately what Youtube servers want.

Knowing this, we can split the two parts: first we launch an ffmpeg instance that always stays on, only encodes the input it reads from stdin and sends it to youtube. Then, every time we need to switch tracks, we launch a second ffmpeg process that decodes the file, pipes it to the other process and then dies.

After including this logic in the python script, I moved all the files on the server and then created a systemd service to start and stop the radio.

A quick systemctl start and… the stream is working! That was easy!

Maybe too easy…

The stream worked like a charm… for the first 2 hours, after that it just stopped. So I stopped the service, and after running top I noticed something odd… an ffmpeg process was still running! Of course, there was only one logical explanation: stopping the service killed the python script, but not the master encoder, which kept running as a zombie process. It turns out I didn’t correctly configure the systemd service, and the solution was as simple as adding this line:

KillMode=control-group 

I restarted the service and everything started working again. That same evening, before going to bed, I checked the stream and… it was down.

The next day I log into the machine and run top again…

And to my surprise… there was an “unattended-upgrades” service just hanging around. Makes sense, I say to myself, the machine tried to update itself, memory ran out and everything crashed. As we’ll see later, I was wrong, but after killing the process and disabling automatic updates, the stream started working again!

The buffering problem

The stream now seemed to be a bit more stable, but when listening to it while working, I eventually noticed something weird and very annoying… it was starting to buffer. Of course, at first I pointed the finger at the usual suspect… my crappy internet. But after realizing that I had the same problem with mobile data (which, believe it or not, it’s usually not as bad), I realized there had to be something wrong with the server.

So I logged into the machine and ran my good old friend top one more time.

Well the process is running, no doubts about that but… wait a second… why is it using 138% CPU??

Time to look at the metrics of my machine…

The machine I chose for this project is burstable, which means it allows you to use more processing power than you rented. But by doing that you consume CPU credits, and when they reach zero, the CPU gets throttled. And this is exactly the root cause of the buffering, and also explains why stopping the machine for a while seemed to temporarily fix the problem. As you can see in the graph above, the credits simply recharge over time.

So it’s time to optimize CPU usage! It turns out that the main bottleneck is video encoding: right now we’re encoding a still image at 1080p 30 times a second. That’s a bit of an overkill! To fix this, I lowered the resolution to 720p and the frames per second to just 2. I also changed the “video preset” option of ffmpeg from veryfast to ultrafast, and that seemed to help as well.

I modified my script and restarted the radio. After hours of streaming, there were still no buffering issues, and the metrics showed the credits recharging, which means we solved the problem!

Now that all the technical issues are solved, there’s still one more variable to consider that so far, we completely ignored…

One last giga-issue

In a cloud environment, every byte that goes from your machine to the internet has a cost. Now if you’re building something like a web application you can usually just ignore it, but in our case, we’re pumping out data every single second. And so we probably would like to know how much that costs.

Let’s look at the metrics:

With the current configuration, we are putting out 450 MB of data every hour. That’s approximately 324 GB of data every month! Considering that on average a GB costs 0.07$, that’s around 23$ of network traffic.

Just for fun, let’s see how much we can bring that number down!

First, let’s break it down even more. 450 MB per hour means that every second we are sending 125 KB of data.

In the configuration right now we have set the audio quality to 192kbps, which divided by 8, makes 24 KB/s.

Now we could decrease the quality to 128kbps (16 KB/s), but that comes with the risk of introducing sound artifacts (which to be honest for a radio it’s awful), and we would only save around 28 MB per hour.

Instead I think it’s a much better option to decrease the video resolution to 144p which is the minimum that youtube allows.

So I applied the new configuration and waited a couple of days…

That’s a huge drop! 121 MB per hour means 87 GB per month, so around $6. At this point, I can say my optimization itch has been finally scratched :)

Conclusions

Unfortunately, the stream won’t stay up for much longer. Azure’s first month trial is almost over, and the data costs still go way over the free tier of the first 12 months (which is 15 GB per month). And, let’s face it, this radio is still pretty niche, people don’t really look it up that much. Still, I had a great time putting this up, solving the issues and applying all the optimizations. I learned a lot, and I hope you have too! Thanks for reading!