So here we go: In the past one and a half weeks, I’ve set up and then started running a BigBlueButton video conferencing server ‘pro bono publico’ in my quality time for a faculty of a non-technical institution. It’s been an exciting week for many reasons. The main one: Like many projects, the number of people who wanted to use it grew quite quickly.
Initially, I started working on a solution that would enable 3 to 4 lecturers throughout the week for some sessions. This quickly grew into 10 lecturers with some sessions in parallel. In the end, 16 lecturers showed up in the first orga call. A quick poll indicated the requirement for a system that could handle up to 5 parallel sessions with 60 people. On top, video between all people was seen as essential. From earlier experiments I knew that a 4-vCore VM could (only) handle a session with up to 20 people with the same number of videos between them. O.k., challenge accepted.
N:N Video Creates a Lot of Streams
When looking at the server load of previous calls and also from a logical point of view, I got the impression that the main capacity driver is not the number of concurrent sessions but the number of people per session and the requirement that everybody should see the video of everyone else in each session. So running 3 sessions with 20 people inside requires 3 * 19 * 20 = 1140 concurrent video streams while 5 sessions with 20 + 10 + 10 + 10 + 10 people only requires 740 video streams. In terms of bandwidth and CPU capacity, that is a big difference!
I Need Those 16 CPUs
So based on these thoughts I calculated the size of the VM I would need and came up with a minimum of 8 vCPUs for my VM. To be on the save side, I doubled the number and scaled-up my setup to a 16 dedicated vCPU VM in Hetzner’s data center in Nürnberg and 1 Gbit/s network connectivity. That’s quite a bit of a setup and costs 166 Euros a month.
Monday Is Coming
O.k., so Monday came and I held my breath. And indeed, by 10 a.m. I had 5 sessions running in parallel with around 50-60 people in total, one with 20 people and the others with 8-10 people each. Most but not all people had their video activated. Those who didn’t had old devices and were struggling a bit. More about that in another post. I’ve put together a rough graph that shows the average CPU load over 16 CPU cores in percent in the top graph from 8 am to 10 pm. The bottom graph shows the network throughput in Mbit/s. The peak sustained incoming datarate was somewhere around 14 Mbit/s, while the sustained outgoing data rate was around 120 Mbit/s.
As most people left the video quality at “medium” when activating their web cam, each uplink video generated 0,35 Mbit/s of traffic, i.e. 2,85 videos per Mbit/s of inbound traffic. Therefore, the 14 Mbit/s inbound traffic was generated by roughly 40 sources. As there were more users, some without video, this is only a rough estimation, but pretty close to reality.
Applying the same ratio to the 120 Mbit/s in the downlink direction results in around 342 simultaneously outgoing video streams. That’s lower than I expected in my theoretical calculation above.
Session Experience
As my setup can still be considered rather small, I knew all lecturers and joined some of their sessions over the week to see how things were going. For those people with good hardware, including me, the experience was excellent. Two hour sessions worked without interruptions, and sound and video quality was excellent. When joining a call, it takes only a few seconds to recognize which people on the call had old notebooks, smartphones or tablets as their videos were not as clear and fluid than those from people with better hardware. Also, some people seemed to have connectivity issues. I’m not exactly sure why, because even in calls with 20 people and 15 videos, the downlink data rate never exceeded 3.5 Mbit/s and even with the video camera switched on, the uplink data rate is a relatively low 0.35 Mbit/s. Even low end DSL lines should be able to handle this. Unless, of course, other people in the household use the network simultaneously…
So based on these numbers and my usage scenario, i.e. number of simultaneous sessions with sizes between 10 and 20 people, with video switched on between most of them, my €160 per month server can likely handle 80 people simultaneously. While that is enough for my purposes with a good margin, the number is far lower than the 200 people given for an 8 core setup in the BBB FAQs. The difference, as far as I can tell, is because the FAQ number assumes a far lower number of simultaneous video flows. That said, I am very happy about the outcome of week one. The server I have setup-up is running perfectly, 60 teachers and students are back working and studying and the gloomy faces when the project started have given way to smiles and fun during calls.
Old Hardware Is A Challenge
While I don’t want to end this post with a negative thought, there is one thing I need to mention though: Around 10% of the people who wanted to use the system have hopelessly old and under-powered hardware. Most of them didn’t have a computer but only an old smartphone or tablet that can’t handle the videos very well. Most of these old devices just drop out of the call after a short time. Videos transmission and reception can be switched-off in the BBB client so audio is not a problem for most of these people. However, seeing the video of at least the lecturer is essential for them so that is not much help to them. So we are working on this and hope that in the next couple of days we can enable as many of them as possible.
I think in some cases, the video / speech drop out problems could be related to poor WiFi – people may have 50 Mbit/s to the router, but once they take their laptop two rooms or upstairs away to the spare bedroom they loose a lot of quality. Add to the fact that in a large apartment building there might be a lot of interference on the WiFi frequencies at the moment, and this can result in even more issues.
Not everyone has a router / repeater per floor – in my case Ethernet from the kitchen to the first floor, and (after one day of working from home), a power line adaptor to the hobby room in the cellar keeps me in quality coverage!
Hi Mark, yes, that is definitely a factor as well. That’s why my first recommendation is to try it in the same room as the Wifi AP. If that doesn’t help, the next thing I recommend is to reboot the router. That has helped one or two persons. In most cases, unfortunately, it is not the Wifi but the devices are just too old and under-powered.