I worked in a start-up that enabled live broadcasts on the Internet, a personal "blog" that broadcasts live. All you need is a computer and a camera.
At first there was a possibility to broadcast, then we added a possibility to record the broadcast. As part of the testing process, we checked the quality of the broadcasts and the recordings, and we had all kinds of methods to check them. We either broadcasted and recorded a clock or simply broadcasted movies.
Over time, we added a new feature: the main broadcaster, or host, could add and manage a secondary broadcaster.
We could still record, when in fact, both recordings were saved as two files, in one folder.
The folder structure included the 'father' folder, named as the name of the main broadcaster, and each specific recording was saved in a sub folder which the name included the recording end time.
The viewers could choose to see those recording synchronized just as they were in the live envirobnent.
We checked the synchronization in the recordings between the host and the secondary channel and many other tests. Finally we were really close to releasing the version.
We started to record some tapes in the staging environment, the day before and in the release day, just in case. Everything looked fine, and I was ready to give the confirmation of the tests. Suddenly, Lena, one of the testers, looked at a recording and said a sentence that changed everything:
"But my shirt is blue." "what?" I asked, not understanding why we went to talk about fashion before releasing such an important version.
"My shirt is blue, but as the secondary broadcaster on the tape I'm in a green shirt," she added. "That's the shirt I wore yesterday."
It turned out that some cases even though the recordings were kept in the right place (we checked!), the secondary broadcaster tape the server has taken the recorded file from another library, an early recording, if available. In the case we recorded on that particular user also a day before with a secondary broadcaster, and so the problem was flooded.
Could we think of such a scenario of checking a secondary broadcaster recording when there is already a library with a secondary broadcaster record? I do not know, but I know one thing: in the reality of the production, it would have been very quickly revealed, because the broadcast with the sub-broadcast and its recording were common, and in reality you would immediately recognize that today's sub-broadcast is actually that of yesterday.