You know you’re in trouble when a “Good Day” consists of saying “Well, the database only crashed once, but since we have a caching proxy almost no users will have noticed. ”
Of course, when that’s a Good Day … Bad Days are freakin’ hilarious
For instance, one day the network router connecting the incoming pipe … got turned off?
The main web server … got rebooted? and didn’t start the web server up? wtf?
The database starts crashing every twenty minutes … and starts eating data occasionally …
Mind you, the guys who got tasked with beating the test servers within an inch of their lives to figure out the crashes seemed to be enjoying themselves!