At stack.io, we love to have a little fun around the holidays! For Halloween, we put up a few spooky decorations and shared some of our favourite spooky stories in the tech realm. Here are some of our favourites.
500 Mile Mail
For some reason, a campus email system could not send emails further than 500 miles...
A Story About ‘Magic’
A suspicious switch at the MIT AI Lab did something it wasn’t supposed to do...
Bonus
A VM Farms original from our parent company
One of the most common horror stories that our CEO, Hany Fahim, said he faced a lot during the early days of VM Farms is something is wrong, but there’s no error!
One day, a client contacted the VM Farms support team saying that their site is down. The VM Farms support member loaded the website, checked their monitoring dashboards, and did all their precursory checkups, but everything was GREEN.
They decided to have another member of the team log on to double check that it wasn’t an isolated issue. The site was loading for them too! Why was it working for some people, but not others?
After further investigating the VPN, testing the connection with telnet, sniffing the traffic using tcpdump, among other debugging techniques, the VM Farms support member suddenly realized that this client had a different upstream provider from VM Farm’s other clients!
They phoned the upstream provider and found out that while making updates, the provider did not configure the host to true passive. Instead, they set the host to active with very high thresholds, which they thought no customer would pass. Well...one of VM Farm’s customers did!
Some lessons that the team members at VM Farms learned were:
Turn on monitoring timeouts with high thresholds
Get notified for all major maintenance events by the upstream provider
Errors are no fun… spooky even, but if you keep looking, you’ll eventually find the culprit!
What’s your spookiest infrastructure story? Let us know.