Social media giant, Facebook, has finally released a statement regarding the sudden outage that happened for more than six hours last Monday, October 4, which caused inconvenience among users.
In a Twitter post made by the engineering department of Facebook, they disclose the reason behind the collapse of the global system. “Now that our services are fully restored, we are sharing more details about yesterday’s outage,” said Facebook.
Now that our services are fully restored, we are sharing more details about yesterday's outage. https://t.co/SwDmBGZaZU
— Facebook Engineering (@fb_engineering) October 5, 2021
According to the statement released by Santosh Janardhan, Facebook Vice President of Engineering, he explained the company’s engineers issued a command which unintentionally disconnected Facebook data centers from the rest of the world.
“During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally,” he said.
The company added that they also struggle amid the recovery of the system since the sudden modification caused a complete disconnection on their server connections between their data centers and the internet which made things worse.
“All of this happened very fast. And as our engineers worked to figure out what was happening and why, they faced two large obstacles: first, it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this.” Janardhan continued.
Janardhan shared the failure that happened in their company will serve as an opportunity to learn and get better, “After every issue, small and large, we do an extensive review process to understand how we can make our systems more resilient. That process is already underway.” he concluded.
Meanwhile, Facebook’s CEO, Mark Zuckerberg apologized for the sudden disruption on their system saying he is aware of how much people rely on their services.
Moreover, Zuckerberg posted a note that he wrote for his company in his official Facebook account.
Source: https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/, https://twitter.com/fb_engineering, https://www.facebook.com/zuck