What Happened to Facebook, Instagram, & WhatsApp?

What Happened to Facebook, Instagram, & WhatsApp?
Share this article with your friends!

Facebook, Instagram, WhatsApp, and Messenger are back online, after the platforms suffered a massive global outage on Monday, affecting tens of millions of users around the world.

The Facebook-owned platforms all crashed early on Monday evening, blocking users from accessing their services.

Facebook’s internal systems used by employees also reportedly went down.

WhatsApp users on both iPhone and Android could not make or receive phone or video calls or send text messages.

Facebook said late on Monday that “the root cause of this outage was a faulty configuration change” and that there is “no evidence that user data was compromised as a result” of the outage.

internet traffic served for facebook

Facebook said late on Monday that “the root cause of this outage was a faulty configuration change” and that there is “no evidence that user data was compromised as a result” of the outage.

The company apologized and said it is working to understand more about the cause:

Widespread outages

Downdetector.com showed outages in densely populated areas like London, Paris and Paris.

The site, which provides real-time status reports for a range of online platforms and services, 73,804 problems with WhatsApp had been recorded in the UK alone in a spike at 5:53 pm CET.

In the US, 36,686 reports were recorded.

Many Facebook users were shown messages including “This site is inaccessible” and “Impossible to find the server address”.

According to Downdector, 57,690 reports of issues were recorded around the same time in the UK, of which the vast majority (around 63 percent) were related to access to the website and a further 25 percent to the app.

In the US, 127,726 reports were recorded.

People who use their Facebook credentials to access third-party apps, including Pokemon Go and Match Masters, were also hit by the company being offline.

Localized outages are not uncommon but global outages are generally considered rare.

The last major internet outage, which crippled many of the world’s leading websites, including Euronews and other media outlets, was in June and lasted an hour.

Content delivery company Fastly blamed it on an undiscovered software bug.

 

What was the likely cause?

Doug Madory, director of internet analysis for Kentik Inc, told the Associated Press that he believes the outage was caused by Facebook withdrawing “authoritative DNS routes” that let the rest of the Internet communicate with its properties.

Such routes are part of the Internet’s Domain Name System, a central component of the internet that directs its traffic. Without Facebook broadcasting its routes on the public Internet, apps and web addresses simply could not locate it.

Jake Williams, chief technical officer of the cybersecurity firm BreachQuest, said that while foul play cannot be completely ruled out, chances were good that the outage is “an operational issue” caused by human error.

Facebook in crisis

Facebook, which has nearly 3 billion monthly users worldwide, is going through one of its worst reputation crises in a fortnight due to revelations by a whistleblower.

Frances Haugen, a former Facebook product engineer, leaked numerous internal documents in the past week, including to the Wall Street Journal.

She further accused the company of “(choosing) profit over the safety” of its users, in an interview broadcast by CBS on Sunday.

On Wall Street, Facebook’s share price, already down at the beginning of the session, accelerated its losses on Monday and fell by nearly 6 per cent, shedding more than $50 billion (€43 billion) in value.

Updates:

Update, 4:37 p.m. ET: Sheera Frenkel with The New York Times tweeted that Facebook employees told her they were having trouble accessing Facebook buildings because their employee badges no longer worked. That could be one reason this outage has persisted so long: Facebook engineers may be having trouble physically accessing the computer servers needed to upload new BGP records to the global Internet.

Update, 6:16 p.m. ET: A trusted source who spoke with a person on the recovery effort at Facebook was told the outage was caused by a routine BGP update gone wrong. The source explained that the errant update blocked Facebook employees — the majority of whom are working remotely — from reverting the changes. Meanwhile, those with physical access to Facebook’s buildings couldn’t access Facebook’s internal tools because those were all tied to the company’s stranded domains.

Update, 7:46 p.m. ET: Facebook says its domains are slowly coming back online for most users. In a tweet, the company thanked users for their patience, but it still hasn’t offered any explanation for the outage.

Update, 8:05 p.m. ET: This fascinating thread on Hacker News delves into some of the not-so-obvious side effects of today’s outages: Many organizations saw network disruptions and slowness thanks to billions of devices constantly asking for the current coordinates of Facebook.com, Instagram.com and WhatsApp.com. Bill Woodcock, executive director of the Packet Clearing Housesaid his organization saw a 40 percent increase globally in wayward DNS traffic throughout the outage.

Update, 8:32 p.m. ET: Cloudflare has published a detailed and somewhat technical writeup on the BGP changes that caused today’s outage. Still no word from Facebook on what happened.

Update, 11:32 p.m. ET: Facebook published a blog post saying the outage was the result of a faulty configuration change:

“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” Facebook’s Santosh Janardhan wrote. “This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”

“We want to make clear at this time we believe the root cause of this outage was a faulty configuration change,” Janardhan continued. “We also have no evidence that user data was compromised as a result of this downtime.”

JOIN OUR NEWSLETTER
Join over 1.000 visitors who are receiving our newsletter and get free eBooks, breaking news, learn how secure your data, your company accounts, database, clients passwords, get the best security advice and more.
We hate spam. Your email address will not be sold or shared with anyone else.

Share this article with your friends!