Gone in Minutes, Out for Hours: Global Outage Shakes Facebook

Facebook‌ ‌and‌ ‌its‌ ‌family‌ ‌of‌ ‌apps,‌ ‌including‌ ‌Instagram‌ ‌and‌ ‌WhatsApp,‌ ‌were‌ ‌inaccessible‌ ‌for‌ ‌hours‌ ‌today,‌ ‌taking‌ ‌out‌ ‌a‌ vital‌ ‌communications‌ ‌platform‌ ‌used‌ ‌by‌ ‌billions‌ ‌and‌ ‌showcasing‌ ‌just‌ ‌how‌ ‌dependent‌ ‌the‌ ‌world‌ ‌has‌ ‌become‌ ‌on‌ ‌a‌ ‌company‌ ‌that‌ ‌is‌ ‌under‌ ‌intense‌ ‌scrutiny.‌ ‌

Facebook’s‌ ‌apps‌ ‌—‌ ‌which‌ ‌include‌ ‌Facebook,‌ ‌Instagram,‌ ‌WhatsApp,‌ ‌Messenger‌ ‌and‌ ‌Oculus‌ ‌—‌ ‌began‌ ‌displaying‌ ‌error‌ messages‌ ‌around‌ ‌8:40am,‌ ‌users‌ ‌reported.‌ ‌

 

Within‌ ‌minutes,‌ ‌Facebook‌ ‌had‌ ‌disappeared‌ ‌from‌ ‌the‌ ‌internet.‌ ‌The‌ ‌Oct. 4 outage‌ ‌lasted‌ ‌over‌ ‌five‌ ‌hours,‌ ‌before‌ ‌some‌ ‌apps‌ ‌slowly‌ ‌flickered‌ ‌back‌ ‌to‌ ‌life,‌ ‌though‌ ‌the‌ ‌company‌ ‌cautioned‌ ‌the‌ ‌services‌ ‌would‌ ‌take‌ ‌time‌ ‌to‌ ‌stabilize.‌ ‌

Even‌ ‌so,‌ ‌the‌ ‌impact‌ ‌was‌ ‌far-reaching‌ ‌and‌ ‌severe.‌ ‌Facebook‌, headquartered in Menlo Park, ‌has‌ ‌built‌ ‌itself‌ ‌into‌ ‌a‌ ‌linchpin‌ ‌platform‌ ‌with‌ ‌messaging,‌ ‌livestreaming,‌ ‌virtual‌ ‌reality‌ ‌and‌ ‌many‌ ‌other‌ ‌digital‌ ‌services.‌ ‌In‌ ‌some‌ ‌countries,‌ ‌like‌ ‌Myanmar‌ ‌and‌ ‌India,‌ ‌Facebook‌ ‌is‌ ‌synonymous‌ ‌with‌ ‌the‌ ‌internet.‌

More‌ ‌than‌ ‌‌3.5‌ ‌billion‌ ‌people‌‌ ‌around‌ ‌the‌ ‌world‌ ‌use‌ ‌Facebook,‌ ‌Instagram,‌ ‌Messenger‌ ‌and‌ WhatsApp‌ ‌to‌ ‌communicate‌ ‌with‌ ‌friends‌ ‌and‌ ‌family,‌ ‌distribute‌ ‌political‌ ‌messaging,‌ ‌and‌ ‌expand‌ ‌their‌ ‌businesses‌ ‌through‌ ‌advertising‌ ‌and‌ ‌outreach.‌ ‌Facebook‌ ‌is‌ ‌also‌ ‌used‌ ‌to‌ ‌sign‌ ‌in‌ ‌to‌ ‌many‌ ‌other‌ ‌apps‌ ‌and‌ ‌services,‌ ‌leading‌ ‌to‌ ‌unexpected‌ ‌domino‌ ‌effects‌ ‌such‌ ‌as‌ ‌people‌ ‌not‌ ‌being‌ ‌able‌ ‌to‌ ‌log‌ ‌into‌ ‌shopping‌ ‌websites‌ ‌or‌ ‌sign‌ ‌into‌ ‌their‌ ‌smart‌ ‌TVs,‌ ‌thermostats‌ ‌and‌ ‌other‌ ‌internet-connected‌ ‌devices.‌ ‌

Technology‌ ‌outages‌ ‌are‌ ‌not‌ ‌uncommon,‌ ‌but‌ ‌to‌ ‌have‌ ‌so‌ ‌many‌ ‌apps‌ ‌go‌ ‌dark‌ ‌from‌ ‌the‌ ‌world’s‌ ‌largest‌ ‌social‌ ‌media‌ ‌company‌ ‌at‌ the‌ ‌same‌ ‌time‌ ‌was‌ ‌highly‌ ‌unusual.‌ ‌Facebook’s‌ ‌last‌ ‌significant‌ ‌outage‌ ‌was‌ ‌in‌ ‌2019,‌ ‌‌when‌ ‌a‌ ‌technical‌ ‌error‌ ‌affected‌ ‌its‌ ‌sites‌ ‌for‌ ‌24‌ ‌hours‌,‌ ‌in‌ ‌a‌ ‌reminder‌ ‌that‌ ‌a‌ ‌snafu‌ ‌can‌ ‌cripple‌ ‌even‌ ‌the‌ ‌most‌ ‌powerful‌ ‌internet‌ ‌companies.‌ ‌

This‌ ‌time,‌ ‌‌Facebook‌ ‌said‌‌ ‌late‌ ‌today,‌ ‌the‌ ‌culprit‌ ‌was‌ ‌changes‌ ‌to‌ ‌its‌ ‌underlying‌ ‌internet‌ ‌infrastructure‌ ‌that‌ ‌coordinates‌ ‌the‌ traffic‌ ‌between‌ ‌its‌ ‌data‌ ‌centers.‌ ‌That‌ ‌interrupted‌ ‌communications‌ ‌and‌ ‌cascaded‌ ‌to‌ ‌other‌ ‌data‌ ‌centers,‌ ‌“bringing‌ ‌our‌ ‌services‌ ‌to‌ ‌a‌ ‌halt,”‌ ‌the‌ ‌company‌ ‌said.‌ ‌

Facebook‌ ‌eventually‌ ‌restored‌ ‌service‌ ‌after‌ ‌a‌ ‌team‌ ‌got‌ ‌access‌ ‌to‌ ‌its‌ ‌server‌ ‌computers‌ ‌at‌ ‌a‌ ‌data‌ ‌center‌ ‌in‌ ‌Santa‌ ‌Clara,‌ ‌three‌ ‌people‌ ‌with‌ ‌knowledge‌ ‌of‌ ‌the‌ ‌matter‌ ‌said.‌ ‌Then‌ ‌they‌ ‌were‌ ‌able‌ ‌to‌ ‌reset‌ ‌them.‌

The‌ ‌company‌ ‌apologized‌ ‌for‌ ‌the‌ ‌outage.‌ ‌“We’re‌ ‌sorry,”‌ ‌it‌ ‌‌said‌ ‌on‌ ‌Twitter‌‌ ‌after‌ ‌its‌ ‌apps‌ ‌started‌ ‌
becoming‌ ‌accessible‌ ‌again.‌ ‌“Thank‌ ‌you‌ ‌for‌ ‌bearing‌ ‌with‌ ‌us.”‌ ‌

The‌ ‌outage‌ ‌added‌ ‌to‌ ‌Facebook’s‌ ‌mounting‌ ‌difficulties.‌ ‌For‌ ‌weeks,‌ ‌the‌ ‌company‌ ‌has‌ ‌been‌ ‌under‌ ‌fire‌ ‌related‌ ‌to‌ ‌a‌ ‌whistle-blower,‌ ‌‌Frances‌ ‌Haugen‌,‌ ‌a‌ ‌former‌ ‌Facebook‌ ‌product‌ ‌manager‌ ‌who‌ ‌amassed‌ ‌thousands‌ ‌of‌ ‌pages‌ ‌of‌ ‌internal‌ ‌research.‌ ‌She‌ ‌has‌ ‌since‌ ‌distributed‌ ‌the‌ ‌cache‌ ‌to‌ ‌the‌ ‌news‌ ‌media,‌ ‌lawmakers‌ ‌and‌ ‌regulators,‌ ‌revealing‌ ‌that‌ ‌Facebook‌ ‌knew‌ ‌of‌ ‌many‌ ‌harms‌ ‌that‌ ‌its‌ ‌services‌ ‌were‌ ‌causing,‌ ‌including‌ ‌that‌ ‌Instagram‌ ‌made‌ ‌teenage‌ ‌girls‌ ‌feel‌ ‌worse‌ ‌about‌ ‌themselves.‌ ‌

The‌ ‌revelations‌ ‌have‌ ‌prompted‌ ‌an‌ ‌outcry‌ ‌among‌ ‌regulators,‌ ‌lawmakers‌ ‌and‌ ‌the‌ ‌public.‌ Haugen,‌ ‌who‌ ‌revealed‌ ‌her‌ ‌identity‌ ‌on‌ Sunday‌ ‌online‌ ‌and‌ ‌on‌ ‌“60‌ ‌Minutes,”‌ ‌is‌ ‌scheduled‌ ‌to‌ ‌testify‌ ‌Tuesday, Oct. 5, ‌in‌ ‌Congress‌ ‌about‌ ‌Facebook’s‌ ‌impact‌ ‌on‌ ‌young‌ ‌users.‌ ‌“Today’s‌ ‌outage‌ ‌brought‌ ‌our‌ ‌reliance‌ ‌on‌ ‌Facebook‌ ‌—‌ ‌and‌ ‌its‌ ‌properties‌ ‌like‌ ‌WhatsApp‌ ‌and‌ ‌ Instagram‌ ‌—‌ ‌into‌ ‌sharp‌ ‌relief,”‌ ‌said‌ ‌‌Brooke‌ ‌Erin‌ ‌Duffy‌,‌ ‌a‌ ‌professor‌ ‌of‌ ‌communications‌ ‌at‌ ‌Cornell‌ ‌ University.‌

“The‌ ‌abruptness‌ ‌of‌ ‌today’s‌ ‌outage‌ ‌highlights‌ ‌the‌ ‌staggering‌ ‌level‌ ‌of‌ ‌precarity‌ ‌that‌ ‌structures‌ ‌our‌ ‌increasingly‌ ‌digitally‌ ‌mediated‌ ‌work‌ ‌economy.”‌ ‌

When‌ ‌the‌ ‌outage‌ ‌began‌ ‌on‌ ‌this morning,‌ ‌Facebook‌ ‌and‌ ‌Instagram‌ ‌users‌ ‌quickly‌ ‌turned‌ ‌to‌ ‌Twitter‌ ‌to‌ ‌lament‌ ‌and‌ ‌poke‌ ‌fun‌ ‌at‌ ‌their‌ ‌inability‌ ‌to‌ ‌use‌ ‌the‌ ‌apps.‌ ‌The‌ ‌hashtag‌ ‌#facebookdown‌ ‌also‌ ‌started‌ ‌trending.‌ ‌Memes‌ ‌about‌ ‌the‌ ‌incident‌ ‌proliferated.‌ ‌
But‌ ‌a‌ ‌real‌ ‌toll‌ ‌soon‌ ‌emerged,‌ ‌because‌ ‌many‌ ‌people‌ ‌worldwide‌ ‌rely‌ ‌on‌ ‌the‌ ‌apps‌ ‌to‌ ‌conduct‌ ‌their‌ ‌daily‌ ‌lives.‌ ‌
“With‌ ‌Facebook‌ ‌being‌ ‌down‌ ‌we’re‌ ‌losing‌ ‌thousands‌ ‌in‌ ‌sales,”‌ ‌said‌ ‌Mark‌ ‌Donnelly,‌ ‌a‌ ‌start-up‌ ‌founder‌ ‌in‌ ‌Ireland‌ ‌who‌ ‌‌runs‌ HUH‌ ‌Clothing‌,‌ ‌a‌ ‌fashion‌ ‌brand‌ ‌focused‌ ‌on‌ ‌mental‌ ‌health‌ ‌that‌ ‌uses‌ ‌Facebook‌ ‌and‌ ‌Instagram‌ ‌to‌ ‌reach‌ ‌customers.‌ ‌“It‌ ‌may‌ ‌not‌ sound‌ ‌like‌ ‌a‌ ‌lot‌ ‌to‌ ‌others,‌ ‌but‌ ‌missing‌ ‌out‌ ‌on‌ ‌four‌ ‌or‌ ‌five‌ ‌hours‌ ‌of‌ ‌sales‌ ‌could‌ ‌be‌ ‌the‌ ‌difference‌ ‌between‌ ‌paying‌ ‌the‌ ‌electricity‌ bill‌ ‌or‌ ‌rent‌ ‌for‌ ‌the‌ ‌month.”‌ ‌
Samir‌ ‌Munir,‌ ‌who‌ ‌owns‌ ‌a‌ ‌food-delivery‌ ‌service‌ ‌in‌ ‌Delhi,‌ ‌said‌ ‌he‌ ‌was‌ ‌unable‌ ‌to‌ ‌reach‌ ‌clients‌ ‌or‌ ‌fulfill‌ ‌orders‌ ‌because‌ ‌he‌ ‌runs‌ the‌ ‌business‌ ‌through‌ ‌his‌ ‌Facebook‌ ‌page‌ ‌and‌ ‌takes‌ ‌orders‌ ‌via‌ ‌WhatsApp.‌ ‌“Everything‌ ‌is‌ ‌down,‌ ‌my‌ ‌whole‌ ‌business‌ ‌is‌ ‌down,”‌ ‌he‌ ‌said.‌ ‌

Douglas‌ ‌Veney,‌ ‌a‌ ‌gamer‌ ‌in‌ ‌Cleveland‌ ‌who‌ ‌goes‌ ‌by‌ ‌GoodGameBro‌ ‌and‌ ‌who‌ ‌is‌ ‌paid‌ ‌by‌ ‌viewers‌ ‌and‌ ‌subscribers‌ ‌on‌ ‌Facebook‌ Gaming,‌ ‌said,‌ ‌“It’s‌ ‌hard‌ ‌when‌ ‌your‌ ‌primary‌ ‌platform‌ ‌for‌ ‌income‌ ‌for‌ ‌a‌ ‌lot‌ ‌of‌ ‌people‌ ‌goes‌ ‌down.”‌ ‌He‌ ‌called‌ ‌the‌ ‌situation‌ ‌“scary.”‌ ‌

Inside‌ ‌Facebook,‌ ‌workers‌ ‌also‌ ‌scrambled‌ ‌because‌ ‌their‌ ‌internal‌ ‌systems‌ ‌stopped‌ ‌functioning.‌ ‌

The‌ ‌company’s‌ ‌global‌ ‌security‌ ‌team‌ ‌“was‌ ‌notified‌ ‌of‌ ‌a‌ ‌system‌ ‌outage‌ ‌affecting‌ ‌all‌ ‌Facebook‌ ‌internal‌ ‌systems‌ ‌and‌ ‌tools,”‌ ‌according‌ ‌to‌ ‌an‌ ‌internal‌ ‌memo‌ ‌sent‌ ‌to‌ ‌employees‌ ‌and‌ ‌shared‌ ‌with‌ ‌The‌ ‌New‌ ‌York‌ ‌Times.‌ ‌Those‌ ‌tools‌ ‌included‌ ‌security‌ systems,‌ ‌an‌ ‌internal‌ ‌calendar‌ ‌and‌ ‌scheduling‌ ‌tools,‌ ‌the‌ ‌memo‌ ‌said.‌ ‌

Employees‌ ‌said‌ ‌they‌ ‌had‌ ‌trouble‌ ‌making‌ ‌calls‌ ‌from‌ ‌work-issued‌ ‌cellphones‌ ‌and‌ ‌receiving‌ ‌emails‌ ‌from‌ ‌people‌ ‌outside‌ ‌the‌ ‌company.‌ ‌Facebook’s‌ ‌internal‌ ‌communications‌ ‌platform,‌

Workplace, was also taken out, leaving many unable to do their jobs. Some turned to other platforms to communicate, including LinkedIn and Zoom as well as Discord chat rooms.

Some Facebook employees who had returned to working in the office were also unable to enter buildings and conference rooms because their digital badges stopped working. Security engineers said they were hampered from assessing the outage because they could not get to server areas.

Facebook’s global security operations center determined the outage was “a HIGH risk to the People, MODERATE risk to Assets and a HIGH risk to the Reputation of Facebook,” the company memo said.

A small team of employees was soon dispatched to Facebook’s Santa Clara data center to try a “manual reset” of the company’s servers, according to an internal memo.

Several Facebook workers called the outage the equivalent of a “snow day,” a sentiment that was publicly echoed by Adam Mosseri, the head of Instagram.

In Facebook’s early days, the site experienced occasional outages as millions of new users flocked to the network. Over the years, it spent billions of dollars to build out its infrastructure and services, spinning up enormous data centers in cities including Prineville, Ore., and Fort Worth.

The company has also been trying to integrate the underlying technical infrastructure of Facebook, WhatsApp and Instagram for several years.

John Graham-Cumming, the chief technology officer of Cloudflare, a web infrastructure company that helps direct traffic to Facebook, said in an interview that his company became aware of the outage early on and saw the incident’s scope. He described the issue as a “misconfiguration.”

“It was as if Facebook just said, ‘Goodbye, we’re leaving now,’” he said.

Mike‌ ‌Isaac‌‌ ‌and‌ ‌‌Sheera‌ ‌Frenkel‌ ‌are technology reporters with The New York Times. Ryan Mac, Nicole Perlroth and Kellen Browning contributed reporting. Copyright, The New York Times.

9 Comments

  1. facebook is bad for you, your business, and your kids

    count this as a blessing and remove yourself from its evil web

  2. Nationalize Google, Facebook and Twitter and use their assets to re-establish locally-based, locally-controlled digital newspapers (or public information and discussion facilities) to replace the local media that these rapacious and voracious corporate actors have systematically destroyed in the past quarter-century. Let’s reconstruct the commons in the digital age with local, participatory, transparent and democratic mechanisms using the architectures and infrastructures (e.g. internet, mobile communications, global positioning systems) that the public sector has researched, developed and built and that have been hijacked by the private tech and venture capital thugs.

  3. What are the odds of two loud mouthed morons spewing their inanities using the same fascination with a Romance language?

  4. Econoclast, Salem, and FACENDO GUAIO are all the same person, they as much as admit via projective accusation…

  5. SJI’s site is having some hick-ups, my comment didn’t take and on the Vax Rate article the name and email from another commenter is showing up….
    Try again later.

    But I agree with “Not Him”

  6. Try #2:
    Not Him, it appears so… the same tired, envy filled argument by people or persons that most likely screwed around in school and possibly chose college majors that were more like a hobby in nature versus choosing a self sustaining career path with positive contributions to society.

    All they are ‘left’ with (pun intended) is resentment and a wanting for…. failed notions of wealth redistribution and ‘Big Government’ to bail them out of their Poor Choices in Life.

Leave a Reply

Your email address will not be published. Required fields are marked *