- Oct 8, 2004
- 17,923
- 24,172
- AFL Club
- Port Adelaide
- Other Teams
- Liverpool
New York Times: Facebook employees reportedly can’t enter buildings to evaluate the Internet outage because their door access badges weren’t working.
An automated process started removing routes on core gateways so the IPs were not accessible. This hit all core routers which is why their internal tooling isn’t working either. They have to deploy physical employees to every router to physically login to those servers directly (no remote access.) There’s also apparently a breakdown in who knows what, so there’s multiple parties involved in that physical process. People who know what is affected, people who know how to fix the problem, people who know the login info, people who can get to the physical router.
Right now, facebook and their entire network doesn’t even exist. The routing is gone. This is going to take hundreds of people to fix and probably up to 24 hours to fix due to the physical access by knowledged people required.
An automated process started removing routes on core gateways so the IPs were not accessible. This hit all core routers which is why their internal tooling isn’t working either. They have to deploy physical employees to every router to physically login to those servers directly (no remote access.) There’s also apparently a breakdown in who knows what, so there’s multiple parties involved in that physical process. People who know what is affected, people who know how to fix the problem, people who know the login info, people who can get to the physical router.
Right now, facebook and their entire network doesn’t even exist. The routing is gone. This is going to take hundreds of people to fix and probably up to 24 hours to fix due to the physical access by knowledged people required.