LogFAQs > #969156846

Tl:dr: I found the cause of a major network problem that affected hundreds of locations.

I'm so proud of myself. I've come a long ass way. In mid November, it will be 10 years since I started working as an entry level NOC tech. A little over 14.5 years since I started doing ISP tech support with only a basic familiarity with Windows. Now I'm a network engineer for a large company and I'm doing pretty well well.

When I started here almost 3 years ago, I didn't know if I'd be good enough for this job. I was coming from a relatively small network environment where I had built in some complexity to the network I set up, but it's nowhere near the level of complexity of this company's large network of over 400 locations.

Today, about 350 locations started having network connectivity issues, and the network team had to figure out why. We had managers from multiple departments on a conference call along with multiple network engineers troubleshooting.

I figured out the specific problem by looking at traffic patterns in the data center and then looking at a handful of branch locations to see matching traffic patterns. Now we just have to figure out the fix as a team, which won't be too hard for us.

Problem:

We have thousands of iPhones in thy environment that are used for various purposes. Every year, Apple makes a new iOS version available around September. Our application teams aren't always ready to start using the new version by then, so they'll delay the update for 90 days and push out the update for each use case when each team is ready for it. After 90 days, we no longer have a choice. Today, we pushed out updates for about 3000 iPhones and it tanked the whole network. It maxed out the throughput of some devices in the data center.

The fix will be putting in a traffic shaping policy to limit the bandwidth for that traffic, but we'll work out those details tomorrow.

Someone on the bridge specifically commented on how I did a great job finding u the needle in the haystack, and my mind is a bit blown. I can't help but think back to the days when I didn't know jack shit about networking. I'm doing it. I'm really doing it.