Current Events > Today, I found a needle in the haystack that caused a major network problem

(+) Yeah! (+)

Tl:dr: I found the cause of a major network problem that affected hundreds of locations.

I'm so proud of myself. I've come a long ass way. In mid November, it will be 10 years since I started working as an entry level NOC tech. A little over 14.5 years since I started doing ISP tech support with only a basic familiarity with Windows. Now I'm a network engineer for a large company and I'm doing pretty well well.

When I started here almost 3 years ago, I didn't know if I'd be good enough for this job. I was coming from a relatively small network environment where I had built in some complexity to the network I set up, but it's nowhere near the level of complexity of this company's large network of over 400 locations.

Today, about 350 locations started having network connectivity issues, and the network team had to figure out why. We had managers from multiple departments on a conference call along with multiple network engineers troubleshooting.

I figured out the specific problem by looking at traffic patterns in the data center and then looking at a handful of branch locations to see matching traffic patterns. Now we just have to figure out the fix as a team, which won't be too hard for us.

Problem:

We have thousands of iPhones in thy environment that are used for various purposes. Every year, Apple makes a new iOS version available around September. Our application teams aren't always ready to start using the new version by then, so they'll delay the update for 90 days and push out the update for each use case when each team is ready for it. After 90 days, we no longer have a choice. Today, we pushed out updates for about 3000 iPhones and it tanked the whole network. It maxed out the throughput of some devices in the data center.

The fix will be putting in a traffic shaping policy to limit the bandwidth for that traffic, but we'll work out those details tomorrow.

Someone on the bridge specifically commented on how I did a great job finding u the needle in the haystack, and my mind is a bit blown. I can't help but think back to the days when I didn't know jack shit about networking. I'm doing it. I'm really doing it.

They gonna throw a bonus your way or what?

Good job, nerd!

CableZL posted...

Problem:

We have thousands of iPhones

One time we accidentally broke our office network by running out of IP addresses because of this.

MFBKBass5 posted...

They gonna throw a bonus your way or what?

We get bonuses quarterly. These calls are stressful AF, but it's awesome when you get to flex your skills in a relatively public manner on a conference call with so many people involved.

I wish my internet people were that quick with things

CableZL posted...

We get bonuses quarterly. These calls are stressful AF, but it's awesome when you get to flex your skills in a relatively public manner on a conference call with so many people involved.

im curious, as I just started working in tech at a large companybut what do these bonuses look like?

MFBKBass5 posted...

im curious, as I just started working in tech at a large companybut what do these bonuses look like?

At my current level, I get a 15% annual bonus that is broken up into 4 bonuses, one per quarter. At the end of the year, it the company does well, there is also an extra bonus on top of that. I also get private stock annually, which normally amounts to somewhere between 2 to 4%, depending on what the company decides to give us.

Promote this man to Network God-King asap