Episode Cover Image

60. AVAYA crashed, not coming back up with Anthon Encao

Dissecting Popular IT Nerds
Dissecting Popular IT Nerds
60. AVAYA crashed, not coming back up with Anthon Encao
Loading
/

Anthon Encao

I have been working with Amazon as an IT manager for just over a year now and loving it! I retired from the Air Force in 2013 and have held one other job since getting out. I spent nearly fourteen years in the Air Force setting up and maintaining networks from small to global in size, Just over four years of that time was spent as an instructor teaching the Computer, Network, Switching, and Cryptographic Systems course where I earned the title of Master Instructor. I previously held a Top Secret / SCI security clearance, last used Feb 2020 My education includes two associate degrees, one BS in Technical Management, and a MS in Information Technology Management.

Disclaimer: The views, thoughts, and opinions expressed by guests on this podcast are solely their own and do not necessarily reflect the views or positions of their employers, affiliates, organizations, or any other entities. The content provided is for informational purposes only and should not be considered professional advice. The podcast hosts and producers are not responsible for any actions taken based on the discussions in the episodes. We encourage listeners to consult with a professional or conduct their own research before making any decisions based on the content of this podcast

AVAYA crashed, not coming back up with Anthon Encao

3 Key Takeaways

Episode Show Notes

Anthon Encao… now IT manager at Amazon. Has years of high-level government security clearance. The last network he managed couldn’t even touch the public Internet.

His raw truths and advice start with, “It’s not easier it’s harder”

“Good IT costs more money and takes more time period.”

But if you want to make more money (as a company) And make more money faster

Then guess what?

You need to invest in IT

Oh yeah, and we cover a great story of his AVAYA PBX crashing

And never waking up… nope… took its final breath

What the heck would you do?

Hundreds of users are down with no reboot

No way to turn it back on

Transcript

Speaker 0 | 00:09.782

Welcome everyone back to Dissecting Popular Nerds. And today we have a very interesting show. We have, first of all, we have Anthony Enco on the show. And I like, you know, honestly, I’m looking at your background on LinkedIn and it looks like the surface of Mars, which is probably appropriate. for this conversation because you are an IT manager that is in charge of a network that can’t have any access to really the public internet. So it’s as if you were building a network on Mars. So I think that’s, you know, that’s what we’re going to title this episode. You know, what would it be like being an IT manager building a network out on Mars without any internet access? Is that like an appropriate comparison?

Speaker 1 | 00:56.512

I think so. As long as you consider there’s other colonies that we connect to, just none of them connect back to Earth or the Internet. Yeah.

Speaker 0 | 01:04.896

Oh, so the other colonies are on like maybe the other side of Mars. Okay, I got you. I got you. So I don’t even know where to begin. So I’m just going to let you maybe throw out some challenges there because you have a vast, you know, you’ve got some vast experience. What is this like compared to, say, I don’t know, managing a simpler network or what life used to be like.

Speaker 1 | 01:32.689

Sure, because I do have experience with that. I’ve built up a lot of base-level networks. And it was challenging until you could get your internet connection. Then Google’s your friend. You can look things up. Very nice. In the environment that I work in now, it’s almost as nice, but you got to swivel the chair to the right maybe a yard, and then you can look things up on Google, but you can’t copy things directly over. So biggest challenges we have are things like importing critical updates, repos, anything involving licensing can become a hassle with some different vendors we have. Adobe, surprisingly, is pretty rough on it.

Speaker 0 | 02:18.399

And just walk me through that. You know, maybe, you know, I’m kind of like like a sixth grader here.

Speaker 1 | 02:25.920

you know learning this so why well um usually if you have something especially with the way uh enterprises are going now uh you install the program uh you put in your credentials and then it reaches out to the web and then validates it downloads a key and validates that your software is legitimate when you can’t reach out to the internet you have to work with the vendors and they’ll sometimes give you a crazy long, like 128 character X decimal P. You have to write down on paper, drag it inside with you, and then manually type that thing in and hope you didn’t miss anything.

Speaker 0 | 03:05.884

Oh, that sounds like fun.

Speaker 1 | 03:09.204

Sometimes, yeah.

Speaker 0 | 03:10.405

That might be like a job opening for somebody. Yeah,

Speaker 1 | 03:14.946

entry level.

Speaker 0 | 03:18.407

Entry level license. updating person.

Speaker 1 | 03:23.090

We’re trying to get around things like that in the future. We’re working on being able to do one-way data diode kind of things. I’ve had them at other places at work. We don’t have one here yet, which is where a lot of the hassle comes from.

Speaker 0 | 03:36.282

Okay, so clearly there’s a lot of benefits to why you’re doing what you’re doing security-wise.

Speaker 1 | 03:44.648

Yeah, it’s definitely far more secure. We actually have two instances of a network configuration this way. One of them is just air-gapped completely. It has no classification level. It’s just for our company’s proprietary data. It makes it incredibly more difficult to exfil or for someone to get into it. But the one that I manage primarily right now is an actual classified network where we have connections to our customer sites and use their network. So as a result, we have to play by their rules. And that’s a big one. Don’t let it touch things.

Speaker 0 | 04:28.141

Pretty interesting. What else do you have? I mean, I’m tempted to ask, how did you get here and why? you know, why you, why qualified? I mean, does it have anything to do with maybe being in, um, I don’t know, were you in the U S air force or, or anything? Yeah. Military experience, you know,

Speaker 1 | 04:47.112

like maybe those are fair. Okay. That’s totally a fair question. So back in 2000, I did join the air force. I was, uh, at the time, perfect cryptographic systems maintenance, which isn’t even a job anymore. Uh, it used to be just a card swap kind of monkey. Fun, fun gig. Uh, from there, that career,

Speaker 0 | 05:06.960

first of all, why was it? Well,

Speaker 1 | 05:09.241

it sounded neat. Cause you were in a, you were a small number of people that not everyone could get the clearances required to do it. You had to learn basic electronic circuits. You had to learn how to use O scopes and multimeters and stuff like that. And, you know, but it was boring, really boring work. Um, so from there moved on to it. it merged up and became a computer switching and telephone networking kind of a job.

Speaker 0 | 05:39.191

What was the telephone stuff? Because that is exciting for me as boring it is for everyone else.

Speaker 1 | 05:44.332

Okay. So I was actually,

Speaker 0 | 05:48.073

uh,

Speaker 1 | 05:49.894

no, no, nothing cool like that. It was actually one that most people probably wouldn’t have heard of. It was a red comm red, uh, red comms. Yeah. It’s based out of, I think Victor, New York. It’s what the airport purchased as part of their theater deployable communications package. And it’s a standard 19-inch rack. I don’t remember the number of U’s, but all the cards fit in vertically. And it was just basic telephone switching services that you programmed on the fly out in the field. And then you have fiber to connect the boxes together. You stack boxes like Legos, and then you have like a whole site. on a couple of pallets worth of stuff.

Speaker 0 | 06:31.822

Old school hub and spoke. What were we doing? PRIs into that? Or how did we get out to the PSTN?

Speaker 1 | 06:37.485

So from there, we would actually hook it up via, oh gosh, testing my knowledge here. We went way back.

Speaker 0 | 06:46.170

Like some old T1 cards or something or what?

Speaker 1 | 06:49.251

We didn’t use T1. We actually had the, we used PRI and we connected that up to satellite terminals. And then those are what got us our links back into the actual. DSTN kind of telephone land where you could dial things long distance. But where we were, it was all internal. We had our own NPAs, our own extensions that we had set up, and these boxes were length daisy chain. They were set up to be in a complete loop for fault tolerance. However, there was no real way, unless you had a main hub, to get more than four connections to it. So it was just a big loop. But at the bottom of each of those, there was also an old Cisco switch. And so each of these boxes would drop off in someone’s tent, and suddenly, boom, they got telephone, like 16 phones, and enough for 48 people to connect to the base network.

Speaker 0 | 07:45.654

No, wait a second. Tent? Like literally tent?

Speaker 1 | 07:50.117

Oh, yeah. I was in deployed communications, so my whole job was to build places and set up… bases where there was fans when we landed okay now i never got to do that part specifically i don’t want to give people the wrong impression but we did go to some pretty cool places we did go to some places where there were no us comms available but i got to spend some nice time on a base in the uae so

Speaker 0 | 08:16.590

spend some time in a rat wait are you saying that you you let me just make sure i understand this correctly so we drop up these switches in a tent for phones Are you saying it’s connecting via satellite?

Speaker 1 | 08:30.614

Yes, we connect these boxes back to the main communications. And then we run some lines out to a little satellite terminal outside.

Speaker 0 | 08:40.377

That’s a different type of network that I’ve never been a part of, but that’s pretty cool.

Speaker 1 | 08:46.198

It actually was. It was a great, fun 2000 to 2006 time of my life.

Speaker 0 | 08:52.000

So you’re in UAE. That’s not too bad of a place.

Speaker 1 | 08:54.457

In the UAE, I got to, that was actually a beautiful place. I’ve been to.

Speaker 0 | 08:59.879

Did they have those man-made islands back then? Did they have those man-made like, you know, like pineapple islands that they made and all that type of stuff? Have you seen those?

Speaker 1 | 09:08.363

It’s funny you ask that. Yeah, I was actually there in 2002. So they were building those.

Speaker 0 | 09:15.466

I mean, it’s man-made islands. It’s pretty, I mean, it’s pretty sweet.

Speaker 1 | 09:19.107

It actually is really cool.

Speaker 0 | 09:21.268

Yeah. I’ve had a bunch of friends that have been over there and or worked over there. And I guess it’s, you know, it’s, uh, I guess it’s pretty wild as far as, you know, I, you know, UAE is, it’s not like, it’s not like it’s, it’s, it’s pretty rich, I guess. I guess you,

Speaker 1 | 09:38.201

it is incredibly rich. Like we used to drive downtown sometimes if we had time to, and there was a giant mall that we’d go to and they had car dealerships in the mall. And we’re not talking like Ford, we’re talking like Maserati.

Speaker 0 | 09:51.072

Yeah. Let me pull out my Lamborghini and yeah. All right. Exciting. All right. So, so you move from there. Um, how’d you, how’d you get from, so how’d you get from there to, you know, where you’re at? So obviously that, that qualified you for what, what I just out of curiosity, cause I hear I worked in DC for a long time and a lot of people go through, you know, have different levels of security clearance and everything. Is it pretty much like an in-depth background check or do they give you like a psych evaluation too? I’m just curious, like, you know, how, how, how much security clearance doth one need?

Speaker 1 | 10:27.088

It. It depends on the type of information you’re handling. In my case, I don’t know specifically the level of background information that they do. I know I have to submit like a 133-page thing, and then they give it to someone. I think it’s a third-party investigation firm. And then they go out and talk to everyone you’ve ever talked to. Oh, wow.

Speaker 0 | 10:47.805

133 people. Do you have to fill that out, or is it like a multiple-choice type of thing?

Speaker 1 | 10:54.390

I don’t know. You’ve got to fill it out. It’s mostly where you’ve ever lived, who do you know, what references do you have, where have you worked, what was your timeline.

Speaker 0 | 11:04.578

Did you ever lie on a high school test, anything like that?

Speaker 1 | 11:10.082

I don’t think they went that deep, but I’m sure they asked someone.

Speaker 0 | 11:13.324

Okay, interesting. Then they go interview everyone.

Speaker 1 | 11:17.747

But yeah, so what qualified me for my position now is back when I was doing that, we had classified and unclassified networks running in these tents. carried my security clearance all the way up through when I was retired from the Air Force in 2013. The last couple years I was monitoring in a giant like enterprise maintenance operation center, monitoring a bunch of classified networks and opening tickets on them and you know sending them out to wherever the ground station that worked on it was around the world for someone to physically go fix something. After that, I came over here to where I work now and spent some time working in networking. That is my background. I go back to it whenever I can because it is what I know the best. Spent three years in networking and then management position opened and was supposed to manage a network team. And I said, hey, you got a clearance. You want to go manage the cleared people? No one’s ever done that for us. Like, yeah, cool.

Speaker 0 | 12:19.375

Okay, nice. So. From a management perspective, from being a network jockey, you know, and pushing cards and this stuff, how was leadership? Where did you get your leadership experience from? And maybe just what are the biggest challenges you think from a technical guy, a networking guy in a leadership role?

Speaker 1 | 12:47.096

Sure. One of the biggest challenges. that I think anyone can face and I definitely did my best on it was you go from a position of working with everyone to we’re in the same company I knew them I pulled a couple people from my old team to work with me and it’s rough managing people that you’ve worked with as friends as co-workers but as long as you understand what your role is and you know what needs to be done you can usually get around that what I think was most challenging for me was coming from a area where my job was to make VLANs, make trunks, create company-wide networks, make sure it’s working, create fault tolerance and redundancy in switches and routers, to managing people whose responsibility is Active Directory, Outlook, Exchange, because I’ve used them. But as far as installing them, knowing when someone’s trying to pull the wool over your eyes on, well, we really need this. It was rough. I had to spend a lot of nights just doing research, trying to get myself up to speed on the different jobs that my team made.

Speaker 0 | 14:00.150

And that’s from a, is that from a kind of trust and verify type of standpoint?

Speaker 1 | 14:06.314

It is. I mean, we hire incredibly smart people. I love everyone on my team right now. And it’s one of those things where I don’t like not knowing what’s going on. And I think that’s a great quality. You have to remain. curious. You got to keep learning. Just because you take on a management position and you’re not expected to push the buttons and make the things happen, you need to at least understand it to some level.

Speaker 0 | 14:30.796

I can only relate as a parent. You know what I mean? I can only relate as a parent and being a homeschooler and never being someone that I thought believed in homeschooling, but really, really believes in it quite a bit right now. When your kids learn at an exponential level, I think. They can learn at an exponential level at home and they very quickly reach a point where you pretty much have to wash your hands of the level of math, at least for me, maybe not for you. You know, maybe it’s English for you. I don’t know. Maybe it’s, you know, English literature or something. But at some point, your kid is going to vastly surpass you, you know, at like a sixth grade level in some subject. So that’s the only way that I can relate, except I have not taken up that additional study.

Speaker 1 | 15:24.869

You have hit incredibly close to home on that one. My kid’s 11 now and the stuff that he wants me to help him with. Oh, I vaguely remember this.

Speaker 0 | 15:32.275

Khan Academy is available. I’m pretty sure Khan Academy is out there. It’s awesome. Okay, so I guess… From a security standpoint, there’s a big difference between security and network. And people have the philosophies that you should definitely have your security guys separate from your network guys. And there should be a checks and balances and kind of balancing back and forth and a good security policy there. What’s your general outlook on security in the environment outside of the complete lockdown? Your network’s almost completely locked down right now, so it might be easier to manage from a security standpoint. The question I really want to ask is, how secure do you think the U.S. government’s network is? And do you even want to answer that?

Speaker 1 | 16:26.649

I won’t comment on their network, mostly because I don’t know personally. However, what I can comment on is our own network that connects to it. And I know that… Um… I don’t know if you’re familiar with the DISA STIG values, the different requirements for different operating systems, different programs, what security they want you to enact, what switches and buttons they want you to click inside the software. We,

Speaker 0 | 16:56.267

of course, know all of that, but for our listeners… But for our listeners, let’s enlighten us.

Speaker 1 | 17:08.933

So, DISA has a set of standards for hardening software, hardware, and enabling security to a level that they want. And it goes down from, you know, making sure that you’re FIPS compliant. A lot of it follows the NIST standards.

Speaker 0 | 17:27.920

A lot of the comments that I get, you know, just from FIPS compliancy and whatever level of encryption they require. I remember, you know, the FIPS compliancy on VoIP standards back. five years ago was some kind of 256-bit encryption end-to-end between sites, you know, whatever it is. And I don’t know what it is now. But a lot of people say, well, yeah, we can check all the boxes, but that doesn’t mean we’re secure.

Speaker 1 | 17:50.790

And that is true. We actually have a lot of, well, several, I’m not sure if I can say them, so I won’t, several different programs that we run scans against our own internal networks at least once a week, sometimes more often. and anything that comes up as a vulnerability, we have to do. Like I said earlier, patching is difficult because we’ve got to get those patches usually hand-carried in, but we are constantly no more than one patch cycle behind, and if we are a patch cycle behind, we need a really good reason for it.

Speaker 0 | 18:24.645

So that has got to be, and you have a very motivated reason to do that all the time, but there has got to be… endless companies out there that are just it’s got to be like just an open an open marketplace um an open marketplace for people to just you know for security vulnerabilities i guess you could say oh yeah i mean we like the word hacker which is kind of weird but

Speaker 1 | 18:53.991

uh you know at the base of it what is a hacker or someone that exploits the vulnerability it’s their job much easier

Speaker 0 | 19:03.345

Their job is much easier.

Speaker 1 | 19:04.485

There’s a lot more to it than that. But at the base level, the way I think of it is they found a way in through whatever means. They’ve exploited the vulnerability to gain access to the things that I’m trying to keep from them. And our job is to keep it from them. So that means making sure all the security settings are set, making sure that we’re meeting all of the requirements at a minimum. But then on top of the DISA standards, on top of the NIST standards, on top of it. Each of our individual customers has additional security requirements that might throw on us.

Speaker 0 | 19:40.170

Give me an example, or I don’t know if I’d say an example, but is there anything out there for people listening to the show that they could walk away with and be like, that is great piece of advice? Well, one thing. Like you should lock this down, or you should do this, or here’s one thing that’s going to make. I don’t know if I’m going to say easier, security easier, but here’s something that you can do that will vastly reduce X.

Speaker 1 | 20:07.603

Well, it’s not easier. It’s harder. And that’s why a lot of people don’t do it. But one of the things that we have worked very hard towards, and at least one of the customers we have demands it, is we make sure that all permissions are verified at least once a month. And that’s not just for people in IT. That’s across the board because we have people working on multiple different projects for multiple different customers. And sometimes they shift between them and they’re not expected to go back. But we want to make sure that they don’t have access to the data that they had before because you can’t cross-intaminate things. So making sure that people’s folder access is just as simple as that or access on SharePoint. And the easiest way to do that that we found is through Active Directory, to be honest. But you have to have a means of validating that need to know. And need to know is something that comes up in the classified environment all the time. You can be cleared, but do you need to know it? If not, we’re not going to let you in there.

Speaker 0 | 21:23.173

It’s changed so much since someone’s BlackBerry server that they kept to their house years ago.

Speaker 1 | 21:29.912

We won’t get into home servers.

Speaker 0 | 21:34.115

I mean, the beds, that was just, that was the thing. That was the thing that many millennials probably don’t know about. Just like,

Speaker 1 | 21:45.323

yeah,

Speaker 0 | 21:47.384

I mean,

Speaker 1 | 21:47.645

uh, things like that, making sure that that’s going on correctly. Um, keeping up with when people are fired or leave the company, you got to make sure to strip their rights immediately.

Speaker 0 | 22:00.064

Yeah, I wonder how many logins I still have at my old companies.

Speaker 1 | 22:04.307

More than you think, probably.

Speaker 0 | 22:06.768

There’s probably some logins out there, like a Salesforce login. Like, oh, this is nice. I can get right back in.

Speaker 1 | 22:14.193

I mean, these are all common sense when you really think about it. You don’t want an ex-employee to have access to a database.

Speaker 0 | 22:20.057

Okay, so it’s common sense, but it’s kind of, it’s like at the end of the day, in whatever restaurant I look in and work in, We’ve got to clean out the refrigerator. We’ve got to do the, what is it? The scheduled maintenance to keep the machine from breaking down. We’ve got to mop the floor. We’ve got to do this. We’ve got to do all these things. Or if whether you’re in a hospital, you got to clean the room every day, you can do it every day. But it’s more, how do you get people to do it every day? What’s the system in place? Is there any management system or anything like that? And I really like the theme of it’s not easier, it’s harder. Because the more you move… I guess the more you move up in life, even the more success anyone experiences, it’s usually due to, A, they’ve worked harder, and life doesn’t necessarily, in my opinion, doesn’t get easier. I guess it could get easier if you’re looking at it from a, I got out of my teenage years and hormones and stuff like that, and that’s back to the family and us dealing with kids again. But do you understand what I’m saying?

Speaker 1 | 23:21.487

I do, and I forget who it was that said it. It’s a pretty popular quote out there, at least. in my circles, it’s you don’t achieve your aspirations. You fail back to the level of your systems, right? I mean, you can want, like for me personally, I want to lose weight, but if I keep eating cookies because they’re available, it ain’t going to happen. I need a system to stop eating the cookies. Maybe not buy them. I don’t know. But so yes, but we have, when I started, I had a database admin.

Speaker 0 | 23:52.058

You need to trick yourself with like fake cookies. You need to use, this is the way, okay, so this is just a complete side note, okay, but let me teach you how the cookie, the cookie fix, okay? I’m just like, I’m a big proponent of like a ketogenic diet and a, even carnivore diet to even go super, if you can go carnivore for 30 days, just try it. Literally, you just eat hamburger patties and that’s all you eat for 30 days, okay? A little crazy. But if you want to, I think the… My wife makes these chocolate chip cookie keto fat bombs. It looks like a chocolate chip cookie dough. It looks like a ball of chocolate chip cookie dough. It’s totally okay to eat. Tastes like cookie dough. There’s sugar-free chocolate chips in it. There’s, I mean, butter in it. It’s great. But there has to be a system. You have to trick the system. You know what I mean? It’s almost like you have to like… Like trick yourself. And if they’re there right next to the regular cookies, I can at least eat the ones to trick them. Anyways, we’re failing back to the level of our systems going.

Speaker 1 | 25:07.864

Right. So one of the things that I did when I came in here, I found out that they were using Excel spreadsheets for inventory. And that’s great if you have a small inventory.

Speaker 0 | 25:17.972

It’s classic. And it definitely was not up in Google Docs. Definitely was not.

Speaker 1 | 25:23.557

No, no, because you can’t get there from here.

Speaker 0 | 25:27.768

Okay. Anyways, so it was a spreadsheet.

Speaker 1 | 25:29.788

I live in North Carolina, right? Can’t get there from here. Bridget and Warsh out. But what we do, I had a-You just said

Speaker 0 | 25:37.150

Warsh. You just said Warsh. I did.

Speaker 1 | 25:38.790

That’s what they said.

Speaker 0 | 25:40.311

Yeah, Warsh. Okay. My mother-in-law says Warsh. We need to Warsh.

Speaker 1 | 25:42.992

I don’t actually say that.

Speaker 0 | 25:44.512

I don’t. Because there was a little bit of sarcasm in there. That’s a behavioral derailer, by the way. From a leadership standpoint, sarcasm is a behavioral derailer. I learned that. Anyways. Ah. Again, go on. I’m drinking a lot. My coffee’s starting to kick in.

Speaker 1 | 26:01.799

Ah, cool. So I had a database admin working for me, super smart guy. And I said, hey, shouldn’t this be in a database where we can make sure that certain data isn’t changed manually? It has to be done through a ticket process. There has to be checks and balances. We have to create internal tickets to go to security to make sure that the equipment’s allowed to come inside before it comes inside. It automatically throws out a request for us. for a sticker to be printed with a barcode that we throw on there automatically forces our death side text to put the stuff in there and put it in place wow that was that was pretty mind-blowing that was pretty mind-blowing it’s actually pretty cool it’s homegrown guy’s super proud of it i’m proud of him it’s a great thing from spreadsheet to amazing system The guys that I know that don’t like it, don’t like it because it forces them to do what they were supposed to do to begin with.

Speaker 0 | 26:59.981

Which is what? And I’m playing.

Speaker 1 | 27:02.583

Accurately inventory things. Make sure that our security team validates equipment before it comes into the area. Put a sticker on it.

Speaker 0 | 27:11.491

Why do you think people use, why do you think people use spreadsheets like that? Why do you think people do that?

Speaker 1 | 27:16.115

So they can do it later. And then later never comes. Or they have a very few things to keep up with and they don’t want to pay for a database. I get that too. But in our case, it forced them on.

Speaker 0 | 27:31.950

I’m thinking totally selfishly here. Because I’m a very visual person when it comes to databases, CRMs, however you want to look at your data. I really like the spreadsheet look. But a lot of databases are like single query. You have to like look things up and then it pops up and it’s just not, it’s not visually like appealing like a spreadsheet is. I’m just wondering if that’s part of the reason why people do spreadsheets.

Speaker 1 | 27:57.420

Well, maybe I could see that because I am a fan of Excel for everything in my personal life. Yeah. Not to, not to plug Microsoft at all, but they make good product. But when it comes to something like this,

Speaker 0 | 28:10.789

I mean,

Speaker 1 | 28:11.109

he made it, it’s a GUI design. It’s easy to follow. You can see what’s going on in it. How long did it take?

Speaker 0 | 28:20.507

How long did it take? Cause this is, this is now we’re getting down in the nitty gritty. How long did it take to build?

Speaker 1 | 28:26.490

Like I said, he’s a smart guy and he kind of had the vision going into it. But if you talk about the time it took to gather the requirements, what security wanted, what we wanted out of it, what our customer requirements were for reporting, it probably took. Four months, five months to get the first beta out. Okay. And that was with one guy working on it.

Speaker 0 | 28:49.621

How much time does it save? Do you think it does? Does it save time?

Speaker 1 | 28:56.823

That’s just it. If you look at the way it was with just the spreadsheets, it doesn’t save time. Because people were not keeping up with it. So what it did was show hidden costs that we didn’t have.

Speaker 0 | 29:09.667

For example, hidden costs we didn’t have? Oh, is that possible? That doesn’t mean that’s a negative. Huh?

Speaker 1 | 29:16.777

Well, when it comes to time, if people weren’t doing the job before, then they weren’t billing time to it. So we weren’t paying for it. But because now they’re being forced to do what we were actually required to do to begin with, they’re spending time on it. So it’s actually costing something.

Speaker 0 | 29:33.567

This is crazy. See, this is the weird, this is the weird type of thing that in the non-security have to be secure world. An IT director would take to a board of directors and they would say no to.

Speaker 1 | 29:51.116

And normally I think they would here too. However, it turns out we’re actually contractually obligated to these customers.

Speaker 0 | 29:56.699

That’s what I’m saying.

Speaker 1 | 29:57.439

That’s the sort of thing. So you don’t sell it on a cost basis. You sell it on a, they won’t give us more work basis.

Speaker 0 | 30:07.750

Now that’s, that’s pretty mind blowing. So we’re selling it on, go ahead.

Speaker 1 | 30:12.574

But being able to print out a report on demand, there’s assets in there that allows us to click a button. It spews out a 25% randomized point that we can then go our own inventory with. and come back. Customer loves that. Because our customer, like I said, they’re very secure, they’re very paranoid people, and they want an inventory of everything touching their network.

Speaker 0 | 30:37.675

Paranoid people. Okay. So the secret sauce is paranoid people plus we won’t get more work. I mean,

Speaker 1 | 30:53.588

what it does is it makes us stand out from our competitors as we have the ability to guarantee you a better than 99% inventory of everything in here. You will know what’s touching your equipment. You will know what’s on your network when we have it.

Speaker 0 | 31:09.121

Yeah, I mean, I have whole companies that their only job is to go out and inventory equipment. And just like inventory, like what does our network have? Literally.

Speaker 1 | 31:17.909

And it’s really hard to get a… third party in here when they all need top secret SDI clearances?

Speaker 0 | 31:27.833

I’m always looking for ways for the complainers in the tech world that are complaining. I shouldn’t say complainers because, I mean, honestly, it’s not their fault. I just feel like saying complainers today. The people that are constantly saying, no one cares about security until we get breached. we lose everything and no one cares until something happens. So what’s the argument? How do you get people to care before they don’t care when they don’t care? And I think part of that argument is not insurance. It’s not the insurance approach. If we don’t do this, we could lose everything. It might be if we don’t do this, we won’t get more work.

Speaker 1 | 32:10.289

Right. And that’s the point of turning it into money. Like I said, it costs more money to do it the right way. It always has. But if your customers are interested in a company that does it the right way, then you’re moving in the right direction.

Speaker 0 | 32:25.285

It doesn’t always cost more money to do it the right way. I’m going to play you back. I’ll get there. I’ll give you that. And the reason being is because there’s a lot of old crap still out there. In other words, sometimes it costs more money because your systems are so legacy and so old that in that case, it doesn’t cost more money. That’s, oh,

Speaker 1 | 32:50.565

sorry. Complete squirrel moment here. You go a couple of years ago, since I know you like the phone switching thing. You familiar with Avaya?

Speaker 0 | 32:59.490

Who are they?

Speaker 1 | 33:01.191

Really?

Speaker 0 | 33:02.291

Of course I know. You mean Avaya? Sorry. I don’t know why. Of course I know. Um, the, uh, uh, sometimes when I get really excited, my microphone shuts off. because I’m yelling too loud. So I’m going to whisper about Avaya here, which I used to say was a sinking submarine going down like a ship under the water, right? And recently they have made one of the best decisions that they’ve ever made, which is to partner up with RingCentral to sell their cloud product. Surprise, surprise. So anyways, go on.

Speaker 1 | 33:38.575

So we had two giant ancient boat anchor Avaya PBXs running… the phone system in our entire company.

Speaker 0 | 33:50.324

Geolocation in various different geolocations?

Speaker 1 | 33:55.388

All in the same state, but they were running it.

Speaker 0 | 34:00.232

Hub and Spoke, MPLS, or some kind of frame relay even maybe? Because I know there’s a lot of frame relay.

Speaker 1 | 34:07.058

Honestly, I don’t know because I was on the network team at the time.

Speaker 0 | 34:11.081

Anyway, big boxes.

Speaker 1 | 34:12.983

We put the… things to try to get onto uh cisco voip setup uh we had a guy ccie he knew what he was doing oh wow i had to propose it multiple times saying do you know how much money we’ll save moving off of these things even if you don’t count the electricity they’re eating and the and it finally got down to a point yeah the maintenance contract is what did it we were the maintenance contract was like three hundred thousand dollars a year And the only reason it was so big is because we had one of two remaining switches that went end of life, end of support in 2012.

Speaker 0 | 34:50.461

Yeah. And what are you paying for?

Speaker 1 | 34:51.762

The only other switch.

Speaker 0 | 34:52.483

1,800 gold pound sand. You’re paying. I mean, really, that’s what you’re calling. I mean, anyways.

Speaker 1 | 34:57.748

The only other switch. that we could get spare parts from was at avaya and the only reason they still had it is because we still have it well you could you can go to ebay we actually did have to go to ebay several times to buy replacement i mean this is this is the state of repair it was in but there is a that’s still like i need to explain it but voice is a huge cost savings if done right yes

Speaker 0 | 35:22.346

It still exists that way. I don’t know if the Cisco call manager is the right, is the right way for everybody. If you have someone that’s certified, which is now you become your, that person has now become your single point of failure. If that guy quits or goes somewhere else, you’ve got to hire people that are very, are very well trained to do, you know, to run a Cisco call manager and stuff like that. So I don’t think that that’s always the best of options. And I think that’s, again, that is the not easier, harder methodology. I do think there’s ways to do things a lot easier in the voice world. I don’t think I absolutely know for a fact. But yeah, it’s, that was a, that was a good story. So anyways, what, what, what did happen? What was, what happened?

Speaker 1 | 36:10.239

Well, with the Avaya specifically, it crashed one time and didn’t come back up and it forced the hand. Oh,

Speaker 0 | 36:16.841

gosh.

Speaker 1 | 36:18.358

There was a power outage at one of our sites. The ups were only good for a couple hours. The generator, apparently no one put fuel in it. I don’t know what happened to it. But it went down. It did not come back up.

Speaker 0 | 36:29.146

Wait, so you were just without phone service?

Speaker 1 | 36:32.469

Luckily, in that campus, because we had been running a pilot program for the VoIP system. Oh, my God. We already had a semi-infrastructure.

Speaker 0 | 36:40.355

You just turned it up.

Speaker 1 | 36:42.036

With our wonderful VAR. We called him up and said, Hey man, we need like 300 of those phones right now.

Speaker 0 | 36:48.659

He like put his phone on, he like put his phone on mute and then like, like yelled and then took it off mute. He’s like, okay, um, it’s going to be tough.

Speaker 1 | 36:58.183

And we’re going to, you know, I, I’m willing to say it to, uh, you can cut it if you want to believe it, whatever. But we, we worked with A&M and those dudes, they put phones in the back of their F-150s and drove them up here.

Speaker 0 | 37:11.569

Absolutely. And I. Absolutely. I would have. We’d have been like, oh man, I don’t know. We’ll make it happen.

Speaker 1 | 37:20.775

They’ve been great partners with us for a long time now.

Speaker 0 | 37:24.397

Awesome. Are they out of Colorado Springs or where are those guys out of? For some reason I don’t know.

Speaker 1 | 37:29.981

I don’t know where specifically. They might even be out of New Mexico. Okay. But I know they have an office up this way somewhere.

Speaker 0 | 37:37.326

Okay. So that was just, you know.

Speaker 1 | 37:41.834

by chance well i don’t want to say by chance but that was like you know i don’t know a blessing that you had that you guys have been testing that and turning that up at the same time well it would have been sunk right and we’ve been working with them we we had confidence that they were going to make the right decision on the board sooner or later uh-huh so the guys it’s

Speaker 0 | 37:59.585

just a chance we’re going to order someday in their stock room yep i think it’s a good example of i think it’s just a good example of look um On-site PBXs are a single point of failure. If you have another redundant switch down the street, great. But those are still boxes in whatever data center or server closet room, whatever you want to call it. And you can say that you’re redundant, but you can only do so much. It’s not like it’s a massive network of pops and various different… you know, data centers and HIPAA compliant data centers or FIPS compliant, whatever it is that you want. It’s just, it’s, it’s a, it’s a dead model.

Speaker 1 | 38:48.351

And it’s important to note, these weren’t the like IP based Avaya. These weren’t the ones that did VoIP. These were POP. Yeah.

Speaker 0 | 38:56.796

RJ11, like still plugging in.

Speaker 1 | 38:58.717

They were all hardwired from wherever it was in the campus, straight back to that PBX.

Speaker 0 | 39:03.980

That had to have been a circus act turning up. that new phone system in that short amount of time. Porting numbers, everything. I can just imagine.

Speaker 1 | 39:13.246

It wasn’t fun. Yeah,

Speaker 0 | 39:15.768

well.

Speaker 1 | 39:17.149

It was a good weekend though.

Speaker 0 | 39:19.250

I think we’re going to leave it on this note. We’re going to leave it on this note. It’s not easier. It’s harder. And, but we want to get more work. So we’re going to work harder. And, or it could, you know, it doesn’t always have to cost more money. You know what I mean? It really doesn’t. But I really like that theme of it’s not easier, it’s harder, especially when it comes to the security piece.

Speaker 1 | 39:43.525

It’s definitely the right way to go, though. I mean, in my mind, someone who’s been security-minded for 20-plus years now, security is the way to go if you want to actually care about your data.

Speaker 0 | 39:53.615

Last piece of advice, last message, whatever it is, to anyone listening out there, what would it be?

Speaker 1 | 39:59.732

If you’re in IT, if you’re a manager or even a first-level worker, be friends with your security department. Be friends with the IA guys because sometimes they want to put things in your best interest. If you team up, you have a better shot.

Speaker 0 | 40:18.628

That’s such a friendly… We’ve got friendly fire. What if they fire back? So I think it kind of goes both ways there, right? But you’re saying it’s easier if… if the network guys are friends with the security guys? How do we do that?

Speaker 1 | 40:34.040

Network and sysadmin.

Speaker 0 | 40:35.621

Yeah, how do we do that? How do we make friends with them?

Speaker 1 | 40:40.044

Well, for me personally, when I took the position, it was letting them know that I cared about security and that I wanted to see what they wanted. It happened that we had similar goals. So making that partnership was a lot easier for me.

Speaker 0 | 40:56.520

Oh man, Anthony, great having you on the show. Thank you so much. I’m leaving it on that note because it’s a great, it’s a great note.

Speaker 1 | 41:01.724

All right. Well, thank you.

60. AVAYA crashed, not coming back up with Anthon Encao

Speaker 0 | 00:09.782

Welcome everyone back to Dissecting Popular Nerds. And today we have a very interesting show. We have, first of all, we have Anthony Enco on the show. And I like, you know, honestly, I’m looking at your background on LinkedIn and it looks like the surface of Mars, which is probably appropriate. for this conversation because you are an IT manager that is in charge of a network that can’t have any access to really the public internet. So it’s as if you were building a network on Mars. So I think that’s, you know, that’s what we’re going to title this episode. You know, what would it be like being an IT manager building a network out on Mars without any internet access? Is that like an appropriate comparison?

Speaker 1 | 00:56.512

I think so. As long as you consider there’s other colonies that we connect to, just none of them connect back to Earth or the Internet. Yeah.

Speaker 0 | 01:04.896

Oh, so the other colonies are on like maybe the other side of Mars. Okay, I got you. I got you. So I don’t even know where to begin. So I’m just going to let you maybe throw out some challenges there because you have a vast, you know, you’ve got some vast experience. What is this like compared to, say, I don’t know, managing a simpler network or what life used to be like.

Speaker 1 | 01:32.689

Sure, because I do have experience with that. I’ve built up a lot of base-level networks. And it was challenging until you could get your internet connection. Then Google’s your friend. You can look things up. Very nice. In the environment that I work in now, it’s almost as nice, but you got to swivel the chair to the right maybe a yard, and then you can look things up on Google, but you can’t copy things directly over. So biggest challenges we have are things like importing critical updates, repos, anything involving licensing can become a hassle with some different vendors we have. Adobe, surprisingly, is pretty rough on it.

Speaker 0 | 02:18.399

And just walk me through that. You know, maybe, you know, I’m kind of like like a sixth grader here.

Speaker 1 | 02:25.920

you know learning this so why well um usually if you have something especially with the way uh enterprises are going now uh you install the program uh you put in your credentials and then it reaches out to the web and then validates it downloads a key and validates that your software is legitimate when you can’t reach out to the internet you have to work with the vendors and they’ll sometimes give you a crazy long, like 128 character X decimal P. You have to write down on paper, drag it inside with you, and then manually type that thing in and hope you didn’t miss anything.

Speaker 0 | 03:05.884

Oh, that sounds like fun.

Speaker 1 | 03:09.204

Sometimes, yeah.

Speaker 0 | 03:10.405

That might be like a job opening for somebody. Yeah,

Speaker 1 | 03:14.946

entry level.

Speaker 0 | 03:18.407

Entry level license. updating person.

Speaker 1 | 03:23.090

We’re trying to get around things like that in the future. We’re working on being able to do one-way data diode kind of things. I’ve had them at other places at work. We don’t have one here yet, which is where a lot of the hassle comes from.

Speaker 0 | 03:36.282

Okay, so clearly there’s a lot of benefits to why you’re doing what you’re doing security-wise.

Speaker 1 | 03:44.648

Yeah, it’s definitely far more secure. We actually have two instances of a network configuration this way. One of them is just air-gapped completely. It has no classification level. It’s just for our company’s proprietary data. It makes it incredibly more difficult to exfil or for someone to get into it. But the one that I manage primarily right now is an actual classified network where we have connections to our customer sites and use their network. So as a result, we have to play by their rules. And that’s a big one. Don’t let it touch things.

Speaker 0 | 04:28.141

Pretty interesting. What else do you have? I mean, I’m tempted to ask, how did you get here and why? you know, why you, why qualified? I mean, does it have anything to do with maybe being in, um, I don’t know, were you in the U S air force or, or anything? Yeah. Military experience, you know,

Speaker 1 | 04:47.112

like maybe those are fair. Okay. That’s totally a fair question. So back in 2000, I did join the air force. I was, uh, at the time, perfect cryptographic systems maintenance, which isn’t even a job anymore. Uh, it used to be just a card swap kind of monkey. Fun, fun gig. Uh, from there, that career,

Speaker 0 | 05:06.960

first of all, why was it? Well,

Speaker 1 | 05:09.241

it sounded neat. Cause you were in a, you were a small number of people that not everyone could get the clearances required to do it. You had to learn basic electronic circuits. You had to learn how to use O scopes and multimeters and stuff like that. And, you know, but it was boring, really boring work. Um, so from there moved on to it. it merged up and became a computer switching and telephone networking kind of a job.

Speaker 0 | 05:39.191

What was the telephone stuff? Because that is exciting for me as boring it is for everyone else.

Speaker 1 | 05:44.332

Okay. So I was actually,

Speaker 0 | 05:48.073

uh,

Speaker 1 | 05:49.894

no, no, nothing cool like that. It was actually one that most people probably wouldn’t have heard of. It was a red comm red, uh, red comms. Yeah. It’s based out of, I think Victor, New York. It’s what the airport purchased as part of their theater deployable communications package. And it’s a standard 19-inch rack. I don’t remember the number of U’s, but all the cards fit in vertically. And it was just basic telephone switching services that you programmed on the fly out in the field. And then you have fiber to connect the boxes together. You stack boxes like Legos, and then you have like a whole site. on a couple of pallets worth of stuff.

Speaker 0 | 06:31.822

Old school hub and spoke. What were we doing? PRIs into that? Or how did we get out to the PSTN?

Speaker 1 | 06:37.485

So from there, we would actually hook it up via, oh gosh, testing my knowledge here. We went way back.

Speaker 0 | 06:46.170

Like some old T1 cards or something or what?

Speaker 1 | 06:49.251

We didn’t use T1. We actually had the, we used PRI and we connected that up to satellite terminals. And then those are what got us our links back into the actual. DSTN kind of telephone land where you could dial things long distance. But where we were, it was all internal. We had our own NPAs, our own extensions that we had set up, and these boxes were length daisy chain. They were set up to be in a complete loop for fault tolerance. However, there was no real way, unless you had a main hub, to get more than four connections to it. So it was just a big loop. But at the bottom of each of those, there was also an old Cisco switch. And so each of these boxes would drop off in someone’s tent, and suddenly, boom, they got telephone, like 16 phones, and enough for 48 people to connect to the base network.

Speaker 0 | 07:45.654

No, wait a second. Tent? Like literally tent?

Speaker 1 | 07:50.117

Oh, yeah. I was in deployed communications, so my whole job was to build places and set up… bases where there was fans when we landed okay now i never got to do that part specifically i don’t want to give people the wrong impression but we did go to some pretty cool places we did go to some places where there were no us comms available but i got to spend some nice time on a base in the uae so

Speaker 0 | 08:16.590

spend some time in a rat wait are you saying that you you let me just make sure i understand this correctly so we drop up these switches in a tent for phones Are you saying it’s connecting via satellite?

Speaker 1 | 08:30.614

Yes, we connect these boxes back to the main communications. And then we run some lines out to a little satellite terminal outside.

Speaker 0 | 08:40.377

That’s a different type of network that I’ve never been a part of, but that’s pretty cool.

Speaker 1 | 08:46.198

It actually was. It was a great, fun 2000 to 2006 time of my life.

Speaker 0 | 08:52.000

So you’re in UAE. That’s not too bad of a place.

Speaker 1 | 08:54.457

In the UAE, I got to, that was actually a beautiful place. I’ve been to.

Speaker 0 | 08:59.879

Did they have those man-made islands back then? Did they have those man-made like, you know, like pineapple islands that they made and all that type of stuff? Have you seen those?

Speaker 1 | 09:08.363

It’s funny you ask that. Yeah, I was actually there in 2002. So they were building those.

Speaker 0 | 09:15.466

I mean, it’s man-made islands. It’s pretty, I mean, it’s pretty sweet.

Speaker 1 | 09:19.107

It actually is really cool.

Speaker 0 | 09:21.268

Yeah. I’ve had a bunch of friends that have been over there and or worked over there. And I guess it’s, you know, it’s, uh, I guess it’s pretty wild as far as, you know, I, you know, UAE is, it’s not like, it’s not like it’s, it’s, it’s pretty rich, I guess. I guess you,

Speaker 1 | 09:38.201

it is incredibly rich. Like we used to drive downtown sometimes if we had time to, and there was a giant mall that we’d go to and they had car dealerships in the mall. And we’re not talking like Ford, we’re talking like Maserati.

Speaker 0 | 09:51.072

Yeah. Let me pull out my Lamborghini and yeah. All right. Exciting. All right. So, so you move from there. Um, how’d you, how’d you get from, so how’d you get from there to, you know, where you’re at? So obviously that, that qualified you for what, what I just out of curiosity, cause I hear I worked in DC for a long time and a lot of people go through, you know, have different levels of security clearance and everything. Is it pretty much like an in-depth background check or do they give you like a psych evaluation too? I’m just curious, like, you know, how, how, how much security clearance doth one need?

Speaker 1 | 10:27.088

It. It depends on the type of information you’re handling. In my case, I don’t know specifically the level of background information that they do. I know I have to submit like a 133-page thing, and then they give it to someone. I think it’s a third-party investigation firm. And then they go out and talk to everyone you’ve ever talked to. Oh, wow.

Speaker 0 | 10:47.805

133 people. Do you have to fill that out, or is it like a multiple-choice type of thing?

Speaker 1 | 10:54.390

I don’t know. You’ve got to fill it out. It’s mostly where you’ve ever lived, who do you know, what references do you have, where have you worked, what was your timeline.

Speaker 0 | 11:04.578

Did you ever lie on a high school test, anything like that?

Speaker 1 | 11:10.082

I don’t think they went that deep, but I’m sure they asked someone.

Speaker 0 | 11:13.324

Okay, interesting. Then they go interview everyone.

Speaker 1 | 11:17.747

But yeah, so what qualified me for my position now is back when I was doing that, we had classified and unclassified networks running in these tents. carried my security clearance all the way up through when I was retired from the Air Force in 2013. The last couple years I was monitoring in a giant like enterprise maintenance operation center, monitoring a bunch of classified networks and opening tickets on them and you know sending them out to wherever the ground station that worked on it was around the world for someone to physically go fix something. After that, I came over here to where I work now and spent some time working in networking. That is my background. I go back to it whenever I can because it is what I know the best. Spent three years in networking and then management position opened and was supposed to manage a network team. And I said, hey, you got a clearance. You want to go manage the cleared people? No one’s ever done that for us. Like, yeah, cool.

Speaker 0 | 12:19.375

Okay, nice. So. From a management perspective, from being a network jockey, you know, and pushing cards and this stuff, how was leadership? Where did you get your leadership experience from? And maybe just what are the biggest challenges you think from a technical guy, a networking guy in a leadership role?

Speaker 1 | 12:47.096

Sure. One of the biggest challenges. that I think anyone can face and I definitely did my best on it was you go from a position of working with everyone to we’re in the same company I knew them I pulled a couple people from my old team to work with me and it’s rough managing people that you’ve worked with as friends as co-workers but as long as you understand what your role is and you know what needs to be done you can usually get around that what I think was most challenging for me was coming from a area where my job was to make VLANs, make trunks, create company-wide networks, make sure it’s working, create fault tolerance and redundancy in switches and routers, to managing people whose responsibility is Active Directory, Outlook, Exchange, because I’ve used them. But as far as installing them, knowing when someone’s trying to pull the wool over your eyes on, well, we really need this. It was rough. I had to spend a lot of nights just doing research, trying to get myself up to speed on the different jobs that my team made.

Speaker 0 | 14:00.150

And that’s from a, is that from a kind of trust and verify type of standpoint?

Speaker 1 | 14:06.314

It is. I mean, we hire incredibly smart people. I love everyone on my team right now. And it’s one of those things where I don’t like not knowing what’s going on. And I think that’s a great quality. You have to remain. curious. You got to keep learning. Just because you take on a management position and you’re not expected to push the buttons and make the things happen, you need to at least understand it to some level.

Speaker 0 | 14:30.796

I can only relate as a parent. You know what I mean? I can only relate as a parent and being a homeschooler and never being someone that I thought believed in homeschooling, but really, really believes in it quite a bit right now. When your kids learn at an exponential level, I think. They can learn at an exponential level at home and they very quickly reach a point where you pretty much have to wash your hands of the level of math, at least for me, maybe not for you. You know, maybe it’s English for you. I don’t know. Maybe it’s, you know, English literature or something. But at some point, your kid is going to vastly surpass you, you know, at like a sixth grade level in some subject. So that’s the only way that I can relate, except I have not taken up that additional study.

Speaker 1 | 15:24.869

You have hit incredibly close to home on that one. My kid’s 11 now and the stuff that he wants me to help him with. Oh, I vaguely remember this.

Speaker 0 | 15:32.275

Khan Academy is available. I’m pretty sure Khan Academy is out there. It’s awesome. Okay, so I guess… From a security standpoint, there’s a big difference between security and network. And people have the philosophies that you should definitely have your security guys separate from your network guys. And there should be a checks and balances and kind of balancing back and forth and a good security policy there. What’s your general outlook on security in the environment outside of the complete lockdown? Your network’s almost completely locked down right now, so it might be easier to manage from a security standpoint. The question I really want to ask is, how secure do you think the U.S. government’s network is? And do you even want to answer that?

Speaker 1 | 16:26.649

I won’t comment on their network, mostly because I don’t know personally. However, what I can comment on is our own network that connects to it. And I know that… Um… I don’t know if you’re familiar with the DISA STIG values, the different requirements for different operating systems, different programs, what security they want you to enact, what switches and buttons they want you to click inside the software. We,

Speaker 0 | 16:56.267

of course, know all of that, but for our listeners… But for our listeners, let’s enlighten us.

Speaker 1 | 17:08.933

So, DISA has a set of standards for hardening software, hardware, and enabling security to a level that they want. And it goes down from, you know, making sure that you’re FIPS compliant. A lot of it follows the NIST standards.

Speaker 0 | 17:27.920

A lot of the comments that I get, you know, just from FIPS compliancy and whatever level of encryption they require. I remember, you know, the FIPS compliancy on VoIP standards back. five years ago was some kind of 256-bit encryption end-to-end between sites, you know, whatever it is. And I don’t know what it is now. But a lot of people say, well, yeah, we can check all the boxes, but that doesn’t mean we’re secure.

Speaker 1 | 17:50.790

And that is true. We actually have a lot of, well, several, I’m not sure if I can say them, so I won’t, several different programs that we run scans against our own internal networks at least once a week, sometimes more often. and anything that comes up as a vulnerability, we have to do. Like I said earlier, patching is difficult because we’ve got to get those patches usually hand-carried in, but we are constantly no more than one patch cycle behind, and if we are a patch cycle behind, we need a really good reason for it.

Speaker 0 | 18:24.645

So that has got to be, and you have a very motivated reason to do that all the time, but there has got to be… endless companies out there that are just it’s got to be like just an open an open marketplace um an open marketplace for people to just you know for security vulnerabilities i guess you could say oh yeah i mean we like the word hacker which is kind of weird but

Speaker 1 | 18:53.991

uh you know at the base of it what is a hacker or someone that exploits the vulnerability it’s their job much easier

Speaker 0 | 19:03.345

Their job is much easier.

Speaker 1 | 19:04.485

There’s a lot more to it than that. But at the base level, the way I think of it is they found a way in through whatever means. They’ve exploited the vulnerability to gain access to the things that I’m trying to keep from them. And our job is to keep it from them. So that means making sure all the security settings are set, making sure that we’re meeting all of the requirements at a minimum. But then on top of the DISA standards, on top of the NIST standards, on top of it. Each of our individual customers has additional security requirements that might throw on us.

Speaker 0 | 19:40.170

Give me an example, or I don’t know if I’d say an example, but is there anything out there for people listening to the show that they could walk away with and be like, that is great piece of advice? Well, one thing. Like you should lock this down, or you should do this, or here’s one thing that’s going to make. I don’t know if I’m going to say easier, security easier, but here’s something that you can do that will vastly reduce X.

Speaker 1 | 20:07.603

Well, it’s not easier. It’s harder. And that’s why a lot of people don’t do it. But one of the things that we have worked very hard towards, and at least one of the customers we have demands it, is we make sure that all permissions are verified at least once a month. And that’s not just for people in IT. That’s across the board because we have people working on multiple different projects for multiple different customers. And sometimes they shift between them and they’re not expected to go back. But we want to make sure that they don’t have access to the data that they had before because you can’t cross-intaminate things. So making sure that people’s folder access is just as simple as that or access on SharePoint. And the easiest way to do that that we found is through Active Directory, to be honest. But you have to have a means of validating that need to know. And need to know is something that comes up in the classified environment all the time. You can be cleared, but do you need to know it? If not, we’re not going to let you in there.

Speaker 0 | 21:23.173

It’s changed so much since someone’s BlackBerry server that they kept to their house years ago.

Speaker 1 | 21:29.912

We won’t get into home servers.

Speaker 0 | 21:34.115

I mean, the beds, that was just, that was the thing. That was the thing that many millennials probably don’t know about. Just like,

Speaker 1 | 21:45.323

yeah,

Speaker 0 | 21:47.384

I mean,

Speaker 1 | 21:47.645

uh, things like that, making sure that that’s going on correctly. Um, keeping up with when people are fired or leave the company, you got to make sure to strip their rights immediately.

Speaker 0 | 22:00.064

Yeah, I wonder how many logins I still have at my old companies.

Speaker 1 | 22:04.307

More than you think, probably.

Speaker 0 | 22:06.768

There’s probably some logins out there, like a Salesforce login. Like, oh, this is nice. I can get right back in.

Speaker 1 | 22:14.193

I mean, these are all common sense when you really think about it. You don’t want an ex-employee to have access to a database.

Speaker 0 | 22:20.057

Okay, so it’s common sense, but it’s kind of, it’s like at the end of the day, in whatever restaurant I look in and work in, We’ve got to clean out the refrigerator. We’ve got to do the, what is it? The scheduled maintenance to keep the machine from breaking down. We’ve got to mop the floor. We’ve got to do this. We’ve got to do all these things. Or if whether you’re in a hospital, you got to clean the room every day, you can do it every day. But it’s more, how do you get people to do it every day? What’s the system in place? Is there any management system or anything like that? And I really like the theme of it’s not easier, it’s harder. Because the more you move… I guess the more you move up in life, even the more success anyone experiences, it’s usually due to, A, they’ve worked harder, and life doesn’t necessarily, in my opinion, doesn’t get easier. I guess it could get easier if you’re looking at it from a, I got out of my teenage years and hormones and stuff like that, and that’s back to the family and us dealing with kids again. But do you understand what I’m saying?

Speaker 1 | 23:21.487

I do, and I forget who it was that said it. It’s a pretty popular quote out there, at least. in my circles, it’s you don’t achieve your aspirations. You fail back to the level of your systems, right? I mean, you can want, like for me personally, I want to lose weight, but if I keep eating cookies because they’re available, it ain’t going to happen. I need a system to stop eating the cookies. Maybe not buy them. I don’t know. But so yes, but we have, when I started, I had a database admin.

Speaker 0 | 23:52.058

You need to trick yourself with like fake cookies. You need to use, this is the way, okay, so this is just a complete side note, okay, but let me teach you how the cookie, the cookie fix, okay? I’m just like, I’m a big proponent of like a ketogenic diet and a, even carnivore diet to even go super, if you can go carnivore for 30 days, just try it. Literally, you just eat hamburger patties and that’s all you eat for 30 days, okay? A little crazy. But if you want to, I think the… My wife makes these chocolate chip cookie keto fat bombs. It looks like a chocolate chip cookie dough. It looks like a ball of chocolate chip cookie dough. It’s totally okay to eat. Tastes like cookie dough. There’s sugar-free chocolate chips in it. There’s, I mean, butter in it. It’s great. But there has to be a system. You have to trick the system. You know what I mean? It’s almost like you have to like… Like trick yourself. And if they’re there right next to the regular cookies, I can at least eat the ones to trick them. Anyways, we’re failing back to the level of our systems going.

Speaker 1 | 25:07.864

Right. So one of the things that I did when I came in here, I found out that they were using Excel spreadsheets for inventory. And that’s great if you have a small inventory.

Speaker 0 | 25:17.972

It’s classic. And it definitely was not up in Google Docs. Definitely was not.

Speaker 1 | 25:23.557

No, no, because you can’t get there from here.

Speaker 0 | 25:27.768

Okay. Anyways, so it was a spreadsheet.

Speaker 1 | 25:29.788

I live in North Carolina, right? Can’t get there from here. Bridget and Warsh out. But what we do, I had a-You just said

Speaker 0 | 25:37.150

Warsh. You just said Warsh. I did.

Speaker 1 | 25:38.790

That’s what they said.

Speaker 0 | 25:40.311

Yeah, Warsh. Okay. My mother-in-law says Warsh. We need to Warsh.

Speaker 1 | 25:42.992

I don’t actually say that.

Speaker 0 | 25:44.512

I don’t. Because there was a little bit of sarcasm in there. That’s a behavioral derailer, by the way. From a leadership standpoint, sarcasm is a behavioral derailer. I learned that. Anyways. Ah. Again, go on. I’m drinking a lot. My coffee’s starting to kick in.

Speaker 1 | 26:01.799

Ah, cool. So I had a database admin working for me, super smart guy. And I said, hey, shouldn’t this be in a database where we can make sure that certain data isn’t changed manually? It has to be done through a ticket process. There has to be checks and balances. We have to create internal tickets to go to security to make sure that the equipment’s allowed to come inside before it comes inside. It automatically throws out a request for us. for a sticker to be printed with a barcode that we throw on there automatically forces our death side text to put the stuff in there and put it in place wow that was that was pretty mind-blowing that was pretty mind-blowing it’s actually pretty cool it’s homegrown guy’s super proud of it i’m proud of him it’s a great thing from spreadsheet to amazing system The guys that I know that don’t like it, don’t like it because it forces them to do what they were supposed to do to begin with.

Speaker 0 | 26:59.981

Which is what? And I’m playing.

Speaker 1 | 27:02.583

Accurately inventory things. Make sure that our security team validates equipment before it comes into the area. Put a sticker on it.

Speaker 0 | 27:11.491

Why do you think people use, why do you think people use spreadsheets like that? Why do you think people do that?

Speaker 1 | 27:16.115

So they can do it later. And then later never comes. Or they have a very few things to keep up with and they don’t want to pay for a database. I get that too. But in our case, it forced them on.

Speaker 0 | 27:31.950

I’m thinking totally selfishly here. Because I’m a very visual person when it comes to databases, CRMs, however you want to look at your data. I really like the spreadsheet look. But a lot of databases are like single query. You have to like look things up and then it pops up and it’s just not, it’s not visually like appealing like a spreadsheet is. I’m just wondering if that’s part of the reason why people do spreadsheets.

Speaker 1 | 27:57.420

Well, maybe I could see that because I am a fan of Excel for everything in my personal life. Yeah. Not to, not to plug Microsoft at all, but they make good product. But when it comes to something like this,

Speaker 0 | 28:10.789

I mean,

Speaker 1 | 28:11.109

he made it, it’s a GUI design. It’s easy to follow. You can see what’s going on in it. How long did it take?

Speaker 0 | 28:20.507

How long did it take? Cause this is, this is now we’re getting down in the nitty gritty. How long did it take to build?

Speaker 1 | 28:26.490

Like I said, he’s a smart guy and he kind of had the vision going into it. But if you talk about the time it took to gather the requirements, what security wanted, what we wanted out of it, what our customer requirements were for reporting, it probably took. Four months, five months to get the first beta out. Okay. And that was with one guy working on it.

Speaker 0 | 28:49.621

How much time does it save? Do you think it does? Does it save time?

Speaker 1 | 28:56.823

That’s just it. If you look at the way it was with just the spreadsheets, it doesn’t save time. Because people were not keeping up with it. So what it did was show hidden costs that we didn’t have.

Speaker 0 | 29:09.667

For example, hidden costs we didn’t have? Oh, is that possible? That doesn’t mean that’s a negative. Huh?

Speaker 1 | 29:16.777

Well, when it comes to time, if people weren’t doing the job before, then they weren’t billing time to it. So we weren’t paying for it. But because now they’re being forced to do what we were actually required to do to begin with, they’re spending time on it. So it’s actually costing something.

Speaker 0 | 29:33.567

This is crazy. See, this is the weird, this is the weird type of thing that in the non-security have to be secure world. An IT director would take to a board of directors and they would say no to.

Speaker 1 | 29:51.116

And normally I think they would here too. However, it turns out we’re actually contractually obligated to these customers.

Speaker 0 | 29:56.699

That’s what I’m saying.

Speaker 1 | 29:57.439

That’s the sort of thing. So you don’t sell it on a cost basis. You sell it on a, they won’t give us more work basis.

Speaker 0 | 30:07.750

Now that’s, that’s pretty mind blowing. So we’re selling it on, go ahead.

Speaker 1 | 30:12.574

But being able to print out a report on demand, there’s assets in there that allows us to click a button. It spews out a 25% randomized point that we can then go our own inventory with. and come back. Customer loves that. Because our customer, like I said, they’re very secure, they’re very paranoid people, and they want an inventory of everything touching their network.

Speaker 0 | 30:37.675

Paranoid people. Okay. So the secret sauce is paranoid people plus we won’t get more work. I mean,

Speaker 1 | 30:53.588

what it does is it makes us stand out from our competitors as we have the ability to guarantee you a better than 99% inventory of everything in here. You will know what’s touching your equipment. You will know what’s on your network when we have it.

Speaker 0 | 31:09.121

Yeah, I mean, I have whole companies that their only job is to go out and inventory equipment. And just like inventory, like what does our network have? Literally.

Speaker 1 | 31:17.909

And it’s really hard to get a… third party in here when they all need top secret SDI clearances?

Speaker 0 | 31:27.833

I’m always looking for ways for the complainers in the tech world that are complaining. I shouldn’t say complainers because, I mean, honestly, it’s not their fault. I just feel like saying complainers today. The people that are constantly saying, no one cares about security until we get breached. we lose everything and no one cares until something happens. So what’s the argument? How do you get people to care before they don’t care when they don’t care? And I think part of that argument is not insurance. It’s not the insurance approach. If we don’t do this, we could lose everything. It might be if we don’t do this, we won’t get more work.

Speaker 1 | 32:10.289

Right. And that’s the point of turning it into money. Like I said, it costs more money to do it the right way. It always has. But if your customers are interested in a company that does it the right way, then you’re moving in the right direction.

Speaker 0 | 32:25.285

It doesn’t always cost more money to do it the right way. I’m going to play you back. I’ll get there. I’ll give you that. And the reason being is because there’s a lot of old crap still out there. In other words, sometimes it costs more money because your systems are so legacy and so old that in that case, it doesn’t cost more money. That’s, oh,

Speaker 1 | 32:50.565

sorry. Complete squirrel moment here. You go a couple of years ago, since I know you like the phone switching thing. You familiar with Avaya?

Speaker 0 | 32:59.490

Who are they?

Speaker 1 | 33:01.191

Really?

Speaker 0 | 33:02.291

Of course I know. You mean Avaya? Sorry. I don’t know why. Of course I know. Um, the, uh, uh, sometimes when I get really excited, my microphone shuts off. because I’m yelling too loud. So I’m going to whisper about Avaya here, which I used to say was a sinking submarine going down like a ship under the water, right? And recently they have made one of the best decisions that they’ve ever made, which is to partner up with RingCentral to sell their cloud product. Surprise, surprise. So anyways, go on.

Speaker 1 | 33:38.575

So we had two giant ancient boat anchor Avaya PBXs running… the phone system in our entire company.

Speaker 0 | 33:50.324

Geolocation in various different geolocations?

Speaker 1 | 33:55.388

All in the same state, but they were running it.

Speaker 0 | 34:00.232

Hub and Spoke, MPLS, or some kind of frame relay even maybe? Because I know there’s a lot of frame relay.

Speaker 1 | 34:07.058

Honestly, I don’t know because I was on the network team at the time.

Speaker 0 | 34:11.081

Anyway, big boxes.

Speaker 1 | 34:12.983

We put the… things to try to get onto uh cisco voip setup uh we had a guy ccie he knew what he was doing oh wow i had to propose it multiple times saying do you know how much money we’ll save moving off of these things even if you don’t count the electricity they’re eating and the and it finally got down to a point yeah the maintenance contract is what did it we were the maintenance contract was like three hundred thousand dollars a year And the only reason it was so big is because we had one of two remaining switches that went end of life, end of support in 2012.

Speaker 0 | 34:50.461

Yeah. And what are you paying for?

Speaker 1 | 34:51.762

The only other switch.

Speaker 0 | 34:52.483

1,800 gold pound sand. You’re paying. I mean, really, that’s what you’re calling. I mean, anyways.

Speaker 1 | 34:57.748

The only other switch. that we could get spare parts from was at avaya and the only reason they still had it is because we still have it well you could you can go to ebay we actually did have to go to ebay several times to buy replacement i mean this is this is the state of repair it was in but there is a that’s still like i need to explain it but voice is a huge cost savings if done right yes

Speaker 0 | 35:22.346

It still exists that way. I don’t know if the Cisco call manager is the right, is the right way for everybody. If you have someone that’s certified, which is now you become your, that person has now become your single point of failure. If that guy quits or goes somewhere else, you’ve got to hire people that are very, are very well trained to do, you know, to run a Cisco call manager and stuff like that. So I don’t think that that’s always the best of options. And I think that’s, again, that is the not easier, harder methodology. I do think there’s ways to do things a lot easier in the voice world. I don’t think I absolutely know for a fact. But yeah, it’s, that was a, that was a good story. So anyways, what, what, what did happen? What was, what happened?

Speaker 1 | 36:10.239

Well, with the Avaya specifically, it crashed one time and didn’t come back up and it forced the hand. Oh,

Speaker 0 | 36:16.841

gosh.

Speaker 1 | 36:18.358

There was a power outage at one of our sites. The ups were only good for a couple hours. The generator, apparently no one put fuel in it. I don’t know what happened to it. But it went down. It did not come back up.

Speaker 0 | 36:29.146

Wait, so you were just without phone service?

Speaker 1 | 36:32.469

Luckily, in that campus, because we had been running a pilot program for the VoIP system. Oh, my God. We already had a semi-infrastructure.

Speaker 0 | 36:40.355

You just turned it up.

Speaker 1 | 36:42.036

With our wonderful VAR. We called him up and said, Hey man, we need like 300 of those phones right now.

Speaker 0 | 36:48.659

He like put his phone on, he like put his phone on mute and then like, like yelled and then took it off mute. He’s like, okay, um, it’s going to be tough.

Speaker 1 | 36:58.183

And we’re going to, you know, I, I’m willing to say it to, uh, you can cut it if you want to believe it, whatever. But we, we worked with A&M and those dudes, they put phones in the back of their F-150s and drove them up here.

Speaker 0 | 37:11.569

Absolutely. And I. Absolutely. I would have. We’d have been like, oh man, I don’t know. We’ll make it happen.

Speaker 1 | 37:20.775

They’ve been great partners with us for a long time now.

Speaker 0 | 37:24.397

Awesome. Are they out of Colorado Springs or where are those guys out of? For some reason I don’t know.

Speaker 1 | 37:29.981

I don’t know where specifically. They might even be out of New Mexico. Okay. But I know they have an office up this way somewhere.

Speaker 0 | 37:37.326

Okay. So that was just, you know.

Speaker 1 | 37:41.834

by chance well i don’t want to say by chance but that was like you know i don’t know a blessing that you had that you guys have been testing that and turning that up at the same time well it would have been sunk right and we’ve been working with them we we had confidence that they were going to make the right decision on the board sooner or later uh-huh so the guys it’s

Speaker 0 | 37:59.585

just a chance we’re going to order someday in their stock room yep i think it’s a good example of i think it’s just a good example of look um On-site PBXs are a single point of failure. If you have another redundant switch down the street, great. But those are still boxes in whatever data center or server closet room, whatever you want to call it. And you can say that you’re redundant, but you can only do so much. It’s not like it’s a massive network of pops and various different… you know, data centers and HIPAA compliant data centers or FIPS compliant, whatever it is that you want. It’s just, it’s, it’s a, it’s a dead model.

Speaker 1 | 38:48.351

And it’s important to note, these weren’t the like IP based Avaya. These weren’t the ones that did VoIP. These were POP. Yeah.

Speaker 0 | 38:56.796

RJ11, like still plugging in.

Speaker 1 | 38:58.717

They were all hardwired from wherever it was in the campus, straight back to that PBX.

Speaker 0 | 39:03.980

That had to have been a circus act turning up. that new phone system in that short amount of time. Porting numbers, everything. I can just imagine.

Speaker 1 | 39:13.246

It wasn’t fun. Yeah,

Speaker 0 | 39:15.768

well.

Speaker 1 | 39:17.149

It was a good weekend though.

Speaker 0 | 39:19.250

I think we’re going to leave it on this note. We’re going to leave it on this note. It’s not easier. It’s harder. And, but we want to get more work. So we’re going to work harder. And, or it could, you know, it doesn’t always have to cost more money. You know what I mean? It really doesn’t. But I really like that theme of it’s not easier, it’s harder, especially when it comes to the security piece.

Speaker 1 | 39:43.525

It’s definitely the right way to go, though. I mean, in my mind, someone who’s been security-minded for 20-plus years now, security is the way to go if you want to actually care about your data.

Speaker 0 | 39:53.615

Last piece of advice, last message, whatever it is, to anyone listening out there, what would it be?

Speaker 1 | 39:59.732

If you’re in IT, if you’re a manager or even a first-level worker, be friends with your security department. Be friends with the IA guys because sometimes they want to put things in your best interest. If you team up, you have a better shot.

Speaker 0 | 40:18.628

That’s such a friendly… We’ve got friendly fire. What if they fire back? So I think it kind of goes both ways there, right? But you’re saying it’s easier if… if the network guys are friends with the security guys? How do we do that?

Speaker 1 | 40:34.040

Network and sysadmin.

Speaker 0 | 40:35.621

Yeah, how do we do that? How do we make friends with them?

Speaker 1 | 40:40.044

Well, for me personally, when I took the position, it was letting them know that I cared about security and that I wanted to see what they wanted. It happened that we had similar goals. So making that partnership was a lot easier for me.

Speaker 0 | 40:56.520

Oh man, Anthony, great having you on the show. Thank you so much. I’m leaving it on that note because it’s a great, it’s a great note.

Speaker 1 | 41:01.724

All right. Well, thank you.

Share This Episode On:

HOSTED BY PHIL HOWARD

Dissecting Popular IT Nerds Podcast

Weekly strategic insights from technology executives who understand your challenges

Are You The Nerd We're Looking For?

ATTENTION IT EXECUTIVES: Your advice and unique stories are invaluable to us. Help us by taking this quiz. You’ll gain recognition good for your career and you’ll contribute value to your fellow IT peers.

QR Code