Speaker 0 | 00:08.742
All right, well, good morning and welcome to another Dissecting Popular IT Nerds, where we’re allowed to geek out with fellow nerds and there’s no iScrolls. So this morning or today, I’d like to introduce Chris Shoemaker, who comes to Dissecting Popular IT Nerds with that same college degree almost all of us have of marine science. So, hey, Chris, tell us a little about yourself and let us know how a marine science degree led to the director’s chair in IT. Because, wait a minute, marine science.
Speaker 1 | 00:42.184
Yeah, it’s, you know, I’ll start with, you know, the biggest problem coming from marine science into IT is just the degree itself. Because 23 years ago, I’ve been in the business 23 years, actually 24 now. that marine science just wasn’t who you hired. You hired an IT guy that graduated with a programming degree or something else within IT. And it’s about getting that foot in the door that first time. But, you know, I always had, my father told me growing up, many, many times, all that matters is you get the sheepskin. It just means that you can learn highly technical material very quickly. Nowadays, it’s not like that. Nowadays, you can learn a lot just watching YouTube and LinkedIn learning and Coursera and things like that. You don’t necessarily have to find that degree. And I’ve also noticed I’ve done a lot of work in BI that more and more people are hiring for BI purposes, art majors, other creative majors that or people that are just really good at art because it’s all about visualization. How do I? show the data in a way that’s most appealing and it’s going to get you to the easiest. Gone are the days of Excel notebook, Excel workbooks, you know?
Speaker 0 | 02:07.192
And kind of a twist on that. One of the things that I’d come to realize in the last few years was, you know, give me somebody with the right attitude and the right mentality, and I can teach them the IT. I can get them that technical skill set. But But having somebody that is customer service focused or, you know, detail oriented, those are some of those things that you need that person or the personality with that. And like I said, I can teach you tech. Teach you tech is like breathing.
Speaker 1 | 02:47.722
Right. Well, and one of the things that I found out, too, is I’ve hired people that have been in tech and not been in tech. And I’ve mentored a lot of people throughout my career. And that’s I look for a culture fit or a fit within my team. Is this person really going to fit? That’s what I look at first. I you know, it doesn’t matter whether you’re a graduate, high school, college, doesn’t matter. At the end of the day, just like you said, I can teach the tech. I can because a lot of times I’ve hired people that have come from other companies and they bring what I would consider non best practices with them. And that just. that just causes wreaks havoc and it goes against policies and things like that. And it, you know, it ends up, you end up having to remove the ties from the company with them at the end of the day, if they don’t change.
Speaker 0 | 03:40.597
You built policies, you had like best practices and wait, it sounds like you had documentation.
Speaker 1 | 03:48.401
Yes, I did. I, I throughout my career.
Speaker 0 | 03:52.802
No,
Speaker 1 | 03:53.022
I’m kidding. So You know, and that’s a good point. Documentation is so huge and it goes into what we’ll be talking a lot more about on cybersecurity and disaster recovery is documentation is key. And the first company I worked for was in IT was ABS pumps. And then I worked for DHL third party logistics. It used to be Excel logistics and then DHL purchased them. By the way, DHL is the world’s When I left, they were the world’s largest third-party logistics company. And we had a saying, if it’s not documented, it doesn’t exist.
Speaker 0 | 04:35.647
Interesting. Okay.
Speaker 1 | 04:37.327
So we documented everything. And I had at Excel or at Excel slash DHL, I actually started off working as an application specialist. Before that, at Texas A&M, I was at the help desk. And so I even worked offshore in marine science, just so everyone wants to know. I did try working offshore. I was 250 miles due south of New Orleans. working on a seismic research vessel for six weeks and i hated the isolation um i was just graduated out of high school just engaged and i didn’t want to be from away from my fiancee for that length of time so i came back on called one of my buddies and he said yeah come on over and i’ve been in i.t ever since so did you only spend one rotation out on the ocean like that One rotation. That’s all it took for me to not like it. Yeah, that was enough. Exactly. Exactly. But, you know, nowadays it’s a lot different. Back then, you know, 24 years ago, you had a ship to shore phone. You didn’t have Internet. You didn’t have anything. So the ship to shore phone was like ten dollars a minute. You had to pay it to right then and there. It was pretty.
Speaker 0 | 05:57.330
And you would have had to have downloaded six weeks worth of videos and. At that point, getting a hold of that kind of stuff and the storage needed for it, you would have had to have had a separate hard drive to plug into your laptop because you didn’t have tablets. Right. Oh, man.
Speaker 1 | 06:16.918
It would have been even worse. You would have had to have several hard drives because think about how much space it takes. You would have had to have probably back then the hard drives weren’t terabytes.
Speaker 0 | 06:26.526
Right. I was just going to say that. They weren’t terabyte size. I think at best or the biggest. was maybe the 512s and even the 512s were huge and expensive and and not to mention the fact that they were the big bricks yep and heavy yes exactly so it just you know it was complete isolation it really was yeah like it’s not like a cruise ship right right and i’m thinking of something else then so this also tells me that your personality is not a personality who likes to stay in the back closet
Speaker 1 | 07:00.095
and have pizza slid under the door and maybe a red bull or a you know what was there a jolt cola right right well you know there there was a time where uh that was the case and that’s during uh our ransomware attack that we had had and we’ll get into that in a little bit i’m sure but it’s you know part of the career path is you never know i’m i’m a christian and i’m a deacon at our church You know, God puts you in situations and in jobs that you you wonder why. And then he makes you change and points you in a different direction. And that’s fine. And it’s been you know, I trusted him and he’s taken me to where I am now. In order for me to stay in marine science, I would have had to go to school for another six years. And I just wasn’t ready for that. I was ready to get out into the workforce.
Speaker 0 | 07:57.871
yeah so you got that that chance so you talk to your buddy he he offers you a chance you slide in as help desk not not really having much experience yet right and so yeah not much experience and and where’d you go how’d you get how
Speaker 1 | 08:14.785
do you get from isolation on a ship to help desk to director so it’s it’s a lot of work it’s dedication obviously A lot of dedication, a lot of hard work, a lot of studying on my own and just school of hard knocks. You know, I think as you go through and you you’re going to have failures. Everyone’s going to have failures, even at director, VP, CIO level. It doesn’t matter. We’re all going to have failures. It’s how you bounce back.
Speaker 0 | 08:46.463
You’re not doing anything.
Speaker 1 | 08:48.163
Right. It’s how you bounce back from those failures, how you react to those failures. And it’s changed throughout the. throughout life and I’ve had great mentors in my life that are still involved in my life. Not just my parent, but career mentors. One of my mentors at the last company, Peter Homschoten, he was a CFO. Great mentor. Showed me and taught me how to interact with board of directors. And I’ve had several other accounting and FP&A people that have been mentors as well as professional level mentors as well i’ve had excuse me i’ve had consultants that we’ve hired that mentored me and showed me and helped me navigate politically uh and show me what i need to learn active directory back then was the huge thing nt and active directory right so that was one of the first things and then what’s funny is on ram active directory oh man yes yes on nt okay keep going sir Yeah, and you know, one of the interesting things was I had the opportunity at DHL to do a lot of things. It wasn’t, it started off as help desk and application support, and then it progressed to well we need this who wants to do it on the team and you can’t be afraid to say i’ll do it i don’t know anything about it but i’ll do it yeah and so that’s that’s what you do and i progressed into a programming situation where i started supporting uh some custom programs that were developed in vb a lot of people probably know vb as dot net nowadays not many people know vb anymore But I did a lot of VB and then we had some database guys leave and they said, hey, you want to do databases, too? I’m like, sure, I’ll do it. So you can’t be afraid to extend yourself. It may not be comfortable, but just know you can learn. Everyone can learn this stuff. You know, you have to be open minded.
Speaker 0 | 10:55.805
If you’re comfortable, you’re not learning. If you’re comfortable, you’re not you’re not stretching yourself. I’m stretching myself if I’m comfortable.
Speaker 1 | 11:05.930
Right. Right. Well, and one of the things, too, is my throughout my whole career, I’ve never wanted to pigeonhole myself into one technology because that one technology at some point in time is going to be legacy. So I want to make sure that I understand what the technology can do.
Speaker 0 | 11:23.998
So hold on a second. Explain that a little bit more, because like when when you say that, I’m thinking of to be honest, I’m thinking of being pigeonholed into one. vendor, not necessarily into one technology like going from ITSM or help desk into databases or into what was the one in between the databases and help desk that you mentioned?
Speaker 1 | 11:49.390
Databases, vb.net or vb, vb program.
Speaker 0 | 11:54.993
Okay, program. All right. So help desk programming now databases, that’s teaching you SQL, that’s teaching you structured query language. Yeah.
Speaker 1 | 12:05.815
t-sql equal well let alone database design yeah um you know and and i actually ended up i never got my certification but i took all the classes for uh master and oracle master and microsoft sql and master and mysql you
Speaker 0 | 12:22.104
know here’s something that i find really ironic in our in in our um careers so we started off i and i’m i remember We had to get structured databases and we had to get rid of the data deduplication and all of these things to normalize the tables. And then suddenly, suddenly, big air quotes around that word, suddenly we’re dealing with, you know what, it doesn’t matter. Here’s let’s go for the star schema. And, you know, those are still semi normalized tables. But we’re starting to duplicate the data again within the database.
Speaker 1 | 13:00.623
for reporting purposes and and it becomes all about the speed of reporting and and you throw that in you throw that in you sit there and and you think from web design standpoint i want things normalized because in the traditional quote unquote traditional way that’s not an efficient way to do things but right you know you start looking at it and going okay it kind of makes sense in some situations And it’s thinking outside of the box. But back to your question, how do you, how do you, what do I mean by pigeonholing into that, into one technology stack, so to speak?
Speaker 0 | 13:40.137
Right.
Speaker 1 | 13:40.737
It’s just like Fortran. Do we even hear about Fortran anymore?
Speaker 0 | 13:45.362
Yeah, on shows like this.
Speaker 1 | 13:47.363
Well, yeah. But you don’t see a job out there that that’s looking for a Fortran expert. Because it’s legacy. And when you pigeonhole yourself and you specialize in one particular area, at some point in time, that technology is going to go away. Unless you’re an AS400 guy. AS400 never dies. So,
Speaker 0 | 14:09.400
you know. In multiple ways. They are solid machines. They never die.
Speaker 1 | 14:13.842
They are solid machines. Yes.
Speaker 0 | 14:17.183
I killed, in my 20 years, I brought down, I, not me specifically, but I was responsible because I was.
Speaker 1 | 14:24.867
the head um we brought down our 400 at least three separate times wow wow ab and just stops um those were bad days yeah they are they are as400 is a very powerful and resilient machine for sure yeah um but i i have a funny story about that that one of my jobs we had an as400 and it kept going down every friday night at seven o’clock eight o’clock around that time frame They looked at all the event logs, power loss, what’s going on. We have no idea what’s going on. This went on for two months. couldn’t figure out what was going on it’s where it was plugged in isn’t it so the the it is but you took my punch line um so no that’s okay at the end of the day they uh the as400 admin said look i’m just going to spend all day friday and all night friday i’m not going to i’m not going to sleep i’m going to monitor it all night he was in the server room and uh he went out he went to go get something to drink came back and the house cleaning company was in there and they plugged in they unplugged the as400 to plug in the vacuum the vacuum in the offices where the as400 was so and then she would vacuum and plug it back in and that’s that’s what was going on i’m
Speaker 0 | 15:51.874
sorry to steal that thunder man that’s okay how how could you because they don’t those machines
Speaker 1 | 15:59.637
don’t drop unless you do something like that yeah exactly exactly so that that’s that’s a funny story it wasn’t it wasn’t you know an it help desk type story but it was definitely one of those things man how do you how do you see that in the logs yeah you don’t you don’t you just see power loss how do we lose power there’s no electric power outage or anything what the heck happened so they even they were actually getting to the point where they’re going to swap out power supplies
Speaker 0 | 16:29.045
Now, let’s take it, instead of it having it under somebody’s desk, even though it’s too big to go under a desk, let’s put it into a room where there’s battery backups.
Speaker 1 | 16:42.732
Yeah, it’s crazy. But that was 23 years ago when that happened, and battery backups were expensive. All that stuff was expensive.
Speaker 0 | 16:51.157
Oh, man. And I keep telling other people that I talk to in an interview, I think the deadliest. thing that I ever added to any of my data centers in a closet were the UPSs. The UPSs have caused me more pain than anything else I ever did.
Speaker 1 | 17:08.568
And then if you want to monitor them, it’s a pain in the butt, right? I mean, if you’re using APC, it’s the smart connect and all that stuff. And it’s, yeah, it’s always a mess, but you have to have them.
Speaker 0 | 17:22.398
Yeah. Well, you should have them.
Speaker 1 | 17:24.980
Well, you should.
Speaker 0 | 17:27.022
best practices are you should have you should have them you have to have them so oh wait a minute sudden sudden connection too so the 400 was in a room that had carpet where people where it needed to be it was it
Speaker 1 | 17:43.918
was tile it was tile but she went in once a week and vacuumed the tile up and so it was like okay just So after that, they obviously put a keypad on it and told her not to go in there anymore. And IT would take care of cleaning the office, that space. So, yeah. Oh,
Speaker 0 | 18:01.161
so now new job for IT because, yeah.
Speaker 1 | 18:04.904
Well, and back then, you know, a lot of times it was, you know, you had a key code pad that you had to put in and the maintenance guy didn’t know how to install it. And so I was like, OK, IT is going to do it. Yeah. You wore many hats back then and it was a small company. So we were used to wearing many hats. Yeah. And I think that’s really what. So going back to the tech stack thing, I mean, really.
Speaker 0 | 18:26.687
Sorry, I never let you back.
Speaker 1 | 18:28.147
No, no, no. That’s OK. I’ll get us back. I’ll get us back there. No, it’s I’ve just always in my career, I’ve always felt like in order to go up, I need to understand not only networking. I need to understand how programs interface, the network interfaces, the databases interface, knowing that interaction from a tech stack standpoint and understanding especially the security side of things. Let’s make sure that the applications that you’re developing are communicating securely. Let’s make sure the network’s secure and allowing the proper traffic through. Let’s make sure that the databases are communicating securely. And we have a lot of compliance stuff nowadays with SOC compliance and everything else that are getting more and more stringent. And it’s easier to understand how all those interact because that allows you to go to that next step much simpler.
Speaker 0 | 19:25.850
Yeah. So help me out here because I know, at least in my world and the peers that I was talking to, I was out of the norm. talking about security and looking into security and knowing what nmap was and those kinds of things all the way through i want to say through like 2010. um so so when you started having this security awareness and blending this into the whole infrastructure and understanding what that security posture was as you looked across the infrastructure um What point was this? Because you obviously, come on, you didn’t have that out of the gate when you were doing your VB programming. You might have heard about some of it. As you went into the databases, you started to hear about SQL injection. I’m betting that was probably about the time that you were starting to do databases was the emergence of SQL injections.
Speaker 1 | 20:26.727
Yep.
Speaker 0 | 20:26.987
So as an overall view of the… of the infrastructure, not just the network, because it’s so much more like you were talking about. Where were you in your career and what time period are we talking?
Speaker 1 | 20:41.376
So from help desk to that was probably about six years to database. And then the realization, once I started learning about SQL injection, and I’d already learned a lot about virus protection and malware at that time was the big stuff and help desk, obviously. And, you know, something in my head just was like, all this is linked. If they can fish and they can steal credentials, they can get into SQL. They can get into here. They can get into there. And it just kind of hit me that, okay, we need to talk to, and I talked to my IT manager at the time. And I said, look, we got to make sure that we. protect the network. We got to look at these programs. We got to make sure that they’re secure because this is going to be a big issue. I can already tell you it’s going to be a big issue. And it turned out to be a big issue that I actually survived. Thank goodness.
Speaker 0 | 21:44.444
So before we get into the survival story, when you said you’re your IT manager at that point, was he reporting to you or were you reporting to him?
Speaker 1 | 21:56.952
I was reporting to him.
Speaker 0 | 21:58.253
Okay.
Speaker 1 | 21:58.826
cool and he happened to be my best friend at Texas A&M so we already had a great relationships okay hey the network works exactly and then um you know we and I learned a lot from him he was a network guy and so we were able to start talking about the applications and what ports were being available we started looking at really using in map and uh angry IP and all that all that fun stuff that people still use thank goodness that’s one technology i think is it’s evolving and it’s it’s it’s really good if you know how to use it right right and we started looking at all of the ports that were being used in the network traffic with more uh with with magnifying glasses really yeah
Speaker 0 | 22:44.357
what’s actually happening you guys were probably segmenting your network internally users on one of one subnet servers on another subnet probably if you’re paying attention to the um i guess they call that the east west traffic cruising across your network um you probably have internal firewalls and everything too yes yes we did and so you know and back then we didn’t have high speed internet it’s a norm yeah
Speaker 1 | 23:13.165
no i mean without high speed internet it was you know you’re more worried about the external traffic but you knew you could probably stop it by pulling the plug on the internet not a big deal but you know it but at the end of the day it was let’s look at the traffic because you know it’s the more traffic that’s going through and we’re not able to if we just leave it wide open like it was for the most part there were we had a lot of things that were blocked standard ports and stuff but um you know we started in starting seeing increase in speed and the erp system started increasing in speed uh it was starting being more responsive and so we ended up actually improving efficiency of the overall system and that that was huge that was huge and then you know the leadership at that location saw that we were able to do something like that and then they brought us in this is something you and I hadn’t talked about uh they brought us in to look at a cost reduction program and we developed uh I was I was a project manager for this and we developed um at that time forklift this was a logistics center so We had a lot of forklifts. They were running on 900 megahertz, green screen on top of the forklift. And at that time, HP had just come out with their tablet, the TC1100, I think is what it was, the first enterprise tablet, so to speak. And we found a way to mount them on the forklifts, and we improved their efficiency significantly. 33.3% by 33.3% on the forklift.
Speaker 0 | 24:54.176
And see, this is typically, we always hear about IT is such a cost center, cost center, cost center. This is where… We really, we as IT really need to gather what that 33%, what the value of that 33% was and bring that back to the organization and talk to others because we’re not a cost center. We are a force multiplier and we can help make, they got 60% increase out of that, you know, a 33% efficiency. So they had to do 33% less work.
Speaker 1 | 25:30.714
but that opened up another 33 to do something else so not only were they achieving the same goal in less time but now they’ve got more time to do more right and well and the interesting thing about it is is our customer uh they saw the success we had with that solution and they replicated it within their own distribution center so it even helped them out so it ended up at the end of the day uh we did some other tweakings as well from a technological standpoint and uh we did some retraining and things like that and uh it ended up saving our customer um about a 25 on their invoice from us okay so
Speaker 0 | 26:15.020
which was in the order of about a million dollars a month all right so you you’ve alluded a couple of times wait a million dollars a month yes that doesn’t suck
Speaker 1 | 26:26.070
No, it doesn’t. It was a very high volume. They were about $42 million a month on their invoice previously.
Speaker 0 | 26:33.131
Wow. Dang. Okay, cool. That is money brought back to the organization from IT, leveraging the technology to accomplish a goal.
Speaker 1 | 26:45.815
But you know, you have to get involved. You can’t stay at your desk. You have to be vocal. And I think that’s one of the things that a lot of IT organizations miss. is you have to go look, you have to let upper management know we’re in, we are going down this mindset and we’re looking at the network. We’re looking at speeding up the network. We’re looking at whatever in order to get there. And when they see those results and you bring back the results, great. Yeah. You have to educate your employee. You have to educate your employees and your board.
Speaker 0 | 27:22.113
Right.
Speaker 1 | 27:23.373
Or the owner.
Speaker 0 | 27:24.422
whoever or the investors whoever it is and sometimes the board is probably one of the more difficult ones yeah because a lot of us get scared to go in there and talk to him oh my god he’s got a suit yeah he’s got us well you know what they’re people just like we are they
Speaker 1 | 27:42.817
are and they’re going to ask the tough questions and if you don’t know you can’t be afraid to say i don’t know the answer but i will get back to you and you get back to them as quickly as you possibly can And that’s one of the things that you have to have that confidence in front of that boardroom. And you can’t get frustrated because it could take three or four board meetings because very few times is there going to be someone on the board that’s a tech guy that understands technology and understands your terminology. So you have to start at that very high level and get them to understand what the acronyms that we use are, understand what they mean and how it affects. your network, your applications, whatever it is, your cybersecurity program, whatever.
Speaker 0 | 28:26.855
Even if you do have that one guy that understands it and knows it, he’s still got to explain it to the rest of them.
Speaker 1 | 28:33.076
Right.
Speaker 0 | 28:33.336
So you’re relying on somebody else to do that explanation for you. They steal it and they get all of the, the accolades for understanding versus you being able to tell them.
Speaker 1 | 28:44.739
And it’s going to take time, right? It’s going to take, in my experience, I’ve seen. I’ve seen it take three or four board meetings for them to, for you to finally get to the thing that you want them to do. Right. And it’s that education that we have to be patient on, you know, it’s always really patient. So yeah,
Speaker 0 | 29:06.511
sure. We are.
Speaker 1 | 29:08.192
It’s it’s I say that just for sure, but you know, if it’s important enough for us to go to the board, then you have to stick with the program. You can’t let a board meeting go one or two months till you come back.
Speaker 0 | 29:22.042
to them you have to you have to be relentless in the education of them right so and the education of the employees too you’ve alluded to this this ransomware attack tell them let’s i i want to hear about that because those those are always fun stories i i’m sure it wasn’t fun to live through but but it’s it’s interesting to find out and and hear about the experience after the fact and and having you know where’s where’s some wood man knock knock on um on wood that that I have yet to be through one um and I hope that whenever I do that I get to say hey I saw it come in land and we killed it well and I have to say something I’m glad you said that
Speaker 1 | 30:04.828
If and when I go through one, that’s exactly it. It is, you know, and that’s one of the things that I drove home with the board of directors. It’s not an if, it’s a when. When are we going to get attacked? And you have to hit that big time because nowadays with AI and machine learning, it’s even, they’re attacking even faster and more sophisticated than they ever have. And it’s not just the big companies, it’s the small companies too. And… You know, we started off really well at this company. It was a startup. And I was in charge of the disaster recovery program. And I started off as a manager or the director of applications, which was the ERP and all the other custom applications we had. And I set up the DR program. We tested it quarterly. We did a yearly full test. It replicated all the information. every hour on the hour. We had redundancy as well. We had two on-site DR servers, and then we had a DR server in a third-party cloud environment. So at worst case scenario, if we lost everything, we had at least one hour of data that we might have to bring back quickly. So this was during COVID. We, gosh, it was six months into COVID. Everyone remembers and everyone probably has heard of the Garmin GPS ransomware attack. We got hit with the same ransomware. And it came through. It came through a very innocent looking phishing email that bypassed Microsoft’s phishing detection. Surprise. Yeah, exactly. And then they. they watched him we worked with a company called kroll i don’t know if anyone i i would hope that some people have heard of them if you haven’t check them out they do they do some really great work um yeah kroll k-r-o-l-l okay and they did a really good job at helping us investigate who patient zero was so to speak and we found out how they got in but during this you know we’re six months into it it’s three o’clock in the morning and I get an alert from one of my servers that was still up that something was going on and it shut itself down. And so I get up and I drive to the data center. I call my network admin. He drives to the data center and we start unplugging everything because we start seeing servers going down. We unplugged it. It took about three hours by time I got the notification and we got through all the security at the data center. And the data center is about an hour away from where i live and my network admin and uh they had encrypted in three hours almost every single server and they had also encrypted the two on-site replicated dr servers they
Speaker 0 | 33:14.844
never got to the cloud server okay so uh we spent uh they i assume they they waited until they had control over the backups also because this sounds like a smart attack i mean it’s not yes
Speaker 1 | 33:31.757
It was a very smart attack. And I mean, to give you an idea of the scope, it was 80 servers that they had encrypted, including our ERP server, our file server, everything. And so it was we got a ransom note, obviously, and we had cybersecurity insurance. So they took care of talking with the ransomware people because. the one thing that they told me is do not email them let us do it on behalf of you because that just opens up a can of worms and um you know having that dr program it’s one of the key things that i want to say if you when you go through this as a leader the number one thing you have to do and you have to show not only to your team but to management is that you’re calm you’re calm and collected you have urgency But you’re calm. You have to keep calm. And it’s really, really hard. Trust me. I’ve been through it. I never want to go through it again. But I know there’s going to be a time where I’m probably going to have to go through it. Because the team and the company and the leadership needs to needs to see you stay calm in the middle of these storms.
Speaker 0 | 34:49.933
Yeah, for sure. I mean, that’s I can’t help but think of George Costanza.
Speaker 1 | 34:59.664
Yeah, exactly.
Speaker 0 | 35:02.628
Exactly. it is your your team needs to see you um breathing and thinking and going through this and leadership needs to see it too um you can move urgently you can move with with um speed and purpose but
Speaker 1 | 35:22.125
but you still got to portray that calm right well and you know you have to think about this this was six months after covet hit so we had all the coven protocols and everything else that we had to go through at the data center. And that’s what took forever to get through. You know, it was just like, oh, my gosh, this is ridiculous. Six feet apart. So my network admin had to be on the backside of those servers and I had to be on the front. We couldn’t be next to each other. It was an interesting time to live through, to say the least.
Speaker 0 | 35:57.087
Okay, yeah. I hadn’t picked up on that part of this.
Speaker 1 | 36:00.886
Yeah. So here’s the great thing about it. Because of our DR program and our best practices and us going through it on a quarterly basis and doing a full restore every single year, we knew exactly what to do. And a lot of people are going to be a little astonished by it. Everyone that I talk to about it is astonished. We were actually able to recover everything within five business days. And I mean everything. uh since it hit on a monday morning there was no weekend transactions so we didn’t lose any data um and we did not have to pay the ransom by the way the ransom was seven million dollars so hey force
Speaker 0 | 36:45.988
multiplier another saving um exactly you know hey so the other day when when we were talking i was at a seminar being held by infragard and the fbi and and they were talking about the ransomware attack that hit Atlanta and how months later, eight months later, they’re still not, they’re still sending out paper invoices and people are still not able to like clock in and clock out. And I don’t, I know I’m not giving the exact details, but it was for a city not to be able to handle this. They talked about Denver who handled it much better, but it was still Um, I think weeks, not days before even they were clear of it and they got it shut down quickly.
Speaker 1 | 37:36.324
Yeah. And I think a lot of that is because, I mean, let’s face it in IT, a lot of times other things take precedence over testing your DR, testing your recovery program, testing your business continuity. And that’s one of the things that was one of our top priorities. It was our number one thing every single quarter at the quarterly business meeting that I went through. This is what we did. This is how long it took us to recover. This is how long it took us to to redo everything to recover from backup, all of that. And you have to be diligent about it has to be a top priority.
Speaker 0 | 38:11.707
And it’s got to be across the organization, because if operations, if whatever your business is, it has some form of operations, the people who are actually doing well, you know, air quotes around this again, doing the stuff that actually generates the money that that group. doesn’t want to stop just so that we can run a fire drill but right i don’t know about you guys but you know i we were doing the same kind of thing we were trying to have um quarterly tests of the um the roll swaps the the data recovery all of the disaster recovery and each time we did it we always found something that we’d forgotten you know something’s broke you get back and everybody’s like okay all the blinky lights are blinking and and
Speaker 1 | 38:56.978
one of the users goes um hey i can’t do closer yeah yep and you go and that’s one of the things that i think is you know it was a private equity firm so in the beginning we had lots of money right they always throw lots of money in the beginning and we had a lot of money to put towards it and we had a dedicated team in operations it every function that we would send out and we would say hey Don’t forget, we’re doing our quarterly. Here it is. This is a date. We’d send them a calendar invite saying block out this whole date. We’d let their managers know it’s communication. It’s you have to communicate and let them know that, hey, this person is not going to be available for a whole day because we’ve got to go through the entire test script. They’ve got to make sure they can create everything. And that’s what we would do. And we would do it on the weekends and we’d bring pizza in and all that kind of stuff. And it was we tried to make it fun. or at least not fun enjoyable right at least you were like i have to be here on a saturday at least give me something right yeah exactly let them have the pizza that they want the sodas being able to come in in the jeans ripped jeans and and their smart alecky it t-shirts and you know just yeah well and all it had to be there too so we had some people that were remote that they would just remote in and zoom all day and it was you know but everyone was there and we could handle the issues right then and there with urgency because oh you can’t get there let me look and see let’s fix it let’s go on is it big yes okay cool Now we know where we’re at and we can move forward, which, you know, it’s it leads to, you know, that I had been going to the board for for several years for a cybersecurity solution other than Microsoft’s because they they hung their hat on Microsoft. And I was like, look, they didn’t let it through. Finally, you know, I had proof that they weren’t as good as they were. And they they approved bringing in Darktrace. So we brought in Darktrace on the network and endpoint solution that they had. And at that time, it was about $172,000 over three years.
Speaker 0 | 41:16.692
Yeah, I was going to say, Darktrace is one of those premium, and I’m making that money symbol, those premium solutions. But private equity, I mean, there’s a lot more money involved there. Yeah,
Speaker 1 | 41:29.903
there is. But I tell you what, after that. the amount of information i mean if you if you ever have the opportunity to test a dark trace test them have them put in their network device have them put it in their office 365 uh everything let them test everything you’ll be amazed at what their software can find amazed we we went from uh so microsoft blocked about uh 10 percent of malicious emails okay and yeah i mean 10 of our emails were malicious that we got and um dark trace was actually added another 25 to that okay so they now now they’re using ai and it’s even better and the great thing about it is it’s sun up to sundown 24 7 365. I would get phone calls from the SOC at one o’clock in the morning saying, hey, we see some weird traffic from this. Oh, I forgot to tell you guys, we installed this new application and it does this. Okay, we’ll only let that application through. You know, it was phenomenal. It was great. There’s a learning curve like to any cybersecurity software. But once you get there and it’s stable, it took us about three weeks to get used to writing the rules. and they helped us all along the way as all the other companies do and uh i mean they it ended up being an asset to the sale of the company because it was private equity owned and and like i said before our our goal was to be an asset to the sale of the company that’s what it was and it ended up being an asset to the company yeah um but it takes leadership it takes communication you know it takes that vision and that strategy to get you there and you just have to be relentless at it.
Speaker 0 | 43:31.215
Right. So, you know, disaster recovery, the fact that you got to try it, you got to not only try it, you’ve got to run it, you’ve got to practice it, you’ve got to, if you don’t do that, you’re not doing it. You know, yes, you can, you can put it on paper, hey, we have a plan, we have these things, but when, if it’s your first time running through it,
Speaker 1 | 43:57.902
when it’s going on that’s a problem yeah if you’ve never tested it you don’t have a dr solution yeah well not a real continual testing yeah you’re not good at it i look at it this way if you if you say you have a dr and you don’t test it you just have backups that’s all you have right
Speaker 0 | 44:16.587
so um quick question you guys found patient zero how long between patient zero and the actual
Speaker 1 | 44:26.582
that that three-hour run on the servers um it was they were on patient zero system for three months for three months doing recon figuring things out finding out where the dr
Speaker 0 | 44:42.126
servers were finding out backups finding out all of these different things so that they knew what to hit so that they could make it hurt the most exactly exactly well and the funny thing is we were testing
Speaker 1 | 44:56.390
We were testing Darktrace right about a month before and Darktrace had said, hey, we see some weird traffic. And so we were looking at and we said, OK, let’s just kind of quarantine it. And then next thing we know, boom. Well, and Kroll had looked at looked at everything and they said it was time bombed. There was a script that was running that was calling back home. And if it didn’t talk over a certain period of time, it kicked off. It kicked off a program.
Speaker 0 | 45:33.230
that was already pre-written oh man so that we that we never knew was there so it was exfiltrating the data while it could and as soon as it recognized that it could no longer exfiltrate then they moved into a different um
Speaker 1 | 45:48.980
stage or a different attack type exactly and it was and it was something that we would have never caught if dark trace hadn’t been there yeah just doing it we were just doing a poc a proof of concept right And they were going to help me with providing data to the board of directors because they have a nice executive dashboard and stuff like that. But it’s, you know, if it wasn’t for them, it probably would have happened at some point, I’m sure. But it would have been more disastrous, I think, and we wouldn’t have been near as protected afterwards. The good thing is, you know, you mentioned they were exfiltrating data. They actually weren’t. They were just pinging the server. They were just pinging their server.
Speaker 0 | 46:32.626
Yeah, and I wondered if I had misspoken when I said that. But in other words, it was, yeah, it was the keep alive, going back to the command and control center, the triple C, and letting it know, hey, we’re still here, we’re still here.
Speaker 1 | 46:47.892
Well, and the thing is, it’s just a heartbeat. It’s just like one kilobyte. We never even saw it on any of our network management stuff. Never saw the traffic. I mean, you’re never going to see a bump in traffic. It’s one kilobyte. You don’t care about one kilobyte.
Speaker 0 | 47:02.602
It’s hidden within a DNS request. It’s hidden within regular traffic. It’s just a separate payload that you have to know every packet and what’s going inside of it to be able to catch something like that.
Speaker 1 | 47:18.669
Right. And let’s face it, if you enable packet inspection on a firewall, it greatly reduces your network speed. It does.
Speaker 0 | 47:26.572
Yeah.
Speaker 1 | 47:27.892
So at the end of the day, what do you do? Right. And and we were lucky that they had there. And and I mean, the great thing about those systems is they’re taking if there’s an attack in Europe, they’re going to already have a solution in place by the time it gets to the US and vice versa. So it’s you know, those are some of the benefits of having a 365 stock sun up to sundown stock like that.
Speaker 0 | 47:53.492
All right. So I want to steal something from one of my co-hosts. He loves to throw out this thing called random access memory. And I want to hit you with a question and you give me whatever comes to mind. And that is, if you could change or improve one thing about IT, what would it be?
Speaker 1 | 48:13.838
Oh, yeah, that’s one of the things I think traditional, and I’m going to go this route, I’m going to think traditional IT is getting out of the norm. thinking outside of the box we’re usually pretty good about it but a lot of times we’re so um we’re so it’s so ingrained in us to do things one way and one way only and um i always have i’ve had a phrase that i’ve used my entire career is uh i never i do not like to say no because there’s always a way um you know if it takes me you know if it takes me honestly i mean when we look at when I look at the amount of time that I spent on that project for the forklifts for DHL, I spent months on it. We tested so many different things because it had to be powered by electric forklift and there were different models and different mountings and we tested so many things.
Speaker 0 | 49:18.600
I want to jump in real quick. I couldn’t agree more because I’ve had that philosophy too. It’s not been one of the ones that I’ve shared externally. But I’ve had this belief that, you know, there’s always a way. But it’s going to take three things to get there. It’s going to take the education. It’s going to take the imagination. And it’s going to take some money.
Speaker 1 | 49:38.412
Yep. Yeah, I was going to say money is the one that everyone gets stuck on. Yeah. But it’s our job as IT leaders to give them the value for the money and to show them what the value is. And we did a pilot with that program, which is two forklifts. And we showed them. how much it’s going to save them. And I mean, ROI is huge and we can’t be afraid of that ROI name, but we have to have the proof behind it in those presentations and those discussions behind the doors. with the leaders that control the finances. It’s not easy. It’s not easy.
Speaker 0 | 50:13.642
Being able to get them to see the new vision. Because, I mean, think about the guys, think about the first ones when Netflix came out and it was like Redbox or, you know, it was the male subscription of DVDs was the beginning of Netflix, let alone the streaming, which is now I’ve cut my cords.
Speaker 1 | 50:36.141
Yeah.
Speaker 0 | 50:37.290
You know, everything’s streaming services nowadays and the Uberfication. I hate naming a company like that, but there’s such a shift in how things are handled. It’s such a radical departure from the norm like you’re talking about.
Speaker 1 | 50:55.437
Yeah, it is.
Speaker 0 | 50:55.817
How to be watching for those and looking for AI. How are we going to apply AI to what we’re doing that’s going to help give us a chance?
Speaker 1 | 51:05.561
Right. And I… I’ve just started learning AI and, you know, the possibilities are just crazy. I mean, they really, really are. And, you know, in one of my classes, the teacher actually brought up something and people are afraid AI is going to take over their job.
Speaker 0 | 51:22.889
The singularity.
Speaker 1 | 51:24.370
The singularity.
Speaker 0 | 51:25.130
I agree with that, too, but hopefully you know what the singularity is. And I’m talking to the audience, not you. I’m pretty sure you know what the singularity is.
Speaker 1 | 51:33.374
But, you know, it’s one of those things she she likened it to the old switchboard operators. Right. So we’ve all seen them in the movies. They have to say, well, who do you want? And they take cords and they plug them into this board that’s got everyone’s phones. And when the auto dialers came, not auto dialers, but the automated operators came out, all those jobs went away. The automated switchboard went and took all those jobs away and they had to switch and change to a different. skill set. And that’s what IT is. We have to continually change and add skill sets. I mean, if you’re an IT and you don’t, if you want to grow in your career, you have to be willing to learn new things and learn them quick. You don’t have to be a master at them. You just have to learn them. And I think as a good leader, I’ve always been interested in new technology and it’s always interesting to go, where’s the technology going to go? I didn’t see AI early enough. I wish I would have agreed, but it came on so quick. It really, I feel it came on very quick and, but the possibilities there are phenomenal. But the one thing we have to remember there’s, there is a human aspect to AI. Someone has to program it.
Speaker 0 | 52:49.634
Yeah. You got to feed it something. You want a lot of something.
Speaker 1 | 52:55.079
Right. There, there is a human behind it. There is an. As IT leaders, it’s our responsibility if we’re going to go into AI and really look at AI as a viable option for our companies, then it is our responsibility, our duty, to actually ensure that AI is operating in a responsible manner and utilized in a responsible manner within our organization. Because AI is very powerful. I mean, let’s face it. you know, the cyber thieves out there are using AI and machine learning. That’s how they can infiltrate so quickly now. And that’s how they can find the holes in your network and your security so easily.
Speaker 0 | 53:44.690
And now they’re going to help try to bypass the AI you’re using to help find them.
Speaker 1 | 53:49.191
Exactly. And that’s, you know, and that’s the thing that’s changed so much because, you know, when I first started thinking about cybersecurity, you know, gosh, 15, 16, 17 years ago, something like that. It was actually not that big of a deal because it was a physical person having to type all this stuff out. It wasn’t near as fast, right? We could stop them. We could see it, stop it pretty quick. Nowadays, it’s at machine speed. It’s at machine speed. And these guys, I mean, I can just imagine they’re probably at a data center with, you know, 20 different Xeon servers out there that have
Speaker 0 | 54:27.102
50 cores in them and they’re running so fast that it’s ridiculous so you have to have a solution that can run at machine speeds as well yeah oh man it just makes my mind i i spin off into all of the different possibilities and like you just said man i wish i had seen or known that it was coming more than when chat gpt and and i started hearing the articles about oh kids are going to be able to um quit doing homework and and are going to be able to cheat and all of i started those things and then i started to become a little more aware of it and um and now the boardroom is bringing it to us and going, hey, I saw this thing on TV.
Speaker 1 | 55:10.762
Yeah. What are you doing? How do you do it? Well, and I think that’s part of the, as leaders, we have to recognize that it’s a reality. We have to recognize new technology as it comes out. We don’t have to be masters of it, but we have to understand two things. Number one, how can we use it in our company to make us more profitable or make us handle information? faster, better, more accurately, predictions, all that stuff, right? But we also have to look at it as that’s the same tool that the cybersecurity threats are using as well. So how do we use it and how do we defend against it? So it’s got to be twofold, but it all goes back to that education. We have to educate the employees, we have to educate the board or the owners of the company and invest the money to make sure that we have all the protections in place and that we have the training in place that everyone needs. And there’s lots of programs out there that can do this, that can help you through all this. And you don’t have to do it all. And that’s one of the things, you know, I had, he was a, he was a controller at one of my companies and he said, a good leader does not know everything. He cannot know everything. He just needs to know who to go to for the right answer.
Speaker 0 | 56:31.959
Yeah. So. Speaking of a good leader, not getting pigeonholed into a single technology, having procedures and doing disaster recovery and making sure that it’s top of mind, cybersecurity and all of these things has gotten you into the director’s chair. But, you know, just like we’re talking about how AI has brought a change to the world, you’ve had another change in your world. So talk to me a little about that.
Speaker 1 | 57:01.316
Yes.
Speaker 0 | 57:01.916
Tell us some more about where we can find out and help with this.
Speaker 1 | 57:05.858
Yeah. So first of all, if anyone’s interested, hit me up on LinkedIn. I’ll be more than happy to talk to you more about my experiences or you want some advice or just talk. That’s fine. So just recently, my my position has been has been terminated. They’re not replacing me with that company at that company. So I feel sorry for the guy that’s taken. They’re just adding on to him. All these projects, I had about nine or 10 projects going on. So now he’s going to be doing it. Of course, I’m in touch with him. I wouldn’t be an effective leader if I didn’t mentor the people that reported to me and people outside of my department. And so I wish him the best of luck. I am currently looking. And so, you know, I’ve I’ve been a leader for. for many years uh i’ve got about uh 10 years leadership experience um official leadership experience so to speak with from a job title perspective but um i think i think all of it is a leader um even the help desk guy is a leader because we lead people to technology and and we teach them yeah solve their problems exactly exactly and at the end of the day you know we have to have that mindset and the mentor mindset in order to achieve what we need to achieve and to achieve a highly, highly functioning team that works together. And that’s that’s my big strong suit is is building those teams up and providing the strategy moving forward, obviously, with with DR and cybersecurity strategy, as well as just I’ve done networking programming. I’ve done it. I’ve done I’ve been in everywhere. I never pigeonhole myself and I never want to pigeonhole myself. And to get there and to be that leader and to provide strategy and vision for a company is my passion. And change management, all that stuff that goes along with it. And integrating IT in with business strategy and vision for the actual business itself and operations and how we can affect and help them. is huge. I’ll spend weeks coming up with a solution, months if I have to, if it’s going to save them 10%.
Speaker 0 | 59:27.208
Right on. Well, you know, I hate to hear that your expertise has been made available to the market. But, you know, after this discussion with you, you’d be a valuable asset to any organization that’s smart enough to pick you up. Thank you. Especially with your concentration around the holistic solution. I mean, it’s not. You know, as leaders of IT, we have to have that holistic view, not just a, ooh, you know, let’s use one of your examples. Let’s go take care of all of the forklifts, and that’s all we’re going to do. No, it’s much larger than just the forklifts. So, yeah. Any last thoughts besides, hey, hit him up on LinkedIn, Chris. Yeah. You make her. Yeah. Anything else you want to promote? Anything else you want to bring up?
Speaker 1 | 60:20.744
No. You know, one of the things I want to say is as leaders, I think we have the opportunity to influence a lot of people within our lives. I’m very heavily involved with the community, and I encourage all of the IT leaders to be involved in their communities. Just so you know, I’m a deacon. at our church. I’m on the board of directors of Fine Arts Society. I’m a school board member, and I am also on the board of directors for a couple other different organizations as well.
Speaker 0 | 60:53.901
Right on. Well, you know what? There’s a gift coming out of this, man. I promise there is.
Speaker 1 | 61:01.288
I know there is.
Speaker 0 | 61:02.168
And I’ll help spread the word for you. And thank you, Chris. As we come to a close on another Dissecting Popular IT Nerds, I need to invite everybody that’s listening to a comment and rate podcast on the iTunes store or wherever you’re grabbing your copy of the podcast from. Really appreciate the support of the program and the time you invested in listening to us. So thanks, everyone, and talk to you soon on another episode.
Speaker 1 | 61:30.135
Great. Thanks, everyone.
Speaker 0 | 61:40.754
Thank you
