Speaker 0 | 00:09.586
Welcome back to Dissecting Popular IT Nerds Day. We have Matt Donahue on with us. And honestly, I’m really excited about this because you’re the IT guy that leaves work every day at 3 o’clock. And I think that that’s kind of like the dream. That’s what everyone wants to do. So you work at… decisive analytics. It’s very top secret. We can’t talk about what you guys do over there and they don’t even tell you what you do over there. They just say, kind of keep all this stuff running. That’s kind of how I imagine it. But anyways, man, tell me just how you got into IT. Maybe what was your first computer and what you actually, maybe if you do know what you guys do over there, tell me a little bit about it.
Speaker 1 | 00:47.690
Well, my first computer was an 8088 that my father liberated from his office. It had a 20 megabyte MFM hard drive. ran DOS 3.3 and had PKZip on it. So I’ve been in the industry a while. I mean that ran WordStar and I thought I was pretty high-tech back then but I think at that point the 386 was the standard computer that was in the marketplace. So I mean we were still several several gens behind but it was still something.
Speaker 0 | 01:24.368
My first real computer that I ordered from a catalog and built, because you’d get whatever it was, PC Magazine, whatever it was. And my first computer, it was my first real computer that was ridiculously high-tech and had a CD-ROM was a 386.
Speaker 1 | 01:40.992
Yeah. I did have a Trash 80. I did have a Tandy Trash 80 that had the tape recorder.
Speaker 0 | 01:47.094
That’s awesome.
Speaker 1 | 01:49.315
You know, like where you would write a program in like, C basic. It wasn’t quite basic. It was like candy basic. And then you would save it to like a cassette or you’d have to like load games like Pong off a cassette. Those were good times too. That was all black and white. Those were good times.
Speaker 0 | 02:11.379
I’ve never loaded Pong off a cassette. I would love to pull one of those out.
Speaker 1 | 02:18.122
You can still find them on eBay and they’re actually very expensive.
Speaker 0 | 02:21.764
Yeah, there’s a ton of really cool. Yeah, there’s a ton of cool, like you go YouTube all day and like, hey, I’m opening up a never been opened before that was hiding in this warehouse. Yeah, it’s awesome. So you’ve been, you said 13 years in IT is forever the last time we talked.
Speaker 1 | 02:38.998
Yeah, 13 years at one company is forever. I mean, I’ve been in IT for almost 25 years now. My background is in, I actually have a degree in history, which doesn’t seem to make a whole lot of sense. but it was what I wanted to study.
Speaker 0 | 02:54.173
Hey, mine was creative writing.
Speaker 1 | 02:56.414
I was in creative writing. So we probably both got into the industry probably by accident. You needed to make money.
Speaker 0 | 03:03.778
I needed a job.
Speaker 1 | 03:04.459
We needed to make money. I looked at $26,000 a year being a history teacher. And I mean, don’t get me wrong. I still have a passion for it. But I was offered like an unpaid internship and I was like, huh. that’s not gonna get it done. And so I fell into the industry by luck. Before the tech boom, before the MCSE boom, where everyone went to boot camps, got MCSEs and things like that. So that was, I want to say, 96. It was around 98 that things really exploded. But I already had two or three years of experience at that point, which kind of set me apart from other people. They simply had paper certifications.
Speaker 0 | 03:52.997
So 13 years at the same company. Yep. What’d you have when you walked in? That’s what I want to know. Like, what was the infrastructure like? What was the network like? Just to kind of give, you know, paint a picture of what it was like 13 years ago. Or, I mean, do you remember what it was when you walked in?
Speaker 1 | 04:07.168
I remember what it was. It was functional-ish. I mean, I’m not, I don’t need to necessarily rip apart. you know, my, my predecessor was a green, you know, who was a green IT worker that, you know, that, that, that didn’t have a whole lot of experience building an enterprise in that respect, but it wasn’t, there were a lot of things that were wrong. And there were a lot of things that, that, that we look at today would be not, not best practices. Um, there was no necessary enterprise architecture in place. Um, So we spent the better part of the first year addressing infrastructure like design. You know, for example, we had an office right down the street. right? Like literally walking distance away. Yet for some reason they had their own, their own windows NT domain and their own domain controller, which just made, and we had a, you know, a layer two pipe to them. And it was like, why, why would they have their own domain, their own security context, their own log on and everything when really their mail servers over here? Uh, it was like a, like a. It was a fractional T, so maybe 384K, somewhere in that range.
Speaker 0 | 05:32.550
Just to put it in perspective, you know, less than a meg.
Speaker 1 | 05:36.133
Yes, less than a meg. Absolutely less than a meg. I think it was a fractional T1, and we did voiceover part of it. They had their own phone system. And, you know, Colorado, my office in Colorado, again, they had their own internet, no VPN back here, and no hard VPN, and their own Windows NT domain, their own Exchange server. And no, there was no… no cohesion to that. We had our authoritative DNS servers, you know, those like NS dot network solutions dot whatever, but the ones for DAC were actually hosted by us at our own office. It was unmanned in Columbia. So, so a little thing like that. So, and I mean, does anyone even know how to change like your authoritative, you know, DNS servers? For your domain, when you’re going to move them from your premise to somewhere else and then change your DNS, your A records and whatever, it was like, whoa, we don’t even know how to do this, right? And so, I mean, but the environment was functional. People sent and received email. People had an architecture. They could log on to things. But we had probably… 30 to 50 gigabytes of used storage across the enterprise. That was our, our storage footprint between exchange, between file services, between everything that was shared on the quote enterprise was roughly 30 to 50 gigabytes.
Speaker 0 | 07:17.487
So it could almost fit on like an iPhone.
Speaker 1 | 07:20.248
Yeah. I mean, I, and I mean, what, what’s fun, there’s people, people are always remember this, but I’m, We had Zip drives as our backup. And it wasn’t being done very often, but there was actually a Zip drive. That was actually how the backup was being done with a Zip drive, not a Jazz drive, but an actual like 250 meg USB at least Zip drive. As you remember, they used to be parallel, parallel ports when they were 100 meg. So, I mean, compare that to our storage footprint now, which is, you know. Well over 200 terabytes. I just, I can’t buy storage fast enough. I mean, I just bought a new NAS with 150 raw terabytes. You know, the nature of what we do and the nature of how we do what we do lends itself towards large, complex data sets that are stored for a very, very, very long time. If it takes you six… 12 months to get a data set from a customer, you never want to get rid of that. You want to hold onto it forever. And so, you know, that’s like, say someone came to me the other day and said, Hey, I need five terabytes. I’m like, that’s, it’s not like a normal thing to ask for, but they only need five terabytes and that will happen regularly. So, I mean, our storage, you know, what 10 years ago, people were just storing everything on external hard drives, which has its… You know, that’s obviously not an ideal situation, but neither is it to have an uncontrolled storage sprawl. But, you know, that’s just an example of how things have changed in the last 13 years. You know, storage is just huge.
Speaker 0 | 09:10.157
So when you looked at it and you came and you looked at it, it just made me think, you know, maybe when people hire IT directors from now on, they should have them first walk through and kind of get a general overview of what’s going on in the place and the network. I wonder if that’s even possible. Do they do that? Do they do that when we hire IT directors? I wonder.
Speaker 1 | 09:27.465
It didn’t happen with me. I mean, we talked a little bit about the architecture. I mean, I had a phone screen and we talked a little bit about the architecture. And what was interesting is, you know, we were a company of about 150 people across, I think, six locations at that time. And so I was like, I had some ideas right off the bat. Right. I had some interesting ideas right off the bat over that phone screen of how we could. How we could do things together. I had some experience in… larger and smaller companies that had similar architectures and similar physical layouts as in you know multiple locations and how we connected those in some sort of quasi mesh network using say sonic wall ipsec vpns um like that and so we hit it off right then and there it’s like hey i have experience direct experience on how i can help you and then i came in for an interview it was a long interview where we chatted a little bit more about about what what the company does and how, you know, his, you know, what he, I guess he sits in a CTO position now, but how, what his vision for the network was at a very, very, very high level. And, and, and it was empower the users. That was really what we had, what the vision was for year one. And then, and then beyond was empower the users with the tools necessary to, to do their, to do their work. And when it comes to. receiving budgetary authority to do projects. It is all in the framework of empower the users. We will never say no to a tool. I mean, within reason that advances an employee’s ability to complete a task.
Speaker 0 | 11:12.343
So how does that come about? Like an employee says, Hey, I need this. And I saw this and can I have it?
Speaker 1 | 11:18.346
I mean, sometimes, um, and sometimes it’s, it’s generally from us. Sometimes it’s we solve a business problem by we invest in storage, in back-end storage or a new storage architecture, a new SAN or a new something that will empower our users to more reliably receive email or reliably get to SharePoint or reliably get to anything. We upgrade our corporate firewall to a honking fast, super unnecessary… um, like Sonic, well, NSA 6600, which can do 12 gig per second. Why do we need to do that? When our internet pipe may be a hundred mag. Well, um, because the internet’s now a gig and we future proofed it so that, you know, when I had 10 people downloading giant data sets from the internet at the same time, now that the Sonic wall didn’t implode and like shut down the internet for everybody else.
Speaker 0 | 12:13.977
And there’s like, don’t buy what you need by what you need, like 10 years from now.
Speaker 1 | 12:18.580
Kind of. I mean, that’s a trend.
Speaker 0 | 12:21.342
And it’s been a poor trend.
Speaker 1 | 12:24.764
It’s been a poor trend in our industry for the longest time that we overbuy, that you only get budget so much, and then you’re going to overbuy and get everything you need, even though you might not need it. We always, you know, the biggest thing is storage, right? We buy a sand that has 50 terabytes when really we need 10 terabytes, but we overbuy because, you know, because we don’t know what the future, the future is murky. And those sorts of things.
Speaker 0 | 12:48.953
So how do you balance that?
Speaker 1 | 12:50.493
So, I mean,
Speaker 0 | 12:51.474
because what you basically what you told me is that you guys are prepared for significant growth. Now, obviously, you forecasted that growth somehow, but you’re prepared for it.
Speaker 1 | 13:01.777
Sure. So, I mean, the future is always murky, but you can tell based on historical trends and a basic assessment of your workload that where your big problem is going to be. whether it’s going to be storage, whether it’s going to be compute, whether it’s going to be memory, what it’s going to be. And then you, you, you buy an architecture that has scalability. So you don’t have to overbuy.
Speaker 0 | 13:24.426
So let’s talk about your recent, let’s talk about your recent upgrade if we’re allowed to talk about that. Sure.
Speaker 1 | 13:29.269
So we looked, um, our, our, our current stack of VMware servers and, and, and architecture. Well, our servers are about nine years old, um, which is again, just as. as long as I’ve been here, that’s forever. So we needed to do a server refresh on the back end and maybe a storage refresh. So we looked at a bunch of different architectures.
Speaker 0 | 13:53.945
Real quick, why were we…
Speaker 1 | 13:55.165
Why…
Speaker 0 | 13:55.866
Other than nine years.
Speaker 1 | 13:57.067
The biggest reason was not performance, was not… was compliance. We had nine-year-old servers. The servers are no longer supported on a VMware platform. So for us to like… put that next version or that next security patch or next whatever could be dangerous. It could move us into an unchartedness where if anything goes wrong, VMware has the ability to say, well, you’re on an unsupported platform. So sorry, right? You can’t, you know, like we can’t help you. And I couldn’t continue to advance the organization’s missions if that loophole existed. So I couldn’t, I couldn’t take, I couldn’t, I could no longer risk except not being able to put. new versions or patches on the hypervisor in the event of failure of the hypervisor or security risk of patches or losing the hypervisor and having that creep into the VMs. I couldn’t I couldn’t risk accepting that anymore. So when we looked at a bunch of different architectures, we looked at some hyper-converged platforms, some dissociated hyper-converged platforms, and some traditional rack servers, and either some converged-ish solutions. And we settled on Dell’s new Kinetic MX7000 chassis and Blade servers. And why we selected that was because we found it gave us the most amount of flexibility to change and tweak our architecture over the next eight to 10 years. If we looked at our server refresh as an every seven to 10 year problem, that was, we needed to plan for that.
Speaker 0 | 15:44.582
And what are you thinking you’re going to tweak? I mean, thinking ahead, like, man, this could happen or this could happen. And, you know, like what… what’s kind of like those things that pop into your head?
Speaker 1 | 15:53.768
So our workload, we know our workload right now. We know what our workload is going to look like for the foreseeable future. But let’s just say for the sake of argument, we’re looking at maybe having to add 700 users to my directory or double or triple or quadruple our size. our exchange organization. So, you know, what kind of problem is that? Is it a compute problem? Is it a memory problem? Is it a storage problem? Largely, you don’t need that much bigger of an exchange server to support 1,000 users than 300 users. Maybe a little bit more memory. Maybe another couple of these, you know, virtual CPUs or whatnot. And maybe you want to split the load off a couple boxes. But largely, it’s a storage problem, right? You have… 200 users with 4 gig mailboxes, or you have 500 users with 4 gig mailboxes, or you have 1,000 users with 4 gig mailboxes. The problem is storage. The problem is storage. So the thing that we might have to tweak and change in the future, sooner rather than later, is what we’re doing for storage. Do we need more power, more speed on the storage? Do we need to move to all flash? Do we need to do… 25 gig to storage rather than the 10 gig connection we currently have. Those are the sorts of things we’re going to have to tweak. Now, the cool part about the MX7000 is Intel’s kinetic Gen Z architecture, which allows us the ability to add resources to the chassis without having to scale in a traditional, hey, I’m out of compute, I need to add a server.
Speaker 0 | 17:41.978
Now, last time you talked about the traditional way that a lot of people do it, and I can’t remember what we were talking about or what you said. It’s just coming back to memory right now. You’re like, we didn’t go this route because.
Speaker 1 | 17:50.124
Yeah. Can you pause for a second? Hey,
Speaker 0 | 17:51.726
go ahead, man. Yeah, let’s do it.
Speaker 1 | 17:54.508
Hey, babe, what’s up?
Speaker 0 | 17:59.432
We’re here. We’re back. Okay.
Speaker 1 | 18:00.953
Sorry. Okay. Okay. So, with the Gen Z architecture, you can add individual components. So say my workload, it becomes a memory thing. I need more memory. Instead of having to, in a traditional architecture, buy another blade, another server, and get all that compute that I don’t need because I don’t have a compute problem. I can just add memory. I can add a daughter board to the chassis and allocate that memory to the workload. Okay, great. So instead of spending $8,000 to $10,000 to $12,000 on a server, I’m spending $1,000 on memory, allocating it to the workload on the fly. Much, much, much better. Much better architecture, much easier. Another thing that we’ll probably be looking at is some type of VDI initiative with GPUs. Again, daughter board, GPU card, assigned to blade, no buying new GPU blades, right? So again, assign the GPU to the workload, to the VDI. Yes, I’ll have to buy compute. I mean, that’s kind of a given. But it’s very flexible. Individual components. that can be added to a chassis and then allocated to a workload allows you a lot more flexibility. So that if someone comes to you with a huge compute problem, again, you’re going to have to go out and buy 10 new servers. You can buy CPUs, insert them to the chassis, allocate them to the blade, and do it on the fly. So that’s the way we looked at building an architecture that was as malleable. and is flexible to look at the unforeseen possibilities that could come. When we looked at hyper-converged platforms, hyper-converged platforms, while they may clean up your rack, They force you to be on a virtual workload for the rest of your life. And yes, there are ways you can expose the storage, but they’re not designed to do that. So as long as your workloads are going to be 100% virtual for the rest of your life, fine. But if you have a physical need, a bare metal need, you really can’t do that with a hyperconverged platform. And it’s even true of a dissociated hyperconverged, where you can scale. Plus, with hyper-converged platforms, you have to scale things you don’t need. If I run out of disk space, I run out of storage, I have to add compute. So you have to add them in Legos, which in our because of the unknown nature of how our company might evolve in the next how many ever years, we didn’t want to stick ourselves on an architecture that wed us to only one type of workload or that eliminated us from going down a path.
Speaker 0 | 20:54.673
It’s smart strategic thinking, but more importantly, how does this make your life easier?
Speaker 1 | 21:02.518
Well, I mean, it doesn’t, it doesn’t. Because now again, I’m looking at those rain clouds, right? Those rain clouds. Well, at some point. in the next, you know, I’ve got a pallet sitting in an office right now. I’m going to have to actually roll out this technology, load out these blades, roll out a new core 25 gig core, you know, core networking to replace our 10 gig core networking. And replacing core networking is not easy. It’s going to be a pain to do. And it’s going to be some downtime here. And I know that people are going to complain about downtime. And downtime means I’m not doing it at 2 p.m. on a Tuesday. Now, again, with the right design and the right architecture in place, with fault tolerance and whatnot, I can do a lot of it during the day. I can roll out a new VMware server, I can add it to my VMware cluster, I can vMotion that things over, and I can lather, rinse, and repeat, and do a no downtime migration of hardware. Okay, that’s great, because I can do it at 2pm on a Tuesday. Do I have to do it immediately? No. I can do it slowly. I can do it. I can vMotion a few hosts to not affect performance. And I can do it all invisibly to the users. And I can do it during business hours. Perfect. That’s going to help me. But that’s a system design thing. That’s building an architecture and designing it to have enough fault tolerance to be able to do these migrations and still be able to withstand a hardware failure. Like one of the servers implodes or I lose something on the storage side or things like that. You can’t, you know, so that, that, so the design makes my life easier. The hypervisor and the, and, and what we’re rolling out makes my life easier so that I can leave by three so that if I’m in the middle, I don’t have to worry that okay it’s three o’clock it’s time for me to get out of the office, but I’m still view motioning three hosts or three, three, three gas machines. Who cares? It’s going to go on without me and I can log on when I get home and make sure it’s done. Right. Cause I’m not going to affect, I’m not going to have downtime. Doing what I’m doing is not going to create downtime. Now, yes, replacing core switching. That’s going to be a, that’s going to be intriguing. But we’ll, we’ll, we’ll, we’ve made good vendor relationships over the last 10, 12, 13 years where we, we strategically ally ourselves with either vendors or top tier partners. Yep. It can assist us with this. I mean, we’re a shop of roughly two and a third people for 200 users across four, five, six locations. So you can imagine the workload could be huge, but it can be less if we do it right.
Speaker 0 | 24:06.342
So how do you think most… Do you know what the situation is out there for the majority of kind of mid-market IT directors that manage that many end users? I mean, you have two and a half people for…
Speaker 1 | 24:16.557
I see it. I see it in the industry that people are working, you know, 10, 12, 14 hour days and whatever. And I mean, maybe it’s a pure workload, but I have to ask the question, what workloads are causing you to have to work an 18 hour day? What things are you… addressing or not addressing that are causing you to have to work long hours? Do you have system downtime events? Do you have viruses in the network? Do you have intrusion events? Do you have security events? What? What things are making you have to work 12-hour days? Is it simply you have a very high demanding user base? Sure, that can happen. But my guess is that most of the time, you’re responding to fires that you could avoid. The biggest one of these is probably viruses, spyware, malware, those sorts of things that force you to have to whack and reload a machine, respond to a security event, document it, prevent it, test it. and isolate it, and that takes hours, right? And if you’re wasting IT manager, IT director time on responding to antivirus alerts and fixing viruses and remediating spyware, you’re not an IT director at that point. Almost no IT directors should be forced to do tier one work. An IT director should be sitting much higher level to be looking at architecture, system design, policy, and procedures to enable the organization. If you’re doing tier one work, then you need to hire staff, or you need to re-architect something, use different technology that frees up your time. We haven’t had, well, I’ll knock on some wood, but we haven’t had a virus or ransomware or any of that sort of thing. in five or six or seven years now. Why? Because we invested in technology that largely makes malware and those virus types of events obsolete. We have multi-level, multi-tiered protection on the machines. So if antivirus, I have another customer that I work with, they’re spending days reloading computers because their computers are getting infected. because they refuse to turn on gateway antivirus protection because they think it’ll slow down the internet. Okay, well, do you know what else slows down the internet? Everyone having spyware on their computer. Right? And so they’re like, well, let’s block YouTube, and then let’s not turn on any gateway protection. Let’s have an on-premise spam prevention solution that’s updated once a month. These are things that don’t compute to high… availability. And then if you have old servers and your system is designed without fault tolerance and you’re constantly having to deal with fixing and rotating machines and rotating servers and rebooting servers, again, it comes down to system design. If you design the system so that you have no single points of failures or very few single points of failures, you can leave it three because you know the system’s not going to go down the moment you walk out the door. You can’t have a VPN that goes down the moment people use it. Hey, it’s a snow day. Great, we’re all working from home today. No one can log in because the VPN doesn’t work. Oh, but it was working yesterday when no one needed it. Awesome, right? That doesn’t help you. It doesn’t help anybody. It doesn’t help anybody in your organization from an end user standpoint. And it certainly doesn’t help you get any sleep.
Speaker 0 | 28:09.646
You said you work to have a life outside of work. where you work for, what was that? What was that? What was that quote?
Speaker 1 | 28:19.752
You said a wise scholar once told me that you work to live, not live to work. I mean, and maybe there are some people that are so into their career and, and, and the technical side of what they do that they, you know, that they really wake up and can’t wait to get to work. And that could be true. But for the most part, the people that have, you know, 18 to 20 years experience and have families and whatnot, I want to have kids and spend time with them. I want to make enough money to be comfortable so that I can give them what they wish and what they desire. I don’t, I don’t want to not, I don’t want to have kids and not watch them grow up. What’s the point? And so… Like, yes, I try to leave every day at three or as close to three as I can so that I can get home. And, you know, if you live to work, you lead a very interesting existence. To me, I work as hard as I do so that I can have a life.
Speaker 0 | 29:38.286
And you live it.
Speaker 1 | 29:38.766
So that on my weekends, I have my weekends off.
Speaker 0 | 29:40.888
And you live in a very affluent area. I mean, I think Washington, D.C., D.C. metro area, Northern Virginia, Maryland is a… very, very, I mean, it’s crazy busy. First of all, the traffic’s absolutely absurd.
Speaker 1 | 29:53.009
Traffic is brutal.
Speaker 0 | 29:54.129
But I lived there for years. And what I found was that everyone works really, really hard, but it’s a very affluent area. It’s very, I think it’s easy to be successful there. Maybe that’s because all the money in the world is spent there. All the tax dollars are spent there. And you see people, you see like where the tax dollars go. That’s the one thing I noticed right away when I lived there for years. But people leave early. They go home early.
Speaker 1 | 30:17.824
Yeah. I mean, so in my case, I have about an hour, I live about 38 to 39 miles as the crow flies from the office. Um, and so if I don’t leave relatively on time, um, you know, I come into the office about six, uh, I leave about three. If I don’t leave on either or whether my commute becomes, I don’t want to say untenable because it it’s, it’s, you know, We all have to deal with it. Death, taxes, and traffic. I mean, no one lives five minutes from the office. But, you know, it becomes unpleasant for me. But, you know, there are times where you have to. But, you know, I guess that’s part of the game. And then you do have to, I mean, it is an affluent area where… You’d think that if you lived in Dallas and made a hundred grand, you’d have a much better living. Whereas here, that same hundred grand doesn’t go as far. I mean, especially with housing what it is.
Speaker 0 | 31:22.491
Dallas is a good example too, by the way, because you can get a nice spread down there in Texas. And I have thought about that numerous times.
Speaker 1 | 31:29.132
You know, I had a, I had a conversation with a, with a large vendor and I was looking at, at, at, at Dallas as a potential place to relocate. And then, yeah, I mean, I’m not sure if I would have thought about that. It’s like, wait a minute, I can get a five-bedroom house that’s all brick and whatever for $300,000 and still make the same amount of money.
Speaker 0 | 31:52.503
Don’t forget the pool with the waterfall and the overhanging outside.
Speaker 1 | 31:56.644
And then you can use the pool for more than three months a year. My mom has a pool in her house here. She relocated to be closer to the grandkids. So she’s got a pool. It’s like, oh man, this is so cool. My mom’s got a pool. And then she’s like, wow. This sucks. I can only use it for, you know, for May, June, July, and August, or the end of May. And then she kept it open until the end of September, but couldn’t swim because it’s too damn cold. So I’m like, you know, it was, it was, it was, you know, it’s like, it’s, it’s a nice, you know, a nice thing to have. But yeah, I mean, if you lived in Dallas, you’d be able to use it for like eight months a year. That would be a lot more ideal.
Speaker 0 | 32:35.240
I’m laughing because I’m up in the Northeast now. So, and what last time I went down to Baltimore, man, I was like, it is sunny down here. It’s warm early and you’re complaining about the cold and alcohol and get a hundred. Oh,
Speaker 1 | 32:45.429
sure. I mean, I’m, I’m originally from the Northeast. I’m originally from New Hampshire. So like, you know, to come to deal with, you know, first of all, in New Hampshire, we don’t have things called heat pumps. Like,
Speaker 0 | 32:57.639
like everything’s oil.
Speaker 1 | 33:00.062
We, we.
Speaker 0 | 33:01.143
Stopping wood.
Speaker 1 | 33:04.100
October in my household was my dad coming home and there was a cherry picker full of logs that was delivered. And my dad was on a chainsaw for two hours. And then it was splitting, stacking, throwing it into the basement, stacking. Basement wasn’t a rec room. It was where we stored wood. Oh, yeah. So, and, you know, it’s different. I mean, like here, just to turn on the heat, you know, it’s like, it’s just a different experience. But at the same time, heat pump never feels warm. So we still burn a wood stove down here. So that’s my roots.
Speaker 0 | 33:35.952
I’m stacking wood this weekend. See? Yeah. So challenging IT, people are putting out fires. So then, okay, I’m going to take this advice. I’m going to ask myself what workloads are causing me pain and anguish and what things are causing fires. And I’ve got this solution and I’ve got… the new design, but I’ve now got to ask for this money to executive management who has no clue what I’m talking about. If I said hypervisor, they’d be like, what? What are you talking about? So how do you make that argument when they say, look, I don’t know, everything seems to be fine. You’re doing all right. You’re working your butt off and we’re still here.
Speaker 1 | 34:26.589
Yeah. So one of the… one of the neat ways that I’ve been able to, to, to, to equate this back to either non-technical people or whatever is once I was talking to a guy and you know, this is my last company. Um, and the, and the guy’s like, you know, one of our contractors out of the state department, he’s like, I don’t work, I don’t work with users. I work on servers. And I’m like, That’s a terrible attitude because everything comes back to a user at a keyboard. Everything, everything that anyone does in information technology eventually gets its way back to a user on a keyboard. So what you if you have a non-technical executive that you’re trying to explain things to, you have to put it in those terms. You have a keyboard in front of you. I am going to make sure that you continue to be able to use that keyboard to do your job faster and more efficiently and better. It’s not technical. Like, hey, your system’s on. I’m going to make sure that it stays on. And I’m going to make sure it stays on more and better. And that your email goes faster and better. And that you remain free of viruses and free of this. And I’m going to make sure. that our data center footprint, instead of it being a whole rack, is now a half a rack. Therefore, the bill is less, right? And bring it down to money. Yes, we have to spend $50,000 up front today, but I’m going to save $2,000 a month. When we switched from hosted PBX in the cloud to a premise voice over IP, it was, wow, we got to make like a $45,000 CapEx hit. That’s a lot. I said, that’s true. But at $3,000 a month, how many months do we have to pay before we hit that break-even point where the CapEx all of a sudden makes a lot of sense?
Speaker 0 | 36:29.723
You’re just paying for SIP trunking and you’re managing the Cisco. Correct.
Speaker 1 | 36:32.585
And when we scale outward, we don’t scale, hey, we just added 50 users. And now all of a sudden the phone bill is $5,000 a month. No, we scaled out in users. We scaled up in licenses a little bit as a one-time expense, not a… every month expense. And so you explain it in those terms, you build an Excel, hey, here’s what our cost is going to be like, as we scale forward with 100, 200, 300, 400, 500 users. Here’s what our costs scale out on a premise solution. Yes, it’s a CapEx hit. You know, but that CapEx is negligible, considering what our costs would be if we did something different.
Speaker 0 | 37:12.120
I do not like that argument being a man that sells hosted voice over IP. No,
Speaker 1 | 37:16.364
I, however, I feel that. I feel that. And well, and you know, I’ve, we’re in that spot where we’re, we’re look, our, the provider we have is, is going down a different path. Yeah. And it might make sense for us to reexamine that. So I mean,
Speaker 0 | 37:32.860
I’m joking a little bit because I understand your argument. I do those arguments all day. And I say, hey, look, in this particular case, because you have the experience, because you have the ability to manage a Cisco call manager and it’s easy for you, even though it’s not easy for a lot of people, I will be happy to sell you SIP trunk and internet. And you can’t argue with that because yeah, the savings is there. But if it becomes like a ridiculous pain in the butt factor to manage this very complicated call manager with call center agents and stuff like that, it can be different. obviously multiple organizations.
Speaker 1 | 38:04.141
Sure. And then again, but the important thing from an IT director standpoint is that you don’t become complacent and stale in your architectural thinking. It’s constant. I am aware that our call manager, although we are running current versions of everything and it is redundant, but maybe at some point we will need to move back to the cloud or do something different, something. as we have a software defined architecture, maybe we can, we can justify moving that way again because maybe the cost has come down. It’s not $32 per user per month. It’s less than that. And we can leverage it and integrate it with office 365 and, and whatever, or teams.
Speaker 0 | 38:45.566
Do you want to know, here’s a little secret. Most providers won’t tell you this, but they can break up the users and call paths. Sure. In other words,
Speaker 1 | 38:53.612
there’s no point, right?
Speaker 0 | 38:54.773
There’s no point in paying, you know, if you’ve only got, you know, you’ve got a thousand people in the organization, but only 20 people are on the phone at any given time. And you’re a very light phone organization. There’s no point paying $35 for a thousand users. It’s just not.
Speaker 1 | 39:09.006
Exactly. And again, it comes back to your, comes back to your partner relationships where you find a company that’s not, you know, like, like a Vonage, right. Where you have an inflexible organization that can’t really provide great service anyway. And, you know, you, you have this, you know,
Speaker 0 | 39:25.728
I’m going to have to put up a disclaimer at the beginning of this. The views and opinions of people on this show are, no. But yeah, no. Yeah, if you’ve got a large bureaucracy that’s inflexible or you can’t get anywhere and you’re calling 1-800-GO-POUND-SAND and that’s what you’re dealing with, that’s where you need a strategic partner. Just like you said, a partner that can get things done that other people normally can’t get done and can leverage different marketing funds and can leverage different architectures on the back end that could break up. breakup call.
Speaker 1 | 39:56.222
Correct. And that’s what an IT director should do. An IT director should be responsible. Instead of fixing viruses and stuff, your IT director needs to be constantly re-envisioning the architecture and then selecting organizations and best-of-breed technology and partners to work with to help make you look good and get you out of the office by three. I love that you said that. We use RSA. We use all big, big vendors, and we have relationships. We’re using products that the average customer is like three to $5 billion a year, but we’re using it. Why? Because I called RSA to say, Hey, look, we need to do this. I understand that you’re a huge and you don’t care about the small business and the small, the medium business and whatever, but we have a business problem. Do you want to help or not? And I got them to bite, but that’s the job of an it director. And again, if you, you can select, you know, select technology that, that may or may disappear. And you will have to deal with the fallout from that if you select a vendor who’s…
Speaker 0 | 41:02.427
I don’t understand why people don’t understand this approach. And I think it’s almost the point of arrogance and the fact that people have gotten screwed over in the past. I don’t know what it is, but I don’t understand why some IT directors try to do all the research on their own and try to go about it on their own. Because I will give you five people on your payroll completely for free that will do all kinds of work for you. And… bring all kinds of, you know, carrier agnostic, vendor agnostic solutions to them because they just want to help you make the best decision. And because we’ve got 300, 400 vendors, it’s in our best interest to give you the best solution.
Speaker 1 | 41:40.570
And largely the best price. Now, I don’t, I mean, like I have a pretty good working relationship with like, say, CDW. Yes. Now, does CDW add any real value add to my architecture? No, not really. they’re selling me a box, right? But what their box that they’re selling me is, is at a better price point because of their purchasing power.
Speaker 0 | 42:02.597
Absolutely. I’m the CDW of the carrier and telecom world.
Speaker 1 | 42:09.340
And it’s great because you have all these resources to you and you buy a lot. I mean, if you go directly to the vendors, you will not necessarily get the best price. You might, but you’re probably not going to get the best price.
Speaker 0 | 42:20.684
But here’s the thing also.
Speaker 1 | 42:21.584
Sometimes you will because they can kill that margin. You know, that’s… CDW margin, but typically CDW is going to get better pricing or whatever vendor.
Speaker 0 | 42:30.108
But here’s the difference though. You’re doing this once. It’s like tiling your bathroom. You’ve done it once. CDW does it thousands of times a day. Correct. Right. So they already have historical pricing. I’ve already got historical data on a hundred megs Comcast in this zip code. And you think you’re negotiating and going back and forth, but I can just give you the data. You can just take the emotion out of it, take the negotiation out of it and just say, Hey, look, this is what they gave XYZ person. Here’s what their rack rate is. And I get 60% below this rack rate. So why’d you do that? And here’s the last 10 people that we, uh, that bought a hundred megs in your area and here’s the lowest price we got. So let’s use that and go against that. And then on top of it, it’s most people don’t have, you know, a lot of people don’t have legal or they don’t have a legal team in house also that kind of understands. how all these carriers and various different equipment vendors work. So if you have legal experience, you can say, hey, well, look at this company. This is what they got in these terms and conditions. And then look at what this company, look at what they were able to negotiate. Now let’s put them all together and summarize them and ask for that.
Speaker 1 | 43:35.326
Yeah, no, that makes sense. A lot of people are worried about outsourcing different components of their job. And I don’t outsource much in terms of the actual work being done. But. Yeah, I’ll go to the big vendors of the world and say, hey, I’m looking for a smart card solution. I’m looking for a multi-factor authentication solution involving smart cards and maybe OTP tokens to use with my VPN and my local authentication for my users. I don’t have time to do it. Go give me, and I’ll meet with a security professional at like a CDW or those sorts of things or an SHI, and they’ll come back. They’ll come back with, all right, we have these three vendors. Who do you want to talk to? Yeah,
Speaker 0 | 44:19.314
because you could look forever. You could look forever.
Speaker 1 | 44:21.175
You could look forever and never get Google people,
Speaker 0 | 44:23.637
talk with a thousand different colleagues, and they’ve got different environments.
Speaker 1 | 44:26.820
And the thing your biggest worry there is, is selecting a vendor that goes out of business. I mean, you remember, I’ll really date myself here. It’s like, and you’re in telecom. So you remember Comcast used to be another company before. It’s Comcast, like their internet. It used to be something called at home, right? You remember at home? So. At home, I bought some stock and at home, and I was really proud of myself. I’m like, I’m going to day trade. This is going to be cool. So I bought 10 shares of at home at like $42. I was like, sweet, I’m going to be a millionaire. Look at me. I bought 10 shares. And so, and then like it went up to like 44. I’m like, oh yeah, this is where it starts. And then I like went to the bathroom and it came back and it was 27 cents.
Speaker 0 | 45:13.405
Here’s, let me tell you. Okay. So here’s the thing. I’ve been in telecom for almost two decades. And when I was in the corporate world and this is another, this is, I say future proofing also. And that’s why I say people should buy their voice and data for me because I don’t care who you’re with for the. I’ll be with you for like until I’m in the grave and then my kids take over. Right. But the point is, is like, I don’t care if you’re with Comcast, I don’t care if you’re with whoever, it’s just about the best possible solution for your company. And if that needs to change because of whatever, great. But here’s the truth. I have been in telecom for two decades and every single company that I have ever worked for has either gone bankrupt, um, been acquired, right. Purchased other companies while I worked for them. and or done some sort of strategic merger. Every single company that I ever worked for is now a different name. That’s just the truth. That is the nature of the business. And every single person that works at those companies in the bureaucracy, the average lifespan is eight months, a lot or less, a lot or three months, and some are two years. So if you actually get a sales rep or a customer account manager, or whoever, you know, when you call 1-800-GO-POUND-SAND, whatever it is, right? If you actually get one of those people that’s there for longer than the life of your contract, then you’re pretty fortunate.
Speaker 1 | 46:38.077
Yeah, because it doesn’t happen. I mean, I joke around because, I mean, for some reason, I’m in charge of copiers as well, like our copier leases and whatever.
Speaker 0 | 46:46.283
And I’m a hand dryer.
Speaker 1 | 46:48.224
You know what’s really good? Copier salespeople are really, really, really good. They’re really good at selling stuff. And then the copier technicians, the people that come here and fix the copier are also really, really, really, really good. Everything else sucks. Your account manager, like I had to find out on LinkedIn that my copier guy, like my account manager, like moved to Arizona. That’s where I found out. I found out by like Googling their name and following their LinkedIn profile to find out that six months ago they moved to Scottsdale. Well,
Speaker 0 | 47:21.655
all telecom. If you remember back in the day, telecom was selling lines and switching long distance and all that stuff. Most of the telecom guys copied the Xerox copy machine model. The, you know, kind of like the presentation, the tie downs, the let’s go through, like, you know, make you feel sick about your current situation, tie you down to that. You know, I mean like the whole, it’s, it’s a science to that. There’s, there’s a lot to that. They are very good, very good at what they do. But anyways it has been an. absolute pleasure talking with you. And I want to give you, like, if you had one piece of advice for IT directors out there struggling or successful or people that are looking to have a life, you know, whatever it is, I probably shouldn’t, I probably shouldn’t try to pigeonhole you into whatever it is, but any piece of advice you want to give, uh, what would that be? Um, select your technology carefully.
Speaker 1 | 48:12.816
Um, I think that’s a good question. That’s really the biggest advice. Select your technology and your partners wisely. If you select technology that works for you and select partners that work for you and with you, your job will be easier. If you select always go bottom dollar or always go insert whatever and you don’t design things well, then you can’t exist on a house of straw. You know, you need to build that brick house on the foundation of concrete, not the straw house on a foundation of straw. And that’s every single piece of your organization, all the technology in your organization, your policies, procedures, and protocols in your organization, your vendors and your vendor relationships, and all those sorts of things are all part of that house. They’re all walls and shingles and windows and doors and pieces of that house. I mean… one of those things can bring the house down. So if you select all of these things, select technology after careful evaluation and more concern about features and capabilities rather than price, you will have a house that stands that lets you get out of the office by three.
Speaker 0 | 49:41.923
It’s awesome. One, I do have to ask one last question because it’s just been something that I’ve been asking everyone that I’ve had on the show in the last, I don’t know, the last three or four shows. Did you ever have any really good IT mentors? Like if you think back in your life, can you think of someone that really, and it doesn’t necessarily have to be IT, but is there someone that really influenced you or had an impact on your IT career?
Speaker 1 | 50:07.918
Um, not, not, not really. I mean, there were some people that helped me, but again, you’ve been in telecom long enough to know what things were like, say 20, 20 or 25 years ago. Um, if you were a tier one help desk or a tier two desk side guy, the server guys treated you like crap. And, and, and most people were so worried about keeping their job that there was almost no mentoring. There was no one that would, they would white border problem out with you. that would, even though you might have had an aptitude for it, there was no one that would explain what a subnet mask is and what it really means to have a 255-255-255-0 subnet or to split that up. No one really explained that to you. So even though you might have had some basal understanding of it, there was almost no mentoring. I’ve tried to change that trend and really grab every opportunity I get to change that trend so that I work with my lower end staff here to… if they’ve got an aptitude to try to grow that. No, I didn’t really have a lot of, I had to learn by baptism by fire. I mean, my first big, after working on a government contract for a while, my first job was Y2K work. And I was working 20 hour days. I mean, I remember falling asleep at the keyboard and waking up and it was done. I don’t know how, but this was. this was, you know, and you had to learn everything baptism by fire. Um, and so that’s, that’s not a real good way. I mean, that’s, that’s again, how one of the ways that you design an architecture of straw is because you didn’t, you didn’t learn to do it the right way. You learn to do it by just figuring it out and no one wants to be that. So I wish I was, I wish I, I mean, I wish I had, I mean, I had a couple of guys that would, it would get me involved in some of the migration projects and that was cool because I get experience. But they didn’t sit down with me and explain things. You had to kind of figure all that out yourself. And things are much different now. There’s a lot more mentoring programs out there. But it was way too competitive. It was way too competitive. Yeah.
Speaker 0 | 52:18.402
For people out there listening, then to be mentors if you’ve never been mentored. Well, thank you, sir. I hope you have a wonderful weekend, whatever you’re doing. I’ll be stacking wood up here in New England.
Speaker 1 | 52:31.709
Enjoy that. I’m glad I live in the mid-Atlantic.
Speaker 0 | 52:36.234
All right, man. Have a good one.
