Speaker 0 | 00:09.584

All right, welcome everyone back to Dissecting Popular IT Nerds. Today we have Todd Shipway on the show. And while you are the director of IT, you are more importantly the director of robot communications. And that is very, very interesting. and you’re working at Bossa Nova in robotics. And really, I mean, let’s be honest. The robots are what it’s all about. The robots are really more important than the end users. The end users are just a headcount. They’re just a number in this, I don’t know, cog of the machine. And it’s really all about the robots. So I’m just going to let you speak a little bit, first of all, about what you do. And truly… why humans don’t matter.

Speaker 1 | 01:00.310

Nice to have you. Thanks for having me. So what I do here at Bossa Nova Robotics, well, first off, a little bit about Bossa Nova. So we make automation, inventory, automation and management systems. And we use robotics to achieve most of that. And what I do here is, like you said, I have sort of two roles. I’m a director of IT where… I manage all things IT, everything down to office, internet, Wi-Fi, laptops, applications, the whole gamut. And then the other side, the flip side of that coin is the robot communications. My background is in network engineering. So I manage all of the things for robot communication. Basically, everything down to the Wi-Fi cards, the LTE. hardware that’s within the robot. The robots have networks within themselves. They have 10 gig networks on board. There’s a whole big piece of networking puzzle that comes into making the robots work both on the robot and getting that robot connected to the cloud where we stream a ton of data. So its connectivity is basically the breadth that the robot requires to operate.

Speaker 0 | 02:24.368

robotics as far as changing the as far as changing the future goes and the internet of things and what we can make them do why well first of all why is this so important and what are some of the things that you guys are doing so

Speaker 1 | 02:39.932

it’s it’s important because robots could do things over and over and over and over and over and they never complain uh they do it to the same thing you know the exact same way every time. So your efficiency, your accuracy, just everything about a robot can be done very well.

Speaker 0 | 03:00.990

So basically, if we can make a robot do a $10 an hour task, that’s a good thing.

Speaker 1 | 03:07.895

Correct. You know, and it’s, yes, it’s putting some, you know, it’s kind of replacing some jobs, but those jobs are now changing, you know, to adapt to the robots. Because, you know, yes. That robot may be doing the job of that person, but now that robot has to have people to support it, has to have, you know, there’s other sides of that. Now that robot’s creating data, there’s going to be people that use that data. And now that person that that robot is technically replacing is now just being used in a different way, you know, if that makes sense.

Speaker 0 | 03:39.652

Yeah. So maybe just give me a couple of examples. Like where have we used robotics to really… Where are we using robots to replace the human tired function or where humans fail? And give me an example of where a robot is better than a human.

Speaker 1 | 04:02.572

Well, I mean, take our inventory management system with the robots. If you look at stores such as Walmart or their competitors with others. There are a ton of different grocery markets specifically.

Speaker 0 | 04:18.581

People with stuff on the shelf, people with money sitting on the shelf.

Speaker 1 | 04:22.343

Yes. Any store can have hundreds of thousands of products sitting on the shelf that when it comes time to know when these things are low, getting low, out of stock, sometimes they don’t realize it until without a robot. They don’t realize it until someone complains or they… have someone that has to go through and scan every one of these items and count how many is on the ship and when you have you know a store that has 430 some thousand products that gets very tedious It’s tired of, you know, maybe it’s just extremely repetitive. And in between there, the human is being asked, hey, where’s the ketchup? Where’s the mayonnaise? Can you help me get you? Do you have these in stock? There’s a lot of interruptions. It’s incredibly inefficient. Whereas the robot comes along, robot scans in, you know, less than 30 seconds. And it, we have our whole AI back in that. Basically looks at all these pictures that the robot is taking and does all of the work itself. It counts what’s in stock, what’s in the road, all of these things in a matter of seconds. And it can do it over and over. No one’s asking the robot, where’s the robot? Over and over again throughout the day. So on the flip side of that is. Yes, we’re taking that human out, but that human is now focused on answering those questions for the people. You know, like instead of scanning, now that human has a little tablet in her hand or, you know, a device. Hey, where’s the ketchup at? And they can look up the ketchup. They can help you. They can help put the things on that are now out of stock, things, you know, more things on the shelf more often. That’s just one job of a robot in that environment.

Speaker 0 | 06:27.450

Okay. All right. Bear with me one second here. I’m switching. I’m switching Wi-Fi connections. Hold on. Irony. Irony. Irony. Are you still there?

Speaker 1 | 06:40.295

Yep.

Speaker 0 | 06:41.676

All right. Beautiful. You know, the irony of the Wi-Fi in my house. And we’re just going to leave this in the show since the next topic is going to be Wi-Fi connectivity. I’m just going to, I am going to leave my Wi-Fi connectivity mesh in my house. And the reason why is I’ve got eight kids homeschooling and I’ve got this wireless network. And every now and then my computer jumps on the kids network and I experience latency issues. And here’s the, in the middle of this show. So, okay. So the. Here’s what’s cool about your job is now you are tasked with this ridiculous number. I don’t know. Do we even call them? Can we call the robots end users, endpoints? What do we call the robots? What do you call them?

Speaker 1 | 07:28.908

Robots? For me, it’s a robot. It’s an endpoint, but it’s an endpoint with other endpoints behind it. So it’s basically just a robot.

Speaker 0 | 07:39.015

I mean, is this like tracking these via Mac addresses or how do you track all these?

Speaker 1 | 07:46.030

Uh, it’s,

Speaker 0 | 07:46.670

well, are we giving them human names? Like if there’s an, if there’s one that’s like really buggy, do we give him like his name’s Satan or something?

Speaker 1 | 07:52.472

Oh, well we do unofficially.

Speaker 0 | 08:00.575

Oh man. It’s good. Okay. So, um, how do we track all these?

Speaker 1 | 08:05.576

Oh, well basically, you know, we assign a name to a robot, you know, we have specific. Depending on the version of robots, we have different, it’s not necessarily code names, but we just have different names for different versions. That way, and then followed by some level of serial number. So we have a name that symbolizes the version and the platform, and then the serial number behind it, which can be anywhere from two to four characters or numbers. And that’s really how we… Yep. define the robots you know because that name will have hundreds of components behind it it’s got a whole network behind it it’s it’s sort of like a mini office if you will and every every uh any of these guys you have out there that you’re managing and kind of in charge of uh well uh a little while ago uh earlier this year we were at around uh 800 uh and then that’s shrunken a little bit because uh covid has you know changed the landscape a little bit uh but you know we’re hoping to get that back up.

Speaker 0 | 09:14.184

Just insane. So you’re managing 100 robots out there. Might as well be humans, but maybe they work a little. Okay, so what about the robots is not human? Well, obviously all of it. But what about the robots is worse than a human? Does that make sense? Like what’s the real problematic thing that just doesn’t, like when they’re not working, they’re not working, right? So um it’s not like you can call them up on the phone and be like hey um could you show up to work like what’s um you can’t call them or maybe you can call them but what’s going on you know what i mean what’s the worst part about it what’s the i mean it really comes down to you know they cost a ton so there’s only one so when it’s not working uh whatever

Speaker 1 | 09:54.951

store or location that that robot is in is now you know Once they get used to having that inventory data, and now they have to go without it while we fix a problem, someone needs to be dispatched, whatever the case may be, that that robot is now not working, now it’s equal to a certain dollar amount. Because now they’re not, you know, they’ve got to do a little extra that they haven’t had to do for a while. So that’s the biggest thing is a robot not doing its mission is the biggest thing. And that can happen for… a ton of different reasons. You know, a robot could run over a piece of plastic and that plastic gets wrapped up in the, you know, one of the motors and, you know, the robot can’t move or something’s happened with a battery or it didn’t charge, it got knocked off charge because someone hit it with a buggy, you know, and overnight we didn’t realize it, you know, things like that. There’s a huge number of factors that keep it from, that can keep a robot from not running. You know, our job is to find out when they happen. and you minimize that as much as possible.

Speaker 0 | 11:01.690

You just made me think of like the, the, the well pump going off and I’m just not having running water anymore as that’s like, that’s like we, the robots aren’t working. We don’t have running water right now. We actually have to go like, I don’t know, we have to go back to manually counting or something. And it’s kind of, you know, yes, no one wants to do that. Um, for You’re in the business of robots. What about companies that need robots that don’t know they need robots, if that makes any sense? Is there a way to, from the standpoint of selling executive management, that technology helps you do things better? That’s basically what the robots are doing. We’re accounting, we’re doing inventory faster, we’re doing inventory more accurately, we’re able to manage shrink, and I don’t know. gross margin and all of these things would be very important from a business perspective. So, but just using the robot metaphor, I guess, even internally in your own company, what’s it like selling technology to the business forces or to executive management that, you know, we need to invest in this and what are the arguments or ways that you like to make those arguments?

Speaker 1 | 12:19.245

Well, it’s not easy. You have kind of two schools of thought in business today. You have the new school and the old school. When you get into the people that are familiar with technology and they understand the good that technology can bring, it’s a lot easier discussion. However, there are still some of those folks out there that think that humans are… can do everything a robot can do a whole lot better because they’re human. The other flip side of that is, robots are a very hard sell in any aspect, outside of warehousing and car manufacturing, mainly because just putting a robot in isn’t enough. That robot is now going to create a ton of data. And if you don’t have the technology team on your side, your customer side, and the know-how and the experience to be able to capture that data and action upon that data and do stuff with that data, well, the robot’s not going to do you any good. So it’s not like it can be used for everybody. There are some requirements that come with it. And because of that, and this is just about any robotic solution, is at the end of the day, a robot is doing a job. You have to be able to… support that robot in any aspect before that robot becomes effective. And for that, robots can be a very hard sell because they come with a pretty, A, the robots have a huge price tag in the first place, but there’s some ROI there when it’s done correctly and efficiently and things like that. But outside of just that price tag for the robot, there’s also a price tag to be able to… support what that robot is doing for you. That makes sense. So it’s a hard sell, especially in retail environments, anything like that. It becomes really hard because now you’ve got to deal with, you’ve got to be able to sell that, hey, the robot can do this and do this really well. You’ve got to be able to show metrics and record this stuff and show them all of this data. But there’s also the safety aspect. Stores don’t want robots running over people. Stores don’t want robots. you know, bumping into things. Stores are afraid of what a human is going to think of, you know, walking alongside a robot.

Speaker 0 | 14:50.919

So this is happening. How big are these things? And this is happening during open store hours?

Speaker 1 | 14:58.049

Oh, yeah. It’s, you know. People are walking around it, and we have some metrics. I don’t know the specifics, but there’s a certain amount of time that it takes for a robot to become invisible, meaning that when a robot first goes in, everyone’s like, oh, wow, and they’re taking pictures and doing things with the robot. And then after 30, 45 days, whatever that number is, the robot becomes more invisible where everyone’s kind of used to it. The fanfare has worn off, and now the robot’s kind of doing its thing, and everyone just works around it or just… You know, doesn’t see it. And there’s different parts of the country. That timeframe is different. You know, West coast is a little shorter. East coast is a little longer. So there’s, you know, there’s some give and take there. But it’s one of the hardest parts.

Speaker 0 | 15:43.933

Are you saying West coast people are more adept at technology or more used to weird technology than East coast people?

Speaker 1 | 15:50.477

Yes. That’s what it’s, that’s kind of what it seems like is, you know, we can see this because, you know, we’ve tracked this, you know, because the one thing that. Robots for decades have been running fine in manufacturing environments. They’re taking off in warehouse environments where there’s no humans around. It’s just there may be humans, but they’re workers and they’re used to this technology. And then you go into a store like a Walmart or a Walgreens or a dollar store or whatever the case may be, and people aren’t used to seeing this robot room around. And now, for example, Walmart, you’ve got inventory robots. But you’ve got. floor cleaning robots. You’ve got pallet robots. You’ve got all these robots around and it’s starting to become average and people are starting to become used to it. It’s still early on in the robot industry.

Speaker 0 | 16:46.018

I watched The Social Dilemma with my kids last night because I want to scare them into I don’t know if you’ve seen that yet. I have. I wanted to scare them into you know you know, whatever you social media and AI and all of this stuff, but there’s probably a lot of, um, I don’t know, is there like an AI learning curve here or anything like that with the robots or the, you know, is there any kind of things that you, any weird things you’ve noticed that you guys have discovered from, from working with them that you didn’t think from the beginning?

Speaker 1 | 17:12.893

Oh yes, absolutely. There’s, there’s a ton of stuff that, you know, over the past three to five years we’ve had to deal with, you know, something as simple as For example, in a Walmart, a Walmart’s got skylights. Well, if you’re familiar with a Walmart, specifically, they’ve got like their apparel is a different floor than their aisles, you know, a different flooring. And then there’s, you know, transition strips to do that. Well, when you’re talking with AI and computer vision and autonomy and how this robot kind of sees and avoids obstacles, well, during certain parts of the day and certain and sunnier parts of the country, you know. the sun would shine that skylight, create a little bit of reflection, and our robot would see an obstacle. And that obstacle is depending on the different types of flooring that were in there. Walmart in one part of the country may have a different color flooring than one in another. So there was a lot of little things like that you normally don’t think about. But we had to address it. We had to solve it. I’m trying to think about a couple others. Transition strips were a big issue because they weren’t standard, as well as just being able to overcome some of the technology shortcomings, like with LiDAR, not being able to see. If someone went in a store in all black, like one of those black suits where you’re just covered head to toe in black, LiDAR is not going to see you. There’s things like that that we just had to overcome and resolve.

Speaker 0 | 18:49.507

So the robot’s profiling people. This guy looks sketchy. He’s all in black. We do not see him. Last time we were together, I can’t remember why we were saying humans shouldn’t be kept in the loop. Or we had mentioned something about that. And what was that?

Speaker 1 | 19:10.712

We call it HITL, H-I-T-L, human in the loop. And that’s basically the overall. pipeline of a robot. So a lot of your, you know, robotic solutions, you know, air quotes around that, um, are saying that we can, our robot can go out and do this and we can give you this data. What they’re not telling you is the robot is, you know, for example, there’s, there’s a robot out there, not ours, a, at one point it was a competitor of ours, um, that scans the space. It looks similar to ours, and it kind of roams around, and it’s looking for spills. And when it finds a spill, it calls across the PA, hey, there’s a spill. Come clean it up. What’s not being told about that is what we call human interlupes. And that means that from the time that robot sees something, the time a data or an action is created, a human is in that loop somewhere. Meaning that for that spill robot, a robot sees a spill, alerts a human with a picture, hey, I think this is a spill. Is it? Yes or no? And the human is saying, yes, that’s a spill. Alert. Or no, that’s not a spill. Ignore. That’s what we call human in a loop. And in the world of robotics, the biggest thing is you want to get as much humans out of the pipeline of the robotics from all the actions along the way. Meaning that a robot should be able to go out and do its job, finish its job, and a human never has to do anything for it to complete the job.

Speaker 0 | 20:51.828

We need to eliminate the humans. So for that robot to have even been successful, it should have just cleaned up the spill itself.

Speaker 1 | 20:57.731

Well, either that or… have the technology and the learning to understand what a spill is. Now, that’s a very hard thing to determine, so I kind of understand that. But every robotics company out there starts with human in the loop. And then as they progress and they kind of work on their solution, they’re constantly working to lessen the amount of humans that are in that data loop. So it’s all about automation. refining your technology to allow the robot to do everything it needs to do completely on its own. And that’s where you get the efficiency. Because once a human gets involved, your efficiency of the robot is dropped. Because humans are not efficient.

Speaker 0 | 21:40.589

Crazy. Mind-blowing. And yet we invented the robots. How can we eliminate all of us? How can we eliminate all of us and just leave it to the robots?

Speaker 1 | 21:52.835

That’s a world I don’t want to live in.

Speaker 0 | 21:56.276

How can we just, you know, so we want to eliminate all of the work, the mundane tasks and everything, and just let the robots work for us, I guess, is the better way of putting it.

Speaker 1 | 22:07.121

That would be ideal, mainly because you don’t want a robot making a whole lot of decisions that you don’t know about. It can make some really crazy decisions based off of some of the data it creates without some guardrails. Once you take those decision guardrails away, A robot could do some really crazy things.

Speaker 0 | 22:25.749

You got any examples? Even that you’ve just heard of in general?

Speaker 1 | 22:29.850

Think about publicly.

Speaker 0 | 22:30.870

Oh, dang. Okay, well, we’ll leave that up to imagination. I’m sure we’ll leave that one up to imagination. As far as networking and wireless networking goes, I’m sure it’s a wireless networking challenge that’s much worse than… I don’t know what many people have dealt with, whether it be Wi-Fi access points in hospitals, hotels, whatever it is. I’m sure yours is a little bit more complicated.

Speaker 1 | 23:01.687

Yeah, so that’s kind of where I came in is that when these robots are created, the roboticists and the hardware engineers have no concept of, hey, I can just connect this to Wi-Fi. The magic happens. But there comes a point where you have to understand what that black magic is. of connectivity is. And that’s kind of where I come in. And, you know, the robot, our robot specifically, you know, is streaming a ton of really high definition images, you know, to do its job to our, you know, our cloud instance where all the AI processing and algorithms take place. But even then, that robot is also sending telemetry data. Now, the safety and navigation of that robot is all on board. So that means, you know, If connectivity goes down, that robot’s still not going to run over somebody’s kid. But as far as any of the telemetry data, when something goes wrong, if someone needs to access that robot, robots get stuck all the time. It may get in a position that it just can’t calculate a way out, whether that’s humans in the way, it gets in a position. There’s a lot of times where we’ve had teenagers on Friday nights take basketballs and break a cage for a robot. and a robot can’t get out of it. So a human has to connect to that robot to get it past whatever obstacle it has. So connectivity for the robot is really important. And when you’re looking at our use case specifically in retail environments, just recently in the past couple of years, these retailers have started putting Wi-Fi in their stores, but when they put it in stores, it’s built specifically for phones. They want somebody to load their app, walk into the store, be able to have really good Wi-Fi access on their phone to look up where this shirt is or where this bottle of mayonnaise is or whatever the case is, which is very light, a few hundred K and you’re done. The session’s over, the person moves on. And then our robot comes in and we’re going to come in and we’re going to stream, you know, 30, 40 gig of data across the, you know, across the wireless network while we’re roaming the entire store.

Speaker 0 | 25:16.549

Hopping from access point to access point.

Speaker 1 | 25:18.590

Yes, we’re roaming everywhere. So one of our big problems was the roaming aspect. You know, we would come in and we kind of blew apart, you know, the way they designed their wireless network. We required them to have more APs to be able to. to have a higher quality. We basically needed a voice quality Wi-Fi network where retailers just aren’t doing that. They’re starting to now because they’re starting to get every tool they have, whether it’s a robot, whether it’s a cleaner, or even the stuff they’re selling, like a Walmart. They’ve got TVs. They’ve got all of this stuff that is on Wi-Fi now. So the Wi-Fi in a store is… becoming critical to just business operations now. So they’re starting to change it. We were a little earlier in that, so there was a ton of work that we had to do. I worked very heavily with Walmart’s network team specifically on, how can we improve things both on the robot? There was a lot of work we did on the robot side. We looked at hardware and we wrote different drivers to roam differently and all this stuff that we did to better utilize the Wi-Fi technology we had today. I mean, this is before Wi-Fi 6 and LTE and all this other stuff kind of really came in. And then we also had to deal with the backup connectivity, which is where LTE comes in. All of the robots have an LTE connection that when Wi-Fi goes away, it can at least get on LTE and use it. But what we ran into there is, of course, that gets really expensive really quick, especially if we’re going to do 30 gigabytes of data across an LTE connection in a matter of… minute.

Speaker 0 | 27:00.656

Yeah, that would be,

Speaker 1 | 27:01.497

it gets really expensive. It’s quick. We go from, you know, a bill of 20,000 a month to, you know, 160,000 a month in a matter of 24 hours. So they come with a lot of interesting challenges, both Wi-Fi and local networking.

Speaker 0 | 27:18.666

You have a very, very cool, you know, even prior to, you know, replacing humans with robots. You have a pretty awesome background, having worked at even Facebook and the data center piece. Are you allowed to talk about that? Yeah. Because again, I’m assuming from since 2013 to now, I’m assuming their data center and rack space and energy consumption has grown kind of maybe a tiny bit.

Speaker 1 | 27:54.609

Oh, yes. They’ve built many more data center campuses. since I’ve left there.

Speaker 0 | 28:00.292

What was that like as far as working in that data center space? I mean, how big are we talking? How big would you say it is right now? Like data consumption, energy. I mean, it’s got to be ridiculous.

Speaker 1 | 28:14.260

Oh, it’s huge. I mean, they would have, you know, like, for example, one data center campus is made up of four buildings. Those four buildings are, I forget how big, somewhere around the four or 500,000 square foot range per building. And then each building would have, you know, 130, 150,000 servers. The biggest piece of that was the network to connect it. You know, we would have this massive fiber network to basically create, you know, this crazy network between all the buildings because everything was split into different, you know, different clusters. And, you know, you, when you go to Facebook, you’re. Images may load from one data center, your videos may come from a different data center, your ads may come from a different data center, your user profile may come from a different one. So there’s this massive network that in a data center world is larger than any regular data center would ever need, mainly because their need is custom. And it was all, every server is 10 gigs, so we had this massive 10 gig.

Speaker 0 | 29:18.616

Are they just way ahead? Are they just, do they just have to be building all the time?

Speaker 1 | 29:24.137

Like just build compute power? Yes, they have to be. And the-I mean,

Speaker 0 | 29:27.198

is it just like right now they’re like, okay, purchase this land, let’s build this, like bring in generators. I mean, are they building their own data center stuff or are they using, I mean-No,

Speaker 1 | 29:36.760

it’s all built themselves. You know what I mean? The generators, you know, they’re off the, let me say off the shelf, but one building would have 18 generators, you know, 18 two megawatt generators, I think it were, two or five. There’s massive- plant of generators per building um so it was you know they’re just everything they have they have to build it themselves mainly because when you get off the shelf even When I was there, we were building out this massive pop network around the globe. We would go into Colos and we wanted to get to the edge, meaning that when a user would go to Facebook, that ACTP connection was terminated as close to that user as possible.

Speaker 0 | 30:18.102

What if you’re on Facebook in, I don’t know, the Congo? Or what if you’re on Facebook in, you know, like where’s the nearest network access point or pop or whatever?

Speaker 1 | 30:28.109

You know what I mean? Yeah, and that’s the big network that… we were building out was we would go into these we’d find data centers and all we’d have a map and we figured out where all the best places would be and we’d go in and put a you know get a cage really relatively larger cage and all these data centers and build this pop network and then that would have a hefty backbone to our data center network where the real data happens but your edge termination happens as close to you as possible to speed this up that way the back end connection Your front end would be local to you or as close to local as we could get it. The back end connection then would be this massive bandwidth behemoth that connects that pop to all of the data centers. That was the big thing. When I was working there, we were turning up circuits, 10 gig circuits by the hundreds. And we got to a point where providers couldn’t do it fast. give us the bandwidth we needed fast enough. So we had to start building our own fiber network, you know, doing all of this crazy stuff because we were just, no one could, no one could give us what we needed as fast as we could get it. You know, we had this massive, uh, cluster of F5, uh,

Speaker 0 | 31:44.831

I’m just blown away.

Speaker 1 | 31:45.592

I get the model, but on the way, the largest one. And I’ve got videos that show this F5, you know, all of this stack of F5s showing us the DOS alert, you know, we’re being DOS attacked. It wasn’t a DOS stack. It was just we were pushing so much data, it just couldn’t handle it. It thought it was being attacked. So it’s things like that.

Speaker 0 | 32:03.294

You’re attacking yourself.

Speaker 1 | 32:05.435

Yeah. So it’s things like that that we just couldn’t use off-the-shelf stuff. Even Cisco and Juniper, we use Juniper as backbone routers. So even those, we were pushing to the limits. We would put it in, and within about four hours, it’s maxed out. So we had to put in… Absolutely. We would never put just one. We put it at a time.

Speaker 0 | 32:27.649

It just had to have been a crazy job. I don’t even know.

Speaker 1 | 32:30.171

It was.

Speaker 0 | 32:31.012

I wonder if it’s slowed down. You think it’s slowed down now because, because Facebook has kind of scaled to where it’s at, you know?

Speaker 1 | 32:37.096

I think so. And they’ve got everything. It’s so automated at the time I was there, it was still growing like crazy. Uh, it was still, still a hot thing. It’s still a hot thing now, but they’ve got most of that on lock and key now to where it’s just a matter of here’s the stamp, go do this. And they’ve got it. So, you know, Yeah. So efficient nowadays that it, you know, it can’t be as crazy as it once was.

Speaker 0 | 32:59.842

I would imagine there’s still like third world country type build out stuff going on. But, um, the, I don’t, and I was trying to think last night cause I was talking, my, my kids are asking me like, as we were watching that movie, the social dilemma and they’re asking me like, you know, well, what if like, cause they, the data center part of the movie came up and was like, well, what happens if you know, like someone cuts one of those courts? I was like, they’re asking me like, how do they know where like every cat five or cat six or whatever, you know, how do they know where every cable goes? I was like, they know where every cable goes.

Speaker 1 | 33:34.944

Do you guys know where it’s the whole system that I mean,

Speaker 0 | 33:39.148

if one cat six cable came unplugged, would someone would something alert and someone would go plug it back in?

Speaker 1 | 33:45.353

Oh, yeah, we would know that even though we had millions upon millions of servers and network devices. Yeah,

Speaker 0 | 33:51.838

yeah.

Speaker 1 | 33:53.948

We had an entire full-time staff of, I think, like 20 to 25 people at each data center that all they do is cabling. Because by the time you build out one data center campus, a year later, because it was all in clusters, you’d have one building is split into four halls, data halls. And then within data hall, you may have one or two clusters of whatever it may be, whether it’s ads, whether it’s videos and messaging, whatever the cluster is. Right. But by the time you get everything built out, now you’re going back through and rebuilding one whole cluster. So now you’re ripping all these servers out and putting all new ones in and you got to recabling all of it. Like it’s, it was this massive thing.

Speaker 0 | 34:35.048

Has Facebook had an outage? I’m trying to remember, like, have they had an outage?

Speaker 1 | 34:39.850

They have, but back, it’s never like a data center outage. You know, the data center, the infrastructure for Facebook is built such that they can actually lose entire data center campuses. And you, the user, would never feel it.

Speaker 0 | 34:54.714

Like really good geo-redundancy.

Speaker 1 | 34:57.437

Oh, yeah. It’s massive. And the data is replicated. everywhere.

Speaker 0 | 35:01.624

So like if the East coast got nuked, like if the East coast got nuked, like they’d be fine.

Speaker 1 | 35:08.148

Yeah. It’s going to run off of all the other data centers. They have plenty of data centers now. You know, at one point it was only one, you know, and then we got to two and then we got to three. So it’s, you know, it slowly got there. It’s so amazing for me to just think about one of the big exercises that I, you know, that we worked on while I was there is the, uh, the ability to, you know, We had a code name, I forgot what the code name was. But we would actually null route an entire data center case and see what happened. And to make sure all of the DR and everything would work, we would actually do full-size, real-world outages.

Speaker 0 | 35:46.959

Look, Mark said, Mark said, we got to flip the switch today.

Speaker 1 | 35:53.264

It’s a very stressful time with that null route, that’s for sure.

Speaker 0 | 35:58.508

Mark just called. It’s two o’clock in the morning. He wants us to flip the switch and make sure everything still works. It blows my mind. It blows my mind that a simple social media idea, what’s behind it? What’s the growth behind it? And people don’t really know unless you’re kind of like in our world or even partially understand our world to really understand the compute power and the energy it takes up. And they. The wires going everywhere. It’s bigger than a movie. It’s bigger than The Matrix. It’s bigger than make-believe, in my opinion.

Speaker 1 | 36:41.661

Oh, absolutely. It’s crazy. You could make a full-length movie on just the infrastructure and run Facebook, and it’d be a hot movie.

Speaker 0 | 36:53.589

It’d be really cool. It’d be really cool. Just traveling. follow the cabling jockeys around. I can see. Really, just like, now that we’re talking about this, maybe someone will find this little podcast of mine and then someone will come up with some reality show of Facebook cabling jockeys. It’ll be a three-season show. How did you get to this point? Do you remember when you were… when you were a little one, like how did this all start for you?

Speaker 1 | 37:29.693

Well, it all started back in, you know, you know, back in high school, I think ninth grade or so, just, I don’t know what year that would be 94 or five-ish. I started, I was the, like the school and I came from a really small town, you know, I think my graduating class was like 25 people. It was a very small school. We didn’t have hardly any technology. Then at the time, I was the local computer guy. I started with a nice 286 and went to 386 and 486 at home. I was the computer guy. I started out just kind of helping out at school, and they had no network there. At the time, I kind of helped the county figure out how to put the network in, and then we got this massive, what we call the distance learning lab, where we had satellites. You’re doing this while in school?

Speaker 0 | 38:22.655

You’re doing this while in high school?

Speaker 1 | 38:24.660

Yeah, this is when I was in like ninth and tenth grade. So I built or I helped them set up this massive distance learning. So now us in this little tiny school out in the middle of the mountains of Maryland can now take classes from a college professor in California. Or in this is, you know, back in, like I said, 95, 96, when all this kind of the remote learning thing is just.

Speaker 0 | 38:50.490

No one even heard it. Everyone was laughing.

Speaker 1 | 38:53.296

What? And it wasn’t used that much.

Speaker 0 | 38:54.717

You can’t learn that way.

Speaker 1 | 38:57.239

Wow. This was in 98? Yeah, this would be 95, 96-ish because I graduated in 97.

Speaker 0 | 39:05.306

How did we even do that? Let’s see. How do they even what kind of internet are we talking about?

Speaker 1 | 39:10.730

It wasn’t internet at the time. This was like direct satellite connections. We had this big satellite on the roof of the school that went up and we would connect. It was a very when I look back, it’s like, wow, this is The school system probably paid a small fortune to make that happen. And it kind of went from there. I just kind of became the school computer guy. And then when I graduated school, went to college, kind of went from there, started building my own BBS, and then I had my own ISP with a wall of 56k modems. And then that turned into a couple of T1s, and kind of went from there where I had my own ISP. And then that sold to… a local larger ISP and it kind of just escalated from there. You know, I was in the ISP world for quite a while. And then basically just built from there to where I’m at today.

Speaker 0 | 40:07.180

Do you have any pictures of that wall of modems?

Speaker 1 | 40:10.683

I do somewhere, but I, cause at the time, these are paper pictures. So I, I’ve got to try to dig them up.

Speaker 0 | 40:16.969

That would be so awesome. It’d be really cool. It’d be really cool to see some of that stuff, a wall of modems and then a T1. You got a T1 man, like you were, you were firing.

Speaker 1 | 40:29.300

Oh yeah. T1, then you get to a DS3. And, you know, at that point I was, I was starting to get out of it cause I was focusing on college and even in college I was kind of, you know, you know, we had a visual basic class. I remember very vividly this visual basic class in college. Yeah. Me, I was, they put us in this, I went to a tech college. So we were in the apartment with my two roommates who were also kind of, you know, on, on the level of me where we were, you know, we already kind of knew the stuff. So. We sat down and finished the entire.

Speaker 0 | 40:59.385

With a bunch of words. Okay. You got, you lived with a bunch. Yes.

Speaker 1 | 41:03.407

Yeah. And we finished, you know, the entire visual basic class, the first night we had school, you know, and we started building things at the school that the school couldn’t have. And, you know, because a lot of that, that we just, you know, and all of that just started building from, from there where, you know, we left there and went to.

Speaker 0 | 41:19.899

That’s gotta be like a feeling of power. Did that go to your head ever? Like this feeling of power. of ability to do something that no one else really do you think that that’s a problem in the tech world like i’ve got power that no one else really they’re clueless it can be uh you know i see it especially not so much in in that aspect like you know i can do this or that um you know i know like you were able to build things and do things that were like very very i mean powerful is what i mean you know and like able to do things that you and communicate in ways that other people are just, you know, really clueless of.

Speaker 1 | 41:59.398

Oh yeah. It’s, it’s, I don’t think it’s much of a problem nowadays, but back then it definitely was. You had, you know, you had some power trips. I tried to keep those kind of low on my end, but I, I’ve seen many of them. I’ve seen people come and go because of power trips.

Speaker 0 | 42:16.207

Yeah. Yeah. Yeah. Outstanding. Been been a lot of fun having you on the show. I could talk about. Facebook data center and stuff forever and walls of old modems. That’s just outstanding. For anyone out there in the industry, people, you know, other it directors, mid market people, anything like that. Any, any, like any piece of advice moving forward into the future or anything, anything that you can offer up?

Speaker 1 | 42:42.860

I mean, my biggest thing, thing that I tell people when I talk to them is, you know, having worked at Facebook, having worked at small companies, large companies, and insanely large companies from an infrastructure and IT and just technology perspective, I always tell people, understand scale and understand how to work at scale. Having worked at Facebook, when I approach a problem today, I approach it completely differently having the experience I had at Facebook from scale. Not that I would solve it the way Facebook would, but it’s just a matter of thinking about things differently. Don’t pay attention to what the book says you can do. The book says what they think you can do. Understand the technology. I’m the president of a local nonprofit called React, which is robotics and engineering in Allegheny County together. And basically what that organization does is we raise money and we provide robotics classes and… uh, they were technology STEM oriented classes. And after, uh, after school activities to the County students. But one of the big things I tell people there is, you know, if you want to be in technology, understand the underlying technology, don’t stop at what the manual says, mainly because once you understand how a technology works, you can now do whatever you’re imagining comes to, uh, with that technology. above and beyond what they say it can do.

Speaker 0 | 44:20.941

Modifying, changing, getting creative, moving beyond. Yes.

Speaker 1 | 44:24.863

Yeah, you can solve problems with things you never thought you could solve problems with. And that is where the value comes in, you know, either in your small little IT organization, you could probably solve problems a whole lot easier, cheaper, quicker, you know, with different things you never thought possible. You know, at scale, you do things, you’re able to understand the technology, the low lying stuff, you can now. build a solution that scales, you know, to hyperscale if you need to do it. It’s all about understanding what you’re working with. And, you know, that’s, that’s what’s got me where I’m at today, you know, from a career perspective. And now that I’m kind of, I’m still, you know, an engineer at heart. I’m still, you know, in command line doing code and things like that. But, you know, it’s starting to, it’s starting to peel away and I’m starting to have to get into more management and architects and budgeting and things like that. But, you know, for all those people coming up, I definitely say, you know, take the time and understand the technology that you’re, you’re working with and find what you love.

Speaker 0 | 45:24.307

Danny, man, thank you so much for being on the show. Really appreciate it. I think people are really going to like this.

Speaker 1 | 45:30.211

Thanks for having me. Thank