Episode Cover Image

249- Automation and the Future: A Look at IT/OT and Control Systems with Brian Romano

Dissecting Popular IT Nerds
Dissecting Popular IT Nerds
249- Automation and the Future: A Look at IT/OT and Control Systems with Brian Romano
Loading
/

Brian Romano

Dr. Brian Romano has over 25 years of experience in industrial automation. As Director of Technology Development at Arthur G. Russell, he oversees control systems engineering projects across the manufacturing sector. Brian holds five college degrees, including a BS in computer engineering, an MS in applied computer science, an MBA, and a PhD in technology and innovation. He teaches automation systems at the University of Hartford.

Automation and the Future: A Look at ITOT and Control Systems with Brian Romano

How can manufacturing facilities leverage automation, data, and analytics to work smarter? Michael and Brian provide valuable perspective on this question in their illuminating discussion about integrating information technology (IT) and operational technology (OT).

They explore key technologies enabling this convergence, from programmable logic controllers to AI-driven predictive maintenance. However, concerns around cybersecurity vulnerabilities and the need for ongoing human oversight emerge.

This forward-looking conversation highlights the promise and challenges of deeper IT and OT integration. Michael and Brian ultimately paint a vivid picture of how crossing cyber-physical boundaries can pave the path to more agile, insight-driven production.

Disclaimer: The views, thoughts, and opinions expressed by guests on this podcast are solely their own and do not necessarily reflect the views or positions of their employers, affiliates, organizations, or any other entities. The content provided is for informational purposes only and should not be considered professional advice. The podcast hosts and producers are not responsible for any actions taken based on the discussions in the episodes. We encourage listeners to consult with a professional or conduct their own research before making any decisions based on the content of this podcast

249- Automation and the Future: A Look at IT/OT and Control Systems with Brian Romano

3 Key Takeaways

Episode Show Notes

A Journey of Five College Degrees [00:00:42]

Designing a Virtual Reality World for IT Professionals [00:02:45]

Solving Technical Issues on the Go [00:15:25]

The Remarkable Story of Remote Troubleshooting [00:18:42]

Skills Gap in College Programs [00:19:44]

The Need for Effective Marketing in Technical Education [00:22:43]

The Fulfillment of Seeing Machines in Action [00:26:05]

The Importance of Precision in Operational Technology [00:37:29]

The Importance of Cybersecurity in Industrial Environments [00:51:52]

The Connection Between Networked HMIs and Cybersecurity Risks [00:57:22]

The Future of IT: Data Analytics and AI [01:04:44]

Fallibility of AI and the need for human oversight [01:14:40]

Transcript

Speaker 0 | 00:07.899

Hi, nerds. I’m Michael Moore, hosting this podcast for Dissecting Popular IT Nerds. I’m here with Brian Romano, Director of Technology Development at the Arthur G. Russell Company, Incorporated. Hi, Brian. How’s it going?

Speaker 1 | 00:21.824

Not too bad. How about yourself?

Speaker 0 | 00:23.445

I’m doing pretty good. I have to say that as I was trying to type your name… My hands got a little tired because Brian Romano is short, but all the letters after your name, AS, BS, MS, MBA, PhD, my fingers cramped up and I couldn’t keep going.

Speaker 1 | 00:47.541

Yep, that’s true. I do have, I have to say I have five college degrees. I started Early on in my career, 1980, I started as a junior in high school. I went to the University of Hartford and got an associate’s in 83 and rolled the credits over right away and started in the EE program. Kind of stopped short there. I have four kids. When my oldest daughter was approaching school age, being very competitive and always been an athlete my whole life, I couldn’t be beat. So I went back to school. and finished my bachelor’s, fell in love with, and I did it in a way that married with what I do for a living. So it flowed together nicely, got my bachelor’s, rolled into my master’s in applied computer science. And again, it was always a means to an end. I did it to figure out the next link in industry and what I needed to do to get there. So that was my master’s. Then I went for my MBA. In between all that, I owned a company for 16 years. So I wanted to be a better boss soon. figure out how things are. But the MBA also had a data analytics component, which has helped me focus on a lot of the things in today’s industry. And finally, PhD. I just got my oral dissertation, my defense, I’m sorry, my defense of my oral dissertation last Wednesday before Thanksgiving, passed that and got the official okie dokie on the PhD. Again, that’s a means to an end. Technology and innovation is the concentration on that.

Speaker 0 | 02:24.503

So really here, should I be calling you Dr. Brian Romano?

Speaker 1 | 02:29.606

You can, but you don’t have to.

Speaker 0 | 02:31.828

Come on. You took all that. You get to say it. Dr. Brian Romano. I love it. No, that’s fantastic. What a learned scholar you are. It’s time for our… our icebreaker segment we call Random Access Memories. I ask a question and you respond with the answer that comes to your head first. Your first question, if you could design a virtual reality world specifically for IT professionals, what features would you include?

Speaker 1 | 03:09.880

AI, for sure.

Speaker 0 | 03:13.063

Requirement now. Yeah,

Speaker 1 | 03:14.404

exactly. Let’s see. Boy, that’s a good one. There are so many hands-on things inside the IT world. How do you make it virtual to where you don’t have the hands-on side? I don’t know. You got me stumped on that one. Sorry.

Speaker 0 | 03:35.841

No, I like it. No, you got a good point, though. It’s actually a really good kind of a stumped question, right? Because you’re saying, okay, you’re creating a virtual reality world. IT professionals. And then you’re like, hey, let’s include all the things that IT would like. And all of a sudden, it’s like, you know what I would put in there? I’d put a cloud.

Speaker 1 | 03:58.428

Okay. Okay. It has to be, right? Yeah, exactly.

Speaker 0 | 04:04.091

We’ll add that. So now we’ve got a cloud and we’ve got yours as well. So we got something. So it’s a start, Brian. It’s a start. Maybe I’ll continue this question with other people and we’ll just keep adding to the list.

Speaker 1 | 04:19.736

There you go.

Speaker 0 | 04:20.056

All right. Here’s the second one. In a futuristic IT-themed movie, who would be the hero? Who would be the villain? And what technologies would they use to battle each other?

Speaker 1 | 04:37.230

Oh, the first thing that came to mind, I’m sorry to say, is Terminator.

Speaker 0 | 04:43.556

That’s a good one. Yeah. I’ll take that.

Speaker 1 | 04:46.317

Arnold over there being the bad guy. I mean, and it’s got the whole AI content part of it. You know, it’s so who would be the hero? Let’s see. They could either be. Matt Damon or Tom Cruise.

Speaker 0 | 05:01.278

Okay. All right. All right.

Speaker 1 | 05:03.959

Maybe Elon Musk.

Speaker 0 | 05:06.239

Oh, boy. You’re going to spark some comments for that.

Speaker 1 | 05:11.321

And what technology? Boy, that’s a good one. Again, it would be, first of all, futuristic. It would be time travel. You’d have to have that element part of it. Okay. And, I mean, in the real world, again, you would have the whole fiber optic led. networking and things like that. But also again, with the whole, I keep using the word AI and I apologize, but I mean, there’s so many different aspects. There’s the language part of it. There’s for the chat GPT, there’s the process part of it, the time series data there’s in part of my world, there’s image acquisition and processing for the AI. So there’s three different aspects of AI for there. So it’s all about math at that point, quite honestly. So, but.

Speaker 0 | 05:58.302

that’s the futuristic part wow holy that was a lot so um i’ve been giving you some loaded questions here i’ll give you an easier one all right this one this one’s easier uh if it processes had their own music genre what would it sound like smooth jazz you know i you know that’s funny because i was looking at it i was like i was i was back and forth i was like is it going to be like a techno is it going to be you know uh um just because i was all you know i was thinking earlier on i’m like earlier on it could be that um you know almost like techno when you had the aol connection right uh but no smooth jazz man it’s get

Speaker 1 | 06:43.076

kenny g up in here right now that’s not part of my personality i’m classic rock all the way but when you said that the first thing that came to my mind is is an it guy sitting at a computer with the headphones on Kind of mellow, just making things happen.

Speaker 0 | 06:58.023

But I like it. No, that’s good stuff. But, you know, I think you’re right. And I think I would just interject probably smooth jazz with the occasional, like, heavy metal, like, out of nowhere, right?

Speaker 1 | 07:13.354

Wiping people up.

Speaker 0 | 07:14.995

Yeah, right. And that would just happen right in the middle of the night.

Speaker 2 | 07:20.919

Hey, guys, this is Phil Howard, founder of Dissecting Popular IT Nerds. I just want to take a few minutes to address something. It has become fairly apparent, I’m sure all of you will agree, over the years, that slow vendor response, vendor response times, vendors in general, the average is mediocre. Support is mediocre. Mediocrity is the name of the game. Not only is this a risk to your network security, because I’ve seen vendors on numerous occasions share sensitive information, but there’s also a direct correlation to your budget and your company’s bottom line. Not to mention the sales reps that are trying to sell you and your CEO and your CFO on a daily basis. That causes a whole nother realm of problems that we don’t have time to address. Our back office program. at Dissecting Popular IT Nerds. We’ve put together specifically for IT leadership and it’s on a mission to eliminate this mediocrity. And the best part is that we’re doing this in a way that will not cost your IT department a dime. So if you’d like us to help you out, get better pricing, better support, and jump on pressing issues in minutes, not days, then contact us now so we can get on. a call with you and conduct a value discovery session where we find out what you have, why you have it, and where you want to go and how we can improve your life, your IT department, and your company’s bottom line. What you’re going to end up with is, number one, just faster support from partners who care about your organization’s uptime and bottom line. And because you’re going to be able to access our 1.2 billion in combined buying power, you’ll be able to benefit significantly from historical data. And on top of that, you’ll also benefit from the skills of hundreds of on-demand experts that we have working behind the scenes that are all attached to our back office support program. So if you’d like, again, none of this is ever going to cost you a dime. At the very least, it’s going to open your eyes to what’s possible. Let our back office team provide you the high-touch solutions and support that your IT team deserves so that you can stop calling. 1-800-GIL-POUND-SAN for support. Now, if you’re wondering, what does this apply to? This applies to your ISPs, your telecom providers, all your application providers, whether you’re a Microsoft shop or a Google shop, what you might be paying for AWS, even Azure, co-location space, any of those vendors that you’re paying a monthly bill to. we can help you with.

Speaker 1 | 10:04.113

Hey, it’s Greg,

Speaker 2 | 10:05.133

the Frenchman secretly managing the podcast Behind the Curtain. To request your one-on-one call, contact us at internet at popularit.net. And remember,

Speaker 1 | 10:14.278

it will never cost you a dime. Well, you know,

Speaker 0 | 10:17.680

you said something really interesting when you were back there. You said, you know, you apologize for using AI. And it’s funny because so many people use it now. And it’s come up as a recurring theme on this podcast. over and over and over. And to the point where people are like, I know, I keep using it. I know I keep using it. But, you know, I thought about this the other day, and I think I know why it keeps popping up, because it hasn’t really been classified and segmented yet. So, for instance, when we say AI, it’s such a broad subject. It’s a broad sweeping subject. And much like what we’re going to talk about today, which is… IT OT, which I have yet to completely understand this, you know, all of this piece. I was taught IT OT, IT of operational technology, essentially. And we’re going to get into that in a minute because I want to know more about the letters I’m saying. Also a recurring theme here. So but AI, such a broad subject. It hasn’t really been. applied, but you were even kind of talking about it right there. You were saying, well, you have AI in this piece, you have AI in this piece, you have AI on this. And that’s kind of, I think, why it keeps coming up because AI is touching so many different things that, and we haven’t yet classified it and put it to the different items that we have. In fact, I think today we’re probably going to touch on it specifically when we talk about ITOT and any fun, new little acronyms that I learned along the way. So, I think it’s okay. I think I’ve come to terms with, we can bring up AI as long as we’re not talking about it in a very generalized sense, as long as we’re taking it and attaching it to something so we can start to have those conversations and classify it.

Speaker 1 | 12:20.565

So,

Speaker 0 | 12:21.706

I want to quickly turn the page though on AI. We will get right back into it. I guarantee it. because I want to start on this ITOT, right? We had folks that don’t know, I have a quick discussion, and not a long discussion, and Brian will attest to this, just a quick discussion up front about, you know, kind of what we’re going to go over, just so we can go over the page. But he started talking to me about ITOT and also CSE, Control Systems Engineering, two very intersecting, uh um items and i know uh i’ve had uh actually a couple friends uh that have delved into uh control systems and you should see their houses they are this i it’s like listen man i have an alexa and uh and to get that thing to work i i’m always constantly having to reprogram things and and it and i looked at like the code that it was just to ring a doorbell like i mean it’s like if i ring my doorbell At 5 p.m., I want it to turn on five different lights, make them a specific color, run an LED program to make them do a design. Then I want the thermostat to turn in a certain spot. And then I want you to change. It was like, I don’t understand how you got this much time to program out your house. I don’t have that. So, Brian, bring us into the world of control systems. and help us understand this?

Speaker 1 | 13:58.050

Well, first of all, real quick, there’s a common, probably personality thread between your friends and myself. We all abide by the notion of he who dies with the most toys wins. So you look in my garage, look in my cellar. I am exactly what you just said. I’ve got Google everywhere. I can turn on my Christmas lights. I can, you name it. My house is wired for sound here.

Speaker 0 | 14:23.490

I don’t doubt it.

Speaker 1 | 14:25.252

Yeah. So ITOT, that’s where we’re going, right? Yep. All right. So information technology, everybody on the podcast here probably knows a good deal about that. And from the office side of things. Now I’ll back up a little bit. I owned a business from 1998 to 2013 before going to work for Arthur G. Russell. And in that time, somewhere around 2004, I had many customers of mine. I was a control systems engineering. what’s called system integration company. I took hardware and software, PLCs, programmable logic controllers, and put them all together to make things happen automatically on the factory floor. As you’re doing that, the customers start asking for reports. They start asking for screens. Somewhere between 2004 and 2007, I had several customers start asking me to connect the factory floor to the business system. And what that means is I want to be able to take a work order from an ERP, pass it down to the factory floor, have an operator pick what work order they need to work on. When they select it on the HMI, the human machine interface screen, when they select it, it takes the work order, takes all the recipe data from the work order and pushes it down to the process. And the PLC then starts to go to work on what it was asked to do. And as it’s now running, it’s gathering data saying how much current am I using on a mixer? What is the weight in my hopper? So all of the things that go along with the… the process parameters and other things are being pulled and brought to a database. Now, back in 2007, we started doing SQL server logging and reporting. We started pulling data from the factory floor, putting it in a place. My first database was, oh my God, it looked like an Excel flat file. I had no idea about relational tables and schemas and all that. That was one of the reasons I got my master’s degree. I wanted to know how to do that right. So, but I mean, customers were starting to ask for this. Okay, so here’s the, in my opinion, the birth of IT and OT. A lot of the things that I just talked about, your IT people are going, yeah, I get it. I’ve got a Cat5 or Cat6 based network. I got routers. I got switches. I’ve got, what I did is I put what I called operation servers on each plant floor that was responsible for interfacing. to a link table on something like a JD Edwards AS400 system. I’d pull the work orders down to a local table. I’d have print servers built into this server. My database was in here. So all of these are all IT things, right? But the OT side now is the operational technology. I’m taking all the information as the factory is working, as the things are being built and mixed and baked and whatever. And all that information is going into a database. Then what you can do is after the fact, when the process is done, you can say, how much material did I use today? How much of this particular type of material did I use? How many rejects did I have? How long did it take to make? How long was it before an operator acknowledged an alarm? All of those things are coming to light nowadays a lot more. But even back in 2007, customers were asking for this kind of stuff. Um, so the, the birth, as I put it, um, of that kind of stuff is now called industry 4.0, uh, is started for me as early as 2004. Give you a little bit of an, another anecdote, which is kind of cool. Somewhere around 2009, I think it was, I had just activated a cell card on my cell phone. Coincidentally that morning, one of my technicians and I were driving up. We were in New Hampshire at a customer’s facility starting up another machine. We’re on our way home and a customer who has a plant in Oregon called me up and said, hey, my laser printer is not etching on the bag. I’m sending the data, but it’s not getting there. Can you help me? My technician’s driving. I’ve got a new laptop with a new cell phone card in it, cell card. I open up the computer. Cybersecurity was a little thinner back then. logged on to their VPN that they gave me. They gave me access to certain areas. I logged on to the VPN. I went over from Connecticut to where I live to Oregon. And because it was a system I put in, I recognized what IP addresses should be what. I started pinging around for the laser. I started pinging around for the PLC. And I got to a point where I could segregate. There was a switch that was bad. So I’m doing this. driving 70 miles an hour, I’m sorry, 65 miles an hour down the highway.

Speaker 0 | 19:30.293

In the passenger seat.

Speaker 1 | 19:31.573

In the passenger seat, yes.

Speaker 0 | 19:32.794

In the passenger seat.

Speaker 1 | 19:35.675

And I got to a point where I’m on the line with the guy and I have my other cell phone, my cell phone, cell phone in my hand talking to him. And I tell him, there’s a little gray box on the bagger. Go open it up and see if there’s lights on it. He opens it up, there’s not. He. I said, go to your office. Do you have any like link six switches or anything like that? He goes, yeah, we got a spare one in a drawer. It’s like an eight port. I said, perfect. So he grabbed an extension cord, plugged the link six switch into the. um, into the extension cord, unplugged all the cat fives, plugged it in. I said, try it again. Wham started working. He was able to, uh, start running his product again, driving down the highway within 15 minutes. I solved the problem across the continent using a combination of ITOT structure.

Speaker 0 | 20:26.773

Wow. Wow. I, I mean, and that’s, it’s amazing. Uh, um, and that’s, and that was quite a while ago.

Speaker 1 | 20:34.675

Yes, correct.

Speaker 0 | 20:35.355

Yeah. That’s what makes the story pretty. pretty remarkable right because this is before because right now i mean we can do that but you know because you’ve got cloud and you can tie it in with you know you can tie it in and connect in over here and you can even connect the msp is you sorry you can even connect the um uh the you know on-premise systems uh right uh with the um you know with the systems uh that are in the cloud as well yeah but back then that wasn’t a thing you know You were creating your own way to get over there. And just the ability to be able to kind of see that, you know, connected in from so far away during that time. That is a pretty remarkable story. By the way, to catch everyone up, again, we’ve learned new things. We’ve got new letters. There’s PLC, programmable logic controllers. right and hmi human machine interface right these are these are new uh new ones to me um i did think i knew about knew the plc but um you know i haven’t worked with much of it to to know so this is pretty interesting stuff now let me ask you a question um you know i’ve been on it for quite a while but um you know i’ve heard of this stuff but i haven’t really put it together um and i’m just sitting here thinking you if i’m uh uh you know uh you know been around the block in it and various different things and even been uh and done some stuff in manufacturing um with erps that you know so if i haven’t heard of this stuff in some of the detail that that you’re describing um have other like uh is this a problem of other people not uh i mean Is this something that you can find in colleges?

Speaker 1 | 22:36.194

No, no. You hit on a hot button topic of mine. So everybody knows what a double E is, electrical engineer. They know what a chem E is. And in my industry and what I do for a living, it’s a blend of a couple of skill sets. It’s a blend of electrical engineering and computer science in terms of programming and networking and IT. So. It’s like two or three skill sets. I’m not saying we’re experts in every one, but we have to have knowledge of these at some professional level to be able to implement them. So you go into any of these programs, you go into a double E program and you’ll get the hardware aspect of it, the design of the hardware, and you’ll get some coding. And you may know what object-oriented code is, but maybe not how to implement it completely. Then you go to, or you go to computer science, computer engineering. and you’ll get a lot of the coding side, but you won’t understand the industrial hardware side. Then you got the whole ITOT thing again. And none of these programs, they may have a basic networking class, which will be good. Don’t get me wrong. That’s great. You need to have that. But all of these things together don’t exist. So anybody in my field, I do presentations around the country for Industry 4.0. And one of the slides brings up a workforce. shortage and a skills gap. And it’s something like if you look at the professional and engineering aspect, it’s something like 40 or 50% of the open jobs right now are in the manufacturing professional engineering aspect. And it’s because there’s a large skills gap. So what’s happening is as the baby boomers are retiring off and this acquired experiential skillset is leaving us, there’s not… any way to fill up the funnel, if you will. So I’ve gone to a couple of universities. I’m an advisor to universities. I teach at the University of Hartford. And I am trying to bring this automation skillset, this ITOT skillset, the things that kind of surround industry 4.0, trying to bring that to industry. So yeah, there’s not. And here’s the hard part. When you grew up in the 60s, 70s, 80s, and somebody said an industrial environment, the first thing that pops into your head is dirty, dank, and dangerous. And it’s dirty. I’m going to get dirty working in greasy pits and stuff like that. It’s dangerous. I might have… presses banging all over the place. And it’s, you know, dank and dreary. That is not the case at all. But that’s the view that a lot of people have, especially the mothers and fathers of people that grew up in the 60s, 70s and 80s, who now have kids going to college, and they want them to get a good job. They want them to utilize their skill sets and everything. And they don’t know what control systems engineering is. They don’t know what IT and OT is. They do know what EE is and again, KEMI and things like that. So the hardest part of all this, I hate to say it, marketing. We’ve got to figure out a way to take everything I’ve just talked about and bring this to light so that the parents out there will talk to them about their kids. The kids will go to the guidance counselors. The guidance counselors have to be knowledgeable of this stuff. in order to start the pipeline to be full and getting full with people that will replace people like me.

Speaker 0 | 26:18.308

Well, there’s so much packed into what you just said there. And wow. Let’s start with the fact that, and I think you hit on it, that some of the generation that is saying, hey, I want you to go out and get your… get your degree and stuff like that and then some you know and and they look at a a manufacturing job or an industrial job and they’re like well no no go into services go into you know something else um but you know having worked both right um they all they both come with their just interesting uh interesting pieces to them um manufacturing is uh um is very tangible. That’s one of the things I loved about manufacturing, right? There was a very tangible quality to it. Services, not so much, you know, but also the giddiness of walking on a floor and watching something get produced. I mean, it was so cool. And it was so neat. This is a thing that… I always thought it’s like I do IT and I watch a thing get made. Right. And part of what I’m doing helps do that. You know, that’s what to me was the most remarkable piece of it, because everyone has a piece in the cog and you’re actually building and making something, which is, you know, that right there. Why is why is that so hard to market?

Speaker 1 | 27:58.596

Oh, you’re absolutely right. It’s not. It’s an unknown. I mean, it’s just it’s a. I don’t want to say relatively new. It’s not really, but the merging of all these skill sets is relatively new. I mean, this kind of stuff has been around for a while. Going back to what you said, one of the classes I teach is automation systems. And some of the students, I walk up and I say, look, you’re going to be doing a lot of paperwork. You’re going to be doing a lot of design, a lot of programming, a lot of late nights getting this stuff debugged and working, but I’m going to promise you something. The minute you walk up and you press the start button and the machine does what it’s supposed to do, what you programmed it to do, and it starts running, there’s no better feeling in the world. And I’m a little bit of an advantage because I can cherry pick the students that I had in my class and have them come to work for me. So one of my better students I had, I asked him, I actually brought him in. He was teaching lab for me at the class. And we were in class one day and I said to him, tell the students what I told you about having the best feeling in the world. He kind of reiterated what I said. And I said, what’s your feeling on that? And he says, absolutely true. So what you said, Michael, is absolutely true. There’s no better feeling than watching something that you put all that effort in coming to life.

Speaker 0 | 29:24.181

And there’s also another piece to it, right? And I think the other piece is. we’ve come a long way in terms of automation. And man, I have seen some robots on these floors do things that, I mean, you just sit there and watch them. And as they swirl around and like arms, they have arms and grip things and put, and you’re like, how is that even? It’s such a graceful movement. It’s almost an art form to sit there and watch these things move in such a way. beautiful. It’s, dare I say, sexy, right? And that’s why I say I don’t understand why it’s hard to market it, except maybe because of the stigma. uh, the old industrial and the old manufacturing still exists, you know, in which, Hey, get yourself out of, uh, uh, you know, uh, a blue collar job and into a white collar job. Right. And that’s, but that’s not the case anymore. Right. That’s that really, that doesn’t really exist. There’s people that, uh, um, that make tangible things and there’s other people that sell things that are not tangible. And, uh, and. Those are essentially what you’re working in, one or the other. You’re either making something or you’re selling something that is vapor, right? But does something, right? But that’s what’s remarkable about it is this is actually a physical thing. And it is really, really, really, really fascinating to see all the different pieces. I get excited like I’m in school again. And I’m like, I’m in. elementary school, seeing something for the first time, I get excited when I walk onto a manufacturing floor because you just never know what you’re going to see.

Speaker 1 | 31:18.315

Yep. No, that’s absolutely true. The Made in America show brought some of that to life. I work for a company that does exactly that. We do automated assembly systems and packaging systems that if you need something put together at slow speed, at medium speed, at high speed, We take all the discrete components and put them together for you. And that takes robots. It takes servos. It takes PLCs, computers. I started putting computers, embedded computers on every machine with software to analyze data and things like that. So, yeah, absolutely true. You’re really kind of taking something and bringing something to life. Yeah.

Speaker 0 | 32:02.938

You had mentioned in there a little bit about. debugging, right? And, you know, anybody that has stayed up way, way, way past their required bedtime to code something and try to fix something because it won’t work because they keep debugging it and it’s still there and everything like that. There maybe is, I would have to say there is an art form to debugging. There is a science to debugging. I’m not good at it. I sit there and I essentially am just like, no, I’m just going to comment out this whole section and just start working this one spot and see if this works. Okay, now that works, great. All right, let’s go to the other one. And I just keep,

Speaker 1 | 32:54.666

you know,

Speaker 0 | 32:55.646

let me find the section that it’s dying at. Put flags everywhere and have it create a little… files so that I, or a little section so I know it got to one spot or another, but, um, it, to me, it feels, it feels random. It, it feels like it’s, um, it, like there’s not a, uh, um, a science to it probably because I’m not, uh, doing it correctly.

Speaker 1 | 33:17.778

You’re not used to doing it maybe.

Speaker 0 | 33:22.620

So I guess, tell us a little bit about debugging and, uh, and, and what the, um, you know, what, what surrounds the art form to it? What, We don’t what is the process? Because in the one thing I just want to kind of get clear on why I’m interested in this topic is because it’s the measure twice, cut once theory. Right. And I read an article that you had linked to. And in that article, I had talked about essentially making sure that the what you what you’re producing and I’m paraphrasing here, but what you’re. making producing or everything is actually meets the actual design requirements uh and and they’re linking those together which i also think when you back up for a minute and talk about project um just general project management um that’s always the goal right you know did we create did we perform and do the thing that that we we said we were going to do or did we scope creep and and just create something completely different right yeah and and

Speaker 1 | 34:29.188

The whole debug essence that you’re talking about. Yeah, you’re absolutely right. There was another article. I’m not sure if that’s the one you were talking about. There’s one that was just released a couple of weeks ago that they interviewed me talking about debug methods, coincidentally. And it was talking about, okay, and it might have been the same article, but we’re talking about starting off with

Speaker 0 | 34:48.984

URS. Is that an assembly?

Speaker 1 | 34:50.966

Yes, assembly magazine.

Speaker 0 | 34:51.907

Yeah, it was the same article.

Speaker 1 | 34:53.068

Okay. We start off with the URS like you talked about, user requirement, the spec.

Speaker 0 | 34:57.716

We got a new one. You got a new one,

Speaker 1 | 34:59.538

everybody.

Speaker 0 | 35:01.660

URS.

Speaker 1 | 35:02.561

URS. Yeah, acronyms are us. So you start off with that and that becomes your… your design guide of how, how things should flow. It will have in there, um, things like you will use this brand X, Y, Z. You won’t use brand, uh, W and Oh, by the way, you have to program in this methodology. Uh, you can’t program in this way. You can’t use flow charts or, or structured text. You have to use ladder logic and blah, blah, blah. And they, they kind of spell out the way that their plant runs. And you’ve got to adapt to it. So that’s kind of the beginning of everything. And once you go through the whole design aspect, there’s checks and balances that go along with that as well. The way that we do it right now is we’re trying to minimize the amount of errors, if you will, that make it to the floor where things are built. So I do a third eye set of eyes for checking. So somebody will design it. Somebody will draw it, two separate eyes. And then somebody who will compare. the bill of materials and the, the drawing, the design aspect, the third person will say, yep, everything matches good release to the floor. So three,

Speaker 0 | 36:16.412

that would be a person.

Speaker 1 | 36:19.375

Right now it does. Yes. I, I can see where you’re going. Cause yes. And in the future, there’s really a way you could, you could code that out for sure. Yes. And then from there, okay. Then somebody builds it, they wire it, they program it. And the programming again is, um, kind of a guided functionality from the customer number one and the needs of what the mechanical engineers have designed. We’ve got to animate it and automate it. So we write the code that animates the mechanical structures. And then from there, the electricians wire everything together. And then we go through, in that article, I talked about a systematic checklist where you go through, all right, here’s all my inputs and outputs. If I fire an output, Does the proper thing happen? Yes. A switch was made. Does that happen? Yes. It’s supposed to trigger an alarm that you program. Does the alarm happen? Yes. So you go through inputs, outputs, alarms, info messages. So systematically working your way through the design, through the programming to the point where now you bring back in other trades and the mechanical assembly guys that will actually now start working with you to bring the machine up to. production where you’re actually starting to make something. Just to kind of give everybody here listening a quick idea, if you were to take a ballpoint pen, a click pen that you have on your desk, and you were to take it and spin it apart and put all the basic components on your desk, you’ll probably end up with like six or seven components. The spring, the nib, the little clicky thing, the barrel on the bottom, the top, all of that. Now, what we do is we take all those discrete components and assemble them. in high throughput or low throughput or whatever the customer needs. If they want to do 10 a minute, we do it. We’ve gone up to 1800 a minute. And we do that by doing many at once. I might, every cycle of our machine may build something like 30 or 40 of this assembly. In this case, we’re talking about a pen. So all of these things are electromechanical. Now, going back to the whole ITOT thing. Everything kind of comes together here. PLCs are all Ethernet networked. All the touchscreen HMIs, they’re all networked. All the encoders that run the machines are Ethernet networked. So you kind of see now that there’s a level of IT technology, the whole I need to know how to put switches and routers and NAT devices on the floor so that it runs my automation equipment. And then I have to put a gateway on there that takes the data that the machine generates and brings it up through the double doors. And we’ll talk about that in a second. And brings it to the other side where people can do some work on the data. So there’s the kind of the combination of everything. It’s systematic, starting with the URS, working its way through a design specification, going through IO checks, debugging. And then. the whole debugging part of it that you talked about with software, you hit it on the head. I mean, you do it in modules and you bring each module online. If there’s a problem, you cut the problem in half and then work on the half that’s not working and keep doing that until you get to a point where you’ve isolated it and go from there.

Speaker 0 | 39:52.418

Yeah, but you said it much better. Yeah. That’s so incredibly interesting. It’s amazing because- You’re forced to have to sit there and make sure that all of the T’s are crossed, all the I’s are dotted. If you do not. it’s not going to work at all uh everything has to work um you know but it but it’s an interesting concept because you have to build that into your time right and i do feel like sometimes um uh when projects happen that don’t require um anything having to do with uh um operational technology or control systems uh you know um they don’t go through those rigorous tests You know, they don’t go through those. They just get it about 80 percent of the way there and we’ll fix it on the way. Right. But you don’t have that. You can’t do that. You can’t fix it on the way. I mean, you’re sending through, you know, how many how many at once. Right.

Speaker 1 | 40:56.276

Did you say you’re sending through 48 every cycle? Yeah. Yeah.

Speaker 0 | 41:00.940

You can’t fix something on the fly. That’s that’s running like that. I mean, you know, you’d have such a high error rate or it just wouldn’t work. So. So it’s actually pretty remarkable. You’re actually forced into this methodology that really everybody should be using. But, you know, the you know, the the requirements to do other types of, you know, I.T. work don’t match up with having to do it 100 percent, mainly because, you know, it’s about time. It’s about get it in, get in quickly. We’ll fix it as we’re going. Right. And you see this and that’s that’s infrastructure. Right. uh infrastructure is uh um build the road while it’s still still there you know and uh just build around it make another lane let them go this way while we fix this thing and then push them back so there’s a there’s an art form to infrastructure and uh and it happens to be having to do things uh um you know probably not the most prettiest way to get them done sometimes but they have to do it but oh wow to do this operational uh technology and control systems it you have to you really have to have all of your stuff planned out and have your tests and checks to be set to be able to do this correctly. And thank you for explaining debugging in such a beautiful manner. It actually sounds really, really good the way you did it. I’m adding that into my brain now, right? So, wow. There’s so much. You talked a few times about…

Speaker 1 | 42:40.484

industry 4.0 yes tell us a little bit about industry 4.0 all right so back like what happened to one two and three is five coming out like what’s happening there yeah um okay the quick primer industry 1.0 also known as the industrial revolution somewhere in the 1700s um things were mechanized eli whitney with the cotton gin samuel colt uh with the firearm things like that uh interchangeable parts mechanizing things. Then industry 2.0 later came where this was more of the Henry Fords of the world, where you took the idea of interchangeable parts and mass production, and you put them on conveyor systems, and then you start mass producing things in various plants and across plants. Industry 3.0 is where kind of everybody listening here, I’m guessing, grew up. 60s, 70s, 80s, 90s, 2000s, where you start… applying electronic controls and control systems and computers, the cyber physical, as it’s called, to the factory floor. Now you’re actually taking in generating data. During that period of industry 3.0, you generated data, were able to do reports, but that’s about the extent of what you could do. Here comes industry 4.0. It’s made up, it was first kind of devised. and announced at a Hanover Fair in Germany in 2011.

Speaker 0 | 44:05.685

That doesn’t surprise me.

Speaker 1 | 44:06.526

It’s kind of crazy.

Speaker 0 | 44:08.287

Yeah.

Speaker 1 | 44:10.267

And when I talked to you about what happened to me in 2007, that takes into account several segments, several aspects of Industry 4.0. There’s nine accepted pillars of Industry 4.0, augmented reality, systems integration, cloud computing, big data, industrial Internet of Things. 3D printing, cybersecurity. We’ll talk about that, I’m sure. Autonomous robots and modeling and simulation. Those are the nine accepted pillars that pretty much there are sometimes you’ll see 11 and things like that. But those are kind of the nine accepted pillars. This is literally taking everything. This is where IT and OT really breathe and live now because I’m taking stuff from the factory floor. I’m taking sensors, controllers. I’m taking rudimentary devices on the floor. If anybody here has heard of things like IO link, or even Wi-Fi or Bluetooth sensing, wireless sensing, bringing that to some gateway, getting that information from a gateway up through to a layer where it’s controlled. From there, it goes up to a layer where it’s kind of concentrated. And then from there, it brought up to the ERP system. Now, with all that information, the whole kind of goal of all this is to take and autonomously bring information from the ERP down to the factory floor. Take the information from the factory floor, feed it back to your planning stages. I need to produce this because I have low inventory here. And all this data interconnectivity is where this comes from. Then you bring it to a level where now if you’ve got it in an ERP system and you’re connected to both your value stream up and down where you’ve got your vendors and your customers, the customer can look at your system and tell how long something it might take to deliver or build or if it’s in stock. Some of all these components already exist. Don’t get me wrong. But now this whole thing is being a lot more autonomous, a lot more automated. Um, and all the things that I had mentioned, all of the nine segments, um, kind of come into effect. Augmented reality is used for support, um, being able to remotely support and keep the downtime down from a, for a machine systems integration is what we talked about, taking data, bringing it up and bringing it through the ERP up to the vendors and to the customers, cloud computing, being able to take your data from the factory floor, bring it up. Do some work on it in the cloud. Leverage the cloud structure. Leverage all the servers that are around the globe. Big data. Big data analytics. Talking AI again. That’s where a lot of this comes from. Back in the 80s, 90s, and 2000s, I could grab data. Couldn’t do anything with it. With the advent of low-cost, high-speed computers, I can now do a lot more with it. I can run it through AI engines. I’ve got all this data. I can train models. I can look at what that might mean for future stuff. And hopefully we come back to that in a second. Then there’s the IIoT, the Industrial Internet of Things. That’s taking sensors, adding sensors that bring not only information about key process indicating variables, KPIV, there’s another one for you, up to you go from there, you’re also looking at machine health. Why is the machine drawing more current than it used to? Why is it consuming more air? What is my bottleneck? Why is one of my stations running slow? All those sensors feed into that information. Then there’s 3D printing. 3D printing is not what you and I kind of think of. Well, to a degree it is. But again, it’s taking information from like an ERP saying, hey, I need 300 of Model XYZ made. This fleet of 3D printers over here are going to do that. So the 3D printers start working. Then there’s autonomous robots in the same vein. If I’ve got a fleet of robots, I can do quick tool changes. I can do expedited manners of changeover for new products. Cybersecurity, you can see that everything I’ve talked about now is taking data from the floor, running it through the enterprise up to the cloud. Guess what? I just opened up my enterprise to the cloud. Now I’ve got to have some level of cybersecurity to be able to protect my data and my hardware on the floor. And the last one is modeling and simulation. There’s a couple names for that. One of them is the digital twin. One way to look at that is if I design something in SOLIDWORKS or a mechanical set of software that represents the physics model of how a machine is built, and then I take my PLC programming and I take that and I marry those two together in a software package, I can literally do a digital twin of… that machine. I can run the machine offline on a computer while the real computer is off on the floor running. If somebody says, you know what, I want to make a change because I think I can get 3% more output out of this. They make the change in the computer. They change the code in the PLC in the computer, run it on the computer, prove it out, then bring it to the machine before the machine has to go down.

Speaker 0 | 49:36.108

It’s virtual modeling.

Speaker 1 | 49:38.009

Exactly. Yeah. I know I’ve talked a lot real quick, but that’s what Industry 4.0 is. Industry 5.0 is more humans working alongside the cyber physical part. People are starting to talk about it. It’s still a long way away. I don’t know if anybody’s done marketing or anything like that or read the book Crossing the Chasm, where they talk about the standard normal curve of marketing, where you have the early adopters, the early… the laggards and things like that. In a lot of these marketing things, you have this thing called a chasm where People aren’t quite jumping yet. You got the people, the early adopters who have invested a lot of money and want to do it because it’s cool. Once they’re successful, then you got the people jumping the chasm and starting to adopt it. We’re at, in my opinion, we’re at or just over the other side of the chasm and just starting to get industry 4.0 adopted in a more general sense.

Speaker 0 | 50:38.941

Industry

Speaker 1 | 50:41.002

4.0 or 5.0? 4.0. 4.0 is still in its infancy. It’s been developed. It’s been talked about. You’re still dealing with 60s, 70s machines on the floor. So there’s a lot of things that need to happen on the floor to really get a whole company on the bandwagon.

Speaker 0 | 50:59.771

In previous podcasts, we’ve actually touched on the cybersecurity concerns with the. an ability to upgrade some of the devices that have been there forever. Hey, we bought this device a long time ago. It does this one thing. It can still do the one thing, but it runs off Windows 3.1 and I can’t upgrade it. We joke, but there are some seriously old machines still running these things or not just old, proprietary. And only one person knows how to do it. And they’re off in, you know, vacationing and in a fun little place, you know, drinking martinis or something. I don’t know. This is what this is the reality of of what happens here. And and cybersecurity needs to be done at a level of when people are building things. Right. Because if you try to just interject it in afterwards, then you don’t get. uh um that seamless you don’t get that you know that seamless uh working workings of it so would you yeah because you’re you’re just you’re like oh yeah and then put put give it a cyber security package and move on i mean that it doesn’t really seem like that works too all too well that’s why that’s why uh when people code things and file things uh um cyber security is built in along with it right what does it need to do okay now now make sure you build that into the system uh and uh and it’s the same thing with when we build infrastructure as well, the same kind of concept when you build cloud. And so I want to ask about this because there are precise components to these things. We’re talking about the programmable logic controllers, right? What PLC, right? We’re talking about these PLCs. Great. I can protect the network. I can protect all this stuff. But let me get down to the actual PLC. Are people trying to hack that?

Speaker 1 | 53:08.465

Yes. Actually, there was some success as early as 2010, I think, somewhere around that area. There was a virus called Stuxnet, and it was aimed specifically at industrial components. And it was kind of a wake-up call for a lot of people. And to your point, it’s like, hey, I got cybersecurity on the IT side. I’ve got my servers protected. I don’t care what goes on on the floor. And, um… There’s a lot of things. We deal, our machines deal with production. And if it crashed, it would hurt a company’s revenue. It’s not going to kill anybody or make people starve or blow up things. It’s just going to stop making product. But this same thing, PLCs exist in gas plants, in power plants, you name it. So if you don’t protect those assets in a way that would… vitally stop people from penetrating it yeah you could have some serious serious concerns so yes cyber security is is something that’s i don’t want to say up and coming it’s here it has to be um believe uh one of the societies the organizations i belong to the international society of automation isa um they partnered with the iec uh and they’ve got a standard cyber security standard six four two six two four four three i think it is and it kind of dictates what this looks like from an industrial environment, how to put together your network. And they take the approach that they do, I guess you can call them blast zones. So you compartmentalize areas. So if something does go wrong, it’s compartmentalized. It’s not going to go free and wild. And that’s kind of the approach that’s taken with that.

Speaker 0 | 55:00.170

Interesting. You look at it. possible reasons why people wouldn’t want to implement this. And there’s a couple, right? I mean, we’ve already implemented it. Now we have to go back and replace PLCs and reprogram them. It’s a lot of work and all this stuff. But then on top of that, too, there’s an overhead to security, which could reduce production.

Speaker 1 | 55:26.913

Yes. Yeah. I mean, even to your point, if I had to take the time to put in to change out something that for the cybersecurity aspect, while I’m doing that, I’m not making production. So it’s kind of a twofold thing. You know, I’ve got the expense of the equipment that I have to put in, but I’m stopping. production while I’m doing it too. So yeah, there’s several reasons why you would want to do it. Yeah, there is overhead that goes along with it. Some of which is people being able to understand. Again, we’re harping on the IT-OT crossover where you have skill sets that cross, but there’s people that are involved with that, that understand this crossover and can support the equipment, both on the IT infrastructure and the OT side.

Speaker 0 | 56:11.583

So if… if I were able to jump in and get into a controller, right, could I crawl my way into the network at all? I mean, that’s the, yeah. So that’s the, and I believe it’s the case because you can crawl your way from the network into the controller, right? So, and you can exchange data with it. So anytime you can do exchange ones and zeros with something, you can hack it, right? So I guess- If that’s the case, then companies should worry about their data, right? They should start worrying about the controllers, these PLCs that are sitting out there that are not. Tell me a little bit, by the way, about HMI, the human machine interface, real quick.

Speaker 1 | 57:03.920

They’re essentially touchscreen computers, whether they’re PCs or embedded computers, but they’re essentially touchscreen devices. A lot of them have standardized software that you basically program displays, push buttons, dials, sliders. The graphical interface that people need to run a machine is what an HMI is. It can have a lot of recipe functionality where I can pull information. It can have data logging functionality. There’s a lot of things that go along with it, but it’s essentially the interface device, just like it says human machine interface. It allows a human to touch parts of the machine through this interface.

Speaker 0 | 57:47.247

So most of the times I’ve seen some things like this, they’ve been pretty locked down. Like this is, you know, you only hit these buttons to do these one things and it looks like they’re programmed specifically for the thing that they’re trying to do. Correct. However, someone has to program that, right? And that program can be altered.

Speaker 1 | 58:08.602

Correct.

Speaker 0 | 58:09.563

Right. When you get up to these HMIs, do a lot of them have logins or do they are just open and ready to do their thing?

Speaker 1 | 58:19.233

Depends on the year. No, quite literally, yes and no, depending on if it’s a standalone, if it’s an isolated island where the machine is on the floor by itself doing its own thing, there’s no need. At that point, you still have the ability to program. operator levels. I don’t want an operator changing set points, let’s say. So I will build in programming levels, operator levels, I’m sorry, password levels that allow different areas of some people can do maintenance on the machine. Some people can’t. Operators are only allowed to hit start and stop. So there’s different levels of that. Yeah. So to your point, if it’s an isolated standalone thing, you don’t necessarily need it. You would want to have those layers in there to be able to protect the machine. When you put it on a network, it becomes even more so. You want only certain people who understand the dangers, I guess, if you will, of going deep into the program to be able to do it. So, again, operators only have cursory levels of operation and control.

Speaker 0 | 59:31.658

So, really, the big piece is… whether or not these are connected into the network and exchanging data. Because if they are, then they can become an entry point, you know, where somebody can, and even remotely get to them, because you’ve already proven that that can be the case, especially if it’s connected to the cloud, you know, then there you go. If you can use that computer as an entry point, then you can use it to infiltrate the system and… and grab data and then extract the data out of the system again. And since PLCs seem to be very good at running. things, uh, in a specific order and doing things a certain way, it seems like a perfect way to even just hide a, uh, um, hide a program.

Speaker 1 | 60:22.212

Yep. You can certainly do that. Yep. Yeah. There, um, there’s going, going back to your, your point on going to the cloud, um, from a cybersecurity aspect, if you look at it, data should be flowing up to the cloud. And if you want to access the data from the cloud, you should be coming in from a layer that just allows you access to that data on the cloud. So you got outbound traffic coming from the PLC, from the factory floor, going up through to the cloud for analytics, for storage, things like that. And while the data’s up there, I can take a PC and then hit the data up in the cloud. No need for me to go down to the floor. So if I’m looking just from the data standpoint, and that’s where a lot of this configuration comes in. If I’m looking just from the data standpoint, part of the cybersecurity is… access levels, who’s touching what, where are things going. But I have opened a gateway. In my case, what I just said, it’s an outbound gateway, but that can be broke too, right? I mean, if somebody is good enough, they can figure out a way to do it. So yeah, there’s certainly a concern that you have to consider when you’re doing this.

Speaker 0 | 61:33.499

I think if we haven’t… connected these items to logging, centralized logging, with a back-end seam and a 24-7 sock sitting there watching what’s happening, there’s still a very big opening there from a threat standpoint, right? I mean, I really think from a manufacturing standpoint, folks ought to be thinking about some of the standard items that healthcare companies and financial companies do. You know, because a lot of those times, you know, you don’t have and this is the trouble, right, is that sometimes you have big manufacturing companies. That’s great. But other times you have small mom and pop manufacturing companies and they don’t have a lot of the resources to be able to implement a lot of this stuff, especially, you know, investing in a in a centralized logging seam and and sock in the back end. So to them, that seems like that’s a. that’s a tough order.

Speaker 1 | 62:40.298

Yeah. There’s industrial ways, quote unquote, to get around that if you want. I don’t want to say get around that. It’s not the right way. To implement it so that you can gain access to it securely. There are devices out there called, one brand that I happen to use is E1, E-W-O-N. It’s a company that makes industrial VPNs and it’s an outbound ping that just constantly pings a secure website. And- just says, I’m alive, I’m alive, I’m alive. And then only certain people have access and people who are granted access have access to that cloud site. And then what’ll happen is me, when I have my client on my side, I ping the cloud as well saying, okay, I’m here. I want to talk to IP address, blah, blah, blah. It goes across the network. We meet in the cloud. We negotiate in the cloud. Then I come down and there’s a couple other levels of security, but one of them is physical. I’ve got a key switch that will kill the enable circuit on the E1. I only can get access to that machine when somebody has asked me to gain access to that machine. So they walk up, turn a key switch. I have access to it. There’s outbound traffic from the E1 saying I’m alive. There’s no inbound at that point. We meet in the cloud, negotiate. My credentials are accepted. I now have a VPN tunnel to this device. And now I can gain access to my PLCs, my vision systems, my HMIs, all of that as if I was plugged into it at the machine itself.

Speaker 0 | 64:18.354

What I love about that implementation is it means that you have to use two different attack methods, which is one, you have to gain credentials and two, you have to use social engineering. So to be able to mix those two together to actually gain access, which is very difficult. So that’s I mean, it’s possible, but the harder you make, that’s the goal of of security is to be harder to implement than the other person. Right. Oh, this house has cameras. I’ll go to the next one. Right. So, you know, because I mean, you can never be fully secure, but you can you know, you can keep adding to your arsenal to make it harder and harder to get that. You can reduce that risk and mitigate that risk down. But yeah, I like the way that. And I like the way that that’s structured.

Speaker 1 | 65:05.842

Yeah, it works. We still have customers that say, nope, it’s connected to the internet. Nope, you can’t have it. So sometimes we have a couple of different ways around that. One of them is we ask the customer, keep a coil of Cat6 to your internet device nearby. If you need us, plug it in. And then when you’re done, unplug it. Then you never have to worry about it. No physical connection, you’re done. The other way is these same devices have cell data. So I can meet in the cloud. I don’t have to hit any of their physical internet infrastructure in their plant. I can just come in over the cell and hit the machine directly without hitting their network. So there’s a couple, three ways to kind of implement this and keep it secure.

Speaker 0 | 65:52.194

Yeah, very nice. Very nice. So we have, we’ve gone over a lot of things so far, and we’ve learned a lot of things. We’ve learned ITOT, CSE, PLC. HMI, URS, KPIV. I’m sure I lost a few in there, right? I’ve started a brand new dictionary after this call.

Speaker 2 | 66:18.636

At Dissecting Popular IT Nerds, we expect to win and we expect our IT directors to win. And one of those areas where we know that we can help you win is internet service providers. As an IT director tasked with managing internet connectivity, few vendor relationships can prove more painfully frustrating. than the one with your internet service provider. The array of challenges seems never ending from unreliable uptime and insufficient bandwidth to poor customer service and hidden fees. It’s like getting stuck in rush hour traffic. Dealing with ISPs can try one’s patience even on the best of days. So whether you are managing one location or a hundred locations, our back office support team and vendor partners are the best in the industry. And the best part about this is that we can help you. None of this will ever cost you a dime due to the partnership and the sponsors that we have behind the scenes of dissecting popular IT nerds. Let us show you how we can manage away the mediocrity and hit it out of the park. We start by mapping all of the available fiber routes and we use our 1.2 billion in combined customer buying power in massive economy of scale to map all of your locations, to overcome construction fees, to use. industry historical data, to encourage providers to compete for the lowest possible pricing, to negotiate the lowest rates guaranteed, and to provide fast response times in hours, not days. And we leverage aggregators and wholesale relationship to ensure you get the best possible pricing available in the marketplace. And on top of all of this, you get proactive network monitoring and proactive alerts so that you’re not left calling 1-800-GO-POUND-SAN to enter in a ticket number and wonder, why is my internet connection down? In short… We are the partner that you have always wanted, who understands your needs, your frustrations, and knows what you need without you having to ask. So, we’re still human, but we are some of the best, and we aim to win. This all starts with a value discovery call where we find out what you have, why you have it, and what’s on your roadmap. All you need to do is email internet at popularit.net and say, I want help managing all of my internet garbage. Please make my life easier and we’ll get right on it for you. Have a wonderful day.

Speaker 0 | 68:38.459

Let’s go ahead and start our IT crystal ball in this segment. And when we do this, normally I kind of free for, sometimes I’ll be specific and kind of talk about the future of IT. And sometimes I just kind of free form, let people go run wild and see what their imagination comes up with. But we… sort of kind of uh already uh figured out i think uh where this one’s going because um you talked about and and we we we’ve touched on it a few times when we said data analytics right and in through the process we’ve kind of sprinkled it through um ai uh is data analytics gone wild right i mean it it is what happens when you can just you know, give a bunch of data to something that can actually make sense out of it, right? And actually has enough of a pattern recognition. That’s where we are right now, right? With AGI, right? This is a pattern recognition, you know, where basically we have figured out that when people do these things, that these are the next things that come so that we’re going to fill it out. with that information. And we’re at the lowest spot of AI right at the moment. It’s like,

Speaker 1 | 70:06.729

you know,

Speaker 0 | 70:07.730

the lowest you can possibly be, which is just scary at just how good this is and how fast it has exploded. And it touches so many things. So… We talked about all of these pieces that make up ITOT and CSE, and we talked about implementing them in. And then we specifically kind of, we touched a little bit on data analytics, but we didn’t really dabble completely into it. And that was by design because I knew that this end part of this podcast was going to specifically hover around the future of IT. with data analytics powered by AI, right? That is essentially where we’re going to go with this thing. And I think you and I are probably in agreement, right?

Speaker 1 | 71:00.540

Absolutely. Yep. Yep.

Speaker 0 | 71:02.941

I’m just, Brian, I’m just going to let you talk because I have no idea. I have ideas about where this could possibly go and I’ll interject when we do it, but I’m going to let you just start going because I just want to see where we go with this.

Speaker 1 | 71:19.567

Okay. Well, there’s already some predetermined paths. There’s already some software out there. There’s already ways and methods to do this. But a lot more data needs to be collected and characterized, imputed, all of that cleaned up. Now, going back to what you had said, you hit it on the head. Predictive analytics, predictive and preventative maintenance. So imagine you’re in a world where… All of a sudden, you get a little alert on your car that says, within the next two weeks, you’re going to lose the bearing on your right-hand tire. You’re still able to run. You got no issues. But now you can make an appointment. You can order the part. You can bring it to the dealer. You can get a rental car. All on your time. You’ve got two weeks notice to be able to do this. That’s all driven by data, data analytics, and the AI side of things. So there’s software out there, there’s methods out there to do exactly that. We had talked about the whole industry 4.0 thing, taking data from the factory floor, taking machine data, taking KPIV, key processes indicating variables, bringing those in, flooding a repository full of data. And then something happens on your machine one day, you lose a motor. There’s all sorts of data that surrounds that failure. There’s temperature, there’s flow, there’s pressure, there’s photo eyes that might have temperature sensors built into them for temperature compensation. And you happen to be reading that. There’s vibration analysis sensors. All of these things are kind of built in and you’re sucking all this data up and then you put it against a trigger. You say on December 1st, 2023, I had a bearing or a motor failure. What were the aspects of the data around that? that might indicate that failure. And now you have a training set of data. Everything that you’ve had in your past becomes now part of your historical training data. Now, in the future, you’re running fat, dumb, and happy, just like the car scenario. And the AI engine’s sitting there now looking at the data real time, and it’s looking for those same triggers, that same data signature, and says, last time I saw… The photo eye temperature rise by three degrees. And I saw the vibration go up to 73 Hertz. And I saw a little bit more consumption. Anyway, you get the idea. When all of these variables come close or match to the limits of the last one, I can do that far enough in advance if I’ve got all the data that led up to that event and put up a flag well in time to say, Mr. Operator, you’re a… about to have a failure, you’ve got an hour a day, a year, a month, whatever that happens to be based on historical data. You say, okay, Mr. Maintenance Man, I need a motor that I’m going to have a failure. I’m going to need a motor. Do you have one in stock? If not, please order one. They order the motor. They have it available. You now gracefully bring the machine down. No downtime, no actual catastrophic failure that’s happened. I haven’t crashed two things together. I’m bringing the machine down gracefully. I schedule that in. Maybe there’s other maintenance things I need to do. I schedule it. Then I fix the thing, bring it back online, bring it back up. No catastrophic failures, minimal amount of downtime. This is all surrounded around this whole predictive and preventative maintenance that’s driven by AI engines. And it’s there today. It’s not used a lot because the… The data’s not quite there yet. There’s not a lot of people that have clean data. They’ve got disparate data all over the place and different. formats, different schemas, different, and somebody’s got to bring all that together, clean it up, characterize it, and then put it in a method in a way that the AI engines, I use the word engine, AI engines can look at it.

Speaker 0 | 75:36.334

I think you summed that up really well. And there’s kind of a couple pieces that you can take from that. So the first one is, if you’re taking all this data, and you’re logging it, to be able to do all of these amazing automations and preventative work and kind of alert to things that haven’t happened yet, then essentially you’re also take that data and throw it against a cybersecurity framework so you can monitor it as well, right? I mean, so the great part is this AI requirement, which is I need data, right? and is also a business requirement, I need data, also meshes with the cybersecurity requirement of I need data,

Speaker 1 | 76:26.246

right? Exactly.

Speaker 0 | 76:27.766

And I need good data. That’s the other thing you mentioned. And this is a constant problem in a lot of organizations because data is entered by humans. And we use computers to make humans enter things the same way every single time. Otherwise, you will have, you know, items that are entering in completely different orders and in different spots. I mean, the idea for good data analytics is to make people enter data in the same way every single time because not having clean data breaks everything. It breaks systems. In fact, you know, many attack vectors are used based on injecting. bad data into systems, right? So this is such an interesting problem. And I actually wonder now if AI can help us even cleanse that data appropriately, which is kind of a question of, do we want that to happen? Yeah.

Speaker 1 | 77:39.828

Is that the trip? Is that the edge? Right?

Speaker 0 | 77:43.071

Because if we’re using AI to cleanse the data, um, you know, that humans entered, right? Now you’re, now you’re like, okay, we’re going to clean the data, we’re going to collect it, we’re going to clean it, and then we’re going to predict on it. And then now you’re going, okay, well, now I’m, I’ve put all my eggs in one basket. Let’s hope it doesn’t, uh, let’s hope that basket’s, uh, sturdy because if it, if it’s not giving me the right, if not cleaning the data correctly, if it’s not, uh, um, predicting the data correctly, if it’s, if it’s patterns are way off, which we all, which we know. Right now, the current AI that exists dreams up, you know, and hallucinates things all the time. I mean, even when I listen, I’m just the most basic one that everyone’s going to understand is I type in a type of email. Right. And or I ask it something that I know is correct. And I ask it just to see if it and it goes, yeah, it’s this. And I’m like, no, it’s not. You’re wrong. Oh, I’m sorry. I’m an AI. I’m just learning. And it is, though. I mean, AI is fallible at the moment. And I don’t know if it won’t. I don’t know in a case if it actually will ever be not fallible. I think it will always be fallible. Because you’re draining. Because ultimately, we trained it on our data. We trained it on ourselves, right? And even though it can improve on that, right, it’s still improving on the data that we gave it to the editor in. So there’s a thought there.

Speaker 1 | 79:27.011

Yep. No, you’re right. I guess you’re always getting better too, right? There’s always new technology generated. And as you take that next step, that data that’s generated needs to get learned.

Speaker 0 | 79:39.922

and sucked into the system so yeah they’re always there i think there’s always going to be a level of uh being having it be fallible yep well i think that that kind of gives us a good uh um a good outlook on the future right uh at least makes us feel kind of nice that uh you’re always going to have to have a human look over uh the ai that’s trained on a human’s uh human’s data

Speaker 1 | 80:07.462

You know, I’m not going to open the can of worms of synthetic data. We’ll leave that one alone.

Speaker 0 | 80:11.485

Oh, no. Oh, no. Nerds. I’m Michael Moore. hosting this podcast for Dissecting Popular IT Nerds. I’ve been here with Brian Romano. Sorry, Dr. Brian Romano, Director of Technology Development at the Arthur G. Russell Company Incorporated. Brian, thank you so much for being on. It was an absolute pleasure and I learned a lot.

Speaker 1 | 80:38.147

I’d be too. Thank you very much for having me. It was great.

249- Automation and the Future: A Look at IT/OT and Control Systems with Brian Romano

Speaker 0 | 00:07.899

Hi, nerds. I’m Michael Moore, hosting this podcast for Dissecting Popular IT Nerds. I’m here with Brian Romano, Director of Technology Development at the Arthur G. Russell Company, Incorporated. Hi, Brian. How’s it going?

Speaker 1 | 00:21.824

Not too bad. How about yourself?

Speaker 0 | 00:23.445

I’m doing pretty good. I have to say that as I was trying to type your name… My hands got a little tired because Brian Romano is short, but all the letters after your name, AS, BS, MS, MBA, PhD, my fingers cramped up and I couldn’t keep going.

Speaker 1 | 00:47.541

Yep, that’s true. I do have, I have to say I have five college degrees. I started Early on in my career, 1980, I started as a junior in high school. I went to the University of Hartford and got an associate’s in 83 and rolled the credits over right away and started in the EE program. Kind of stopped short there. I have four kids. When my oldest daughter was approaching school age, being very competitive and always been an athlete my whole life, I couldn’t be beat. So I went back to school. and finished my bachelor’s, fell in love with, and I did it in a way that married with what I do for a living. So it flowed together nicely, got my bachelor’s, rolled into my master’s in applied computer science. And again, it was always a means to an end. I did it to figure out the next link in industry and what I needed to do to get there. So that was my master’s. Then I went for my MBA. In between all that, I owned a company for 16 years. So I wanted to be a better boss soon. figure out how things are. But the MBA also had a data analytics component, which has helped me focus on a lot of the things in today’s industry. And finally, PhD. I just got my oral dissertation, my defense, I’m sorry, my defense of my oral dissertation last Wednesday before Thanksgiving, passed that and got the official okie dokie on the PhD. Again, that’s a means to an end. Technology and innovation is the concentration on that.

Speaker 0 | 02:24.503

So really here, should I be calling you Dr. Brian Romano?

Speaker 1 | 02:29.606

You can, but you don’t have to.

Speaker 0 | 02:31.828

Come on. You took all that. You get to say it. Dr. Brian Romano. I love it. No, that’s fantastic. What a learned scholar you are. It’s time for our… our icebreaker segment we call Random Access Memories. I ask a question and you respond with the answer that comes to your head first. Your first question, if you could design a virtual reality world specifically for IT professionals, what features would you include?

Speaker 1 | 03:09.880

AI, for sure.

Speaker 0 | 03:13.063

Requirement now. Yeah,

Speaker 1 | 03:14.404

exactly. Let’s see. Boy, that’s a good one. There are so many hands-on things inside the IT world. How do you make it virtual to where you don’t have the hands-on side? I don’t know. You got me stumped on that one. Sorry.

Speaker 0 | 03:35.841

No, I like it. No, you got a good point, though. It’s actually a really good kind of a stumped question, right? Because you’re saying, okay, you’re creating a virtual reality world. IT professionals. And then you’re like, hey, let’s include all the things that IT would like. And all of a sudden, it’s like, you know what I would put in there? I’d put a cloud.

Speaker 1 | 03:58.428

Okay. Okay. It has to be, right? Yeah, exactly.

Speaker 0 | 04:04.091

We’ll add that. So now we’ve got a cloud and we’ve got yours as well. So we got something. So it’s a start, Brian. It’s a start. Maybe I’ll continue this question with other people and we’ll just keep adding to the list.

Speaker 1 | 04:19.736

There you go.

Speaker 0 | 04:20.056

All right. Here’s the second one. In a futuristic IT-themed movie, who would be the hero? Who would be the villain? And what technologies would they use to battle each other?

Speaker 1 | 04:37.230

Oh, the first thing that came to mind, I’m sorry to say, is Terminator.

Speaker 0 | 04:43.556

That’s a good one. Yeah. I’ll take that.

Speaker 1 | 04:46.317

Arnold over there being the bad guy. I mean, and it’s got the whole AI content part of it. You know, it’s so who would be the hero? Let’s see. They could either be. Matt Damon or Tom Cruise.

Speaker 0 | 05:01.278

Okay. All right. All right.

Speaker 1 | 05:03.959

Maybe Elon Musk.

Speaker 0 | 05:06.239

Oh, boy. You’re going to spark some comments for that.

Speaker 1 | 05:11.321

And what technology? Boy, that’s a good one. Again, it would be, first of all, futuristic. It would be time travel. You’d have to have that element part of it. Okay. And, I mean, in the real world, again, you would have the whole fiber optic led. networking and things like that. But also again, with the whole, I keep using the word AI and I apologize, but I mean, there’s so many different aspects. There’s the language part of it. There’s for the chat GPT, there’s the process part of it, the time series data there’s in part of my world, there’s image acquisition and processing for the AI. So there’s three different aspects of AI for there. So it’s all about math at that point, quite honestly. So, but.

Speaker 0 | 05:58.302

that’s the futuristic part wow holy that was a lot so um i’ve been giving you some loaded questions here i’ll give you an easier one all right this one this one’s easier uh if it processes had their own music genre what would it sound like smooth jazz you know i you know that’s funny because i was looking at it i was like i was i was back and forth i was like is it going to be like a techno is it going to be you know uh um just because i was all you know i was thinking earlier on i’m like earlier on it could be that um you know almost like techno when you had the aol connection right uh but no smooth jazz man it’s get

Speaker 1 | 06:43.076

kenny g up in here right now that’s not part of my personality i’m classic rock all the way but when you said that the first thing that came to my mind is is an it guy sitting at a computer with the headphones on Kind of mellow, just making things happen.

Speaker 0 | 06:58.023

But I like it. No, that’s good stuff. But, you know, I think you’re right. And I think I would just interject probably smooth jazz with the occasional, like, heavy metal, like, out of nowhere, right?

Speaker 1 | 07:13.354

Wiping people up.

Speaker 0 | 07:14.995

Yeah, right. And that would just happen right in the middle of the night.

Speaker 2 | 07:20.919

Hey, guys, this is Phil Howard, founder of Dissecting Popular IT Nerds. I just want to take a few minutes to address something. It has become fairly apparent, I’m sure all of you will agree, over the years, that slow vendor response, vendor response times, vendors in general, the average is mediocre. Support is mediocre. Mediocrity is the name of the game. Not only is this a risk to your network security, because I’ve seen vendors on numerous occasions share sensitive information, but there’s also a direct correlation to your budget and your company’s bottom line. Not to mention the sales reps that are trying to sell you and your CEO and your CFO on a daily basis. That causes a whole nother realm of problems that we don’t have time to address. Our back office program. at Dissecting Popular IT Nerds. We’ve put together specifically for IT leadership and it’s on a mission to eliminate this mediocrity. And the best part is that we’re doing this in a way that will not cost your IT department a dime. So if you’d like us to help you out, get better pricing, better support, and jump on pressing issues in minutes, not days, then contact us now so we can get on. a call with you and conduct a value discovery session where we find out what you have, why you have it, and where you want to go and how we can improve your life, your IT department, and your company’s bottom line. What you’re going to end up with is, number one, just faster support from partners who care about your organization’s uptime and bottom line. And because you’re going to be able to access our 1.2 billion in combined buying power, you’ll be able to benefit significantly from historical data. And on top of that, you’ll also benefit from the skills of hundreds of on-demand experts that we have working behind the scenes that are all attached to our back office support program. So if you’d like, again, none of this is ever going to cost you a dime. At the very least, it’s going to open your eyes to what’s possible. Let our back office team provide you the high-touch solutions and support that your IT team deserves so that you can stop calling. 1-800-GIL-POUND-SAN for support. Now, if you’re wondering, what does this apply to? This applies to your ISPs, your telecom providers, all your application providers, whether you’re a Microsoft shop or a Google shop, what you might be paying for AWS, even Azure, co-location space, any of those vendors that you’re paying a monthly bill to. we can help you with.

Speaker 1 | 10:04.113

Hey, it’s Greg,

Speaker 2 | 10:05.133

the Frenchman secretly managing the podcast Behind the Curtain. To request your one-on-one call, contact us at internet at popularit.net. And remember,

Speaker 1 | 10:14.278

it will never cost you a dime. Well, you know,

Speaker 0 | 10:17.680

you said something really interesting when you were back there. You said, you know, you apologize for using AI. And it’s funny because so many people use it now. And it’s come up as a recurring theme on this podcast. over and over and over. And to the point where people are like, I know, I keep using it. I know I keep using it. But, you know, I thought about this the other day, and I think I know why it keeps popping up, because it hasn’t really been classified and segmented yet. So, for instance, when we say AI, it’s such a broad subject. It’s a broad sweeping subject. And much like what we’re going to talk about today, which is… IT OT, which I have yet to completely understand this, you know, all of this piece. I was taught IT OT, IT of operational technology, essentially. And we’re going to get into that in a minute because I want to know more about the letters I’m saying. Also a recurring theme here. So but AI, such a broad subject. It hasn’t really been. applied, but you were even kind of talking about it right there. You were saying, well, you have AI in this piece, you have AI in this piece, you have AI on this. And that’s kind of, I think, why it keeps coming up because AI is touching so many different things that, and we haven’t yet classified it and put it to the different items that we have. In fact, I think today we’re probably going to touch on it specifically when we talk about ITOT and any fun, new little acronyms that I learned along the way. So, I think it’s okay. I think I’ve come to terms with, we can bring up AI as long as we’re not talking about it in a very generalized sense, as long as we’re taking it and attaching it to something so we can start to have those conversations and classify it.

Speaker 1 | 12:20.565

So,

Speaker 0 | 12:21.706

I want to quickly turn the page though on AI. We will get right back into it. I guarantee it. because I want to start on this ITOT, right? We had folks that don’t know, I have a quick discussion, and not a long discussion, and Brian will attest to this, just a quick discussion up front about, you know, kind of what we’re going to go over, just so we can go over the page. But he started talking to me about ITOT and also CSE, Control Systems Engineering, two very intersecting, uh um items and i know uh i’ve had uh actually a couple friends uh that have delved into uh control systems and you should see their houses they are this i it’s like listen man i have an alexa and uh and to get that thing to work i i’m always constantly having to reprogram things and and it and i looked at like the code that it was just to ring a doorbell like i mean it’s like if i ring my doorbell At 5 p.m., I want it to turn on five different lights, make them a specific color, run an LED program to make them do a design. Then I want the thermostat to turn in a certain spot. And then I want you to change. It was like, I don’t understand how you got this much time to program out your house. I don’t have that. So, Brian, bring us into the world of control systems. and help us understand this?

Speaker 1 | 13:58.050

Well, first of all, real quick, there’s a common, probably personality thread between your friends and myself. We all abide by the notion of he who dies with the most toys wins. So you look in my garage, look in my cellar. I am exactly what you just said. I’ve got Google everywhere. I can turn on my Christmas lights. I can, you name it. My house is wired for sound here.

Speaker 0 | 14:23.490

I don’t doubt it.

Speaker 1 | 14:25.252

Yeah. So ITOT, that’s where we’re going, right? Yep. All right. So information technology, everybody on the podcast here probably knows a good deal about that. And from the office side of things. Now I’ll back up a little bit. I owned a business from 1998 to 2013 before going to work for Arthur G. Russell. And in that time, somewhere around 2004, I had many customers of mine. I was a control systems engineering. what’s called system integration company. I took hardware and software, PLCs, programmable logic controllers, and put them all together to make things happen automatically on the factory floor. As you’re doing that, the customers start asking for reports. They start asking for screens. Somewhere between 2004 and 2007, I had several customers start asking me to connect the factory floor to the business system. And what that means is I want to be able to take a work order from an ERP, pass it down to the factory floor, have an operator pick what work order they need to work on. When they select it on the HMI, the human machine interface screen, when they select it, it takes the work order, takes all the recipe data from the work order and pushes it down to the process. And the PLC then starts to go to work on what it was asked to do. And as it’s now running, it’s gathering data saying how much current am I using on a mixer? What is the weight in my hopper? So all of the things that go along with the… the process parameters and other things are being pulled and brought to a database. Now, back in 2007, we started doing SQL server logging and reporting. We started pulling data from the factory floor, putting it in a place. My first database was, oh my God, it looked like an Excel flat file. I had no idea about relational tables and schemas and all that. That was one of the reasons I got my master’s degree. I wanted to know how to do that right. So, but I mean, customers were starting to ask for this. Okay, so here’s the, in my opinion, the birth of IT and OT. A lot of the things that I just talked about, your IT people are going, yeah, I get it. I’ve got a Cat5 or Cat6 based network. I got routers. I got switches. I’ve got, what I did is I put what I called operation servers on each plant floor that was responsible for interfacing. to a link table on something like a JD Edwards AS400 system. I’d pull the work orders down to a local table. I’d have print servers built into this server. My database was in here. So all of these are all IT things, right? But the OT side now is the operational technology. I’m taking all the information as the factory is working, as the things are being built and mixed and baked and whatever. And all that information is going into a database. Then what you can do is after the fact, when the process is done, you can say, how much material did I use today? How much of this particular type of material did I use? How many rejects did I have? How long did it take to make? How long was it before an operator acknowledged an alarm? All of those things are coming to light nowadays a lot more. But even back in 2007, customers were asking for this kind of stuff. Um, so the, the birth, as I put it, um, of that kind of stuff is now called industry 4.0, uh, is started for me as early as 2004. Give you a little bit of an, another anecdote, which is kind of cool. Somewhere around 2009, I think it was, I had just activated a cell card on my cell phone. Coincidentally that morning, one of my technicians and I were driving up. We were in New Hampshire at a customer’s facility starting up another machine. We’re on our way home and a customer who has a plant in Oregon called me up and said, hey, my laser printer is not etching on the bag. I’m sending the data, but it’s not getting there. Can you help me? My technician’s driving. I’ve got a new laptop with a new cell phone card in it, cell card. I open up the computer. Cybersecurity was a little thinner back then. logged on to their VPN that they gave me. They gave me access to certain areas. I logged on to the VPN. I went over from Connecticut to where I live to Oregon. And because it was a system I put in, I recognized what IP addresses should be what. I started pinging around for the laser. I started pinging around for the PLC. And I got to a point where I could segregate. There was a switch that was bad. So I’m doing this. driving 70 miles an hour, I’m sorry, 65 miles an hour down the highway.

Speaker 0 | 19:30.293

In the passenger seat.

Speaker 1 | 19:31.573

In the passenger seat, yes.

Speaker 0 | 19:32.794

In the passenger seat.

Speaker 1 | 19:35.675

And I got to a point where I’m on the line with the guy and I have my other cell phone, my cell phone, cell phone in my hand talking to him. And I tell him, there’s a little gray box on the bagger. Go open it up and see if there’s lights on it. He opens it up, there’s not. He. I said, go to your office. Do you have any like link six switches or anything like that? He goes, yeah, we got a spare one in a drawer. It’s like an eight port. I said, perfect. So he grabbed an extension cord, plugged the link six switch into the. um, into the extension cord, unplugged all the cat fives, plugged it in. I said, try it again. Wham started working. He was able to, uh, start running his product again, driving down the highway within 15 minutes. I solved the problem across the continent using a combination of ITOT structure.

Speaker 0 | 20:26.773

Wow. Wow. I, I mean, and that’s, it’s amazing. Uh, um, and that’s, and that was quite a while ago.

Speaker 1 | 20:34.675

Yes, correct.

Speaker 0 | 20:35.355

Yeah. That’s what makes the story pretty. pretty remarkable right because this is before because right now i mean we can do that but you know because you’ve got cloud and you can tie it in with you know you can tie it in and connect in over here and you can even connect the msp is you sorry you can even connect the um uh the you know on-premise systems uh right uh with the um you know with the systems uh that are in the cloud as well yeah but back then that wasn’t a thing you know You were creating your own way to get over there. And just the ability to be able to kind of see that, you know, connected in from so far away during that time. That is a pretty remarkable story. By the way, to catch everyone up, again, we’ve learned new things. We’ve got new letters. There’s PLC, programmable logic controllers. right and hmi human machine interface right these are these are new uh new ones to me um i did think i knew about knew the plc but um you know i haven’t worked with much of it to to know so this is pretty interesting stuff now let me ask you a question um you know i’ve been on it for quite a while but um you know i’ve heard of this stuff but i haven’t really put it together um and i’m just sitting here thinking you if i’m uh uh you know uh you know been around the block in it and various different things and even been uh and done some stuff in manufacturing um with erps that you know so if i haven’t heard of this stuff in some of the detail that that you’re describing um have other like uh is this a problem of other people not uh i mean Is this something that you can find in colleges?

Speaker 1 | 22:36.194

No, no. You hit on a hot button topic of mine. So everybody knows what a double E is, electrical engineer. They know what a chem E is. And in my industry and what I do for a living, it’s a blend of a couple of skill sets. It’s a blend of electrical engineering and computer science in terms of programming and networking and IT. So. It’s like two or three skill sets. I’m not saying we’re experts in every one, but we have to have knowledge of these at some professional level to be able to implement them. So you go into any of these programs, you go into a double E program and you’ll get the hardware aspect of it, the design of the hardware, and you’ll get some coding. And you may know what object-oriented code is, but maybe not how to implement it completely. Then you go to, or you go to computer science, computer engineering. and you’ll get a lot of the coding side, but you won’t understand the industrial hardware side. Then you got the whole ITOT thing again. And none of these programs, they may have a basic networking class, which will be good. Don’t get me wrong. That’s great. You need to have that. But all of these things together don’t exist. So anybody in my field, I do presentations around the country for Industry 4.0. And one of the slides brings up a workforce. shortage and a skills gap. And it’s something like if you look at the professional and engineering aspect, it’s something like 40 or 50% of the open jobs right now are in the manufacturing professional engineering aspect. And it’s because there’s a large skills gap. So what’s happening is as the baby boomers are retiring off and this acquired experiential skillset is leaving us, there’s not… any way to fill up the funnel, if you will. So I’ve gone to a couple of universities. I’m an advisor to universities. I teach at the University of Hartford. And I am trying to bring this automation skillset, this ITOT skillset, the things that kind of surround industry 4.0, trying to bring that to industry. So yeah, there’s not. And here’s the hard part. When you grew up in the 60s, 70s, 80s, and somebody said an industrial environment, the first thing that pops into your head is dirty, dank, and dangerous. And it’s dirty. I’m going to get dirty working in greasy pits and stuff like that. It’s dangerous. I might have… presses banging all over the place. And it’s, you know, dank and dreary. That is not the case at all. But that’s the view that a lot of people have, especially the mothers and fathers of people that grew up in the 60s, 70s and 80s, who now have kids going to college, and they want them to get a good job. They want them to utilize their skill sets and everything. And they don’t know what control systems engineering is. They don’t know what IT and OT is. They do know what EE is and again, KEMI and things like that. So the hardest part of all this, I hate to say it, marketing. We’ve got to figure out a way to take everything I’ve just talked about and bring this to light so that the parents out there will talk to them about their kids. The kids will go to the guidance counselors. The guidance counselors have to be knowledgeable of this stuff. in order to start the pipeline to be full and getting full with people that will replace people like me.

Speaker 0 | 26:18.308

Well, there’s so much packed into what you just said there. And wow. Let’s start with the fact that, and I think you hit on it, that some of the generation that is saying, hey, I want you to go out and get your… get your degree and stuff like that and then some you know and and they look at a a manufacturing job or an industrial job and they’re like well no no go into services go into you know something else um but you know having worked both right um they all they both come with their just interesting uh interesting pieces to them um manufacturing is uh um is very tangible. That’s one of the things I loved about manufacturing, right? There was a very tangible quality to it. Services, not so much, you know, but also the giddiness of walking on a floor and watching something get produced. I mean, it was so cool. And it was so neat. This is a thing that… I always thought it’s like I do IT and I watch a thing get made. Right. And part of what I’m doing helps do that. You know, that’s what to me was the most remarkable piece of it, because everyone has a piece in the cog and you’re actually building and making something, which is, you know, that right there. Why is why is that so hard to market?

Speaker 1 | 27:58.596

Oh, you’re absolutely right. It’s not. It’s an unknown. I mean, it’s just it’s a. I don’t want to say relatively new. It’s not really, but the merging of all these skill sets is relatively new. I mean, this kind of stuff has been around for a while. Going back to what you said, one of the classes I teach is automation systems. And some of the students, I walk up and I say, look, you’re going to be doing a lot of paperwork. You’re going to be doing a lot of design, a lot of programming, a lot of late nights getting this stuff debugged and working, but I’m going to promise you something. The minute you walk up and you press the start button and the machine does what it’s supposed to do, what you programmed it to do, and it starts running, there’s no better feeling in the world. And I’m a little bit of an advantage because I can cherry pick the students that I had in my class and have them come to work for me. So one of my better students I had, I asked him, I actually brought him in. He was teaching lab for me at the class. And we were in class one day and I said to him, tell the students what I told you about having the best feeling in the world. He kind of reiterated what I said. And I said, what’s your feeling on that? And he says, absolutely true. So what you said, Michael, is absolutely true. There’s no better feeling than watching something that you put all that effort in coming to life.

Speaker 0 | 29:24.181

And there’s also another piece to it, right? And I think the other piece is. we’ve come a long way in terms of automation. And man, I have seen some robots on these floors do things that, I mean, you just sit there and watch them. And as they swirl around and like arms, they have arms and grip things and put, and you’re like, how is that even? It’s such a graceful movement. It’s almost an art form to sit there and watch these things move in such a way. beautiful. It’s, dare I say, sexy, right? And that’s why I say I don’t understand why it’s hard to market it, except maybe because of the stigma. uh, the old industrial and the old manufacturing still exists, you know, in which, Hey, get yourself out of, uh, uh, you know, uh, a blue collar job and into a white collar job. Right. And that’s, but that’s not the case anymore. Right. That’s that really, that doesn’t really exist. There’s people that, uh, um, that make tangible things and there’s other people that sell things that are not tangible. And, uh, and. Those are essentially what you’re working in, one or the other. You’re either making something or you’re selling something that is vapor, right? But does something, right? But that’s what’s remarkable about it is this is actually a physical thing. And it is really, really, really, really fascinating to see all the different pieces. I get excited like I’m in school again. And I’m like, I’m in. elementary school, seeing something for the first time, I get excited when I walk onto a manufacturing floor because you just never know what you’re going to see.

Speaker 1 | 31:18.315

Yep. No, that’s absolutely true. The Made in America show brought some of that to life. I work for a company that does exactly that. We do automated assembly systems and packaging systems that if you need something put together at slow speed, at medium speed, at high speed, We take all the discrete components and put them together for you. And that takes robots. It takes servos. It takes PLCs, computers. I started putting computers, embedded computers on every machine with software to analyze data and things like that. So, yeah, absolutely true. You’re really kind of taking something and bringing something to life. Yeah.

Speaker 0 | 32:02.938

You had mentioned in there a little bit about. debugging, right? And, you know, anybody that has stayed up way, way, way past their required bedtime to code something and try to fix something because it won’t work because they keep debugging it and it’s still there and everything like that. There maybe is, I would have to say there is an art form to debugging. There is a science to debugging. I’m not good at it. I sit there and I essentially am just like, no, I’m just going to comment out this whole section and just start working this one spot and see if this works. Okay, now that works, great. All right, let’s go to the other one. And I just keep,

Speaker 1 | 32:54.666

you know,

Speaker 0 | 32:55.646

let me find the section that it’s dying at. Put flags everywhere and have it create a little… files so that I, or a little section so I know it got to one spot or another, but, um, it, to me, it feels, it feels random. It, it feels like it’s, um, it, like there’s not a, uh, um, a science to it probably because I’m not, uh, doing it correctly.

Speaker 1 | 33:17.778

You’re not used to doing it maybe.

Speaker 0 | 33:22.620

So I guess, tell us a little bit about debugging and, uh, and, and what the, um, you know, what, what surrounds the art form to it? What, We don’t what is the process? Because in the one thing I just want to kind of get clear on why I’m interested in this topic is because it’s the measure twice, cut once theory. Right. And I read an article that you had linked to. And in that article, I had talked about essentially making sure that the what you what you’re producing and I’m paraphrasing here, but what you’re. making producing or everything is actually meets the actual design requirements uh and and they’re linking those together which i also think when you back up for a minute and talk about project um just general project management um that’s always the goal right you know did we create did we perform and do the thing that that we we said we were going to do or did we scope creep and and just create something completely different right yeah and and

Speaker 1 | 34:29.188

The whole debug essence that you’re talking about. Yeah, you’re absolutely right. There was another article. I’m not sure if that’s the one you were talking about. There’s one that was just released a couple of weeks ago that they interviewed me talking about debug methods, coincidentally. And it was talking about, okay, and it might have been the same article, but we’re talking about starting off with

Speaker 0 | 34:48.984

URS. Is that an assembly?

Speaker 1 | 34:50.966

Yes, assembly magazine.

Speaker 0 | 34:51.907

Yeah, it was the same article.

Speaker 1 | 34:53.068

Okay. We start off with the URS like you talked about, user requirement, the spec.

Speaker 0 | 34:57.716

We got a new one. You got a new one,

Speaker 1 | 34:59.538

everybody.

Speaker 0 | 35:01.660

URS.

Speaker 1 | 35:02.561

URS. Yeah, acronyms are us. So you start off with that and that becomes your… your design guide of how, how things should flow. It will have in there, um, things like you will use this brand X, Y, Z. You won’t use brand, uh, W and Oh, by the way, you have to program in this methodology. Uh, you can’t program in this way. You can’t use flow charts or, or structured text. You have to use ladder logic and blah, blah, blah. And they, they kind of spell out the way that their plant runs. And you’ve got to adapt to it. So that’s kind of the beginning of everything. And once you go through the whole design aspect, there’s checks and balances that go along with that as well. The way that we do it right now is we’re trying to minimize the amount of errors, if you will, that make it to the floor where things are built. So I do a third eye set of eyes for checking. So somebody will design it. Somebody will draw it, two separate eyes. And then somebody who will compare. the bill of materials and the, the drawing, the design aspect, the third person will say, yep, everything matches good release to the floor. So three,

Speaker 0 | 36:16.412

that would be a person.

Speaker 1 | 36:19.375

Right now it does. Yes. I, I can see where you’re going. Cause yes. And in the future, there’s really a way you could, you could code that out for sure. Yes. And then from there, okay. Then somebody builds it, they wire it, they program it. And the programming again is, um, kind of a guided functionality from the customer number one and the needs of what the mechanical engineers have designed. We’ve got to animate it and automate it. So we write the code that animates the mechanical structures. And then from there, the electricians wire everything together. And then we go through, in that article, I talked about a systematic checklist where you go through, all right, here’s all my inputs and outputs. If I fire an output, Does the proper thing happen? Yes. A switch was made. Does that happen? Yes. It’s supposed to trigger an alarm that you program. Does the alarm happen? Yes. So you go through inputs, outputs, alarms, info messages. So systematically working your way through the design, through the programming to the point where now you bring back in other trades and the mechanical assembly guys that will actually now start working with you to bring the machine up to. production where you’re actually starting to make something. Just to kind of give everybody here listening a quick idea, if you were to take a ballpoint pen, a click pen that you have on your desk, and you were to take it and spin it apart and put all the basic components on your desk, you’ll probably end up with like six or seven components. The spring, the nib, the little clicky thing, the barrel on the bottom, the top, all of that. Now, what we do is we take all those discrete components and assemble them. in high throughput or low throughput or whatever the customer needs. If they want to do 10 a minute, we do it. We’ve gone up to 1800 a minute. And we do that by doing many at once. I might, every cycle of our machine may build something like 30 or 40 of this assembly. In this case, we’re talking about a pen. So all of these things are electromechanical. Now, going back to the whole ITOT thing. Everything kind of comes together here. PLCs are all Ethernet networked. All the touchscreen HMIs, they’re all networked. All the encoders that run the machines are Ethernet networked. So you kind of see now that there’s a level of IT technology, the whole I need to know how to put switches and routers and NAT devices on the floor so that it runs my automation equipment. And then I have to put a gateway on there that takes the data that the machine generates and brings it up through the double doors. And we’ll talk about that in a second. And brings it to the other side where people can do some work on the data. So there’s the kind of the combination of everything. It’s systematic, starting with the URS, working its way through a design specification, going through IO checks, debugging. And then. the whole debugging part of it that you talked about with software, you hit it on the head. I mean, you do it in modules and you bring each module online. If there’s a problem, you cut the problem in half and then work on the half that’s not working and keep doing that until you get to a point where you’ve isolated it and go from there.

Speaker 0 | 39:52.418

Yeah, but you said it much better. Yeah. That’s so incredibly interesting. It’s amazing because- You’re forced to have to sit there and make sure that all of the T’s are crossed, all the I’s are dotted. If you do not. it’s not going to work at all uh everything has to work um you know but it but it’s an interesting concept because you have to build that into your time right and i do feel like sometimes um uh when projects happen that don’t require um anything having to do with uh um operational technology or control systems uh you know um they don’t go through those rigorous tests You know, they don’t go through those. They just get it about 80 percent of the way there and we’ll fix it on the way. Right. But you don’t have that. You can’t do that. You can’t fix it on the way. I mean, you’re sending through, you know, how many how many at once. Right.

Speaker 1 | 40:56.276

Did you say you’re sending through 48 every cycle? Yeah. Yeah.

Speaker 0 | 41:00.940

You can’t fix something on the fly. That’s that’s running like that. I mean, you know, you’d have such a high error rate or it just wouldn’t work. So. So it’s actually pretty remarkable. You’re actually forced into this methodology that really everybody should be using. But, you know, the you know, the the requirements to do other types of, you know, I.T. work don’t match up with having to do it 100 percent, mainly because, you know, it’s about time. It’s about get it in, get in quickly. We’ll fix it as we’re going. Right. And you see this and that’s that’s infrastructure. Right. uh infrastructure is uh um build the road while it’s still still there you know and uh just build around it make another lane let them go this way while we fix this thing and then push them back so there’s a there’s an art form to infrastructure and uh and it happens to be having to do things uh um you know probably not the most prettiest way to get them done sometimes but they have to do it but oh wow to do this operational uh technology and control systems it you have to you really have to have all of your stuff planned out and have your tests and checks to be set to be able to do this correctly. And thank you for explaining debugging in such a beautiful manner. It actually sounds really, really good the way you did it. I’m adding that into my brain now, right? So, wow. There’s so much. You talked a few times about…

Speaker 1 | 42:40.484

industry 4.0 yes tell us a little bit about industry 4.0 all right so back like what happened to one two and three is five coming out like what’s happening there yeah um okay the quick primer industry 1.0 also known as the industrial revolution somewhere in the 1700s um things were mechanized eli whitney with the cotton gin samuel colt uh with the firearm things like that uh interchangeable parts mechanizing things. Then industry 2.0 later came where this was more of the Henry Fords of the world, where you took the idea of interchangeable parts and mass production, and you put them on conveyor systems, and then you start mass producing things in various plants and across plants. Industry 3.0 is where kind of everybody listening here, I’m guessing, grew up. 60s, 70s, 80s, 90s, 2000s, where you start… applying electronic controls and control systems and computers, the cyber physical, as it’s called, to the factory floor. Now you’re actually taking in generating data. During that period of industry 3.0, you generated data, were able to do reports, but that’s about the extent of what you could do. Here comes industry 4.0. It’s made up, it was first kind of devised. and announced at a Hanover Fair in Germany in 2011.

Speaker 0 | 44:05.685

That doesn’t surprise me.

Speaker 1 | 44:06.526

It’s kind of crazy.

Speaker 0 | 44:08.287

Yeah.

Speaker 1 | 44:10.267

And when I talked to you about what happened to me in 2007, that takes into account several segments, several aspects of Industry 4.0. There’s nine accepted pillars of Industry 4.0, augmented reality, systems integration, cloud computing, big data, industrial Internet of Things. 3D printing, cybersecurity. We’ll talk about that, I’m sure. Autonomous robots and modeling and simulation. Those are the nine accepted pillars that pretty much there are sometimes you’ll see 11 and things like that. But those are kind of the nine accepted pillars. This is literally taking everything. This is where IT and OT really breathe and live now because I’m taking stuff from the factory floor. I’m taking sensors, controllers. I’m taking rudimentary devices on the floor. If anybody here has heard of things like IO link, or even Wi-Fi or Bluetooth sensing, wireless sensing, bringing that to some gateway, getting that information from a gateway up through to a layer where it’s controlled. From there, it goes up to a layer where it’s kind of concentrated. And then from there, it brought up to the ERP system. Now, with all that information, the whole kind of goal of all this is to take and autonomously bring information from the ERP down to the factory floor. Take the information from the factory floor, feed it back to your planning stages. I need to produce this because I have low inventory here. And all this data interconnectivity is where this comes from. Then you bring it to a level where now if you’ve got it in an ERP system and you’re connected to both your value stream up and down where you’ve got your vendors and your customers, the customer can look at your system and tell how long something it might take to deliver or build or if it’s in stock. Some of all these components already exist. Don’t get me wrong. But now this whole thing is being a lot more autonomous, a lot more automated. Um, and all the things that I had mentioned, all of the nine segments, um, kind of come into effect. Augmented reality is used for support, um, being able to remotely support and keep the downtime down from a, for a machine systems integration is what we talked about, taking data, bringing it up and bringing it through the ERP up to the vendors and to the customers, cloud computing, being able to take your data from the factory floor, bring it up. Do some work on it in the cloud. Leverage the cloud structure. Leverage all the servers that are around the globe. Big data. Big data analytics. Talking AI again. That’s where a lot of this comes from. Back in the 80s, 90s, and 2000s, I could grab data. Couldn’t do anything with it. With the advent of low-cost, high-speed computers, I can now do a lot more with it. I can run it through AI engines. I’ve got all this data. I can train models. I can look at what that might mean for future stuff. And hopefully we come back to that in a second. Then there’s the IIoT, the Industrial Internet of Things. That’s taking sensors, adding sensors that bring not only information about key process indicating variables, KPIV, there’s another one for you, up to you go from there, you’re also looking at machine health. Why is the machine drawing more current than it used to? Why is it consuming more air? What is my bottleneck? Why is one of my stations running slow? All those sensors feed into that information. Then there’s 3D printing. 3D printing is not what you and I kind of think of. Well, to a degree it is. But again, it’s taking information from like an ERP saying, hey, I need 300 of Model XYZ made. This fleet of 3D printers over here are going to do that. So the 3D printers start working. Then there’s autonomous robots in the same vein. If I’ve got a fleet of robots, I can do quick tool changes. I can do expedited manners of changeover for new products. Cybersecurity, you can see that everything I’ve talked about now is taking data from the floor, running it through the enterprise up to the cloud. Guess what? I just opened up my enterprise to the cloud. Now I’ve got to have some level of cybersecurity to be able to protect my data and my hardware on the floor. And the last one is modeling and simulation. There’s a couple names for that. One of them is the digital twin. One way to look at that is if I design something in SOLIDWORKS or a mechanical set of software that represents the physics model of how a machine is built, and then I take my PLC programming and I take that and I marry those two together in a software package, I can literally do a digital twin of… that machine. I can run the machine offline on a computer while the real computer is off on the floor running. If somebody says, you know what, I want to make a change because I think I can get 3% more output out of this. They make the change in the computer. They change the code in the PLC in the computer, run it on the computer, prove it out, then bring it to the machine before the machine has to go down.

Speaker 0 | 49:36.108

It’s virtual modeling.

Speaker 1 | 49:38.009

Exactly. Yeah. I know I’ve talked a lot real quick, but that’s what Industry 4.0 is. Industry 5.0 is more humans working alongside the cyber physical part. People are starting to talk about it. It’s still a long way away. I don’t know if anybody’s done marketing or anything like that or read the book Crossing the Chasm, where they talk about the standard normal curve of marketing, where you have the early adopters, the early… the laggards and things like that. In a lot of these marketing things, you have this thing called a chasm where People aren’t quite jumping yet. You got the people, the early adopters who have invested a lot of money and want to do it because it’s cool. Once they’re successful, then you got the people jumping the chasm and starting to adopt it. We’re at, in my opinion, we’re at or just over the other side of the chasm and just starting to get industry 4.0 adopted in a more general sense.

Speaker 0 | 50:38.941

Industry

Speaker 1 | 50:41.002

4.0 or 5.0? 4.0. 4.0 is still in its infancy. It’s been developed. It’s been talked about. You’re still dealing with 60s, 70s machines on the floor. So there’s a lot of things that need to happen on the floor to really get a whole company on the bandwagon.

Speaker 0 | 50:59.771

In previous podcasts, we’ve actually touched on the cybersecurity concerns with the. an ability to upgrade some of the devices that have been there forever. Hey, we bought this device a long time ago. It does this one thing. It can still do the one thing, but it runs off Windows 3.1 and I can’t upgrade it. We joke, but there are some seriously old machines still running these things or not just old, proprietary. And only one person knows how to do it. And they’re off in, you know, vacationing and in a fun little place, you know, drinking martinis or something. I don’t know. This is what this is the reality of of what happens here. And and cybersecurity needs to be done at a level of when people are building things. Right. Because if you try to just interject it in afterwards, then you don’t get. uh um that seamless you don’t get that you know that seamless uh working workings of it so would you yeah because you’re you’re just you’re like oh yeah and then put put give it a cyber security package and move on i mean that it doesn’t really seem like that works too all too well that’s why that’s why uh when people code things and file things uh um cyber security is built in along with it right what does it need to do okay now now make sure you build that into the system uh and uh and it’s the same thing with when we build infrastructure as well, the same kind of concept when you build cloud. And so I want to ask about this because there are precise components to these things. We’re talking about the programmable logic controllers, right? What PLC, right? We’re talking about these PLCs. Great. I can protect the network. I can protect all this stuff. But let me get down to the actual PLC. Are people trying to hack that?

Speaker 1 | 53:08.465

Yes. Actually, there was some success as early as 2010, I think, somewhere around that area. There was a virus called Stuxnet, and it was aimed specifically at industrial components. And it was kind of a wake-up call for a lot of people. And to your point, it’s like, hey, I got cybersecurity on the IT side. I’ve got my servers protected. I don’t care what goes on on the floor. And, um… There’s a lot of things. We deal, our machines deal with production. And if it crashed, it would hurt a company’s revenue. It’s not going to kill anybody or make people starve or blow up things. It’s just going to stop making product. But this same thing, PLCs exist in gas plants, in power plants, you name it. So if you don’t protect those assets in a way that would… vitally stop people from penetrating it yeah you could have some serious serious concerns so yes cyber security is is something that’s i don’t want to say up and coming it’s here it has to be um believe uh one of the societies the organizations i belong to the international society of automation isa um they partnered with the iec uh and they’ve got a standard cyber security standard six four two six two four four three i think it is and it kind of dictates what this looks like from an industrial environment, how to put together your network. And they take the approach that they do, I guess you can call them blast zones. So you compartmentalize areas. So if something does go wrong, it’s compartmentalized. It’s not going to go free and wild. And that’s kind of the approach that’s taken with that.

Speaker 0 | 55:00.170

Interesting. You look at it. possible reasons why people wouldn’t want to implement this. And there’s a couple, right? I mean, we’ve already implemented it. Now we have to go back and replace PLCs and reprogram them. It’s a lot of work and all this stuff. But then on top of that, too, there’s an overhead to security, which could reduce production.

Speaker 1 | 55:26.913

Yes. Yeah. I mean, even to your point, if I had to take the time to put in to change out something that for the cybersecurity aspect, while I’m doing that, I’m not making production. So it’s kind of a twofold thing. You know, I’ve got the expense of the equipment that I have to put in, but I’m stopping. production while I’m doing it too. So yeah, there’s several reasons why you would want to do it. Yeah, there is overhead that goes along with it. Some of which is people being able to understand. Again, we’re harping on the IT-OT crossover where you have skill sets that cross, but there’s people that are involved with that, that understand this crossover and can support the equipment, both on the IT infrastructure and the OT side.

Speaker 0 | 56:11.583

So if… if I were able to jump in and get into a controller, right, could I crawl my way into the network at all? I mean, that’s the, yeah. So that’s the, and I believe it’s the case because you can crawl your way from the network into the controller, right? So, and you can exchange data with it. So anytime you can do exchange ones and zeros with something, you can hack it, right? So I guess- If that’s the case, then companies should worry about their data, right? They should start worrying about the controllers, these PLCs that are sitting out there that are not. Tell me a little bit, by the way, about HMI, the human machine interface, real quick.

Speaker 1 | 57:03.920

They’re essentially touchscreen computers, whether they’re PCs or embedded computers, but they’re essentially touchscreen devices. A lot of them have standardized software that you basically program displays, push buttons, dials, sliders. The graphical interface that people need to run a machine is what an HMI is. It can have a lot of recipe functionality where I can pull information. It can have data logging functionality. There’s a lot of things that go along with it, but it’s essentially the interface device, just like it says human machine interface. It allows a human to touch parts of the machine through this interface.

Speaker 0 | 57:47.247

So most of the times I’ve seen some things like this, they’ve been pretty locked down. Like this is, you know, you only hit these buttons to do these one things and it looks like they’re programmed specifically for the thing that they’re trying to do. Correct. However, someone has to program that, right? And that program can be altered.

Speaker 1 | 58:08.602

Correct.

Speaker 0 | 58:09.563

Right. When you get up to these HMIs, do a lot of them have logins or do they are just open and ready to do their thing?

Speaker 1 | 58:19.233

Depends on the year. No, quite literally, yes and no, depending on if it’s a standalone, if it’s an isolated island where the machine is on the floor by itself doing its own thing, there’s no need. At that point, you still have the ability to program. operator levels. I don’t want an operator changing set points, let’s say. So I will build in programming levels, operator levels, I’m sorry, password levels that allow different areas of some people can do maintenance on the machine. Some people can’t. Operators are only allowed to hit start and stop. So there’s different levels of that. Yeah. So to your point, if it’s an isolated standalone thing, you don’t necessarily need it. You would want to have those layers in there to be able to protect the machine. When you put it on a network, it becomes even more so. You want only certain people who understand the dangers, I guess, if you will, of going deep into the program to be able to do it. So, again, operators only have cursory levels of operation and control.

Speaker 0 | 59:31.658

So, really, the big piece is… whether or not these are connected into the network and exchanging data. Because if they are, then they can become an entry point, you know, where somebody can, and even remotely get to them, because you’ve already proven that that can be the case, especially if it’s connected to the cloud, you know, then there you go. If you can use that computer as an entry point, then you can use it to infiltrate the system and… and grab data and then extract the data out of the system again. And since PLCs seem to be very good at running. things, uh, in a specific order and doing things a certain way, it seems like a perfect way to even just hide a, uh, um, hide a program.

Speaker 1 | 60:22.212

Yep. You can certainly do that. Yep. Yeah. There, um, there’s going, going back to your, your point on going to the cloud, um, from a cybersecurity aspect, if you look at it, data should be flowing up to the cloud. And if you want to access the data from the cloud, you should be coming in from a layer that just allows you access to that data on the cloud. So you got outbound traffic coming from the PLC, from the factory floor, going up through to the cloud for analytics, for storage, things like that. And while the data’s up there, I can take a PC and then hit the data up in the cloud. No need for me to go down to the floor. So if I’m looking just from the data standpoint, and that’s where a lot of this configuration comes in. If I’m looking just from the data standpoint, part of the cybersecurity is… access levels, who’s touching what, where are things going. But I have opened a gateway. In my case, what I just said, it’s an outbound gateway, but that can be broke too, right? I mean, if somebody is good enough, they can figure out a way to do it. So yeah, there’s certainly a concern that you have to consider when you’re doing this.

Speaker 0 | 61:33.499

I think if we haven’t… connected these items to logging, centralized logging, with a back-end seam and a 24-7 sock sitting there watching what’s happening, there’s still a very big opening there from a threat standpoint, right? I mean, I really think from a manufacturing standpoint, folks ought to be thinking about some of the standard items that healthcare companies and financial companies do. You know, because a lot of those times, you know, you don’t have and this is the trouble, right, is that sometimes you have big manufacturing companies. That’s great. But other times you have small mom and pop manufacturing companies and they don’t have a lot of the resources to be able to implement a lot of this stuff, especially, you know, investing in a in a centralized logging seam and and sock in the back end. So to them, that seems like that’s a. that’s a tough order.

Speaker 1 | 62:40.298

Yeah. There’s industrial ways, quote unquote, to get around that if you want. I don’t want to say get around that. It’s not the right way. To implement it so that you can gain access to it securely. There are devices out there called, one brand that I happen to use is E1, E-W-O-N. It’s a company that makes industrial VPNs and it’s an outbound ping that just constantly pings a secure website. And- just says, I’m alive, I’m alive, I’m alive. And then only certain people have access and people who are granted access have access to that cloud site. And then what’ll happen is me, when I have my client on my side, I ping the cloud as well saying, okay, I’m here. I want to talk to IP address, blah, blah, blah. It goes across the network. We meet in the cloud. We negotiate in the cloud. Then I come down and there’s a couple other levels of security, but one of them is physical. I’ve got a key switch that will kill the enable circuit on the E1. I only can get access to that machine when somebody has asked me to gain access to that machine. So they walk up, turn a key switch. I have access to it. There’s outbound traffic from the E1 saying I’m alive. There’s no inbound at that point. We meet in the cloud, negotiate. My credentials are accepted. I now have a VPN tunnel to this device. And now I can gain access to my PLCs, my vision systems, my HMIs, all of that as if I was plugged into it at the machine itself.

Speaker 0 | 64:18.354

What I love about that implementation is it means that you have to use two different attack methods, which is one, you have to gain credentials and two, you have to use social engineering. So to be able to mix those two together to actually gain access, which is very difficult. So that’s I mean, it’s possible, but the harder you make, that’s the goal of of security is to be harder to implement than the other person. Right. Oh, this house has cameras. I’ll go to the next one. Right. So, you know, because I mean, you can never be fully secure, but you can you know, you can keep adding to your arsenal to make it harder and harder to get that. You can reduce that risk and mitigate that risk down. But yeah, I like the way that. And I like the way that that’s structured.

Speaker 1 | 65:05.842

Yeah, it works. We still have customers that say, nope, it’s connected to the internet. Nope, you can’t have it. So sometimes we have a couple of different ways around that. One of them is we ask the customer, keep a coil of Cat6 to your internet device nearby. If you need us, plug it in. And then when you’re done, unplug it. Then you never have to worry about it. No physical connection, you’re done. The other way is these same devices have cell data. So I can meet in the cloud. I don’t have to hit any of their physical internet infrastructure in their plant. I can just come in over the cell and hit the machine directly without hitting their network. So there’s a couple, three ways to kind of implement this and keep it secure.

Speaker 0 | 65:52.194

Yeah, very nice. Very nice. So we have, we’ve gone over a lot of things so far, and we’ve learned a lot of things. We’ve learned ITOT, CSE, PLC. HMI, URS, KPIV. I’m sure I lost a few in there, right? I’ve started a brand new dictionary after this call.

Speaker 2 | 66:18.636

At Dissecting Popular IT Nerds, we expect to win and we expect our IT directors to win. And one of those areas where we know that we can help you win is internet service providers. As an IT director tasked with managing internet connectivity, few vendor relationships can prove more painfully frustrating. than the one with your internet service provider. The array of challenges seems never ending from unreliable uptime and insufficient bandwidth to poor customer service and hidden fees. It’s like getting stuck in rush hour traffic. Dealing with ISPs can try one’s patience even on the best of days. So whether you are managing one location or a hundred locations, our back office support team and vendor partners are the best in the industry. And the best part about this is that we can help you. None of this will ever cost you a dime due to the partnership and the sponsors that we have behind the scenes of dissecting popular IT nerds. Let us show you how we can manage away the mediocrity and hit it out of the park. We start by mapping all of the available fiber routes and we use our 1.2 billion in combined customer buying power in massive economy of scale to map all of your locations, to overcome construction fees, to use. industry historical data, to encourage providers to compete for the lowest possible pricing, to negotiate the lowest rates guaranteed, and to provide fast response times in hours, not days. And we leverage aggregators and wholesale relationship to ensure you get the best possible pricing available in the marketplace. And on top of all of this, you get proactive network monitoring and proactive alerts so that you’re not left calling 1-800-GO-POUND-SAN to enter in a ticket number and wonder, why is my internet connection down? In short… We are the partner that you have always wanted, who understands your needs, your frustrations, and knows what you need without you having to ask. So, we’re still human, but we are some of the best, and we aim to win. This all starts with a value discovery call where we find out what you have, why you have it, and what’s on your roadmap. All you need to do is email internet at popularit.net and say, I want help managing all of my internet garbage. Please make my life easier and we’ll get right on it for you. Have a wonderful day.

Speaker 0 | 68:38.459

Let’s go ahead and start our IT crystal ball in this segment. And when we do this, normally I kind of free for, sometimes I’ll be specific and kind of talk about the future of IT. And sometimes I just kind of free form, let people go run wild and see what their imagination comes up with. But we… sort of kind of uh already uh figured out i think uh where this one’s going because um you talked about and and we we we’ve touched on it a few times when we said data analytics right and in through the process we’ve kind of sprinkled it through um ai uh is data analytics gone wild right i mean it it is what happens when you can just you know, give a bunch of data to something that can actually make sense out of it, right? And actually has enough of a pattern recognition. That’s where we are right now, right? With AGI, right? This is a pattern recognition, you know, where basically we have figured out that when people do these things, that these are the next things that come so that we’re going to fill it out. with that information. And we’re at the lowest spot of AI right at the moment. It’s like,

Speaker 1 | 70:06.729

you know,

Speaker 0 | 70:07.730

the lowest you can possibly be, which is just scary at just how good this is and how fast it has exploded. And it touches so many things. So… We talked about all of these pieces that make up ITOT and CSE, and we talked about implementing them in. And then we specifically kind of, we touched a little bit on data analytics, but we didn’t really dabble completely into it. And that was by design because I knew that this end part of this podcast was going to specifically hover around the future of IT. with data analytics powered by AI, right? That is essentially where we’re going to go with this thing. And I think you and I are probably in agreement, right?

Speaker 1 | 71:00.540

Absolutely. Yep. Yep.

Speaker 0 | 71:02.941

I’m just, Brian, I’m just going to let you talk because I have no idea. I have ideas about where this could possibly go and I’ll interject when we do it, but I’m going to let you just start going because I just want to see where we go with this.

Speaker 1 | 71:19.567

Okay. Well, there’s already some predetermined paths. There’s already some software out there. There’s already ways and methods to do this. But a lot more data needs to be collected and characterized, imputed, all of that cleaned up. Now, going back to what you had said, you hit it on the head. Predictive analytics, predictive and preventative maintenance. So imagine you’re in a world where… All of a sudden, you get a little alert on your car that says, within the next two weeks, you’re going to lose the bearing on your right-hand tire. You’re still able to run. You got no issues. But now you can make an appointment. You can order the part. You can bring it to the dealer. You can get a rental car. All on your time. You’ve got two weeks notice to be able to do this. That’s all driven by data, data analytics, and the AI side of things. So there’s software out there, there’s methods out there to do exactly that. We had talked about the whole industry 4.0 thing, taking data from the factory floor, taking machine data, taking KPIV, key processes indicating variables, bringing those in, flooding a repository full of data. And then something happens on your machine one day, you lose a motor. There’s all sorts of data that surrounds that failure. There’s temperature, there’s flow, there’s pressure, there’s photo eyes that might have temperature sensors built into them for temperature compensation. And you happen to be reading that. There’s vibration analysis sensors. All of these things are kind of built in and you’re sucking all this data up and then you put it against a trigger. You say on December 1st, 2023, I had a bearing or a motor failure. What were the aspects of the data around that? that might indicate that failure. And now you have a training set of data. Everything that you’ve had in your past becomes now part of your historical training data. Now, in the future, you’re running fat, dumb, and happy, just like the car scenario. And the AI engine’s sitting there now looking at the data real time, and it’s looking for those same triggers, that same data signature, and says, last time I saw… The photo eye temperature rise by three degrees. And I saw the vibration go up to 73 Hertz. And I saw a little bit more consumption. Anyway, you get the idea. When all of these variables come close or match to the limits of the last one, I can do that far enough in advance if I’ve got all the data that led up to that event and put up a flag well in time to say, Mr. Operator, you’re a… about to have a failure, you’ve got an hour a day, a year, a month, whatever that happens to be based on historical data. You say, okay, Mr. Maintenance Man, I need a motor that I’m going to have a failure. I’m going to need a motor. Do you have one in stock? If not, please order one. They order the motor. They have it available. You now gracefully bring the machine down. No downtime, no actual catastrophic failure that’s happened. I haven’t crashed two things together. I’m bringing the machine down gracefully. I schedule that in. Maybe there’s other maintenance things I need to do. I schedule it. Then I fix the thing, bring it back online, bring it back up. No catastrophic failures, minimal amount of downtime. This is all surrounded around this whole predictive and preventative maintenance that’s driven by AI engines. And it’s there today. It’s not used a lot because the… The data’s not quite there yet. There’s not a lot of people that have clean data. They’ve got disparate data all over the place and different. formats, different schemas, different, and somebody’s got to bring all that together, clean it up, characterize it, and then put it in a method in a way that the AI engines, I use the word engine, AI engines can look at it.

Speaker 0 | 75:36.334

I think you summed that up really well. And there’s kind of a couple pieces that you can take from that. So the first one is, if you’re taking all this data, and you’re logging it, to be able to do all of these amazing automations and preventative work and kind of alert to things that haven’t happened yet, then essentially you’re also take that data and throw it against a cybersecurity framework so you can monitor it as well, right? I mean, so the great part is this AI requirement, which is I need data, right? and is also a business requirement, I need data, also meshes with the cybersecurity requirement of I need data,

Speaker 1 | 76:26.246

right? Exactly.

Speaker 0 | 76:27.766

And I need good data. That’s the other thing you mentioned. And this is a constant problem in a lot of organizations because data is entered by humans. And we use computers to make humans enter things the same way every single time. Otherwise, you will have, you know, items that are entering in completely different orders and in different spots. I mean, the idea for good data analytics is to make people enter data in the same way every single time because not having clean data breaks everything. It breaks systems. In fact, you know, many attack vectors are used based on injecting. bad data into systems, right? So this is such an interesting problem. And I actually wonder now if AI can help us even cleanse that data appropriately, which is kind of a question of, do we want that to happen? Yeah.

Speaker 1 | 77:39.828

Is that the trip? Is that the edge? Right?

Speaker 0 | 77:43.071

Because if we’re using AI to cleanse the data, um, you know, that humans entered, right? Now you’re, now you’re like, okay, we’re going to clean the data, we’re going to collect it, we’re going to clean it, and then we’re going to predict on it. And then now you’re going, okay, well, now I’m, I’ve put all my eggs in one basket. Let’s hope it doesn’t, uh, let’s hope that basket’s, uh, sturdy because if it, if it’s not giving me the right, if not cleaning the data correctly, if it’s not, uh, um, predicting the data correctly, if it’s, if it’s patterns are way off, which we all, which we know. Right now, the current AI that exists dreams up, you know, and hallucinates things all the time. I mean, even when I listen, I’m just the most basic one that everyone’s going to understand is I type in a type of email. Right. And or I ask it something that I know is correct. And I ask it just to see if it and it goes, yeah, it’s this. And I’m like, no, it’s not. You’re wrong. Oh, I’m sorry. I’m an AI. I’m just learning. And it is, though. I mean, AI is fallible at the moment. And I don’t know if it won’t. I don’t know in a case if it actually will ever be not fallible. I think it will always be fallible. Because you’re draining. Because ultimately, we trained it on our data. We trained it on ourselves, right? And even though it can improve on that, right, it’s still improving on the data that we gave it to the editor in. So there’s a thought there.

Speaker 1 | 79:27.011

Yep. No, you’re right. I guess you’re always getting better too, right? There’s always new technology generated. And as you take that next step, that data that’s generated needs to get learned.

Speaker 0 | 79:39.922

and sucked into the system so yeah they’re always there i think there’s always going to be a level of uh being having it be fallible yep well i think that that kind of gives us a good uh um a good outlook on the future right uh at least makes us feel kind of nice that uh you’re always going to have to have a human look over uh the ai that’s trained on a human’s uh human’s data

Speaker 1 | 80:07.462

You know, I’m not going to open the can of worms of synthetic data. We’ll leave that one alone.

Speaker 0 | 80:11.485

Oh, no. Oh, no. Nerds. I’m Michael Moore. hosting this podcast for Dissecting Popular IT Nerds. I’ve been here with Brian Romano. Sorry, Dr. Brian Romano, Director of Technology Development at the Arthur G. Russell Company Incorporated. Brian, thank you so much for being on. It was an absolute pleasure and I learned a lot.

Speaker 1 | 80:38.147

I’d be too. Thank you very much for having me. It was great.

Share This Episode On:

HOSTED BY PHIL HOWARD

Dissecting Popular IT Nerds Podcast

Weekly strategic insights from technology executives who understand your challenges

Are You The Nerd We're Looking For?

ATTENTION IT EXECUTIVES: Your advice and unique stories are invaluable to us. Help us by taking this quiz. You’ll gain recognition good for your career and you’ll contribute value to your fellow IT peers.

QR Code