Speaker 0 | 00:00.500

Welcome back to today’s episode of Dissecting Popular IT Nerds. I’m your host, Doug Kameen, and today I’m talking with Ian Johnson, the head engineer of Fullstack at BizScout. Welcome to the show, Ian.

Speaker 1 | 00:12.123

Thanks, Doug.

Speaker 0 | 00:13.963

So, Ian, BizScout, you’re a startup. You’re working in the startup space, but your history is not in the startup world here. Can you tell us a little bit about BizScout, what you’re doing now, and anything you want to share? I’ll just open the floor up to you.

Speaker 1 | 00:27.904

Yeah, thanks. Yeah, I think we could probably work backwards. So much like other technology trends, things sometimes become full circle. And my career has kind of done that as well. So I’ve worked in enterprise development. I’ve worked in fintech. I’ve worked in nonprofit, but I started in a smaller sort of implementation background with my own company. So right now what I’m doing is I’m working with Node.js and a small team of developers, and we’re using Node, Next.js, and a couple of other technologies to quickly… deliver software for an aggregator, which is basically something like what you would see in biz by cell or, you know, online marketplaces that have businesses for sale. So the, so the goal of our platform is to sort of put buyers in, in, in touch with sellers and then also have brokers in the mix somewhere so that they can make their buck off of whatever it might be. And so the founder is really prominent on social media. And so she’s done a lot of things that bring people together to to allow them to buy businesses. And so her goal is to make sure that people that, you know, the people that you wouldn’t normally think would own a business are capable of buying businesses. So that’s the goal of the platform.

Speaker 0 | 01:36.581

So people like me, like I could go buy a business at Bisco.

Speaker 1 | 01:39.463

Absolutely. And you could probably join our Slack space and talk to her directly because she’s very engaging. Or you could go through the site and look at the marketplace and put yourself out there as well. So that’s the goal. Anyway, we’ve been working pretty hard to make sure that It’s available to everybody. And there’s been some speed bumps. Obviously, there’s always going to be speed bumps when it comes to things like this.

Speaker 0 | 02:02.356

The startup.

Speaker 1 | 02:03.717

Exactly. You have funding. You have timelines. You have people that are expecting certain things. So that’s what I’m working on now. But if we were to wind the clock back and just go all the way back and just think about where all this started in Austin, Texas around 1997 when you and I were riding bikes right prior to that.

Speaker 0 | 02:24.751

Right. So for the listeners, Ian and I have known each other for a long time. We grew up in Austin, Texas together, and we’ve known each other since we were like 12. But we don’t always, we live at different ends of the country and only speak very periodically nowadays. So it’s great to have you on the show here.

Speaker 1 | 02:41.320

It’s good to talk to you, Doug. We even got in a fight on the sport court one time, I remember. Doug’s dad worked for IBM. long ago and I had the privilege of knowing a few people that were into technology that got me really started with computers back in those days and so really what I was doing was chatting on AOL and Prodigy and those those modem-based systems back in the day and just trying to understand like where I was in the world and just you know growing up and we were on we were on modems we’re on dial-up modems right so we had 2400 baud modems and then I remember a distinct conversation with Doug at one point We were trying to send a file over a 2400 modem and he said something. Man, if we just had that 14.4 we could just leave it. It wouldn’t take any time at all.

Speaker 0 | 03:26.340

The world would be our oyster, man.

Speaker 1 | 03:28.081

I don’t remember what we were sending. It was probably some kind of EXE that had Pamela Anderson in it. I don’t remember what it was. But I do know that when we went back to Best Buy I took the 2400 modem back in its packaging and returned it and got the 14.4 for the same price because they couldn’t tell the difference. And it was a game changer. So we went from chatting on these boards, BBSs, things like that, to cable modem. And the first software that I wrote was a spam operation that essentially would log into AOL and find active users and send them email. And I made probably, I don’t know, anywhere from $5,000 to $10,000 a month doing that for a period of time while I was younger. And it was a blessing and a curse because I could have probably started a legitimate business, but I was doing spam. And I just think, you know, the… the money kind of sidetracked me away from what I could have been doing, which was building web applications. So this was a desktop application. And once Time Warner shut us down, I went back into building web apps. And web apps was sort of like, I really enjoyed it. Just the idea of deploying to browsers. And if you think about your compile time on a desktop, you’ve got a runtime that runs on a desktop. Well, now you’ve got a runtime that’s in distributed browsers, and you’ve got to cater to. any number of them, Chrome, Netscape, all these different ones. And that to me was a real challenge. It’s gotten a lot easier these days. There are packages and frameworks that allow you to do things very quickly and easily. But back then it was really a challenge. So I got away from that and went into working sort of in e-commerce. And I got my first development job, I think, in… 2006 after all that settled down and i moved to mississippi and i worked for somebody called excuse me a company called fnc we did loan origination software and loan servicing and i to you know interact with different lender banks wachovia centrust all that and guess what right around that time was the big crash and we saw all of those banks kind of fall off and disappear and also can consolidate so some of those brands people don’t really No, the Bank of America and a couple of the other ones were kind of, they had their hands in different things. And so when you talk about the CDOs and everyone’s seen the movie with the CDOs and all that, you can see the technical side of these banks asking for requests that would literally just get thrown away because our contract with the banks was to make as many updates as they wanted. And if they wanted to throw the updates away, they could. There was no cost for that. So they were spitballing whatever they wanted at us. And it gave us, you know, the ability to code quickly. do what they wanted and just watch things that didn’t work fall away. And to me, I think that was a huge value because there are many teams, many engineering teams that don’t follow that same mantra, I guess. And what you’ll see is, you know, they dictate specific requirements and it’s a hard line and there’s no flexibility. And what I really missed about that was the flexibility aspect of it. It’s like, if this doesn’t work, it’s just going to go away. So went through all of that, came back to Austin. I decided, look, I can’t live in Mississippi forever. Look, I loved Oxford, Mississippi. Great, beautiful girls. awesome place. Really cool tailgating. Sorry about the cursing. I knew that was going to happen.

Speaker 0 | 06:35.729

Yeah, that’s fine. Our listeners will get over it. Well,

Speaker 1 | 06:39.712

hey, it was a great time for me. And so I came back in 2010 and worked for another loan servicing company here in Austin owned by a guy named George Starry called Mortgage Dashboard. And we were operating as a startup. And I said, I talked about full circle, right? And so this whole I would consider that to be my baseline, not necessarily the work in Mississippi. But I started there, went from there to Volusion, which was one of my favorite places to work. Volusion is an e-commerce competitor for Shopify. We had 2,000 store merchants. I learned a lot of things very, very quickly from the developers that I worked with there. They came from Simi Valley, California. And we all worked in the same area. You had situations where you would break something and become the hero to fix your own bugs. But then you had also situations where you needed to build something new. And so we produced something that had a patent. It was, I don’t know if you’re familiar with PCI compliance. Oh, yeah.

Speaker 0 | 07:37.363

So PCI compliance. I’m on the board of a, I’m the chair of the audit board for a large credit union based up here in our area. So, you know, PCI compliance and all those types of things, those are all part of the part and parcel of the day.

Speaker 1 | 07:50.534

Okay. So you know about storing card information at rest and all those different. And so we were violating one of their principles by storing card information at rest. It was secure to us, but it was secure to them. And they came in like the mafia. And here’s what’s funny. You pay them a certain amount of money to, I guess, audit you. And all they do is fine you. They don’t really, they can either approve you or take more money from you. At least at that time, that’s what they did. So what we did was split the card numbers in half and we created a patent around this. So part of the card number would be stored in one database. The other part of the card number would be stored elsewhere. And there would be a knots or a ID that would, you know, form the card number at the time of checkout. And so the one page checkout was what I owned as a developer. I saw that happen. And when the PCI people came and wanted to audit everything, they said, where are the card numbers? Well, one’s over here and one’s over there. They’re not stored at rest anymore because they’re not card numbers. They’re separate. Well, they’ve since changed all of that. And I’m pretty sure that we probably can’t get by by doing those sorts of things anymore. But we’re coming up with ways to just. move quickly and stay with the industry. So we did all of that, integrated with 10 different payment APIs, 10 top ones, and put a facade on top of it. I think it was Chase Payment Tech, Skipjack, Authorize.net, a few of the other ones, and that was one of the biggest challenges I think I’ve ever faced as a developer, just because good God, those things are so bad. Those payment gateways are just really bad. So not spending too much time on that, kind of got away from working in the e-commerce space and moved on, went into a nonprofit. I worked for the Michael and Susan Dell Foundation for about three years directly with Michael and Susan Dell on their grant making system. They were two of the major logins on our system. So we always had to have uptime. We had to make sure that we had automation and availability. And whenever we saw one of the golden users, which was Michael or Susan Dell pop into the system, we got alerts. They’re in there doing something.

Speaker 0 | 09:49.891

Everybody, all the screens come up and everybody’s making sure that this system is running. Exactly. And running like a top.

Speaker 1 | 09:56.374

And it’s funny because I remember one specific case where Susan had gone to Hawaii. And she was on a machine that hadn’t been updated in like five years because it was on some property that they didn’t ever go to. And it had an old, old version of the browser. So when she logged into the grant making system, it didn’t work properly. So they have a tech team that’s part of the family. And then the next layer is us. And then the next layer of that is my management. And then it comes on all the way down to me. So I would get like this game of telephone where it’s like, hey, Susan can’t log into Internet Explorer. It’s like, okay, well, what version of Internet Explorer is it? And I would wait. And it would have layers of people, and then eventually it would come back. It looks like it’s IE6. And I’d be like, well, that’s your problem. And so, you know, it would go back, and I would have to wait and wait. And finally, we figured out that it was just a really, really old device and that we had to send someone from her personal tech team to go update everything, and once they did, it worked. But that was the kind of thing. And this was at like 5 in the morning, by the way. I don’t know what. why it was five in the morning, what they were doing, but we had to have full availability. So I built a lot of automation around, like, since we had a small team, I would take the production database and make sure it was updated down into low environments via a lot of scripting and things like that. And so working with a small team like that, you want to have a replica of your prod environments that you can actually code against it. And I learned a lot about making sure that I had a mirror from my production environment down to my lower environments. And every morning, When something new showed up, I would just run a few tests and I had, I had unit tests and things, and I would, I would smoke things out by myself. Just like, okay, we, we released a product and we’ve got a little bit of an issue. I don’t know how that got by our test before, but now we found it and let’s fix it. So I learned a lot about that in that regard and became a lead there, ended up leaving and went to Charles Schwab. Charles Schwab, let’s just, I just want to make this a short conversation. Right place to work, but you got to be on the right team. That was the place where you learn that one little mistake can cost hundreds of thousands of dollars in no time at all. Hundreds of dollars, if not millions. And so I learned to work very, very quickly, work with product. And I learned the value of what product wants to have delivered versus what you’re trying to do as an engineer. Because sometimes you can make an engineering environment your, I guess, mad scientist layer. You can start to do things that you think are something fun for you to do, but doesn’t necessarily translate to. uh, what the business wants. And, and so that’s where I really learned like quick, quick, fast and efficient is the best way to do measure twice caught once. And that’s where I learned that. And I went from senior staff engineer to lead engineer. I actually left and went to Dell for a period of time. They brought me back. They said, the SAC is falling apart. Please come back. I came back and then became a director. And once I became a director at Schwab, I started to learn another valuable lesson. And that was when you take your hands off the code and you take your hands off of anything. Yep. You start to become more of a voice than an influence. And it’s harder to what I always say is like teach a man to fish. Right. You know what I mean? Don’t don’t tell them, you know, teach a man to fish. And and I had a real problem with that once I got to that level. And I felt a hard time.

Speaker 0 | 13:12.533

That’s a hard transition to make. Like is people move when you move between one your space of of being a hands on leader to being. a leader of influence. You say you have a voice, but you don’t necessarily have hands on the keyboard anymore in the same way. So you have to be able to effectively lobby the people who have their hands on the keyboards and make them want to do the thing that you want them to do. I mean, you can always order them to do it, but that may not work out in your favor either.

Speaker 1 | 13:42.209

That’s absolutely right. And where do you draw the line, right? You can’t micromanage people. First of all, I was an IC, so I wasn’t a manager. I was an IC the whole time. You can show them what you think that things should, how things should turn out or what you think they should do. Or you can even make a group decision. Now, what happens when a week later, the group decision that you thought you made was not the same thing and they implement something different? So you come back and you say, hey, I thought we decided on this. It’s all written down plainly in this document that we all agreed on and you wrote something else. Now, what do you do in that scenario? The time has already been wasted. You’ve already trusted the people around you that you think should be implementing the software that is. good for the business, the stockholders, the shareholders, everybody that, you know, anybody that is, has a stake in this. And it’s, it’s painful to watch sometimes now on good teams, that doesn’t happen. And I think good engineering teams are very, very rare because there’s one factor, there’s one factor and it’s just trust and respect of each other. Um, it’s just, you just have to trust each other. And if you can’t trust your team, you know, there’s Any level of micromanagement is going to come death by a thousand cuts, and the team will disintegrate. And I’ve seen that happen as well. I saw it happen at Schwab. I saw it happen after that. I saw it specifically happen at Q2 Software. So after I left Schwab, I went to Q2. Q2 is what’s called ATG, which was an advanced technology group, and reporting to directors and SVPs. All I worked with was engineers because we were like the SWAT team of different teams on Helix. Helix did 50 million transactions a day with Visa and ran mostly on a SQL Server database platform with some.NET Core. And we were doing some containerization and trying to rewrite a jobs platform, which I had a lot of input on and hands-on input on. But the hardest thing to deal with there was the engineers. And I hadn’t seen it. Like, I had a lot of trust at Schwab. the varying levels of trust but you know what i’m saying when i got to q2 as a newer person i don’t know what it is what do you have to do to prove to these people like do we have to go out and and and and and beat each other with sticks to prove that we’re actually you know worthy of our positions because that’s what ended up happening at that place we started to kind of fall into this like i don’t trust you i don’t know you this and that everybody was pretty new and for lack of a better word we were measuring uh things so anyhow that being said i couldn’t i couldn’t handle it it was two years i had hundreds of thousands of dollars in stock i had a good financial position other people were really uh golden handcuffed the same way i was i could tell they were miserable everybody was miserable and i said i gotta get out here and so i went to this startup so that brings me back to the full circle right back in the startup space back to the startup space I said, you know what, this is a place where I can leverage my influence and be heard and be seen and actually just have hands on, no blockers, get things out quick and solve problems.

Speaker 0 | 16:45.244

There’s a couple things I think about and I appreciate you sharing that journey. So a couple things that came to mind for me. One is you talked about the teams and how the differences in teams, the effectiveness of teams. Where I work, we just went through an exercise. We have a. uh, an executive coach who helps us and kind of, you know, help steer us for the last year or so, uh, which has really been helpful for us. Uh, but one of the things, the exercises that we did recently was we walked through rating in understanding that the ideas of the, I guess I call it the team performance curve. So this is a pretty common chart where, uh, and of course, we’re on an audio podcast, so I’m just going to have to verbally describe it, but on your, your bottom access, you’ve got the, the effectiveness of the team and then the the vertical axis is the performance impact of the team so you start starting on the left you start with like a working group and and as the team effectiveness declines it gets like a pseudo team and then as it starts kind of rounding that bend and you know coalescing it becomes a potential team uh and but only you know in the performance impact is going up and you get to a real what they call a real team and then finally a high performing team and like to me it sounds like what you’ve experienced is like all aspects of that graph. You’ve been on places where it’s a pseudo team. If you’re a team in name only and the people are, they’re not in it as a group, they’re in it for themselves. And there’s a lot of fighting and challenges going on and stuff like that. But you’ve experienced the other side of that, of being on either a real team or a high performing team where you’re really seeing the difference. And the biggest challenge in that space that I see is that once you’ve been on one of those high performing teams, It hurts so much more to be on the pseudo team.

Speaker 1 | 18:31.812

That’s absolutely right. That’s why it’s so painful when you see it happen again and again. So you don’t want to be the chicken little. You don’t want to be the one that says the sky is falling in front of people and say, what you’re doing is wrong. It’s going to fail. But there are a lot of people out there, including managers, especially managers that don’t listen to that type of input and they just wait to fail and they’ll actually learn from their own mistakes. So you can say that the decisions we’re making right now are going to be a problem and you try to fix it. Well, they don’t know that you’re actually. trying to help them. They just think that you’re someone that’s calling out issues that may not exist. And then once they realize that you were correct, now they have some kind of a vendetta against you because you’re the one that called it out before and didn’t do anything about it. But it’s not that I didn’t do anything about it. You’re the one that’s making the decisions. I’m trying to give you the input to make those decisions. So it’s a really tough balance. I think that what you’re getting at is what does it take to build a high-performing team? And I think I watched a video with Steve Jobs not too long ago. And he said that his, you know, way back and way before he was, was even older, he had hair and everything. And he said, he said that, uh, he said that his biggest, his biggest, uh, job role was recruiting because he needed to figure out how to get, you know, the right people with the right other people to build the correct teams. And this is something that I think is lost in today’s industry. I think that like, you know, a lot of times this is overlooked and. It’s not something that usually happens with planning. It happens naturally. Sometimes it just happens and you have this great team and sometimes you just don’t. And what forms that? I don’t know all of the tangible… details, but I do know that the main thing is trust and being able to trust your coworkers and not, you don’t play political games. You don’t do any of the things that are not part of an objective engineering team. You literally just try to get the jobs done and you eat your crow whenever you have an issue. Like if you’re wrong about something, just be wrong about something. Don’t take it all the way to as long as you want to take it. Just be humble. And sometimes that results in fast moving delivery and fast moving delivery is what. everyone wants. I don’t care who you ask on any engineering team, they want the software out quickly and they want it to work. Those are the two things that everyone wants. It needs to function and it needs to exist. It doesn’t exist if you can’t release it. So for me, I’ve seen managers standing in the way of that. I’ve seen project managers stand in the way of that. I’ve even seen scrum masters stand in the way of that. People need to loosen the grip a little bit. from every different role and every perspective on these teams. And if they can, you loosen the grip and then you’re relying on people’s skill sets. Now that’s where things become a little bit hairy because if you have a bunch of green people on your development team and you trust them too much, they over-deliver, they break things in prod and you’re going, why did I trust this person? I don’t know why I did that. Well, you have to. That’s the answer. You have to trust your team. So build the right teams. That’s all. Now, how do you do that? Well, then it becomes even more complicated. Pay grades are so high now. I mean, think about what it takes to employ a top-notch engineer. A lot of people now, I’m reading posts on LinkedIn, you know, oh, you know, the junior devs are better than the senior devs ever were. The junior devs now are awesome. They’re using ChatGPT, and I’ll just say, okay, fine. What I’ve noticed, and this is not a knock on junior developers at all, is problem-solving capability of senior engineers is way beyond the problem-solving capability of… lower-end engineers. Yes, they can develop software quickly.

Speaker 0 | 22:07.485

The skill isn’t in the code. The skill isn’t in the ability to write the code. And look,

Speaker 1 | 22:12.247

being an old dog, I don’t want to be the person that’s like, yeah, yeah. But look, if I pull up a notepad and write a JavaScript closure for you and ask you why this is a closure, you better be able to tell me quickly without opening chat GPT, okay? That being said, if I say, hey, man, I have a high-level problem. We need to integrate with Braintree and process the payment, and I need you to add some webhooks so that we can process the responses from the payment processor. Can you please do that and just create a prototype? I don’t want to see a separate application with 35 webhooks. I want to see a direct integration and one endpoint so that we can actually see how this works. What I’ve seen, and this is just me, and maybe I’m wrong about this, I’m not saying I’m correct about all this, but junior developers tend to over-engineer things, and you have to watch them so much more closely because they don’t want to ask questions. They, they’re afraid that if they ask questions, they get into the imposter syndrome thing. And then after that, it’s like, oh, well, he thinks I don’t know what I’m doing. And I never do that. I always, like I said, I err on the side of trust. So, um, I don’t know if I’m down a pit.

Speaker 0 | 23:13.610

No, that’s okay. So I’m glad you brought up imposter syndrome. Cause like, like, so as a leader, how, how do you help your teams overcome some of those blockers like imposter syndrome? Well, I’ll focus on imposter syndrome. Just start with there. Like You’ve got team members that got imposter, but you’ve been in this, you’ve been in lots of different fields here. How do you help your teammates, especially the junior ones, overcome that and feel confident in their space?

Speaker 1 | 23:36.821

Yeah, that’s a great question. It’s not really difficult. What you have to do is create some rapport with them and let them own their successes and their failures. If you start to persecute someone for failures that you perceive on your side that aren’t necessarily failures or whatever they might be, you’re actually becoming a negative influence on their career because… When they do things that are correct, they don’t hear anything. When they do things that are wrong, they hear all this negative input. So you have to kind of do both sides of that equation. There’s successes and failures. Don’t steal people’s credibility or credit for things that they do right. Always call them out and give them, I would say, give them props for what they’ve done correctly so that they know they’re doing the right thing. But if something negative happens, pull them aside and say, hey, look, you should have ran more tests. And look, I had to fix this in production. I have a great example that I just did this yesterday with one of our. contractors, he didn’t put a null check on a string length check and broke our entire profile page in production for one simple call that was basically to see if you had discount codes on your account. And I could have just eaten that and not told him, but I had to bring him aside and just be like, hey man, why didn’t you run this one use case? He said, I thought I did. I thought I ran that. I said, okay, well, I’m glad that you thought you did that, but I did it and it was in production. It was broken for about 15 minutes and users probably saw it. I had to fix it myself. What are we going to do to fix it? And he came up with a good plan to fix it. He came up with some tests and some other things. So positive outcomes can come from negative, I guess, reactions. I don’t know if that makes sense.

Speaker 0 | 25:04.939

Yeah. And I’m thinking about, so I work in the IT operations delivery space. I’m a technologist in delivery of things, network switches, gear, servers, that type of stuff, but I don’t develop. So you’re in the development space. And so I’m thinking about some of the differences in leadership and how you have to lead people who are developers. And it’s, I don’t want to call it like totally different skill sets because leadership is still leadership. But. some of the tactics and the things that you you put forward i i think there’s some really important lessons and valuable lessons there like what you just shared you pulled a lot aside this gentleman and you highlighted to him where the gaps were but you asked him for a plan so like i i always describe this as um giving an exit like you give somebody an exit ramp to to for lack of better put it to satisfy you or please you you know so if i’m the leader and i’m going to say i’m going to i’m going to offer um some sort of critique i make it pretty evident what the path is to get back into like like i would call it you know whether it’s to impress me or anything else like that and it’s not that they need to like kiss the ring it’s just that that’s the like the human nature of it like if i if i got criticized for doing something i want to know how do i get back into the good graces with that individual to know that i’m on a good place and and that goes a long way towards building the rapport And the understanding and the trust that you’ve highlighted as being so important in so many of those teams. Yeah,

Speaker 1 | 26:31.529

I mean, I think it’s simple, but it’s also complex, right? This is not an exact science and it’s not something that everyone is really good at. But, you know, being a human being and not being a robot as a person that is a leader in the space is very, very important. At the end of the day, we all check out at 5 p.m. or 6 p.m. and we go do our own thing. We have a life outside of this space. It’s not. Unless you’re working with pacemakers or something, no one’s going to die if you implement a bug or you send a bug to prod. So let’s just find a way to be better. And so a lot of times just finding a progressive way to be better and not being super negative and just having these histrionic fits about certain things is very, very important. You have to be the level-headed person. You have to be the one that doesn’t freak out when there’s a problem. And, you know, if… prod is down and everyone is running around with their hair on fire, you need to be the one that goes into the database and runs some queries and figures out what’s going on and then brings it back to the team. So leadership, hands-on technical skills usually go hand-in-hand with that level-headedness. I’ve seen some managers that are people managers that don’t have that acumen and they literally will sometimes lean on certain SMEs. Let’s just talk about management for a moment. I think that management can go wrong a lot of different ways. And one of those ways is when they, and I talk about trust a lot, but if they trust the wrong people based on a lack of skillset. So if they really don’t know how to solve specific problems or haven’t had the experience in solving specific types of problems, they might trust the wrong people. And when they trust the wrong people, that is a detriment to an entire organization because those people become propped up artificially and they make decisions that are bad for everyone. And everyone knows it, but they can’t say anything. And this is one of the bigger questions, I think. How do you, and I’ve had problems with this before, I don’t like to jump rank. I don’t want to go up and complain to an SVP over something. I usually like to just do my job and make sure that everyone under me is happy, all of that. But sometimes you feel like you need to jump rank and call somebody out. Now, how do you do that when someone has more authority than you, higher pay grade than you, and you know they’re doing something wrong? What do you do? I mean, that’s a real question that I’ve struggled with. You pull them aside and talk to them in a vacuum. You don’t want to call them out in front of the team. That’s unprofessional. And to me, even talking to them about these things can be unprofessional because maybe there’s no fix. You know what I mean? So that’s something that I’ve that’s a question that remains in my mind that I’ve dealt with several times. Some of these leaders that are higher, higher up, I’m talking about founders, SVP. We’re talking about the upper echelon C-suite people. They can almost feel like they’re not approachable at all. And I just wish that some of them would come down to the team level, make themselves approachable and say, how do we fix? I know we’ve got some issues. I don’t know where the issues are. How do we fix it? They rarely do that. They’re usually up on their pedestal somewhere and they don’t do that. And I encountered that at Schwab. And Charles Schwab has a real problem with that. SVPs have no clue what the teams are doing. And they try. a couple of people to tell them. So that’s another subject. I will leave that one as kind of an open.

Speaker 0 | 29:50.471

Yeah. Okay. That’s okay. Yeah. And it’s true now. So I don’t, I, you know, I’m a, I’m a C-suite executive, but I work in a small nonprofit. So like, I’m, you know, I’m, I, you know, we’re 500 staff, not 50,000. So like, it looks different. Um, I will say that. At least in my space, I try hard to make sure that I’m accessible. But it’s a lot easier when there’s 500 people in the organization to feel that way than there is when there’s 50,000. And that’s different. But I mentioned I’m on the board of a credit union, and they have about 1,000 employees. And they have a very great work culture, so this is not an explicit knock. But even the jump between one to the other and the difference in industry. there’s there’s there’s absolutely a stratification there where like the c-suite executives and like the line staff are much further disconnected um just by nature of the business you know so like that that can play a part i i did want to i wanted to pivot slightly you mentioned just kind of offhand but you talked about junior developers this isn’t a question about junior developers specifically but you mentioned about junior developers like potentially using say like chat gpt and some other tools to do this in your space so of course this is the talk of the moment all the AI tools and all the things that are coming down the line. I’ve been exploring this in my space and in the nonprofit space. But for you, there’s some really, I would argue, compelling support cases for some of these AI tools. But I’m just curious what your feelings are on them. How have you been exploring them and what are your thoughts?

Speaker 1 | 31:22.422

Yeah, thanks for that question because it comes up constantly. You have two different schools of thought on this. You have the people that are more, actively developing software with their hands. And you have the people who want the software delivered quickly and don’t have any idea what you’re doing when you’re digging the dishes, right? So the people that are not hands-on are the ones that really love this idea of Chad GPT and AI. Like, hey, eventually we don’t even need developers. I can just tell them to do something. That’s right. Developers are paying me ass because all these technical details come out every time I ask them something. And then you have the developers who really want to use these tools. There’s nobody that I know that doesn’t want to use them. The problem that we have is quality of what comes out of those engines. We had one at Q2 that was based on our internal software, and it was trying to suggest things for us to do, and we ran into one specific. And I’ll ask you this question. I think probably you’re going to be able to identify it very quickly. If you’ve got a legacy code base, you’ve got a new code base, you unleash an AI on all of it, and it’s making suggestions. How does it know the difference between crappy code and good code? Right?

Speaker 0 | 32:27.776

Or does it hallucinate or create some sort of hybrid between the two where it’s like, some of this is JavaScript and some of it is like, you know, RPG, convert old AS400.

Speaker 1 | 32:38.480

And sometimes it’ll reach out to the open market, which is the main code base that they leverage. And you’ll get things like entity framework that you’re not even using. It’ll be a suggestion. So we did a pilot test on this internally at Q2. And we found that from day to day, the suggestions that this… AI was making were completely different. So one day it would be, here’s a suggestion for this code block and you would go back to the exact same code block the next day and it would give you a different suggestion. And to me, right then and there, that’s the main quality problem. From my understanding, the human input needs to be relative to what you do, like say you’re on a Reddit thread and you’re upvoting and you’re downvoting things. You need to be able to upvote specific good answers and downvote the bad ones. And I don’t… excuse me, I don’t think the learning aspect is there for a lot of these LLMs. They’re really good at taking a question and digesting it and producing some LLMs and language back, but they’re not great at being consistent and actually giving you valuable data. They’re just giving you data that’s in their database somewhere. So if we could improve on any of these tools, it would be more human input about quality. We actually got rid of that AI at Q2 because of this reason. Well, there was another reason. Some of the upper brass was a little bit… scared that it had too much access to some things like kubernetes clusters and things like that and didn’t like that at all but it was just logs we wanted to have access to all of our logs the idea was that mainly it was a code base tool and a log tool so if you wanted to if you wanted to digest logs you could do that which is great they’re really good at digesting logs in fact that’s probably the main use case for an ai that i know of that’s very very valuable give me the log for the last 10 days on these 10 different machines only sql server and give me all the event loops you And any type of stack traces that you see, boom, they’ll give it to you. Okay, that’s perfect. Then you digest that data. But if you say, give me a ORM layer for this application that needs to connect to MongoDB, but also be swappable for Postgres and even have SQL Server in the background for another different use case, the thing just on itself because there’s too many different things. And I apologize for the language, but that’s what happens. And so for me…

Speaker 0 | 34:50.606

Passionate, that’s all. That’s all. We’re passionate about this one.

Speaker 1 | 34:54.609

It’s because I would love to trust these things, but I don’t, and I can’t. Objectively, you just can’t yet. Now, someday they will be better. Someday they can actually solve problems, and someday they are going to be capable of writing small applications. But they’re not going to be capable of leveraging senior engineer experience. There’s always going to be a human being pushing buttons to put pieces together. And that’s where I’d like to be. You know, I love writing software. I love the craft. I love being able to make things simpler and make them as simple as they can, but not more simple. But sometimes, man, there’s boilerplate things that you’ve done before and you’re just like, I wrote this HTTP client class like a year ago and I remember doing this, but I don’t remember all the details. Do I have to start over? Where did I put that code? I don’t know where I put it. You could push a button on the AI and say, you wrote this a year ago and here’s your block of code and you just started here. Thank you.

Speaker 0 | 35:44.015

Perfect. And even. Even Geordie LaForge had to be Geordie LaForge to run the enterprise, right? Like, you know, in front of all that technology.

Speaker 1 | 35:52.301

Still pushing buttons and even data still had to push buttons. And a captain would know that.

Speaker 0 | 35:57.905

So I appreciate your thoughts on that. I think that was insightful from an alternative perspective. I don’t say alternative to me because I work just in the infrastructure space. So it’s a little different to think about how it impacts in the development space. Another thing I was, so just some maybe more fun, humorous discussion points and things to talk about. So have you ever, you tell us a story, one that’s safe to share, I guess I would say, a time that you’ve done something, like in your professional world where say like something went wrong and you’re like, oh, crap, like we did it wrong. And like, it was almost embarrassing, but you get to laugh about it now. And I’ll… I usually start with a little bit of an example. So way back in the day, it’s funny now, I guess it was almost 15 years ago now, maybe a little more, I was at a customer site replacing some servers, and their server room was really just 1980s classic server room, like wood paneling and wires on the floor, nothing underneath and stuff. So behind their racks were just stacks and stacks and stacks of wires, and you had to step on the bundles of wires to get the stuff in. So I had to tiptoe through it, and I’d done this many times. But one time I did it, and the bundles of wires, there was the… one rubber pulled on the next rubber and the holes on the next one it had unplugged the expansion chassis for for their as400 so like you know where here i am the network admin and i are in there working and all of a sudden like the as400 guy comes in and he’s like what the hell’s going on what the hell do you guys do we’re like we didn’t do anything what are you talking about and what ended up happening was the plug had come out so we look and we’re like nothing’s unplugged like it all looks fine and the plug had had been pulled just enough to disconnect it, but not enough to make it fall out of the outlet. So we couldn’t see at first that it was done. And I was a contractor here, you know, so like here I just took out their, you know, their manufacturer, like they could probably quantify how much money that stake cost. I don’t, I never got to know that number myself, but yeah. So yeah, like open the floor.

Speaker 1 | 38:04.013

Yeah, there’s two that come to mind, but the one that really, are we almost done?

Speaker 0 | 38:08.532

Yeah, you got a little while. You still got a few minutes.

Speaker 1 | 38:11.395

Okay. So the one that comes to mind is I worked in a data center too called Data Foundry, which was Texas.net back in the day, and that was downtown Austin in the old building. So I worked the night shift. I would come in at 8 p.m. sometimes on a Friday, and everybody was going downtown to party, and I was going to work. And so I ended up in this building, this data center all night. And we had biometric scanners. And so… All this cool tech. And so every once in a while, somebody would get lost. It was under the Museum of Art in downtown Austin. Someone would get lost and try to put their hand on the biometric scanners. Uh-huh. I’m going to give you two little instances. This is kind of funny, but I’m going to go to what you were asking after that. This guy came by with his girlfriend, and the biometric scanners only worked for us and a couple other people, people that had access to the servers. He goes in there and stands up real straight. And I’m looking at the cameras. I can only see people down there. And we’re talking about it was an old bank vault. So the old bank vault door. was this thick two-foot steel automated. It had a man trap, and if you opened it, you would see a fluorescent sign that said Data Foundry. Very, very ominous. Like if it opened, it was like, what in the hell is this thing, right? So this guy puts his hand on it, and nothing happens. So he takes his hand off. His girlfriend comes up, puts her hand on it, and I push the button to open the door. You get in there real straight, and the door opens, and he looks in there and sees that fluorescent sign. And I was sitting there just laughing because I’m the only one in there at the time. And the door shuts automatically because you can’t get through the man trap anyway. And it’s not a security issue. And so door shuts. He puts his hand on it again. Nothing happens. Nothing happens. She puts her hand on it again. It opens up again. It opens up again. And he looks at her like, who are you? Right? Like, you have access to this and I don’t. What in the hell is this? Right? And I come on the speaker. Can I help you? And he’s like, we’re good. We’re fine. And he just. bug out. But when I worked in that place, we had a Halon system, and the Halon system had sensors on the floor. The sensors would sense moisture because the machines, obviously no moisture was allowed by the machines. This was a big deal. People paid a lot of money to be in that. that co-location and one of the guys went out there and he’s the larger guy had been hired and he was it was hot day and he went out there he was sweaty and uh i was thinking in my head i was like man i hope he doesn’t hope he doesn’t sweat on the on the tiles out there you know could cause a problem he goes out there the halon system goes off and just covers the whole room with no way yeah and i couldn’t find him so i’m in i’m in the op center and i’m looking out there and i’m like i can’t even see the dude and he finally came walking back with his hands out like this and made his way back So that was one thing that’s similar to what you said, but failures in software, the worst one was at Schwab, our trade blotter one time failed and it was out of sync with the open market. So we have as a trade blotter, you put trades in on our system, Schwab Advisor Center has the ability to make trades. And so hundreds of thousands of trades were in this blotter about to go out to the open market while it failed and it couldn’t be processed. Turns out the trades were a net positive for the people making those trades because they’re good traders. I mean, they operate on this platform. And we lost in 30 minutes close to a million dollars just because that trade blotter was behind. And once it went out to the market, it was a net loss, but it should have been a net positive. Yeah. And so they have insurance that is made for fixing that. But this was a very simple software bug. It was something that had to do with mapping fields, and the fields weren’t mapped properly to go from one step to another. So every time I tried to move forward in production, it literally just couldn’t map a set of fields to a shorter set of fields that wasn’t properly. And so it would fail, but it was a silent failure because it was very old. We also had mainframes. So we had these mainframe services that nobody touched. Usually they think they were written in cabal or something. And once it got to that layer, there’s no testing it until it gets to the live market. How do you test a live market? You can’t replicate a live market down into lower environments. You just have to make sure that it works when it gets there. So that was a big learning for us then. Another one, another big failure that, so you talked about, so that was one. You’re talking about how did you fix it, right? So what were the two questions? One was what was the failure and one was what?

Speaker 0 | 42:19.057

Oh, just in general. I was just interested, mostly interested, just the humorous stories that we all have, like these war stories where we did something like boneheaded and we’re like, oh, that cost a lot of money or this thing went down or, you know.

Speaker 1 | 42:33.767

Well, we thought we were fired at that point. Yeah. But it turns out they had insurance backing that was like, guys, it’s not a big deal. Just don’t fucking do it again. Please don’t ever do that again. it’s been a while since we had to dip into this pot to pay for some stuff like this but and then the other thing was when I was at Volusion we had 40,000 store merchants and I don’t know if you’re familiar with jQuery but jQuery has a so jQuery for those who aren’t familiar with it is a cross platform or cross browser library that allows you to write code that works on pretty much every browser back in the day the event structure was a little bit different so they encapsulated into this little package that everyone used. And so back then, I think it was like jQuery 3, or just before jQuery 3, maybe 2.1. We were building some things for all of the store merchants on the core platform, and I was writing the code, and I saw this dollar sign. Well, the dollar sign is shorthand for the main jQuery object in most environments. Use a dollar sign. Okay. Well, it turns out there’s another JavaScript library called Prototype. Prototype also uses the dollar sign. What we didn’t know or what I didn’t know is that there were some people that were skinning our storefront as a third-party, I guess, offering and being paid to do so and using Prototype. Well, Prototype used the dollar sign. Well, it turned out there was like 30% of our stores, and I didn’t even know it. I decided, you know what? I’m going to change this dollar sign, make it more explicit. We’re going to move it back out and just make it say jQuery. No, it was the other way around. I’m going to take jQuery and turn it into the dollar sign. Well, the dollar sign overlapped with Prototype. Didn’t break any of our tests. didn’t break anything went all the way out to prod and i broke 30 of the stores and we’re talking about 40 000 merchants yeah soon as that goes out to prod the system goes nuts support calls everybody’s freaking out what the hell is broken what happened and i’m going oh damn i was like was that me that can’t be me it couldn’t be why and there were why the dollar sign yeah i just changed it from literally typing out j query to a dollar sign for shorthand to make things easier to read or just shorthand And it broke everything. So I went down to one of the sites and I give me an example. I was like, I need, I need an example like for this out. It was one of the sites. It was an airsoft company. One of our biggest sites was an airsoft sales company that sold these airsoft guns. And it was just completely just borked. Right. I go in there and look at the source and I was like, oh man, I was like, okay, I see it. I see it. Prototype, prototype. Where’s prototype? Why are we at prototype? Who’s got prototype? Well, the third party they pay for has prototype. Okay, fine. Whatever. One quick release and I fixed it. It took no time at all. We weren’t down for very long. I just. rolled back what I did and pushed it back out to prod. And I started receiving things from these, uh, from these merchants. Uh, like I was, I got a wine and cheese basket. I got all these other things. And I broke it. I fixed it. But I became the hero still.

Speaker 0 | 45:21.883

That’s great. That’s awesome.

Speaker 1 | 45:24.104

And the dev team was like, what did you do? I’m like, I broke everything. That’s what I did. I just fixed it.

Speaker 0 | 45:29.666

So you’re like, I don’t mean to compare you directly to an arsonist. It’s like the arsonist dude that puts out the fire, you know, comes back later. So I learned right away that,

Speaker 1 | 45:39.648

like, yeah, it’s not always as big of a deal as you think it is. But, uh. But yeah, that was one of my biggest memories at that place.

Speaker 0 | 45:46.628

Nice. So we’re coming up to the end of the podcast here. Just got a few minutes left. I’m going to ask this question to you more explicitly because we have a shared history in this particular timeframe. So we’re both technology people. But what was your first technology experience, your first computer experience? How did you get into being in the space?

Speaker 1 | 46:08.694

I had a 286 IBM that my mom bought me, and I think it had something to do with you guys talking about IBM computers all the time because I’m pretty sure you had one before I did. It was a 286 that was flat, and it had a little flip thing on it. Before that, it was all video games for me. And I went into that thing. I started going through everything. I started going through every file in the system. There used to be something called autoexec.bat that was on Windows machines. I started the entire machine. I would go through every executable and run it. I would run everything that I had and see what it did. And I ran that autoexec. I said, it doesn’t do anything.

Speaker 0 | 46:43.141

I deleted it.

Speaker 1 | 46:45.523

I tried to restart the machine, and the machine wouldn’t do anything. The OS just completely paused and broke. And I had to call my friend, Rio Marcantonio, at the time. He had another machine just like it. And I said, what did I do? He said, what did you delete? I said, I deleted the autoexec.bat. My mom’s going to kill me. Can you figure out what I did? He said, you don’t delete that. That’s the bootstrap for the whole OS. I didn’t know that. That was probably the moment that I realized don’t go around screwing with it too much until you really understand how things work. Then once you understand how things work, you can make surgical changes. That’s how I approach software with legacy code bases now because of that trauma. So that machine. I had that machine in that apartment in Spicewood Springs. You and I used to chat back in the day. That was the machine. Later on, I had a Packard Bell 386.

Speaker 0 | 47:35.414

you know the rest is history yeah yeah i remember when like when my dad bought like that ibm ps1 that had had the 2400 modem and it was like a big deal that we all got on aol and prodigy and like it was sticking around in night field like we’re gonna go on this but at the time we didn’t it didn’t dawn on us how

Speaker 1 | 47:54.464

expensive it was at first remember when it was like 6.95 an hour and we were a little late so that was one of the biggest scans is that they were charging for the minute and aol and prodigy you know the main users of those services were children at that time. It’s obvious that most of the kids were so curious about these online services that they were on there using them. And I got my ass chewed multiple times for huge bills.

Speaker 0 | 48:22.099

And that’s where it’s the story of the late Gen X, early millennial kids, right?

Speaker 1 | 48:28.323

My mom, she called it glorified walkie-talkie, right? She was talking. And occasionally she would pick up the phone and start talking and the model, you know, but then I would hear my mom from the PC speaker. Hello. Hello. I’d be playing doom or something, trying to shoot someone. I’m like, what are you doing? I have to go one out. What are you doing? You know, and luckily none of that ever happens anymore, but I gigabit feed into this house and as much internet as I could ever want. So.

Speaker 0 | 48:55.079

Right. How times have changed. So, so, so last, last thoughts here, then we gotta, we gotta wrap up, but, uh, For the listeners out there, what leadership advice do you have for people that are coming up in the space? You know, whether it’s being, in your case, being a development leader. You know, what advice do you have to share with the folks that are getting into their first leadership roles and what things they need to pay attention to?

Speaker 1 | 49:17.714

I would say, and this is probably the first thing that comes to mind, is aptly, aptly. Okay. Respect your engineers and your SMEs. Find the ones that really are. your keystone to your environment and make sure that you trust and leverage those individuals and never get yourself into a position where you’re undermining decisions from people that you trust as an SME. establish your SMEs, your subject matter experts in different areas, and make sure you’re leveraging them and respecting them and not undermining their decisions. One of the biggest problems I see with management is that they think that they are the end-all be-all and that engineers don’t matter and that whatever they say is secondary to their decision-making skills. The best management and the best leadership leverages the input from their engineers and their subject matter experts and makes decisions based on that. That’s all you can ever do. And don’t ever believe that you’re the best. that it’s you that is the end-all be-all. You have to leverage your engineers. That’s what I would say. And that goes for every engineering org. And that’s one of the biggest gaps I’ve ever seen. I’ve seen great leaders do that. And you loosen the grip and everyone, the stress level goes down. The organizations will run like butter and you just don’t have to worry about that much. But if you have anxiety about talking to your leadership and about them changing your decisions and changing your input and not listening to you, that means you’re not valuable as a… as an employee or a professional and no one wants to feel that way. No one should do that way. You hire people based on what they can do and you trust them after that. And that’s, that’s my perspective on that. Thanks.

Speaker 0 | 50:51.219

Awesome. I appreciate that insight. Ian, thank you so much for investing your time with us on the podcast today.

Speaker 1 | 50:57.080

Well, thank you, Doug. It’s been really good talking to you, man. And we should stay in touch.

Speaker 0 | 51:00.881

Absolutely. So that’s a wrap on today’s episode of dissecting popular IT nerds. I’m Doug Kameet and we look forward to coming to you on our next episode.