AI Won't Solve Your Toughest Engineering Problems | Honeycomb’s Charity Majors

Charity Majors 0:00 The superpower of engineering is doing and always has been. Software engineering is not about writing code. It is about solving business problems with technology.

Conor Bronsdon 0:16 Welcome. We are back on Chain of Thought, and I am your host, Conor Bronson. Today, I am joined by a powerhouse in the world of engineering and observability. You may know her Charity Majors, the co founder and CTO of Honeycomb. Charity, welcome to the show. Oh, thank you so much. I'm getting all excited. Powerhouse. I mean, I think it's a really fair phrase. I have been delighted to have the chance to chat with you before and watch a ton of your content, read quite a bit of your content. You're very well known for your strong opinions,

Conor Bronsdon 0:46 for your insights on everything from DevOps to observability and the software industry as a whole. And many of them musings, you've been kind enough to post at your website, charity.wtf. It is the best domain. Right? I am kind of jealous, actually. It makes me think of changing it every time I see your domain. I'm like, oh, should I have connor.wtf? Like, haven't Yes. I done that

Conor Bronsdon 1:06 You should. So you're inspiring me. I mean, this time we'll actually go through with it. This is gonna happen. Alright. Sidebar. I'm just curious now. What what inspired the .wtf

Charity Majors 1:14 domain for you? I mean, as soon as I saw it, I knew it must be mine. It's like Romeo and Juliet. You know? You see each other. Your eyes meet. You're like, I must have you forever.

Conor Bronsdon 1:24 And and honestly, there are too many good posts to list on your website. You've been writing for ages. You clearly use that as a way to not only express yourself to the world, but, like, organize your thoughts and kind of teach. And if you're a listener to the show and you haven't already, you should absolutely go check out Charity's website. Again, .wtf,not.com.

Conor Bronsdon 1:45 You will not only learn something new, but come across plenty of colorful language in the process, great reading all around. But today, I wanna start by talking about a piece you wrote that was also posted to Stack Overflow's blog titled generative AI is not going to build your engineering team for you. And too often, AI, for all its importance, has a way of taking up the oxygen in any room where it's being discussed.

Conor Bronsdon 2:09 But as you have pointed out, it won't solve every problem that we have. And it's changing what it takes to build a great technical team in today's environment. So let's start simple. What was the inspiration behind that article in particular?

Charity Majors 2:24 There's this great book that was written recently, and I'm blanking on the name of it, but it was for developers who are interested in writing, just like kind of pitching them on why should you write. And for me, like, I did a little blurb for their book. And so I've been kind of thinking about this recently and realizing just how much I figure out about my own thoughts. I crystallize them when I'm writing. Like, there are things that I didn't realize I thought until

Charity Majors 2:48 I put it down and I looked at it and was like, well, that doesn't seem quite right. You know? And I feel like I'm I always have like three or four things that are overdue. So like, why do I sign up to write anything for anyone else? Well, it's because if I'm writing for Honeycomb or writing for myself, it's like topics that I chose. And and when someone else asks you to write something, they typically have a prompt. And that often, I feel, tickles. It shakes something

Charity Majors 3:15 loose in me. It it break like, I never would have thought to write this if Stack Overflow hadn't reached out to me and been like, would you like to write a post? And I gave them a few ideas. So I appreciate that about the writing process. It's like, figure out what I learn. I figure out, you know and and the one that you know, this was this was oh my god. This was almost a year ago now that that this was published.

Charity Majors 3:38 And it's crazy. It feels like the industry changes so much, like, every six months. Like, it's an entirely different industry. But you asked what the inspiration behind the article was, and it was really just kind of trying to unpack what does this mean for hiring? Because because a year ago, there was so much excitement about can we replace engineers? You know? And can we automate them out of jobs? And I sit in this weird place where I'm kind of in the cross

Charity Majors 4:09 section of worlds between, know, there's engineers, there's SREs, which I very much identify as, and then, you know, founders and and VCs. And those are very different conversations.

Conor Bronsdon 4:22 How do you think that conversation has evolved in the last year? And

Charity Majors 4:27 would you change anything that you wrote today? Or I guess what's your perspective today on it? Well, the other the other thing about that article was it was kind of a Trojan horse to just like it said AI in the title, and then it just talked about all of my strongly held beliefs about why you should hire junior engineers and build great, You know? Because I because I honestly don't feel there's so much about the present moment

Charity Majors 4:47 that is not actually new. It's just intensified and accelerated. There are so many sentences where you could just replace AI with automation and be like, Oh, it's the same thing. Only a little bit faster and more intensely than before. It's 100% true. It's like nine times out of 10, you can do that. There are novel elements. But so much of it is think that engineers I think that people are being disingenuous when they say, oh, it won't replace people. I mean, automation

Charity Majors 5:16 always the point of automation is that it replaces people. And so you need to pair it with, you know, what are people doing with that time? You know? And I feel like I this was especially true a year or two ago. I see it less now, but I was seeing creative folks go like they had so much fear about losing their jobs that they were refusing to use the new tools. Yeah. And that is like

Charity Majors 5:41 that is the fastest shortcut to not having a job. You know, you can't you can't do that. You have to learn to embrace the acceleration, the, you know, the the amplification, the what can I do that's exciting and new with this? Like you that that's the number one tip I would take for people is it doesn't matter. It's the same with AI as it was with automation.

Charity Majors 6:03 The fastest way to make yourself irrelevant is to not embrace the new.

Conor Bronsdon 6:07 I can definitely hear this historical context you're applying, this understanding of the Wild West of of old tech, as you put it, and these different waves that have come before. What are the other lessons that you would advise folks who are navigating this new hiring environment, this new team building environment to take as they try to both leverage AI in their day to day jobs and understand how to differently move

Conor Bronsdon 6:32 in the world?

Charity Majors 6:33 Oh, it's such a good question. And the answers are all over the place, depending on whose perspective I'm supposed to be answering or representing. Right? Well, let's think maybe with the developer side first, and then maybe let's talk also about the CTO leadership side, since those are the two worlds I see you at the cross section of, as you put it. If you are an engineer and you're a senior engineer, let's say, I think that

Charity Majors 6:56 the superpower of engineering is doing. It always has been. Software engineering is not about writing code. It is about solving business problems with technology, which has historically included a lot of writing code. Right? But I think taking a step back and being like, it's about solving problems and how can we but like, it's also been something we've known for decades is that engineers are the value generating

Charity Majors 7:23 engines of modern businesses. We have a lot of power. And I think sometimes we don't inhabit that power or recognize it or flex it in the ways that we should. And I think it's increasingly like we have to. We have to represent, you know, we have to go to bat for junior engineers because they don't have the power yet to go to bat for themselves. Right? Like, we have to be the ones that point out this is how you build high performing teams with the range of levels. You build organizations that are continuously learning,

Charity Majors 7:52 working on interesting, hard new problems, pushing their boundaries, like the work that you and then I and I feel like senior ICEs are the ones with the most power and relevance and context, and we can't abdicate this responsibility. Know? On a personal level, I think if you hate AI, you should really embrace it. Like, get to know it. Like, get to know it up close, right? Nobody cares about the critique of somebody who is, like,

Charity Majors 8:22 way away from it. They care about the critique of people who are who are knowledgeable, who are in the midst. Right? So the more you hate it, the more you you should embrace it. Absolutely. And it's an overdone phrase, but knowledge is power and

Conor Bronsdon 8:36 engineers need to flex that power as you put it. So one thing I hear a lot of, and I think it's a fair take is that senior engineers and up are best positioned to leverage AI to accelerate their impact. But there's this risk happening where so many companies now seem to be just hiring senior engineers, just trying to find the most senior people because, oh, they're gonna be the most effective.

Conor Bronsdon 9:00 And as as you have alluded to here, that that feels like a potential weakness, not only for the industry, but individual teams.

Charity Majors 9:06 This is an apprenticeship industry, full stop. You know? Whether you've got a CS degree or not, like, over the course of your career, almost everything you learn, you learn at work. And and we learn from each other. Right? And I feel like it this isn't a new thing, but it's like, again, more so now than ever that people are looking for the most senior engineer that they can get for the least money, which is, I think,

Charity Majors 9:34 so shortsighted because it's really about the ability of the engineering team to ship. And that does not actually correlate tightly. I don't know. I feel like the best engineering orgs are the ones where everyone is leaning in and interested and curious. And the systems that you're building are rewarding that curiosity. And the highest performing teams that I've ever worked on were the ones with a range of levels where people were growing. I worked on some teams that were staff plus engineers,

Charity Majors 10:07 and they cut corners. And they're bored a lot of the time. And and nobody's asking the really interesting hard questions. Because, like, if you're implementing a login page for the thirty thousandth time, you're on autopilot. You're not actually that interested in it. Right? Yeah. It's not like when you do your best work. You do your best work when you're deeply interested in what you're doing. It sounds like,

Conor Bronsdon 10:30 in your viewpoint, this narrative that's out there right now of, like, oh, we need just need more senior leader in teams is overblown.

Charity Majors 10:39 I also think that, like, today's juniors are sharper, more educated, more skilled than juniors have ever been. You know, when I was a junior, it was like, oh, can you drive the unit's command line or write some HTML? Cool. We'll drop you into the deep end and you'll figure it out. Right? Today's juniors, they can write tests. They know multiple languages. They are more And the thing is that AI can be really valuable to juniors too. I have a junior engineer, Roozy, who has written a great blog post about how they spend all day in the old chat GPT window just asking questions like, what about this? As long as they don't ask questions of the senior engineers next to them. But like by the time they're ready to go to the senior engineer, they have tried a bunch of like easy things. They've crafted a really great question.

Charity Majors 11:27 They're growing and learning faster than I think I've ever seen junior engineers learn, because you have the ability to iterate so much faster. You have the ability to have like your own personal coach. You know, I feel like there's this perception in the industry that if you hire a junior developer, it's going to be a net negative for the next who knows how long? And who can afford that? And I agree. No one can afford that. But that's not actually the reality. Like, the reality is when you hire anyone,

Charity Majors 11:56 they're going to be a net negative for a period of time. It takes a while to train and to transfer skills and to teach people the ropes. You know, if you if you hire a senior engineer who's seen these problems before, it might take you a couple of months. You hire a junior engineer, might be a couple months more. Like, honestly, most of the problems that most engineering teams have to solve are not that hard. You know, a lot of it just needed to get done. Hey. Hey. We can't say that on the podcast here. We gotta

Charity Majors 12:25 don't reveal that to the industry. I know. Right? And and this is not uniform across companies or industries. Yeah. You know? But there's a lot of work out there that just needs to get done. And I've said before that intermediate engineers are often the workhorses of the industry if you're measuring productivity in terms of shipping features, responding to customers, because they come in in the morning, they put on their headphones,

Charity Majors 12:48 and they crank all day. And your staff engineers, your principal engineers are in meetings. They're doing cross team collaboration and coordination and which is what they want to be doing. Typically, they spent ten or twenty years in that with their headphones on cranky and they're, you know, they want to solve new problems now. But like, you need people who can just put their headphones on, write code, ship shit all day long.

Charity Majors 13:14 And then often your intermediate engineers are the ones who can really be, to use your word, the most powerhouses.

Conor Bronsdon 13:21 Let's say someone comes to you and they say, Hey, I'm just going to build, I'm building a startup and I'm just hiring senior devs. Of these AI code gen tools, they can just ship so much faster and then I can have them solve these big business problems because they have this broader context understanding.

Charity Majors 13:38 How would you advise them to build their team differently? What would be the How do we write? I don't think I would advise them to do anything. Interesting. When Honeycomb was small, when we were starting, the first two engineers that we hired are now our only two principal engineers. You know, the first three engineers we hired were solidly staff plus principal engineers. Right. But there comes a point because the biggest risk that any baby startup has is

Charity Majors 14:01 that they're going to go out of business because they're not going move fast enough. Your time horizon in a startup, you can measure in weeks or months, you know, but like you can't plan for more than, I don't know, six months if you're doing great, right? You can't see around that corner. You don't know what you're going be working on. And trying to bring a junior engineer into that environment isn't actually serving them. I'm generalizing grossly, but like, typically

Charity Majors 14:23 but by the time you're planning not for weeks or months, but you're planning for years in advance, you should be able to bring some juniors along. They're an investment in the future. They are not an investment that takes years to pay off, but they are an investment. You know, they need a bit more stability, you know, in the same way that like you don't have four zero one ks matching for, you know, you're

Charity Majors 14:48 you're not paying them much, you know, money. You're paying them in. You know, there are a lot of things that are kind of weird about early startups that we know are not how you build a long term sustainable company.

Conor Bronsdon 15:01 And I think this is an important piece of nuance is a lot of folks who are giving us advice of like higher senior plus Yeah. Are coming from, oh, I'm just building my first app here, and I want to run quickly with it. I'm building Everything a small

Charity Majors 15:16 when it comes to advice. It should come with, like, a mandatory disclosure label. This is my background, and this is where I'm coming from. Honestly,

Conor Bronsdon 15:23 we would be a smarter world for that. It wouldn't be nearly as good for hot takes. So you know it's not going to happen. True. And I've heard you talk about this idea that this is something I think a lot of us feel, which is as people accelerate to, let's call it, IC leadership, the senior plus, the staff, the the principal, you know, they start solving these new problems because they have the context.

Conor Bronsdon 15:48 Maybe they don't want to spend, you know, eight hours a day in front of their keyboard or they want to spend a couple hours a day. Like the way they solve these problems changes. They try to, multiply their impact in different ways. And part of that is is something you've said, which is that writing the code is the easy part.

Charity Majors 16:05 Can you expand on that? It's a little spicy. It's a little, you know, it's it's not always true. But I always said I come from ops, right? So that's my bias. It's my background. It's my context. And the fact is that writing code compared to like maintaining it, operating it, instrumenting it, you know, extending it, tracking down really weird bugs, you know,

Charity Majors 16:30 writing the code, greenfield code is typically the fastest, easiest part of the lifecycle. And I think it's no accident that, you know, new technology always reaches for the, you know, the lowest, lowest hanging fruit. What's the fastest, easiest place that I can like show big impact and get raise a lot of money? Well, of course, they reach for code generation because

Charity Majors 16:49 generating code is a lot easier than owning it over the fullness of time. You know, it was the entry point that was easiest and made the most sense. And I think that, you know, Gen AI is what, two, two and a half years around now. Like, I think you're start we're starting to see companies start to, you know, change their focus to what happened after the code has been deployed. And how can we help people understand it faster and make sure it makes sense. You know, and I do think there's going to be this enormous sort of split. I think there's going to be an immense flowering of effectively disposable code where no one's ever going to understand it or be expected to understand it. It's just going to be

Charity Majors 17:29 generated despec. And if they don't like it, they'll destroy it and generate 10,000 more, you know? But then there's the code that runs the world. Right? The code that is itself like a big risk to mitigate. Right? The banks and delivery companies and anything with transactions. Right? And at the end of the day, somebody's got to understand that shit. Like, I can't really see past five years out. I don't I don't I'm not a proclaimer. Right? But, like, I don't see a path within the next five years to just be like, cool. Nobody's gonna understand my banking code. Like, I don't I don't see that. This is interesting because I I I do think there is this evolving

Conor Bronsdon 18:06 difference between, like, consequential systems and less consequential systems. And a lot of the narrative we hear in media and online circles is around con or non consequential systems. Because it's it's more spicy. It's more provocative. Move fast. Yeah. You know, let's break some things. Wow. We're we've been trained to. Yep. Yep. But I mean, like, the advice you'd hear from the CPO of Databricks would be, yeah, build for the startups, but sell the enterprise. And to do that Love that quote. In a B2B industry you might might be in, like, obviously, you need to understand the consequential piece. You need to understand governance. You need to understand security. You need to understand the whole life cycle.

Conor Bronsdon 18:47 I guess I would wonder, as we start to bring in all of this AI generated code, as we start to just adjust how the speed of code generation works, speed of certain parts of our team works, What are you seeing around changing team dynamics to adapt to these new pain points?

Charity Majors 19:08 Yeah. That's interesting. Changing team dynamics.

Conor Bronsdon 19:13 Can can you say more about that? Yeah. I I guess what I'm thinking about is, you know, like, I think we've for a long time, at a business level, it's you know, the reason to hire a bunch more engineers has been, oh, I have a lot more code to generate. And like, yes, I need some of them to be like solve these business problems because I do hire principal staff for a reason. But like, I need to generate all this code. I need to create more features faster. And now, like the actual code generation part doesn't appear to be as big of a problem. Yeah. I think we're seeing that pretty rapidly. Like, sure, debugging, it might be a bigger problem now depending on how you're generating that code. Yep. But, like, the where we have

Conor Bronsdon 19:50 I guess we've we've shifted right in this case on, like, where the problems are. Yeah. Is now it's like it's like, oh, post cogeneration. Yeah. Because now we're having these challenges. Yeah. Like, how do we have to rethink solving that? Yeah. I think that, like,

Charity Majors 20:05 the teams that I think of as being really good, high performing engineering teams, for for them, this is not a new thing. But I think Right. That's not most teams, to be perfectly honest. I I think that the center of gravity being production, not preproduction, not your IDE, not your test, not you know, but like what's happening in production is like pretty central. Right? And the phrase

Charity Majors 20:31 test and prod or live a lie, which I have a T shirt that says that, it's like everyone tests in production, but a lot of folks have denied it for a very long time, which means that they haven't invested in it. And you can try that. I feel like you can often tell where a company's real priorities are if you read their engineering job ladder. I know of so many big companies that I will not name where it was just known that if you really wanted to be an E7 or a principal engineer, you needed to ship a lot of features and give zero fucks about whether or not they actually did anything. Right? You didn't stick around for the long run.

Charity Majors 21:06 And I mean, there are risks on the other side too. If you if you overemphasize operating stuff, then you can, you know but they're one not ones that are as common, you know? And so like you really do need you need a balance. And and the problem is that when you try to codify something that is as nuanced as this, it always comes a lot of it will always come down to individual judgment, which makes it fragile

Charity Majors 21:32 as a system in a different way and in a way that business types really are less comfortable with. Right? It's a lot they're more comfortable with being wrong, but like solidly wrong in a direction that's fairly predictable than with like you're being nuanced and well, you have to make sure that the people you hire are good. But I actually think that this is sort of related to why

Charity Majors 21:52 the expectations of engineering managers have really shot up in the past few years. I think there are a lot of reasons for this. There are a lot of different ways. You know, for a while, was like, well, engineering managers do the people problems and engineers do the technical problems, which does not work because you can't actually disentangle or distinguish them. Right? So like our expectations of managers and their technical abilities has escalated. Our expectation of managers and their ability to, like, create an emotionally safe environment.

Charity Majors 22:21 And, you know, they've escalated pretty rapidly. And I think that this is kind of part of it because we're just I think socio technical systems are extremely complex and that some of that complexity is irreducible.

Conor Bronsdon 22:37 As we rely on AI assisted tools, as we rely on AI agents and, you know, these other AI workflows to do different roles for us to automate different parts of what we're we're looking seeking to do. We're abstracting ourselves away from some important aspects of of different key delivery points. And I've heard you mention this concept of the cognitive decay. Others have talked about this and the risk around this with overuse of AI tooling without structure around it. What do you view as the implications for the long term development of

Charity Majors 23:11 engineering of the discipline? Yeah, there's a lot of science around this that shows that, you know, if you move from doing the work to just supervising the work, your ability goes down, like your expertise decays. This is not like an inevitability, though. It's something that, again, this has been around as long as automation has been around. Right? And, you know, I'm writing this talk right now for SRECon later this week because SREs

Charity Majors 23:38 sit in this interest they're risk managers. Right? And my pitch to SREs is like, we, with our elevated awareness of risk at all times, we need to make sure that we are like good lawyers. We don't tell you no. We tell you how to get to yes. Right? We help you choose tools with workflows that will help make you a better engineer because it's bringing you along with it. Right? You can imagine like an SRE agent agentic AI thing that's like supposed to be your SRE.

Charity Majors 24:09 By the way, this is good, right? Most engineers don't have access to world class SREs. If you bring an agent that's only 20 or 30% as good as, you know, a great world class SRE, great for the world. But you can imagine an agentic SRE thing that's like, does it for you. Right? And you not only don't become better as an engineer, you weren't asking the questions, you become worse as an engineer. And you can imagine ones that like are like, Here's what I'm seeing. Have you tried this? And sort of bring

Charity Majors 24:39 you with it so that the longer you interact with it, much like if you're interacting with a great SRE, you'd be learning things about service discovery and exponential back off and retries and monitoring alerts and paging alerts and, you know, so much of software. You you touched on this a few times, and I don't think we really said it out loud, but like, software systems are feedback loops, right? The whole solution of your technical system is a big feedback loop made up of a lot of other feedback loops. And feedback loops can be dampening or leveling or amplify. And most of these are amplified, which means that the farther upstream you find it,

Charity Majors 25:19 the bigger of an impact it has downstream. Right? Co review is feedback. Right? Deploys are like the grandfather feedback. Right? And the more you can tighten these feedback loops, the less cognitive overhead it takes, the faster you can move. You know, I use and observability is really like the feedback loop because it's the meta it's always feedback looping while you're feedback looping. Right? And like analogy I use a lot is like glasses. Like I'm whole blind. Right? So when I want to go driving down the freeway, I put on my glasses. Right? So that I can make course corrections

Charity Majors 25:54 it and doesn't even feel like a woah,

Conor Bronsdon 25:56 what?

Charity Majors 25:57 You're just driving, right? And great source of your technical systems have good observability and good tooling. It feels like you're just building, right? Doesn't feel like you're off down rabbit holes just trying to you know, there's so much toil involved. And I feel like part of what gets missed in the discussion of who we hire and why we hire them is that

Charity Majors 26:22 our company is our feedback loop. They're sociotechnical systems, too, right? It's not like the person enters and they are who they are. They will always be that person, you know. And yet we talk about it that way. Instead, it's like someone enters

Conor Bronsdon 26:37 and they become someone else while they are there working. And they change the social techno systems that are occurring in there potentially, too. Every person that changes, like, the melting of the technical systems and the social ones can change. Yes.

Charity Majors 26:51 Exactly. And and there's this, like, there's this internal incoherency in most hiring managers' minds where they really want a senior engineer to show up on their doorstep who's already done all these things before and could just be dropped in, pick it up immediately at the drop of a hat. But actually, really long term, they want to hire people who don't just want to do the same thing over and over

Charity Majors 27:15 and don't just want to be faced with the same you know, those are not your high performers. Your high performers are people who push themselves, who are you know, in the last time I was going out looking for a job, I got so pissed off because everybody wanted to hire me to manage an engineering SRE team that it was using MongoDB. And I'm like, I just spent three years doing that.

Charity Majors 27:38 You just it's just really what you want people to just that's not who I want to hire. I want to hire people who are restless, people who learn, people who aren't satisfied and comfortable with they're doing. Right? And so I feel like you kind of have to pick. Do you want to hire people for their potential and their hunger? Or do you want to hire people who want to do exactly what they're doing? Right? Because

Charity Majors 28:01 I think I'm being a little bit binary. It's not actually that binary. A little bit of both, as you said earlier. You want to hire people who are fast learners, who wanna do new things, who are up for new challenges, who are who are energized and excited by that, not not not frightened by that.

Conor Bronsdon 28:19 Yeah. And and I think it's something we often forget in these conversations, that are occurring right now about how to change teams, how to adapt to AI. Like, often it focuses on the leverage you can get from adding these technical systems to the social system of the, like these senior engineers. And there's a ton of leverage you can deliver there. No question. But

Conor Bronsdon 28:41 if you want multiplicative leverage, you're probably gonna find it from bringing someone on who is more junior, who has an opportunity for more growth in their career Yeah. And giving them a technical system that can also multiply that leverage. 100%. Much more cheaply too.

Charity Majors 28:55 100%. And if we're talking about really squishy things and the loyalty and the energy and the like, these people aren't coming in just like, well, I'm gonna put in six months here, get my blah blah. Like, if you invest in them,

Conor Bronsdon 29:11 people often want to invest in you back. Yeah. And one of the crucial things to enable them with is something you brought up earlier and something we've kind of touched upon in this conversation throughout, which is observability into their system so they can understand how to grow, how to improve, and and how to set up their team for success. And, obviously, you and Honeycomb have been leaders in this space.

Conor Bronsdon 29:31 But the concept of observability and engineering is evolving, especially in the context of AI driven systems. What are you seeing? You know, we talked earlier about how the center of gravity is shifting from preproduction to postproduction.

Charity Majors 29:44 Like, the source of truth is what is happening right now. And I think that a lot of software engineers haven't really fully internalized that. And they feel very comfortable in their IDE, in their editor, running tests. And then, oh, that's someone else's. But I feel like in some ways, one of the biggest challenges that is being generated by this influx of generated code is

Charity Majors 30:08 teams are used to debugging using one path, which is, something just changed. Who changed it? And who knows exactly what happened and why? I will go find the expert in this thing that just happened and they will figure it out. And what if there's no expert? What if no one actually understands it or ever understood it? And what if, you know, what if you need to understand production on its own

Charity Majors 30:32 terms? And that's all you've got. And there are some really interesting things, implications here for instrumentation and for what metadata you attach and all this stuff. But like, this was kind of the founding story of Honeycomb almost a decade ago because Christine and I met at Parse, mobile back end of the service, million mobile apps running on it. And every day, you know, we had developers all over the world just like writing snippets of JavaScript and uploading them. And we just had to make it all work.

Charity Majors 31:01 We couldn't track them down. You know, and these are you know, some of them are interns, some of them are principal engineers you didn't know. Right? And like, what you need to understand that code when it hits your system is very different. And this is, I feel like, really what the entire world is grappling with at this moment.

Conor Bronsdon 31:19 I'd love for you to dive a little deeper here in your perspective on these AI driven systems and how we need to do more on the observability side or maybe adapt. Obviously, adding this element of nondeterminism adds a lot of complexity. What's your perspective on what's right for this future of AI systems?

Charity Majors 31:43 You put your thumb right on the pressure point there, which I said so much of this is just like automation. Know, we've seen the story before.

Conor Bronsdon 31:51 For that 10%, know. Except for that little bit about nondeterminism,

Charity Majors 31:55 right? Yeah. And like the whole premise of APIs is that you put in the same thing, get the same thing out. And I, you know, my my coworker, Philip Carter, wrote this great blog post about I think it's called All Under Weird New Kind of Computer. I don't know if you saw it, but it's incredible. It's been like stuck in my I owe him rent for sure. It's been stuck in my head for like two weeks now. But like, it's so true because, you know, where traditional computers are really good at very precise things and very bad at fuzzy things,

Charity Majors 32:26 OLMs are like really bad at pretty very precise things and really good at fuzzy things, which is a lot more human shaped than traditional computers. And like that intersection of determinism and nondeterminism, I don't have any great answers here. If I did, I would probably be a multimillionaire. I am not. But it is definitely the interesting thing to be watching over the next couple of years.

Conor Bronsdon 32:48 Definitely. Yeah. Our our research team has been doing a lot of work around like essentially trying to increase the efficiency of LLMs being used as judges within this context. But it's it's so important to not solely rely on these, like, automated nondeterministic judges, even as they judge other nondeterministic systems. Yeah. You have to have that continuous learning loop with human feedback inserted. Yeah. In order to, like, really effectively evolve these systems, I think. Pitching it as an

Charity Majors 33:17 analog to analogous to the advent of the computer is I think, you know, a lot of engineers out there are pretty grumpy about AI stuff. They're sick. You know, they're hit the hype, the you know and like, I get it. But like, it's kind of a reality that in tech, the scale of the hype often correlates with the eventual scale of impact. I think there's a reason people compare it to the Internet. Right? There's a reason people compare it to the Internet or the advent of the computer. And

Charity Majors 33:43 not dealing with it or writing it off as just type is not going to help. It's not going to be you know, and I feel like for me, I've come a long way on this. I started pretty grumpy about it. And now I'm like, it is an opportunity. It is a massive opportunity. And it is one that if people are choosing to ignore, they should have their own very good reason for it. But I feel like some people, a lot of people are writing it off because they're cranky about it or tired of hearing about it. I don't know that that's a good reason.

Conor Bronsdon 34:14 Yeah. I I think it's really easy for us to get stuck at a certain point, whether it's in our careers or in our lives. And this to me speaks to I know it's very common to use this phrase of lifetime learner or growth mindset. But I think it's especially crucial in this current era where we're seeing this massive transformation

Charity Majors 34:34 once again from major technology. It's not going be the last time we see this either. There's more waves coming here. And I get it. I did not grow up with a growth mindset, and sometimes it's exhausting. Sometimes you need to be able to get it. And it's tiring, you know? I get it. And and also this is I feel like why it's important to be a well rounded adult with other life skills, like cats

Charity Majors 34:55 and family members. And, you know, you've got to be able to turn it off. You've got to be able to ground yourself because if you want to have a long career in technology, this is what it takes.

Conor Bronsdon 35:06 Absolutely. And as a brief aside here, if you're listening to us right now and you're not watching us on YouTube, you're absolutely missing some incredible cat action in the lower left corner of my screen where you can't quite see, but you can maybe a little bit there there we go. Oh, hello,

Charity Majors 35:22 Pretty. Oh, a pretty kid. Oh, anyways.

Conor Bronsdon 35:26 All right. Back to the upside. Charity and I get completely distracted. So as this as you've put it before, unprecedented influx of new software occurs as we enable everyone in the world to build software more easily, even more easily than we did with like low code tooling. Now it's just like, hey, I, I have a buddy I can talk to that's going to do this for me. I have that co founding engineer. How does this affect

Conor Bronsdon 35:55 Honeycomb's mission the kind of goals of what you're seeking to do next?

Charity Majors 36:01 Yeah, it's a great question. You know, I think that over the next five years or so, you know, we talked earlier about this sort of bifurcation into like disposable code and the code that runs the world. We have always been for people who need to understand the code. Right? I think the big question mark in my mind is around I'm still trying to wrap my head around agentic stuff. Like, I think it's interesting. Philip and I have had a bunch of interesting spirited conversation and arguments about whether or not I think it's entirely possible that the action will move out of browser

Charity Majors 36:33 based SaaS, Right? And into I don't know. Previously, I was like, maybe I'll move into the IDE and we'll be able to just like, as we're writing code, see what the impact would be on production if I made this change. That would be super interesting. I agree. But now maybe it's moving out of the you know, I don't know. I don't know. What I'm not scared about is

Charity Majors 36:54 a future where people who understand software are not valuable. Yes. Yeah. I don't know exactly what it's going to look like. But if there's anything I am sure about, it is that understanding how to fix software and and garden it and care for it over its life cycle is

Conor Bronsdon 37:15 still going to be needed. And observability is, I think really the broad term for what you're saying here is this ability to understand what's going on and the tooling that Honeycomb provides and that startups like us are working on for AI is going to be continually valuable, no matter whether you're an engineer or a product manager or whatever else. There is an opening here for sure. And I think like the trajectory of like the observability industry, I think, is going towards, you know, instead of all these really expensive little pillars, we're going towards, you know, a single consolidated

Charity Majors 37:50 data lake shaped storage where engineers interact with the data, but also it sort of unifies business data, marketing data, sales data, product analytics, because context is what makes data valuable. The more connected a piece of data is, the more valuable it is. And right now, I think we're going to look back in ten years and go, God, can you remember when every request entered

Charity Majors 38:14 our business, we stored in like 25,000 different places? And we tried to make decisions at right time about what questions we were going to need to ask years and like, that was insanity. It was so costly and so fragmentation of like, you know, and like, we've spent so much more time arguing about the nature of reality than we did about the problem we were trying to solve, and it's just absurdity. Right? Absolutely.

Conor Bronsdon 38:37 Yeah. I won't name names, but, I was talking to, these engineers at a racing team when I was at NVIDIA GTC last week, and they were talking about exactly this problem. They're like, yeah. You know, as we've developed our vehicles over time, we have all this data from thirty years ago, from twenty years ago, and it's it's like, it doesn't all work together. There's all these different eras.

Conor Bronsdon 39:01 We need to figure this out. Like, we wanna apply Gen AI to maybe, like, help us do some of this cleanup, but also, like, what are we doing here? Yep. And, I I see that problem across the board, whether you are, an internet native company that has just added instrumentation, added new layers of observability over time, or whether you are a company that started pre internet and has these other implications where they've got hardware and software involved. And it it just does seem like there is this incredible opportunity for

Conor Bronsdon 39:30 data to become increasingly useful, increasingly stitched together, and for the context to continue to just drive where we're going here. Yeah.

Charity Majors 39:41 Agree. Plus one.

Conor Bronsdon 39:43 Charity, this has been fantastic. I really appreciate you coming on the show and sharing these insights. It's always so fun to chat with you. So fun. Do have any closing words you want to share with us or advice for engineers who are navigating today's

Charity Majors 39:56 AI explosion? We started out by talking about my article about generative AI is not going build your engineering team. And I want to close by kind of circling back to that and saying that I have met several junior engineers in the last year or two who were hired in, who have bootstrapped, who have come up to speed. Every single time, it happened because not a manager, not a director,

Charity Majors 40:22 because a senior engineer decided to make this important to them personally. And they lobbied for it. They volunteered to help source, interview, train, mentor. And you look at the smiles on their faces and they're just like, Yeah, it was worth it. It's great. We're going to do more. We have total support from management. And I just want to do it. I know this is traditionally seen as a management problem.

Charity Majors 40:48 And maybe it is or should be. But I think it's an engineer solution. I see individual contributor engineers as the ones who are driving the only real solutions out there. So if you're out there, it just matters to you, know that I think you can do it.

Conor Bronsdon 41:05 I love that. That's a fantastic message. For everyone who's been listening or watching, make sure you follow Charity's work, whether at charity.wtf, honeycomb.io, and some amazing stuff they're doing, or all across the Internet. As always, thank you so much for listening to Chain of Thought. And if you are listening, consider watching every episode on the Galileo YouTube channel so you can see my cats

Conor Bronsdon 41:26 and others as I accidentally spook her out of her sleep. Episodes drop on Wednesday. It's the same time they drop in your podcast feed. Charity, thank you so much for joining us. So much for having me. It's super fun.