Ep 32: Revolutionizing Business Operations With AI and Automation with PagerDuty CIO Eric Johnson

Saam: Hi there, and welcome to Enterprise Software Innovators, a show where top tech executives share how they innovate at scale. In each episode, enterprise CIOs share how they've applied exciting new technologies, and what they've learned along the way. I'm Saam Motamedi, a general partner at Greylock Partners.

‍Evan: And I'm Evan Reiser, the CEO, and founder of Abnormal Security. Today on the show, we’re bringing you a conversation with Eric Johnson, CIO at PagerDuty, the digital operations management platform. 

Eric is a longtime technology leader and has been the CIO at many notable companies throughout his career, including Informatica, DocuSign, and SurveyMonkey

In this conversation, Eric shares his thoughts about the impact of data strategy on AI adoption, transforming business operations with automation, and best practices for using generative AI in the enterprise,

Well, first of all, and I mean this very sincerely, Eric, I'd really appreciate you taking the time to, uh, join us today. I've been looking forward to this conversation for a while. So first of all, you, you've been a CIO of many notable companies, including Survey Monkey, DocuSign, Informatica, and recently PagerDuty.

Can you give us like an overview of your career, maybe how you got to where you are today? 

Eric: Sure. Um, I actually really started my career in the management consulting area. I, you know, first, first job out of school was, um, working at Deloitte Consulting, which was a great way to start your career. Lots of industries, lots of different people, lots of different challenges.

It was almost like a graduate degree and in, you know, how to, how to work, uh, in business. And so it was a great start. And from there, um, really took that mentality around management consulting and, and looking at, you know, the holistic approach to how you deal with technology. Technology, in my opinion, it's one piece of a puzzle and I don't think this is anything new to people, but I think sometimes it's forgotten in this world we live in where it's technology, technology, technology, that there is such an element of people and change management involved. And I think even more so now, along with process, I can't tell you how many times I've, I've walked into organizations or situations where people are screaming that the technology is broken only to come to find out the process is broken, pretty severely. And the people are broken pretty severely. And there's a whole lot of work there before you even talk about the technology. So I say that because I sort of learned that pretty hands on when I was at Deloitte. And I think that has carried with me through my career, you know, working for a lot of innovative tech companies that grew in all kinds of different ways.

I've worked with some amazing leaders. I've worked at some amazing companies and I've got to really experience sort of what good looks like. And I think you can't ask for more in your career if you, if you've had that opportunity. 

Evan: Do you mind sharing a little bit about what PagerDuty is and kind of how you help your customers?

Eric: Yeah. PagerDuty, it's a company that started out as this notification technology that allowed folks to be aware of an issue going on, finding out who's on call and making sure that things were being addressed. And, uh, people were alerted to a situation. Obviously the company has evolved quite a bit since then.

And as the company continues to evolve, it goes from not just being able to tell you there's a problem and alerting you to a problem, but getting into a place where it could even see that there's a problem, understand what the problem is, and potentially have workflow that you could build to help you resolve the problem.

And then I think the longer term play here is, do we get to a world where you have all the sensors and all the information coming from your infrastructure, but you can also identify, hey, there looks like a problem that's about to happen. And actually being able to stop a problem from occurring. Maybe you got a infrastructure a server or some data integration, Whatever it may be that looks like it's not working like it should and there's early indications You could have a problem and being able to fix the problem before it causes an outage. So I I think you could start to see this kind of Evolution of the solution from just simply telling you, Hey, we got a problem and alerting you to helping you then to fix the problem in an automated way.

And so I think it's a really interesting space that they're in, and I think it's a space that's going to very much be a must have technology or at least a must have area of many companies business operations. Because the complexity of what we have today has just gotten so complex. You know, the days of having a server sitting in your data center, you could go over and look at, see if the light's blinking those days are for many companies that day's over and it's still there, but they also have stuff in the cloud that they're having to talk to.

At PagerDuty, I think there's a strong belief and I think a really strong position that the company's in around really being able to support companies that are starting to move into a much more complex and more modern architecture with processes and things that are going on even now in a world where and where it's very much a remote world.

People aren't sitting in offices as much anymore. So having technology that can bring together an alert and identify where problems are across a wide network of folks in a lot of different locations, I think is also going to become pretty important. 

Saam: Eric, like there's a lot to, to double click on. And I think I want to zoom up a little bit and talk about the technology trend of, of the moment of the year, maybe of the century, which is around AI.

What are the top two or three areas you think CIOs need to be thinking about today in order to set themselves up for success with AI in the next 18 months? 

Eric: Yeah, you know, this is obviously something super top of mind. At the end of the day, I'm someone who leverages technology as one of the key tools that I use to drive business impact.

And the reason I start with that comment is because I think at the end of the day, CIO's job is to drive business impact. Measurable business impact. So I think it's always something that many CIOs are cautious about. We've all lived through lots of different trends in technology. There's always a hype curve that occurs when technology comes out.

AI is not something that's necessarily new. AI has been around for a while. And so if I was to talk to, you know, my CIO peer group, the first thing I would talk about is taking a step back. And really thinking through your broader data program, one, how are you using data to drive business impact, two what are very clear use cases that you could be focusing in on around generative AI? 

And the reason I say that is because I think what's going on right now, at least my experience is there's a lot of excitement around generative AI, which there should be because it's a very powerful, very sophisticated set of solutions that I think are gonna have really change not only just like the business world that we live in it's going to change people's lives. But I think right now if you're not careful, you can get sucked into this Situation where you're spending a lot of time and energy running around looking for a problem to solve and so my experience so far to date has been more of a purposeful approach to what, what business problem are we going to solve?

What opportunity are we going to go take advantage of with this new technology? So I think you do need to really start with the problem you're trying to solve as opposed to, Hey, I have this new fancy piece of technology. Let me go look for a problem to solve. So, I think the last thing I would add to that too is I have come across companies and organizations where data and management of that data still is not something that is seen as a top priority.

And the reason I bring this up is because if you have a data program that is not incredibly diligent in managing the quality of the data, the security of the data, and really the curation and trust around the data, I don't know how strong of an AI program or generative AI program you're ultimately going to have.

It kind of goes back to some of the fundamentals. If your fundamentals aren't in good shape, it's going to be really hard to take advantage of some of this really, really cutting edge, innovative technology, and I think data is going to be something that CIOs are going to have to get really maniacal about and super focused on making sure that it is world class.

Saam: When I think about AI's impact to transform organizations, I think about it on two vectors. One is transforming how you operate your business, and the second is transforming your products that are used by your customers to operate their businesses. And so, are there one or two ideas you would share with the audience that could be a quick win in terms of how someone can use AI to transform the way they're running their business?

Eric: Yeah, I think that there's things that can be done today. One of the things I found that I don't know if it's a quick win, but I think it's a win is really sitting down and having some very specific conversations with the business, especially around the world of sales and marketing and customer success.

You know, what are some of the key business drivers that they have and how are they measuring things like churn? How are they measuring things like expansion of their base, you know, of customers, how are they identifying new logos, having an understanding, and this is kind of gets to that business impact piece.

Really having a deep understanding of how the business is measuring success. And then building data products to support those things and move the needle. I think that's kind of the key takeaway. You know, I think there's a lot of I. T. organizations out there that are really great at like delivering things, but not really great at measuring the true impact that they're making.

Evan: As you mentioned, a lot of the excitement around AI is around some of these generative capabilities. We'd love to hear how you envision, you know, PagerDuty using some of these new capabilities, you know, just maybe generally AI capabilities to optimize, you know, the product today that you mentioned earlier, something around kind of AI ops. You know, I'm just curious what that is and you know, what you kind of see is like the future opportunity for PagerDuty to take advantage of these technologies to improve the product or offering. 

Eric: Yeah, I, I think there's kind of like phases of it. For example, today, being able to identify all this information and synthesize it into what was the actual problem?

Where did it actually occur, what was the root cause of the problem? And then bringing all that information together, coupled with automation to then very, very quickly drive channels to be open, documentation, to be pre-built and, and being able to move from an incident occurring to an incident being closed and, and remediated.

So, using AI to be able to identify all of these data points, all of this information coming in, we believe helps our customers go from, I've identified a problem to I've actually remediated and closed a problem much, much faster. So I think that's kind of like today. And that's coupled with not just AI, its workflow, there's a bunch of things that sort of come together to build that solution, but AI has a role in that.

As you move out into the future where you can start to identify where a problem has occurred, you have workflow that can then address the problem for you on your behalf. You could start to then even see that time shrinking even more. Right. And that's also working with AI and data to be able to correlate a lot of different data points, identify what's going on and then, and then having very intelligent workflow that can actually go in and, and, you know, turn a server off or block that account or whatever it is, it's causing the incident on your behalf. And I think the complexity of what's going on in this day and age around infrastructure and process and people being distributed, I think only contribute to the needs of companies to be able to have technology like this.

And then as you breach into the opportunity around more advanced AI, where you could start to actually identify where there's a problem that's about to happen. It's seen pattern recognition. It's seen data from historical models that say, Hey. If we continue to see these things happening, this is going to lead to an outage or a problem.

Let's go ahead and fix that issue or stop that, whatever's happening that could contribute to that outage ahead of time. I think that's where the world of PagerDuty is going to be really leaning on, you know, being able to look at lots of data. And we're talking a lot of data too, right? We're talking about sensors and all kinds of information and alerts coming in from all kinds of different systems and tools.

And partnering up with, you know, a lot of other logging and monitoring solutions that are out there to be able to actually then drive. I think that's where I see PagerDuty in the short time I've been there. I've been really impressed with not only what they're delivering today, but also what they're building into the product for the future around this whole concept of, uh, you know, cloud ops as kind of a bigger. Solution. And then there's a lot of capabilities, AI ops being one of them. But, um, I think, you know, any CIO out there is looking at not only the AI side of it, but also the automation side of it. I think those in many cases, those go. Can be very much go hand in hand. I find that a lot where I'm like talking about Oh, you know, we're gonna we're gonna see something.

For example, we're gonna see a customer that's maybe about to churn. Great. I have a signal now, what do I do with that signal? I need to actually drive some automation and the business to actually do something with that signal, and more and more I think that's gonna start to I think I see a world where we're going to be automating a lot of the enterprise, right?

A lot of the operations in a, in a, in a, in an organization more and more is going to be done through automation and the use of data and AI. So, That sounds a little scary, I think, for some folks. I see. Remember I talked earlier about change management. That's going to be the big challenge, I think, as we move into, into the next, you know, five to seven years.

I think it's, we're going to have a ton of really powerful technology, but you're, you're, you're going to then have to really focus on the people and process side of it and say, what does this mean to the new workforce that's, that's out there, you know, in the future? How are they going to react and how are they going to coexist with this technology?

I, I think we're underestimating how much impact this is going to have. 

Evan: Eric, do you mind sharing a little more about that? As a technology leader, right? Part of your job is to think about the organizational developments, right? To make sure you're equipped for the technology of the future, not just, you know, how do you get stuff done today?

And so what are your thoughts around preparing your organization or maybe the broader company to embrace and leverage some of these new AI technologies. Like, is there some training or enablement? Or do you see kind of new roles appearing in your team? How do you think about how the average IT organization looks different two years from now versus today due to some of the kind of changes in these technologies?

Eric: Early and often, Evan. I think one is, is really doing everything you can to start to build a communication with the company, not only your own department, but other departments. Change is something that you just have to understand is, is a constant. The pace of technology change is fast. And now with more and more new technology coming out that I think companies are starting to get better and better at managing and leveraging things like automation and AI and some of these other tools or that are topics of the conversation today, I think you have to sit down with the organization.

You got to kind of talk about what that kind of change is going to be in the organization and making sure the executive team is on board with that. If you can't get that done, then you're, you're probably not going to get very far on that sort of transformation. So let's assume that everyone's on board.

And then I think it's a really a matter of, and I've saw this in a, on a number of companies I've been at, where, you know, as you get into department by department automation changes, using technologies like RPA with bots, and I've had experience with that, sitting down with the leaders in those functions and explaining what the benefits going to be and what some of the change management challenges will be.

And a lot of times it's educating people that we're not going to put you out of a job. We're going to change your job. Your job's going to be different. There's going to be new skills you're going to develop. There's going to be opportunities for you to, to potentially work on higher value programs and processes.

Um, but the world of like maybe doing accounts payable or accounts receivable is going to fundamentally change. And I think that was the beginning of it. And I think some people were on board with that. And they said, yes, I, I, I'm, I'm interested in sort of what this means. Others were 100 percent not interested in it.

I saw it even in my own IT organization, doing things like a virtual agent, which is now a very popular thing. And a lot of CIOs are doing where it's like, when you call the help desk or you engage with a... Uh, the, the IT helped us through Slack, a bot now is responding to that, similar to like, you remember the airlines used to go and ask, you know, Hey, I need help and you go to chat and it wasn't a person, it was a bot, you know, that, that sophistication around virtual agents is only getting better and better and more and more vendors are showing up with that sort of capability. And it's actually, I've seen some amazing results with technology like that, but what does that mean? That means there's a fundamental shift in people in what they're doing. So the answer to your question, I think you, one, you have to kind of really be clear about at the highest level about what this change, like, what does this look like, where do we want to go?

Why do we want to go there and then get that executive support? And then you need to work into the organizations and really, truly be honest and open with people. And you also need to kind of look very carefully at the change management around re skilling people in many cases. 

I do think there's going to be new roles. I mean, do you think there was ever going to be a role of someone that was going to be sort of a bot administrator? Did you ever hear of a job like that five years ago? But those are jobs that people have now. So, It's one of those areas that I do spend quite a bit of time on because if you screw that up, honestly, I think it really creates a lot of friction for the organization.

Saam: I feel like one of the things I hear from CIOs often is like both this excitement around AI and automation, but also this frustration around how do I translate that into concrete ways I can change and transform my business. Obviously, PagerDuty plays an important role in the software development lifecycle, and we've seen, you know, GitHub, Copilot and these other examples of how generative AI can impact software development.

But I'm curious from your perspective, both as you think about empowering internal development teams at PagerDuty and at other companies you've worked at, as well as kind of what a company like PagerDuty could do to make developers more productive, what are two or three of the ways you think generative AI could impact the software developer?

Eric: I mean, you spoke about a number of them. There's a bunch of generative AI tools that are out there that are helping, you know, the copilots of the world that are helping developers to write code. I think you're going to start to see more and more stuff around even security around doing, looking at, you know, is, is code being written securely almost in real time?

I mean, we've talked a lot about PagerDuty and sort of where it fits in the world of, you know, helping as developers build infrastructure and us being able to tap into that to be able to do the things that I talked about earlier, that PagerDuty, I think, is going to ultimately be playing a bigger part of. I think we have to be careful though.

There's this belief that AI is going to transform. And I think there are elements of it for sure, but I don't know how great AI is at taking a bunch of old legacy code and untangling that. This gets back into the, the whole problem that I think when I talked about the hype curve earlier, where gotta be careful not to assume that AI is sort of the, the utopia, generative AI is the utopia, um, because I do think there are still a lot of limitations to it.

Structure data, not great at. Some of the generative AI stuff, at least what I've read. Talk to some of the development teams. It's great. If you want to write new code, not as great when you're trying to untangle legacy code. And Oh, by the way, most of the code out there is old legacy code, right? I mean, generative AI is just showing up on this scene, but you know, stuff that was built 20 years ago, you still need a human to kind of go through and untangle that many times.

So. There's lots of data companies out there that, you know, AI has been around for a while data science. There's, there was a lot of passion around that. Now we have generative AI. I think the next big wave of technology is going to be, how do we now use all of this technology to really help to drive amazing efficiencies for the people that are doing the work. And I think you sort of tied into that, like developers being able to have these tools that are guiding them through the process and saying, Hey, I've prebuilt this code for you. Um, and as they're, as they're in there doing it, like you said, it's, it's, it's telling them based on your policies and security standards, you need to fix these things or we've already fixed them for you so that you're developing really, really highly secure, highly efficient code. And I'm not a developer, but I can imagine that that's going to drive efficiencies and productivity gains. Also, the thing I just talked about are RPA technologies.

All of a sudden you have a finance organization that's able to process 10 times the volume of transactions they used to with the same number of people. So as you're growing the business, you become more and more efficient. 

Customer success people and IT people. People are calling them, asking questions. They're able to do, you know, 10 times the amount of tickets that they used to with the same number of people because they have bots that are supporting them. So does it remove people completely out of the equation? Maybe someday, but I think for the foreseeable future, it's going to be something that really compliments our ability to drive amazing efficiencies and scale and guess what's happening right now and Saam, you know, this, Evan, you know, this, I see it every day.

The world of like growth at all costs is over. It's about profitability, being efficient and growth together. So we're like back to what was logical and normal before the craziness that occurred over the last 10 years. And so what's that mean? It makes things like process efficiencies and really driving economies of scale sexy again.

There was a period in time over the last 10 years where it was like, buy every piece of technology, throw it in, use 5 percent of it, who cares? It's all about growth, growth, growth, growth. We got to go get more customers at any cost. Well, I'm finding now. People are coming and saying, Hey, I need you to help me to right size all these technology investments.

I need you to help me right size our business process. I need you to help us right size the way, uh, we're, we're running the organization to become much more efficient. And so I think some of the, what we're talking about, this technology is really, really going to help us do that as CIOs and as, as leaders. In our respective businesses. 

Evan: Appreciate you sharing, Eric. I want to switch over to kind of our lightning round because I have 10 minutes left and uh, looking for some kind of punchier, uh, one tweet responses. Saam, why don't you kick us off? 

Saam: Yeah, sure. So Eric, maybe to start, how do you think companies should measure the success of a CIO?

Eric: What business impact have they created and measured? Simple as that. 

Evan: What's one piece of advice you wish someone told you when you first became a CIO? 

Eric: I think I always knew this, but I think it's been reinforced time and time again. You only win when you have the best team. No technology out there that's going to do it.

There's no process out there that's going to do it. It really does come down to having a great team. And that means chemistry. That means the right skill set. That means the right attitude. But really, as a CIO, like that, that is one thing time and time again that I have learned. Where I've had success, I've always had the best team.

I've hired the right complement of people with the right personalities, the right attitude. And we built a great culture about, you know, delivering a result and delivering value. And I think, you know, when you build teams like that, I think it truly is probably the most important thing you do as a CIO, period.

Saam: How do you think CIOs should position themselves to best collaborate with the rest of the C suite? 

Eric: Don't talk technology. 

Evan: So Eric, maybe switching gears more on the personal side, what's a book you've read that's had a big impact on you and why?

Eric: There was a book around five dysfunctions of a team, I think was the one that I'd read recently, you know, starting a brand new role, uh, that, that, that is a very time consuming process to onboard into an organization, especially when it's, um, you know, growing so quickly and has had, has so many opportunities in front of it.

So I've been over the last, you know, four or five weeks been incredibly consumed with that, but that was probably the last one that I remember. 

Saam: What's an upcoming new technology? And it doesn't need to be AI related that you're personally most excited about. 

Eric: I love the virtual assistant stuff. I mean, my life is busy as hell.

I'm sure both of yours are too. A lot going on at home, kids, people going off to school, college, just like a general busy life and, you know, across the board. It's very fascinated by this world of personal assistants that could be virtual. It's a bot. It's something you could say, Hey, make a dinner reservation over here.

You know, make an appointment over there. Deal with this, deal with that. Like, it'd be interesting to see what happens, especially the world of generative AI, I think is gonna like supercharge that all of a sudden. Can you imagine a world where we have our, we have our own personal assistant that, you know, you pay like three dollars a month for a subscription and they just handle like a bunch of your Life's to-do's.

I don't know about you guys, but I had usually as I'm sitting here, I have a laundry list on a piece of paper of all the to-do's I have to do around sending an email to this person and checking with this and checking with that I'm so on a personal level, I'm really fascinated and and and interested to see how that world changes.

Evan: I think when people think about virtual assistants, they kind of think about it in terms of the use case today. I think about these virtual assistants almost like this universal adapter between the digital world and the analog world. I have the world's best doctor, but there's no like technology interface.

If I want to schedule an appointment, I have to call up, you know, his office and set it up. It's kind of like disconnected, right? But the virtual assistant can bridge that through the phones and the fax machines and the emails, right? And things that you. They don't have like APIs, right? So I think about this like universal adapter between, you know, software, right?

And kind of like the, the real world we live in. So I think there's a lot of opportunities 

Eric: there. And that real world we live in is, is, is the bulk of the, it's a bulk of what we deal with. The bulk of the world we have today is analog in our personal life. Doctors and your insurance agent and your property taxes and your everything is still very analog.

There's so much of our lives that are still analog and I agree with you. There's going to be this continued shift around how do you move all this amazing digital to be able to engage with that analog. There's things that we can't even imagine today that are going to be possible in the next decade.

Evan: Eric, unfortunately we're out of time, but I really appreciate you making time to, uh, chat with me and Sam. Great to see you again and looking forward to chatting again soon. 

Eric: Nice to see both you guys. Take care of yourself. Have a nice rest of the week.

Evan: That was Eric Johnson, CIO at PagerDuty. 

‍Saam: Thanks for listening to the Enterprise Software Innovators podcast. I’m Saam Motamedi, a general partner at Greylock Partners.

Evan: And I’m Evan Reiser, the CEO and founder of Abnormal Security. Please be sure to subscribe, so you never miss an episode. You can find more great lessons from technology leaders and other enterprise software experts at enterprisesoftware.blog.

‍Saam: This show is produced by Luke Reiser and Josh Meer. See you next time!