Ep 26: The AI Playbook with Databricks CIO Naveen Zutshi

‍Saam Motamedi: Hi there, and welcome to Enterprise Software Innovators, a show where top technology executives share how they innovate at scale. In each episode, enterprise CIOs share how they’ve applied exciting new technologies and what they’ve learned along the way. I’m Saam Motamedi, a General Partner at Greylock Partners.

Evan Reiser: And I’m Evan Reiser, the CEO and Founder of Abnormal Security. Today on the show, we’re bringing you a conversation with Naveen Zutshi, CIO of Databricks. Databricks is a leading AI enterprise software company enabling organizations to build, deploy, and maintain data solutions at scale. In this conversation, Naveen shares fascinating use cases from the next generation of large language models, how AI can transform every aspect of enterprise work, and the best ways to stay knowledgeable about the cutting edge of AI. Alright, so maybe you can kick us off, do you want to share a little bit about the background of your career and what your current role looks like at Databricks?

Naveen Zutshi: Yeah, so a technology leader, did my BE in computer science, so have been in the Bay Area now over 20 years, so time flies when you’re having fun. I’ve worked both in technology companies, both in startups, in R&D, and then in IT, as well as in retail companies like GAP. I was CIO at Palo Alto Networks before this, and now at Databricks I’ve been here for close to 18 months. The charter at Databricks is that the company is growing really fast, we crossed $1 billion in sales in the middle of last year, growing at 80%. We just announced DB SQL, which is our warehousing product, grew at 100 million ARR in less than three years from zero, so really fast growth there as well. So as a CIO, the first CIO that they hired, really my focus has been on how to help the company scale and do transformation, and I used to think that transformation was the “big T” Transformation. It is now a “bigger T” Transformation, and that is happening in the industry.

Evan Reiser: And Naveen, one thing that’s so unique about your role, a lot of people we talk to, they’re either at really large companies that are kind of working on digitizing and transforming or they’re at smaller companies really focused more on innovation, you guys are doing both, right? Like you’re already at high scale and you’re constantly innovating. Can you share some of what’s unique about Databricks that might differ from other organizations and how that affects your role relative to maybe some of your peers?

Naveen Zutshi: Yeah, extremely engineering-led focus at Databricks. While we have over 5000 employees, it feels like a startup in many ways. A lot of first principle based thinking because most of the founders are still with the company, and that makes us think about the long term rather than just the short term. And then I meet a lot of customers in my role and it’s pretty amazing to see the enthusiasm and energy behind data and AI from our customers and what they are achieving using Databricks as a platform.

Evan Reiser: I have to ask, are there any anecdotes about some of the surprising things you’re seeing your customers use, or surprising applications of AI that have just been impressive or exciting for you?

Naveen Zutshi: Yeah, it’s been really fascinating, like since November, we are in the throes of LLM, and over 600 customers are already using Databricks for large language models and NLP-based use cases, so that’s one. If I look at companies like Rolls-Royce, and this use case is public, they actually have a very interesting selling model, and their selling model is they sell the engines for a very low price because they actually charge the customer per fly run, per hour of flights. So their whole premise is they want to keep the engines really in great working order and they want to do a lot of preventative maintenance to prevent or to reduce the amount of downtime for the engines because their entire revenue is tied to this. They use Databricks’ machine learning and AI for all of those purposes, so it’s a fascinating way of thinking you changed a commercial model and that commercial model is what made Rolls-Royce quite successful now as the aircraft engines and at the same time, the way they built that platform on Databricks. I was actually in India just a few weeks ago and I was meeting with two IPL companies, I don’t know if you’re familiar with IPL, this was one of the biggest sporting events in the world, and there are two companies that are both Databricks customers, both which help with the gaming aspect of that and they have massive adoption, like over 30 million concurrent users using their platform and they’re using machine learning and data science using Databricks as their platform. So it’s fascinating to see that it’s not just a question of CDOs and CIOs that are interested, it’s the CEOs that are interested in leveraging data and insight to drive their business forward.

Evan Reiser: Yeah, I’ve seen the same thing. Surprisingly, we’ve had a lot of our customers come say, hey, this other person in our organization who normally is not interested in IT or technology is now coming in and saying, hey, what are the applications in FP&A about how we can be using AI? So it’s really interesting how it's changed from more of just a technology thing to like a business transformation opportunity, so that’s an exciting wave we’re on.

Naveen Zutshi: Yeah, and I think like, Andreessen, Marc Andreessen had this note, right? Software eats the world and I think data and AI is going to eat software, and that is proving more and more true.

Saam Motamedi: I thought maybe let’s take a step back for a moment because the three of us have been talking about AI for a long time. Databricks obviously has been a leading AI company for many years now, yet when I talk to both enterprises and consumers, this wave of AI feels different, right? And it’s kind of like, we had this almost iPhone moment with ChatGPT, and now like all anyone wants to talk about is how’s AI going to impact my business? Can you help our listeners understand how we got here and what’s similar and what’s different about, let’s say, the last six months of AI and what lies ahead versus everything that Databricks was focused on before that?

Naveen Zutshi: Yeah and  I can relate this to my own personal journey here in the US, you know, I came to the US in ’94. I think the first wave of the Internet, which was ’95-2000, and I think like I was early on in that, but I was not in the Bay Area and I felt like I missed out on what was happening in the Bay Area, and even though I was in the periphery, it was not as impactful. I feel like in the last six months, every day I’m learning something new, both from my customers, from myself, as well as what we are doing in a company, which seems so radically different. In fact, I would profess to say, this is just as transformative as the Internet was and more transformative for enterprises than mobile. The reason I say that is, on the mobile side, a lot of B2C customers really had a radical change for them through mobile, whether it’s social media or the consuming of apps on mobile devices, but enterprises still are tethered to their laptops quite a bit and that hasn’t really changed to a large extent. But this is going to transform the way we do user experience, the way users will interact with systems. This will transform the way each one of us can have a personal bot or a personal AI assistant. We used to talk about RPA bots and having RPA bots as a personal assistant for you, and that didn’t really work out that well. Whereas I believe we are at the cusp of having AI bots that are truly your AI copilots or AI assistants. Just in software development itself, I have two children, both in software engineering, and the concern is like, what kind of work will software engineers be doing in the coming years? Because, even at Databricks, we are seeing through Copilot and CodeGen, even in technology like Salesforce, in Apex code, we are seeing really good benefits, both from code debugging, test gen, as well as software development. So that’s one. I think there was a step change improvement and that is what I think the biggest aha was in terms of how conversations can be much more realistic than they ever had been, and that was the aha moment for a lot of companies. Because before that, you were using NLP and AI, but it was used  primarily by data scientists and data engineers. It also is now democratizing this whole notion of LLMs to the entire company. I can’t tell you how many business leaders, whether they’re in sales and other groups within Databricks,want to start using the technology. In engineering alone, we had a hackathon and then I think we had over 100+ use cases demonstrated in that hackathon using LLM. So that just gives you an example of the speed with which companies like ours are moving in this space and the enthusiasm that we see across the board, not just within Databricks, but across our customer base.

Saam Motamedi: Maybe just to follow up on that, because many people listening may have not done those hackathons inside their companies and are likely thinking about, okay, I’ve used ChatGPT, I seek the Copilot use case for developers, from my vantage point, that’s the one that certainly is having real impact today. But maybe I’m not a developer. Maybe I’m a product manager. Maybe I’m a sales rep. Maybe I work in an IT and technology organization, and I don’t yet fully grok how it is going to change my life. And so Naveen, can you help us peek into the future? Other than the developer use cases, what are some of the other ways you think this Copilot model will change enterprise work?

Naveen Zutshi: Yeah, I can think of four other use cases. One is to transform and simplify the user experience for businesses. For example, search, and if you look at engagement scores and enterprises, finding information is probably one of the lowest engagement scores for enterprises. I think we can turn complex requests to English-based requests that can serve the need, without the need of engineering help and a lot of backend work once the prep is done, so I think that can make not just access to information, which is hindsight information, but also hopefully help with some of the predictive information as well. That’s one. I think automation through prevention and self-service. So I think that’s a pretty common use case, we are seeing a lot of use cases there. For example, we built a  knowledge base: kb.databricks.com. You vectorize it, you embed it, you do the LangChain, and then you do the summarization. And you are reducing, you’re improving accuracy of results, you are grounding the prompts, and you are coming out with a much more conversational bot that will help with both self-service for our customers as well as reduce the amount of tickets that need to be addressed by our customer support reps. That is just one use case you can see, whether it’s sales support, whether it’s HR support, whether it is even creating job descriptions. We are doing a pilot where we can help you create job descriptions and also have interview questions upfront created, those will help recruiters in driving much better questions. So that is just a simple example of something that an intern can do in a few weeks. Personal assistant for every employee, and I mentioned this before. I think if I’m a sales rep or if I’m an XDR or a BDR, or if I am someone in legal, I need a legal copilot that can look through my contracts and actually help me summarize those contracts and help me find the issues that I need to focus on as an employee. So I feel overall, we can have a lot more applications that will be intelligent and those intelligent apps will help turbocharge productivity for us as a company.

Evan Reiser: Anytime you have a new technology, right? First, it's really exciting, you can imagine the immediate applications. Then as people’s understanding and imagination starts getting broader, it naturally takes us to maybe scary things, right? Any new technology, you start worrying about, oh, here’s all the bad applications, right? Sometimes that can overshadow some of the opportunities. There’s certainly going to be a growing discussion around to what extent AI should be regulated or not. There’s a risk that we over-rotate, and we may underestimate some of the upsides, as we get more clarity on some of the downsides. Any thoughts you have about what we should kind of keep in mind or maybe some of the applications at world level that we should maybe not underestimate as we think about the future impact AI has on the world?

Naveen Zutshi: Yeah, I think from a regulation standpoint, my thought is, and this is just my personal experiences, often regulation done poorly can lead to rising costs for consumers and lead to incumbents being the winners. So the question is, you do need some level of regulation, what is that right level of regulation and how do you roll it out? And even for enterprises, we need to think, and we are thinking long and hard to make sure that one, control of data, like you have PII data, you have data that is your customers data, you don’t want to train on that data without getting their permission, and then also you want to make sure that you have really good access control, security controls over that data. You have to think about the models themselves. Do you want to have smaller models or do you want to have large models? Do you want to build an abstract layer that will help you automatically pick and choose the kind of model that you want to use? So there is a lot of consideration there. I think I mentioned a lot around the accuracy of results and grounding your prompts. I think you have to think long and hard there because, especially if you’re using it as an automation mechanism, accuracy rates need to be quite high. If you are using it as a compliment to humans where humans are going to review it before it’s shared, then you can have somewhat lower accuracy because the humans are in the loop in everything. So you have to think about that. I think you have to think about bias and toxicity. How are you addressing that? How are you managing through that process? And then do you have the right model? Do you have the right architecture? Often for enterprises, what I’ve learned being in many enterprises myself, the challenge is not just the shiny object, the challenge is still the data prep, the data architecture, the integrations, having data cleansed, having all of that process available. Without having really strong foundations, or at least having good foundations, it’s very hard to build really good models on top of them.

Saam Motamedi: Naveen, maybe one question which I’m asking somewhat selfishly and somewhat also for our listeners. Earlier, you drew the analogy between this and the internet wave and your personal experience with that wave. One thing that’s been striking for me in the last six months, relative to my career in technology, has just been how fast everything is moving and how something that three months ago felt important can suddenly be made redundant by a new product release or new research release. How do you personally stay educated? What’s your reading diet on this wave of AI? How do you stay up to date? I feel like I need to just take a few weeks off work and catch up on everything going on.

Naveen Zutshi: It's so funny because we were having a dinner conversation with some startups and we were discussing that we need to take time off and just listen to podcasts and then do work And every day, I listen to a few podcasts like yours, there is No Priors by Sarah, it’s a good one, and Cognitive AI, that’s another good podcast. I think trying it is probably the best way and, if you’re more technically minded, reading the white papers, because there’s a lot of new white papers that are being published as well. And then the number of models, you used to think one model to rule them all and now suddenly there is a plethora of both open source models as well as smaller models with fewer parameters. Then also the cost curve has come down so dramatically, so quickly, it’s amazing to see. What used to take, I think GPT-3 was $5 million, it now can be trained at $300,000 to $500,000. So it’s like there are so many vectors of learning and then on top of that, we are trying to figure out, how do you have the vision and the internal vision for enterprises for LLMs? What would that look like? And there is no playbook here. You’re trying to figure that out and you’re probably in a very interesting space trying to figure that out. I would also say you collectively lean on the collective experience of your teams, of others. I think that’s been really fascinating for us as well. Not only doing these hackathons, we are also going to do a business and IT hackathon, we are having weekly learning sessions between teams where other groups are presenting and sharing their insights as to what they have developed or what they are building or what they are learning, and that’s a great shortcut to learning as well. And I’m sure there are other examples of ways you can learn too.

Evan Reiser: You know, your role is to inspire and activate innovation, right? Not just within your team, but across the company. As we think about the rate of change, the growing magnitude, the importance of some of these breakthrough innovations, any guidance on how you build a culture of experimentation, of learning, of innovation inside Databricks, or in your experience at Palo Alto?

Naveen Zutshi: Yeah, I think, like it’s not just me, I think we have Ali, we have a lot of founders in the company who are doing an amazing job of encouraging innovation. I think we are a very open culture, which is one of the things that really helps, like there is a lot of transparency in the culture. So as I mentioned, even if we have 5000 employees, there is a “no idea is a bad idea” approach to things. And then it’s not a very top-down culture either, it’s a very bottom-up culture in terms of innovation. That helps. I think encouraging teams to take the time to actually do work is quite important. So the more people we can incorporate, the more people we can bring along the journey, I think that helps a lot. I think these brown bag sessions that we are doing helps a lot. Having these Slack sessions helps a lot. So I think more than anything else, it’s also incumbent on every leader to spend a lot of time learning themselves and listening to others and then sharing their perspective. It’s quite interesting also, what I’m seeing when I talk to Ali and others, the amount of CEOs who are asking their CIOs and others to help them educate about large language models, NLPs, and AI. Traditionally, CEOs would spend very little time with their CIOs and they’re spending a lot more time with their CIOs and their teams, not just CIOs, CTOs, CDOs and others, because they are themselves getting really interested in what this can mean to them and their companies.

Evan Reiser: I totally agree with that, Naveen, from my perspective, just talking to a lot of CIOs. I think every company is trying to become a software company and that already was a growing trend. I think now with AI, that’s kind of spiked up where the CIOs’ role around the executive team, for every business, even outside of software businesses, is kind of more and more important. I think that we’re going to see that trend continue.

Naveen Zutshi: I know of some companies who are doing weekly stand-ups with their CEOs on understanding what new models, what new applications we are driving with AI.

Evan Reiser: There’s a pithy saying, I’m sure we’ve all repeated once, which is that every company is becoming a digital business, every company is becoming a software business. The natural extension to that is every company is becoming increasingly more of an AI business, or at least AI will empower more parts of the business.

Saam Motamedi: Maybe just staying on this thread of like, CIOs listening and Naveen, like you’re on the forefront now in your current role, but if you kind of think back to your days at prior companies or some of your peers who are not sitting in AI-native companies, I think one of the big things I hear from those folks is obviously a lot of excitement and enthusiasm, but also two particular flavors of concern. One is, hey, I’ve been sold the AI dream before and it didn’t deliver, and there were these prior generations of AI tools that promised that they would revolutionize my forecasting or different aspects of my business, but they just weren’t quite good enough, and so I spent a lot of social capital internally and the results weren’t there. And then the second is like, hey, back to our point on how fast everything is moving, like, what if I go build on OpenAI today and it turns out six months from now, the right decision was to build on open source or it was to build on Anthropic or one of the other vendors, how do I actually make decisions when the earth feels like it’s constantly moving under me? And so I’m curious if you have any advice for people who are listening who are wrestling with those types of questions.

Naveen Zutshi: Yeah, I think on the second one, the question is, are these two-way decisions or one-way decisions? And I feel like these are often two-way decisions. You’re not marrying for a long term with a company. You are using it for certain use cases, and in many cases, you can have different models for different use cases. You can work with many companies rather than just one company. That would be my advice to CIOs because you’re right, the world is changing very rapidly and, whether ChatGPT’s in the front lead in terms of models, there are so many new companies that are coming up, Anthropic, Cohere, and others, as well as what Google is doing, what Microsoft is doing, what we are doing. We built OpenAI, which we open sourced, DALL·E, in a matter of weeks and on a very, very limited amount of GPU spend. So you are seeing this in the industry where financial companies are building their own models as well, or other companies are building their own models. So I think I would not do the opposite, which would be like, let’s wait, rather than let’s experiment now. And I think on the first question, let’s say you are doing 100 experiments and let’s say 20 turn out to be really amazing and 80 you have to discard. That is 20 more ways to turbocharge your revenue and productivity than you had before. And that’s the way I think about it and I think not all experiments will work because maybe the data is poor, the data set is not there, or it’s not a good use case for large language models. But we have seen a threshold change, to your point, Saam, that we had not seen before with AI. The other thing is the complexity of building these solutions has come down dramatically as well. This is demonstrated by engineers building it because they’re coming back from a week-long hackathon and have an amazing product to show and maybe that’s not fully production grade, but it is an amazing idea and it works. For us to build  our knowledge base, I think it was two to three weeks, it wasn’t more than that to have engineering and support build it and put it in production. So some of these changes don’t take months and years, so I think the cost is lower as well. That’s what I would go for. And then I think for longer term changes, which will be multi-months, multi-quarters, yeah, you’re going to be more thoughtful about those and make sure that those use cases are well thought out before you embark on those projects.

Saam Motamedi: That’s all really well said. The thing you said I hadn’t thought about is the speed at which you can actually build some of these things, given all the tooling that’s emerged from folks like Databricks, is very different than in the past. So in that sense, maybe in the past, a lot of AI projects took a long time to get live, the value was questionable, it was hard to even get to the point where you knew if the experiment worked or not. Now, to your point, if you can get there in weeks, for people listening, you can go and post wins on the board.

Naveen Zutshi: Yeah, absolutely and I think there was this misconception because we have all suffered from building data science projects that didn’t work, like forecasting is a great case, where you wanted 100% accuracy on a forecast for production, and you’re never going to get that. And so the question is what level of tolerance and what level of error rate were you willing to handle? And I think with large language models, especially on language, I think that accuracy is much better, especially if you’re grounding it on your data. So you’re preventing hallucinations or at least reducing hallucinations quite a bit. And I think there is still going to be more research, I have a lot of confidence there is going to be accuracy-related algorithms also built, which will help. I guess in one way, I’m completely bought into the hype, and in other ways, I should be more practical about it, but that’s just me.

Saam Motamedi: It’s a good tension to balance.

Naveen Zutshi: Yeah. I think that has been the biggest learning for me personally as well in the last six months, right? Not only the pace of change, but the speed with which engineers are building interesting things with it. Otherwise it would be a lot of hype and then you will use it. The one thing that I would say that I'm still working on is like, am I changing my daily behavior as a result? And I think that would be an interesting test for all of us, whether my daily behavior is changing because of how GPT is incorporated into my daily life, and that is taking some time. That will take some time. But I think there’s an interesting part around that, then I would say it is more mainstream, like the way you write emails, the way you do Word documents, the way you research, the way you find information, the way you interact and socialize and collaborate with others.

Evan Reiser: I actually totally agree with you and that’s a litmus test that I think is going to separate different types of AI products or AI features: is it a productivity enhancement thing we’re already doing or is it really transforming the day-to-day work of some role? I think we’re going to see AI products that all legitimately use AI; there will be a separation between what’s augmenting the work, and what’s really replacing the work and unlocking new capabilities.

Naveen Zutshi:  I think like Saam, probably as you’re thinking about investing in AI companies, this is where it’s also pretty hard to figure out which companies to invest in because the barrier to entry is so low. And the question is what is the thing that they’re going to deliver that is going to lock the user in a good way to that product?

Saam Motamedi: Yeah, I agree. It’s also worth noting, we’re just super early, right? And one thing I was reflecting on last night is, and I want to go back to that kind of iPhone analogy, right? So let’s say ChatGPT in November is kind of this iPhone moment. So the first iPhone came out in June of 2007 and, I’m sure you guys remember, there’s no app store, there’s only first-party apps, it’s like the camera was super rudimentary, et cetera. And then you think about the kind of applications that got built on top that changed the way we live, and Naveen, to your point about how does GPT change the way we live? So Instagram is one such application. Instagram was founded three years after the release of the first iPhone, like in mid 2010 right? Uber was founded late 2009 but didn’t really kind of emerge in its form factor until 2011, 2012. And so I bring that up to say like, one thing that’s so exciting is it feels like so much has happened and yet we’re six months into kind of the post ChatGPT world. And I think Naveen to your point on the entrepreneur side, I think,for entrepreneurs listening, the opportunity is really like, not to think incrementally, right? Not to build a thin layer on top, but to really ask like, okay, I have a new set of capabilities on how people can interact with computing, what can I do with that? And like what’s the analogy for an Instagram or an Uber that we wouldn’t have thought of as even conceivable prior to the iPhone? And I think that thinking is so early, I’m optimistic, and I think over the next few months and quarters, as people really play with these models, we’ll see a new wave of applications that feel very different and then hopefully very durable.

Naveen Zutshi: Yeah, and I think Saam, this point that we should have some patience as well, in terms of how quickly these will change the way we work, change the way we play, and if you look at the mobile examples, it was several years, not a matter of days and months. Whereas right now, the immediacy of being in it, you feel like every day is going to be something new.

Evan Reiser: Yeah, that is kind of like mind-opening and also very exciting, right? It’s like, we’re just at the tip of the iceberg, it feels like the whole iceberg, right? But it’s just the tip. So we’ll quickly do a quick lightning round, then we’ll close it off. Saam, you want to start?

Saam Motamedi: Yeah, absolutely. Naveen, maybe to start, how do you think companies should measure the success of the CIO?

Naveen Zutshi: On the value produced.

Evan Reiser: And, Naveen, you’ve had a long career, multiple times as a technology executive. What’s maybe one piece of advice you wish someone had told you when you went to your first CIO job?

Naveen Zutshi: Say yes more.

Evan Reiser: I like that one.

Saam Motamedi: Both of those answers were crisp and to the point, and I agree with both. Moving to the next question, how do you think CIOs can best position themselves to collaborate with the rest of the C-suite and leadership team?

Naveen Zutshi: Understand the business deeply. I think it starts there. And then bring your technology acumen to the business conversation.

Evan Reiser - 00:29:20: We’ve got two more questions real quick. This is a little more on the personal side but is there a book you’ve read recently that had a big influence on you? And if so, which book and why?

Naveen Zutshi: Not recently, but The Innovator’s Dilemma is a book that I really loved and it made a big difference to me. I’m a big sci-fi fan, so The Three-Body Problem is the book that I really love.

Evan Reiser: Naveen, we have to have a conversation in the future to like sync up on some sci-fi books, because that was one of my favorites. Did you read the whole trilogy?

Naveen Zutshi: Yeah. It’s a great book and it's a great series.

Evan Reiser - 00:29:51: It’s great, yeah.

Naveen Zutshi: Maybe the second and third are not as good as the first one, but still really, really good. Project Hail Mary recently is pretty good too, from The Martian, Andy Weir.

Evan Reiser: Yeah, Saam actually bought me that for the holidays, I think, last year.

Saam Motamedi: Great book. I also recommend rereading Ready Player One.

Evan Reiser: Saam, you want to maybe close it off with a final question?

Saam Motamedi: Yeah, maybe one final kind of more personal question like, Naveen, we’ve obviously talked a lot about exciting things coming on the technology front in this episode, if you were to name one thing you’re most excited about in terms of upcoming new technology, it doesn’t need to be related to your work at Databricks, that like you’re personally most excited about, what would it be?

Naveen Zutshi: Oh, self-driving cars. I’m still waiting for those, but I would really enjoy them if that turned out to be a reality.

Saam Motamedi: Yeah, me too.

Evan Reiser: Well, Naveen, thank you so much for joining the show. As always, I love chatting with you and really appreciate you sharing your views on how AI is going to change the world and how it’s affecting us today.

Naveen Zutshi: Thank you, Evan. Thank you, Saam.

Saam Motamedi: Thanks a lot for joining, Naveen.

Evan Reiser: That was Naveen Zutshi, CIO of Databricks.

Saam Motamedi: Thanks for listening to the Enterprise Software Innovators podcast. I’m Saam Motamedi, a General Partner at Greylock Partners.

Evan Reiser: And I’m Evan Reiser, the CEO and Founder of Abnormal Security. Please be sure to subscribe so you never miss an episode. You can find more great lessons from technology leaders and other enterprise software experts at enterprisesoftware.blog.

Saam Motamedi: This show is produced by Luke Reiser and Josh Meer. See you next time.