Episodes · S2 E16
Information Symmetry: DevRev's Bet on AI-Driven Enterprise Decisions | Manoj Agarwal
AI AgentsMulti-Agent SystemsRAG & RetrievalAI Evaluation & ReliabilityEnterprise AI
Key takeaways
- Manoj Agarwal frames enterprise dysfunction as an information-symmetry problem: most meetings exist only because attendees hold different data. As he put it, “if the same information was available to every single person in that room, did we even require this meeting to make a decision?” The cost is decisions driven by “the biggest title and loudest voice” over the best-informed view.
- DevRev’s answer, per Manoj, is a “business-centric knowledge graph” — not a dump where retrieval “magically” works, but a structure built around the only two things he says matter to a company: “the product or services that you provide and the customers you serve.” On top sit a conversational layer and agentic actions that two-way sync back into primary systems via DevRev’s patented Airdrop tech.
- Manoj’s cost-collapse claim, with his own hedge intact: over “eighteen months” a million tokens went from “dollars 50 down to what? 12¢ or something,” and for advanced models “more than 99% reduction.” His point isn’t the exact figure — it’s that LLM inference is now nearly free, so the real enterprise challenge is uniting private data, not affording compute.
- Manoj’s model for agentic AI is programmable skills: what people do in a meeting are skills you can “write in plain English” — “I have 20 skills” — then stitch across data for end-to-end work. His example is triage: managers weigh tier-one status, customer health (“red or orange”), and strategic value to set P0–P3; capture that judgment as a skill and run it at the scale of 200 reviewers.
- On trust, Manoj splits internal from customer-facing AI. Internal users inspect the “chain of thought,” correct it, and “commit” — teaching the machine. Customers “don’t have that luxury,” leaving reinforcement learning plus continuous answer-checking. Yash Sheth adds that humans convey their confidence level, while a machine states every sentence with equal certainty — so even 10% wrong erodes trust fast.
- Yash Sheth’s three-year forecast: every enterprise system — Service Cloud, HR, Jira — exposes its own agent, but ROI arrives only when those agents “seamlessly talk to each other.” The missing piece, he says, is a standard for inter-agent communication, including “what is DNS for agents?” — discovery — which is why Galileo and others formed a collective to build an open agent protocol.
Frequently asked questions
- What does Manoj Agarwal mean by “information symmetry,” and why does he say it matters inside enterprises?
- Manoj’s premise is that the open internet — now accelerated by LLMs — already gives people roughly equal access to public information: he and Yash “can have the same search, get the similar result.” Inside companies it breaks down. Decisions require pulling people into meetings because each holds different data from different tools, systems, and side conversations, so outcomes get driven by “the biggest title and loudest voice.” Manoj argues this asymmetry is a major source of enterprise cost and wasted effort, and that closing it — giving everyone the same information — could remove the need for many meetings entirely.
- How does DevRev technically tackle scattered enterprise data, according to Manoj?
- Manoj describes a layered approach. First, connect siloed sources — structured systems like Salesforce Service Cloud, Zendesk, Jira, Atlassian, and GitHub, plus unstructured collaboration in Slack — using DevRev’s connectors, which he calls Airdrop technology. Second, organize that data into a business-centric knowledge graph built around the company’s products and customers, with a personalized rather than generic schema. Third, add a conversational AI layer for retrieval. Fourth, make it agentic — take actions and write changes back into primary systems through two-way sync. Manoj notes both Airdrop and the knowledge graph are patented technology DevRev built.
- What concrete example did the guests give of turning human judgment into an automated agent?
- Manoj uses ticket triage. Managers deciding whether something is P0, P1, P2, or P3 “stare at the data” — which customer it’s from, whether they’re tier one, the health score (“red or orange”), and strategic value even from non-paying accounts they want to win. He argues you can program an expert’s decision style as a “skill” and run it at the scale of 200 triagers. Yash Sheth says Galileo has seen this productionized: a customer getting thousands of tickets daily where expert engineers’ intelligence is baked in so tickets are auto-triaged and next steps carried out by LLMs — “baby steps,” but real and in production.
- Why does Manoj distinguish between building trust for internal versus customer-facing AI?
- For internal users, Manoj says you can expose the AI’s chain of thought and the linkage showing where information came from. Users inspect it, correct it when they disagree, and “commit” when it’s right — effectively teaching the machine how they think. Customer-facing AI removes that feedback loop: “they’re not the ones who are going to come and train.” There the tools are reinforcement learning — did the answer land or not — plus an internal system continuously checking whether what was given to the customer was correct, which gets much harder for complex, reasoning-heavy queries that demand more trust.
- What does Yash Sheth say is missing for a multi-agent enterprise future, and what is being done about it?
- Yash predicts that within about three years every enterprise system will expose its own agent, but value only materializes when those agents can “seamlessly talk to each other.” Today, he says, everyone builds agents in silos and the infrastructure between them is missing — both a shared communication standard (an “agent protocol”) and discovery, which he frames as “what is DNS for agents?” He notes other gaps like state passing and authentication. To address this, Galileo, with other companies in the space, formed a collective to build an open standard with everyone contributing — not a closed, single-decision-maker project.
Chapters
- 00:00Welcome to Chain of Thought
- 00:57Information Symmetry in Enterprises
- 02:03Challenges of Decision Making
- 03:41Recency Bias and Product Management
- 04:58Data Silos and Information Waste
- 05:23Structured vs. Unstructured Data
- 06:04Collaboration and Data Retrieval Issues
- 08:17DevRev's Approach to AI and Data Integration
- 09:23Building a Business-Centric Knowledge Graph
- 10:00Conversational AI and Automation
- 12:57Agentic Interactions and Skills Programming
- 20:05Multi-Agent Systems and Future Vision
- 21:25Challenges in Multi-Agent Communication
- 25:10Data Cleanliness and Governance
- 28:14Trust and Reliability in AI Systems
- 36:58Conclusion and Future Outlook
Show notes
What if everyone in your organization had equal information at all times? Would meetings even exist?
This week, we dive into the concept of information symmetry with Manoj Agarwal, co-founder and president of DevRev. Manoj, along with hosts Conor Bronsdon and Yash Sheth, explores how DevRev is connecting data, personalizing schemas, and automating complex tasks, offering a glimpse into the next generation of AI-driven workflows. This is revolutionizing enterprise data and decision-making by breaking down the silos that create information asymmetry.
Learn how AI is reshaping business outcomes and collaboration, moving us closer to a world where everyone has the information they need.
Chapters:
00:00 Welcome to Chain of Thought
00:57 Information Symmetry in Enterprises
02:03 Challenges of Decision Making
03:41 Recency Bias and Product Management
04:58 Data Silos and Information Waste
05:23 Structured vs. Unstructured Data
06:04 Collaboration and Data Retrieval Issues
08:17 DevRev's Approach to AI and Data Integration
09:23 Building a Business-Centric Knowledge Graph
10:00 Conversational AI and Automation
12:57 Agentic Interactions and Skills Programming
20:05 Multi-Agent Systems and Future Vision
21:25 Challenges in Multi-Agent Communication
25:10 Data Cleanliness and Governance
28:14 Trust and Reliability in AI Systems
36:58 Conclusion and Future Outlook
Follow the hosts
Follow Atin
Follow Conor
Follow Vikram
Follow Yash
Follow Today's Guest(s)
LinkedIn Manoj Agarwal
Check out Galileo
Transcript
99 segmentsSpeaker 0:00 If the same information was available to every single person in that room, did we even require this meeting to make a decision? I truly believe that inside the enterprise, there is this big amount of silos that exist because of the data silo. Is the big reason also that we spend so much money today.
Conor Bronsdon 0:24 Welcome back to Chain of Thought, everyone. I am your host, Conor Bronsden, and I'm here with my fellow cohost, Yash Sheff, COO and co founder of Galileo. Yash, always great to have you on the show with me. It's been too long. Thanks, Conor. And, you know, we we have an esteemed guest today, Manoj Agarwal,
Speaker 0:40 who's, you know, co founder, president of DevRev. Prior to that, Manoj has been SVP of engineering and GM for hybrid cloud at Nutanix. And so, you know, really excited to have this chat, Manoj, and thank you for being here. Thank you, Yash and Connor, for having me here. I'm looking forward to the conversation. I've heard you talk a lot about this idea of information symmetry and your passion for it and how it's informing a lot of your decision making around
Conor Bronsdon 1:06 the approach to AI and what DevRev is thinking about. Can you elaborate on the concept of information symmetry and its importance in how DevRev is achieving business outcomes today? I would say that like if you just go and look at today
Speaker 1:20 through the internet, the amount of information that exists, people have access to like within whatever is available to them freely. I mean, can just go and search for the information and can get that information at the tip of my finger fingertip and just like not distinguished between the two people. If you just think about me and Yash can have the same search,
Speaker 1:44 get the similar result and can go and act on. Now that has been made easier by the use of now RLMs. Like obviously OpenAI with the GPT that what we see that has started to happen where I can get the information in more succinct way that I can go and consume it. Now start to think about within the enterprise, within the company, within the businesses what happens.
Speaker 2:10 Most of the time for us to be able to make a decision, we require the different people to come in the meeting and they all bring some information with them. They have access to the different set of information that not necessarily everybody has the same set of information. And I like typically that I've what I've found and which when you have to make a decision,
Speaker 2:37 a lot of time this fight, people can call people can call it okay, people with the biggest title and loudest voice, they start to make decisions. I'm sure that you all have seen it somewhere or the other that happens. And for me it was like it was not about the title. It was not about that shouting game. It was mostly around the information that they carried
Speaker 3:00 was very different compared to the other people. And because they have access to the different data, different tools, their different systems, they are into different meetings where they're also gathering this information. And the question was, if information symmetry was there, if the same information was available to every single person in that room, did we even require this meeting to make a decision?
Speaker 3:25 Yeah. That was the crux of it. I truly believe that inside the enterprise, there is this big like amount of silos that exist because of the data silo that exists. It's the big reason also that we spend so much money today. I'm also curious about like you know being in a large organization myself earlier right and having been in these meetings, like recency bias creeps up quite a bit as well apart from like the loudest voice.
Speaker 3:55 This happens to us in a startup as well, like, hey, like this customer is asking for feature x like let's go and build it but like actually going back and thinking about you know what let's look at all the feedback that we've received around this and holistically make that decision like I think that's huge. Yes, like you were just touching something on very important.
Speaker 4:18 Have you heard like a product manager running to you and saying that, okay last night I heard at beer bash this conversation and the customer said, this is one feature that I need. Yeah. And they come back and say, this is the highest priority in the company that everybody should work on it. Three months later, you deliver that feature and go and ask the same product manager what's happening, how many people are using it and so on. They actually have the list of new features from somebody else. They don't talk about the ones that you just delivered.
Speaker 4:50 Yeah. Because it was all the time. So amount of waste is just like this crazy. Yeah. Like hard to look at it and you can save a lot. I mean, if you just had the information symmetry, you start to collect this information in the right way. So how do you solve this, Manoj? This data, first of all, they exist in the organization, and we call it like the structured data, unstructured data and so on. And now when I say structured data, think of like, okay, if I'm in customer support, I'm using Salesforce Service Cloud, or I'm using Gendesk,
Speaker 5:20 or I'm using some other systems to keep this data, organize this data. If I'm in software development, I'm probably putting that in Jira or Atlassian or GitHub or some other places that I'm going and putting that data. The one is basically these are restructured data. Now the conversation that I'm having, all this collaboration that I'm having, for that there is a completely different system. Like today people, especially
Speaker 5:43 you start to look at this, Slack is very popular. I mean, go and look at like the, even if you are coming from the software development side, when put some information and they jump back to the Slack. As soon as I say, can you go and look at this particular Jira ticket ID? Okay. And that's when the collaboration happens, right? Yep. Yep. And can you imagine two days later, comes
Speaker 6:07 and asks, okay, how did you really make that decision? You can potentially still find the person who can come and answer that. Three months later somebody new joins comes and they want to find this information out. Like how do they find it? First of all, like the person who actually has done the work also, they have no idea. They probably will not remember anything.
Speaker 6:30 A search also is such a big problem today on Slack that you can try to find that information. So the question is, all this information that we are talking about it and there are so many silos already getting created. How do we bring it together? How do you connect it to the right thing? Which is like where we call it for the existence of the company, there are only two things that matters.
Speaker 6:53 It's the product or services that you provide and the customers you serve. So how do you really organize your data around those two things? To that point, this is a daily thing on our team as well where there's discussions on Slack and they pertain to a ticket and then an engineer is like always saying that hey let's go back and put this on the ticket and we've even built a plug in in Slack to like take this chat and then throw it in the relevant ticket because it's very important. It's extremely important too. In Slack itself, any collaboration platform, whether it's Slack or even offline in meetings and on conference calls,
Speaker 7:31 things are scattered extremely and you tend to lose that. And there have been times where I've gone back and said like hey we built this product feature why did we do it this way? Like you know why did we do it this way and not that? And there's no context like it's so hard to find even that information on why did we make that decision. This is a slack channels that you're aware of.
Speaker 7:55 There are other ones that you're not aware of the same conversation happening. Crazy yeah. I think coming back to the point of like you know Conor's point of like how do we solve for this niche like organizationally what do we need to do and you know there with AI of obviously we there's a step forward where we can distill all of these into something more structured but you know how are
Speaker 8:19 you guys at DevRev approaching this? Yeah so we actually addressed this I think beautifully and I'll say when you look at the data outside on the internet through the LLMs now that we are able to go and do so much, right? And even the cost when you start to think about like in eighteen months it has gone down like what? For a million tokens dollars 50 down to what? 12¢ or something?
Speaker 8:48 Now, for much more advanced models also, more than 99% reduction in the cost. Now with that, we are able to, I mean, at least consumers, they can just go and use it. Don't even worry about the cost. The question is, how do you do that in the context of the data that is there in the enterprise? Could you really bring that data together? Is something that first thing that we are tagged on. You said, okay, people have these tools, systems,
Speaker 9:18 conversational systems also. How do you bring this data together? How do you really build a knowledge graph? And the knowledge graph, when we say it, it's not just any knowledge graph where you just dump the information and magically it will go and figure it out. Could you really build a business centric knowledge that understands your product, how you sell your product in the market, how you organize your product.
Speaker 9:44 And then it also understands customers. So once you organize your data, and this is where that a lot of help from the AI also comes in, with the LLMs, part of LLMs that comes in, that it can understand the context and go connect it with the right areas. Once you brought that data together, now you can talk about the conversational AI on top of it, just like chat GPT.
Speaker 10:08 Can start to talk about that quick. Even the automation, when you talk about it with the agentic AI, you can start to now imagine the kind of things that you can come and do. Without AI, I can imagine an engineer going and understanding all my data and you know the concepts, the product concepts, the teams, the customers and then creating a custom schema for the knowledge graph.
Speaker 10:33 But you know now you can have a personalized schema for the knowledge graph from the AI itself and then add a conversation layer on top of that so that retrieval becomes more personalized as well, right? Yes, Yash, one thing that I want to point out, there is one basically just you just bring the data together, which is like a one way I can consume that data, I can maybe even flatten that data, put it in a database,
Speaker 11:00 go and link it and so on. The other part that you have to think about is, it's not just one way, which like what the data warehousing solutions always did with the ETL, on which that you can build all kind of report, analytics, dashboards that you can go and create. Now, conversationally also you can search and get this information out. We are talking about like when you have to take action on this data,
Speaker 11:24 when you have to also go and create new set of information, whether it's okay in the context of customer support that things that I'm not able to, let's say, deflect L1 support. Customers coming and asking a question with their knowledge, I will be able to deflect now, but there are things that I will not be able to do. I have to go and get into the database itself or my primary system,
Speaker 11:49 gather that information out, bring that information out in real time, process it, converse with the customer, with the conversational AI interface, and then customer will say, okay, maybe let's say I want to order something, X, Are you able to also go and update, like take that order, being able to go and update the primary system, right? So now we are talking about the, you can start to see that amount of things that are happening is all through conversational AI interface and that requires two way sync
Speaker 12:24 with the primary system. It has to understand, I mean, what exactly is the order? What does order even mean, this system? So what we have built is not our patented airdrop technology and the in fact knowledge graph is also patented technology that we have, which does So two way thing with those it is kind of a mini agentic interaction like you know it's taking your request serving that the right information and then making the right change back into the system.
Speaker 12:58 How do you see that growing? So I think of this as almost like this is an AI platform you're building on top of the data and you're talking about like three things so far connecting the data, personalizing the schemas, adding the conversational layer on top and now making it agentic. Where do you see this growing? Just think about like when we are sitting in the meeting room,
Speaker 13:23 people obviously they have different set of information but they also have skills, things that they do and those are skills. Can you start to imagine those skills now that you can program? This is the way certain things are done, every single skills and doesn't have to be one skill. I have 20 skills and I can go and write those skills in plain English, right?
Speaker 13:49 So if you can start to think about, okay, every person or kind of profile that there are skill set, you can program in the system. Now on top of that, now think about the entire meeting that is happening. Every time there is a question, people are bringing different skills and that's what is the agentic AI to us. So if I could go and stitch this skill set along with the data
Speaker 14:16 and between the different skills to do end to end work, then I think that I have solved it and that's the state that we are looking at. Yeah. And the good news is that this is possible. No. Yeah, early stage of that actually I like your chain of thought, just the name itself, platform also that must provide you as you are going and experimenting with these things is there's a chain of thought. We'll give you these
Speaker 14:47 are all the skills that we are bringing together for the questions that you have asked just now. If you approve it, I will go and run that chain of thought and then give you the information in a way that you can consume it through the conversational AI. You know at Galileo we're constantly thinking of you know when developers are building agentic workflows like the one we're talking about,
Speaker 15:13 in order to make sure that these workflows are performing more reliably there is a developer and a subject matter expert building that feature And now if we can distill that subject matter experts knowledge into metrics that this is how the end to end agent should be behaving and here are the three things I would check manually but do that at scale for a 100 different agents
Speaker 15:38 that's a very similar analogy of like how you're thinking about transforming skills into actions and for us is like automating evaluation. Makes sense. I mean at the end of the day what are we trying to automate? Most of the mundane things that you do. And those mundane ness now with the reasoning that the system is able to do, I mean and you stitch together
Speaker 16:04 this mundaneness like every single person do is like more complex task they will do now through the GDPR. I'd
Conor Bronsdon 16:11 use a really basic example I saw recently. You know, Text to SQL has been definitely a use case for LLMs where it's like, oh, I'm talking to Claude or ChatDPT, and I need to query this database. You know, here's what I need. Help me write the great SQL command. I don't wanna spend any time doing it. Awesome. Let's go do that. And now we're shifting to this agentic world where we can use agentic querying through function calling,
Conor Bronsdon 16:35 and it uses the the structured queries of the predefined function calls of JSON. You can just put your natural language processing in and get the information you need without having to do this, you know, in between filter steps. And that's just like one example of something a lot of people may do where they have to query a database at some point. And there are so many potential areas where this can just speed up to to your part, but I was where it's like, hey, I just gave you the natural language. I get the output I need. And then somewhere along the line, we have human feedback that comes in about the the outputs
Conor Bronsdon 17:07 so that we can have continuous learning based off that human feedback, and then based off of elements judge opportunities. And to me, that really speaks to the opportunity here where we can create agents that continuously learn to better solve our problems for us. Yeah. Absolutely. I mean,
Speaker 17:26 when you spoke about also Yash, the a question that is asked and when it is asked to Manoj, let's say it is asked to Yash, like how will Manoj respond versus how will Yash respond and so on. Now start to think about it, like when we sit in the triage meeting, a lot of managers, they sit in the triage meeting, okay, What should be the priority of this? Should this be P0? Should this be P1, P2, P3? Like, what do they look at? Well, let's just look at, stare at the data. Okay, so I see it's coming from this customer,
Speaker 17:59 this high paying customer, say tier one. I look at the health of the customer, maybe it's like red or orange or whatever health score that you look at it. Strategic customers, maybe, even though they're not paying any money right now that I want to go and win this account or the conversation that is happening. So typically based on this structured information, unstructured information,
Speaker 18:22 they're making certain decisions. Now can you imagine that what Yash with such a high level when he looks at it, he'll make the decision this way. Could I go and program that? That'll be nice. Now there are like the 200 people who are doing the triage on their own or maybe sitting in this room and like just multiplied by 200 and machine is doing it at scale.
Speaker 18:48 Yeah. That reminds me of an actual use case we worked with one of our customers is like you get thousands of tickets every day on things that are not working from many customers. How can we bake in human intelligence for those expert engineers triaging the tickets? And this is a very small example, but it's very much in vain of we discuss at a high level. And these tickets can be auto triaged and also like next
Speaker 19:21 steps, automated steps can be carried out by the LLMs and we've seen this and productionized this already with customers. So this is baby steps, but now if you generalize that to like every function in the organization and have experts be able to program their intelligence that's amazing. I think that's doable this year right like in you know it's something that we
Speaker 19:47 see coming now rather than like five years later. It's happening now we all see it. I'm sure that Kooner you hear this from every single guest who comes here. We are experiencing it right now. It's a very interesting time I must Yes. And now if you think one step beyond Manoj right like now what's next after that when there are many agents doing many tasks in the enterprise and
Speaker 20:15 you know let's say from this particular use case of programming skills for from people and making them operationalizable in the platform. Every enterprise also has a lot of different systems and let's say now let's skip three years, I'm not even saying five years because things are moving so quickly, but three years from now, you know all of these systems will have their own agents.
Speaker 20:40 Your service cloud agent will have to talk to you know your HR agent and then you have your engineering ticket system whether it's Jira or something else has its own agent. Now these agents will be exposing certain tasks and activities and actions that they can take. And let's say now this is like you know let's say DevRev is kind of the central hub where people can just get work done right.
Speaker 21:10 How do you see the like DevRev agents communicating with all the other agents and this is actually relevant now because we there's some level of standardization efforts going on in this space. But I'm curious about like how do you see that vision pan out? I mean first of all we absolutely expect like on the siloed systems that also there are going to be agents that people write that does one thing
Speaker 21:37 really well. May not have any other context about the other areas. The way that the system actually in the end has to play out is when you think about the agents, these are really the skills we are talking about. And if you could go and bring those skills and can execute it in near real time. Now the real question is just going to be when do you have really a lot of latency that you have to go and figure it out in real time that can you go and execute, get all that data, process all of that data together,
Speaker 22:12 pass it onto the next system, next system, next system, right? So anywhere that you have to go and do joins between the different systems to make head and tail out of it is where that you start to see or we learn quite a bit. Now today, the approach that we have taken is build agent on top of DevRef. Yeah. Bring the data sources connected as something that we'll, we provide the connectors. We call it Airdrop with Airdrop technologies,
Speaker 22:43 structured data on structured data, and then start to build your agents here. But I mean the architecture has stood in my opinion also expanded to that okay it can go and talk to other agents also. One thing we've been seeing in this space across many different companies is like everyone's building their agents in silos and as you said like there's like actual infrastructure components missing between multi agent communicating systems and that's one of the reasons why like you know Galleo along with some of the other folks in this space have created a collective
Speaker 23:20 to have standardization. What is like an agent protocol where every agent can talk to each other. Apart from that there's like what is DNS for agents? Like how does an agent discover other agents? In DevRev you can like, you have full control. So you can say like, hey I have my head of finance as an agent and I can tap onto his or her skills. You can pretty much
Speaker 23:47 create a structure within the product but when it's talking to other systems how can I detect like in my organization I have and what's my the people ops system that I want to or HR system that I want to tap into? There are certain components and there's also state passing and authentication extension. So these are smaller things that are being worked on now but as this collective
Speaker 24:15 we are creating like a way for everyone to come in and extend it. Like it's not just open source with a closed decision maker, it's like everyone contributing to the standard. So we're seeing that there's a huge amount of interest in doing that and as part of you know the multi agentic future I mean I would say like to your point about everyone is in silos and like building their own agents
Speaker 24:42 in each doing one good thing. The true ROI between, multiple systems will come when, like, agents can seamlessly talk to each other. I 100% agree with that statement.
Conor Bronsdon 24:54 I'm curious what other challenges you foresee as we build this multi agentic future. Obviously, inter agent communication, protocols like, you know, model context protocol can help with tool selection, these type of things, but what other challenges should we be preparing for?
Speaker 25:11 Very first thing that I feel inside the enterprise, the data and data silos. Also just the cleanness of that data, that in itself will take a little while. At least I think this is the first step. What we are experiencing right now, the initial euphoria that every single company thought that they can just go and build something, a chatbot. That was the very easy answer. Even there the companies found very, very quickly
Speaker 25:44 that, you know what, for the same question three times I'm asking, I'm getting different answer. Yeah. And either way, I don't know which answer was correct. I need to go and ask the subject matter expert now. Like Rag is not same. Built by one company versus second company versus third company. What does the context itself means? Like how good are your embedding models?
Speaker 26:07 So some of these things that we are corner like right now, just going through. For us, because we started in 2020 and we, at the time that we were looking at commercially available vector database and something that can scale for very small company to very, very large company, nothing existed. So we had to build our own and we went through like just the three iterations of the rag itself. Now back then nobody was calling it rag.
Speaker 26:35 Now we are calling it rag, so it's easy for people to understand. But yeah, I mean the challenge is a very very normal, very very early days right now. Initial thing was just going and doing the semantically search something that is context aware. And then the second thing was that okay is this thing especially for the enterprise from the data governance perspective, the permission perspective, RBAC and all that,
Speaker 27:03 does it like really follow that? Then the third thing is that every time that you're doing it, especially in the context of the organization, there is law of the land that there are regulations that you have to go and follow. Is there an audit trail that you're going and keeping on what AI is doing and not doing? How it is retrieving that information? What exactly is the information that it's providing? So there are many areas right now, Connor,
Speaker 27:31 in my opinion that we all need to work. The big picture that we all know that the conversational AI is here. We resonate a lot with that point at Galileo because our focus is always to help build trust in these systems. And so the very first point you made about like non determinism in the system, like that's what got us motivated to even start the company because even at Google we're trying to launch
Speaker 27:58 early versions of Gemini and it's incredibly hard. If Google gave you a different answer every time, would you trust Google to be your search engine? No, right. You know, it comes to the search engine for the enterprise, DevRev has the same issues. How are you and your teams thinking of instilling that trust in the end user? Because let's say if I ask the agent Manoj for
Speaker 28:26 hybrid cloud data, how do I know whether it's like whether I can rely on it? It's a great question by the way. For the internal people, you can always this chain of thought that you want to provide, the linkage, where you're getting this information from because and that's the way that you also Learn, I mean that then you start to commit. This is the way that it should be you trust.
Speaker 28:59 You actually teach the machine. This is the way that I think. In fact you are right. This is your chain of thought makes sense to me. Let's commit. Or when you disagree that you can ask it to go and construct it differently. When it is in a customer facing world where customers are interacting, you don't have that luxury. They're not the ones who are going to come and train. Only thing that you can do is maybe just getting the, I mean, obviously reinforcement learning, I mean, whether you got the answer or not, and then there is system internally that is continuously checking
Speaker 29:32 that actually what information that we gave to the customer, that was the right one or not, especially for the more complex one. Knowledge base and all those are the early days, but I mean, when you start to get into the really, really complex thing, you start to look at the reasoning stuff and so on I as mean, you'll have to build a lot more trust.
Conor Bronsdon 29:51 I'm curious how you're thinking about how other business leaders should be taking action. What advice would you give to other leaders that you talk to as they think through implementing AI in their organizations and trying to solve these information challenges that are coming up throughout?
Speaker 30:11 That's a great question, Conor. Like when I talk to the business leader, don't go just after the AI. Obviously, you'll have to think about what's happening in the world, but it has to be the use case, the pain point that I have. Recognize those pain points in the context of, let's say, support, customer success, when you start to look at it. Okay, today my cost for
Speaker 30:37 every single call that is coming from the customer, there's certain amount of cost that I have. My business is growing. I expect to have more cost over time and I somehow have to reduce the cost. Now that is a great business problem, pain point that should we now look at something from the AI? Can I really go and solve for that? Now, can I really go and even
Speaker 31:03 take it to the next level, which is the information that what L1 used to provide, which was just based on the information, could I extend that to L2 also? So if 30% deflection suddenly start to look like 60% deflection, so that will be the extension. Okay, I was able to start, I was able to get the 30% cost reduction right away, but now I can get to the 60% level.
Speaker 31:30 So there are ways, I mean, that's where that you go and start to insert. You can't go and say that, okay, I'm just going to go and change everything. So that's one that we advise. Second, would say, Conor, is quite related. Companies, they'll have to look at their own workforce. If they are AI enabled themselves, will they embrace this or not? Because to me,
Speaker 31:58 a human with AI is certainly going to displace human without AI, and it's not like something this had been taught as part of the learning coming out of the college or in the organization today. So the question is that okay, are we also going and training inside the organization? People, like the same people, will you hire them today or not with AI? So I mean, there's a huge amount of responsibility and work that is ahead of us in every single enterprise to,
Speaker 32:30 train our people in the the use of AI.
Conor Bronsdon 32:34 I I think we're honestly underrating how much effort there is there. Obviously, we're already seeing people who are effectively leveraging AI systems, whether it's agents or simply great, you know, prompts they're creating, are outperforming many other people within the workforce. But the systemic challenge of how do I get my whole organization engaged in a way that is positive
Conor Bronsdon 32:57 and has reciprocal benefits that stack upon one each other can be really challenging. An example of success with this I would give would be an app like Cursor, where as, if you can get on enterprise license and you have all your engineers using Cursor or Copilot, the self reinforcement of that learning about the code base, about how your engineers are coding, can not only speed up an individual engineer's code delivery, but also will improve the tool's ability to
Conor Bronsdon 33:22 actually write great code for your code base and align to your needs. And I think we're gonna see a lot more of that happening, at least particularly in the technical area. But I think we're maybe not doing enough of that for some other less technical organizations. I obviously, there are a lot of firms spinning up for marketers of, oh, we'll solve your content problem, do x y z, but the training across the board has felt ad hoc so far. And okay, great customer success, brought up earlier is, we're starting to solve that like first question problem, or like we got a lot of the early queries, or can be solved by an agent, but how are we enabling our CS folks across the board to leverage AI? And there are certainly organizations that are doing it well, but it seems like there's a gap here right now.
Speaker 34:10 Have you heard, Connor, like, inside the organization that from machine, they want perfect answer always. But from human, we are okay to get different answer.
Conor Bronsdon 34:23 Yes.
Speaker 34:24 Right? It happens all the time. Like, you go and teach people the same content. You ask a question after that, you get different answers. And we are working with that. And with machine,
Conor Bronsdon 34:37 particularly, we expect that it has to do the perfect job. Satya talked about this in a podcast I listened to recently about this trust challenge where, you know, we have a built in sense of trust with human beings, whereas we have this higher expectation of our co pilots, of our agents. It's less of a like, hey, can we get agents in at scale? We're figuring the tech out, like that is coming,
Conor Bronsdon 34:58 but there is a trust challenge that we have to achieve. And to me that speaks to the need for evaluation, for observability, for security, for all the things that, organizations like LA are working on. It's definitely an exciting time, but there's a lot of work to be done.
Speaker 35:12 I'll add just one quick point there, right? Like I think just going one level deeper into that issue, we face the same problem on our teams where like AI driven features, like some folks on the team will adopt it very quickly. The others will be skeptical and I think the paradigm of conversation is very different when you type something in a text box versus there are
Speaker 35:38 physical cues to know that okay this person may be not sure so like you know I may say hey I think it's supposed to be done this way so does that mean I'm sure or not right but every answer that we get from a machine it seems like it knows it all right and when it does it so when if I say every single sentence confidently this is what you do this is what you do and maybe even 10% of the time it's wrong you'll start doubting whether I'm the right expert
Speaker 36:08 for it. But if I can tell you something that hey look I think it's like that is in that direction but I'm being fair and confident I'm being fair in conveying my level of confidence That is missing in machines. Absolutely. I mean obviously that's why we doubt it. Remember that there are there's always this 1x developer versus 10x developer that in the industry we talk about. The question is,
Speaker 36:37 the machine can make this 1x developer to maybe 6x, 7x developer. We'll at least deploy the AI there first before we start to talk about, okay 10x will become, don't know what. We always want it to be 10x. Now on 10x are the people they are doubting the most, saying that, sorry, it's not able to do the way that I do it. Manoj, I wanna drill in on that vision you're talking about,
Conor Bronsdon 37:02 of the relationship between humans and AI and the data you're getting from customers and from other sources. Let's say we solve that flow of information and we enable both humans and machines. We have solved information symmetry. How should companies look? What is that vision of the future for you? At the end of the day,
Speaker 37:25 any technology that you build is supposed to go and help human, period. In that process, businesses are also that they need to have the top line growth and bottom line growth, both. That has to happen. That will happen through this. But if you just focus on the first point itself, amount of time today that is spent on just getting the right data, the number of people that you have to go and talk to today.
Speaker 37:56 I mean, just think about like the software development, how percentage of time that people are spending in writing the code versus just trying to go and gather the information, the log files and talking to the people, what exactly has happened there to find the root cause before that just going and changing that line of code, even changing the line of code that I need to find out all the changes that has happened in the system around that time. Can you imagine that work can be done by the machine
Speaker 38:25 easily? Machine doesn't get tired. Can do that work for you. So at least the way that I visualize this, the data, the fact that the machine can now bring it together, can make a head and tail out of it, can understand the customer, can understand the law, can understand the code, can understand wherever these things has happened, distill that, bring out the information,
Speaker 38:51 give it to the human. This is the way that I think probably where the problem could be
Conor Bronsdon 38:57 with higher certainty, one, number one, number two, number three, and so on. We got a lot of work to do, but I'm very excited to see everything that DevRev is putting out. For folks who are listening, what's the best place they can go to to find out more about DevRev?
Speaker 39:10 Obviously devrev.ai is a great place to start. We do have our applications that is available. It's a SaaS offering. You can just still sign in. There's no cost, no question asked at least. You can try it out yourself. We are available. I mean you can schedule a demo. We have DevRevU or it's like U for university. We do a lot of teaching, learning community that has been built through that to talk about this topic also and even start to talk about this conversation around the product and customers and team intelligence
Speaker 39:47 and all of good stuff commerce.
Conor Bronsdon 39:50 Fantastic. Well, Manish, thank you so much for joining Yash and I today. This has been a really wonderful conversation. It's been a ton of fun. Likewise.
Speaker 39:57 This was awesome. You know, learning from you about, you know, where the future of information is going. It's it's pretty exciting. Looking forward to building.
Conor Bronsdon 40:09 Definitely. And you can always tune in to more Chain of Thought episodes or watch them on our YouTube to dive into a lot more about the future of information and what we are all building. So make sure that you have subscribed to Galileo's YouTube channel for more content, including webinars about how to do agentic evaluations, what the future of multi agent systems look like, and much more. Check out our events, deep dives, and of course, every episode of this podcast will be there as well, and you can watch us interact with our lovely guests. Manoj, thank you again for joining us. It's been fantastic.
Speaker 40:40 Really enjoyed the conversation. Thanks for having me.