Got Agents? Agentic Workflows & Architecture | Weaviate, Unstructured & CrewAI

Speaker 0:13 Hey folks. Welcome back to the Chain of Thought podcast episode two. We want to start by giving a massive thank you to everyone who has already rated and reviewed the podcast after episode one last week. We've been overwhelmed by the support. Thank you so much. This week on the podcast, we're bringing you Got Agents, a session from our Production Ize two point o virtual event. It was recorded live in front of an audience of over 2,000 people from more than 60 countries around the world. AI agents have captured everyone's attention and imagination,

Speaker 0:45 emerging as the next hot thing in AI. To help explain agents and why they've exploded onto the scene, we're joined by Brian Raymond, founder and CEO at unstructured.io Bob Van Laoght, cofounder and CEO at Weviate, Joe Mora, founder at CrewAI, and Yash Sheff, cofounder and COO of Galileo, is our session host. Yash, take it away. And thank you all for joining us today. In the last three weeks itself, I've met several teams across different verticals, several leadership teams as well,

Speaker 1:22 and the term agents itself is quite overloaded. As we go deeper into it, I'd like to set the record straight and set a baseline of what are agents? What do we mean by agents? And who better than Joe here, founder of TrueAI, to tell us about his perspective there? All right. So AI agents, that's a good question. I feel like everyone kind of like agrees and disagrees at it at the same time. So it's a little weird. For me, it's very simple. For me, having an agent is about having agency.

Speaker 1:54 So regular flows that is if this, that, that call alarms, that is not an agent for me. Me, an agent is something that controls the flow of the application. It uses a MLL m as a brain to kind of like decide and have some reasoning on how to go about things. But in the end of the day, deciding what tools to use it, when to use it, how to format the answer, how to go about a thing. And that would be for me, base definition of what an agent is, like, something that has agency.

Speaker 2:26 Absolutely. And to think of that, you know, there's, you know, we think of agents as like a planning phase and an execution phase, but when we go into the architecture itself, like what are some of the key components, and this is a question for all three of you, when we think about building agents, what are some of the most essential components of the agentic architecture?

Speaker 2:46 And it's, as we, you know, it's already been established as well beyond Rag plus Prompts. Right? So love to get your perspectives there. Look, I think the interesting thing is, is that, you know, this is a kind of a special moment. We're almost at the two year mark on the from the release of ChatGPT. And we started out by using just what's in memory in the models. Right?

Speaker 3:05 And we dusted off that meta paper and we hooked that sector up, right, to, to all the crazy stuff Bob's been building. Right? And the thing was alive. Right? So he helped counter some of the three problems with just using what's in memory and and the models, which that models are frozen in time. They don't have access to private data and they tend to make stuff up. Right? And so

Speaker 3:24 RAD became, you know, a dominant paradigm, interestingly, like, you know, in order to actually put these things into use, but to actually drive towards like multi step process automation, which is kind of how I think about the business value of it. Right? You need to take the models, you need to take the prevailing Rag architecture, and then you need to be able to set that into motion. I think that's, what's really exciting about what the folks over at Crewitt are doing and others, which is highly variable depending on

Speaker 3:55 the types of agents that you're working on. Right. And that you're wanting to set in motion. And so, you know, from like, you know, from that standpoint, it depends on like, you could think about like, what's going into like a Tesla and the self driving mode, right. As a genetic, it perceives the world that makes decisions and then takes action. Right. Which might be different

Speaker 4:14 from a trading bot that a hedge fund might use. Right. And so depending on the types of agents, right. Different frameworks are needed, different model hosting and serving capabilities are required. Right. And so what going all the way down to like the cloud architecture and the hardware, I think it's fairly variable depending on what, you know, what the actual

Speaker 4:36 business or operational use case is for the agent and like what data needs to make available to it and a host of other sort of related issues. Bob, you want to go next? Yeah, I don't have much to add there, so thanks for having me by the way, Josh. But I think what Joe said is very interesting, right? So it's like, if something has agency, so it can make a decision, are we going go left or are we going to go right,

Speaker 4:59 right, and rather than saying, if this, then go left, else go right, there's some reasoning to it, right, so I think it's as simple as that and the reason why it's so important is because if we look at data management and those kind of things, we kind of got stuck in that world, right, so low data quality, not getting to insights from your data, that's just still the number one problem with all the exciting stuff that's happening.

Speaker 5:28 So I recently had a conversation with a with a CIO and he said, like, he said, it's all great what you guys are building over there. I said, wonderful. He said he said, but I still have like 15 ERP systems and I can't, you know, I can't make heads or tails of it. Right. So if you can help me solve that, then that's the, and that's where, you know, the agents or the, as I like to say the agentic way of doing stuff

Speaker 5:50 comes in. And that's where the value is going to come from. And let's bear in mind that Rag, you know, I run Weeviate, so I'm very happy with Rag, right? Of course. You can imagine why. But the thing is it's also one dimensional, or one directional I should say. And there's a lot of exciting stuff that's happening, know, thanks to this agentic way of thinking,

Speaker 6:11 where we're gonna loop that information back into the database, right, and that we really start to build smarter systems, and that is extremely exciting. And it goes back because now I can go back to that CIO and say like, you know what, maybe there's a way to help you solve that problem that you have. So it's exciting times and that's what the agentic way of thinking is

Speaker 6:31 solving. Yeah. And to add a few things to that, I think you got a new in the head, like we definitely have seen kind of like a few plateaus as we go, but like we keep like finding these other things that unlocks more stuff. And honestly, I think that like agents, like as we start are usually kind of like straightforward, right? Like, Hey, I have an LLM. I have tools. I have a task. I would give you an outcome. What I find though, is that as you bring these things into production, especially like an enterprise company, then another, like a few other needs start to pop up. And those are more like regular kind of like a, how can I say a regular technology needs? We have seen this before, right? Since they're like, oh, I want a caching layer. I want my agent sees the same tools over and over. Oh, I want a memory layer. So it's not only about the caching, it's about being able to do the rag, pull the context,

Speaker 7:22 create kind of like a long term memory of everything that I have done and the things that I have learned. So I do need to keep relearning them and everything along the way. And then of course you go into the regular guardrails and you want to have a validation and you want to have everything in between. So I do find that these architectures, at least they tend to start simple, but once that you go into production, especially under your company, that

Speaker 7:47 security is another layer and distribution is another layer, then like you start to add a bunch of other components that kind of like can get the complexity to, to skyrocket a little bit. Exactly. Right. I think that's, that's, this is great. Cause like, I wanted to cover, I mean, data is the backbone of our agentic flows, but I think what I'm also seeing is that, for example, APIs, let's say we want an agent to call an API effectively.

Speaker 8:14 Do we have a standardized spec for that API? Is that API changing over time? How do we make sure that the agent is able to call that API effectively? Like these are, I mean, this is just one small example that as Joe mentioned, like security permissions, if the, if you want the agent to act on our behalf, how do we make that auth token sort of pass through towards the end system and get that, get the action approved from the end system, like who's authorized to make these agents take actions on their behalf.

Speaker 8:47 Some of these things like, you know, we are still in the early stages, but in order to like truly make agents operational in production, Some of the questions from the audience was also similar, like how do we have agents interact with ERP systems like effectively, right? Yeah. I can quickly say something there, Josh, it's like, I recently had a conversation with somebody who was super interesting and this person was he was proposing

Speaker 9:12 to like have an add on to the, to the RESTful architecture. Right. So that he said, like, if we see the return headers that are coming in an API call, he was proposing ideas where he would say like, who's going to consume this information? And if then the return header would say, well, it's actually not a human or like system button, a model that's going to consume the information,

Speaker 9:34 then how you would format what was coming back would be different. Right? And so I love how people are thinking about that and how people are even going back in time to, this case, restful architectures to say, like, can we do something there to rather saying like, you know, it's gonna be consumed by humans, it's gonna be consumed by an application or by model.

Speaker 9:56 So it's like, there's a lot of exciting stuff happening there. And because I think that more and more of these interactions will be handled by models. Absolutely. And I think one other kind of aspect that came up also in our previous discussions is with agentic workflows, the dependency on multimodal environments becomes even broader and even bigger. Like, I think question for you, Brian, on like, how do you see kind of just the challenges with digesting

Speaker 10:26 multimodal information being kind of a blocker in some ways for some agentic workflows today? What are we doing to get past that? I kind of break it down into three different buckets and you could break this down in a number of different ways. First is data availability. So for example, like Bob and I were at a conference last week where a CIO said, actually my number one problem right now for adopting generative AI is RBAC. It's role based access controls and just managing that across, you know, our, their enormous organization.

Speaker 10:54 What a boring topic and a boring issue to be blocking generative AI adoption, but like, it's real. Why didn't you say that to Ryan? Why didn't you say that during the past? But a lot of you struggle, right? Like this stuff. So data availability, data quality, structuring it. That's what Bob was just talking about. Right. And that's where we see, you know, embedding natural language or, you know, image data, whatever it might be generating triples, for example, indexing it in graph, try and make RAD work better, whatever it might be, you know, changing how APIs deliver payloads. Then three, like how you're actually serving that to the model.

Speaker 11:31 This, you know, right now, like for example, when we're using multimodal models, we're thinking like with best in the business, we're still looking at like twenty or thirty seconds for inference on just like base on basic stuff on just tiling PDF pages and boring stuff like that, that we do. Right. If you want this to be sub millisecond for, you know, like, oh, I have like a, almost a human feel to this as, as sensor data is coming in. Right. Then you'd need to be really thoughtful about like what sort of cocktail of models that you're using to power those models. Oh, because

Speaker 12:05 this is something that Chip and the other panelists was talking about the last, in the last panel, which is that like, look, sub 7,000,000,000, sub 2,000,000,000, sub billion text to text models might work absolutely fabulously for a lot of tasks. Right. And so being really thoughtful about which models you're giving, which workloads, in order to deliver that, that business value, but you gotta solve across that whole value chain. If you're going to do that, I think stepping back,

Speaker 12:31 what I'm seeing at least from like the, over the summer and into the fall are lots of startups emerging to go grab data in the wild. So it's like kind of solve old problems, go, go rip, you know, go scroll, scrape websites, rip HTML, deliver to the models. And then the stuff that companies like, you know, I'm in part of the value chain and unstructured, which is like get private data and deliver that and, you know, as as clearly structured of of a manner as you possibly can

Speaker 13:01 to help the models, be successful. Because at the end of the day, the math is never gonna change. You want to deliver to the context window exactly what the model needs, nothing more as efficiently as possible if you want to minimize cost, time, and also maximize performance. And accessing data is absolutely important. Like, yeah, just a question keeps coming up is like, how do we even let the right person access the right data and manage access that way? But from the accuracy standpoint, right? Like Brian, that you mentioned,

Speaker 13:33 maybe Bob, I'd love to, and I know you have your hand raised as well. So, you know, feel free to chime in here, but Bob, how do we ensure like agentic workflows are going to need more specificity on the data that is provided as part of the rag outputs and you know, how are folks, how are developers solving that today? Before answering your question, really would like to comment something on the previous point because the, we

Speaker 13:57 have a lot of people now listening and building, you know, coming up with ideas and building these kinds of applications And I think so things like RBAC, etcetera, are more like, you know, things that people need to go into production. Right. I really would like to urge people to just don't think back in time. Don't think backwards. Right. So we see that a lot of,

Speaker 14:18 a lot of things that I see is like how people try to implement this is that they just basically slap on the old techniques to something new. But I really believe that with AI, we are like in this new paradigm, how we do stuff. Right. So there are even new ways of doing, for example, RBAC. Now, what that means and how that works, I don't know, because if I would know,

Speaker 14:38 you know, we probably would be doing something there, but the point is that this is the time, right, to come up with new ways of, to think about, because you know, RBAC is very binary, right? So it's like access or not. Right. But what if it's like certain type of information or nuances of information that you're getting from the model? And I think that, you know, startups that will emerge

Speaker 15:00 that solve that, right? So, but not in like the old way, but like in a new way, just leveraging the models to do, for example, RBAC or whatever that's, I really would like to urge people to, you know, to keep thinking that step ahead. And then to answer your question, like how do people do it? Well, so one of the challenges that we have right now with like with chunking and those kinds of things

Speaker 15:25 is that sometimes the true value comes from more information where we don't have any access to, right? So let me give you an example. So let's say that you have like email and so we have like we've yet we have like a customer with a big use case in email and I happen to be using that application as well. So if I go like, okay, you know, what time is my flight today? Right. Then, you know, it does like a hybrid search to the database. It's properly junked. It goes into the model and then it will return, I don't know, a 01:15PM,

Speaker 16:02 right? But if I then ask the question, the follow-up question, so, you know, what terminal is it from? Then, you know, the answer I get like, don't know, that information is not available. And one of the things that we've seen right now is that the junking, and yes, sometimes it's hard and sometimes it depends on the information, but the kind of the techniques are now kind of here, right, how to deal with that,

Speaker 16:26 is that bringing together also had like what stuff with crew, etcetera, is doing is like bringing together different ways and different sources of information to really truly help the end user. That is stuff where I really would love to see more being built. So Rack, one dimensional Rack. So I often just draw it like as a line right from the query to the answer.

Speaker 16:49 It's brought us very far, but now we need to start to bring in loops, right? So what if you have more questions? What if we need to query multiple times? And that is something we at We've Hit, we put a lot of work in that. So we'll probably get to that later, but the feedback loops to get might be multiple sources of information, do multiple queries from the database.

Speaker 17:09 That is going to be tremendously important because that's where the true value sits. And let's not forget if we talk about what you just mentioned, right? With this one dimensional structure is like, we only talk about reading, right? Reading information. So I have a query, you get a result and it's presented to you. But what about writing? What if the agent you're talking to or interacting with is, you know, what time is my appointment? And it says, oh, you don't have an appointment today. And what if I then say, okay, can you book an appointment for me? Right. So that kind of stuff, a lot of work needs to be done there and it's happening. Of course it's happening, of course, but I would love to see everybody that's also listening to us right now,

Speaker 17:49 when we're done with our panel to get their hands to the keyboards and help us build more of these kinds of tools. After the panel, after the panel, not now. Don't start yet. Yeah. I was just going to share just for sake of just candor, a failure and a success that we've had, or let's say a hope, a probable success on the failure, or at least we haven't cracked the code yet. We've been trying to use agents to help automate

Speaker 18:16 the setup of connectors. So I'm gonna pull from Salesforce or S3, etcetera. Right? And it's a pain in the ass. Like we have, we've sat live with like lots and lots and lots of users. And like the fastest folks have been able to set up a connector is like fifteen to twenty minutes a lot because it requires like emailing an admin to get, you know, certain permissions and so on and so forth. It's a giant pain in the ass. Right? And we're like, oh, well, let's use some, let let's use, you know, agents for this. And it's proven incredibly difficult to have that work. So that's like the one we haven't cracked quite yet in order to like productionize it and put it, put it out in the wild for our users to leverage.

Speaker 18:56 On the success side, we're hoping to move this into prod somewhere time around, like after Thanksgiving around like reinvent December 1, but this isn't like prop. I, it depends on how strict you are with the definition of agents, but with the definition I gave you earlier, we're looking at like dynamic pipelines where models are sampling data. Like say you have a fresh, you're doing a fresh pull from S3.

Speaker 19:19 You sample it, you spin up a, you run a bunch of different chunking techniques in parallel. You have models judge, which, which kind of chunking strategy it prefers for that particular type of data. And then decides to implement that for that particular job to do the same thing for embedding model selection as like you could, if you wanted to for better model selection, which might be problematic, but like for all these cocktails and models that we,

Speaker 19:44 that our, our users rely upon to go from raw to like rag ready data. So I can do a baton task to Bob. Right. Well, you may have six or eight different models along that. And it might vary wildly depending on the type of data that you're pulling from. Right. On like which cocktail models and which settings, make the most sense. And so we're trying to automate that or at least make those dynamic by inserting

Speaker 20:05 one or more models to help perceive, you know, decide and then act in terms of like adapting pipeline configurations autonomously. Amazing. Joe. Yeah, no, I wanna, I wanna touch you one thing that Brian said that I think is super relevant and that is we are seeing a lot of people kind of like going after like use cases and agenda again and getting burned. Right. And sometimes it's not even about the truth that they're cheesing is about like, it's,

Speaker 20:31 I get it. I think it go back to what we're talking. Starts very simple, but once that you get, like, you're like, all right, now when I get a reliable, consistent results, you're like, oh, all right. So this, there's a lot more that I need to put into this in order to get what I actually need from this. So there's definitely something that we are seeing where like, some of our customers are coming to us saying like, Hey, we do this. We have this thing. It kind of works is very promising, but it's not up to our standards. Like when it affects your thing and other customers come up with like this

Speaker 21:00 crazy use cases, right? They are looking for a 100% accuracy. And honestly, right now it's not a time for that. You don't get a 100% accuracy with AI agents. That's not how things goes. You can definitely get humans in the loop and that brings you very close to it, but it's something that you to be mindful of. So whenever somebody shows me, he's like, Hey, I want to do accounting. I was like, well, you can definitely try it, but you're going have some cute little loop there. They don't pretend that you're going like automate the whole thing end to end.

Speaker 21:29 Absolutely. And Joe, to that effect, right? Like moving on from the data to the application side, like what are some of the learnings or best practices for thinking about framing agentic flows so that you're not going down a rabbit hole, the best practices that we can share with all our listeners today on getting their agentic strategy right from the beginning.

Speaker 21:53 I love to talk about that. So folks, everything that we're seeing out there, are seeing that it's definitely, it's definitely cross vertical. So like there's a lot of internal automation and operations that are like going on. And then you have like sales marketing research and coding. So there's kind of like a, there there's a hell big distribution, but what we're finding is that there is a, there is basically a normal distribution on

Speaker 22:20 the pattern of the use cases. So a lot of them are usually a combination of research analysis, summarization, reporting. Sometimes it's not all of those together. Sometimes it can like mix and match, but those kind of like those work like a charm. So if it's like, Hey, I want to pull data from somewhere from an ERP, from the CRM, from somewhere else. And when I conduct some sort of analysis, I want to have a summarization of sorts, and I want to put off a report, either Jason or whatever it might be so that I can push into another system. Those ones kind of like work pretty well. Then the other thing that people are doing that are kind of working great is this idea of like bringing human the loop when necessary. Like, take this kind of like a use case where we actually want to have human the loop. Like we're already saving a bunch of time on this, but I want to make sure that we get someone in there. I would say that's the other the other thing. And the final thing that I want to mention is

Speaker 23:15 the way that you roll out these things in companies. So we had a, we have a great interview that went out last week with the commercial CTO of Gemini from DWC. And that's something that they had to talk like from the ground up, because it's a big company and we are rolling out things like that. People start wondering like, all right, what role this AI agent's playing in here and how do I interact with them? So there's definitely a human component of like the companies that are promoting people to feel like you manage these agents now, and they're too, you're too sad. I can like having more sessions with that than others.

Speaker 23:53 I know we are coming up on time, but the one thought that I'd love to end with is for each one of us, highlight what needs to be addressed to make agentic workflows successful. Like what each one of us think is kind of one thing that we would love to have solved in the coming months for agentic workflows to see, you know, the light of date literally across all kinds of use cases.

Speaker 24:19 Cheryl, from everything that I have seen, the one thing that I think is a must have for this future of GenTech to live to its full potential is it needs to be fast and simple for companies to view them. There's, there's many other things that I could mention here, like enterprise readiness, security, everything. And like, now all the things are true, but I think being fast and simple is the most important thing

Speaker 24:50 because in the market that's moving so fast, if you say that companies need to spend like five, ten engineers for three months with a product manager at design or to get like something like a POC done that doesn't work. But if you make it that simple and fast to try things and throw things in the wall and see what sticks, then it screws up the economic in a good way.

Speaker 25:15 Where now companies can move as fast as the market and then double down on what works. So I would say that would probably be the most important thing from my point of view for the future of agents can like live to its full potential. All right, Bob, you're next. It gets harder for you and me and Brian. I'm going to be a little bit self serving answering this question, but the

Speaker 25:39 Advivator is super bullish on this concept of generative feedback loops and very soon you can already read about it and I'll share a link in a bit, but the very soon you can play around with it. And it basically means that you can, rather than prompting the model, prompt the database. What you're basically going to say is that, for example, when you create a collection in the database, let me give you the simplest example that you can think of. They say, okay, let's say that you have products in e commerce, you're like every product should be in English, right? American English.

Speaker 26:06 Then if you store something in Spanish, then it just updates that for you. It says like, well, know, and then you can choose as you use, do you want to keep the original content or do an upsert or those kinds of things? But now also think about, for example, data management, right? So we have like, we now work with a data sets like a factory data set and it says like, you know, all temperatures from the oven should be in Celsius.

Speaker 26:29 And then if somebody stores in the Fahrenheit, it just does an upsert and it's just like, well, you know, it's now in Celsius as well. And so you prompt database rather than the model and then you get full CRUD support. So the database can read more from the database. It can update in the database. If you can even delete if you wanted to, and it can create. So that is something we're super, super, super bullish on. And that's just that's in the domain of the database.

Speaker 26:55 So basically if you would be interacting to the stuff, you know, that the folks at that crew or at the structure are building that basically then the database gets smarter. Right. And we call that agentic workflows, right? So the generative feedback loop is a form of an agentic architecture. And that is something I'm extremely bullish about. So very soon more for and also for people to play around with.

Speaker 27:18 That'd be awesome. Can't can't wait to try it. I think just the the pace with those models are improving is still really exciting. You know, rumors are that GPT five is gonna drop in December. Right? And and we're gonna have even more performant and faster multimodal models. I think that's the long pole in the tent here for a lot of this. I think the stuff that Jao is working on is absolutely essential to being able to operationalize this and put this in the hands. But, like, at the end of the day, like, it comes down to to performance, which is what we were talking about earlier. And a lot of the performance rides on

Speaker 27:52 power and quality of these models, like as we figure out ways to shrink them back down again, that's great. Right? It means that they could proliferate, we could push them to edge devices. But at the end the day, it's like, my mom thinks it's a piece of crap and she doesn't use it. Like she doesn't use Siri. Like we're not there yet. Right? And so I think that we're driving in that direction. I think we're like, it's a shallow trough of dis it was a shallow trough of disillusionment on, on agents from like, for like six months in 2023.

Speaker 28:19 And like, they're kicking ass, but like, I think that we're driving hard towards that reality. And I'm, I'm bullish that in 2025, we're going to start getting there with a higher win rate. Amazing. And I'll, I just add on to that because like, you know, it's great to be able to build agents quickly in a more robust, you know, data driven manner. But in the end, from my perspective, I think we all want to build applications that can make it into production very quickly. And in the spirit of this conference today, I think

Speaker 28:48 one of the things that I would love to really solve for personally as well as the wonderful team at Galileo is how can we enable not only building agents fast through a better data driven perspective, improving access to more data, but also having the right set of metrics that people can rely on right from the beginning so that you're not building a POC in three months and then you're spending a year trying to productionize that, trying to make it better and useful as you said, Brian, it

Speaker 29:19 should be robust and useful right from the get go and if you don't have the right signals for the entire agentic chain, because it's even more complex than a simple rag setup, it's going be very hard for us to go from that POC to production stage without the set of observability and metrics. So, you know, our team at Galileo is also innovating and training models and systems on algorithms on what do we call agentic evaluations.

Speaker 29:46 And we had a Atin a talk earlier with Chip and Vivian about this, but we've been thinking hard about this. We have some ideas too. Lots more to come in this space, but I think without the right set of trustworthy metrics that teams can rely on, along with the data infrastructure and an easy way to orchestrate them, these are all the key building blocks here to build the agency. Yeah, and we now know that the best evil for an agentic framework is Brian's mom, so that's Love very

Speaker 30:18 it. Love it. Amazing. Well, thank you all. This was an amazing chat. You all had fun and I'm sure our listeners and audience as well. And you guys have been super engaging on the chat as well, which is amazing. Thank you all for being here. This is awesome. Thanks, Bye bye. Cheers. Thanks, everyone, for tuning in to episode two of Chain of Thought. If you want to check out more sessions from Production Eyes, you can find them at galileo.ai.

Speaker 30:48 We'll be back next week with predictions about Gen AI in 2025 and much more. If you are enjoying Chain of Thought, leave us a quick rating and review on your podcasting app of choice, or share the show with a friend that you think might enjoy it. Thanks so much, and we'll see you next week.