LLMs as Intelligent Assistants

Link to slides

 Do I think it's time to get started, so I'm gonna bring Sarah onto the stage. There she is. What's going on Sarah? Hey, thanks for having me. I'm so excited that this actually worked out and that we get to learn from you today. And as you know, your talk was. Uh, officially supposed to start four minutes ago, so I'm just gonna go ahead and jump off stage and I'll be back in like 20, 25 minutes.

Awesome. Thanks so much. Hey everybody. Very, very excited to be here today, hoping to share with you some of the amazing use cases that. Allow us to see how business users are thinking of using large language models in production, uh, specifically in the context of intelligent assistance. So I am at Salesforce.

Um, I lead a team of machine learning engineers and data scientists. And so very excited to bring to you today the work that we've been doing and, um, what you'll be seeing from us going forward. Now I have to start every talk by reminding you that Salesforce is a publicly traded company, and so if you are indeed a customer or thinking to make a purchasing decision, um, please do that off of products and services that are currently available as this may be very forward-looking content that you're seeing here today.

All right, let's give it a go. So no surprise to everyone here. Uh, but of course, generative AI is dominating all of our headlines. And while to many it may have seen like much of a buzzword, there is absolute consensus that this is a game changer. It's the type of revolution in tech that we haven't seen in the last two decades, really.

There have been several that have come along, but this one is really a game changer with far reaching opportunities. Um, lot of. Benefits that we can see her. And of course, risks, um, like Demetria has called out around things like hallucinations. What we're seeing is that companies of every size and every industry are really racing to understand how they can use generative AI to help propel them into the future.

Now, by no means is this necessarily a brand new journey. Um, AI has been here for quite some time. Um, Salesforce, of course, has been working on this space for a very, very long time, but there is just a ton of pressure, um, to make sure that with AI you can meet your customers expectations. So in the past, there were opportunities for customers to believe that it's acceptable, uh, for long response times.

Those absolutely those days are gone. So, Customers expect for you to respond to them with speed. Um, that it is seamless in the communication that you are available. Um, of course they also now assume that. You need to have personalized content. It used to feel like one size fits all, and you could kind of get something close to what you wanted, but today, customers really expect you to anticipate your needs.

62% of the ones, uh, surveyed have this expectation at this point, and of course, All the time consuming tasks. The expectation is that everything should be convenient and automation of tasks needs to be present. So what we have really is a change where AI is no longer a nice to have. It is truly the expectation that AI power tools are going to be a standard for helping shorten your response time, personalizing those experience, and eliminating these very time consuming tasks that could indeed be automated.

Now, while those are the expectations, there is absolutely still a gap. So 70, 76% of execs, they say they still struggle to deploy AI on production. It's actually staggering to hear how often products that are, products that are worked on don't make it into production. And there's several reasons for that.

Everything from data limitations and quality. So if you bring in garbage data, of course, that's what you get out. There are integration challenges. So companies need to figure out how to bring all that data together, make sure they're connected, and of course, making sure that all of your processes can plug into these new tech.

So for Salesforce, we have really been in this space for quite some time, and I myself have been on this journey with Salesforce for, uh, coming up on seven years here. So what's exciting is that Salesforce has always been thinking about how to make it easy to democratize ai, truly making sure that everyone has access to AI as they are using Salesforce in order to improve those customer touch points and making them more intelligent.

What we've done at Salesforce is, Today we ship 1 trillion predictions a day, or sorry, 1 trillion predictions a week, which is kind of staggering when you think about it. Um, these are everything from opportunity scores, uh, to being able to, you know, pr um, predict a label for a case, for classifying it for triage.

Um, what this ensures is that you're able to convert leads faster for lead scoring for service Cloud. You can. Have on average a 27% reduction in support costs by using Einstein and bots. Uh, in marketing we see an increase in engagement for emails and in commerce. Of course, product recommendations allow major uplift.

Um, We knew nearly a decade ago that AI was gonna be extremely important, and so we invested in in-house research. Um, today we have over 200, 200, uh, AI research papers published and over, uh, 200 AI patents. Uh, we acquired, um, different AI innovators along the way. Everything from, uh, relate IQ and Prediction IO and MetaMind.

And of course we developed a lot of technology in-house. Um, I myself was able to work on the automated machine learning technology, um, different forecasting methods, and of course multiple large language models. The LMS. What I think is so exciting now is, aside from the world and the journey that we've been on for democratizing ai, I truly believe that at this moment we are seeing massive democratization of AI by these large language models, by putting them into the hands of every developer.

Even my aunt is using them, um, to try them out and to see what is possible. And the unlock that comes with that, of course, is everyone is able to envision how to make their lives easier, how to be assisted by AI in order to move faster. But of course with all of that, uh, there are still a lot of things that you have to consider.

So while IT leaders believe that generative AI is a business priority and they absolutely need to fast track it, and we do believe it's a game changer and will transform the customer relationship, management, the analytics, and help automate parts of the business, uh, there is still a lot of concern, like Demetrio has pointed out.

Uh, there are some risks around, uh, security, around trust. Um, bias, misinformation, the inaccuracies that come with hallucinations, and companies are worried about how their data is gonna be used, that everything is done in a secure way. And of course for Salesforce, that is front and center. How can we put this technology into the hands of our customers and ensure that it is completely secure and that they can safely deploy this new tech?

So for that, we have Einstein G P T, that is the world's first generative AI for crm. It allows sales teams to reach out, marketers to create landing pages and developers can write code. So I'm really excited to be able to show you some demos, uh, that we've recorded for you, just to make sure that we don't have any tech ledges here of what it is that we're working on.

Um, so that you can really see the art of the possible and think about how the work that you may be doing, um, could be changed every day. Now I do wanna pause and talk a little bit about our architecture because of course it's very, uh, important to think through how it is that we are building this. Um, so we did announce earlier this week, just on Monday, our AI cloud.

Um, it is this complete and unified architecture. Um, of course everything is grounded in trust. So we are starting on hyper force, which is our trusted infrastructure, um, that is secure and compliant. And on top of it, we have our platform that gives low-code tools to developers, um, to build out. Uh, then above that we have our large language models.

You can see it's embedded into our platform, and what it does is it allows us to offer it then to our builders. Who are then able to construct apps on this very secure layer. And of course then we ship our own apps, which are gonna be the ones that I'm gonna be highlighting for you. So very excited to talk to you about how we can think about this going forward.

Now I'm just gonna take a brief detour to make very clear how we think about trust and how we're putting that first. Um, so we do focus heavily on making sure that there are controls in place to make sure that we are able to leverage customer data. So customer, it's their data and they can use it, but we also wanna make sure that the trust is built in.

And so we have various, uh, various components that we are putting in place that allow the secure retrieval and the use the masking toxicity detection. So that we can deploy, our customers can deploy these powerful tools, but feel that there is complete trust that it stays within our boundary of trust.

All right, let's get to the use cases. So this is about how LLMs can be intelligent assistance. So let's take a brief pause and um, for those of you that aren't as familiar with, Salesforce, um, customer relationship management is really all of the touch points that you have with your customers, whether it's in sales, uh, where we're working to be able to close deals faster, um, by helping create emails, generate them, uh, service where it's about how can I, when a customer service interaction is happening, resolve faster.

Any request for marketers to be able to res to create more content. Uh, that is more personalized and more relevant in commerce when you're building ways to be able to interact with those customers to make it faster, to generate product descriptions, and of course, in it to be able to develop faster, to be able to write code.

Um, we have our force.com platform that allows our customers to be able to build applications on top of our platforms as well. All right. With no further ado, I'm about to take you through a tour of the types of products that we are building. Um, All right, so we're gonna start off in a sales use case. Um, we have a seller who's in the Salesforce experience here and on their homepage, um, seeing that a new account was transferred.

So they go ahead and ask, can I learn more about my, uh, new account here? Outreach and love. Would love to see any recent news. So the Einstein assistant is going to create, A summary using available data and points out as well that hey, scape has some expansion in the US market. It pulls up a lightweight description there on the left and offers you the opportunity to update that description.

So this allows the Salesforce data to be kept fresh with information, making it easy for the user to update as they see necessary. Now, since the news is there about the expansion, and my role here is as a seller, I'd love to learn more, and that's being offered to me. So here we see that we're presented with two individuals that are involved in this expansion.

Some of them are not yet in Salesforce, and so we are able to go ahead and create the contact directly again from this assistant. In another case, Mia, who's leading this expansion, I have the opportunity to compose an email. So with large language models, we're able to ground the information and. Create a personalized and relevant email reaching out to Mia.

Now we can edit it or we can actually also ask Einstein assistant to make it a little bit shorter for us, so make it a little bit less formal is the first request that we'll go ahead and make. We can see that we're presented with a new version that we can go ahead and edit live there. Um, I'd also love to actually send a link to Mia so that I can connect with her live in Slack.

So we go ahead and ask for that to be created in private channel and added on to that email. What's so powerful is how, this is all in the flow of work. You're able to draft an email, make any edits that you want, but you can also go ahead and send once you're satisfied with it. So it's that easy. Very efficient, the assistant is helping you get through this task quickly.

Now Salesforce, other product of Tableau. Um, actually the goal here is to allow the assistant assistive technology to be able to generate actual analytics for you. So asking how out scope is performing. We see that we are able to see a summary of the graph that is also the chart that's being generated for us there.

And I love the call outs of the colors that can actually happen down below. We see that relevant other related. Charts are being presented, other dashboards that you can click on, and of course you can mouse over and see all of the individual, uh, portions of that, um, dashboard that are interesting to mouse over.

So, All right, let's hop on over into service. We've all, um, interacted with, um, customer service representatives. Um, here's an example where somebody's coming in and asking about the temperature rating of K three Alpine jacket. So, um, we see that we are offered a reply, um, that of course we can edit.

Fantastic. Um, so it's retrieving relevant information and making sure that we can, with the request for night skiing and the temperature rating now understand what else we would like to provide back. So what's powerful about this is that as the requests are happening, we're, we're getting recommended replies that are using knowledge articles and content.

At the end, it can even help me create a summary for closing the case. And what's super powerful about is you also can create knowledge articles that can now be shared. Again, this is generated off of that conversation and the replies and can now be used to share with the rest of the team so that the next team member that has to answer similar cases can handle it.

All right. Another part of customer relationship management is marketing. So here we're gonna take a case where, um, this company NTO is creating a campaign. Um, so we're here trying to set up a new campaign, and again, we're going to use the assistant in order to build out a page. So we're gonna go ahead and say that we're gonna create a landing page, and it's for a V I P opening at the Park City Store for nto.

So on the left we can see that there is a page that's created with it. We have experience builder where you can go in and edit, but actually we can continue editing it right here in the assistant. So we'll start off by again, asking for a campaign message to be generated. Pretty great. Gotta also say that we can add it on to the left, but it's, um, maybe looking a little wordy there for us.

Um, so we'll make it a little bit shorter. Again, we see this generated content. You can always go ahead and add it and edit it as you please, but it's allowing us to iterate enough a lot faster. So turns out that we have the opportunity to meet the US Olympic team so we can add that into the content as well, and this iterative nature where we can both edit.

But ask the assistant allows me to move a lot faster, so we're gonna even generate an image for the header. Um, it's wintertime, so we're gonna go ahead and change it. Make it a little bit snowy.

All right. And then what other touches do we want? Let's add a title. Let's add a form.

It's pretty, it's pretty incredible. Uh, all of all of this experience is something that will really allow our users to move a lot faster. They can, of course, open it and experience builder and make edits there, but they can also just go ahead and publish. And so the level of efficiency that is gained, um, is truly dramatic, truly remarkable.

Last but not least, our developers, um, we have the ability to also assist right here, um, while we are, are writing our code. So we've been asked to create a controller, um, to be able to retrieve contacts. Um, so what we're gonna do is go ahead and. Write some comments. What's great is it's giving us the, um, auto complete, the the hinting.

Um, so we're gonna create a method to list contacts with name and contact information. Um, we're gonna limit the query to 10 records, so, Um, continuing to do the smart hi comment hinting, which is fantastic. Um, we have lightning components that we want, um, this to be invo from. And with that commenting, boom, we're actually able to get the code.

Um, what's really powerful about that is it's actually doing everything that we asked for. It's making it invo. With that decorator, we see that it's. Selecting all of the fields. Um, it's making sure that it's creating that right limit, and this can now be available, but of course, We need to test it in order to ensure that we're confident with the code that we're deploying.

And so in addition to being able to generate code, well it turns out we can also generate test scaffolding. Um, so let's go ahead and take a look at how that might work. So right now, uh, we're, uh, again, going ahead to write code, but starting with our comments. So let's start by typing that. We would like to create the tests for contact controller again, uh, hinting there happening for us.

Um, and. What's powerful about this is, again, we're learning while we're doing, so we can see that that scaffolding is just generated for us. Um, it allows our, it allows developers to be thinking about how they're working. Of course, any edits that they need to make can be happening right here, but it really, again, allows, it assists in being much more efficient so you can see how everything is available to us.

Um, really a game changer for, for our customers here. Now I know we went through that pretty quickly, so going to highlight a few of those again. Um, we started earlier talking about all the different parts of the customer relationship management platform that we have. So again, with sales, G P T, we're really focusing on how to sell faster and thinking again, how the LLMs and what has happened with making it so available, how it's been so much easier to think through.

What sort of, um, increase in productivity we can get by unleashing this AI for our sales context. Um, of course we also have service that we mentioned, that service experience. Again, how can we accelerate the. Resolution of any cases with AI as an assistant to move faster, um, help make your customers much more satisfied and making sure that there's a lot more efficiency unlocked.

So we saw how we were able to be assisted in the responses to any requests that were coming grounded in the information that is available in crm. For that individual customer. And of course then generating more knowledge to make it more efficient and speed us up as we go forward. And then lastly, we saw marketing.

Uh, it's pretty exciting to see how we could create some campaigns quickly. Everything from the image generation down to the content, um, and making it as easy as possible to create. Anything that we like, right in that experience around text and around images, and accelerating the time to be able to publish it.

All right. Well, that actually completes my whirlwind tour. Uh, the goal really was to show all of the different places in the context of customer relationship management where we are able to accelerate and help our customers be better, serve and interact with their customers. So hopefully that really unlocked, uh, some ideas for you all and shows the art of the possible, um, and really what sort of differentiation we can get out of bringing this to the, to everyone.

So awesome. Thank you so much Sarah. So there's some cool questions that are coming through in the chat and I want to ask 'em real fast. Uh, we are on a bit of a delay, so like we talk right now and then people see us in 20 seconds. So, So I think th I'll ask you the first question and then by that time, more questions will have come through and, uh, you are awesome on timing, which is you were, you are the clock today, and that's great.

You're making my job easy. So, question for Sarah. What was the input for the analytics demo data in rdbms? Ah, so one really important thing about Salesforce is. Um, customers use that platform and it is their data. So everything is per tenant separated, every organization. Um, and so we have a product, Tableau.

Um, customers are able to access data, um, securely have access to their data. And I really encourage you to check out, um, Tableau to learn more about it, um, both in terms of where the data can be that is being accessed, and of course, um, how you can also rely on, uh, data that's stored in Salesforce, um, to be able to access it in a variety of different places, including Tableau.

So, cool. So another one that came up is how is the input form created? Is it selecting from some pre-programmed set of available forms, or is it the l l m generating code for the form if the later, is there any validation of the generated code for the landing page example? Uh, the landing page example?

Yeah, so, um, a lot of this again, are to the possible, but for generate forms are something that are commonly used and generated. I really, again, encourage you to check out Marketing Cloud and the platforms that we have. A lot of what we're showing here is a combination of data that's available in Salesforce.

So again, looking at, for example, the sales example, uh, where you're able to ground in the data that is stored in Salesforce. And then of course there's the work that the team is putting together on being able to generate the right prompts and being able to make sure that. Um, that data is available, it's secured and, um, permitted to for use of course.

So, um, Salesforce actually has a lot of great control and the ability to, uh, generate content from that data that's available, but only your own data, um, and only really, um, sort of, uh, with a human in the loop since that is a very, very critical part of everything that we're building. Yeah. So, uh, I, there's, there's also awesome questions coming through here, and I have, uh, one that I want to ask just to selfishly, because in the report that we created, a few people were talking about how what they're thinking about with LLMs is where do we augment someone's workflow and where do we completely replace it?

And I imagine you've been thinking deeply about that for the past, what, whatever, seven or 10 years. So how do you look at those trade-offs? Yeah. For Salesforce, um, human in the loop is so, so critical, and especially as you rightly pointed out in the beginning, um, making sure that everything that is being generated is accurate.

Um, making sure that it is reviewed, um, making sure that it can be trusted. Um, and, and this is true for our, um, Pro or generative products. But of course it's also very, very true to your point for the AI models that we've been shipping for quite some time. So with that, aside from putting, you know, with the, the builders that we showed, um, in, in that graph that really allow our customers to build, again, models on their own data with just a few clicks, um, it's very important for us to also have that explainability element where they can go and check the reports.

Check, uh, for an understanding of what data is driving the predictions and how accurate those are. And even at the individual predictions, uh, having explainability. So the, the import of the importance of humans in the loop cannot be understated. Um, because they're truly what we're showing are assistance.

They're assisting in making us more efficient. They're assisting in helping us think and bring in more context, um, so that indeed the right decisions are made. So, Edits can be made, but it still is a, a process of speeding up, of prioritizing work. Um, that is the, the main differentiator. Um, and putting aside kind of maybe the repetitive tasks of, in the service case, needing to search knowledge articles and look for it, well, what if we can just surface them?

Of course, that's still there, of course, gonna be very complex use cases and, and requests that are coming in. Um, but wherever possible to leverage what's available in your knowledge, articles that are created, um, and then the multiplier that you saw of even generating new knowledge articles, um, but still with the human in the loop that curates and, and make sure that you know everything is true, trusted, and tried.

Yeah, that is so awesome. And it's, uh, sometimes it can get a little bit sticky, like you don't know, is this good enough? Is the technology good enough right now to fully say we can trust this and trust the output nine times outta 10 or 10 times outta 10, if it's a very, very important use case. Right? And, uh, and so.

I think that's awesome. One last question for you before we jump, because it's been like a few people in the chat have been like, yes. Plus one to this question, plus one please. And I know Salesforce, I've talked to actually a, a few people at Salesforce before and I know how serious you take customer data and the privacy of data.

So, uh, Brahma is asking, can you comment how you're keeping the customer data private? And especially this is me talking now. There are so many ways for these LLMs to kind of leak data and talk about different, um, data that maybe you don't want out there. So how are you thinking about that? And I think there was a whole layer where you're like catching things.

Before when I looked at the stack, Yeah. Uh, absolutely. So one, one very important thing to understand is everything that Salesforce does is focused around making sure that customer data is secure. Um, so your data is yours, your data is not our products. Um, everything that we do is precisely to avoid that.

Um, so going back again to the AI products that we build, like the builders, it is on your data. It is only your data built by you. Um, and that's so, so critical in order to make sure that. Um, you know, for the reasons of trust and then also for the reasons of relevance. All that matters is really your data and the signals in your data.

Um, so this is how Salesforce focuses on it by keeping data separate, by building models for customers on their data, because it is really their. Product that they are building and we are offering a platform and tools to make that possible in a way that is fully secure. Mm-hmm. And maintaining trust. Um, there was a lot of focus.

I encourage you to watch back our AI day where we really go deep on how it is that we're doing that and preventing exactly what you described, your customer, your data. It's not being retained to train and fine tune models. That is not the goal. The goal is to make sure that we can ground in the data that you have retrieve relevance.

And then make sure that we are able to generate content that is hyper-relevant to you. Awesome. So good. Well, Sarah, thank you so much for coming on here. I'm so glad that this worked out. This was like, uh, made my, made my whole conference that you were able to do this, and I really appreciate you showing us, uh, what you are up to at Salesforce and how you're thinking about things, especially in this new l l m paradigm world and how you can incorporate that into product.

So thank you so much. Thanks for having me. Really appreciate it. We'll see you later and.