Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Android | TuneIn | RSS
Rebecca Evanhoe practices, teaches, and writes about conversation design, a key UX practice that is taking on fresh importance in the age of chat-based AI applications.
Since the publication of her book Conversations with Things (co-authored with Diana Deibel) three years ago, the tech and media worlds have fundamentally transformed, but the conversation-design principles that she teaches remain as relevant as ever.
We talked about:
- the conversation design and UX writing courses she teaches
- reflections on the book she co-wrote several years ago, “Conversations with Things” and the changes in the conversation-design world since
- how the focus on principles in
- a framewwork set out in their book that helps designers decide on whether or not and how to ascribe personality to a chat agent
- her identification as a UX designer
- how she’s incorporating LLMs into her course curricula
- her take on the misappropriation of the term “prompt” in new practices called “prompting” and “prompt engineering” and their divergence from traditional use in the conversation design field
- the differences in the conversation designer role in the LLM world compared with NLP
- the linguistic concept of “conversation repair” and how it manifests in “bot land”
- how to adjust confidence level in conversation design
- how intent classification in NLU works
- her preference for humans and human conversation
- the importance of including people with a humanities background in conversation design
- the ongoing importance of humans in the content and conversation design process for our ability to think strategically about how to maximize the success of conversational technology
Rebecca’s bio
Rebecca Evanhoe is an author, teacher, and conversation designer. With degrees in chemistry and fiction writing, she’s passionate about how interdisciplinary thinking can combine arts, humanities, sciences, and tech. She teaches conversational UX design as a visiting assistant professor at Pratt Institute, and co-authored Conversation with Things: UX Design for Chat and Voice (Rosenfeld Media, 2021).
Connect with Rebecca online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Content and AI podcast, episode number 19. We’re all talking to computers a lot more these days – telling Alexa to set a timer, asking Midjourney to create an image for a party invitation, or prompting ChatGPT to draft an outline for a slide deck. Rebecca Evanhoe is an expert on the interaction design practices that guide these conversations. Three years ago, her book “Conversations with Things” set out a principles-based approach to conversation design that remains super-relevant in the age of large language models.
Interview transcript
Larry:
Hi everyone. Welcome to episode number 19 of the Content and AI podcast. I am really happy today to welcome to the show Rebecca Evanhoe. Rebecca is really well known in the conversation design world. She’s a conversation designer. She’s the co-author of the really excellent book Conversations with Things that came out a few years ago, and she teaches conversation design and other kinds of design work at Pratt University in New York. So welcome to the show, Rebecca, tell the folks a little bit more about what you’re up to these days.
Rebecca:
Yeah, hi Larry, it’s nice to be back. Yeah, these days I am teaching, I think you said conversation design, and specifically this semester I’m teaching a class in UX writing, which I love because it doesn’t matter what kind of writing I’m teaching, it’s like a chance to think about language and celebrate how cool language is with my students. And yeah, I’ve been teaching, I am doing some work at a cool place that I won’t get into here. But yeah, it’s been a really interesting couple of years.
Larry:
Yeah, because we last talked right before your book came out, I think it was maybe a few months before the book came out. And since then, I mean conversation had been a thing. I had talked to Phillip Hunter and several other content designers before I had you and Diana on the show, but it seems like I’m going to guess that more has happened in the last four years than in the four years before you wrote the book. Is that accurate?
Rebecca:
I think that’s definitely accurate. Yeah, our book came out in April of 2021, and I think that ChatGPT became publicly available in November of 2022. So our book has been amazingly well received, tons of enthusiasm. It really seems to be sticking around and people are finding it useful. But if you control F and search our book, there is not one mention of the term large language model. And I think there’s only one mention of natural language generation.
Rebecca:
It’s been interesting to look at our book through the lens of the fact that technology keeps changing. And I think, and other readers think as well that it’s based enough in principles that it really applies to any conversational technology, or at least that’s the hope. And when I think back about the things that have happened in the last few years, when the book came out, I remember people kind of wanting us to put a couple things in the book that we didn’t.
Rebecca:
People really wanted more information about how you build an Alexa Skill or a Google Action. Those were very visible at the time. People also wanted us to put a list of prototyping tools for conversations into the book. And we didn’t, and I think things like that future-proofed it a little bit, because Alexa Skills and Google Actions… Like Google Actions aren’t around anymore, Alexa Skills are very much de-emphasized. And a lot of the prototyping tools that we had a few years ago were acquired or were kind of sunset. So yeah, I think we made some lucky decisions to future-proof it. But certainly it doesn’t have a mention of LLMs, which is-
Larry:
Well that’s really… I got to say, this is super interesting because I remember from, we talked on the Content Strategy Insights podcast about this, that you and Diana both emphasized principles. And I don’t know that you specifically stated that, but in retrospect it’s like, yeah, that’s way better than focusing on any specific technology or practice. Can you talk… I remember you covered those really well in the book. But is it possible to do a quick overview of some of the guiding principles? And maybe more to the point, how are they helping you through the arrival of LLMs and generative pre-trained transformers and all that stuff?
Rebecca:
Absolutely. I think that one of the concepts from the book that has become even more important today is the idea… And in our book we call it level of personification, and it’s in the personality chapter. So I think a lot of people are thinking more about personality design, but also specifically how much of a character, how much of a mind the AI is sort of presenting itself as.
Rebecca:
So is it presenting itself as a fully realized character that’s your friend and it refers to itself as I, or is it behaving more like a machine? So the example that I always give is if you have a remote control where your voice is the input, it doesn’t need to be named Sandy, and it loves, Thanksgiving is its favorite holiday, and… It doesn’t need to be a character in mind. So thinking through that spectrum really for any AI experience you’re creating, I think is really important. How much of a person should it present itself as? I think that becomes a lot more visible.
Rebecca:
And an example that I would give for the LLM world, it’s like, if you talk to chat GPT or Claude, those bots use I. And you can ask them a little bit about themselves and they’ll tell you, they’ll generally clarify like, oh, I’m an AI so I don’t have feelings, but I can describe feelings or talk to you about feelings, stuff like that. But then there are other LLMs that they don’t have any personification at all.
Rebecca:
So for example, Perplexity AI is a platform that is an LLM and you can talk to it and it does all the LLM-ey stuff, meaning you could ask it to summarize, you can ask it to give you bulleted lists, you can ask it to imitate a turn-taking conversation with you, but it doesn’t really present itself as a character at all. And I think those kinds of decisions are still very much ones that conversation designers should be involved in, because that level of personification really impact user expectations, how they’re going to behave toward it, and then how successful those interactions are going to be.
Larry:
Yeah, that’s really interesting. How do you make that decision? Because I can picture making the wrong decision for good reasons. Oh, we like our customers, we want to be close to them, so we’re going to act like their friend, where it would probably be more appropriate in a business setting to not be that way. Are there sort of guidelines around that, how you decide the kind of personality?
Rebecca:
Yeah, I mean in our book there’s sort of a framework that walks through a lot of the facets of it. But generally I would say I think people over-personify these interactions. They think that having a character must be more interesting and fun, and they forget that people want their thing fixed, they want their task completed, they want their problem solved. And people also forget that lots of people are very happily solving these problems already through an app, through a website. People do like to solve their own problems, and conversations are not necessarily easier and more efficient unless they’re designed to be so.
Rebecca:
So yeah, I think one of the things that we think through in the framework is first defining interaction goals that are independent of the conversation. So an example that I always use is, if you’re making a voice bot that takes orders for a drive-through, which is a use case that lots of businesses are chasing, and maybe even closing in on, finally, the interaction goal is efficiency.
Rebecca:
So that’s what the business wants to do with this voice bot. Make drive-through orders efficient, or maybe even efficient and more accurate. And so if you have a bot that is like, “Hey, my name’s Becky and I’m here to serve you, how’s your day going?” It’s very personable and warm. You are actually drawing people into a chattier space, and perhaps harming that interaction goal of efficiency. You might actually be making it less efficient. So thinking about what you’re really trying to accomplish, the personality is supposed to be in service of that, as opposed to making a personality that just seems fun to the people designing it. Because what we think is fun as tech industry nerds is not what everybody thinks is fun.
Larry:
That’s right. And the most successful designs just kind of disappear into the background. You don’t even know that it’s there. And as you were talking just now, you kind of reminded me that I think if I pinned you down and made you identify the highest level, top level, classic categorization of what you do for a living, you’d call yourself a UX designer. And you just happen to be manifesting that as conversation design, UX writing. Is that accurate?
Rebecca:
Yeah, I always say that conversation design is a form of UX. And it follows the same process of researching the problem. What is the problem? How are people solving it today? What are they struggling with? The classic design concepting, hypothesizing the best way to solve the problem. Testing that hypothesis through prototypes, and that is supposed to be an iterative process. So that’s what UX is, and that’s what we do.
Rebecca:
Although I should say that not everybody… I don’t have a formal UX background. I have a writing background, and then started working in conversation design, and then realized that I needed to learn the UX skillset for it to be successful. So I think not every conversation designer feels comfortable saying like, oh yeah, UX is my thing. But I think it’s inherently the framework that we work in.
Larry:
And as you talk about that, as you say that I’m thinking of… I’m old, I’ve been around forever. And I remember when talking to Alan Cooper and Jacob Nielsen before there was a thing called UX, they were usability guys back in the 90s. And so it’s like it’s not even that old of a profession. I don’t know, it’s just really interesting that we struggle with this all the time in content strategy and content design is like what even to call ourselves, but just that idea that it’s always in service to people.
Larry:
And in your case, among those people, this is something I want to circle back to, are your students. And as you’re talking, you’ve been setting out some of these principles and talking about best practices and things. How does that manifest in the curriculum at Pratt when you’re teaching convo design or UX writing?
Rebecca:
Yeah, I definitely have to keep updating, I never get to set it and forget it with my courses. What I’ve been doing, I mean industry has not fully decided what to do with LLMs. Conversation designers are still figuring out and experimenting with where these fit in, and whether they become the focus, or whether they’re just a tool in the toolkit, part of the tech stack. So that’s something people are thinking and disagreeing about and having discourse on.
Rebecca:
With my students, basically what I’m showing them is, my class is about how you think critically about interactions with conversational tech. So no matter what the tech is, we’re looking at it and going, was that a good experience for the user? Was that a good conversation based on principles that we know about cognitive load and plain language and all these content delivery concerns? Did the person get the content that they needed in the way that they expected?
Rebecca:
So I’ve added in a lot of stuff. I sort of added an LLM component to most of my assignments. So I’ll have my students do an activity where they look at real training data, and they kind of do an intent classification architecture. So they sort of experiment with that process, and then I have them use an LLM to do the same thing, and then compare which one did it better. Was the LLM as good as you at this thing that you just learned how to do in my class? I’ll have them write their own sample scripts and then have an LLM do the sample script. And usually they find that the LLMs aren’t doing exactly what we would like for them to do in the conversational space. That’s kind of where the tech is today.
Larry:
Interesting. And as you say that, I’m wondering, these are learning models. Are they getting any better at it as you go? I don’t know how many classes you’ve taught, but it would be interesting over time, each cohort of students, how well do the LLMs do versus them? Are they getting any better, or is it just an inherent human capability that machines might take a while to get up to speed on?
Rebecca:
Yeah, I don’t know. I mean, I think they will get better over time. I don’t think I am suddenly seeing things turn a corner through my specific lens, but I think we have to assume they will get more and more capable over time.
Larry:
As we talk about LLMs, I’m realizing over the past year we’ve all become super aware of the term prompt engineering and prompts. And that’s how you interact with, I’m in ChatGPT every day, and that’s what you do, you write a prompt. And I went back and listened to our… We spoke on my other podcast about almost four years ago now, and the word prompt came up four or five times in there. So that’s always been just part of your lingo as a content designer. Is it used the same way now, or tell me a little bit about that.
Rebecca:
I think it got a little co-opted. So in our book, and we define every term we use, and we define a prompt as something that the bot says. And people we researched for the book, people use that term a little differently even then. Some people meant prompt as the whole response, and some people meant it as the cue that comes at the end. So we just defined how we were using it in the book as kind of the whole response.
Rebecca:
But yeah, now when people talked about prompting and prompt engineering, they’re not even talking about a turn-taking conversation. They’re talking about writing an instruction you give to an LLM to maximize the chances that you get the behavior and content that you want from the LLM. So I actually have sort of shifted what I… I don’t say prompt anymore, I say response. I just refer to it as, when I’m crafting a response that the bot’s going to say, I just call it a response. Because we’re starting… It’s so dominant to think of prompt engineering and prompting now, that yeah, it seems to have taken a swerve.
Larry:
That’s so interesting. And it kind of seems unfair that you have to change because they’ve co-opted the language, but doesn’t sound like the end of the world for you. But that’s super interesting. And as we’re talking about that, I’m reminded that these are large language models, and you’ve pretty much established your language cred just in all the things you’ve done and studied. How far deep in the stack have you gone with these? Or have you got to the point of working with the engineers and data scientists around training stuff? Or are you still mostly in the UI part of it?
Rebecca:
For LLMs, that seems to be, I as a conversation designer have only experimented with training an LLM in the sense of creating a knowledge base. So I don’t work very deeply with the original neural network concepts very much. Which is interesting because when we’re working on a system that uses classic NLU, natural language understanding, intent matching, a conversation designer is often, not always, but often the person curating, testing, tuning that system. And so with LLMs, we sort of have potentially a different relationship, and people are using LLMs in different ways or looking at it in different areas of potential. So there are people who are looking at an LLM as replacing the whole system. So since an LLM can take any input and produce, generate output, some people are thinking about LLMs as sort of replacing the whole stack. I’m not one of those people.
Rebecca:
I feel like the largest tech companies probably have… Google and Amazon probably have a lot going on in those ways, getting their voice assistance to be more large language model based. That’s how it seems. But then I’m looking at it a little bit differently where it’s like, can we use the LLM not to replace the whole stack, but to replace parts of it that will kind of plateau.
Rebecca:
So the two examples there are companies who are using LLMs for intent classification or for entity extraction. Can we replace the NLU part with the LLM? There are people who are creating the model where an intent matching classic kind of bot system would handle known requests. And then if a request is not matched to anything, having an LLM referred to a knowledge base to generate a response. And then there are, if you look in Voiceflow or Dialogflow, they already have beta features for just using generative AI for conversation repair. So there’s a lot of different… Some people are looking at injecting it into certain places in the pipeline, and some companies are looking to see if it can be the whole pipeline.
Larry:
Got it. And when you say, I want to back up just a little bit, conversation repair. I think I know what you’re talking about, but can you define that and talk about what it’s fixing, what it’s repairing?
Rebecca:
Yeah, absolutely. In linguistics, this concept of conversation repair is like when people talk to each other, they never fully understand. So sometimes we indicate that we need more information or we disambiguate. We say like, “Oh, I didn’t hear that word.” Or, “I heard what you said, but I didn’t understand what you meant.” Or, “I missed the last half of that sentence.” Those are kind of human conversation repair things. And then when we translate that concept over into bot land, conversation repair would be all the things that we build in so that if the thing the person said didn’t match or it matched to two things and we needed to disambiguate, we set up systems for handling that.
Larry:
Got it. Okay. And that role, it seems like that kind of repair, both the process and the content of it seems like, is that in your purview as a conversation designer, or is that something where you’re collaborating with engineers?
Rebecca:
That would be in my purview, and sometimes it is collaborative. So for example, the confidence value where… There’s a thing called the confidence value at confidence level that tells you on a scale of zero to one, how similar the utterance is to the training data it’s comparing it to. So you can play with that confidence level like, oh, we only want stuff that’s 80%, 0.8 or above, to be matched. Otherwise they enter a conversation repair state where we try to clarify or get more information.
Rebecca:
So we often test that threshold, and that is something that at this stage in my career, I could do on my own, but earlier in my career I would work with developers to experiment with. So yeah, an experienced conversation designer who has a career that has allowed them to be really close to the training data would be able to design experiments and know the more technical aspect of tuning and training.
Larry:
Got it. Has that changed? You mentioned earlier that we’ve kind of gone from natural language understanding to LLMs. And the LLMs are doing it, they’re doing it a different way, right? The LLMs are doing word embeddings, vectorizing them, then predicting stuff based on that. I don’t know how NLUs work. How did they understand what was going on?
Rebecca:
Oh yeah. So basically you give… And the NLU systems, there are other types, but I’m most familiar with an intent classification type of NLU. And you basically give it curated sets like bundles of training phrases that are grouped together by their semantic meeting. So you’re giving it a set of different ways to express the same thing, which we call an intent. So there’s a bunch of different ways you could say, “Turn on the lights.” Your training data set represents a representative sample of different ways you could say that. And then when the algorithm is doing its job, it’s taking a new utterance from the user that’s not necessarily in the training data, and just finding based on the words and the word sequences, and what’s found in the training data, it’s finding the most similar set that individual utterance is close to.
Larry:
And that sounds like an intuitive description of what natural language understanding would be. Like, hey, I’ve heard this said a bunch of ways. That sounds like it’s in the ballpark, you must intend to say this. That’s sort of the idea.
Rebecca:
Yeah.
Larry:
Got it. Okay. Yeah, this is fun to noodle on. I mean, how much fun is this for you? Because your book was entitled Conversations with Things. And so I mean, the intent here is for this to be proxies for human conversation, but you’re talking to computers a lot.
Rebecca:
Generally, I do have fun. But people are actually more interesting to me than the tech, and the tech is only interesting because it’s trying to imitate people, and a lot of the times badly. So the part of why I teach it is is it fun to help my students have experiences where they learn and form their own opinions. They’re usually more enthusiastic about the tech itself than I am, which is really interesting. But yeah, human conversation is very valuable to me. And so seeing my students’ opinions, seeing how they view things, I always learn something or think of something, or make a connection through what my students are observing. So yeah, I definitely like being in this field and talking about all of this stuff, but it’s always because people are interesting.
Larry:
Well, now you’re reminding me of something we talked about in the prior interview when we talked… I can’t remember if it was you or maybe both of you, I think it was you or Diana, but you talked about the importance of finding people with backgrounds in the humanities to do this kind of work. Tell me about your students. How many of them come from a humanities background versus other academic disciplines?
Rebecca:
Yeah, but the students at Pratt are in the School of Information and they’re graduate students, and most of them are getting the UX degree that’s called Information Experience Design. So it’s like UX, but with an information architecture backbone. And, yeah, a lot of my students are coming from all kinds of places. Some of them are interior designers or architects, and some of them are writers, and some of them have some UX experience or maybe some web design experience they want to expand into UX. But they, yeah, very interdisciplinary group. I often have some students who are, I think this semester one of my students is a screenwriter. So a lot of interdisciplinary thinking in the classroom, which is really nice.
Larry:
And so that tie between your focus on humans and the study of humanities, that’s sort of like, I don’t know, is there anything beyond the etymological similarity? Are there still things about people who study the humanities or the characteristics of the humanities curriculum that you think we need to be getting more of in conversation design?
Rebecca:
Yeah, I think that if someone comes from a strict computer science background, that means you spent most of your time studying this one kind of system, and usually a lot less time thinking about language itself, and to what extent it is and is not a consistent system. And I think it is very easy if you have not studied language or studied literature or just been in some field where reading and thinking about things from a more social or humanity based perspective. If you don’t spend time in that space, then it’s really easy to underestimate how nuanced language is, how complex people are, where meaning resides when we exchange language. Those kinds of things are very visible to anybody who’s from a humanity background, and really surprisingly invisible to someone who’s just studied computer science, for example. In my experience.
Larry:
Yeah, no, probably half of my friends in this world are engineers of one kind or another, and they’re all great people, often lovely human beings, good social dancers, good drinking companions and stuff. But there is a difference in, there’s people who’ve been word nerds buried in the novels their whole lives, often are a little different from them. But it’s also a team sport, and I think it’s really great that we all come together in that.
Larry:
Hey, Rebecca, I can’t believe we’re already coming up close to time, but I always want to, before we wrap up, is there anything last, anything that we haven’t talked about or that you just want to make sure we get to before we wrap up?
Rebecca:
Yeah, I think my message for this year is a lot of people are asking the question like, “Oh, are our content designers still a thing? Are conversation designers still a thing?” And my answer always feels like an obvious, “Of course.” I look at what LLMs produce, and it is not as good as what people are producing, and I do think that will always be true. I think that people view a person like a conversation designer or a content designer, if they think of the role a little too minimally, they think of us as producing artifacts, but we’re actually thinking critically about the effectiveness of the interaction. And even if we’re using an LLM partner to draft content, which to me that’s not even that compelling, but it’s happening, you still need people who know what good looks like, to think strategically to solve the problems that LLMs are still perpetrating.
Rebecca:
So just to folks who are in content, I know that there’s a lot of fear, potentially layoffs, it’s always a strange time in tech when new technology introduced. But I think if you can explain that your skills are not just about producing content or producing responses or messing with training data, but your skill is strategic thinking to maximize the success of the conversational technology, then of course we’re still relevant. Hopefully that’s inspiring to some folks who are listening.
Larry:
Yeah, that comes up… Well, two quick things about that. One, this comes up all the time in other conversations that haven’t aired at the time we’re recording this, but they’re out there, that these are language models and we’re language people. So there’s that. There’s security for us in that. But wait, there was a second thing I was going to say. Anyhow, it’s not important. Well, thank you… So one very last thing, Rebecca, what’s the best way, if folks want to stay in touch, what’s the best way to follow you online?
Rebecca:
These days? The best way is to follow me on LinkedIn.
Larry:
Okay, great. I’ll put that in the show notes as well. Oh, I know the other thing I was going to say. A couple of episodes ago on the other podcast, I had Relly Annett-Baker from Google on, she leads a big team of folks who do technical content at Google, and she was talking about to the point you were just saying about that… She talks about it like 20% of what we do is writing, 80% of it’s figuring out what to write. I think that’s in the same ballpark of what you were just talking about. So I’ll link to that as well. Well, thank you so much.
Rebecca:
Yeah, absolutely.
Larry:
Yeah. Always great to catch up, good to talk to you.
Rebecca:
Yep. Thanks Larry. Bye.
