Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Android | TuneIn | RSS
The arrival of generative AI gives content designers a whole new toolkit. As with any new set of gear, there’s some learning that comes with the new capabilities that the tools afford.
At Expedia, Chelsea Larsson is leading her team of content designers into the AI design future with fresh takes on the planning, design, and evaluation skills that designers have always relied on.
We talked about:
- her work as a senior director of experience design at Expedia
- how she is facilitating with her teams the shift from product development design to AI design
- how she has identified new capabilities that AI brings and is incorporating them into product road maps
- how content strategists and architects help them decide whether to use generative AI or structured-content methods
- their shift from front-end content design to working with back-end engineers and architects
- how new LLM-driven applications of conventional content-evaluation criteria permit them to scale up their content design work
- their goal of creating good-quality content at scale
- how content designers are shaping the future of conversational ecosystems
- how AI lets content designers do more strategic thinking, in particular about how to apply their insights at scale
- her take on the recent rounds of tech layoffs
- one of the new roles that are emerging for which content professionals are well-suited, like the new position of model designer
- the origins of their AI program in a simple application of gen AI to partner content creation
- how to bootstrap the implementation of AI content practices in your org
- how to identify opportunities to help your customers by matching their content use cases with your AI capabilities
- her message to content designers: “don’t be afraid” and keep learning
Chelsea’s bio
Chelsea Larsson is a Sr. Director of Experience Design at Expedia Group where she leads the B2B Content Design team, partners on strategic design initiatives, and builds AI travel tools. Chelsea loves to chat about Content Design in genAI and UX design for travel. She shares her thoughts on both topics via the Smallish Book newsletter and conference stages around the world. Her favorite book to gift loved ones is the delightful Chirri and Chirra series. Her favorite sandwich is a turkey club.
Connect with Chelsea online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Content and AI podcast, episode number 27. The arrival of generative AI tools gives content professionals a whole new palette of design capabilities. Learning how to take advantage of these new opportunities so that they can shift from product-development design into content-driven AI experience design challenges many content folks. Chelsea Larsson sees these challenges as a chance for both her and her team at Expedia Group to stretch and grow and to scale their impact as design professionals.
Interview transcript
Larry:
Hi everyone. Welcome to episode number 27 of the Content and AI Podcast. I am really delighted today to welcome to the show, Chelsea Larsson. Chelsea is a senior director of experience design at Expedia Group. And welcome Chelsea, tell the folks a little bit more about what you’re up to these days.
Chelsea:
Thanks for having me, Larry. As you said, I’m a senior director of experience design. I lead the B2B content design team at Expedia Group. So we call that the partner content design team, because we work with Expedia partners. I also lead the Generative AI Experience Design Program, which we’ll get into later and lean in on a couple of strategic initiatives at Expedia.
Larry:
Cool. And I think one way we were talking before we went on the air is we were talking about the idea of these AI learning machine, and that seemed to resonate with you as a way to describe what you’re up to. Can you tell me about the machine you’re building there?
Chelsea:
Yeah, so when I first started getting into AI, which I think was around a year ago, and talking about generative AI here, of course, I saw a kind of paradigm shift in how content designers specifically could work in AI fields, and it kind of led me to create what you called the learning machine, because when you’re working with AI features, the planning is different, the designing is different, and the evaluating is different. It’s not fundamentally different, but there are new layers to consider.
Chelsea:
And those layers led to a lot of questions on, well, how do we plan for the right AI opportunities in our product roadmap? How do we design these AI interactions, questions, when do we disclose that AI is being used? How do we signify that AI technology is being used without words? So what kind of iconography do we use? And then how do we evaluate the output differently than we would evaluate the output if humans had generated the content? So, when you think about those three different pillars of work, planning, designing, evaluating, we were led to, and I spearheaded this, create a program of critiques, guidelines, leadership forums, ways of working, which kind of has created this learning machine as you called it, which I love, and I can get into that a little bit.
Larry:
Yeah, and I love that, they sound like familiar practices, but talk a little bit about how each of them manifests differently in the AI world.
Chelsea:
Yeah, so when you think about, they’re absolutely familiar processes and it’s what we’ve been doing as product development designers for a long time, but there are new considerations to take into account. So when you think about the planning, let’s start there. Your company’s not going to just put AI into the experience. AI is not a solution. It is a avenue to get to the outcome that you want as a business, but you do have new capabilities now. You have text generation, you have text classification, summarization, you have multimodal content generation. You can create photos, you can create videos, you can pull out sentiment analysis. So with these new capabilities, you can matrix those to the outcomes that you already want to have or the user problems that you have in place. And by matrixing those with the new AI capabilities, it results in a change to the roadmap.
Chelsea:
You can plan for new outcomes because of the capabilities that you have with AI. Without understanding those capabilities, that is a hard conversation to have. So that was the change that needed to happen is educating our designers, our content designers and our product folks on what these new are. And that education at Expedia has kind of come about in these forums that I spoke about where I have taken machine learning scientists, product people, and designers and kind of for the first time, put them in a shared space and critique where we are sharing with each other ideas, capabilities and user problems. And those are kind of coalescing in new road mapped opportunities. So that’s kind of a different way that we’ve started approaching planning these AI opportunities.
Larry:
Right, and I’m wondering if each of those parties you mentioned the ML folks, the design folks and the product folks, do they each bring different perceptions of those capabilities and is the mix different than it was before in those kind of relationships?
Chelsea:
Yeah, so there’s also a fourth person, a fourth role who I’ve partnered with quite a lot in the past year, which I guess I’ll call them a content architect at Expedia. They’re called content strategists, which I know is going to be super confusing for this community. They’re people who are really highly skilled in structured and unstructured content. So they’re the NLP experts of the world. They understand BERT, which is a bidirectional language assessment pattern. They understand structured content in a way that makes it really easy and valuable, to have them on your team, to let you know as a content designer, if your solution should be generative AI or if it should be structured content. And that partner brings that knowledge to the table, they let you know kind of what the content landscape is and what the best content tool is to use.
Chelsea:
I think in the future, our roles will probably become one because they also usually typically have a writing background, a taxonomy background, a library science background, but they also have kind of a data, an engineering understanding. If content designers could lean more into that content modeling, content architecture side of things, these roles would basically overlap. But right now, those are two different roles where I work and are very helpful for partnership.
Chelsea:
The machine learning scientists, they bring all of the LLM knowledge, so they’re helping us understand the base model, the behavior of the base model, what we can expect. They’re helping us fine tune the model based on our prompts, based on our system instructions, the definition of good that the content designers are creating. They also help us understand the cost of scaling out some of the proposals that we have. We have to pay for the tokenization of the outputs, so how expensive is it going to be to generate this type of content?
Chelsea:
So they’re the system and scaling experts, and we work with them really closely on the behavior and the output. We work with the content architects where I’ve talked about before on the inputs. What does this content need to be, how does this content need to be structured as an input and with the machine learning experts, how can we fine tune this output to get to the place that we want? All of these people understand the input and output, but they all have different levels of expertise where I work, I think it’s different at different companies.
Chelsea:
And then the product folks are still doing what they’ve always done, which is helping us align around a product experience vision, the fit to the market, how we should be presenting this to the industry and how it aligns with our overall product strategy. But I do find myself working more closely with the machine learning experts and the content architects even more than I do with product designers at this point, because in some cases, the UI actually already exists and it doesn’t need to be redesigned, we’re just changing the way that the content is generated in that UI. I can’t go into specifics because these are products that haven’t launched yet, but it’s just interesting to only work with content and engineering people for a product experience.
Larry:
Yeah, I do content modeling for a living, so I could nerd out on that all day, but-
Chelsea:
Yeah, I know you do. Yeah, you would definitely, you are that person I described.
Larry:
But I spend all my time with content designers and want to help them get their heads around this and the way you were just describing that whole situation and the shift sort of like, because I’m just trying to get the shift from the… It’s often been product driven and you spend a lot of time with product, just is in my experience, but you said this new shift to working so much with the content architect and modeling folks and the modelers. And I guess the thing about that is, to my mind, the structured content folks, the content architects, they have an idea, a way to make the content ready from the get-go. You said they’re the input folks, whereas the output part, if it’s a machine learning thing, it’s probably algorithmically and it’s like the whole vectorization mathematical prediction stuff going on. Long-winded way of saying how do you align these two streams in a way that sort of matches up to our usual concerns around voice and tone and style and consistency and all that stuff?
Chelsea:
Yeah, so that kind of gets to that third, so we kind of talked about how you plan differently now and then the designing differently is that different design team, the evaluation piece is that new part of the learning machine that I’m in, we evaluate the content differently based on criteria that we’ve always used as content folks, so grammar, accuracy, granularity, but with LLMs, we are able to augment our evaluation to make it 40 times faster than a human being can evaluate big bodies of text in order to scale out high quality text. And that is kind of the difference between a content designer writing copy, taking it to critique, getting human feedback, iterating on it, and publishing it. What we’re doing now is we’re training a model to create whatever the output is that we want to have. Let’s just say it’s a description of a hotel.
Chelsea:
That model is creating thousands of these. A content designer cannot go in and annotate thousands of outputs, so the content designer also needs to lean in on that evaluation algorithm, which we call the evalmetrics, and that is where the content designers at Expedia has really leaned in with those content architects and the MLs, the machine learning partners to say, here are the most important things for the model to evaluate on grammar, on granularity, on accuracy, on representativeness, on style, on voice, and we grade it on a scale of good to bad, I’m not going to go into our proprietary way of scoring.
Chelsea:
But our job is less about looking at the output now, wholesale, and it’s more about looking at how we’re measuring that output and then taking sample sizes and ensuring that those measurements are accurate and that we are outputting at the highest level possible, which is a really interesting shift for me as a content designer, because I feel like I’m teaching a machine how to write and how to evaluate content instead of me doing it myself, but it’s actually been super-interesting and fun and I’m enjoying it a lot. I hope that answered your question.
Larry:
No, that’s interesting. And that’s sort of like one level up from a thing that a lot of people have said, like Ethan Mollick and Noz Urbina and a number of people observed that when you have these agents, it’s like all of a sudden you have 10,000 interns, but you kind of have the level above intern, like entry-level writers in these machines and you’re kind of training them to get better and better all the time. Is that a correct analogy?
Chelsea:
Yeah, that’s the goal is good quality content at scale, and then where I see content designers coming in is, wow, how can we drive the strategic vision for this content. For content all over the platform? Interfaces are going to be more voice-centered in the future. People will be able to speak to products more regularly. We’re going to have more assistants. I’ll have an assistant, Larry will have an assistant. Those assistants will be talking to other assistants at different products in different product spaces. What are those conversations looking like? What is this whole conversational ecosystem where we’re using AI powered devices and algorithms and I see content designers really shaping what that looks like. We have adaptive content, content that changes based on what behavior the user is taking the experience.
Chelsea:
So something like that could be, we see that now with advertising. We see you clicking on something and then later, you’re setting a signal for that company, you’re going to get that type of content, but it could be even faster in real time. We see more personalized content coming out. We see augmented productivity tools that just help you do things faster. We already see that now, use this AI macro to write an email. So content designers can really think about what those experiences are because they’re all content based. And that’s why I’m not worried about us letting LLMs do more writing and drafting because we’re actually driving the vision for what companies can do with content in order to build better relationships with their customers. And that’s where I get really excited about the whole AI space right now.
Larry:
It does sound exciting the way you described that a minute ago, that this frees people up to do more strategic work because I think I’ve never met a content designer who was convinced they were doing everything they could in their job, because they’re doing so many detailed things. And now some of those details are both delegated to the machines, but also the machines are doing them at scale so you can strategize about more. Does that make sense?
Chelsea:
Yeah, absolutely. And it’s like you can scale out your thinking and your perspective. How many times has a content designer had a really strong perspective, but they felt like their work was so constrained to the strings that they were writing, that it didn’t enable them to make big change in the product. Now you could have a perspective for a content-led feature, and you can actually create a proof of concept that scales it out to thousands of outputs that you can test and learn from and launch a beta that was just your idea, you have to work with a product person, of course. But content is the currency right now, and it is what all of the AI work is being built around. So for a content expert, this is super exciting.
Larry:
I’m hoping you can help us circle around that because there’s so much anxiety in the workforce right now. There’s been… I was talking to somebody, I can’t remember who the other day about the conflation between layoffs and the arrival of gen AI. I think a lot of people conflate that, but I’m not convinced that they’re linked. I think it’s some kind of coincidence.
Chelsea:
No, I mean I think the layoffs are because tech companies are prioritizing shareholder value and stock buybacks and more hyper capitalism than AI. I do think that AI will create shifts in how people are employed, but I’m not sure that we’ll lose roles. They might just evolve. I’ll give you a really good example. Valerie Qi, who was the head of content design for Slack and then became the head of content design for Airtable, she’s a good friend of mine. She is now at OpenAI as a model designer. She is working in the product org. Her boss is a PM and she is building the behavior for ChatGPT, and she’s hired because she’s a content expert who is working on a content product. That is a really cool job that we could not even have envisioned a year ago. And that is a direct jump from being a content design leader to this role, which is a model designer.
Chelsea:
So I guess what I’m saying is, I don’t think we’re going to lose relevancy as content people, but our roles might change, our titles might change, and that is fine because as content designers, we have evolved every three or five years in tech anyways. We’ll still be content marketers or tech writers or information architects. So we are kind of used to evolving, and I think this is just another point of that evolution, which takes us back to the learning machine, which is how are you empowered to learn? How are you empowered to do this work in your own company and can you experiment? Can you drive experimentation within your own teams? And if you can’t, that is a frustrating place to be because I do think that content designers need to lean in immediately if they’re interested in this work.
Larry:
Well, I think I’ve got to say, I think content designers are lucky, at Expedia, are lucky, it’s amazing that you’re there consciously crafting this new approach to practice because I don’t think that exists in a lot of places. Can you give folks a feel for maybe how to bootstrap it yourself or DIY it?
Chelsea:
Yeah, I can, and it wasn’t like this before. I was really hungry for this. I brought it to my VP of product design, Greg Perez. I’ve always been fortunate to have leaders who supported me in my crazy endeavors, so I’m glad he did. But we did kind of bootstrap it at first because we went back to the discussion that I said earlier, what are the capabilities we have on hand? At that time, which was like a year ago, we were like, okay, well, we can generate text. We have ChatGPT. What are the obstacles we have for some of our users? In VRBO, our hosts have to create a headline and a description for listing a property. Okay, can we use text generation to make that easier for them? Yes, we can do that. That was the start of this whole program because that idea that I had with a product person, we brought it to the VP and SVP and said, can we just try this out?
Chelsea:
And they’re like, sure. I guess. And from that I learned a lot. I started creating relationships with MLs, with those content architects. Other projects sprung up. I pulled those into the larger narrative of, this is what we can do with AI and content, and we have been able to spin up tests across the company. How can you do that on your own? I think if you can find a partner to bootstrap it with, a product person or an ML, there’s got to be something, somebody in your company who is interested in this. Partner with them on just proof of concept, which is why these tools are so exciting, is because you can create a really quick video, photo generator, text generation. You can create really fast proof of concepts. You can take those proof of concepts with your UXR partner, to do really fast user interviews, to see if there’s any kind of desirability there.
Chelsea:
You can talk to machine learning folks about the feasibility of building something like this, and then you can pitch it to whoever can help you get it on a roadmap. That is the scrappiest way I would say, to do it. And at Expedia, there was all kinds of people doing this work across Traveler and Partner, it wasn’t just me, but I think one of the things that I did, which helped, is that I kept pushing them all into the same narrative. So if you can do that as a content designer, look across your company, look at patterns, are other folks leading into this, wrap those up into one big narrative. This is what we are doing with AI. This is how we can differentiate our brand with AI and bring that to the most receptive leader who you can get executive sponsorship from.
Chelsea:
Stories change minds, stories change hearts and minds. So if you can build a story around why AI matters and why you think that it can make the business better and you have evidence of that through your proof of concepts, it’s hard for me to imagine a leader just being like, no, I’m not interested, if you have a really strong story.
Larry:
I love that example, and especially because VRBO, that was a really well-chosen case because this is just somebody who happens to have a room or a house they’re renting and need to write, and that’s a great place to have content design wisdom ensconced in a AI gadget, that helps them write a better description. That seems perfect, and I don’t think it’s going to be hard for people in other industries to imagine their version of that.
Chelsea:
Yeah, absolutely. It’s like you want to think about as a content person, where is my user having to generate content that’s tedious? Where is my user having to scale out content that they’re unable to scale it out because of a manual process? Where could my user benefit from insights, summarizations, classification of text and content, where they’re having to look at the Amazon summaries of products. Instead of having a user go through all of those product reviews, now you have a summary. So where can I shortcut clarity for my users via content? And that is our bread and butter. So I bet you, in anyone’s company, there’s a hundred different opportunities there. It’s just who’s the most receptive person to support you in your endeavor?
Larry:
And the way you’re talking about that, it’s I’m increasingly thinking about – my podcasts have always been about practice stuff – and now I’m thinking more about mindset thing and the way you just said that, establishing this mindset of looking at this landscape in a way that permits, sort of facilitates, these kind of insights. What’s the tedious thing? Oh yeah, I don’t have to worry about tedious tasks anymore because I have all these machines to help me.
Chelsea:
Yeah.
Larry:
Yeah, have you articulated that as then – right back to practice? Is there a three point checklist or how can you kind of embody that in your day-to-day practice as a content designer, just that kind of thinking?
Chelsea:
Yeah, so that kind of goes back to that learning machine again is we are trying to, a couple of content designers at Expedia, of course, like the documentation nerds, we’re trying to document those ways of thinking. I worked with a program manager, Jason Black, shout out to Jason, on a list of steps for how you can get into that mindset of pitching a product idea. And I can’t share those because they’re kind of proprietary, but I think in your own company, it’s pretty much as simple as, I’ll go back to you, what are those obstacles our users have that are content-related? We’ll just stick with content right now. What are the AI capabilities that we have available?
Chelsea:
Which one of those pairs is a quick win, which needs more scaling with ML support and which is something that has to happen in a phase three? We’re talking about a personalized adapted interface that changes based on who’s using it. That’s a way in the future proposal, if you can just think of that use case plus capability equation, you will come up with some ideas and those ideas can lead to test and learns, and those test and learns can lead to case studies, and those case studies can lead to a product roadmap. So that is what I would suggest for people to explore with their ML partner or their content engineer or whoever is most like-minded around generative AI at their company.
Larry:
Yeah, the way you just said that, you can kind of picture if it was a scaling issue, you might spend more time talking to the ML people. But yeah, I love that you can articulate that, I kind of put you on the spot there, thank you, you crushed it. Chelsea, I can’t believe it. I could literally talk to you all day about this stuff, but we’re coming up close to time. Is there anything last you want to share before we wrap up?
Chelsea:
I just want to say, don’t be afraid. I feel like I have learned more in the past six months, than I have in the past six years. And every day I go home feeling like a full dumb dumb, and it’s a good feeling because it means I’m learning a lot. So just be hungry, be curious. Don’t be afraid to ask your ML partners a million questions. There’s a whole new list of terminology to learn, RAG, LLM, ALM, you’re going to get all into that, but don’t be afraid because the more you know, knowledge is power. So just go out there and get it.
Larry:
Fantastic. Hey, one very last thing, Chelsea. If folks want to stay in touch, what’s the best way to follow you or connect online?
Chelsea:
Yeah, I do have a newsletter called Smallish Book that is on Substack. I write with my friend V, that is in a pause right now, but it’s coming back in a couple of months. And then you can always connect with me on LinkedIn.
Larry:
Fantastic. Yeah, I was wondering where Smallish Book was. I’ll keep my eyes peeled for the revival.
Chelsea:
We had to take of small hiatus over the holidays, but we’re coming back.
Larry:
That happens all the time. Yeah. Thanks. Thank you so much, Chelsea. Always fun to catch up. I really enjoyed the conversation.
Chelsea:
Yeah, thank you for having me. I love your podcast and I hope this is helpful to some folks.
