Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Android | TuneIn | RSS
In her work as a content designer at Indeed.com, Rebecca Nguyen is finding new opportunities to assume a leadership role on teams working with generative AI.
Rebecca feels fortunate to work with teams that recognize the value of writing and design skills. She’s also finding that generative AI is the perfect place for content design to take the lead.
We talked about:
- her work as a senior UX content designer at Indeed and her recent shift to focus on product teams using generative AI
- how well-suited content designers are to AI products
- the unique challenges of working with non-deterministic large language models
- their process for designing prompts and how they evaluate them
- her learning curve around the loss of some language control that you get in conventional content design
- the main differences between prompt engineering (the how) and content design (the what)
- her ability as a content designer to lead more in the AI space than in prior design roles
- how they balance the use of outsourced LLM solutions like OpenAI versus developing their own models
- the lack of genuine intelligence in LLMs
- how her fear and concern about AI is eased the more she works in the LLM world
- how the evaluation component of designing content for AI creates more work for content folks
- one of the main benefits of LLMs – their ability to take on tedious rote content work
- the child-like nature of LLMs
- the surprising liberating effects of simply not worrying about whether or not you have a seat at the proverbial table
Rebecca’s bio
Rebecca Nguyen (she/her/hers) is a Senior UX Content Designer at Indeed. She’s been part of marketing, UX, and product design teams at Bankrate, Northwestern Mutual, and LPL Financial, where she established the content strategy practice. A Confab speaker and workshop instructor, Rebecca is also an award-winning memoirist.
Connect with Rebecca online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Content and AI podcast, episode number 15. Just as content design was emerging as its own craft and profession, along came generative AI. At first it looked like ChatGPT and large language models might displace content designers (unfortunately, it appears from recent layoffs that some executives may still think this is the case), but at Indeed.com, Rebecca Nguyen has found that working with LLMs has given her more work, not less, and that her content design efforts are now more interesting, rewarding, and impactful.
Interview transcript
Larry:
Hi everyone. Welcome to episode number 15 of the Content + AI podcast. I’m really happy today to welcome to the show Rebecca Nguygen. Rebecca is a senior UX content designer at Indeed. Welcome, Rebecca. Tell the folks a little bit more about what you do at Indeed.
Rebecca:
Hey, thank you so much, Larry. Great to be here. Yeah, I’m a senior UX content designer at Indeed. I’ve been there for a couple of years now, going on two years, and I work on product teams to make sure their content is useful and useful and accessible and inclusive and all those goodies that we’re used to. And in the past six months or so, my role has really shifted and I’ve been almost exclusively focused on working with product teams who are using generative AI in their products.
Larry:
And that’s why I wanted to have you on the show is we talked about this a while back. And that’s one way to think… One way I think about that is all of a sudden we have new collaborators in two senses. One, we have these new, we’re talking to machines in our work because they’re generating some of the language we work with, but there’s also a lot of other new collaborators. Tell me a little bit about how the people around you have changed over the last six months.
Rebecca:
Yeah, that’s such a great point. So we’re probably, if we’re working in product content, we’re used to working with product managers, we’re used to working with UX designers, engineers. And that has shifted in that the team that I am partnering with now is made up of engineers and product managers, but we’re also working really, really closely with data scientists and we do not have a UX designer or UX researcher on the team right now. So UX content design is really the entire voice of UX in this group, which is really cool.
Larry:
That’s really interesting because often we’re the last one in. How does that feel going in there as a sole UX person?
Rebecca:
It’s exciting. It’s been a little bit intimidating, but I haven’t found myself feeling completely lost or anything. I think it’s been great. As we were chatting earlier and you said we’re really… We’re creating a content product when we’re working with these language models. The output is text and language, and so who better could be suited to drive and design the language when working with one of these models? It’s been a really natural fit. And then the activities and tasks and approach has been different from anything I’ve done before, but it’s well-suited to a content designer skills, I would say.
Larry:
Well, that’s it. So what has that transition been like? You said the activities and the tasks differ. It sounds like it kind of rhymes with your old conventional product work, but how is it different now?
Rebecca:
Like that. Yeah, I sort of talk about it as if we think of a sandwich and in that content creation moment, that’s the meat. That’s sort of like when we’re going through a design thinking process, we’re doing some discovery or research or we’re deciding on the problem that we want to solve, and then we get to that moment where we make the thing, we design the thing and we might be writing words. And after that we are iterating and getting feedback and seeing how it performs and measuring and iterating more, et cetera.
Rebecca:
The difference for me with generative AI has been spreading my focus out and becoming more of the bread. So instead of the meat, that creation moment, when you’re working with a language model, the model takes on that task. They’re the ones creating the content. And your focus as a human is all of that stuff on the periphery of that, so the prepping, which we would be sort of the prompt engineering and design where we’re telling the model what we want it to do, and then the evaluation piece where we’re looking at what the model did and saying, “Okay, was it successful? Did it follow directions? Could we do it better?”
Rebecca:
So it’s almost like you become a teacher of content design instead of a content designer where you’re actually making the thing yourself.
Larry:
Interesting. I have not heard it articulated that way, but that makes perfect sense because… Well, they’re called learning models and you’re the teacher. That’s great. And you mentioned both prompts and one of the things you just said made me think that people always talk about prompt engineering, and you talked about engineering and designing prompts. Do you go into prompt creation with a designer hat on because you’re working with engineers? Do you think more as a designer in that world?
Rebecca:
Yeah, I definitely think so. Particularly as a content designer, thinking about how does the language inside the prompt impact the output and to make sure that content design considerations are represented in the prompt as much as possible to make sure that we’re getting the output where we want it to be. We’re sort of preemptively correcting mistakes or anticipating mistakes that could happen.
Rebecca:
For example, when you get familiar with a model like ChatGPT, you can see, and we all can see as content designers sort of that out-of-the-box tone that the model assumes, the model that’s been trained on the internet. So it’s a very casual tone. It is, in my opinion, it’s overly friendly in a way that can be kind of annoying. There’s lots of exclamation points, there’s a lot of celebration for small things that may not require such celebration. It helped you with a task and it’s like, “You’re so welcome. Awesome.” And you’re like, “Calm down.”
Rebecca:
That tone and that voice isn’t always appropriate for a product. And so when you’re getting in there and designing prompts, you have this opportunity to modify as best you can. And the cool thing about prompt engineering is that you can do a lot of playing around and you can see how different instructions impact the outputs and then tweak and adjust from there. But that was surprising to me because I think that on my team and at my organization, at first we were sort of thinking about this on the other end, once we see the output, then let’s evaluate it and give feedback. But the problem is that once the output has happened, it’s too late. It’s not like working with a human where you can revise it and create this static thing. It’s always going to be different every time. It’s that non-deterministic nature of a large language model. And so really we want to get in at the prompt stage to try and drive and direct before the output happens.
Larry:
That’s so interesting. But you’re still getting some feedback from it, too. You mentioned earlier how one of the pieces of bread is about iteration in your sandwich. And then, as you’re talking there, I’m also reminded back when you said that you don’t have UX researchers on the team. Are there more automated ways of getting feedback? Like for you, because you’re still looking at it after the fact to see compliance with… Not compliance, but sort of alignment with voice and tone and that kind of thing. I guess tell me a little bit about that loop.
Rebecca:
Yes. So we have content design at the beginning, which would be the… Actually, even before we do prompt design, a content designer can create a mock in Figma or whatever you’re designing as this north star. So if the model did everything we wanted it to do, what would that look like? What would be this perfect piece of content, for example?
Rebecca:
Then we go in with the prompt design and make sure that we are anticipating content issues around voice, tone, style, toxicity in the content and addressing those in the prompt where it makes sense. Then after we get an output, the content designer can create heuristics, like measures of quality by which we would evaluate how successful that output was. So we have to define what is the voice that we’re using or what is the tone that we’re using or what kind of readability score are we looking for, all those regular things that a content designer would look for, and how successful were we? And if we were not successful, we need a system and a process in order to quantify that and measure it and then have a plan for what do we do.
Rebecca:
In my case, it’s been difficult because in some experiments that we’ve run, sort of a baseline level of, okay, we’re not doing great with style or tone of voice, we’re not doing awesome with clarity, but it’s not toxic, so let’s ship it. That’s a tough pill to swallow if your whole job is around elevating content and increasing quality. But I’ve had to really recalibrate my brain around this and say, when we’re working with these models, we have to start somewhere and we can’t start at a hundred. It’s not going to work. We have to learn and fail and go and say, this is our bare minimum, and then get better and better from there.
Larry:
That’s so interesting. It’s hard, I’m trying to picture an analogous thing in conventional content design, and it probably wouldn’t happen just… That’s good enough. It’s not horrible, so ship it, and it’s not toxic, so ship it. That’s really interesting. But it sounds like you don’t seem put off by that. It seems like that’s just part of the whole learning curve.
Rebecca:
I was at first, don’t get me wrong. This has all been, it’s been a huge learning curve for me and mostly around just, it’s a very different way of thinking because you have even less control than you might might normally when working with your PMs, but you find these other pockets where you can really drive instead.
Rebecca:
And then you also can make sure that you’re not satisfied, because the risk of that is, well, it’s not toxic, ship it and then say, maybe it’s performing better than what was there before. So then the risk is, oh God, are we going to just continue to ship this crappy content that happens to not be toxic or making sure that you have a plan in place and a process in place to continuously improving and get better. And we’re doing that now.
Rebecca:
So we had to go at it first with, “Eh, it’s okay, it’s not toxic, that’s good.” Now we can do some fine-tuning, and that’s where a content designer is in partnership with a data scientist, and we’re fine-tuning the model and continuously revising and taking these outputs and rewriting them and feeding them back into the model to continuously improve the output. So that’s been a really exciting and powerful area for content design to be involved and really get a little bit more of that control over the output.
Larry:
Yeah, because that seems like if you can control it algorithmically and with all the things that we bring to the game. I’m also curious, as you said that I was thinking about these new collaborators we have. It sounds like, are you collaborating more with engineers when you’re doing prompt stuff and more with data scientists when you’re doing fine-tuning stuff? Is that sort of how things break down in that world?
Rebecca:
Yeah, I think so on a high level and also the product manager at the prompt design stage as well. And it’ll depend too on who has that skillset, who really understands these models the best. And it could be an engineer, it could be a product manager. I think it could be anyone, but it’s who has the best grasp so far on how these things work.
Rebecca:
And I do notice, too, that with prompt engineering, that is a stage where folks tend to maybe not think about content design because they’re thinking so much about what do I want the model to do? And a content designer, the value we add is, but how do you want to do it? So again, voice, tone, style. So because we’re dealing with language, that is absolutely crucial and we can’t not address it at all. If we don’t address it at all, we default to whatever the model has been trained on, which might not be appropriate for our use.
Larry:
Right. And it’s interesting because we have the skills to get that in there. But I guess, let me ask you about this because it occurred to me a few minutes ago as you were talking that we’re all learning this stuff as we go, but I wonder if the data scientists and the software engineers and the machine learning folks, because they’re probably a couple of years ahead of us in this or they’re more likely to have had more exposure to it, do you feel like some of this is content people playing catch up, or do you feel like it’s a pretty even playing field, that we’re all learning new things all the time?
Rebecca:
That’s a great question, and I was actually, I might just be really fortunate at Indeed and the team that I’ve been on, but it has really felt like everyone learning together, even the data scientists who specialize in this stuff. And maybe part of the reason is because the models themselves are evolving all the time. OpenAI released ChatGPT, then they’ve released GPT-4, and they’re constantly improving and changing. And then there’s this whole world where building what you need to know to build your own model as opposed to leverage someone else’s model.
Rebecca:
And so I came in knowing absolutely nothing, and I’ve never felt, I mean, don’t get me wrong, people talk about stuff and it goes over my head every day, but I have never felt this sense of, “Oh, you’re just a content designer. You wouldn’t know.” The people that I’ve been working with are very much, they’re hungry for, “Tell us, is this how you would write it? Is this good or bad? Why?” And they’re very helpful in, “Okay, okay content designer, if that’s where you think we need to go here, we’re going to go over here and tweak and pull levers to get us there instead.”
Rebecca:
So it’s been really refreshing, actually. I feel like I’ve had more input and been able to lead more in this space than I have in previous interactions with a typical design process with a product manager.
Larry:
That’s interesting. Now I’m picturing a clean slate where we’re all coming in and scribbling on the blackboard together. And yeah, that’s a little more field leveling than these old… Because I’ve certainly been in situations where the product and engineering people rule the roost. So that’s super interesting.
Larry:
Hey, I want to go back just a second, too. You mentioned a minute ago, and we promised, we talked before we went on the air about how you’re not super technical and don’t want to go too deep in the stack, so just stop me if this is getting into that. But when you talked about there’s both of training of the models and the data sets that you’re drawing on, is Indeed using OpenAI’s access to their APIs? Or are you developing your own models or combination or…?
Rebecca:
Yes, I think I can talk about… Yeah, so we’re using OpenAI. And it’s interesting because when you’re using OpenAI, there’s an associated cost. And so that’s been another layer of complexity in thinking of, does it make sense for us as a business to use this or to build our own, and also building our own model in-house. But again, that’s time-consuming and very complex and takes time. So there’s a lot of really smart people who I think are working on the checks and balances of cost versus time versus outsourcing versus doing it in-house. Yeah, that’s a whole world, for sure.
Larry:
And we’re not going to figure it out and they’re not even going to figure it out anytime soon. I was just really curious about that though, because I’ve been just hearing more and more lately from people I know in the AI community about, for example, my friend Paco Nathan, who was on the second or third episode of this podcast, he went to pretty much every AI conference in 2023, and one of his upshots of that was the increasing prevalence of one, people using their own data and Indeed must have tons of great data. So I’m assuming you have plenty of that to work with. But one of the other four things he observed was the increasing prevalence of smaller models and more specialized models. And I don’t know, but again, that’s kind of deeper in the stack than we were going.
Larry:
But another thing I wanted to come back to is when we very first started talking about this, you talked about that there was initially at Indeed, and I think just in the whole field in general, there was some almost fear about non-content people doing all this stuff with content. And it sounds like from what you said so far, it sounds like you’re making good headway on getting into these teams, but tell me a little bit about that because I think a lot of people are probably going to be not as far along this path as you, and if there’s anything you can do to assuage their fear going forward.
Rebecca:
Yeah, I was definitely in that camp where I felt like, oh, great, they already don’t know what we do and now they see this ChatGPT and that can generate text. And if they think all a content designer does is type, they’re like, “Well, the robot can do it faster, so why do we need to work with those people?” And when I think about it now, a couple of things.
Rebecca:
So, first of all, I think that these models are not intelligent. It’s not as sentient intelligence. The more you dig in, the more you realize it’s simply words are weighted by algorithm of how likely is one word to appear after another. It is not intelligent. It comes across that way. It comes across as if it has a personality. But when you dig in and start to understand how they work, you realize very quickly like, “Oh, this is not competition for a content designer.” We’re a long, long way away from that.
Rebecca:
And the other thing is, I think if people already don’t understand what you do or the value of what you bring, it’s sort of irrelevant whether they’re going to not understand and not include you because of generative AI or because of some other reason, they’re not going to include you anyway. It’s not like generative AI is going to increase the amount of people, not including you. I don’t think that’s true. I think the folks who value content design, they understand the importance of bringing us in.
Rebecca:
I work with really smart people and they don’t always know how to solve content problems, but they can see them. This reads funny, this sounds weird, oh, that seems strange. And so I think there’s so much opportunity for us, and I completely understand the fear, but the more that I’ve learned and the more that I’ve done this work, the less afraid I’ve become because it’s been so obvious that there’s such a big need for content designers in this work, and we can add so much value. In my case, it’s given me a lot more freedom, and I feel more when I’m doing this work that I get to do my job. I get to come to work and actually do my job instead of fighting to try to get to do my job.
Larry:
Nice. You’re reminding me, I don’t know if… I do another podcast on content strategy insights, it’s a much longer running one, and I just dropped an episode recently with Relly Annett-Baker at Google, and she was talking about kind of to what you were just saying, that the writing is such a small part, she said in that interview that, “Yeah, that’s 20% of what we do. The 80% are figuring out what the heck are you going to write?” And it sounds like there’s an analogous thing here. The whole process is different. You’re not motivating and aligning human writers and designers and engineers on how to do something. It’s a whole different set of collaborators.
Larry:
But it seems like, I guess this isn’t mathematically accurate, I’m sure, but that general idea of 80-20 stakeholder alignment discovery where all the other stuff that comes before and then 20% of it actually creating, is there something analogous percentage-wise emerging in the gen-AI content world?
Rebecca:
I think so. I love everything you just said, and I think that it’s sort of this idea of all of that discovery and prep work and what should it be? What should that output be? What problem are we solving? And then the evaluation piece. The evaluation piece has been this just complicated beast that I never could have anticipated, and it creates a ton of work for content designers. Everything from what are the quality measures, the heuristics by which we’re going to measure if this output is successful or not, developing those. And sometimes those have to change for every use case. If the output changes, the way we measure if it’s, quote, “good or not” will change too. So we’re constantly developing these new heuristics.
Rebecca:
There’s a lot of manual review, human review of the outputs themselves and providing feedback. And then, from there, we’re creating processes where we do internal and external and machine review of the same text, and then we compare to make sure that we are all measuring by these same measures of quality. And then from that, once we have that going back to the beginning and saying, “Okay, now that we know what we know, what needs to change in the prompt?” It’s a different cycle, for sure.
Larry:
Yeah, that’s so interesting. The way you just said that too, that internal external, and then machine review, because I guess the point of that latter too of as the machines get better at this, there’s going to be more and more that we can hand to them, and then that 80-20 rule might expand that it’s 90-10, that they’re able to do more and more stuff, and we have 10% more brainpower to devote to strategy and things like that. I guess maybe, do you think I’m being Pollyanna optimistic or do you think that’s likely?
Rebecca:
No, I don’t think so. Because what I’ve realized too in working with the models is that I think the biggest fear is they’re going to take away the thing I like to do. Let’s say the thing I like to do is write. They’re going to take away stuff you don’t like to do, which is awesome. An example would be, let’s say you have a really, like a thousand word job description, and you want to know if this resume, a job seeker has a resume, and is there enough relevant experience on this resume to make the job seeker qualified?
Rebecca:
Now, are you going to go through word by word and you’re reading, reading, reading a thousand words, and let’s say you have a hundred of these to do, it’s going to take you months. The model is faster than us at that. And I don’t know about you, but I don’t want to sit there and read document after document after document. That’s not fun for me. So it’s also faster and better at stuff that I don’t want to do. It can synthesize, it can summarize.
Rebecca:
I think writing is the thing it’s worst at, actually. It’s not a great writer, especially now when we think about what is good UX writing or good product content. It does not excel. It’s very wordy. There’s lots of adjectives and filler. It fails. But it’s really good at other stuff that can save us time, so then we can focus on the quality of the output, making that better.
Larry:
That’s really interesting. Now, we just have to figure out how to work with these new, they’re kind of a savant in one way, and we’re just like, yeah, but I really need… It’s just like figuring how to work with this new colleague who’s from not just another country, maybe another planet or something, and we have to figure out how to get them on board. Yeah, that was-
Rebecca:
Oh, sorry. I just wanted to add that the product manager I work with the most told me to think about the model as a child, a fourth grader. And I love the analogy because it’s like, okay, can you get a fourth grader to follow directions? Yeah. Are they going to follow directions all the time? No. Sometimes they’re going to just go off and do what they want to do, and they don’t care what you said or they’re going to forget, or it’s going to take a lot of different times for them to learn and for it to stick. And so there’s some unpredictability there. And I thought that child analogy, you feel like you’re a teacher and they’re the student. The model is the student.
Larry:
That’s so interesting. As you’re saying that, for some reason I’m thinking of is it just a matter of adding decimal places to the vectorization to age those models to where you’re talking to a fifth grader, then a sixth grader? I don’t know. It’s fun to think about.
Have you seen the models and the algorithm, have you seen things get better? It’s only been six months, so maybe not a huge opportunity, but have you seen improvements in the kind of results you’re getting over the last six months?
Rebecca:
Yeah, for sure. Right off the gate, if we’re looking at those different, like a GPT, the 3.5 to the 4, it’s like night and day. It’s more expensive, so we can’t always use the better model, but it is kind of amazing how these different improvements they’re releasing really do make it better. But the fine-tuning seems to really help now.
Rebecca:
So I’m working with data science and we can go through and look at these outputs. And as a content designer, there’s so much that can be done to be sort of fed back into the model to improve the outputs and get it closer to what we want. Anything from writing out detailed explanations of why the content is bad, which is so fun, and that sort of trains and teaches the model, this is what bad looks like. And then you can ask it to say, “Okay, was that bad or good? And why?” And it can learn. It’s not really learning, but it will spit out something that’s closer to the way that you’re thinking. So defining bad and okay and good, what does that look like and why? And then actually rewriting a bad output to make it good and then feeding that all back in has improved the outputs for sure.
Larry:
It’s like a new kind of crit. That’s so interesting that that’s how it works. Hey, I can’t believe it, Rebecca, we’re already coming up close to time. These conversations always go way too fast. But before we wrap up, is there anything last, anything you’d like to revisit from the conversation or you just want to make sure we talk about before we wrap up?
Rebecca:
Yeah, I think I’ll just share a quick little anecdote of a friend of mine, a wonderful content strategist and content designer who works at Intuit, who told me that the idea that this whole mental model around content design, not having a seat at the table and being left out, et cetera, I was complaining about that, as we all do. And she had been doing this for 20 years and said, “Rebecca, just drop it.” She said, “Just let it go.” And no one had ever said that to me. Everyone had always said, “Oh, me too, and I’m not included either. And it’s so frustrating.” But she was the first person who said, “What would happen if you just drop it?”
Rebecca:
And as soon as I did, it was a little bit miraculous that I just let go of that idea, and my expectation was that I would be included. My expectation was that I was a partner, and so I behaved that way. And if I wasn’t in a meeting, I said, “Oh, I think you forgot to include me. See you there,” without any animosity or pouting. And so with this stuff too, I think it’s the same story, our fear of not being included or not being valued, not being understood, and we can drop that and lean in and learn as much as we can about it.
Rebecca:
The more we learn, the less afraid we’ll feel. And the more we learn, the more we’ll be able to speak intelligently to our partners and lead this space. It’s the perfect, perfect space for our content designers to take the lead.
Larry:
I love that. That might be the best thing that’s ever come up in any of these podcasts. Just, “Hey, just let it go.” And then all of a sudden you’re empowered because you’re writing that letter and saying, “Hey, oh, you forgot to include me. That’s funny.” Rather than… Yeah, that’s just great.
Larry:
Well, thank you so much, Rebecca. Oh, hey, one very last thing. If folks want to connect with you online or follow you, what’s the best way for them to find you on the interwebs?
Rebecca:
Yeah, at RebeccaAnneWrites, W-R-I-T-E-S. That’s on all the socials. You should be able to find me.
Larry:
Excellent. I’ll put that in the show notes as well. Thank you so much, Rebecca. Really enjoyed the conversation.
Rebecca:
Me too. Thanks for having me.
