Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Google Podcasts | Spotify | Android | TuneIn | RSS
The arrival of AI affects every area and aspect of content practice.
In the technical documentation field, Sarah O’Keefe sees three immediate impacts on the work she does for her clients:
- how AI agents can support technical documentation workflows,
- the ability to create content with generative AI, and
- the ways that AI is changing the delivery of technical content
And wherever she looks in the content and AI landscape, she sees the need for governance guardrails and strategic thinking.
We talked about:
- her work at Scriptorium, which focuses on scalable, efficient technical documentation
- her take on the current impact of AI on technical content
- the unique concerns about generative AI that arise in the technical communication world
- how chat-based user interfaces will change the delivery of technical content
- how users will always hack systems to use them as they wish
- the looming role of trust and reputation as important factors in online interactions
- how techniques like RAG (Retrieval Augmented Generation) can help LLM-based applications deliver better results
- the importance of thinking about the content life cycle as you assimilate and integrate AI into your practices and workflows
- a very simple AI-risk-analysis heuristic
- open questions – many of them complex and non-obvious – around copyright issues in the AI world
Sarah’s bio
CEO Sarah O’Keefe founded Scriptorium Publishing to work at the intersection of content, technology, and publishing.
Today, she leads an organization known for expertise in solving business-critical content problems with a special focus on product and technical content.
Sarah identifies and assesses new trends and their effects on the industry. Her analysis is widely followed on Scriptorium’s blog and in other publications. As an experienced public speaker, she is in demand at conferences worldwide.
In 2016, MindTouch named her as an “unparalleled” content strategy influencer.
Sarah holds a BA from Duke University and is bilingual in English and German.
Connect with Sarah online
- info at scriptorium dot com
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Content and AI podcast, episode number 4. The arrival of generative AI, large language models, and other AI technologies obviously affects us all. In the world of technical documentation, Sarah O’Keefe sees three immediate impacts on the work she does for her clients: how AI agents can support technical documentation workflows, the ability to create content with generative AI, and the ways that AI is changing the delivery of technical content – and across them all, the need for guardrails and strategic thinking.
Interview transcript
Larry:
Hi, everyone. Welcome to episode number four of the Content + AI podcast. I’m really happy today to welcome to the show Sarah O’Keefe. Sarah is the CEO and founder at Scriptorium, which is a company that does technical communication and documentation stuff. Sarah, tell the folks a little bit more about your work there at Scriptorium.
Sarah:
We’re interested in the question of how do you apply systems and technology to what we call enabling content, which is technical product learning, knowledge base, all of the things that are content that enables you having purchased a product or service to actually successfully use that product or service. And we do a ton of work around content management systems, translation management systems, and basically helping companies scale their content operations into something that works. Right? Because typically, somebody shows up on our doorstep and says, “Well, we’re doing it this way and we’ve been doing it this way forever, but this way isn’t working anymore. We acquired a couple of companies. We got a lot bigger. We’re doing more and more localization and we’re just drowning in content, drowning in inefficient content processes. Please help us.” That’s our typical problem set.
Larry:
You’re the cavalry riding in to save the day. That’s great. But I think like anybody in the content world these days, there’s this new kid in town, the AI, especially the large language models that are kind of… They seem to be finding their way into every corner of the content world, and the reason I wanted to talk to you is you’re probably the first person I thought of when I thought of technical content. What’s going on with AI in the technical world? So, a pretty broad question, but we talked a little bit before we went on the air, but I think just the context for how AI is affecting your work.
Sarah:
First, I’ll say that, I mean, people are trying all sorts of things and experimenting and working through what does this look like and what are we doing with it and how are we going to make it work for us? Fundamentally, I think this is going to end up being a pretty straightforward tool, and I compare it usually to a spellchecker. Right? I mean, it would not occur to anybody in this day to write content without using a spellchecker. It’s just part of the groundwater. It’s part of the fundamental tool set, that bag of tools that you’re sitting around with when you actually go to create content. So, in many ways AI is going to be that spellchecker.
Sarah:
It’s going to be, “Hey, can you write an abstract for me? I wrote the article, but I was told to do a summary and I don’t feel like doing it, so can you write it for me? Show me all the places where I’ve used jargon in this article that I should not have,” this sort of supporting tool set that identifies patterns, good or bad, and then helps me work through cleaning up those patterns. Or, “Hey, does this follow the same pattern as that other article I wrote? Are they consistent? Show me that.” You had a great example about, “Can you rewrite this in the style of Dr. Seuss,” which sounds super fun and possibly not totally productive, but I’m here for it. So, those types of things. There’s this idea that the AI world in general can give you the ability to take your tools or to take a bunch of tools and apply them to what you’re trying to do and make it better, faster, cheaper.
Sarah:
Now, separately from that tool set, we’ve got generative AI, and I think that the technical content world is looking at GenAI a little bit differently than most of the rest of the universe because, especially if you’re doing content that is compliance related, so in other words, there’s a regulatory body somewhere looking at it, and/or you have products where there are health and safety implications. So, I mean, the obvious example of this is medical devices. It is important to me and to you and to all of our friends and family that the instructions for how to use a medical device are in fact correct because some very, very bad things could happen if we generate a bunch of instructions and they are wrong, and in the worst case, it could injure or maim or kill somebody, and that seems bad.
Sarah:
Now, we’re not talking here about AI with bad intent. We’re just saying that if you plug a bunch of stuff into ChatGPT and say, “Generate instructions for using this product,” there’s a pretty good chance that it’s going to generate some nonsense. And nonsense is okay for certain kinds of scenarios, but it is not okay when I’m trying to figure out how to configure a pacemaker. I mean, that’s just bad.
Larry:
Yeah. Although, at the same, I do want to say one thing right there. We were talking before we went on the air that one of the things, it’s not like you don’t rewrite it into solid Dr. Seuss, but you can sort of tailor communication. One of the beauties of the… You mostly work with DITA, this very structured content, and by having the core knowledge and information that you’re working with there, you can do stuff with it. One of the examples you gave was this notion of a customer, an end user looking at this documentation going like, “Well, that’s not quite how I understand or how I learn. Can you give it to me a different way,” sort of the way we often will prompt to ChatGPT or something like that to rephrase things. Is that an existing use case now or something that’s coming soon in the tech comms world or…
Sarah:
That’s kind of the third use case because we’ve got the AI tool support use case. We’ve got the help me generate content use case, which is again, terrifying, but provided that you have enough guardrails around it and you have people checking what it’s generating could be useful. But the third use case for particularly chatbots and generative AI is this universe of not authoring but rather delivery. So, as a consumer of the content, I might look at it and say, “Oh, I don’t understand this content. It’s too complicated,” and I would ask ChatGPT to rewrite it for me now, into say, a seventh grade level or into simpler English, or I could tell it use simplified technical English only, and explain this thing to me. So, there’s a whole bunch of stuff I could do there.
Sarah:
Now if I’m the producer of the information, the owner of the product who’s producing all this tech comm content, then it’s pretty likely that what I would actually prefer is for that content or that process to take place on my website within my guardrails and sort of my controls. So, I deliver all this content. I put it in a data store of some sort, and then I wrap a generative AI, a chatbot over the top of that, and I point it at only my approved content. But of course, what’s pretty likely to happen is that your end user is going to access a webpage somewhere that you and I carefully produced and vetted and approved and et cetera, and they’re just going to copy and paste it into ChatGPT, and tell it to rewrite it. Right. You really can’t stop that.
Sarah:
So, I do think that not so much the large learning models per se, but the idea that we can use a chat engine as an information access point is going to be an interesting one. So, downstream the end consumer instead of resorting to search is resorting to, I’m going to use this chat interface because this feels like a more conversational way of handling it. And ideally, in a scenario like that, we really want them to rely not on the internet, the large learning model, the enormous amount of content that’s sitting inside that chat interface, but rather on the vetted, approved, and probably restricted content for a particular topic. In other words, if I make a particular set of products, I want you to rely on that database not on the generalized generic big AI stuff. I want you to process that smaller, more targeted set of information to look for answers. Now, I don’t always get what I want.
Larry:
No. But as you say that, the thing I’m really reminded of there is that the ability of human beings to just dismantle whatever your grandest plans were and do things their own way, and you kindly refer to as different access points like they’ll use a search engine result to navigate directly to a page and a website. That was the old-school way, but as you describe that now, it’s like that idea of cutting and pasting a page and running it through ChatGPT to get what they want. In my world, the next generation of technical content services do that because in one sense, the other thing I was thinking of as you said that we’re really familiar with this notion of variants. The earliest quickest one being localization, translating content into another language, it seems analogous to that. So, do you think that’s something that would be fairly quick to accommodate like say, “Oh, we see you cutting and pasting and this into ChatGPT. Let me do that for you”?
Sarah:
Well, I would like to say yes, but I think the answer is no, because I mean, think about this for a second. If you go to any website in the universe and they have a search button on them and you search… I did this morning. I was searching for something on a particular website and I couldn’t find it. Well, okay, I couldn’t find it in my first two searches, and at that point I said, “Okay, I don’t like you anymore.” And I went over to Google and I typed in the same exact search with a site: example.com.
Sarah:
So, basically, go search that site over there, the one I’ve been searching where I used their search button and it didn’t give me the results I wanted. I shove it over into Google and do the same thing and tell it to target that page or that website, and it gave me the right result immediately. So, I think the answer is that unless your search, or in this case, your chat engine interface is at least as good as or better than what’s out there in the world, you’re not going to have any luck telling people not to use the public stuff.
Larry:
As you’re saying that, another thing that occurs to me that I’m thinking out loud here, but just in the case of your content, the kind of content you work with, it’s not unreasonable to assume that people have justified confidence and that the answer they’re looking for is there. That seems to change that whole wayfinding exploration. Does that make sense?
Sarah:
Yeah. I mean, it’s an interesting point because when we say you have to get the content right and ChatGPT makes up stuff, which it 100% does, okay, well that’s true, but we are starting from the point of view of your technical content is good and accurate and up to date and a couple of other things, and of course, that’s not always the case. So, when we compare the results that we get out of a large learning model generated thing to what you get out of your, we’ll call it, traditional search of the tech comm corpus that a particular company has created, in order to make that a fair comparison, you first have to ask the question of how good is that source content anyway? I mean, we’ve all experienced terrible out of date, not accurate, didn’t answer the question at hand technical content, and I don’t know about you, but I’ve many times resorted to weird third party sites that wrote unauthorized content that was better than what the company, the official product owner was putting out.
Sarah:
Infamously, I was doing some research on this, and I was looking at a company that made tractors and other kinds of farm equipment, and I was looking at their website and I mean, this is a big company, many billions of dollars. Okay. Well, they had a bunch of stuff on their site, but it was all kind of bad, oh, and it was locked up. You had to log in. You had to have a dealer ID or you had to… I don’t know. There was a firewall or not a paywall but you had to log in. You had to have an ID, and you had to be credentialed. But then if you kept searching, what you would find pretty quickly was this dude in Warsaw, North Carolina, which is not a large town, who was making his own videos on how to do maintenance on these tractors, just because, and his stuff was performing a lot better in search than the official corporate site was.
Sarah:
Now, I don’t have a good sense of how accurate the tractor maintenance stuff was because I’m not an expert on that or really anything whatsoever mechanical. I’m not picking on tractors. But just from looking at this YouTube channel and looking at the kinds of comments he was getting, which were like, “Oh, thank you so much for doing this video. I couldn’t figure out how to replace the gimzelflapper, and you helped me and this was great, and thank you, thank you.” There was a lot of that. So, there was a gap in the market, and this person ended up in that gap because the content that was there wasn’t very good. So, when we turn up our noses at generative AI and say, “Well, it’s going to make up stuff,” I mean, it’s absolutely going to make up stuff, but is it actually worse than what’s being produced that’s maybe not very good already? That’s a question that keeps me awake.
Larry:
Well, that points to another interesting thing that comes up a lot in the world of generative AI and chatbots in particular, and just conversational interfaces, that notion of trustworthiness. And this tractor aficionado in North Carolina with his iPhone tripod videos of how to fix the tractor, that probably has some kind of authenticity, almost regardless of the exact quality, but it probably is actually helping people fix tractors better. But does that make sense?
Sarah:
Mm-hmm.
Larry:
And I’m wondering how kind of stodgy publishing people like us can make our content feel as engaging and trustworthy as a guy like that.
Sarah:
I think that one of the key factors in this AI world that we’re apparently moving into is going to be exactly that, trust and reputation. “Oh, we know that Larry generates good stuff. I read his last article and it was interesting, so I’ll read the next one.” As opposed to when you search now, you get just a lot of mush. I mean, we already know that fake reviews, fake product reviews are a huge problem, so it’s gotten to the point where when you go to certain large e-commerce sites and you read the reviews, you just count them, right? Because you know that well, they’re probably AI generated. Half of them say, generated by ChatGPT, at the bottom because the people who generated them didn’t even bother to strip off the identifiers. But we’ve lost trust in those reviews, so now the reviews don’t have any value.
Sarah:
I think that going forward, what’s probably going to happen is that you and I and everybody else will need the reputation for producing good content, for doing good work for whatever that thing is, so that people can cut through the AI-generated optimized stuff, the deepfakes, the synthetic audio, all the rest of it, and get to a point of this is a real person and they produce real stuff, and I am going to work with them.
Larry:
Yeah. You’re reminding me of something we talked about earlier in a more general context, but I kind of want to apply it to this, is the role of metadata in this and this nicely structured content that you’re working with. And it does have authority behind it because you made the product and you’re telling people how it works and how to repair it or use it or whatever, but the thing that can help people find that is the tagging. And I just wonder in terms of as we’re all navigating these environments, there’s this kind of truism that or not truism, but among a lot of my friends, it’s a truism, that well-structured, meaningfully tagged content is going to serve LLMs or any artificial agent better. Does that make sense, and can you expand on that if it does?
Sarah:
This takes us away from talking about authoring and generating content, and more into this question of, if I have existing content and I’m using it as the foundational source material for something that I’m using a chatbot or something for, what does that look like? Basically, we worry about chat interfaces generating incorrect information because they reach inside what they’ve got and they come up with these relationships that make sense but aren’t real, or they make mathematical sense, but they aren’t real.
Sarah:
So, if I have a collection of articles and every single one of them has a chunk that’s tagged, summary, and then I ask it to give me the most important points from these various articles, I could reasonably expect that the summary would contain those points, and probably I can write a prompt that says, “Look in the summaries,” as opposed to process all of the articles. So, what metadata and semantics, these labels that you’re applying to your content, what they give you the ability to do is essentially provide a roadmap or a set of guardrails or whatever you want to call it, to the engines that are processing that content, so that they can more successfully and more accurately reach in and get the information that you want them to get.
Larry:
You just alluded. You said something a minute ago about the math involved in this, and I assume, are you referring to the vectorization of the embeddings like that, where you get that?
Sarah:
Yeah, it’s just math.
Larry:
Yeah, it’s just math, and you can do different things with math. You can do a probabilistic thing like an LLM does and say, “Hey, these things that are close together in vector space often unfold in this way, so this is the most likely next word.” But you can also more intelligently vectorize something and with an ontology or some other mechanism to say like, “Oh, no, we know from our modeling of this domain and how we’ve expressed it in this ontology that those things are related, and therefore, this is going to help you answer that question better.” Are we to the point of actually doing stuff with that yet?
Sarah:
Yes. This gets us into RAG, Retrieval Augmented Generation, and I am something like the opposite of an expert on this, but essentially, what we’re talking about here is a scenario where you don’t just say, “Hey, large language model, give me everything,” or, “Write stuff with no context.” You say, “Oh, over here, I have a set of facts for you to work with. These are things that I know.” So, you have a knowledge graph or you have a database or you have various kinds of things, and essentially, you’re saying to your chat interface, “Use that as a starting point,” or, “Use that to validate what you’re generating, so that you don’t start making stuff up because I’m going to give you the framework of what the known facts are that we’re working with.” Right?
So, in a scenario where you ask it for a history of a particular country, you can point it at data points that are sitting in this database or in, as you said, ontology, knowledge graph, whatever, and also, then allow it to vectorize. So, you give it a set of… I don’t know if it’s guardrails or a starting point, but something like that. We’re right on the edge of things that I do or don’t understand, but the acronym for this is RAG, Retrieval Augmented Generation.
Larry:
Yeah. I’ve come across a couple of good articles that explained that. I’ll put the links to them in the show notes (What is retrieval-augmented generation?, IBM;
Unifying Large Language Models and Knowledge Graphs: A Roadmap) because it’s basically that notion of not interrupt, but just kind of injecting into the LLM’s workflow like, “Hey, you’re about to…” It’s almost like Clippy is repairing. “Hey, it looks like you’re about to predict the next word. Why don’t you consult this authoritative source before you answer?” That’s my understanding of it anyway.
Sarah:
Yeah. That sounds right.
Larry:
Yeah. That’s interesting. Well, are there others? Every day there’s some new acronym or something like RLHF, reinforced learning with human feedback, and there’s all these things, and I heard you say something, which is one of many reasons. I was going to have you on the podcast regardless, but you said something about like, “Oh, yeah, great. We learned it this morning. It’ll probably be obsolete this afternoon.” I guess, let me ask it this way. How are you keeping up with this stuff?
Sarah:
Yeah. I’m not totally sure that I am, so that’s a concern, but I think to me, it’s helpful to sort of step back and look at this in the context of a content life cycle. You create stuff. You format stuff. You deliver stuff. There’s a couple other steps, but basically, right? So, when you look at technology and tools, well, what can we do? How does this change the various facets of the content life cycle? And if you start breaking it down that way and thinking about how does this change distribution, how does this change production, what does it look like to automate those pieces, and what are the pros and cons of doing it that way, that’s been very helpful. The other thing is that when we look at business strategy, and I picked up sort of a five-piece pyramid that somebody else did, but basically you’ve got things like compliance and cost avoidance and revenue growth and competitive advantage and branding.
Sarah:
Okay. Well, right now, a lot of the conversation around AI is focused on cost avoidance, right? Everybody’s talking about cost avoidance. “Oh, I can automate this. We can fire all our writers. It’ll be great.” But tech comm is largely talking about compliance like, “Yeah, you can do that, but then you’re going to ship your doc to the FDA, and they’re going to take a very dim view of a bunch of hallucinations in your content.” What I think is more interesting is what about revenue growth? What about competitive advantage? What about branding? Because I can automate. We can automate anything. The question is what are the trade-offs of doing that automation?
Larry:
I’ll include a link to that pyramid you’re referring to because, like you said, the lower down in that pyramid, the easier it is to automate it. The higher up you go, the harder it is. But I’m going to guess that you’re assuming that the return is higher, the higher up you go in that, so it’s probably worth figuring it out. Is that how you look at it?
Sarah:
I would say the higher up you go, the more sophisticated your strategy needs to be. Yeah. I’m not sure that necessarily… I mean, compliance is one of these all or nothing, right? If you don’t conform to the regulatory systems, you tend to get shut down. We had an example of that pretty early on where there was an attorney that got in big trouble because they used ChatGPT to write their brief, and they brought the hammer down on that guy pretty hard. That’s a pretty decent example of, “Oh, this will speed up my workflow and it’ll be great.” There’s some discussion that the attorney who did this didn’t really understand that it was something you had to vet and verify and check on.
Sarah:
I really like the machine translation analogy. I think that’s very helpful because I think we all understand that machine translation is really fast and basically free and not always accurate. And if there’s an article out there in the world in Icelandic, which I really want to read and my Icelandic is lacking, well, I can use machine translation and it’ll give me that article. Okay. And then, I could read it and get the general gist of what’s going on there. But I would strongly advise against trying exactly that, and then signing a contract based on a machine… “Oh, sure, I machine translated this Icelandic contract and now I’m good.” No. Right? So, the stakes are different. The stakes are higher. And I think we have to look at the entire AI landscape with that exact perspective, which is what are the stakes for this content and what are the stakes if it’s wrong, and how do I address that and how do I put in some governance and some guardrails to make sure that we get it right at the level that is important. Right.
Sarah:
I mean, if you use ChatGPT to generate 200 LinkedIn posts and pre-schedule them, a couple of them are going to be wrong. So what? Who cares? It doesn’t matter. If you use it to generate, again, your medical device documentation and it’s wrong, you’re going to be in big trouble first with the regulatory bodies, and second and maybe worse, if it slides through, people die. Right? Because the instructions are wrong, and that device can kill people if used incorrectly.
Larry:
Yeah. No. That gets into the whole thing around ethics and safety and all the stuff around that. But hey, Sarah, I can’t believe it, but we’re already coming up close to time, but I always like to, before we wrap up, give my guest, is there anything last, anything that you want to make sure we get to or that you want to revisit from the conversation?
Sarah:
No. I think you’ve covered it, and I think really the piece that… Oh, two things. One is that I’m spending a lot of time talking about risk, and I think that’s really the correct lens to look at this. What is the risk if it goes wrong? If the risk is low, then fine, have fun, enjoy. That’s one.
Sarah:
The other thing that I will mention, and we haven’t really touched on this, is that the question of copyright is still an open one, and there are a couple of different issues here. There’s the question of are these large language models going to get shut down because they sourced content that’s not theirs to source? That’s question A, and there’s a whole bunch of legal issues around that. And then, question B, which I think, if anything more problematic, is that according to the U.S. Copyright Office, currently, you cannot copyright something that was generated by a machine, period.
Sarah:
All right. So, you have your corpus of all this content that you carefully curated and created and approved and vetted and translated, and it’s got all this value, and then you stick an engine on top of that and say, “Summarize this,” or, “Generate a summary,” or, “Generate a Dr. Seuss version,” which I just am going to use forever because I think that would be super fun. Okay. The thing that you just generated is not copyrightable because the machine generated it, but the underlying source belongs to you, so did you just strip the copyright off your content, not the underlying content, but the end result? I mean, the plain language regulation says yes. I tell people this and they’re like, “Well, that won’t happen.” Well, okay, but that’s literally what it says right now.
Larry:
Yeah. Having spent-
Sarah:
So, I find that concerning.
Larry:
No. Having spent a lot of time… I used to work in book publishing, and man, I spent a lot of time with lawyers, various things. Yeah. Both sides of that that you just mentioned are definitely going to be in play. Yeah. Yeah. Well, hey-
Sarah:
We’ll see.
Larry:
Yeah. Well, lots to unfold and I think that’s right, and that doesn’t come up as much as you’d think it would, that whole copyright issue. I’ll definitely pay more attention to that as I go forward.
Sarah:
Everybody’s just YOLO-ing it, so I don’t know.
Larry:
Yeah. That’s how it feels. Yeah. One very last thing, Sarah, if folks want to connect or follow you online, where should they go?
Sarah:
Okay. Our website is scriptorium.com, and from there, I mean, I’m on LinkedIn and a couple of other places, but the easiest way to reach me is probably just to send an email to info@scriptorium. I do see those and reach out that way.
Larry:
Cool. Well, thanks so much, Sarah. Really good conversation.
Sarah:
Thank you. It was fun.
