Show Us Your Agent Skills, Episode 1
This transcript and summary were AI-generated and may contain errors.
Summary
I was the first guest on the inaugural episode of Show Us Your Agent Skills, a livestream hosted by Hugo Bowne-Anderson and Thomas Wiecki where practitioners screen-share their actual agentic development setups. The other guests were Jeremiah Lowin of Prefect and Randy Olson of Good Eye Labs. My segment runs through roughly the first third of the episode; I had to leave for a family dinner and didn’t make it back, so the remainder is Jeremiah and Randy walking through their own workflows.
The hosts opened with three questions for each guest. On what I love most about agents, I said it’s not having to write code anymore — which surprised me, because I’ve loved writing code and produced an enormous amount of it over 20-plus years. What I didn’t expect was how much of a relief it would be to be freed from the boilerplate and tedium of building and delivering software. I mentioned I’d tuned out of LLMs for all of 2023 and 2024, partly because they weren’t good enough yet, and that catching up caused a bit of a personal identity crisis last year. On what’s most frustrating: they don’t listen, they lie, and they make the same mistakes repeatedly. There are perpetual conspiracy theories about labs quantizing models to squeeze capacity out of their GPU clusters, so you wake up not knowing whether today is smart Claude or dumb Claude — and a lot of days it’s dumb Claude.
Then I shared my screen. Over the previous six months I’d generated on the order of a million lines of code across about a dozen projects, and I was averaging something like 1.3 to 1.4 billion tokens a day. I started with spicytakes.org, which summarizes and pulls the spiciest quotes out of 11,773 blog posts across 34 blogs, because I like reading blog posts and nobody has time for them anymore.
The frame for the rest was a quote from Jesse Vincent, who created the superpowers skills framework: the difference between vibe coding and agentic engineering is planning, architecture, and caring about the output. My stack is built around superpowers plus a series of tools I’ve written to solve the peripheral problems of running many projects in parallel — observability into what the agents are doing, quality assurance on what they produce, and management of the development pipeline.
I demoed roborev, a daemon that reviews every commit your agents generate. I have a hard rule in all my CLAUDE.md and AGENTS.md files that agents commit every turn, so reviews run asynchronously in the background and accumulate in a ledger. Agents periodically invoke the roborev fix skill, which picks up open reviews and applies fixes. The reviewer itself is Codex with GPT-5.5 at extra-high reasoning — I’ve found it the strongest code reviewer available. For very large plans I have the sub-agent-driven development skill invoke roborev fix every five tasks, because I’ve run superpowers specs of 45 tasks and had one implement for 14 hours without stopping, which produces an enormous amount of code to review.
I also showed AgentsView, a session database with full-text search and token analytics across agent sessions; Middleman, a local GitHub dashboard with a threaded activity view for maintainers, built because GitHub is slow and painful and puts recent activity at the bottom of long pull requests; and kata, a local terminal issue tracker I wrote after getting frustrated with GitHub issues and having beads damage some of my Git repositories.
Asked how much code I actually read, I said almost none directly. My mantra is that roborev reads every line of code that gets generated, and it gets read several times — by the time I merge a pull request, agents have read the code four or five times minimum. What I look at is structural: does it look right, is it too complex, does it need simplifying, is there inappropriate scope creep. I’m still very engaged in what the software does and whether it’s doing the right thing, because sometimes agents go off the rails and build something completely inappropriate and you have to nuke it.
The constraint I keep hitting isn’t compute, it’s me. I said I’d like to turn this into more of an automated software factory, but I’m bandwidth-limited on decisions — between parallel spec interviews I’m already at my decision-making capacity, at the point of “don’t ask me another question, I can’t make another decision today.”
On safety, I admitted I run in YOLO mode more or less all the time, and that being maximally productive with agents currently requires being a little unsafe. We’re all walking honeypots with our SSH keys and agent sessions, and prompt injection and supply chain attacks are real problems. Eventually sandboxing will be ergonomic enough; in the meantime I’m praying to the token gods.
After I left, Jeremiah talked about using OpenClaw as a second brain fed by daily voice memos, his frustration that agents reviewing pull requests are biased toward acceptance — which is wrong for a framework, where the bar should be much higher — and his “explain skill,” whose one load-bearing sentence asks the agent to explain a change the way you would to a colleague who knows the project. Randy walked through a data visualization skill that encodes Tufte principles as an LLM-as-judge verifier loop, researching data from the web, generating chart variants, and iterating until the output passes programmatic and visual checks.
Key Quotes
“I guess not having to write code anymore. I love writing code and wrote an enormous amount of it over the last 20-plus years. But I didn’t expect that not writing code would be such a relief.” – Wes McKinney
“They don’t listen. They lie. They make the same mistakes over and over again.” – Wes McKinney
“Each day you wake up and you fire up Claude Code and it’s like, what kind of Claude am I going to get today? Is it going to be smart Claude or dumb Claude? And a lot of days it’s dumb Claude.” – Wes McKinney
“The difference between vibe coding and agentic engineering is planning, architecture, and caring about the output.” – Jesse Vincent, quoted by Wes McKinney
“I almost don’t read code now. The mantra is that roborev reads every line of code that is generated. By the time I’m merging a pull request, the code has been read by agents four or five times minimum.” – Wes McKinney
“I’m bandwidth limited in terms of my ability to make decisions throughout the day. I’m already at my decision-making bandwidth. It’s like, don’t ask me another question, I can’t make another decision today.” – Wes McKinney
“To be maximally productive with agents requires you to be a little bit unsafe. We’re all just walking honeypots with our agent sessions.” – Wes McKinney
“The best contribution is a great issue.” – Jeremiah Lowin
“How do you say no to something that is technically good, but not aligned with the purpose of your framework? Agents turn out to be really bad at that.” – Jeremiah Lowin
“You don’t want to just tell it what to do. You also want to tell it how to check it.” – Randy Olson
Transcript
[Show intro]
Show us your agent skills. What are people at the top of the game building with AI agents, and how are they doing it? Are they Claude-maxing with eight terminals open at once, or adversarially testing Opus 4.7 generated code with Codex? Do they define suites and swarms of sub-agents, or use agents, markdown, and agent skills? We are on a mission to find out. First on the roster are Wes McKinney, the creator of pandas, Jeremiah Lowin from Prefect, and Hilary Mason of Hidden Door, along with a few more surprise guests. Join us to find out what all these token maxers are doing. Think Excel World Championships meets Eurovision. See you all in the first episode.
Hugo Bowne-Anderson: Welcome everybody to our very first episode of Show Us Your Agent Skills. So excited to be here today with a lot of old friends and familiar faces — Thomas Wiecki, Randy Olson, Jeremiah Lowin, and Wes McKinney.
Wes McKinney: What is up, everybody? Love that intro.
Hugo Bowne-Anderson: And Wes, thanks for sharing that video of your workflow that we could put in the trailer as well. It’s really great to see you in 90s 8-bit side-scrolling glory.
Wes McKinney: Of course. It was a surprise, but I enjoyed it. I showed it to quite a few people. I even changed my GitHub avatar. So thank you.
Hugo Bowne-Anderson: Fantastic. We’ve already got 30 people watching, so thank you so much for joining, everyone, for our first episode. The real motivation here is that we’re all agent-maxing at 3 a.m., surrounded by glowing lights, and it feels like we’re alone in Times Square at 3 a.m. building. We want to share what everyone’s working on — and not always the hottest, shiniest workflows, but things that are really impactful.
Feel free to ask questions in the chat. We’re doing it in Discord so we can continue the conversation afterwards. YouTube live chats are ephemeral, but agents are forever, as we know. Also please like and subscribe if you like this type of stuff. And leave a comment if you do — if you don’t like it, don’t leave a comment. It’s really easy to not do that, you would think.
Wes McKinney is an entrepreneur and software developer focusing on many things including analytical computing. Famously wrote the beginnings of pandas surrounded by rats in a small East Village apartment — correct me if I’m mistaken. His greatest work yet, though, is spicytakes.org, which we’re going to look at today.
We have Jeremiah Lowin, the founder and CEO of Prefect, where they’ve been building some of the most popular tools for managing data and AI workflows. And we have the one, the only, Randy Olson, co-founder and CTO of Good Eye Labs, pointing frontier AI models at a lot of business problems today.
We have a little thing we want to do, which is ask everyone three questions before we dive in. What do you love most about working with agents?
Wes McKinney: I guess not having to write code anymore. I love writing code and wrote an enormous amount of it over the last 20-plus years. But I didn’t expect that not writing code would be such a relief. What’s nice is that it relieved me from the things I didn’t like about software engineering — a lot of the boilerplate and tedium involved with building and delivering software projects.
Hugo Bowne-Anderson: I can relate to that a lot.
Wes McKinney: Especially some of that pandas code I had to write in 2017, 2018. It caused me to have a bit of a personal identity crisis last year, but it’s all good now. I’m good.
Hugo Bowne-Anderson: Just last year?
Wes McKinney: I was kind of tuned out of LLMs for all of 2023 and 2024. I was just like, F this.
Hugo Bowne-Anderson: That explains why you have this resurgence now. You’re making up for lost time.
Wes McKinney: Yes. But then, they couldn’t write code. They weren’t that good.
Hugo Bowne-Anderson: What do you find most frustrating about working with AI agents?
Wes McKinney: They don’t listen. They lie. They make the same mistakes over and over again. The ignoring instructions and the occasional stupidity, which I think is sometimes the result of — there are lots of conspiracies about the AI labs quantizing and doing things to get more capacity out of their existing GPU clusters, and in the process dumbing down the models. So you never know. Each day you wake up and you fire up Claude Code and it’s like, what kind of Claude am I going to get today? Is it going to be smart Claude or dumb Claude? And a lot of days it’s dumb Claude. That’s pretty annoying.
Hugo Bowne-Anderson: Without a doubt. And dumb is like — sometimes I’m trying to do something simple and it will bamboozle or discombobulate for seven to eight minutes, then come out with nonsense.
Third and final question: if all AI agent conversations somehow got leaked to the public, what would you be most concerned about becoming public?
Wes McKinney: Mostly private personal projects that I work on becoming public. Not that there’s anything so crazy, but it would expose a lot of private details of my personal life and projects I’ve built for myself that contain non-public information — things I don’t publish on the internet. Nothing bad, but stuff I wouldn’t want published.
Hugo Bowne-Anderson: You may be one of the only people who has nothing bad in there.
Wes McKinney: Having all of my Claude.ai or ChatGPT sessions public — that’s another matter, since people ask everything there. Health issues, all kinds of things.
Hugo Bowne-Anderson: Wes, do you want to share?
Wes McKinney: Claude is now running on Grok data centers, so maybe it’s coming sooner.
Hugo Bowne-Anderson: He who controls the flops controls, et cetera. There’s a not-so-distant future in which we’re all just paid in flops by agents too. And we’ll probably be paid in bad flops half the time.
Wes McKinney: All right, you want me to take it away here? I can reveal my stack.
First of all, I’ve been building a lot of stuff with AI over the last six months or so — probably generated on the order of a million lines of code across a dozen projects. It’s been a lot of work, some serious projects. Let me share my screen.
As Hugo mentioned, one of my fun things is this website called spicytakes.org, trying to solve the problem that people don’t have time to read blog posts anymore. And I actually really like reading blog posts. So I made a system using AI, of course, to summarize and pull the spiciest quotes out of 11,773 blog posts across 34 different blogs from people I follow online.
You can go here and read Martin Fowler’s takes on the mythical man-month, just published this week — not very spicy. John Gruber. Armin Ronacher increasingly has been quite spicy. If we look at Joe Reis, “Why token maxing is for fools: a rant on fake productivity.” Lots of fun stuff to check out there.
Speaking of which, I am quite the token maxer myself. This is a token-maxing leaderboard. In the last seven days I’ve averaged something like — let me see if I can do math — 1.3, 1.4 billion tokens per day. So I’m burning quite a lot of tokens.
I wanted to explain a bit of my setup and how exactly I’m doing that. I’m working on a talk I’m giving next week in San Francisco at AI Council, so I’ll give a little preview. Jesse Vincent, who created the very popular superpowers skills framework, has this quote which he gave me permission to use: the difference between vibe coding and agentic engineering is planning, architecture, and caring about the output.
Essentially I’ve been building systems to do agentic engineering and not just vibe coding. When I started out it was just, okay, I’m going to open Claude Code in the terminal and type prompts and see what happens. And at a certain point you’re like, wait, this is not creating things that are good. I can look at the output, I can run the code, I can see if it makes sense — but how do we manage all this at scale? If you work on six projects in parallel, how do you keep the train from crashing into the side of a mountain, or whatever the right metaphor is?
My agentic productivity stack basically centers around the superpowers skills framework, created by Jesse. Over the last six months I’ve been building a series of agentic engineering projects that solve the peripheral problems associated with developing lots of projects in parallel: wanting observability into what all the agents are doing, quality assurance on the work they’re producing, and being able to manage the development pipeline. How do you observe the software factory you’re producing? More recently I’ve gotten into granular project management and issue tracking, after being burned by beads and having beads destroy some of my Git repositories, very annoyingly. I hear that it’s gotten better now that it’s on Dolt.
A lot of this runs in the terminal.
Hugo Bowne-Anderson: Could you just zoom in a bit on the terminal?
Wes McKinney: Give me a minute here. Just got to sort out the 12 terminals.
Hugo Bowne-Anderson: Relatable. I’ve never been able to figure out those multi-terminals in one, I always just do tabs in iTerm.
Wes McKinney: At some point I became very comfortable with the split — split vertically, and that’s all I can handle. When I run out of vertical panes, then it’s a new tab.
Hugo Bowne-Anderson: The whole horizontal thing, my brain doesn’t handle it.
Wes McKinney: All right, we’re telescoping. So very briefly — I don’t have great time to go into all these things, but I built a system called roborev, which is a daemon that runs in the background and reviews all of the code your agents generate.
I created this “show us your agent skills” repository here. I run roborev init to add post-commit hooks to this repository. Now whenever the agents commit — and I ask my agents to commit every turn, that’s a hard rule in all of my CLAUDE.md or AGENTS.md files; I use both Codex and Claude to do development.
I built this little macOS widget which shows my work over the last 30 days. You can see I’m about three-quarters Claude Code and one-quarter Codex. If I were paying API rates I’d be paying about $21,000 a month in tokens. I do pay less than that on account of subsidized tokens from the labs. So thank you very much, Sam Altman and Dario Amodei.
roborev is a terminal UI. Whenever we create a readme and commit it in this repository, we can see it commit and see what happens. The idea is that agents generate code, it gets committed when the turn is over — or while the superpowers plan is implementing you might have lots of commits — and those all get reviewed in the background. Then you can ask the agents, while they’re implementing or executing, to periodically invoke the roborev fix skill, which picks up the code reviews that have been running asynchronously in the background and applies the fixes.
You can see here there’s a commit that was generated by the agent — it was a minimal readme placeholder, so there’s really nothing to review. I can close that review and it’s hidden from the roborev queue. If I hit H it unhides the closed reviews.
We could ask the agent, while we’re looking at this, to do something trivial. “Hey, I want to make a simple dashboard showing frequency of recent spicy takes on the spicytakes.org repository. Just stand up something simple, commit frequently, use standard tools.”
Hugo Bowne-Anderson: Make no mistakes.
Wes McKinney: Of course.
Hugo Bowne-Anderson: Don’t lie.
Wes McKinney: Right, don’t lie. Exactly.
Hugo Bowne-Anderson: One quick question. Last time we spoke you mentioned you were writing a lot of code with Opus, and getting GPT-5.2 or whatever it was to do the reviews. Is that still your main setup?
Wes McKinney: Yes. You see Codex doing the review. If we look at the review prompt, you can see it’s using codex exec, GPT-5.5 model, reasoning extra-high, with this prompt — “you’re a code reviewer,” here’s the review prompt, and it inlines the code change. I’ve found that Codex is the strongest code reviewer out there, at least GPT-5.5, that model specifically.
So I’ll let this grind. roborev is nice because you have this ledger of code review feedback you need to address. You have to explicitly close reviews, or if you invoke the roborev fix skill it will look in the ledger, pick up all the open reviews, fix them, and commit the fixes.
The next project in my stack is called AgentsView. It’s a fancy agent session database that provides full-text search and analytics on all your agent sessions. So you can find stuff in previous sessions as well as get information about your token use. I’m not going to pull up my AgentsView because it has data from a number of projects I don’t want to show on the livestream, but it is super useful. If you look at the website, it supports a ton of agents. It’s got a nice web application and a desktop application.
You can see here we’re seeing the superpowers spec interview. A lot of my process is — I work on four or five projects during the day, I’ll run parallel spec interviews with superpowers. I’ll be bouncing between different terminals, which are all on my large monitor. I’ll participate in these spec interviews, build the implementation spec with superpowers, and then set it implementing. Once it’s done implementing, I’ll invoke roborev fix to address all the code quality problems that have accumulated.
For very large plans I’ll ask the superpowers implementation skill — called sub-agent-driven development — to invoke the roborev fix skill every five tasks, so that it pauses and addresses code review feedback that has piled up while implementing. I’ve implemented superpowers specs that run 45 tasks. I had one that ran for 14 hours without stopping. You can imagine, 14 hours of code generation is a lot of code to review, and a lot of code review issues that pile up.
Another project we’re building, which is a bit newer and I haven’t been advertising much, is called Middleman. It’s basically a fancy local GitHub dashboard, because let’s face it, GitHub is dying and slow and painful to use. It’s also difficult to get a single pane of glass across all of your projects.
Middleman has a threaded activity view which, if you’ve ever been a maintainer on open source projects, is a godsend — to be able to see at a glance all the activity, the pushes, the comments, the commits happening on different pull requests without having to go visit the pull request on GitHub. You can see I’ve set up roborev as a CI reviewer on a lot of these pull requests. Contributor working on this project, force push, here’s a roborev review, force push, another review. You can even take these reviews and fold them right back into your agent session.
You can see here that superpowers has written a spec for this spicy takes dashboard design, and I’m asking it to go ahead and write the plan. Meanwhile down here roborev has been dutifully reviewing the code being generated by the agent — which right now is just specs, but it does provide feedback on the specs.
In here you can directly merge pull requests. Here’s a pull request I created a little while ago. I don’t even have to go to GitHub, I can just merge it from here, and it disappears from my activity feed. To be able to do development and not have to go to GitHub and deal with their flaky mess is pretty great.
Hugo Bowne-Anderson: Also, Git has always been kind of tough in a lot of ways, and the fact that Git is something agents are really good at for the most part — you’ve got to be careful with a few things, but 95% of the patterns you’ll need, agents can do, and you don’t need to worry too much about it.
Wes McKinney: In here we’ve implemented a diff viewer, and we’ve got a repository dashboard which shows statistics about the different projects we’re working on. You can browse all the pull requests for different projects, clicking around instantly.
One thing that’s nice is the activity is shown in reverse chronological order — the new activity is at the top of the pull request. It seems obvious to me. The joke is that GitHub is in league with Big Scroll: they really want you to have to scroll to the bottom of a pull request with 100 comments every time. So we put the recent activity at the top so you can see what’s going on at a glance.
Another project I built recently, as I got frustrated with beads and with GitHub issues — I wanted a local issue tracker for my projects. Here I can run kata init in this project, and then I have a terminal issue tracker. For this “show us your agent skills” repository there are no issues right now, but I can ask the agent to read the kata quickstart, which explains how to use kata and how to create issues. You can think of it as a local replacement for beads that’s a lot simpler and doesn’t create a lot of complexity for having your agents keep track of things in a local issue tracker.
I also use a terminal workspace manager which creates Git worktrees. For all my projects I can flip between different Git worktrees and different projects I’m working on. Every Git worktree has a stack of terminals — typically my stack is Claude Code and/or Codex plus roborev running in every worktree. So I can flip back and forth and say, okay, what still needs to be done here? Oh, there are two failing issues that need to be addressed in this worktree. And I can invoke the fix skill and have those issues fixed.
Let’s check in. Still writing the plan. One of the pros and cons of superpowers is that it generates amazing software, but it also takes a long time to generate very detailed implementation plans. The idea is that it doesn’t really trust leaving that much up to the agent in terms of making decisions about how to implement things. So it tries to figure all that out up front and review it before handing off to a bunch of sub-agents to implement different stages of the project.
Hugo Bowne-Anderson: I’ve got a quick question. One thing you said you loved about agents is you don’t really have to write code anymore. How much code do you read, and how much do you think you’ll be reading in five years?
Wes McKinney: I almost don’t read code now. My approach with roborev is that it’s my code reader. The mantra is that roborev reads every line of code that is generated. And it gets read multiple times — whenever I push up a pull request the branch gets re-reviewed, so by the time I’m merging a pull request into a repository the code has been read by agents four or five times, minimum.
I look at the code in terms of structural detail: does it look right, is it too complex, does it need to be simplified, does it have scope creep that’s inappropriate. I ask all those types of questions and often pose those questions to the agent. But I’m pretty in the loop — I’m very engaged in what the software does. Is it doing the right thing? Is it doing too much? Does it need to be pared back? Because sometimes the agents will go off the rails and build something completely inappropriate, and then you have to nuke that.
Let me go back to my screen share. I just wanted to show what superpowers looks like. I’m going to set it to work with sub-agent-driven execution. This implementation might take another 20 or 30 minutes. I can stay on the call, or I have to move locations — I’ll drop back in and maybe I can show off the dashboard a little later.
All the while, down here, roborev is reviewing the code. It isn’t perfect, but it does catch — you can see there are seven tasks in this plan — it does catch quite a lot of the most serious issues. Certainly from a code correctness standpoint, and from a sleeping-well-at-night standpoint. I’m shipping a lot of code. A million lines of code in six months is a lot of code, but I’ve been burning a tremendous amount of tokens to make sure it’s reviewed very thoroughly. So far so good.
I’m thinking about ideas for how I could turn this into more of an automated software factory and build even more software. But the trouble is that I’m bandwidth limited in terms of my ability to make decisions throughout the day. I feel like in all these spec interviews I’m already at my decision-making bandwidth. I can’t make any more decisions. It’s like, don’t ask me another question, I can’t make another decision today.
Hugo Bowne-Anderson: A question on that. We are seeing decision-making or judgment being transformed into intelligence at a pretty interesting pace. It’s a silly example, but I think it’s telling — auto mode on Claude Code, where it will accept certain write things but won’t accept delete things, rm -rf, these types of things. Remember we used to have to sit there and go accept, accept, accept, or do the YOLO mode where you might get everything deleted. So this is something that previously was judgment and we had to make decisions, and now, because we have training data, it’s incorporated into intelligence. I’m wondering how we can supercharge this process.
Wes McKinney: I’m just in YOLO mode all the time. My approach is: back up, try not to store too much sensitive data in my home directory, that kind of thing. But the trouble is SSH keys and so on — we’re all just walking honeypots with our agent sessions.
So I think it’s a balance between do you want to be productive or do you want to be safe. Right now, to be maximally productive with agents requires you to be a little bit unsafe. At some point we’ll figure out a way to make the sandboxing work in a way that’s ergonomic. The number of agent sandboxing projects out there is pretty intense, so it’s going to take some time to figure that out. In the meantime I’m just praying to the token gods to be kind to me — let me not be prompt injected. That being said, supply chain attacks are also a big problem, and as the agents get more sophisticated, the nature of the security attacks gets more sophisticated, so that’s even more of a problem.
Anyway, I will yield the floor to the next guest, but I appreciate you all listening, and look forward to seeing you all in the token mines.
Hugo Bowne-Anderson: All praise be unto the token gods. We need a token god totem. Thank you for coming out of the token mines and bringing your AI lasers to the streets to help us battle all the agentic systems we’re working with as well. And do feel free to pop back.
Wes McKinney: Of course I will.
Hugo Bowne-Anderson: Now we’re going to have Jeremiah. I’ll remind everyone, if you’ve just joined, come and chat with us in Discord so we can continue the conversation afterwards.
Thomas Wiecki: Very excited to welcome Jeremiah. I think we met over 10 years ago at an amazing startup called Quantopian in Boston. Already back then I remember we talked about the sorry state of data pipelining and orchestration, and then of course you went ahead and actually fixed it with Prefect and built an amazing company around that. And now also with FastMCP, I think you just made MCP usable for the first time. Welcome to the show.
Jeremiah Lowin: Thank you, Thomas.
Hugo Bowne-Anderson: We have our usual questions to start. What do you love most about working with AI agents?
Jeremiah Lowin: I thought about this, because it feels too obvious to say the productivity gains, and I think Wes has already covered a more interesting second answer. For me, it’s that I use it as a second brain. I have big expectations about the information that I put in at a moment coming back out later. This is one of the reasons I use OpenClaw, for example — so I can go muck around with its memory in a way that works for me. I don’t know if my way would work for anyone else. That’s kind of a theme of what I want to talk about today: custom software, just-in-time software.
I’ll go into Claude or ChatGPT as a very convenient, especially mobile, way to answer a question or investigate something. But most of the real things I want to know about or work on, I have trickled information in over weeks or months. I started prepping for this years ago. I started recording my meetings, which many of us do now. But the thing I do that folks are still surprised by, and that’s very important to me, is I record a voice memo almost every morning of what I’m thinking about and what I want to do. If I’m driving to my office and dropping off kids at school, I’ve got 30 minutes in the car, so I tee up my day. When I’m working from home it’s a much shorter voice memo. In a previous era of LLMs that would be my system prompt for the day, and it affects everything. Now, with more modern memory systems, it’s an update on what I’ve accomplished that might be out of sight for my agents.
That’s been a huge unlock, and a really valuable habit. I’d love to tell you that I put a meeting in and now I know everything, but the truth is I’ll be walking down the street and think, here’s an idea, we should name it that, we should do this. And that can go in. So that’s what I really love — pouring information in and then working to get it out. That’s sent me down all kinds of rabbit holes around memory systems and knowledge and retrieval.
Hugo Bowne-Anderson: I completely agree. Having the context and the ability to pull all these different threads together is a superpower. The way OpenClaw does this is a cron job. As I always half-joke, there’s a reason Dr. Dre called his album The Chronic — it’s cron jobs.
Thomas Wiecki: So what have been your pet peeves?
Jeremiah Lowin: I have a particular one that goes beyond the mainstream hatred of the things it does. When you build software that’s a framework in particular, how you take modifications and contributions into that software is, in my opinion, very different from how you take contributions into an application. A framework is for building applications, and then we have applications.
The way agents review code, for whatever reason, seems to have a bias that the PR should be accepted — and if only you change these things, it will be accepted. This is the wrong approach for a framework. A framework should not be modified unless whatever’s coming in is so overwhelmingly useful to an overwhelming majority of users that the framework should take it on.
I live this with Prefect, and I’m living it now even more with FastMCP, which is a great framework, but people are doing a lot of things with it — a lot of really cool things that don’t belong in the framework itself. So we get a lot of PRs saying, it should do this thing that is useful to me. And I agree it is useful, it is cool, but I don’t want it to go in the framework. This has been a real problem with deploying agents to review the code. The code is fantastic, but what it does is not something that belongs in the larger framework, and the agent will say, if you just do these two things, I will accept this. And I’m like, no, this is rejected on its face.
I wrote a blog post about this that somehow climbed the Hacker News ladder and had a surprisingly positive reception. It’s called “An open source maintainer’s guide to saying no.” This was hard enough when code was expensive to write, when the act of writing and contributing code was the admission ticket to interacting with the maintainer. Now the code is so cheap and it’s just getting lobbed over. There’s a real imbalance as a maintainer, which you feel and don’t want to take out on the contributors, to be clear. But how do you say no to something that is technically good, but not aligned with the purpose of your framework? Agents turn out to be really bad at that, and that drives me nuts.
Hugo Bowne-Anderson: That’s a great post, and I’ve just linked to it in the Discord. Everyone should check out Jeremiah’s blog, Mostly Harmless. I messaged Jeremiah earlier that he probably should change the name these days to Mostly Harness.
I quite like this movement toward saying don’t send PRs anymore, send issues — because code gets stale so fast.
Thomas Wiecki: If your agent skills got leaked, what would you be most concerned about?
Jeremiah Lowin: It’s the same second brain that I love so much. It’s all there — all my deepest and darkest fears, and how I hate spiders, and all of that is probably in the brain of this thing, and I don’t want folks to know about that. I think it speaks to how prevalent these things are that I almost think that’s the most boring question, because it’s everything. It’s worse than my internet history.
Hugo Bowne-Anderson: While you’re bringing that up — you hate spiders. In Australia we have a lot of poisonous spiders. Thomas is in New Zealand where they have a lot of spiders, none of which are poisonous. I don’t know which I prefer.
Jeremiah Lowin: Size of spiders is one of the reasons that, as much as I want to come visit you, I never have.
Hugo Bowne-Anderson: We’ll get you here one day.
Jeremiah Lowin: Please don’t prank me any other way that I might actually die on the spot.
Okay, so I’m bringing my screen here. We are in the FastMCP repo. Randy, because you asked about this — I want to give a lot of credit to Bill Easton, one of the maintainers of FastMCP, who wrote a blog post which literally says shut down PRs, we do not accept PRs. We didn’t go quite that far, but we wrote this contributing doc a couple of months ago, and this is kind of the headline: the best contribution is a great issue.
Bill and I did a FastMCP maintainers podcast about a month ago and we were talking about this, and he provokingly said, how dare you say your agent is better than my agent at actually doing this? If I’m — why is it that my Claude implements the MRE in the issue better than your Claude opening the PR? I think that’s where some of what you want to talk about on this show, which is clearly nakedly just about stealing people’s agent skills — and we’re cool with that, that’s okay — why is it, what skills have I put in, what biases have I put into it? The funny thing is, not that much, but just enough that it’s stylistically aligned in a way that’s going to make it easy to maintain the code later. Folks forget about the importance of the maintenance of this stuff.
There are a lot of skills I use in particular to work on FastMCP, where I’m the core maintainer, which are hard to show, so let’s talk through them. My workflow here is extremely agent-driven, but as I said, I don’t like how the agents interact with a lot of the contributions, so my workflow is surprisingly manual on FastMCP. I will spin up agents both to write my code and review others’ code, but ultimately I will step in and look at it and work on it. It’s probably a lot slower than it would be otherwise, and I keep wishing for an agent that won’t make that sentence true — but the codebase is a lot better off as a result. As a sole maintainer who does not yet trust an army of agents to do this, I kind of need that. You should see the things that the agents have tried to sneak into this codebase.
One of the skills I value most, because I operate massively in parallel in this hybrid agent-human workflow, is this “explain” skill. There’s nothing grand about it. This is not like the superpowers collection Wes was talking about a moment ago. This is a minor, stupid, natural language prompt for the agent — as far as I can tell it’s about 80 lines long, but there’s only one sentence that actually matters in it, and it’s been in every version of the skill. It says: talk to me like you’re explaining this to your colleague who knows about your project but wants to understand what you just did.
That sentence changes the tenor. I’m running 10 different agents on 10 different things over 10 different timeframes, and I’m cracking open my laptop with a coffee thinking, what’s this one? If I just ask what are we doing here, it’ll say, oh, we renamed this variable on this line to do this — and that’s no good. If I step back and ask what does this code do, I’ll get an explanation of the code, but not the change we’re making. So this skill, with that key sentence, gives me the right mental model of not just the broad grounding, but what are we up to here, in a necessarily technical but not overly verbose way. This skill has become my workhorse. It is referenced in every other skill I have.
Hugo Bowne-Anderson: A lot of people still ask me, when do I use skills and when do I use MCP servers? I like to think of MCP as when you want to solve distribution problems, and serve data or certain types of tools to other people. But as someone who works so deeply with both, I’d be interested in your thoughts.
Jeremiah Lowin: Skills are awesome ways to steer behavior. They go into the agent’s brain in the exact same way that a message from you does. MCPs are great ways to distribute business logic from a central place.
Obviously this leads into the MCP versus CLI style debate, which I think is stupid. If you’re an individual, use whichever one you prefer. It’s a non-starter if you’re an enterprise — we’re not installing a bunch of CLIs on people’s machines. We never have, we never will. It’s a nightmare. We’re going to distribute the business logic centrally. So I find that debate kind of uninteresting.
The prevalence of it speaks to an interesting problem in MCP land, where most people think of MCP as a way for companies to reach their customers — it’s for reaching out to third parties and fanning out. I will tell you that overwhelmingly the use case for MCP is distributing internal business logic to internal teams in enterprises. So you have this real misrepresentation, where the MCP versus CLI debate is insane to most users of MCP. We don’t talk that way. But for the average person who knows what MCP is and probably uses it in their IDE, the representative use case is something like the GitHub MCP server, where it is a third-party relationship. And this debate over CLI actually makes a lot of sense when you’re one user deciding how you want to access that. That’s my soapbox.
Hugo Bowne-Anderson: Can you talk us through your skill now? This is the first time in this session we’re looking at an agent skill, and some people may not have even seen agent skills. Maybe you can talk us through the markdown and the idea of progressive disclosure.
Jeremiah Lowin: Let’s do a little anatomy of a skill. Skills are shockingly simple for how effective they are. They have two pieces of front matter: a name, which is really important — that’s how you invoke it — and a description. The description is going to always be seen by the agent. The rest of it, and this is the genius thing about skills, why they’re not just canned prompts: everything down here is only seen by the agent if and when it decides to invoke the skill.
So your description actually really matters. That’s why in a lot of skills you’ll see a very clear instruction of when to use this, because there is a tool call, in a sense, to load the skill and see all the rest of it. That progressive disclosure is the magic of skills. You can have tons and tons of skills that are all shown as these little tiny descriptions, and then the agent says, I need this skill.
The downside, if you will, is that while a skill can tell an agent what to do — and more complicated skills can include scripts and other actual business logic — fundamentally the skill is natural language instruction masquerading as a workflow. The way I like to put it is that it’s a polite note to your agent, and usually it does what the skill says.
Hugo Bowne-Anderson: And they’ve gotten a lot better as well. Even six months ago, agents and LLMs have gotten a lot better at interpreting descriptions than they were when we had our moment last October.
Jeremiah Lowin: It’s true. And yet a lot of times I have to say, use your explain skill, even when I say explain this to me. It’s not perfect.
Hugo Bowne-Anderson: I still use slash commands a lot as well for that reason. And something else lovely here with skills — you don’t just throw everything in context. We’re told we have million-token context windows, but effective context windows are far smaller, so we still need to be mindful. We need to context engineer constantly as developers and users of AI agents. You can put anything there — if you need to do this, use this MCP server, or involve the human in the loop, ask the user. I also love that you’ve included what not to do.
Jeremiah Lowin: This skill, you can probably tell by reading it, is LLM-authored. It has all the little telltale negative contrasts and all of that. But I like to think of skills as living documents, which is one of the reasons it’s nice that they’re on your machine — though that also kind of makes distribution hard.
I have 20-odd skills in this folder. Most of these skills are changing, as I go, oh, this didn’t work. I have one here called GitHub reply, which is intended to make it respond using my voice in a reply. It’s not because I’m trying to masquerade it as me — I’m usually pretty obvious if I’m using an LLM to draft a reply. It’s because I think there’s a right way to treat people and the LLM doesn’t do it. These are all little annoying things, like: don’t say “great work” followed by a rejection. That’s confusing.
Hugo Bowne-Anderson: I thought that skill was just going to say, say no, close the PR.
Jeremiah Lowin: No, we have to politely say no.
This is probably the first skill I ever wrote and the one I use the most, and it gets me in more trouble: “ship it.” As you can see, I say a lot here. It means open a PR. It does not mean merge the code. That’s bitten me a lot — most LLMs think “ship it” means merge it. That’s not how I use it. This worktree stuff is probably a holdover from when these things didn’t know about that. You can see from the arguments that this started as a slash command.
Why does this skill exist? It’s not a useful skill. I haven’t even read this skill in a year, probably. It seems like a really stupid skill, actually. But the reason it exists is because I want to write the words “ship it” and have the right outcome happen, and this skill is my bridge to ensuring that. And here’s a skill for creating skills — I don’t think I wrote this, I cobbled it together from a lot of things I found online.
Thomas Wiecki: There are a lot of things coming together, right? One is GitHub just being too easy to have huge PRs that then you need to review or decline, as well as having your own preferences of how you want things done. I think it was Peter Steinberger who said that in GitHub repos you’d like to not even have a feature to submit any PRs, but only prompts — and those prompts are easier to review, and they kick off code generation once you’re happy with the prompt, using your machinery and your custom skills. Do you think that would be a good model?
Jeremiah Lowin: I’m into personal software, basically, in the most true sense. And in a funny way I think that’s also resulting in very personal decisions of how you achieve that software. You have your harness — OpenClaw, or whatever you want to use as your base substrate. And then the way people pile on functionality, features, customizations, skills, is deeply personal.
You look at things like Claw Hub, which is the skills repository for OpenClaw — huge power law in popularity in those skills. A lot of people are using a small number of them. If you go look at them, I ask my OpenClaw all the time, do you want this skill? And it’s like, no, this is garbage, I already know this, it doesn’t affect my behavior. So I think there’s a lot of work to do in how people acquire these customizations.
Let me blitz through two things and then I’m going to yield some of my time. I desperately wanted to create MCP apps in Python. That meant I needed a Python frontend framework that didn’t require a backend, which almost every one of them assumes. We don’t get that here — we have an MCP server. So this started as a mini project inside FastMCP and has now become its own thing as of a couple weeks ago.
What was cool is that because of the same insight that MCP has product-market fit in enterprises, it means MCP apps also have product-market fit as data dashboards and for distributing information internally. This is a Python DSL that generates generative interactive UIs. It’s all Python and it looks pretty bizarre. But this and generative UIs has become my new spelunk — more than a side project, not quite a career. All for the goal of building interactive dashboards that can stream from an agent’s brain and don’t have to be hand-coded.
Hugo Bowne-Anderson: I’m really excited about Prefab, because I really think that’s the future of where things are heading. We’re starting to see Claude pop up interfaces right in the chat, and even when it’s developing things and scoping, you can start putting a prototype right there. Enabling that through MCP is amazing.
Jeremiah Lowin: My team uses this for slides sometimes. Someone published a dashboards overview and included Prefab in it and styled his Prefab.
Hugo Bowne-Anderson: While Jeremiah is bringing that up — something we’re talking around is why we should be building for agents as well as humans, and everything we build for a browser should be headless as well. Anders, who works over at dbt Labs, published a dashboards overview and included Prefab in it.
Jeremiah Lowin: There’s this MySpace theme, and then the other was a Windows theme. And I just love this. This is just base Prefab underneath. I want to talk to Anders — literally, I saw this three hours ago — and make these default themes in Prefab.
And then, on the theme of custom software, this is another experiment of mine. This started a few months ago where I wanted custom slide software, mostly because of the way I give a talk — the speaker notes, I want them a different way. This is very live, this came off the agent presses last night, so I haven’t fully used it.
This is the latest version of a piece of software I call Cardboard, which is for laying out my conference talks as cards on a board. It follows a vocabulary I’ve developed purely for me — no one else should use it — where I think of my talks as having acts and beats, and within those beats we have slides. This is the top of the Cardboard, where I lay out the acts and the beats. This was a keynote I gave at the PyAI conference. If we scroll down we get the actual slides, which here are just mocks. But I’m working on the actual speaker notes, which for me always have this blue, gray, yellow and pink format. This is my custom slide software, and then this turns into the deck.
This has been really fun — to have a piece of software that makes talks exactly the way I want them, the way I like to give them. I don’t think anyone else would use it. Maybe they would, I don’t know. Most importantly, this is not an interactive UI. This is read-only. I’m clicking on this stuff and it jumps down; that’s the one interaction it has. I interact with this entirely over an API or an MCP server, talking to it from any agent that can connect to it. So exclusively this is recording a mini voice memo, putting it to an agent, seeing the changes here, reacting in voice. I can’t edit this if I wanted to.
Hugo Bowne-Anderson: And the brain behind this is OpenClaw?
Jeremiah Lowin: This one I principally work with with my OpenClaw, because that way I can work on a talk — there’s a talk I’m giving in three weeks for PyData London. I can feed in something tonight, close it, don’t worry about it, talk to the agent about a thousand other things, and then come back and pick right up because of the memory substrate there. So I typically use my OpenClaw for stuff like this, which is really distributed over time.
Hugo Bowne-Anderson: Is that the only one you use, or do you mix between OpenClaw and Claude Code?
Jeremiah Lowin: I use OpenClaw as my main personal interface because of how I’ve customized its memory. When I’m working on code I use Claude Desktop and Codex Desktop, which I migrated to from the CLIs mostly because of how much better they are at managing parallel sessions. Those are my two, and I have my own heuristic about which is better for what.
Hugo Bowne-Anderson: Very cool. Thank you once again, Jeremiah, for coming and showing us your agent skills. We should talk more about the future of ephemeral software, just-in-time software. The ability for everyone to generate things that are helpful to them and their communities — we’ve just got the tip of the iceberg here.
I am so excited for Randy Olson to show us what he’s up to. There are so many ways to introduce Randy. One of the first podcasts I did nearly a decade ago was about automated machine learning and all the wonderful work you’re doing on TPOT. I remember I asked you, if you want builders to take away one thing from this conversation, what would it be? And you said, just stop doing basic grid search, there are so many other interesting things to do. And yet, look where we still are — LLMs still doing grid search.
A lot of the other things I’ve loved about your work are all the amazing things I’ve learned from you about data visualization. You’ve been moderating the subreddit on data viz for a long time. But also your work at Good Eye, everything you’re thinking about with respect to evals, and the work you’ve done helping non-technical people build with AI.
So, our three questions. What do you love the most about working with AI agents?
Randy Olson: I agree with what everyone has said so far — of course the productivity gains are huge. I’m a builder at heart. I love building things. I have so many ideas and I’ve always been limited by capacity in the past. Now it’s just fire up a new chat, fire up a new tab, boom, you can build that in an hour.
And also treating it as a thought partner. I have a growing memory base where I throw in ideas and track everything. That’s invaluable, especially because most LLMs, even nowadays, still tend to be sycophantic — they’ll just agree with you. When I load an AI agent with what I call my digital twin, it knows I actually want someone who’s not necessarily harsh but won’t just agree with me, who will push back. I want a thought partner. You can induce that by prompting LLMs.
Another thing I really like is how you can be working while being outside touching grass. Some days I’m better at that than others. Like Wes was showing us earlier, I think we all use the superpowers flow to do development — it’s kind of slow, but you can have four tabs running in the background and say, you know what, it’s beautiful out, it’s time to go for a walk. And you’re still getting stuff done. I’ll self-admit, I’m a workaholic and I’m also a hyper-focused person — I can spend 12 hours straight just hacking away at code. It’s great to be able to step away.
Hugo Bowne-Anderson: Without a doubt. I totally agree with being able to do it while I go for walks on the beach, jump in the water, come out, check what’s up. I do think it can mean my time away isn’t always time away. But I’m also interested in the future of being able to chat with these things better, so I don’t need to look at my phone all the time.
Randy Olson: I think that’s feasible nowadays. You just invite a meeting recorder, listen, and then send it off to an AI agent. Go build.
Thomas Wiecki: Same question — what do you find most frustrating about working with agents?
Randy Olson: I think it is the natural tendency for them to be extremely agreeable. Like we were talking about earlier — the default assumption is, oh yeah, we’re going to merge this PR. The default assumption is yes, we’re going to do something, yes, we’re going to give in to this demand. They’re too agreeable by default and you really have to fight them to get them to not be agreeable.
I’ve written about this a little. There are ways you can prompt them — you can take advantage of their agreeableness to be disagreeable, or be constructive. But it’s really frustrating. I think it’s getting better over time; the post-training is starting to focus more on combating it, but that is a very ongoing frustration for me.
Hugo Bowne-Anderson: And if all agent conversations somehow got leaked, what would you be most concerned about?
Randy Olson: Of course all the personal detail stuff. Maybe some of the tics that I have when I’m interacting with AI agents. I always find myself typing the same things. One of my most typed phrases is probably “be concise and unambiguous,” and I’m typing that over and over. I should just turn that into a skill. So probably funny little quirks like that.
Hugo Bowne-Anderson: I feel it’s unfair to ask that question without answering it myself. The thing I’d be most concerned about is some of the words I use when I lose it — when I’ve been engineering for too long at the end of a long session and it does something really dumb, and some of the words which would get me banned from YouTube. Words that are acceptable in Australia but definitely wouldn’t be over there.
So without further ado, let’s jump into all the amazing visualization stuff you’re up to.
Randy Olson: First I thought I’d share — I have this visualization skill workflow I want to show, but since we were talking about skills in general and there might be folks who are very new to skills, let me screen share.
This is the actual skill, which we can make available after this. You have your basic front matter that is used for progressive disclosure. Here are some things I think about when I’m working with skills.
At a high level, with AI agents in general: you can open up whatever AI agent tool you’re using, and I highly recommend connecting as many data sources as you can to it. The more information you give your AI agent — whether through MCPs, CLIs, whatever — the more it’s going to be able to do on your behalf. If you really restrict it down to just one folder or one codebase, it can do a lot, but if it can read your codebase, check all the changes you made today and write a report automatically onto Slack — I have a ton of cron jobs that do things like that, that automatically send off reports to colleagues. Of course, getting their permission first, and with permissions your IT admin is okay with.
Another one that’s really useful: I highly recommend that in every skill, if you need to do anything, make sure you have a phase at the start of the skill that checks your environment is actually set up. In this skill it’s going to be using some unsurprising libraries, so it needs to install those first. Otherwise, if you just tell it to go do this thing and it’s trying to write Python code, it’s going to crash and have to figure it out on the fly.
Another thing I highly recommend, which is not in this skill but is done in many well-designed skills, is to design your skill as a thin driver. This is the idea of progressive disclosure. This skill is relatively very long and has lots of stuff in it — every single one of these phases could be its own reference markdown file. Then if half of what’s in the skill is not relevant to what you’re doing right now, it doesn’t have to load that into context. If I just want to jump straight to phase four, it can load the phase four file into its context and not waste context on phases one through three. If you ask Opus to do that it’ll do it by default usually nowadays, and that’s really good skill design.
I would also say design your skill to be super unambiguous. This one is relatively unambiguous — it’s very specific, at this step do this, at this step do that. It tells exact commands to run and provides exact code snippets. The more specific you can be in your skill, the more repeatable it is in the future. Because when it comes down to it, a skill plus an LLM is kind of a program. LLMs are good at handling stuff that’s a little bit ambiguous, whereas if you need an environment set up, you can just tell it to run these commands.
Hugo Bowne-Anderson: And providing it deterministic scripts to run at certain points as well. So we have degrees of agency there and autonomy.
Randy Olson: A hundred percent. If I were to decompose the skill into a local file, all this code would be a separate Python file. I just put it all in one file so it’s relatively easy to see here.
The last thing I’ll comment about this is: always, at the end of your skills, do a reflect-and-improve loop. You really should treat your skills as a living artifact. They’re not a one-and-done thing, they’re something you need to improve over time. When you run through a new use case, you’re probably having to correct how it does things — how it visualizes data, how it thinks about data, how it makes a good data visualization. If you explicitly put in your skill, at phase seven, “reflect and improve” — how did this run go, how can we improve the skill to make it better — then it just becomes part of your loop. Every single run you’re learning something new and putting it into the skill, and it’s compounding. That is a super powerful thing about skills in particular, that compounding nature.
Hugo Bowne-Anderson: So many wonderful tips and best practices. Something came to mind that’s tangentially related — the idea of a skill being something that’s continuously updated and iterated on. Something I’m trying to help people understand is that a harness isn’t something that a lab ships and you use. For those who haven’t heard this term, think of the LLM as the brain, and the harness is all the things that do stuff from the brain — tools that it uses, whether that’s write to a file, execute something, connect to an MCP server, use this skill.
What happens is that when you write a skill or connect to a different MCP server, essentially they become part of your harness. So the idea of iterating constantly on a skill is part of the mental model of continually building and rebuilding your harness. I’m interested if this is how you all think about that.
Randy Olson: I totally agree. Unless you’re hardcore and running your own local models, the model’s fixed. You can choose between a few frontier models, but there’s not much you can do with that. Even the harness, unless you’re also hardcore and making your own harness, that’s kind of fixed too. The harness helps orchestrate things like when to use sandboxes, when to access the MCP, when to hit the model, when to think, when to take an action. Whereas the skills are really the thing that can evolve with you, and they can guide the harness plus the model toward what you actually want it to do. And they’re just text files. It’s great.
Hugo Bowne-Anderson: Totally. And the models have got — we had that moment with 4.5, and then that week where the three labs all released models that put us in what Simon Willison calls the inflection point. They sucked up the harnesses that we were all building. They did reinforcement learning via verifiable rewards on all the traces, and meant we didn’t need all the tool calls we were using. At that point, the amount you can get away with a bash tool and a search tool and a bunch of skills — Bob’s your uncle, as we say. There’s a post by Amp, a really interesting coding agent and product, where they wrote that harnesses are dead. It’s provocative, but the basic idea is that models are getting so good that harnesses are better when extremely minimal now.
Randy Olson: I agree with that too. It still makes sense to keep building harnesses and keep building skills on top of harnesses, but the great part is that can just become training data for the frontier model. And then straight out of the model it becomes able to make beautiful charts. I would love it if they trained on my skill and then we don’t need the skill anymore.
So, for those who are unfamiliar: back in the early 2010s I was in grad school, looking for any excuse to do anything other than work on my PhD. Hopefully my advisor is not listening, although I think he knows. I was working on data visualizations like crazy. I’m a builder, I’m a very curious person, so I’d always go find a dataset, try to find an interesting question, and make a blog post about it.
I worked on very insightful things, like: what are the deadliest films of all time by on-screen death counts? There’s a forum where people actually watch movies and count every single death. I solved “Where’s Waldo” — or Wally, in other parts of the world — using machine learning to find where he’s biased toward being, and then optimize the path. I optimized road trips. Really important spurious extrapolations, like the frequency of the use of the word “novel” in academic research papers, where if we extrapolate, 100% of research papers by 2130 will use the word novel. And there was a short period where we were all obsessed with marble racing and data visualization on that.
What I was always trying to do was find something interesting, or sometimes something silly, and through data visualization make machine learning, make AI, or even just make data topics more accessible.
Eventually I realized, hey, AI agents can do this. When I first tried to have an AI agent just make a beautiful data visualization and make no mistakes, what popped out was honestly unimpressive. That’s when I realized I have to encode what makes for a good data visualization, at least in my opinion. Having done this a long time, I’m highly opinionated on what makes a good data visualization. If you’re familiar with Edward Tufte, a giant in the field, he’s very opinionated — no chart junk, has to be a clear story, clear annotations, straightforward use of color, and so forth. And I said, what if I encoded those principles into an AI agent?
The idea of what I built is: you start with an idea, and then the AI agent researches the web to find data and other stories that support or touch on that. Then it tries to create a valid data visualization and tell a valid data story from that.
Here’s a prompt into Claude Code. I always recommend still using the frontier models. “Execute this workflow to visualize the history of marriage and divorce in the USA.” I ran this ahead of time because it takes a while. It loads the workflow, and then it immediately sets up the environment and does the dataset discovery. I’ve biased it toward certain sources — the CDC, government and educational institutions — for more reliable data rather than some random dataset a rando posted. It researches a whole bunch. It finds marriage rates between this period, needs to find divorce rates, and researches. It looks at PDFs, various web pages, and so on. Then finally it pulls it together into a single dataset.
This is already pretty cool. It took an idea, researched the web, and turned it into a dataset that we can go back and look at the sources for. This is amazing, because I’ve done this chart before — it literally took me an entire weekend of scraping through PDFs. This data is not cleanly organized out there. And it just did it. I think this part took maybe five or ten minutes.
The next stage is it creates several variants to try out different chart types — line charts, small multiples, an area chart — and then it looks at them and, using the data visualization criteria I give it, says which one it thinks is better. This time it chose small multiples. I actually ended up overriding it. But it starts with that prototype, because the first thing you want to do when you’re visualizing data is just look at it. How does this actually look? What is the distribution? It’s getting a sense of what stories it can tell and what forms of visualization are useful.
Then it runs what I call a verifier loop. This is another thing I didn’t mention: another thing I’ve learned with skills is you don’t want to just tell it what to do. You also want to tell it how to check it. That’s why there was all that code in that skill — that code is used to verify certain things about the data visualization, like a good DPI. I also use LLM-as-a-judge evals that encode Tuftian principles to look directly at the image and say pass or fail, and if it fails, give feedback on how to improve. It loops over and over. This time it was an easy one so it passed pretty quickly, but it’ll loop and keep fixing. These are basically guardrails on the skill, because what I’ve told it to do opened up a massive realm of possibility, and this scopes it back down.
So I looked at the variants and said I actually prefer the dual-line variant. This is the resulting data visualization, and it’s pretty decent. It got the data, it shows an interesting story, it calls out interesting aspects — these are well-known stories in marriage and divorce rates over time in the US, like the post-World War II peak and how it’s been going down. It puts a minimalist annotation here. Other things that are heavily enforced: very minimal background details, no chart junk, labeled axes, using just the ranges that it needs, making sure everything is clear and telling a clear story, to make a shareable thing.
Hugo Bowne-Anderson: And I just want to be very clear — this is a result of this agent running the skill with no human intervention yet.
Randy Olson: Exactly, zero. Well, in this case there is, on the verification loop.
Hugo Bowne-Anderson: Can we go to the chart you just showed? It’s incredible that you’ve been able to encode all of this in a skill. There’s one thing that I’d change, and it’s the final frontier of making sure data viz looks cool. It’s this freaking overlap.
Randy Olson: It’s so annoying, isn’t it?
Hugo Bowne-Anderson: This is a problem in all kinds of image-related things. If anyone solves this consistently, you’re a billionaire. I don’t know why AI models are so bad at detecting when an annotation is over a line.
Randy Olson: So this is the final frontier of human in the loop. This is basically what I do. By the way, I run this skill every single morning, and that’s how I make that post series. Most of what I do is like, oh, I’d like that image more, or hey, an annotation’s overlapping. Otherwise looks good, post it. Maybe that would be another embarrassing thing if all my chats got out — you assume I’m typing genius stuff into my prompts, but actually I’m like, hmm, looks good, let’s proceed.
Hugo Bowne-Anderson: Would you mind if I quickly shared my screen? I ran this skill the other day when you sent it to me and I think it works so wonderfully. The idea I put in was: no horses broke Secretariat’s 1973 Kentucky Derby record. Where did I get that idea? From Randy’s blog.
So I put in that idea, and you can see the setup it did. This is a log I wrote — I often get my agents to write brief logs. I’ve got a skill which is write log, make it human readable, concise yet comprehensive. So I can quickly see everything that happened. It found the data. It found a Randal Olson CSV. Then it did three parallel variants, verifier loop.
It put notes for the human reader — “the Randal Olson coincidence,” classic LLM style language there. The very first search for the dataset surfaced a blog post called “No horses broke Secretariat’s 1973 record,” and found Randy’s. And then it said his post used the same source, considered downloading it, but I’m going to use Wikipedia instead. I found that really interesting.
Once again, pretty good — not quite as nice as Randy’s, and a bit of overlap here. The one other thing I’ll mention is that Randy’s generator-evaluator loop is something that when Anthropic originally wrote their “Building Effective AI Agents” blog post, they recognized as a really important workflow, one of five workflows. It’s great to see this in the wild.
I do have a question, Randy. We talked with Wes briefly earlier about turning judgment into intelligence. You’ve taken things which normally we’d need to be applying judgment in the loop and encoded them in a skill, with the verification loop. How many things can we do that with? And how do you approach encoding judgment into intelligence?
Randy Olson: How many things can we put in there? I’m trying to push the boundary of that every day.
I really approach it like a data scientist. Let’s just try it first. Literally, the whole Tufte test idea, where I created an eval for a Tufte test — I knew the principles, so let’s put that into an LLM that can look at an image and apply it, and just see if it works. Then I keep tuning from there. I try more examples. I build a full set of data visualizations, ones I know are good and ones I know are bad, and I keep tweaking until — essentially I’m treating the eval itself as a living document. Try things out, this worked well, this didn’t, let’s encode that. And now I have not just a skill but an eval, or a verifier, that is hyper-focused on judging that one thing and saying yes, no, this is good, and here’s why.
So that’s how I approach a lot of that: it’s all down to data, experimentation, and continuously improving things — accepting that it’s never going to be amazing at first, but if you have that compounding loop then it’s going to get better over time.
Hugo Bowne-Anderson: We’ll wrap up in a few minutes. I don’t know if Wes is going to join again to show us what happened with his runs, but I am interested in people’s general thoughts around when we do build evaluators and verifiers, the ability of agents to game them, and how vigilant we need to be. Do you have any opinions about that, Jeremiah?
Jeremiah Lowin: Quite vigilant. It depends on your objective and your risk tolerance. Some years ago we were working on an agent framework called Marvin, which to this day I still think is the greatest way to work with agents, but it didn’t quite catch on. We had a lot of LLM-as-judge situations in there, especially because we didn’t know how to unit test our own — evals weren’t a thing, so we reached for the thing we did know, which is we have a magic talking computer now, let’s ask it if it did a good job.
The thing I took away from that is that LLMs have a little wiggle room in what they return to you, and when you’re trying to nail down something kind of subjective that needs to be run very frequently, that variation really plays out in a way that you don’t notice in a one-off conversation. It forces you to watch what happens over 10,000 runs of the same thing. Today we call that an evals suite and there’s a science around it. But back in those early days I got a very healthy respect for the fact that that variation really is there. You just don’t notice it in any one instance.
Randy just showed us how valuable it can be for honing — it’s like hill climbing to an outcome, so it’s really useful. I think you just have to temper it for the precision you require in your outcome, like any good model.
Randy Olson: It’s so dependent on the context. If it’s a guardrail and something absolutely cannot happen, then you’re probably going to want to find something more deterministic. But even still, if an eval is wrong 20% of the time, it’s still directionally valuable. It’s like the Tufte test — I’ve not measured it, but I have seen it judge things wrong. Still, on the whole, it’s really valuable to have in the loop, because it catches a lot of the obvious stuff. And that’s one less time I have to spend my thought tokens. I’d much rather spend Claude’s thought tokens doing something like that, especially if we think about doing this at massive scale.
Hugo Bowne-Anderson: Absolutely. And the other point we’re talking around is that evals aren’t static, like skills. They’re something you constantly iterate on, and perhaps more at the start, and then build trust with them.
In a few weeks Hamel Husain is going to join us to take us through his AI eval skills, which is going to be super fun.
Everyone, thank you so much for joining. Please check out the Luma calendar, which I’ll link in Discord — it’s in the YouTube description for our upcoming events. We’ve got one next week where Hilary Mason is going to show us some of her skills from Hidden Door, Eric Ma, who leads research data science at Moderna Therapeutics, Brian Bischoff, and Tomasz Tunguz from Theory Ventures as well. And then we’re going to Europe for our Eurovision special.
Do check out the agentic data science course that Thomas and I and some of our colleagues are teaching. Thank you all for joining. But most of all, thank you to our wonderful guests for coming and sharing your agent skills with us.
Jeremiah Lowin: It’s a pleasure. Thanks for having us.
Hugo Bowne-Anderson: If you enjoyed it, share it with a friend, hit like and subscribe. And as Dr. Dre said, see you in the next episode.