James Blair Part 2: Solutions engineering, critical thinking, and staying human
This transcript and summary were AI-generated and may contain errors.
Summary
In this second part of our conversation with James Blair on The Test Set, we explore his journey from longtime RStudio user to solutions engineer to senior product manager at Posit. James describes how attending RStudio Conference in 2017 led him to discover the company behind the software he loved, and how the solutions engineering role became a perfect fit for someone who excels at finding ways to make things work at the intersection of tools and user needs.
James shares an innovative approach he’s developed for creating targeted demos using AI. Rather than relying on generic datasets like mtcars or DC bike share data, he built an internal tool that uses Claude Code to generate synthetic, industry-relevant data and then create full demo projects including Shiny apps, Quarto documents, and ML pipelines. This allows solutions engineers to have more meaningful conversations with customers using data that resembles their actual work, reducing the cognitive burden of having to mentally bridge from generic examples to domain-specific problems.
The conversation turns to how we each use AI coding tools in our daily work. I discuss using Claude Code as my primary tool for building bespoke utilities and tackling development work I would never have attempted before, like terminal UIs and CSS improvements. James emphasizes the importance of remaining an engaged participant when using these tools, reviewing code carefully and being able to defend any solution you submit. Hadley suggests using AI to tackle long-deferred issues that seemed too daunting. James closes with advice for people entering data science: your humanity is your greatest strength. The value we produce was never the code we could write, but the questions we asked and how we thought about data.
Key Quotes
“The value that we produced was never the code that we could write in my opinion. It was the questions that we asked and the way that we thought about data.” — James Blair
“Remain curious, stay human, use these tools to help you be better at what you do best, be better at being curious, be better about asking the hard questions, be better at critical thinking, because now we have an opportunity… Don’t use that as an escape valve to be lazy, use that as an opportunity to be better at the things you’re uniquely suited to do.” — James Blair
“If I’m going to put my sort of engineer hat on for a minute and use some of these tools to try to pull a solution together for a problem that I feel passionate about, then I want to make sure that I can defend what I’ve submitted and the solution that I’ve proposed because I understand it.” — James Blair
“This shouldn’t be convenient for me and a burden for someone else. It should feel like it was a convenience for both of us.” — James Blair
“These tools really only work as well as you can articulate yourself and know what to ask for. And so if you aren’t articulating clearly and in great detail what you need the agent to create for you, and then if you aren’t looking at the code that it’s generating and scrutinizing it, and actually giving good quality feedback about what it’s doing wrong or what it’s not doing, it can really create a gigantic mess.” — Wes McKinney
“There’s this increased urgency for everyone to be much better at code literacy and reading code and to spend more time doing analysis of code that they didn’t write and to be able to read and give feedback on code because we’re all going to be writing a great deal less code than we did in the past.” — Wes McKinney
“Find an existing project where there’s something you’ve always wanted to do for a long time, but you keep putting off because every time you look at it, you’re like, oh, this is going to be way too hard. And just throw Claude Code at it.” — Hadley Wickham
“I think there’s something about that nostalgia that is like comforting to engineers who are a little bit worried about AI taking over their jobs.” — Michael Chow, on Claude Code’s terminal interface
Transcript
[Podcast intro]
Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning, digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
This episode is part two of a conversation with James Blair, Cycling Boss, Data Junkie, and Senior Product Manager at Posit.
Michael Chow: I’m so curious to get into your use of AI because I know you’re a preeminent AI tinkerer and doer. We’ve already talked about bytes and filament. I think it’s really clear that you have experimented with AI. I think for people to really appreciate the types of problems you’re solving at Posit, I wonder if it’d be useful to talk just a tiny bit about what a solutions engineer is and what it looked like for you. Because my understanding is you are solving a lot of interesting problems for solutions engineers now as a product manager.
James Blair: Yeah, no, that makes sense. And I think some context here, so when I joined Posit, which was RStudio at the time, seven and a half years ago, I joined as a member of the solutions engineering team, which was tremendous. It was this accidental perfect fit where I had been a longtime user of RStudio and the software, and I attended the RStudio conference in San Diego in 2017. And I was just blown away by the quality, not just the quality of the conference, but the community, the people, and particularly the people of the company, which I had never thought of RStudio as a company until that point in time. I’d always just think it’s the software that I really love to use. But then I put two and two together and was like, oh, there’s people making the software. And I knew some of them, like I knew Hadley from his contributions to the space. And so I knew some of the names, but now all of a sudden I saw these people in real life. I had a chance to interact with them. And I was like, there’s a tremendous culture. There’s good vibes at RStudio. It was kind of the way that I left that. So I knew from that moment, I would love to work for this company and was just kind of really trying to figure out how I would fit.
At the time, RStudio didn’t have data scientists really working internally. It was open source teams and then people building software. And I was not a software engineer and I was not a prominent contributor to the open source ecosystem of our packages. So I was just like, where’s my entry point? And I found this listing for solutions engineer one day and the word engineer in the title almost scared me away. I was like, engineer is not me, but I was desperate enough. I clicked on the link and I read the job description and I was like, this is me. This is exactly what I want. I know RStudio software. I know the ecosystem and I’m really good at finding ways to make things work in that intersection. So I came in that way, came in on the solutions engineering team, spent a number of years there, loved it. Absolutely felt at home, felt fulfilled, felt everything that I wanted to and more from working for the company there.
Michael Chow: Do you mind recapping just for folks, what a solutions engineer is and does?
James Blair: So there’s two sides of this. There’s what it was when I was there and then there’s what it is now. So when I was there, it was very much like we worked with customers who found themselves in interesting predicaments. So we’ve got a great support staff at Posit, but then every now and again, there’d be issues where it’s like, particularly when it was like, this is an intersection between they’re using our commercial tools and they’re also using some of the open source stuff. And they’re trying to mesh these two things together in ways that we just are trying to make them successful at. And we would come in, solutions engineers would come in and kind of understand the problem, work with the customer and kind of ideate on solutions. And a lot of times what that meant is that we would get really interesting feedback that we could funnel back to open source development teams. We could funnel back to the commercial product teams because we were kind of in the trenches with customers doing really interesting work.
And the fascinating thing is data’s everywhere, right? So you’d work with a significant pharmaceutical company one day, and then you’d be working with a New Zealand dairy farm the next day to solve very, very different problems, but with a very similar set of tools. And so it was really fascinating to just kind of think about how flexible open source is when it comes to solving problems and really, in my mind, reemphasized the significance of the open source philosophy that we continue to hold ourselves to as a company. So that was a lot of it.
And then the other thing that was really beneficial was there was a lot of autonomy in the role at that time. So you’d work on customer issues, but then sometimes there wasn’t a fire to be put out. And so you just kind of had a chance to contribute to open source if you wanted to, to build out proof of concepts, write blog posts. So there was all this, you kind of choose your own adventure. And again, for me, that just worked really well. I ended up working with the Plumber package for quite a while and did some things there and really dove into that world. I ended up working with a lot of the partner groups that we have here at Posit, which led to the role that I’m in today on the product side. So it just kind of found my way through various projects on the solutions engineering side.
Michael Chow: I didn’t realize too, you were opening PRs to Plumber and you got a little—yeah.
James Blair: Oh, cool. Working closely with the team that managed that package. Kind of figuring out, I made the Plumber cheat sheet that still is in existence today, which was a ton of fun. And then ended up kind of as a spinoff of that work, worked directly on the Plumber Tableau package, which is an interface between Plumber APIs and Tableau extensions, which was a lot of fun and kind of very similar to the partner work that I’m still engaged in today.
Michael Chow: Yeah. I feel like the thing is too, with cheat sheets, it’s like people love cheat sheets. I feel like nothing hits as hard as a good cheat sheet somehow. RStudio has this whole repo of cheat sheets and there’s a whole bunch of them. If they went away, people I think would riot and burn the internet down.
James Blair: It was also one of the times I remember working on that. I have a giant, my monitor is a 37 inch behemoth and I use all, normally I have it split in four panels as if it’s four separate monitors. I don’t know, it’s 43 inches. It’s huge. Anyway, I remember working on the cheat sheet. I used all screen real estate and I just had like, I think I built it in Keynote. It was kind of a design and software choice, a little bit odd, but I just remember sitting there at my desk for days on end building out this cheat sheet in painstaking detail. And I’m so glad I have this giant monitor because I could step back from my desk and see the whole thing and then get back into whatever part of it I was putting together. It was a fun time.
Michael Chow: I mean, I think it’s very real. The fact that you required every inch of your monitor for the cheat sheet, I think does really speak to the desire to shrink information down so small that it can be printed on a little page. And you have to think critically about what should be included, what is really important as a reference point, what’s not important and people can kind of look at other documentation sources for. So yeah, that was a fun project.
I want to get back to the point that you brought up originally, which is on the AI side and the work that I’m doing now. So coming from the solutions engineering side, it’s interesting at Posit, a lot of the product management team is made up of former solutions engineers, just kind of the path.
Michael Chow: Is there a reason for that? Do you think?
James Blair: I think part of it is, there’s something to be said about solutions engineering really helps you learn how to listen to customers and users, right? I keep using the term customers, but this extends beyond just people paying for software. This is people using software and with our commitment to open source, there’s a lot of users that are equally as important to us. And so really learning to listen to feedback, understand pain points, understand use cases, and then be able to step back from that, look at the bigger picture and figure out ways that we might be able to solve for those things. I think you really become good at that in solutions engineering, and that’s a really critical skill for product management as well. So I think it just kind of became this path that developed on the Posit side and the product team has been built up over time. We didn’t really have a formal product team when I joined and that has since changed where we do now have a well-organized, well-built out team of product managers. And so I think over time, people as they’ve had the opportunity and the desire have made that migration over to the product team.
Michael Chow: Sorry to keep derailing you. I know you were getting to something about AI and products work at Posit.
James Blair: Yeah. So I think the AI saturated world that we live in is I think challenging to navigate in the best of circumstances for a number of reasons, but there’s also, in spite of all of the noise and chaos that I think exists right now, there’s also quite a bit of excitement if you kind of know how to look for it. I think it’s an incredible time to build and think about solving problems because it’s really easy to build proof of concepts when before it would take a while. And so there’s ways you can kind of operate in this world that I think are really unique.
One of the things just to give a glimpse into some of the stuff that I’ve been involved in recently, one of the things that I’ve thought about for quite a while, particularly given the background in solutions engineering, is this idea of we’re constantly kind of trying to tell the story of what our software does, again, on the commercial and the open source side. And so we’re often sitting in front of a Zoom call or in person with an organization trying to kind of help them catch the vision of what sorts of problems we can solve. And traditionally, we’re really good at telling that story with mtcars data. It’s like, here’s some pretty simplistic data, but if you can abstract this enough, you can kind of see how this could apply to you, whoever you are. And that puts a lot of burden on the learner to kind of build the right mental bridge between what they’re looking at with the difference between four and six cylinder cars and their miles per gallon. And the genomic data they work with day to day, right? That’s a big leap to make, but we ask them to make it.
And so I’ve always had this, man, I wish we could be better at delivering meaningful, engaging demos and walkthroughs and content, but it takes a lot of time to build that stuff. And in solutions engineering that was one of our primary things was building content for demos and blog posts. And what would happen is we build something, we invest a lot of time, a lot of cycles into this thing. And then that would become the new thing that we reused over and over again. For a long time, I think this still exists. We had this bike share example that used open bike share data from Washington, DC. And so we had a Shiny app built around it. We had a machine learning model that was trained to predict the availability of bikes at certain stops. And then all of a sudden, every customer got to hear about Washington, DC bike shares because it was the example to show, but they weren’t a bike share company. So they still had to make the mental bridge between what is that and what is the data they actually work with.
So a few months ago, I was having a discussion with somebody internally and I was like, I feel like the tools and the things are there for us to be better at this, to be able to build targeted content more efficiently than what we used to do to the point where maybe we could start being a lot more targeted with the conversations that we have and the data that we show and all these pieces. So I talked to my boss about it, said, here’s this, I’d love for us to do this. And his response was something along the lines of, we’ve tried that. And yes, that’s a great idea. But how do you find the data? How do you find industry relevant data? Kaggle, right? There’s certain sources that people typically will turn to, open GitHub, whatever. You can find data, but I know that game and I know that I’ll be like, yeah, let me go find data. And six hours later, I’m like, there’s no good data here. It’s really hard to find good open data.
And so I was on the road at the time. It was actually in San Francisco, but I remember I was in the hotel room that night. I was like, okay, how do we find the data? And I had a, I think I had Positron Assistant up and running and I was like, hey, Positron Assistant, can you make me some data for this industry? And then it did it. And the key thing about this was it didn’t just spit out a bunch of data. It wrote a bunch of code to generate the synthetic data. So now all of a sudden, I’m not bound by what the LLM can regurgitate. I’m letting it write some code that writes some synthetic data that can actually be surprisingly high quality. It’s synthetic. And there’s no getting around that. And I’m actually, I think that’s fine. I think it’s fine to be like, this is some made up data, but it’s really cool to say, here’s some made up data that is really potentially indicative or familiar to you versus here’s some information about bike share usage in Washington, DC. And so there’s this shift where I was like, oh my gosh, we can use AI to help us make data up that can be really illustrative and really powerful.
And so then we built out this sort of whole internal tool chain that basically says, hey, help me make some data. And then on top of that data, help me write a Quarto document and a Shiny app and build an ML pipeline. And that way I can show some of these things off with data that’s so much closer to the reality for the person I’m talking to versus some of these more static assets that we’ve used in the past. So I’ve been really excited about that. I’ve had a ton of fun kind of exploring and building that out. We’ve leaned heavily on Claude Code to kind of help us build out what we now have today.
In fact, we just had an update to this tool yesterday or two days ago where Anthropic just announced agent skills as a new thing that you can manage inside of Claude Code. So this is now available as an agent skill that you can just enable inside of Claude Code. And then you can be like, hey, write me a demo for this industry. And it goes and makes a bunch of data and then writes a Shiny app or a Streamlit app or whatever you want. And then you’ve got this fully encapsulated little project that you can show off and it takes 30 minutes of Claude just kind of doing its thing to get there. It’s phenomenal. Today I can talk about this, of course this happens. But when I first started building this out, it was repeatedly just me with my jaw on the floor being like, I have no idea that we’re at this point with AI where this kind of thing can happen so quickly because I know that if I were to try to do this myself, it would be like two weeks of work to get all this together the right way. And now it’s an afternoon of me telling Claude what to do while I go eat lunch and I come back and I have something really cool.
Michael Chow: Yeah, I love that you went from it sounds like from trying to find the one perfect example to creating the example generating process that can just fire up great examples kind of on the fly.
James Blair: The thing that’s been the best about this, in my opinion, is there’s kind of this pattern that’s emerged where we’ve got Positron Assistant that we’ve put out there. We also have DataBot as this sort of exploratory data agent that I love, I think, is a fabulously designed, well-built tool. And so one of the things that’s been really fun is to kind of join these conversations with users who are coming from all kinds of backgrounds. They have their domain expertise. And we say, hey, here’s some synthetic data that is perhaps illustrative of the kind of data you work with day to day. And here’s what it contains. Again, it’s all made up, but this can help us have a good conversation. And then we pull open DataBot and we’re like, hey, let’s explore this data together. And I’m sharing my screen or showing them my laptop and I’m explaining this data. And I give it a couple of prompts so we look at some things.
And then it’s been so fun to ask, what would you ask this data? What would you want to know? And then they respond with, oh, I often look at this, or I’m really interested in, I’d be interested in understanding how these two things compare. And then you let them participate in the process. And all of a sudden, this goes back to what I talked about earlier, like my level of teaching, right? You can just see the light bulb go on where they’re like, oh my God, this is what my day to day could be like. I could just be chatting with this thing that’s helping me explore the data. And then it gives me the cognitive freedom to think more critically about the types of questions I should be asking here. Instead of me worrying too much about writing the code and remembering the exact syntax for which ggplot geom I want to invoke here, I instead can think a little bit more deeply about how I can apply my understanding of this domain to the particular data that I’m working with right now.
Michael Chow: It’s interesting to hear, it’s like you have an entourage, if I understand, of bots. You’ve got this thing, which I think is called DemoBot, if I’m allowed to say, your internal tool. Then you’ve got DataBot, which is the Positron data analysis assistant. You’ve got a bot posse kind of in tow for different parts of the process. I wonder if it’d be interesting to talk a little bit about what tools you’re using day to day to interact with AI. So it sounds like you’re using Claude Code a bit for your demo creation. Maybe you could give us a little bit of context on which tools are you using? And then we can go around and maybe talk a bit about which bots, which tools we’re kind of working most with right now.
James Blair: I have an interesting, maybe it’s not interesting, but my philosophy there is, I think there’s high value in exposing myself to a lot of tools. So I have lots of things that I kind of rotate and cycle through. So I really enjoy Claude Code. I like to use that as kind of my quote unquote daily driver for sort of agentic style work. And then their ecosystem, Anthropic continues to do all kinds of really interesting things with what Claude Code is capable of. And so there’s all kinds of interesting community contributions that I’m constantly kind of tinkering with and exploring.
I use Codex on the OpenAI side as just, hey, how does this compare? What does this feel like in comparison to some of the other tools I know?
I’m a big fan. Wes pointed this or mentioned this in an internal Slack thing a while ago, but there’s a tool called Conductor that I really like that is essentially a wrapper around Claude Code, but it is really well-designed in terms of it lets you just run multiple Claude Code sessions against different work trees for a given project. So you have these independent agents running on their own clean copy of your code and they can all be running on totally different tasks without the risk of interfering with one another. And then once something has reached a point where you want to kind of absorb those changes back to everything else, you can submit a PR and the mechanics of it is really nice for a multi-agent sort of workflow. I’ve found that to be a really nice tool to use and one that I kind of continually go back to.
Michael Chow: Oh, wow. That’s a wild one. Yeah.
James Blair: Wes, what are you using these days?
Wes McKinney: I mean, like James, for the last maybe six or seven months, I’ve been solidly using Claude Code as my quote unquote daily driver for doing most of mundane work, building bespoke tools. I feel like there’s this fun analogy between 3D printing, like building an exact widget that solves a problem that you need and using a coding agent to build the exact script or specialty tool that solves the problem in exactly the way that you want or automating. So I’ve been going through and building all kinds of stuff and building bespoke tools to automate my day-to-day work, because I think in my open source work, I abhor drudgery and inefficiencies. And so now having the ability to use agentic coding tools like Claude Code to build, essentially to dive into types of development that I would never have touched before.
I’ve built recently, I’ve built terminal UIs. I’ve done lots of work on CSS. I’m improving all of my websites, making CSS improvements, design improvements that I never would have touched before. I would have needed to consult a designer or somebody because I’m just not good at that type of work. So I’ve been treating AI really as a tool for leverage and for delegating work that I find to be low value for me and to help free me up to spend more of my time thinking about some of the higher level concerns.
But the big learning I have from the last six or seven months is that these tools really only work as well as you can articulate yourself and know what to ask for. And so if you aren’t articulating clearly and in great detail what you need the agent to create for you, and then if you aren’t looking at the code that it’s generating and scrutinizing it, and actually giving good quality feedback about what it’s doing wrong or what it’s not doing, it can really create a gigantic mess. And that can be a huge liability. And so what I’ve been telling everyone the last six months or so is that there’s going to be this increased urgency for everyone to be much better at code literacy and reading code and to spend more time doing analysis of code that they didn’t write and to be able to read and give feedback on code because we’re all going to be writing a great deal less code than we did in the past. But I should broaden my horizons and learn more tools. But as long as I’m productive using Claude Code, it’s been sufficient for my needs in recent times.
Michael Chow: Yeah, it’s super interesting. I definitely think on the theme of picking up new tools, that’s something I struggle with. And I have to admit, I really want to get into Claude Code, but haven’t yet. And I’m so ashamed to say that out loud.
Wes McKinney: I mean, you’re missing out. That’s all I can say.
Michael Chow: Do you have tips? How would you suggest someone start, dip their toe into Claude Code?
Wes McKinney: Try building something totally greenfield. Look at your, it could even be something that’s not work related. Look at your personal life. Really anything just, to sit down and build something, especially something that you would never have built yourself. I’ve heard of people building Android applications, people with zero Android application development experience. And the ability to break through and tackle problems in spaces where you have little experience, or maybe not little experience, but maybe little interest in developing expertise. I think that’s really interesting.
In building production code for enterprise software, that also comes with a downside that if you step outside of your lane where you’re unable to give good quality code review and feedback on code, maybe I would be less enthusiastic. I also find it’s as a tool to do refactoring and augmentations of existing unit test suites. You can say, hey, look for code duplication in this file, create some helper functions, refactor things, make things clean, normalize doc strings. The sky’s the limit. And it’s all limited by your ability to identify things that you want to do and ask the right questions. Yeah.
Michael Chow: Yeah. It’s interesting to hear too, the Android app, like the difference between what you could imagine yourself doing. And if Claude Code could build an Android app, it would really freak me out because there’s no world where I see myself doing that. So maybe that’s like really freaking myself out with Claude Code. Maybe it’s a good, yeah.
James Blair: I think it’s fascinating. I think the greenfield example is a really good way to dip your toes in the water because it’s low stakes. And honestly, it’s just a matter of describing as clearly as you can, what you’re trying to build to the model. And then learn, there’s all kinds of, there’s roughly 8 billion blog posts about how to use these tools correctly and some are valuable and some aren’t. But I think that the biggest thing going back to an earlier point is to just put hands on and experiment, find something that’s of interest.
An example from my own experience has been, I’m not a web developer by any stretch of the imagination. And it’s something I could certainly spend the time to get better at and get good at, but I’ve had a few different websites that I’ve built out that I’ve just used Claude Code and kind of vibed with. And to Wes’s point in the process of doing that, I feel like my ability to kind of understand TypeScript and CSS and some of these other things has elevated as I’m reviewing. Do I understand everything that’s written here? No. Could I reproduce it myself? No, but I know more now than I did when I started this project and continue to iterate.
I think there’s this intentional effort to say, I’m not just going to tune out and let this thing do the work for me. I’m going to stay an engaged participant. And what that looks like is kind of what Wes described. My role is now as a reviewer and as a tester and as somebody who needs to communicate back to this model, what is not right or what, you know. And it’s been really, again, just that greenfield example, I think you can end up in a place where you find where these things are really great. And there’s areas where the capabilities of these tools are phenomenal. And then you’ll find yourself in something that feels so similar and all of a sudden the model is so bad at it. And you’re like, but you were so good at this other, and you just realize that there’s this very jagged abilities curve to what these things can do. And so just through experience, you start to kind of understand, okay, here’s where these tools struggle and I have to kind of scaffold them this way to make it work right. And then these other areas, they’re a little bit better to just do their thing. They’re pretty strong in those capabilities there.
Hadley Wickham: I think my advice would be different to getting started. I would say instead, find an existing project where there’s something you’ve always wanted to do for a long time, but you keep putting off because every time you look at it, you’re like, oh, this is going to be way too hard. And just throw Claude Code at it. And there’s maybe a 20% chance it does the right thing or answers it well. But even if it doesn’t do a good job, just seeing someone tackle this problem that in your head is really complicated and seeing that maybe it’s got 80% of the way there with way less effort or it’s done something obviously wrong. And then you want to step in and correct it. It’s just such a great way to get unblocked when there’s some task that you’re just kind of stuck on psychologically. So that’s to me where I found it the most valuable. Certainly I use it for all this stuff and lots of routine changes and it’s great for refactoring or when you’re trying to do basically the same thing in a bunch of different places, but that kind of unlocking I found to be super, duper valuable.
Michael Chow: Do you think, are there any nice aspects of the Tidyverse that are thanks to AI taking something that you dreaded doing or felt stuck on and blasting it?
Hadley Wickham: Not the Tidyverse here, but I did a bunch of work on TS that there were issues that had been sitting in TS for like 10 years, not necessarily because I thought they were so incredibly complicated, but every time I’d look at them, I’d just be like, this is going to be really hard and the payoff’s not that big. So I just kick the can down the road. And I keep doing that for multiple years. And so just having it have a go at some of those things. And my experience was maybe a third of the time it would come up with a, I was like, oh, this is a really easy solution. And it looks great. But a third of those times I’ve later discovered that its solution was actually too simplistic and it didn’t really fix it correctly, but it’s still kind of got me over the hump. And then I had to do more work, but at least it got me unblocked. And then other times it’s just, sometimes having an obviously wrong answer in front of you is more compelling than having no answer at all, because you’re like, okay, this is wrong and then fix this.
Michael Chow: Oh, I was going to ask, are you using Claude Code? What kind of things are you—
Hadley Wickham: Yeah. Claude Code. Yeah.
Michael Chow: Nice. Yeah. Okay. I’m the odd one out.
Wes McKinney: I think the thing that’s great about Claude Code is you’re just doing it in a terminal. It doesn’t require any special integration with your editor. I also sort of wonder, Claude Code has this very, I don’t know, kind of 1990s user interface. And I think there’s something about that nostalgia that is comforting to engineers who are a little bit worried about AI taking over their jobs.
Michael Chow: It’s interesting. I would let AI take over my job if it felt like Neuromancer, you know, maybe that’s the key.
Wes McKinney: Yeah. The problem is right now, the reality seems to be more like, I’ll take over your job by producing a ton of stuff, but then you have to carefully go through it line by line.
Michael Chow: Yeah, that’s fair. Giving me a new job.
Wes McKinney: I also wonder, I remember years and years ago, I’ve always, I think like many programmers, really been into keyboard shortcuts instead of using the mouse. And there’s this kind of scorn for, why would you use the mouse? It’s so much slower. But I remember reading an article that actually keyboard shortcuts aren’t actually that much faster. They just cognitively feel faster. And it does feel a little bit like this with many of these AI tools. It feels like you’re moving much faster, but are you actually moving that much faster? Or there’s certainly your speed is much greater, but is your velocity greater? I don’t know, but absolutely we, I believe every data scientist, every software engineer needs to be engaging with these tools and trying them out and learning. So much of what you read is just hype or abject skepticism. You have to try it out yourself and learn.
James Blair: I think too, from a product management perspective, it’s been really interesting because there’s a lot of discourse right now in the product management space of what does this mean for the future of product management? Do we see this convergence of engineering and product work where product people are the builders and vice versa kind of thing. And who knows how that’s all going to shake out and materialize. I do know that it’s made certain things like prototyping and putting proof of concepts together a lot more accessible to the product side, which is really nice.
And then the other thing that I’ve seen in practice that I’ve started kind of adopting is every now and again, I’ll find rough edges from customer conversations or whatever it is. I find something that I’m like, this should, it would be nice for this to be changed, or it would be nice to have this feature. And if it’s approachable enough that I feel confident in my ability to define the problem. And if it’s in a part of the ecosystem that I feel like I can operate safely within, I’ll take a first pass at it with myself and Claude Code and build out something and then make sure that I have a full understanding of what I’ve built and have gone through the code review process myself, and then pass it back to whoever the maintainer or whoever the owner is from an engineering standpoint and say, hey, here’s this issue that I filed. I’ve also created this pull request in response to it. I understand what’s going on. I’ve tested this, but happy to chat about it, but it kind of helps me meet the engineers where they are.
What I’ve had to do though, very specifically is avoid the kind of natural inclination to just vibe something out and then toss it into a PR and be like, hope this works because then you’re just shifting cognitive load to somebody else, right? This shouldn’t be convenient for me and a burden for someone else. It should feel like it was a convenience for both of us or however many people are involved. It should feel like we all won versus like, hey, I got a shortcut, but my shortcut was actually making you do more work. I don’t think that’s the right way to use this. And so I’ve been really conscious of saying, if I’m going to put my sort of engineer hat on for a minute and use some of these tools to try to pull a solution together for a problem that I feel passionate about, then I want to make sure that I can defend what I’ve submitted and the solution that I’ve proposed because I understand it. And that is the kind of critical piece for me. If somebody, if an engineer wants a vibed solution to their problem, it’s really easy to just wire up Claude Code through GitHub actions and tag it to go solve an issue and give you a draft. They can self-select into that sort of feedback. What I can bring is, as a human, I’ve reviewed this, I can talk about it and we can reason together about if this is good enough or if we need to do more.
Michael Chow: Yeah. Keeping the human in the loop a little bit, it sounds like. I want to really quick, I know we’re running up on time, but I was hoping really fast just to go around. Because I know Claude skills are relatively new and there’s been a lot of chat about Claude skills. Maybe we could just go around and for 30 seconds a person just quick, get people’s impressions of Claude skills. Because I know there’s been a lot of internal discussion and things like that. Wes, what do you think?
Wes McKinney: We’ve been using, I mean, we’ve been using just prompts, markdown documents in the dot Claude folder to teach Claude Code how to do stuff, develop and workflows and help with different specific areas of Positron. And so I think what’s cool about Claude skills is it’s a more structured approach to something that we were solving in a more ad hoc way. And so I see it as something as Anthropic responding to something that’s a genuine need, especially in more complex projects where your project has different ways of working and there’s complexities involved with development workflows and how you also interact with the project, how you create issues, things like issues in GitHub and things like that. So I haven’t used it, but there’s already a pull request up in Positron to port some of our existing Claude Code machinery to use the new Claude skills stuff. So I will be looking at that and getting more familiar with it since I haven’t used it myself yet since it’s only a few days old.
Michael Chow: Yeah. It sounds like it’s kind of a natural interface across this pattern of having markdown files to explain things. Hadley, what do you think?
Hadley Wickham: Yeah, I haven’t used it yet either. I’m excited about just adding a bunch of skills, the kind of bits of, particularly kind of recent changes to the way we develop packages and stuff. So for example, recently I was using Claude Code to generate some tests and I wanted to use mocking and I had no idea how to do mocking with testthat because it’s a relatively new feature. So that to me, the potential of skills to kind of be like, okay, you need to use this. You don’t know about this. Here’s a quick briefer on how you use mocking with testthat. Here’s how you use snapshot tests. Here’s how you use S7. Just so if I’m going to use really new features, Claude knows how to use them.
Michael Chow: Oh, awesome. See James.
James Blair: It’s rare that I feel like I’m ahead of the curve because I have used skills and I think there’s a lot of upside here related to a couple of things. One is portability, right? Skills can be really portable and shared really easily. And I know that the dot Claude folder is part of that, but also with combined with Anthropic’s new plugin interface within Claude, you can easily just install new plugins and that can include new skills.
And then the other thing, the thing that I think is probably most significant from my perspective is that skills operate in this way that Anthropic describes as progressive disclosure, where when Claude starts up, it only familiarizes itself at the highest possible level with the description of what each skill does. And only when it determines a skill needs to be invoked, does it load the rest of the information from that skill into context. So you can have this huge catalog of skills that takes up a very small fraction of your context. So your context stays really clear. And then you ask some request and Claude says, oh, there’s this skill that I should use for this thing. And then it invokes that skill.
And then even there, there’s continual progressive disclosure where it will read the markdown file or files associated with that skill. And if certain facts, like let me give you an example. In DemoBot, you could choose to use either Python or R for a demo project. And there’s instructions for each one, how to build a Python project and how to build an R project. So if you ask Claude Code, hey, build me a project and use R, it won’t even read the Python documentation. It knows that that is not part of what it needs to know and understand for this particular task. So it keeps that out of context and just reads the bits that are relevant to the specific ask. So I think that part of it, the portability, and then this ability to define lots of skills without clouding context is going to be a big deal.
Michael Chow: And I guess sort of to wind down, any last advice for people getting into data science today you’d give?
James Blair: I think it’s just an echo of kind of what I’ve already said. The fact that, this sounds maybe stupid, but the fact that you’re human is your greatest strength. The humanness that you bring to data science is the value. The value that we produced was never the code that we could write in my opinion. It was the questions that we asked and the way that we thought about data.
I look at my background and history and everything. And one of the great gifts that I think data science has given me in the pursuit of this field as a career was the ability to just reason about and think about data. And that has nothing to do with programming necessarily. It’s just, how do I map data mentally to think about what questions can I ask? How do I structure those questions? And then eventually that translates itself to code. But to Wes’s point, when we were talking about AI, your ability to clearly articulate what you’re trying to accomplish to the model is really critical. And that has very little to do with code. It’s just communication. It’s about how well do I understand this data and using my curiosity and the things that make me a human, my critical thinking capabilities, all these things. I can then describe what I’m trying to understand and let a model kind of write the code for me because the code that’s written was never really the point. It’s the outcome that we’re driving to, the understanding that we’re trying to get to. What is this collection of information telling me and what do I do as a result? That’s what we’re trying to get to. And we have to stay human to do that the right way.
So that’s my advice. Remain curious, stay human, use these tools to help you be better at what you do best, be better at being curious, be better about asking the hard questions, be better at critical thinking, because now we have an opportunity, we have these tools that kind of take care of some of the other stuff for us. Don’t use that as an escape valve to be lazy, use that as an opportunity to be better at the things you’re uniquely suited to do.
Michael Chow: Yeah, James, really appreciate it. I think hearing about your path from robotics to journalism to data, and especially with things like product management and that whole sort of path is really interesting to hear. And I think when you wrap it all up with all your tinkering and your work with AI, it’s interesting to hear how all of it’s kind of come together and that you’re very much the human in the loop, doing a lot of this interesting data work. So really appreciate you coming on to The Test Set and excited to see all the stuff you do.
James Blair: Thanks for having me.
Hadley Wickham: Thanks, James.
Wes McKinney: Thank you.
[Podcast outro]
The Test Set is a production of Posit PBC, an open-source and enterprise tooling data science software company. This episode was produced in collaboration with Creative Studio AGI. For more episodes, visit thetestset.co or find us on your favorite podcast platform.