Mine Cetinkaya-Rundel: Teaching in the AI era and keeping students engaged
This transcript and summary were AI-generated and may contain errors.
Summary
In this episode of The Test Set, Michael Chow, Hadley Wickham, and I chat with Mine Cetinkaya-Rundel, a professor of statistics at Duke University who also works part-time at Posit as a developer educator. Mine shares her path from actuarial science to data science education, including discovering her passion for reproducibility before she even knew the term existed.
The conversation centers on teaching data science in the era of LLMs. Mine describes a challenge many instructors face: students submitting AI-generated code that human TAs then grade, with feedback that nobody reads. Her approach is to use LLMs to provide immediate, formative feedback on low-stakes assignments, potentially freeing human TAs for higher-value interactions like office hours and problem-solving sessions. The goal is to reduce grade anxiety while creating space for genuine learning and practice.
We discuss the muscle memory question: will students who rely heavily on AI tools develop the intuition needed to recognize when results look wrong? Mine notes that even for experienced programmers, AI autocomplete can be disruptive when working in an unfamiliar language because you lack the context to quickly evaluate suggestions. Hadley draws an analogy to learning musical instruments or Japanese hand-tool woodworking, suggesting there may always be intrinsic value in hands-on craft even when automation is available. The episode closes with reflections on why data science matters, from personal decision-making with uncertainty to living in a world where rules and policies are data-driven.
Key Quotes
“I changed all of our reports to be actually Excel files that when you print them out look like Word documents, but the cells were automatically calculated. I think that’s sort of what I later found out was called reproducibility, but I just didn’t know. I was just like, I don’t trust this process.” - Mine Cetinkaya-Rundel
“The idea was, could we use LLMs to give immediate feedback? Because I already write very detailed rubrics… giving you some safe space to practice by yourself, and then we have the LLM reporting to you, giving you some feedback. So not grading, but giving feedback.” - Mine Cetinkaya-Rundel
“What happens is then we have human teaching assistants who give feedback to machine-generated responses that then no one reads, really, on average… because they never wrote the code in the first place.” - Mine Cetinkaya-Rundel
“I think some of it is about reframing it to be more like learning to play an instrument. Lots of people learn the guitar or piano, and 100%, no matter what song you want, you can download a professional recording of it that’s going to be way better than you. But there’s still some intrinsic joy in learning an instrument.” - Hadley Wickham
“For our stats students, we recommend… try to be really good at one language and make sure that that’s not the only language that you are at all comfortable with.” - Mine Cetinkaya-Rundel
“I think about it as people who rap in multiple languages. All of a sudden you have more words you can rhyme… like you’ve expanded your options and it both sounds cool and also gives you options.” - Mine Cetinkaya-Rundel
“I predict that we’ll probably see like, you know, probably some success stories, but also some horror stories of how somebody pushed a button or made a business decision. And it was based on kind of vibe coded slop.” - Wes McKinney
“At a high level, I would like to live in a world where if there’s going to be rules and policies, those are data driven. And the folks who are making them, but also the folks who they apply to have a sense of how we arrived at those decisions.” - Mine Cetinkaya-Rundel
“Making sense of uncertainty… I think that’s something we need to get people to understand as well, because at very critical times of your life, that might be the sort of statistics someone tells you and you need to make a medical decision or something.” - Mine Cetinkaya-Rundel
Transcript
[Podcast intro]
Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning, digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field. On this episode, we chat with Mine Cetinkaya-Rundel, professor of the practice of statistical science at Duke University and a professional educator at RStudio.
Michael Chow: I’m Michael Chow. Thanks for joining us. All right. Hey, everyone out there. Welcome to The Test Set. I’m Michael Chow, and I’m joined by Hadley Wickham and Wes McKinney, and we’re here with our fantastic guest, Mine Cetinkaya-Rundel, who’s a professor at Duke and works on education as well at Posit. Mine, we’re so excited to have you, and thanks for coming in.
Mine Cetinkaya-Rundel: Yeah, thanks for having me.
Michael Chow: Yeah. I wanted to ask, just to start out, could you just tell us how you arrived in your roles?
Mine Cetinkaya-Rundel: Yeah. I wish I had a very exciting story. I don’t, but I had an undergraduate degree from a business school, honestly, because my parents wanted it, and I didn’t really know what to do in business school, so I thought I’ll do the most mathematical thing one can do, so I was an actuarial science major, and I worked as an actuary for two years in New York, and at the end of that, I decided, I’m not loving this. What can I do for free? Yeah. PhD is a thing. If you can get into it, you can do for free, and I also really enjoyed teaching when I was an undergrad. I used to do a lot of tutoring. That’s how I paid a lot of bills, so I thought this could be an interesting thing to try, and just looking at my background and also a little bit of the work that I was doing when I was an actuary, I really enjoyed the pieces of the work where we did things with data, which I hadn’t really done a lot while I was an undergrad. I thought I’ll apply to some stats programs, and so that’s how I decided to go to a stats PhD, and fell in love with teaching even more, and also really realized that I wrote my first line of code when I was in graduate school, I think, which is late, but whatever. I really enjoyed that as well, and then decided to stay in academia, and at some point, I was doing lots of sort of public-facing things as well, and using a lot of the tools that are developed by then RStudio, now Posit, and so I work part-time at Posit as well as a developer educator.
Michael Chow: Oh, awesome. And this actually reminds me, I don’t know a lot about actuaries, but you mentioned writing your first line of code in grad school. Our actuaries aren’t slinging a lot of code, is that?
Mine Cetinkaya-Rundel: Yeah, I mean, actually, that’s not 100% fair. There was one piece of software we used, I believe it was called Ginsu, like a knife, it was for like cleaning data, so one of the—
Michael Chow: You’re like a chef.
Mine Cetinkaya-Rundel: Yes, exactly. One of the clients we had was the United Nations, so we did the retirement plan for United Nations. The data came in literal boxes of CDs, so we had to load them up, and you had to check for things like did the, and each row of your data set is like an employee, did the employment date change, did the birth date change from last year’s data to now, because those things shouldn’t change, so there was a lot of like data quality checking and stuff, and I remember using that software, I don’t remember anything about it, and I believe it was an in-house software, to be honest, but what I do remember is that you would write some, or someone else previously had written these like tests, did the birth date change, and it would print out on a dot matrix printer, so I would go through the paper, like looking at the results, and then go back to the client and ask, actually, can we confirm these lines, so I really enjoyed that piece of the work. There was the other piece of the work, which was more like regulatory, like you have to do these filings, sort of like an accountant, which was just like doable, but wasn’t the as enjoyable part, but, so I did sort of work with code others had written, but didn’t really write code myself as an actuary.
Michael Chow: I love that you’re, though, it’s very like farm to table, you’re like actually in the data physically, like striking things out with a pen.
Mine Cetinkaya-Rundel: Yes, and I thought that was really interesting, and then another thing I really enjoyed doing was, there’s like all this analysis that you do, and then all of it needs to go into a report, and oftentimes, those reports were Word documents, so someone would take a number from this printout and put it into the Word document, and that really bugged me, so I had changed all of our reports to be actually Excel files, that when you print them out, they look like Word documents, but the cells were automatically calculated. I think that’s sort of what I later found out was called reproducibility, but I just didn’t know, I was just like, I don’t trust this process.
Michael Chow: I do like you’re like, there’s all these little hints, you’re like, maybe I’m a data person, you’re like, this spreadsheet should maybe be automated, you’re like sleuthing around.
Mine Cetinkaya-Rundel: Yeah, yeah, because I was an actuary because my mom had read a US news thing that said actuary is the best job you can have, like that was the reason why I was an actuary.
Wes McKinney: It’s funny, like I actually also had a dalliance with actuarial science, like I had an internship as an actuarial science firm, also was in the business of certifying pension plans, which I didn’t know a lot about, but it’s kind of a dying field now that defined benefit pension plans have moved to 401ks, and people are saving for their own retirement as opposed to being supplied with a pension, so I saw some of the same issues around like, can’t this be automated, like this seems really error-prone, and I thought that I would be doing more math and science, but there was so much like data entry and like filling out tax forms and double-checking numbers, and really, yeah, it felt, I was like, by the end, like I hadn’t written very much code at the time, and so it was like, can we write code to do this? But I think they, you know, actuarial firms, like they also are in the business of staying in business, and so if you automate too much of the work, then there’s less work to do, and so it was almost a counter-incentive to make things too efficient.
Michael Chow: I love that being actuaries agitated both of you into data science.
Mine Cetinkaya-Rundel: I don’t know if there, there might have been like a mild causal relationship, but I did take like two of the actuarial exams, and so it’s kind of a weird corner of my history, but yeah, I decided that actuarial science was not for me after that, a little too, yeah, a little too dull for my tastes.
Michael Chow: Hadley, this is your chance to have almost become an actuary. Okay, you weren’t ginsuing or striking things.
Hadley Wickham: I almost, I almost became an accountant. I don’t know if that, maybe that’s a totally different field, but similar reasons.
Mine Cetinkaya-Rundel: Similar, similar, I think, yeah, yeah, but you have to take these exams as an actuary, and they say, we’ll give you time off to take the exams, and then you end up making up that time by staying over time to do the work anyway, so that was another motivation for me being like, if I’m going to study hard, maybe I should study for a more generalizable skill, as opposed, so you can just, if you go down that path, you can just be a higher level actuary, like there’s no necessarily other path, or at least it wasn’t obvious to me. I’m sure there are other paths, but it wasn’t obvious to me, and you would probably stay in the financial sector, which wasn’t the most thrilling for me.
Michael Chow: Yeah, that’s fair. And I could see maybe it’s changed throughout the years, and like shifted. I’m really curious about too, so you joined, so you’re at Duke, and you also do work at Posit. How did you get involved with Posit stuff?
Mine Cetinkaya-Rundel: I took a year leave to sort of try out industry work, and there are parts of that work that I really, really enjoyed, like working on open source packages, thinking about how do we communicate about them, I really like writing documentation, that sort of stuff. I really enjoyed that piece, but I really missed classroom teaching during that year, and I also realized that some of the things that used to motivate me to open issues and the repos of these packages was always observing students getting confused or making a mistake, and then me realizing, actually, that’s totally the fault of how this is written. This could be more clear, as opposed to just like a complete misconception by the student, and I realized that I, not being in the classroom with students, I had lost that. Now, of course, you can do workshops and stuff, which is something I still do and did during that year as well, but there’s so sort of like short-term engagements, you don’t end up seeing how someone’s, where they start and get better, and to me, that’s the joyous part of teaching, like seeing students in their first year to their fourth year and how they develop, so I missed that part of it, and that’s why I tried to negotiate, well, can I do a little bit of both, so that’s how I settled into what I do now.
Michael Chow: Oh, neat. And you’ve been at Posit for almost eight years, is it?
Mine Cetinkaya-Rundel: Yeah.
Michael Chow: So I guess the negotiation succeeded, because Posit really loves having you.
Mine Cetinkaya-Rundel: Yeah, well, thanks.
Michael Chow: Yeah, I’m curious, I feel like this has come up before, too, this like making contact with students. I know we talked a bit with Roger about teaching and students. Yeah, Hadley, I’m curious about your, what you think of that, this kind of dynamic of like students, you mentioned like students kind of like keeping you grounded or like getting you in touch with kind of these realities.
Hadley Wickham: Yeah, I really enjoyed that at Rice, and especially like when you’re teaching, like for me, like teaching this Stat 405, which is basically an introduction to data science, you know, you teach this, you get a fresh batch of students every year, so you kind of get to see like, is my teaching gotten better, has my tools gotten better, so they get a little bit further this year than last year. And I found that like, really, and just being really, like, if you pay attention, you like learn, what are the things that I just assume that everyone knows that are just things that I know, and that I have to like communicate to the students. And I thought, yeah, so and after I left Rice, I was definitely worried for a long time that I just kind of like drift away into like ivory tower land with nothing to like pull me back to reality.
Michael Chow: Yeah.
Hadley Wickham: But that doesn’t seem to have happened, and I’ve stopped worrying about it.
Michael Chow: Oh, nice. Mine, I know you were doing, so Mine, I know you are an author on the second edition of R for Data Science. Do you feel like part of teaching in the classroom influenced kind of what you brought into that book? Like how did that?
Mine Cetinkaya-Rundel: Yeah, totally. I think that we reordered some things. We also sort of like restructured the book to have this like whole game part, which was actually inspired by another book of Hadley and Jenny’s, the R Packages book I think has that as well. But this idea that like, you know, if you’re only going to get so much of it, which I think about it in an academic sense of like maybe you’re at an institution with a quarter, or maybe the course has other learning goals, and you’re not going to get to see like all the intricacies of the tooling that Tidyverse offers, for example, that can be a really good start. But also, I find that sort of getting to a complete story quickly can be really motivating for students. And then we go into the details and the rest of the book. So that was fun to write and sort of think about. The tooling had also evolved over time. So this, I think, also was an opportunity to sort of rethink what aspects do we want to highlight and what aspects maybe we don’t need to as much because some of the things that were like call-outs or warnings could actually just be removed because the tooling does the right thing. It already gives you the warnings anyway.
Hadley Wickham: Have you read the teaching book that the whole game idea came from?
Mine Cetinkaya-Rundel: Yes. Yeah.
Hadley Wickham: I found that like the—
Mine Cetinkaya-Rundel: It’s a baseball.
Hadley Wickham: Yeah. I was like, you know, you don’t teach, like when you teach kids to play baseball, you don’t have them like swinging, you know, you don’t teach them how to like swing the bat for like three weeks. And then you teach them how to catch the ball for three weeks. And then you like teach them how to run around for three weeks. Like you introduce like a simplified version. And then like that’s more fun and it like gets the… And that was like, that was like really influential, I think, for a lot of our like teaching materials recently. Like give people the experience of the whole thing, give them a quick first pass and then kind of loop back and go deeper and deeper each time.
Michael Chow: Yeah. It’s a really striking, I think, feature of the books where this was added. I noticed like, yeah, a bunch of the books now for different tools or different parts of the Tidyverse use the whole game.
Mine Cetinkaya-Rundel: Yeah.
Michael Chow: And it is, I almost feel like being able to see it called the whole game has been really helpful too to like just click in.
Mine Cetinkaya-Rundel: Yeah. And it’s difficult. I mean, it’s difficult in general to write books about open source software, especially software that is relatively new and is changing. You’re also writing a book that’s in the context of all of the other tools that it’s related to, which includes like how you install the software and other packages that you mention in the book.
Wes McKinney: And so like I saw this very acutely with Python for Data Analysis when it came out in 2012. Pandas was a pretty new package and it’s clearly, it’s evolved a lot in intervening 10 years. Like just how you get Python up and running and all the packages installed has changed a lot. So you definitely have to go back and revisit and redo all of that and whenever you do subsequent editions. But as Hadley mentioned, like I think a key thing is like not losing, you don’t want to lose sight of like the things that people struggle with when they’re learning how to use these tools. Because as a tool builder, it is easy to take things for granted and to like not realize that like this might be easy for you, but it’s not easy for, it may not be easy for everyone or intuitive for everyone. And so I think even when you’re writing, like one thing that I found is like the importance of how you, like the language that you use and how you describe things to be more, to not make assumptions, like not suggest that things are easier, that things are simple. And so I remember when I went through like Python for Data Analysis, second edition, I removed like every “just” and every, “this is easy,” like “this is simple,” because I’m like, well, this is obviously, no, it’s not, it may not be obvious, like it may not be just a matter of writing this line of code. And so you really have to put yourself in the shoes of, you know, you as a practitioner or tool builder, you’ve been, you know, doing this for, you know, years and years, but somebody who’s brand new, it’s, yeah, and so I think the student experience helps, you know, really helps with that.
Michael Chow: Yeah, it’s really cool. And it’s neat, all these books, like Python for Data Analysis and R for Data Science are both open, so you can just go to the repository and, I’m not sure if issues are enabled, but you, like anyone could.
Mine Cetinkaya-Rundel: Yeah. Yeah. People do.
Wes McKinney: Yeah. I mean, I think, you know, now versus, you know, when we were starting out, there wasn’t, you know, there wasn’t GitHub and there was open source, but not nearly, the open source development process wasn’t nearly so accessible. So, you know, I’d be curious, like, I imagine students are wanting to, you know, learn how to not just use open source, but be involved in open source and engage with like the tools and packages that they use.
Mine Cetinkaya-Rundel: Yeah. It’s something I demo in class as well, and not sort of like necessarily the whole, like, let’s clone this repo and rebuild the thing, but it’s like, you caught a typo, you can literally go on to GitHub and edit it. And like that makes a pull request. If I remember correctly, I did this in a talk. You like shared a lot.
Hadley Wickham: Hadley merged the request during the talk.
Mine Cetinkaya-Rundel: This was a talk sort of generally about contributing to open source software. Like that was a prearranged thing. But just to be able to see that that’s there and, I don’t know, we always try to make a point to say thanks when we merge the issues. And even if we don’t, we always try to say thanks for the feedback. I think that’s helpful for any sort of learner, student or not, to just be like, oh, I can be part of this and I can help make it better.
Hadley Wickham: I do think it’s, oh, yeah. Do I remember that you had like an advanced, more advanced course where students had to submit all their homeworks?
Mine Cetinkaya-Rundel: Yeah, as part of their assignment.
Hadley Wickham: Through GitHub?
Mine Cetinkaya-Rundel: They do. My intro data science course is like that. Every single assignment is a GitHub repository and they each get submitted as such. And from semester to semester, I go through like, what do we do with the artifacts? Like there were semesters, particularly during COVID, where I wrote more automated tests to check things. Now it’s more like human feedback or I don’t know, I’m working on a project where it’s like, could I give feedback, something. But ultimately, every single thing is a GitHub repository and the tooling is there to make that happen. Now, they’re not doing all GitHub things, like there’s no pull requests, branches, whatever. It’s just like one repo you push to it. But it is actually, the students even mentioned that it’s quite nice to be able to be a first year student, maybe applying for internships and have like Git and GitHub on your resume. And that’s like very fair. It’s not an exaggeration. They really do have the basic skills. It is nice.
Michael Chow: So you are kind of like playing the game, like the whole game as part of the course, like rather than talking about how they could do it, you’re almost just getting them in and doing it.
Mine Cetinkaya-Rundel: Yeah. Yeah. And let’s be 100% honest here. There’s self-preservation as well. The alternative is a course management system that I have to deal with. So I’m doing it for their learning, but also to protect my sanity a little bit.
Michael Chow: You mentioned building an LLM assistant to give feedback, which I think is maybe super intriguing. Could you say a little bit more about what that looked like?
Mine Cetinkaya-Rundel: Yeah. I think it’s one of those projects that I’ve been working on over this year that’s fueled by frustration, but trying to do something productive with it. And so I teach introductory data science to students who have not coded, or at least not coded in R. But nowadays, for a university student, it’s hard to say every single student has never coded before, because they may have had some exposure to programming in high school, for example, but rarely to working with data and code at the same time prior to this course. And a lot of the content is sort of what’s in R for Data Science, and also sort of modeling as well. So about two-thirds of the course is sort of data wrangling, visualization, exploratory data analysis, and the rest is sort of a little bit of modeling and inference. The tasks that we ask students to do are simple, particularly the first few weeks. But maybe as LLMs get better further into the semester, are things that many LLMs can generate somewhat reasonable answers for. And I feel like over the last couple of years, the quality of those answers have been sort of increasing, although when you look at a homework assignment that’s submitted, it really seems like someone with multiple personalities have written it, like from question one to question two, like very different styles. But none of them wrong, necessarily, just not ideal or doesn’t conform to the principles we teach in class.
Michael Chow: Could you give a quick example of something they might answer?
Mine Cetinkaya-Rundel: Yeah, so for example, take this data set, pivot it longer, so then you can then make a visualization, and also when you pivot it longer, maybe you need to sort of separate the columns into two to get the year and the month out, or something like that. Now asking this question at a very high level, and not giving interim steps, I think would make it harder to just get the exact answer, sort of like we’re looking for. But because these are students who are just learning, it doesn’t seem 100% fair to not scaffold it a little bit for them. And then soon as you scaffold it, I feel like that’s basically better prompting for an LLM. So I’ve done that work for it, so then the little pieces of code work pretty nicely. So I’ve been thinking about, and that’s been frustrating, because what happens is then we have human teaching assistants who give feedback to machine-generated responses that then no one reads, really, on average. This is all on average, not talking about every student, but because they never wrote the code in the first place. But I really don’t think saying don’t use AI tools is the right approach, either. One, I don’t think that’s preparing them for the right thing, and it’s unrealistic. So that doesn’t seem productive. Anyway, the idea was, could we use LLMs to give immediate feedback? Because I already write very detailed rubrics. So these 20 TAs that I have in this 300-person course can grade consistently anyway. So building it such that the LLM gives you feedback with the idea that maybe if the student knows that they can get immediate feedback for low-stakes assessments through an LLM, they will maybe attempt it themselves first. Because I don’t know who wants to be feeding these things to each other. You may want to for fun, but I think that I’m hoping that we can remove a little bit of the grade anxiety and then maybe bring in, let’s give you some safe space to practice by yourself, and then we have the LLM reporting to you, giving you some feedback. So not grading, but giving feedback. And then break things down so that some things happen collaboratively in class, where I don’t think that’s going to be the way they work through it. Or even if they do, I think if you’re giving some prompts, I don’t know, ChatGPT is giving you something, and you’re looking over it with a friend and analyzing, that’s still good learning, in my opinion. It’s more the copy-paste, just because it happened to run, is the thing that I’m trying to reduce. And then spending human TA time grading that, that’s just a waste. So the idea is that that freed up time can go into more high-touch things, like more office hours, maybe problem-solving sessions. These are things students always say they enjoy quite a lot. The TAs and myself get really tired of reading LLM-generated code, I think. Just because their job is defined as, to do a good job, you need to write good feedback. It’s really hard to motivate yourself to write good feedback if you don’t think a human has written that code in the first place.
Michael Chow: Yeah, you see that nothing’s going to learn from that. You could be fighting the battle of who could care less, I guess.
Mine Cetinkaya-Rundel: Yeah, in a way. And I’m choosing to continue to care, so this is the thing that I want. So I’ve been working on testing it, and building it, and turning it into an R package. So it’s not about students copy-pasting, but maybe they can highlight it when they’re in the IDE, push a button, and then we get the response back to them immediately. But I will class-test it in the fall.
Michael Chow: That seems great. Wait, just to double-check that I’ve got it. Are you saying, like it used to be they were getting grades back for the assignments, but now it’s shifted to be more feedback? It’s like a shift, I guess people sometimes call it summative versus formative. Now it’s like a formative assessment where they’re just being assessed for feedback, so they can learn really quickly.
Mine Cetinkaya-Rundel: One thing I haven’t yet figured out, but that seems like it’s figurable, audible, is some artifact that can be submitted to say, I have done this. And then some reflection, because as I said, I have some hypotheses about how this may be motivating. I don’t know if these are true. It would motivate me in that way, but I don’t know if that’s true. So collecting some data from the students, and also giving them some bounties, like if you catch feedback that’s wrong, submit it, because that can be fun. People love catching other people’s or machine’s mistakes. And students will do a lot for extra credit. If you make that extra credit, find a case where that line was wrong. And I think find a case where it’s wrong, and then articulate it. How would you make it better? That’s a paragraph you might write, but I think that’s intellectually quite high value work in my opinion.
Michael Chow: And are you developing the tool with Elmer? So there’s R library that it maintains.
Mine Cetinkaya-Rundel: Yes, so that’s the package that’s doing the sending the code to the model. And also the model we’re using is one that actually is built with the open source models out there, plus content from R for Data Science, and my course materials. Because the idea is that we don’t want students running out of tokens, although the idea of a cost is also intriguing to play with. Should you really get limitless feedback? Or should you have to maybe think sometimes as to I’m ready for feedback? I haven’t thought about that before. And to me, that’s a more cognitive thing, interesting to think about, not necessarily my expertise area. But this way, the model can be hosted at the university, so that we don’t want any sort of differentiation between students based on what they can afford to pay for.
Wes McKinney: I mean, one thing that I think about a lot lately, and I don’t know how much it has been on your mind, but I have perhaps old fashioned ideas about thinking that developing muscle memory around the really basic mundane details of data manipulation and using, for example, dplyr’s API or using pandas’ API. And so with LLMs in the mix now, I feel like a lot of students and new programmers just won’t get as much of that basic hands-on, doing really simple, mundane tasks. And so they won’t build up that base of muscle memory, where you see a data set, you need to do x, y, and z. And in the old days, before LLMs, you would have to learn those basic commands. And you would see the structure and say, OK, this sequence of transformations with dplyr, like this sequence of pandas operations. And again, maybe it’s an old fashioned feeling that somehow something will be lost, or maybe this will result in people being less effective. But if they always have an LLM available to write that code for them, maybe it doesn’t matter. So I don’t know if you have some way to predict longer term what effect that will have on the practice of data science.
Hadley Wickham: I do think some of it is about reframing it to be more like learning to play an instrument. Lots of people learn the guitar or piano, and 100%, no matter what song you want, you can download a professional recording of it that’s going to be way better than you. But there’s still some intrinsic joy in learning an instrument. And I think for most of us, there’s that intrinsic joy in programming too, even if you could automate it. The other thing I still think about is Japanese hand tool woodworking. Where people do this with hand tools, they’re like, I’m not going to use nails, I’m not going to use screws. Of course, it’s way less efficient, but there’s still that joy of doing it by hand.
Michael Chow: You mean like intricate joins?
Hadley Wickham: Yeah, rather than just nailing things together or using a power saw to cut things. You’re making beautifully intricate dovetails that just fit together. But it feels like some of that is happening to coding now. We’ve got these power tools where you can just saw through whatever. But with the AI-powered autocomplete, sometimes the LLM will predict what it sees you about to do, and then will essentially suggest the code that you’re going to write. And so you press tab and you move on. And maybe it’s people, sometimes it’s not what you want at all. And you say, sorry, ChatGPT, or sorry, GPT 4.0, you’re drunk.
Wes McKinney: Yeah.
Hadley Wickham: But I think too, with autocomplete, for example, to say that’s not what I want requires background knowledge and experience. That at least for the students I work with, they don’t necessarily have that.
Mine Cetinkaya-Rundel: And I see this myself as well. I would say I’m a pretty experienced programmer in R, and much less so in Python. And to me, I can’t use the autocomplete stuff when writing Python code. It just feels like some annoying person is interjecting into my thoughts too quickly. And I don’t have the experience to be able to just say, that’s not what I want. So I then run the code. And running the code and getting an error is more frustrating. Now I feel like I have to debug someone else’s idea, if that makes sense. And that’s a harder skill.
Hadley Wickham: It’s also like having a conversation where the other person is constantly finishing your sentence for you, which is just so irritating.
Mine Cetinkaya-Rundel: It is so irritating.
Michael Chow: It does remind me too of like, if you’re driving in a car, you can have a conversation really easily. It’s not very cognitively demanding. But if you’re driving in a storm, you might have trouble talking or traffic. And if the person’s in this car with you, they actually might regulate how they talk to you, because they know that. So they might actually talk to you in a different way. But I could see with LLM feedback, you just like, you mentioned Python, you’re just kind of getting hit with autocomplete. It maybe is not as sensitive to where you are and your need to maybe work through things.
Mine Cetinkaya-Rundel: Yeah, yeah, yeah. So I turn that off when I’m doing something that I’m not as comfortable with, versus when I can really immediately see if it’s the right thing. Sure, it saves me some time writing characters.
Wes McKinney: I think another thing, though, is if a lot of this stuff can be automated and you’re not in there looking at the data, you’re looking at the interim output, I am not sure how one then develops a sense about the data set so that when at a later point in your analysis, say you fit a model and you’re looking at the coefficients, you might say, that doesn’t seem right to me. And I still do lots of statistical data analysis where I do actually look at the coefficients, not did we predict right or wrong, but actually those numbers. And sometimes the magnitude is so much larger, and I don’t know that I would have a sense to be able to evaluate that if I hadn’t spent hours and hours looking at interim output. And maybe I could be disciplined enough to, there are these packages even, and I guess with LLMs you can do it more too, like give a data set, let me give you some preliminary summaries. I don’t tend to use them because I’m like, I want to go through that at my own speed to get to know the data, you know? But it might be old school.
Michael Chow: It does remind me of your, I don’t know if this is a deep cut, but I noticed you did like a few episodes on YouTube pair programming with people for an hour, like students for an hour. And what you said reminded me of that, that actually even so much is nonverbal in data analysis, like where people put their eyes, what they look at, and even how they shift things around. And I’m curious how you, what did you take from those, do you have any thoughts on what people get out of that type of activity versus like LLM feedback? Or maybe just how that went.
Mine Cetinkaya-Rundel: Yeah, that activity was fun. I wish we could have kept it up, but video making is always such a like high activation energy thing that we didn’t. But the idea there was that we would take a data set, like one of the Tidy Tuesday data sets maybe, or maybe something else that the student was interested in. And we would have a conversation about like how to mostly visualize it or like ask some other questions. And in some of the episodes, I am the person coding and the student is saying, I wonder if, and then we talk through how we might do it. Or in other cases, they’re the one coding and then I’m the one sort of trying to generate ideas. What was nice about that, I think, and also the feedback that I sort of like got from students after chatting with them, like after the recording was that they’re like, it takes them a bit of time to like think about something and to be able to implement it. That’s they sometimes give up along the way.
Michael Chow: You mean like when they’re driving, it’s that.
Mine Cetinkaya-Rundel: Yeah, yeah. But but to see over and over that I also didn’t get there immediately was very helpful, which is something I do live coding in the classroom as well. It’s just I teach large classes, so it’s hard to get direction in that way in a large class. But when you’re one on one or with a small group, they can give you ideas and they see you like having to look at documentation or having to, you know, like run into errors and correct them. I imagine, for example, for folks who are more sort of better at using these AI tools for coding to be able to do that sort of thing would also be very instructive, like how does an expert programmer use these to their advantage as opposed to as an annoying voice that gets in the way?
Wes McKinney: I mean, it probably won’t probably won’t surprise anybody to know that there’s a whole, you know, kind of ecosystem developing of startups and companies building basically vibe data science. And so some version of vibe coding for for data science, so much so that some of the tools they’re generating code. But there’s no point even to show you the code by default because most people using the tool aren’t going to read it anyway. And so you’ll just look at the output and say, well, this seems wrong or, you know, could you double check that or like that doesn’t seem right. So you could give feedback to the to the agent that’s doing the that’s doing the coding for you. But a lot of the users aren’t actually going to read the code. And to me, it feels like a bit weird, like rather dangerous, like you may actually you get to the point of like actually making like a real life decision that impacts people. And you’ve got this whole, you know, vibe coded analysis that you haven’t read, haven’t read the code. And so I predict that we’ll probably see like, you know, probably some success stories, but also some horror stories of how, you know, somebody somebody pushed a button or made a business decision. And it was based on kind of vibe coded slop. So, yeah, yeah, I could see that.
Michael Chow: Geez. I was thinking maybe we could shift. I’m really interested because I know you, you do both R and Python conferences for some of your workshops. Is that right?
Mine Cetinkaya-Rundel: Yeah, since we have a lot of R and Python, both R and Python representation.
Michael Chow: Yeah, I’m super curious. You’re yeah, how you’ve experienced sort of R and Python conferences. What’s that been like to go kind of across the languages?
Mine Cetinkaya-Rundel: I think I’ve been to more R conferences than Python conferences, but last year I went to SciPy, which I really, really enjoyed. I think I learned like I was at the talks, even if it wasn’t my sort of like focus area, I really enjoyed it. It reminded me of useR from back in the day when it was like about the same size. Maybe I don’t know, 2018, something like that. I really enjoyed that. I enjoyed the workshops. I think they were like really thoughtfully designed, and I felt like I learned a lot. I’ve been to a couple of other Python conferences as well, where I wasn’t exactly sure where I fit. I could see what I was teaching in a tutorial could be useful for folks. So, for example, one of them was like take a Jupyter notebook and turn it into a website. Well, you can use Quarto to do that. Like that’s I think like a thing that’s a useful tool for a lot of folks coming from many different avenues, or turn it into a book or something like that. But in terms of the focus of applications, in some of the other conferences I’ve been to, I found less applicability. But that being said, I really I think the right thing to say is learning Python still, but in a very much data science context. And I think that when you go to a Python conference, that’s not the only context, even if it has the word data in it, I think. Versus I do think that most R conferences that at least I have been to have always had the data or stats sort of like ingrained in it. So it’s a little easier for me to see how I fit in in that ecosystem and what I can get out of it. Versus the other one is like nice for exposure, but a little harder for me to walk away with. These are things I can do. And I often try to measure the value, at least personally to me, of a conference with like did I walk away with some new things that I can actually use like tomorrow. So I will I think never forget the useR in, I think it was in Nashville, where I learned what R Markdown is. And I literally stopped listening to the talk so I could start converting my course materials. It was that useful to me immediately.
Michael Chow: Wait, I think I saw a little like thread of that. That’s back when it was called NidR, is that?
Mine Cetinkaya-Rundel: Yeah, so that’s the, yeah, it was like R and W files.
Michael Chow: Like 2012 or something.
Mine Cetinkaya-Rundel: 2012 sounds right, yes.
Michael Chow: Wait, that’s neat. It might be useful too for us to unpack because I think there are so few people in the data world who will both have spanned useR and SciPy. Like that slice of the Venn diagram is like mostly just all border. How, maybe we could unpack at the table like what is useR and what are SciPy and.
Mine Cetinkaya-Rundel: Yeah.
Wes McKinney: Yeah, I’ve been to a SciPy too. They feel like the people are very similar. Like it’s pretty academic. It’s, you know, people struggling to understand their data sets and talking about it. Pretty academic, but also I think with a, I would say a strong commitment to maintainable open source software that can serve others as well. I feel like there’s a subset of academic, which I totally was when I was writing my PhD. I’m like, I need this to work for me so I can get here. And the rest, I don’t know, once this is published, you know. But I think in these conferences there is a lot of like academic motivation for the start of the projects. But a commitment to building tools that others can use as well. And that’s the piece that I think is enjoyable to hear about.
Michael Chow: Oh, neat. Yeah, like a lot of caring about reproducibility. Like thinking, yeah, like the long term. Like how can I make sure people can still reproduce my results in like 10 years or 20 years.
Mine Cetinkaya-Rundel: Right, right, right.
Wes McKinney: I think SciPy is the Python related conference that I’ve been to the most over the years. And there was a time where it was the only conference that existed where people came to talk about scientific computing. And like there was some emergent work in doing statistics and statistical data analysis. So at the first two talks that I, Python talks I gave in 2010, one was in, I went to PyCon in Atlanta. And then SciPy in Austin. And so that’s where, initially where like I networked with and met like Fernando Perez from IPython Jupyter. Brian Granger, Travis Oliphant from NumPy, Peter Wang, you know, Travis and Peter went on to found Anaconda. And so this was like kind of the original community that spawned like the much larger now PyData and another ecosystem. But I think it’s interesting because Python has always had this pretty passionate scientific computing community. Like high performance computing, more doing like high energy physics. Like they were refugees from like MATLAB and Fortran. And they were just happy that they could wrap all their Fortran libraries in Python and script them that way. And so I’m happy that that community still exists. And there are still, you know, people in academia and in research labs, you know, doing hard science and scientific computing and talking about scientific data formats. So it hasn’t totally been taken over by, you know, data science and machine learning and AI.
Michael Chow: Yeah, yeah, it’s neat to hear that that too was like one of the first places you debuted. Was it, it was a Pandas talk, is that?
Wes McKinney: Yeah, yeah. So there was a, yeah, and there was like a paper. So SciPy still, I think, still has paper submissions. And so the first like academic style paper publication about Pandas came out as part of the SciPy proceedings in 2010. So when people cite Pandas, they usually cite that, cite that paper. Had to all be written in LaTeX, of course.
Michael Chow: Yeah, that’s wild. Yeah, the other thing that’s sort of mind-blowing to me looking back now is like, you know, a lot of the early days of R and Python were like rejecting this idea that you should like have to pay for a programming language. Like that used to be the norm. Like you would go and buy MATLAB, you’d buy a Fortran compiler. And now that idea that you would like buy a programming environment, it just seems like it’s bizarre.
Wes McKinney: Yeah.
Michael Chow: But like that was where we started from. I’m also curious how interest in Python has shifted in universities. So do people have thoughts we could do?
Mine Cetinkaya-Rundel: I think one thing that we tell, so I’m in a statistics department. We primarily use and teach R. Our students who are, let me talk about undergraduate students, who are thinking that they will go to industry after and would like to do something data science related. Also are advised and know that they should know some Python. And generally, my recommendation is try to be really good at one language and make sure that that’s not the only language that you are at all comfortable with. So that’s how our curriculum is designed. There’s also the fact that just about everyone at any university nowadays learn some Python in a CS101 course, because just about everyone at every university seems to take a course like that. So for our stats students, we recommend that quite a few of them do stats and computer science together. Some out of genuine intellectual interest and some, I think, hedging their bets in terms of where will the hiring market be by the time I graduate? Will it be more data science and modeling machine learning or will it be that, like, might I get a software engineering job? So they tend to sort of do both. And so they do get exposure to that at the graduate level. A lot of our PhD students, for example, still ultimately write code in R and use R packages. But I feel like I see more of them, particularly coming to a graduate program after a couple of years in industry, coming back with R and Python skills. And the thing I really appreciate in some of these students is how they’re good at going between languages, like very versatile. I feel like knowing the ecosystem and knowing that there’s this package or library that I can leverage and I can figure that out sort of enlarges your ability to do things. It’s like I think about it as people who rap in multiple languages. All of a sudden you have more words you can rhyme, you know, like you’ve expanded your options and it both sounds cool and also gives you options.
Michael Chow: It’s a nice metaphor.
Wes McKinney: I was thinking, like, a little more broadly, like, you know, I’m sure you get a lot of feedback from students about, like, what they want, what they think you should be teaching them. Like, what are the things where you’re like, that you just tell them, like, you know, they think it’s a good idea and you know it’s not and you tell them that. But also, like, what are the things where you’re like, oh, yeah, we should change our curriculum to teach that?
Mine Cetinkaya-Rundel: So one example for the latter I can say is that, you know, I feel like ever since the R Markdown ecosystem and now Quarto has been around, I’ve, like, made my course websites with one of these tools. I’ve made slides with one of these tools. It used to not be a learning goal for the intro data science course. They did write, like, my students write Quarto documents for their homework assignments and so they know how to write computational documents. But, like, turning it into a website, for example, was not one of the learning goals because, like, they already have to learn so much. It seemed like putting a lot on them. But then I realized that they were, I would hear things from students like, oh, you can only do data analysis with R, but with Python you can do everything. And I feel like Shiny changed things a little bit. Oh, now I can make web applications with R. Like, that’s a thing that’s useful outside of doing my homework. And so now, for example, in the intro data science course, the project that they do at the end actually is a Quarto website. And there’s very little additional overhead to make this happen if you, like, set it up for them and they’re just, like, putting their content in there. But I feel like just that knowledge that with this language, that if you look it up on Wikipedia, it says statistical programming language, you can build more things, is actually really useful. Because it, I think, maybe I don’t then teach them every single other thing you can do with R, but it motivates some curiosity for them to be, like, I wonder if we can do this with R as opposed to thinking there’s no way I can do this with R.
Michael Chow: Yeah, I could see how if you put one demo of kind of, like, doing the thing, the set of things they thought was impossible, it really kind of opens the door.
Mine Cetinkaya-Rundel: Yeah. And I think you’re writing a book on Quarto, so at some point people will have, hopefully, a whole game of Quarto to go through.
Michael Chow: Yeah, we do have that. I am writing a book on Quarto. I figured, I think since we, I feel like we’ve touched so much on stuff that students will get so much out of, I don’t know, maybe we can just go around really quick with the time we have left and say, yeah, like, why does data science matter to you? I think because so many, like, because I do think this is so useful for students, I’d love to hear, like, why data science? Since we’ve talked so much about learning and conferences. So, Hadley, do you want to start?
Hadley Wickham: I just think it’s, like, so empowering. Like, there’s so much data around us, like, data that you’re generating, data that things that you care about are generating, like, learning a little bit of data science just gives you this amazing power to kind of, like, dig in and learn stuff. And I think it’s, like, so rewarding and so fun when you find, like, that data you really care about. And now you’re empowered to learn more about yourself. And that’s, like, super, super cool.
Michael Chow: Thanks. Wes?
Wes McKinney: I mean, I think, it’s a big question, but I think that data literacy and statistical literacy in general is probably, you know, something that’s missing from a lot of basic education, especially in the United States. Like, we learn to do trigonometry and geometry and algebra and I don’t know what else is in the general high school curriculum. But I did not learn any statistics or data literacy in elementary school, middle school, high school. And so when you go out into the world, I think people are missing this foundation of, like, how to make judgments, how to interpret information they’re receiving from the standpoint of data. Like, how do you understand taking risks? Like, how do you understand your finances? And, like, everything, you know, whenever somebody presents you a fact, like, you should be asking the question, like, is that fact supported by data? If so, like, what’s the data? Can I have a look at it? If you could get access to the data set, maybe you could explore the data set for yourself and see if maybe the analysis is cooked or has been spun in a way to, like, support a narrative. And so I feel like equipping people with tools to make data analysis more accessible, you know, feels like just a really valuable thing to do in the world so that more people can be data literate and can be equipped to ask more, you know, ask their own questions and have the tools to answer them.
Michael Chow: Yeah, thanks. And Mine?
Mine Cetinkaya-Rundel: Yeah, I think, like, at a high level, I would like to live in a world where if there’s going to be rules and policies, those are data driven. And the folks who are making them, but also the folks who they apply to have a sense of how we arrived at those decisions. And then at a personal level, like, if you take a more micro level, I want to be able to make decisions about myself with some data as well. So I’m always, like, intrigued by these, like, blog posts where someone’s like, I took my fitness data and learned this about myself. Like, I think those are just, like, cool little things you can do to learn either about just you or the world around you that’s helpful. And I do think that that’s working with data. There’s also sort of the more statistics aspect of things, making sense of uncertainty, which I think is even harder, which is a thing I try to keep, like, in my sort of teaching as much as possible, both because I teach in a stats department, but I also think that, like, it takes some time to be comfortable with making decisions around risk and whatnot when there’s uncertainty around the estimates that you’re looking at. And I think that’s something we need to get people to understand as well, because at very critical times of your life, that might be the sort of statistics someone tells you and you need to make a medical decision or something. So having some experience with that sort of thinking, I think, is very helpful.
Michael Chow: Yeah, thanks. Yeah, it’s so helpful to hear the, like, need for kind of, like, code first, data science, and the power of data literacy. And, yeah, like how you might need to make, like, personal decisions with data or, like, big decisions with data and having a sense for things like uncertainty are really important. Yeah. Yeah. Thanks. Thanks so much, everyone. I think this has been so helpful to hear.
Mine Cetinkaya-Rundel: Do you want to answer the question too?
Michael Chow: What’s the question?
Mine Cetinkaya-Rundel: Why is data science important?
Michael Chow: It’s such a good, wow, what a great question. Yeah, I think data science is important. I mean, I think going back to what Mine said, like, I do think everybody interacts with data, whether it’s like a little bit or you’re being hit with data. And I think, like, all of our interactions with data matter. Like, we use data for so many things. And so I think that just, I think data literacy really equips people, you know, whether it’s statistics or coding, to just handle all the situations that might be thrown at us. Whether you’re, like, tracking coupons in an Excel spreadsheet, which is, like, my dream scenario, or you’re, like, yeah, having to make an important decision that involves a statistic, which will have uncertainty. I just think that it can really improve so many people’s lives in so many ways. And some of those are really critical decisions that impact a lot of us. So, yeah, I really appreciate everybody taking the time. And, I mean, I feel like if I were a student, this is exactly what I would have wanted to hear. So I think from going to actuarial science to data science and the role of LLMs in education. Yeah, Mine, I really appreciate you coming on and just hitting us with so many things to think about. Thank you.
Mine Cetinkaya-Rundel: Thanks for having me.
Michael Chow: Thanks, Mine.
[Podcast outro]
The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with branding and design agency, Adji. For more episodes, visit thetestset.co, or find us on your favorite podcast platform.