The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering
This transcript and summary were AI-generated and may contain errors.
Summary
I joined Dan Beech on the Data Engineering Central podcast for a conversation that covered my whole career arc and then spent most of its energy on what AI is doing to software engineering.
We started with Apache Arrow, which Dan had picked up through PyArrow around 2015, back when almost nobody was talking about it. I described Arrow as one of those projects where the reaction on first hearing about it splits two ways: people say “that’s obviously a good idea, why doesn’t it exist already,” and then immediately follow with “you’ll never get anyone to adopt it.” The standards problem is real — there’s the xkcd comic about 13 competing standards becoming 14 — but Arrow worked because it was the only credible attempt at the problem, and because adoption compounds. The format hasn’t changed much in five or six years. We mostly had to wait. Now DataFusion alone underpins something like 30 or 40 companies, Cloudflare acquired Arroyo, Apple uses DataFusion Comet to accelerate Spark, and dbt’s new engine is built on Arrow and ADBC.
That led into what I think is the most interesting technical claim in the conversation: projects like DataFusion and DuckDB are the ones most resistant to being vibe-coded. LLMs are averaging machines. I compared the difference to injection-molded plastic toys versus a fine Swiss watch. I’ve talked to Hannes Mühleisen about this and he just laughed at the idea. What I did try to convince Hannes of is that he should use AI for the unpleasant parts — generating test cases — rather than hand-coding everything.
The trust argument connects to that. A lot of open source value is a track record of taste and reliability built over years, and people relying on you to guard against supply chain attacks and fix serious bugs. I said I feel like my credibility is on the line every day, which means I can’t ship vibe-coded slop — I have to hold to the standard I set before. Doing that in a world where you can generate 10,000 lines of code a day is genuinely hard. I also noted that open source ends up being more of a cult of personality than it should be: people still call me the pandas guy even though I haven’t been heavily involved in development since 2013, and they should be thanking and sponsoring the large group who actually maintain it.
On origins: I was a gamer, got into GoldenEye 007 speedrunning in the late nineties, and maintained speedrunning fan sites by hand on GeoCities without knowing how to program — which is a little painful to think about now. Watching Derek Kisman automate all of it with C++ gave me my first look at what a genuinely good programmer could do. I studied pure math at MIT, felt inferior to the child prodigies around me, learned enough Java to decide programming was terrible, and only got serious in 2007 at a quant hedge fund where I expected differential equations and found Excel spreadsheets. The reaction that started everything was simply: this should be code. Not MATLAB, because you’d have to buy a license for every production server — it should be open source and in a language built for systems.
We talked about why pandas ended up built on NumPy — it was the only game in town, and interoperability was mandatory to reach Matplotlib and SciPy — and how that forced anything non-numeric into Python objects inside NumPy arrays, with all the indirection and overhead that implied. That limitation became a direct motivation for Arrow. I also admitted pandas was not especially well engineered internally. It was focused on human ergonomics, and the internal messiness created real maintenance burden. Projects don’t have to be architecturally perfect to succeed.
Asked whether data problems have actually changed, I said we’re fighting many of the same ones: moving data from A to B, converting formats, loading into memory, running queries efficiently. I didn’t anticipate pandas becoming the load-bearing infrastructure it became, and there is probably more pandas code being written by LLMs now than was ever written by humans. My summary of the last two decades of progress is that we mostly just needed to listen to the database people — read the papers, go to the conferences, implement what’s in them. We went through a painful NoSQL era and produced scalable systems that a good single-threaded implementation on one machine could trounce, which is the point of the McSherry “Scalability! But at what cost?” paper. Now the modern data stack is essentially ruled by database technology and everything is columnar.
On advice for new graduates I’m still optimistic, though I had a small existential crisis last year about whether I had a future as a software engineer. My view is that AI tools filter people by agency. They hand you a B-plus, middling approach to every problem, and if you don’t bring taste and judgment you end up in what I’ve called the agentic tar pit. So the study time moves from learning to write Python or Java toward software and data architecture — knowing how systems should be designed, and being able to articulate precisely what you want. If you arrive at the prompt and can’t explain the problem or recognize when the agents are heading the wrong way, you’re in deep water with no compass.
The failure mode I keep hearing about is decision fatigue. Work used to be socialized — sprint planning, story points, shared conviction built with colleagues, and then you just did the implementation. Now all of that agile process is compressed into plan mode, and there’s nobody else in the room. Engineers are making ten times as many decisions a day. AI will not fix that. Relatedly, giving AI to someone without taste or experience mostly turns them into a slop cannon, and a lot of businesses struggling to see returns added AI while the people stayed the same.
We finished on token economics. I mentioned that at API rates I’d used $37,000 in tokens over the previous 30 days, far more than I actually pay the frontier labs, which makes the current moment an epic subsidy. I’d recently run GLM 5.2, one of the newer Chinese open-weights models, on physical infrastructure, and it worked well — but it needs something like eight B200s, so a quarter to four hundred thousand dollars of hardware. What I’m looking forward to is the point where open weights get good enough and hardware cheap enough that you can put a server in your closet, pay for the power, and just plug the machine into the wall and get coding.
Key Quotes
“These projects that are finely, intricately constructed by expert artisans like DuckDB — it can be very difficult to build that with AI. Maybe someday, but not anytime soon.” – Wes McKinney
“I feel as an open source developer that every day my credibility is on the line. I have to defend my track record. Especially now, building software with AI, I can’t release vibe-coded slop.” – Wes McKinney
“The AI tools separate people and filter people based on their level of agency.” – Wes McKinney
“They’re going to provide you a very average, middling, B-plus kind of approach to solving every problem. But if you don’t bring good taste and judgment, you’re eventually going to end up with a big quagmire that the agents are going to start to suffocate and choke to death.” – Wes McKinney
“All of that agile methodology is now being crammed into plan mode in Claude, and there’s nobody else in the room. It’s just you, and you have to make all the decisions.” – Wes McKinney
“If you give AI to somebody who doesn’t have the taste and judgment and experience to know what the good thing to build is, in general it will just turn them into a slop cannon.” – Wes McKinney
“One of the major outcomes is that we just needed to listen to the database folks. Read the database papers, go to the database conferences, and just implement what’s in the papers.” – Wes McKinney
“My joke now is that I waited just long enough to learn Rust that I will never have to.” – Wes McKinney
“At API rates, in the last 30 days I’ve used $37,000 in tokens. And I don’t pay nearly that much to the frontier labs.” – Wes McKinney
Transcript
Dan Beech: Welcome to the Data Engineering Central podcast. I’m Dan Beech, your host. Today we have Wes with us. How’s it going, Wes?
Wes McKinney: It’s going great.
Dan Beech: What’s the weather like down in Tennessee? Is it super hot already, or still spring?
Wes McKinney: It’s pretty nice today — a little bit muggy. It’s humid in summers here. Not super hot yet, but I figure the rest of the summer we should be cooking pretty good in the 90s with high humidity.
Dan Beech: I personally prefer being hot to being cold.
Wes McKinney: I’m the opposite. I’ve done hot yoga for the last 15 years, so I purposefully go out of my way to go into a hundred-degree room and spend an hour, hour and a half there. When it’s 85 or 90 outside it doesn’t bother me that much. But I get that for other people it’s not exactly their cup of tea.
Dan Beech: Do you get outside much in Tennessee? Go for walks, or are the bugs too bad?
Wes McKinney: There are some nice parks, and east of here you have the Smokies and the Blue Ridge Mountains. It’s nice to get out and walk around in the woods, touch grass, look at some birds, look at the lake.
Dan Beech: Think about what your agents are working on while you’re walking in the woods, maybe.
Wes McKinney: I haven’t gotten to the place where I’m talking to them on my phone. A lot of people set up remote control on their phone, and I’m like — do I want that? The solution to balance is to monitor your agent sessions from your phone while you go for a walk? I don’t know about that. I can just let them go, and I’ll check in on them when I get back.
Dan Beech: Can you introduce yourself? I’m assuming everybody knows who you are, but give a high level.
Wes McKinney: I’m Wes McKinney. I’m best known for creating the Python pandas project about — gosh, what year is it? It became an open source project in 2010. I wrote a book called Python for Data Analysis, which has been pretty popular and kind of a staple of the Python data science ecosystem.
I spent a bunch of years getting the Python data science ecosystem going, and then I became an entrepreneur and started a combination of companies and business models to try to support open source development. We started a company, DataPad, in 2013, and ended up getting acquired by Cloudera. I started the Apache Arrow project while I was at Cloudera with a bunch of open source developers, spent a bunch of years working on Arrow and building the Arrow ecosystem, got pretty involved in Parquet development, and started another company, Ursa Computing, which turned into Voltron Data.
Voltron Data is wound down now, but I’ve been involved with Posit for the last eight years, since the early days of Arrow, and I still have a role there. This year I founded a new company called Kenn Software focused on developer tooling and infrastructure for AI. So I’m primarily focused on that now — building AI developer tools rather than data science libraries and systems, and pretty dialed into what the future of software engineering looks like. I spent the last almost 20 years building system software for data science and data processing, so I learned a lot about software engineering, and now I’m interested in how we do more of the good stuff more productively and use AI to do it. I’m trying to figure that out, but having fun in the process.
Dan Beech: The first time I ran across your name was — I’m trying to think — I’ve been blogging a long time, maybe 2015. I started playing with Arrow through PyArrow. Barely anyone was talking about it at that time. It’s amazing how far Arrow has come and how it’s exploding in the data space now. I remember nobody talking about it for six years or so, and then all of a sudden, boom, it was everywhere.
Wes McKinney: It’s an interesting case study in how Arrow started and happened. It’s one of those projects where when people found out about it, they’d say “oh, that sounds like a good idea,” and then either “that’s going to be really hard to make popular and successful” — because how do you get people to invest the time and energy to adopt it, and where do the first adopters come from? When the ecosystem of Arrow users and consumers is small, there’s just not that much incentive to adopt it.
For people who don’t know, it’s a universal in-memory data substrate for tabular data. It makes transferring and processing data in memory a lot faster and makes systems more interoperable. So people say, “that’s an amazing idea, why doesn’t that exist already? Surely something like that must exist.” And then when they find out nothing exists, they say, “you’ll never be able to get anybody to adopt and use that, and you’ll never get a community of open source developers to agree on one thing — they’ll want to invent other things that are slightly better for their use case.”
So the fact that we’ve been able to get a lot of people to adopt and use Arrow has been kind of amazing. It’s one of those open source success stories that you hope for, because it doesn’t always work out like that. There’s an xkcd comic — there are 13 competing standards, I’m going to create one standard to unify them all, and you end up with 14 competing standards. That’s often what happens. But Arrow is a category-defining piece of technology, and because it was the only credible attempt at solving the problem, just building reliable software that people could use and waiting was the only real solution. Arrow probably hasn’t changed that much in the last five or six years. We just had to wait for people to adopt it. And the more people adopt it, the more exponentially valuable it becomes.
Dan Beech: I watched it grow in popularity — DataFusion and the rest, and then eventually Spark. It’s funny to see people building companies on top of Arrow.
Wes McKinney: There have got to be 30 or 40 companies that are based on DataFusion — not DataFusion as the product, but DataFusion as part of theirs. People want to build in Rust and they need a query engine, and rather than building one from scratch you can customize DataFusion, which is what a lot of companies have done. There was a company called Arroyo building a streaming data engine, and they got acquired by Cloudflare. So now you’ve got DataFusion and Arrow at Cloudflare powering part of their streaming data infrastructure. It’s everywhere. Apple is using it to accelerate Spark with the DataFusion Comet project.
What’s also interesting is that things like DataFusion and Arrow are the types of projects most resistant to being vibe-coded. You aren’t going to show up and vibe-code a replacement for DataFusion. Same with DuckDB — forget about it. I talked to Hannes about this, one of the creators of DuckDB. I asked him whether he worried about people building stuff with AI and he just laughed, because it’s exactly the type of project where LLMs — they’re averaging machines. They’re not going to be able to build a piece of cutting-edge software. It’s like comparing injection-molded plastic toys with building a fine Swiss watch. These projects that are finely, intricately constructed by expert artisans like DuckDB — it can be very difficult to build that with AI. Maybe someday, but not anytime soon, I don’t think.
Dan Beech: And the goodwill they’ve built in the community — talking about DuckDB or Arrow — it’s not just the tool itself, it’s the way they’ve gone about marketing it and the whole community they’ve built. That’s almost priceless. You can’t replace it.
Wes McKinney: A lot of it is building trust, demonstrating that you have good taste, and acting in the best interests of the community over a long period of time. For better or for worse, open source ends up being a bit more of a cult of personality than it should be. I haven’t been super involved in pandas development since 2013, and yet people still say, “oh, you’re the pandas guy.” The project is developed by a pretty large group of people, and people should go thank them and send them GitHub sponsorship money.
But a lot of open source is building trust and credibility and a track record of building software that’s tasteful, well designed, and reliable — so people know this person is maintaining the project and is going to look out for me, guard me against supply chain attacks, fix serious bugs. I feel as an open source developer that every day my credibility is on the line. I have to defend my track record. Especially now, building software with AI, I can’t release vibe-coded slop. I have to build software that’s consistent with the standard of quality I built in the past. Doing that in a new world where we can generate 10,000 lines of code a day is an interesting challenge, I have to admit.
Dan Beech: Switching gears — take me back to the beginning of Wes. Did you grow up in Tennessee? How did you get into tech? Were you playing video games as a kid?
Wes McKinney: I was always really interested in computers and technology. I was a gamer — I played Nintendo and Super Nintendo, and in the N64 era I got really into GoldenEye 007.
Dan Beech: I’m right there with you, man.
Wes McKinney: In the late nineties there was GeoCities and Angelfire and these site-building websites. When I was 13 or 14 I was tinkering with building little derpy HTML websites and putting them on GeoCities. I made my first GoldenEye fan website in 1997 or the beginning of 1998. Then I found there was a community of speedrunners, and I got pretty involved in the speedrunning community for GoldenEye. But I didn’t program at all. I think back on it and it’s a little painful that I was maintaining all these speedrunning websites by hand and should have been writing C++ programs to help.
There was somebody who came along, Derek Kisman — a Putnam fellow, a TopCoder programming-competition kind of guy — and he professionalized a lot of the speedrunning websites, built a whole bunch of C++ programs to automate updating them. That gave me a preview of what a really good programmer looks like and what they can accomplish.
I grew up in Tennessee and then Northeast Ohio. I went to high school in Cuyahoga Falls, a suburb of Akron. I was more into math in high school — I did math competitions and tinkered with computers and played video games. Then in college I went to MIT and was around a lot of elite programmers who had been child prodigies, programming and doing elite hacks since they were 12. I felt really inferior at that point. I thought, I can’t possibly hope to catch up to these folks who’ve been tutored by parents who are computer science professors.
I programmed a little. I learned that Python was a thing that existed, but I didn’t write any Python then. I learned Java, and I found Java to be awful. So I thought, this programming thing kind of sucks, I don’t want to program in Java, it’s just terrible. I didn’t really get seriously into programming until after I graduated. I got a pure math degree and did a little theoretical computer science — complexity, algorithms — but not much programming. I only really got interested in programming when I recognized it was a tool I could use to build leverage for myself: solve problems, and build tools to solve problems better.
Dan Beech: What year was this, roughly, when you started your first job?
Wes McKinney: 2007. I was 22 and I worked for a quant hedge fund. I was expecting pen and paper, A Beautiful Mind, differential equations on the glass. It turned out to be mostly Excel spreadsheets. I was surprised at how manual and un-automated a lot of the research and analysis work was. I thought, this should just be code. Why is this not code?
There was some code being written — people doing MATLAB, people doing R — but it was supplementary alongside a lot of the Excel work. And you’re not going to put the MATLAB code into production. People tried, but you’d have to buy it: oh, you want to run it on that server over there, you have to buy a license for that server, and it’s really expensive. So it should be code, it should be in a programming language designed for building systems, and it should be open source — you shouldn’t have to pay a license fee to MathWorks to deploy to prod.
That got me thinking about how to solve interactive data analysis in a programming language that’s good for building systems, so you could solve the research-to-production workflow. A colleague said I should check out Python and some scientific computing libraries. I went down that rabbit hole at the beginning of ’08 and tried to replicate some of the research work we were doing in Excel and legacy systems — rebuilt all my research tools in Python. That was basically proto-pandas, and that’s where I really got hooked.
I liked the feeling of being more productive and building tools. I like building tools for other people, and seeing other people become more productive using the stuff I built. It became very addictive, honestly, in the way that in my teens I was addicted to playing video games and speedrunning and being really efficient — beating Facility on 00 Agent. I enjoyed the efficiency feeling of making the tools better, waking up and making things a little bit better every day.
But also making things simple, because it isn’t just about making myself productive. That’s one of the funny things right now — people are building highly personalized tools that are so complicated and convoluted that nobody else can use them. They show you their tool set and it’s a Winchester Mystery House that nobody else can navigate. I really value building tools that are simple and coherent and intelligible to other people. It’s almost as much a human interface design problem as it is software engineering and architecture.
I’ve said at the outset that pandas was not very well engineered. It was focused on human ergonomics and usability — really intelligible, easy-to-write code — but internally it was a bit of a mess. Some of that internal messiness created a lot of maintenance burden for the community over the years. But pandas has been super successful, so it goes to show that projects don’t need to be perfect pieces of software architecture to be really successful.
Dan Beech: At that time, in 2007, 2008, you probably didn’t have many options. Was NumPy even a thing then?
Wes McKinney: NumPy was created by the fusion of Numeric and NumArray by Travis Oliphant. Originally there was one multi-dimensional array library in Python, Numeric, started by Jim Hugunin in the mid-90s, shortly after Python was released to the public. There was a fork of Numeric that added features like memory mapping and various things for doing MATLAB-y stuff on large array data. Then Travis merged them together to create NumPy around 2005 or 2006.
But it’s really focused on numerical data. It wasn’t designed for building a database. It was: you’ve got a big multi-dimensional array of doubles or integers. Because NumPy was the center of the universe for all the Python stuff — Matplotlib, SciPy, statistics, linear regressions — things needed to end up in NumPy arrays to have access to the ecosystem. Interoperability with NumPy was mandatory in that era. So it made sense when I was starting pandas to base it on NumPy and figure out how to work around NumPy’s limitations for non-numeric data.
As a result, anything that was not a number or a Boolean was a Python object embedded in a NumPy array. You’d have arrays containing Python objects with all the overhead and indirection that entailed. For working with string data it just wasn’t very efficient. That obviously became a big motivation later to create Arrow and have an alternate in-memory foundation for building things like pandas. But in 2008 NumPy was the only game in town, so you either integrated or you were irrelevant.
Dan Beech: I came up through the LAMP stack and Perl, around the same time.
Wes McKinney: I wonder how many people these days know what LAMP stands for.
Dan Beech: Probably not that many. Linux, Apache, MySQL and Perl. Is that right?
Wes McKinney: Correct.
Dan Beech: At that time I really didn’t feel like there was a data community. It was mostly web stuff, or what I’d call systems development today. Then all of a sudden — probably before 2010 — you had this split where data warehousing was a thing, SQL Server, totally separate from the software engineers. And then around 2015, 2016 you start to get Hadoop, Pig, Hive, this weird quasi stuff, Parquet. And then Python got popular and there was an explosion. I don’t even know if data science was a thing — it was business intelligence. Spark got crazy popular. It just exploded.
Wes McKinney: There was a whole crop of venture-backed companies that spent a lot of money convincing every company in the world that they needed to become a data company, that they needed to build data infrastructure like Google. Which is true for some companies. It led to massive investment in open source projects, which is good — I was certainly a beneficiary. Without that infusion of venture capital to fund all these big data projects, Arrow probably wouldn’t have happened, or it would have been much harder to get the right people together.
I think part of why Python became suddenly very popular for data science was this big global scramble to figure out what data science means. What is a data scientist? Where do we hire these elusive data scientists who are seemingly good at business and statistics and computer software? Python made data science approachable and accessible to a bigger audience. You could have somebody show up who maybe didn’t have much programming experience but knew statistics, and you could hand them a book — here’s Python for Data Analysis, here’s scikit-learn, learn how to use these things — and they could be productive doing useful stuff at work within two to three weeks. If you had to learn Scala, the learning curve would have been longer.
Dan Beech: I’m proud to say I have gone my entire life without writing a line of Scala, and I think I will go the rest of my life without writing a line of Scala.
Wes McKinney: The same thing also happened with Rust, by the way. My joke now is that I waited just long enough to learn Rust that I will never have to.
Dan Beech: So tell me more — you went to the quant thing. What happened after that? Did you shift into open source?
Wes McKinney: When I was still at AQR we decided to release pandas as an open source project. That was on New Year’s Eve 2009, so the beginning of 2010. I went to my first PyCon to give a talk about pandas in February 2010 — you can still find the video and the transcript of that talk on my website. That was where I first started getting involved in the Python community.
I actually went back to grad school. I started a PhD in statistics at Duke in 2010 because I wanted to learn more about being a data scientist — I felt I needed to learn statistics to do it well. But then I saw this groundswell, this need for more stuff in Python. Financial firms were starting to contact me saying they’d seen my talk at PyCon and were really interested. It became apparent that there was an opportunity for pandas to cross the chasm and become a useful, accessible tool. It was very minimal in those days, with a ton of missing functionality — it couldn’t do all the join types. You can still install pandas 0.1 and see what’s there. Not a lot by today’s standards.
I felt so overwhelmed by the feeling that there was an opportunity that I took leave from grad school and moved back to New York to figure out how to make pandas happen. In 2011 I tried to build a financial technology company powered by pandas. We decided not to do that. I started writing my book. Then Chang She, who’s now the founder of LanceDB — which is doing really well nowadays — he and I founded a company called DataPad, and that’s what led us to move to the West Coast.
We did DataPad for a couple of years. It was a BI company, and if you weren’t Looker in 2013 it was very hard to be a BI company. We saw the writing on the wall and took a soft landing at Cloudera, which turned out to be professionally stimulating — a lot of tremendously smart people at Cloudera in those days. Chang still works with a bunch of ex-Cloudera people at LanceDB. That’s where I got connected with the Impala crew.
Dan Beech: They were the first Hadoop company, right?
Wes McKinney: They were the first Hadoop company. That’s where I got interested in doing Arrow. I got to know Ryan Blue, who went on to create Tabular and Iceberg and is now at Databricks. A lot of people doing really leading, cutting-edge work these days at Databricks and other companies were at Cloudera in those days. It was a cool place to be, and for me it was the springboard for the rest of my career.
We started Arrow there at the beginning of 2016. The enterprise software market was a little tough in the late 2010s, so there wasn’t a lot of budget to build an Arrow team for such experimental technology. I got a lifeline from Two Sigma in New York, who offered to hire me and move me across the country to work on Arrow, make Arrow work well inside their systems, accelerate their data processing workloads, make Parquet better, all the things. That was a very fruitful collaboration for the next couple of years — roughly 2016 to 2018.
By 2018 there were a bunch of corporate interests really leaning in and looking at Arrow’s potential, and everyone wanted to figure out how to put capital to work to fund the project. They were saying: Wes, you’re doing all this development, help us take our money and hire people to work on Arrow. Being at a financial firm, it was logistically difficult for me to manage money coming from third parties for compliance reasons. So we realized it made sense to set up a nonprofit — an independent entity that could take in corporate sponsorship for Arrow development. That was Ursa Labs, created with Two Sigma and RStudio, now Posit, in April 2018.
We ended up with half a dozen corporate sponsors — hardware companies, financial firms — all really interested in Arrow, including NVIDIA. I built a team of eight people at Ursa Labs before we realized there was more commercial opportunity and pivoted to create a startup, Ursa Computing, in 2020, during COVID.
Dan Beech: Do you feel like the data problems you were working on in pandas 0.1 have changed that much today? Data’s an interesting world to work in — the classic software engineer is always saying, oh, data people, they don’t put anything in Git, they have the worst software practices. Do you feel like data has been solved, or are we still fighting the same issues?
Wes McKinney: We are still fighting a lot of the same issues. When I started working on pandas I didn’t anticipate that it would become nearly the load-bearing slice of infrastructure it’s become. I didn’t realize how much data in a business setting would end up flowing through pandas DataFrames. I don’t know what percentage it is, but it’s a lot at this point — especially when you consider that there’s probably more pandas code being written by LLMs now than was ever written by humans.
The struggles are very much the same: moving data from point A to point B, converting from one format to another, loading data into memory, running queries efficiently, doing transformations efficiently. Things like DuckDB are just manna from heaven. If you showed something like DuckDB to somebody 20 years ago their heads would explode. They’d say: something like this is available in the future, and it’s possible, and it’s free, and you can install it anywhere, run it on the phone, run it in your web browser? It’s actually kind of crazy to think about.
But there was a need to struggle for decades to build the motivation to build things like DuckDB. We had to struggle for a long time before there was a realization that we needed to build Arrow. A lot of good ideas came out of big internet companies — Parquet was inspired by work that happened at Google — born of solutions created to solve problems of cost and time at their scale.
Dan Beech: Parquet underpins every lakehouse ever.
Wes McKinney: It’s not the best format ever created, but it’s much better than what we used to have. The fact that systems support it means people aren’t that motivated to move, even though there are better specialized alternatives like Vortex and Lance. People say, well, Parquet’s good enough — and compared with the way things were 20 years ago, it’s nothing to complain about.
In terms of the hierarchy of needs, these days data engineering has been a little bit — you don’t hear quite as much about it. Pete Soderling’s DataEngConf became Data Council, and this year became AI Council. I spoke there, it was a great conference, and I’ve been friends with Pete for a long time. But data engineering is not as popular or in vogue as it once was, so it’s hard to steer energy toward that domain. In a sense that’s great, because open source developers building data tools can continue somewhat undisturbed — keep their heads down, focus on their craft, don’t get distracted by the shiny objects, resist the temptation to vibe-code everything.
I was trying to convince Hannes from DuckDB to use a little more AI in his work and not hand-code everything — at least use AI to help automate writing test cases, doing stuff that’s a little unpleasant or not a good use of his time.
We still have a lot of problems to solve. Databricks just released a whole new data engine, and I hardly heard about it — I saw some stuff on LinkedIn. I feel like something like that would have been much bigger news before, would have been all over Hacker News. But these days stuff in the data world barely makes a blip compared with the news feed coming out of the AI labs.
Dan Beech: It feels like full circle. I grew up in that pandas world, then Spark and Hadoop, then Redshift for a while, BigQuery, Snowflake, Databricks, ClickHouse — and now we’re back to Daft, Polars, DuckDB.
Wes McKinney: It’s funny, because one of the major outcomes is that we just needed to listen to the database folks. Read the database papers, go to the database conferences, tune in to the work coming out of TU Munich and CWI and MIT and CMU and Berkeley, and just implement what’s in the papers, and collaborate, and listen.
We went through a painful era of NoSQL and non-relational storage, and then there was the famous paper from Frank McSherry — “Scalability! But at what cost?” We had all these scalable big data systems, but they introduced so much overhead that you’d end up with a system that’s scalable but can be trounced by a good single-threaded implementation running on a single machine.
Dan Beech: How many people actually need petabyte scale? I think AWS did that famous study — a lot of people are just piddling. They’re not running petabytes.
Wes McKinney: Now the modern data stack is more or less ruled by database technology. Most things are using Arrow — dbt’s new engine is built on Arrow and ADBC, Polars is Arrow, Daft is Arrow, DataFusion is Arrow. Even Photon at Databricks is columnar; I don’t think it uses Arrow, but it’s columnar. Basically the leading solutions to a lot of these problems have ended up having more or less the same architecture and the same types of approaches. Which is great, because we can agree that this is the way we should solve these problems, and dedicate our energy toward higher-level concerns like how we actually make the software usable.
I was reading one of your Substack posts the other day and laughing at the comical complexity of setting up and using Spark. Though Spark was even less complex than using Hadoop. We went from this absolutely insane Java era — we’re just cool with all these XML configuration files, fiddling with all these knobs and buttons, and to actually get anything done requires two days of configuration.
The thing that convinced people about Spark was that word count is one line. Of course, you have to set up all this infrastructure in order to use it, and configuration files and so on. But we can do a lot better. The type of simplicity and form factor you get out of DuckDB is a good model — it should just be this simple. Even in building distributed systems, there should be reasonable defaults and a reasonable configuration that works well out of the box. DuckDB even has a client-server protocol now, and they’re delving into that database-as-a-service space. They also work closely with MotherDuck.
Dan Beech: You mentioned LanceDB a couple times. We’ve got AI, everything’s changing constantly, everybody’s on a lakehouse — Iceberg, Delta, MotherDuck. That was mostly designed around a tabular world. Data-frame-ish, Parquet. And now things have started to change. Vector databases have gotten popular. With AI there’s an explosion of types of data that don’t necessarily fit — unstructured documents, video, audio. Do you think there’s a problem there, where the lakehouse architecture we’ve built is getting funny now because of unstructured data and semantic layers?
Wes McKinney: The multi-modal data lakehouse problem that LanceDB is working on is a legitimate issue, because the data stores you’re building for generative AI, model training, and fine-tuning at large scale in an enterprise setting aren’t just Parquet files anymore. You’ve got images and video and text and documents.
Any time you generate embeddings off a large corpus of unstructured data like image or video data, you can put an economic value on the generated embeddings that came out — I burned this much electricity, or this much GPU time, to produce these embeddings. There needs to be a good data store for storage and stewardship and processing of that data in a production environment. Having every company roll their own multimodal data store is not particularly efficient. You’re going to end up with something that has bugs, or loses your data, or has performance problems.
So it’s the type of problem where it makes sense for a software vendor to solve it well and make a good solution available so other people don’t have to solve it over and over. That’s partly why — yes, you can build and manage your own Iceberg tables, but it turns out it’s better and simpler to have them managed by a vendor, whether Databricks or Snowflake or AWS. That’s one less piece of operational complexity to worry about. The same is true with the multimodal data powering generative-AI-based systems.
Turbo Puffer is a heavyweight in vector search on documents and high QPS, powering things like Notion’s AI vector search. But there’s a lot of room in that space for solving different layers of the data problems emerging in generative AI. It’ll probably be good when things settle down a little and people can get some of the frontier lab IPOs out of the way, and then focus on what we can do that’s economically sustainable and adds value to humanity — and isn’t just generating slop cat videos.
Dan Beech: Are you generally pessimistic or optimistic about the future of software and data? Say somebody’s listening and they just graduated, they’re brand new, they’re scared — are they going to have a job in five years? How would you tell them to proceed? Should they still learn stuff, read books?
Wes McKinney: I’m still optimistic. I went through a small existential crisis last year where I wasn’t certain that even I had a future as a software engineer, or interesting work to do and contribute to society. But I realized that the AI tools separate people and filter people based on their level of agency.
The AI tools provide a tremendous amount of value and leverage to people with high agency — people who spend time thinking about what they want to build and develop good taste about knowing when you’ve built something good, or what the good thing looks like. To develop that taste and build agency, people are going to have to spend a lot less time hands-on. It used to be that you would build taste and agency by being hands-on building stuff. The experts would have the technical skills but would also be able to visualize the efficient path to build the thing that solves the problem. You can still do that, but knowing what the efficient path is, or what the solution looks like, requires a degree of experience and taste and judgment that the LLMs are not going to provide you.
They’re going to provide you a very average, middling, B-plus kind of approach to solving every problem. But if you don’t bring good taste and judgment, you’re eventually going to end up with a big quagmire that the agents are going to start to suffocate and choke to death. That’s what I call the agentic tar pit in one of my essays.
So instead of learning how to program, learning how to write Python or Java, I think we’re going to spend a lot more time studying the architecture of software systems and data systems — understanding different data engineering architectures. What’s the Lambda architecture? What’s the Kappa architecture? How do these systems work, and how should they be designed to solve different classes of problems? And then being able to articulate and communicate what you want exactly.
That’s the big problem now. If you arrive at the prompt and can’t articulate what problem you’re solving or how things should work, and can’t recognize when the agents are going in the wrong direction, you can very quickly find yourself in deep waters with no compass. You’re like, oh my gosh, I’m drowning and I have no idea what to do. I’ve built some things where I don’t know enough about what I’m doing, and at a certain point the agent is asking me, do I want A or B — and honestly, I couldn’t tell you whether A or B is better.
So to new grads, my recommendation would be to spend time with books about software design, software architecture, and designing data systems — depending on what your goal is, whether it’s data engineer, data scientist, or software engineer. Because unfortunately we’re going to spend a lot more of our time talking to prompts and less time writing code by hand. But if you can’t explain what you want, you’re not going to get it.
That’s the thing a lot of people don’t realize. They think AI is a magical panacea that gets sprinkled on organizations and they just become more productive and more economically successful. The reality is that a lot of businesses are struggling to see return on investment from their AI investment, because they just added AI and the people stayed the same. If you give AI to somebody who doesn’t have the taste and judgment and experience to know what the good thing to build is, in general it will just turn them into a slop cannon. I just learned this term the other day. You can produce a lot of stuff, but none of it’s very good, and a lot of it becomes a liability that other people have to deal with.
Dan Beech: AI just amplifies who you already are as a developer. It’s going to say, sure, of course, I’ll do that. Yes, that’s right, let me do that for you. I agree that systems architecture and high-level design are super important now, because I don’t see a future where software engineers are just writing some little function to add A plus B and return it. It’s going to be more soft skills — communicating with product and marketing, understanding the C-suite, taking requirements and turning them into a plan.
Wes McKinney: We’re of the same generation, so you probably remember all these coding interviews where you go in and implement a double-ended queue in C++. I had to do plenty of those. I don’t even know what interviewing looks like now for developers and data scientists. The next five years or so — two years even — is going to be kind of confusing and a bit chaotic.
The thing that will hurt you the most is essentially decision fatigue: having low agency, struggling to arrive at conviction about what to do next. This is something I’ve been hearing a lot about — engineers who are paralyzed by having to make ten times as many decisions in a day as they did in the past.
It used to be that you would have meetings and discuss with your colleagues and do sprint planning, and you’d build shared conviction about every piece of work that was going to be done. You’d have planning poker and story points, and work would be agreed upon and divvied up amongst the team. Then you’d say, okay, it’s all been decided what’s going to be done, I just have to do the implementation. Now all of that agile methodology is being crammed into plan mode in Claude, and there’s nobody else in the room. It’s just you, and you have to make all the decisions.
So you’re either going to react well to that level of demand and ambiguity — you can think quickly and build conviction about the efficient path forward — or you can be frozen in place, paralyzed by the uncertainty of not knowing what to do. Unfortunately AI is not going to fix that.
Dan Beech: And it’s harder to hide in an organization, too.
Wes McKinney: I predict that companies are going to continue to struggle to get returns. They’re going to pay a lot of money to Anthropic and to OpenAI and wonder where the results are. I think there are results — some that are positive — but also a lot of waste and slop, and consequences of the slop. As long as the foundation labs sell tokens and a lot of people are still paying for the tokens, they’re happy. But as companies put harder constraints on token budgets, it’s going to be really interesting to see how things evolve.
Maybe there’ll be some radical technological innovation that makes producing tokens a lot cheaper. I got my hands on GLM 5.2, one of the latest open-weights Chinese models, running on physical infrastructure. It worked great. I’m just wondering, how long will it be until I can have a server in my closet that runs one of these one-trillion-parameter open-weights models, and I just pay for the power for that server? It’d be nice for that server not to cost fifty or a hundred thousand dollars — I think that’s about what a server that can run these models costs right now. But maybe in the fullness of time the open-weights models will get good enough and the hardware affordable enough that there won’t be a need to send all of our money to Dario or Sam Altman. Just plug the machine into the wall and get coding. I’m looking forward to that.
Dan Beech: I tried some of these small language models with OpenCode a little while ago. Everything’s slow. There’s a lot of room.
Wes McKinney: GLM 5.2 is not small — you need like eight B200s to run it. It’s pretty beefy.
Dan Beech: I don’t want to know what a B200 costs.
Wes McKinney: A single B200 is thirty to fifty thousand dollars. So you need a good quarter of a million, or three or four hundred thousand dollars in hardware, to run this model well. It’d be interesting to see what ends up being economically sustainable, because clearly we’re living through epic subsidies for the tokens we’re using. Personally — not to brag about my token burning — at API rates, in the last 30 days I’ve used $37,000 in tokens. And I don’t pay nearly that much to the frontier labs.
Dan Beech: There’s always a question of adoption, too. We can be in a bubble. There are a lot of companies where boards and CTOs are still talking about it and haven’t done anything, and you’ve got this long tail of people actually adopting it. It’s hard to know what the truth is.
Thanks for joining me today, Wes. I took more of your time than I meant to, but I really appreciated talking to you and hearing about your background. It’s fun to hear the full story.
Wes McKinney: Of course. Thanks for having me on.