I wish we did a more formal study, but at $previous_job we rolled out AI tools (in that case it was github copilot) and we found that for 6-8 months productivity largely stayed the same or reduced slightly, but after that it sharply increased. This was rolled out to hundreds of developers with training, guidance, support etc. It was done in what I would consider the right way.

Was that project fairly early-days? The current impression seems to be that AI is useful for accelerating the development of smaller and simpler projects, but slows things down in large complex codebases.

The sample size isn’t the individual participants, it’s the hundreds of tasks performed as part of the study. There’s no indication the study was conducted incorrectly.

Except that the participants were thrown into tasks cold, seemingly without even the most basic prep one would/should do before throwing AI at a legacy codebase (sometimes called "LLM grounding" or "LLM context bootstrapping"). If the participants started without something like this, the study was either conducted incorrectly or was designed to support a certain conclusion.

  LLMs.md
  ├── data_model.md
  ├── architecture.md
  ├── infrastructure.md
  ├── business_logic.md
  ├── known_issues.md
  └── conventions.md

By the time all of this is written, I'm familiar enough with the code to fly over it (Hello, Emacs and Vim). But by then, your tasks are small and targeted fixes, because any new feature requires lot of planning and stakeholder discussions that you can't just go and work on it.

> I simply cannot square this with my last few weeks absolutely barnstorming on a project using Claude Code.

I don't know, but the interesting data in the study is that they all said the same thing you are saying, but their actual time was 19% slower.

And yes, right now it's the only study that seemed to have a good methodology that I've seen that has any data positive or negative.

This study was about “246 tasks in mature projects”. I would expect AI to fare much better in a study about new projects or brainstorming.