Adam Curtis talked about Forester and the study of “cybernetic systems” (to use the contemporary term) in general, in episode 2 of “All Watched Over By the Machines Of Loving Grace” [0].
The core argument (which is alluded to in the Logic Magazine article) is that in the 1960s when computers were beginning to be integrated into companies and governments, there was a strong desire to have the computer solve all the world’s problems. Of course, the problems were not well understood (and still often aren’t), while at the same time the computers were woefully underdeveloped. Therefore simplifications were made so that the computer could calculate something that looked somewhat reasonable. Of course the models were symplistic, and the data encoding symplistic and biased as well. But hey, the computer “solved” the problem, and as we all know, “Computers don’t make mistakes.” Eventually (and Forrester encouraged) the simplistic world was confused for an actual complete description of the real one. Most damningly, industrial leaders and politicians, started trying to make people and society fit the simplistic model, instead of improving the models to better fit society. Why? Well, there was a computer model made by Very Smart Men(tm), and it gave cover for what was already decided, so it must be right.
It’s as if the proverbial dairy farmer that received the report that begins, “Suppose you have a spherical cow...”, started selective breeding and cutting his cows to optimize for sphericality, as opposed to milk production, because the report says “spherical cows.”
It’s a similar pattern we’re seeing play out with ML, where the assumption is that the classifier is correct regardless of its actual performance.
The core argument (which is alluded to in the Logic Magazine article) is that in the 1960s when computers were beginning to be integrated into companies and governments, there was a strong desire to have the computer solve all the world’s problems. Of course, the problems were not well understood (and still often aren’t), while at the same time the computers were woefully underdeveloped. Therefore simplifications were made so that the computer could calculate something that looked somewhat reasonable. Of course the models were symplistic, and the data encoding symplistic and biased as well. But hey, the computer “solved” the problem, and as we all know, “Computers don’t make mistakes.” Eventually (and Forrester encouraged) the simplistic world was confused for an actual complete description of the real one. Most damningly, industrial leaders and politicians, started trying to make people and society fit the simplistic model, instead of improving the models to better fit society. Why? Well, there was a computer model made by Very Smart Men(tm), and it gave cover for what was already decided, so it must be right.
It’s as if the proverbial dairy farmer that received the report that begins, “Suppose you have a spherical cow...”, started selective breeding and cutting his cows to optimize for sphericality, as opposed to milk production, because the report says “spherical cows.”
It’s a similar pattern we’re seeing play out with ML, where the assumption is that the classifier is correct regardless of its actual performance.
[0] https://archive.org/details/AllWatchedOverByMachinesOfLoving...