Unmanageable

People, power, and organizations.

Expertise and Incompetence

The 1960s were a period of ambitious and assertive interventions by government and policy elites: Johnson’s War on Poverty institutionalized federal social policy in a host of new offices and initiatives. From his perch in the Department of Labor Daniel Patrick Moynihan produced his controversial diagnoses of “pathologies” in The Negro Family. The Kerner Commission offered evidence-based recommendations to professionalize and rationalize policing and end riots in urban centers. The Rockefeller and Ford Foundations bankrolled the Green Revolution, bringing Westernized, industrial mono-cropping to the developing world. World Bank and IMF economists offered development loans to poor nations conditioned on detailed free-market prescriptions for fiscal policy. Most of all, the “Military-Industrial Complex” rained devastation on the people and land of Southeast Asia, all on the advice of “idea men” in the White House and the RAND Corporation. In each of these cases and many more, policies were increasingly developed and promoted by high-status experts ensconced in government offices, large multi-national corporations, and rapidly expanding universities.

All of these interventions suffered persistent setbacks, either failing to achieve their stated goals or creating serious new problems in their wake — or, as in the case of the Vietnam War and its spillover campaigns, untold destruction and carnage. Popular critiques fold blunders like these into a general disdain for experts and government, along the lines of Reagan’s famous quip:

The nine most terrifying words in the English language are: I’m from the Government, and I’m here to help.

Press conference, Aug. 12, 1986

This kind of simplistic critique served as justification for decades of deregulation and privatization of government functions. While the 1960s might represent a high water mark for a certain kind of expert intervention, such interventions were not the sole province of government — and, tellingly, government “deregulation” has not reduced the number or significance of bureaucratic experts in American society. If anything, government devolution to non-profits, “public-private partnerships,” and local governments has expanded the overall population of experts while reducing their accountability to the public. Like most of the discourse around “small government,” Reagan’s superficial dig at bureaucrats distracts from more substantive issues. The question remains, from everyday annoyance to catastrophes, what are the sources of bureaucratic blunders? How do experts get it wrong?

To tease out this paradox, we’ll briefly look back at the rise of experts and especially at the varieties of expertise that bureaucrats possess. Clarifying what bureaucracy is and how it works helps us draw out the specific pathologies that lead experts to endorse terrible ideas.

From Incompetence to Rationalization and Back

Karl Marx was a keen observer of the bureaucracy of nineteenth-century Europe, which, where it existed, was confined primarily to the state. Marx believed the state bureaucracy was fundamentally incompetent, unable to formulate and implement sound policy programmes because of a “circular” breakdown in communication and responsibility: the higher-ups assumed the lower functionaries would adapt policies to specific circumstances, while the lower functionaries assumed the higher-ups possessed a global vision that made the policy they handed down generally applicable. In other words, the different levels of officialdom each abdicated responsibility for the results of policies they implemented. This basic incompetence was then exacerbated by all the other pathologies of the bureaucracy, not least their pretensions to serving a general interest rather than their own particular interests and those of the bourgeois state. Marx also noted the prevalence of turf battles and internal competition among departments. It’s impressive (and more than a little disappointing) that the dysfunctions Marx identified are still familiar almost two centuries later. Nevertheless, his account is of limited use, developed at a point when the bureaucratic phenomenon was both less widespread and much less credentialed than it is today.

By the turn of the twentieth century, bureaucracy had developed much further, especially in the large corporations of the US and Germany. Spurred by declining profits, these firms consolidated and integrated vendors and buyers, creating sprawling chains of supply, production, and marketing that were coordinated by new ranks of middle managers.1 Yet ‘scientific management’ gained an audience in a broad range of organizations and institutions, in the state and civil society alike.2 Max Weber developed the concept of rationalization to describe this increasing application of a technical-scientific mindset to social and economic life, institutionalized in a host of new and rapidly growing bureaucracies. Weber’s analysis of bureaucracy is brief and idiosyncratic, made even more challenging by the way he approaches bureaucracy as an ideal-type (preferring description so generalized that there was, and is, probably no actually existing bureaucracy that fits his account).

Sticking to our guiding question about bureaucratic incompetence, it’s not clear Weber would recognize our inquiry as valid. He believed bureaucracy possessed a “purely technical superiority,” maximizing “precision, speed, unambiguity, knowledge of the files, continuity, discretion, unity, strict subordination….”3 The officials in his model were objective experts, performing their duties impersonally and according to rules developed for optimal efficiency and success. Weber would likely explain away specific examples of incompetence as evidence that a particular bureaucracy was not fully developed, as measured against his ideal type.

But Weber also believed that the process of bureaucratization had a natural limit in society, since this rule-bound order was in his opinion ultimately deadening to the creative energies and personal independence that provided the vital core of a capitalist economy — a claim that has become the enduring hobbyhorse of right-wing critiques of bureaucracy, developed especially by von Mises and Hayek.4 Of course, their claims were especially directed at the specter of the USSR and its planned economy; there’s much to say about the motivations underlying their analysis as well as its afterlives, but we can save that for another time. Suffice to say, Weber is not an especially rewarding source if we want to understand the observed incompetence of institutionalized expertise. His broader claims about bureaucratization imply some shortcomings that become evident at some advanced level of development, but that’s about it. Nevertheless, later thinkers expanded on some of Weber’s — and Marx’s — characterizations of bureaucracy to provide interesting insights about institutional incompetence.

Engineers and Managers

Claude Lefort stands out among these later authors. In Lefort’s view, Weber misses the trees for the forest: that is, Weber is inclined to describe a broad phenomenon of bureaucratization and the apparent commonalities across many different bureaucracies, and this strictly formalist account refuses “to accept that the bureaucracy has its own dynamic and instrinsic goal” and thus “fails to investigate…the ways in which it is rooted in its social being and increases its power.”5 Lefort proposes an analysis that’s more historical, more concerned with the details of how a given bureaucracy develops and how it works.

A key part of this analysis is clarifying what kind of ‘expertise’ underwrites professional and official practice. Recall that Weber identified tendencies toward “rationalization” as the driving force behind the growing application of technical-scientific expertise to social, political, and economic life.6 But properly technical or scientific expertise is different from administrative expertise.7 One is the purview of engineers and scientists, the other the purview of managers. As Lefort points out, bureaucrats often have a bit of both kinds of expertise; but what makes them a bureaucrat is the predominance of their administrative duties. Engineers are specialists engaged in productive tasks, while administrators are engaged in maintaining and coordinating the different parts of an organization and, especially, the lower ranks of employees. This distinction is crucial, and it’s pure folly to equate the two. Indeed, the history of managerial thought is full of misguided pretensions by social scientists to derive mechanical formulas for the management of people and groups, the sort of predictable formulas that are deployed more appropriately only in the natural sciences.

What does it do for us, to distinguish between different kinds of expertise? It clarifies the workings of power. Engineers and scientific specialists can have some level of autonomy because their skills are necessary to the organization’s production goals and their work is hard for their managers to understand. Compare this to departments that are strictly administrative:

Here, at the bottom of the scale, we find clerks without real qualifications, employees whose professional training is rudimentary or non-existent. Between these employees and the managing director of the firm, the hierarchy of jobs is a hierarchy of power. The relations of dependence become determinant and to have a function is to define oneself, at each level, with regard to a superior, whether he is a branch supervisor, a departmental supervisor or a manager.8

At some level, any ambition to get promoted requires buying into the power structure of the organization. As even Weber could see, the primary criterion for advancement is not technical skill but discipline.9 As far as top-level managers are concerned, mid-level administrators can perform their duties without technical knowledge but not without loyalty and obedience to their superiors. This dynamic suggests one key route for technically incompetent workers to attain ranks of high status and thus bring their incompetence to bear on important organizational decisions.

But we must also make this point in a less individualistic way, since personnel decisions are just one way this dynamic manifests: bureaucratic units are fundamentally concerned with maintaining and building their power, ahead of all technical or mission-oriented concerns. I say ‘units’ because competition takes place not just at the highest, inter-organizational level, but also within organizations between divisions and departments and teams. Of course, affinities and cooperation are possible at any of these levels, and conflicts are most often subdued or quietly simmering. Yet power (as competition among individuals and among units) and discipline (within units and networks) are the basic structuring principles of bureaucratic life.

Power and discipline are also a primary source of bureaucracy’s never-ending unintended consequences, from ‘mission drift’ in the non-profit sector to misguided government aid programs to ‘collateral damage’ in war-making. Power struggles explain much (if not all) of the dysfunction of bureaucracy, and they are a primary reason why we cannot take bureaucrats’ claims of expertise at face value. In Lefort’s words,

Thus, behind the mask of rules and impersonal relations lies the proliferation of unproductive functions, the play of personal contacts and the madness of authority.10

Flights of Fantasy, Within Bounds

Since bureaucracies are fundamentally driven to build and maintain their power, as well as the sources of their power, it’s no surprise that bureaucracies are basically conservative. Because their worldview is at some level fused with the concerns of their fiefdom and bounded by its horizons, administrative professionals become profoundly incurious about the world and unable to question the conditions of their own knowledge or the values encoded in organizational goals. The institutional filtering mechanisms which prize loyalty and a ‘shared framework’ over knowledge or a well-formed analysis only exacerbate this problem, creating echo chambers where only high-status specialists are allowed to speak and be heard — and here we see how, clarifying my earlier characterization, there is no simple or absolute divergence between technical and administrative expertise. In other words, power principles strongly shape what kinds of intellectual projects are possible.11 In this context, claims based on non-sanctioned frameworks — much less critiques of the institution — are rendered incoherent and fall outside the bounds of what can be discussed or known. Neoclassical economics, for example, bears exactly this sort of relationship to Marxian and even liberal Keynesian economics today, such that the financial executives and academics who run the US Treasury and Federal Reserve treat as completely incoherent and irresponsible any fiscal policy beyond bailouts for large corporations and austerity for the rest (assuming we allow ourselves the fantasy that they take a truly disinterested attitude towards their own material gain and that of the firms they lead).12

Conservatism also drives bureaucratic organizations to avoid implementing any mandate for change, no matter what the origin of that mandate. In firms, this tendency manifests frequently as an aversion to adapting to changes in demand or supply, whether in goods, materials, or labor. The fields of advertising and public relations grew out of large corporations’ attempts to create consumer demand where none previously existed when corporate profits began to stagnate.13 Of course, the pathology of public relations is most obvious in cases like fossil fuel companies’ decades-long efforts to obscure, deny, and derail the science of climate change — again, all in the name of avoiding market- or democratically-imposed limits on their well-established business models. Similar dynamics are familiar in government, as well: since the early twentieth century, the US military has taken full advantage of the most advanced methods in public relations for their efforts at “perception management,” in order to avoid politically-imposed changes to their war-making goals. Elected politicians show this same tendency: the two major parties work hand-in-hand using procedure to reduce their accountability to the voting public, by narrowing electoral choices and maneuvering party insiders into power broker roles, all in the name of avoiding legislating on the generally popular issues which, if addressed, would undermine their fundraising efforts. For the most part, American politicians are party officials first, elected representatives second.

Thus we have a fatal combination: conservative tendencies rooted in a desire for power; an unwillingness to take seriously dissenting views or inconvenient facts; and a highly-developed institutional capacity to project one’s own version of reality. In normal times, bureaucrats use their institutional might to try to shape the context in which they operate; in crisis, they’re driven to astounding flights of fantasy. Henry Kissinger’s famous quote illustrates this sinister tendency:

There are two kinds of realists: those who manipulate facts and those who create them. The West requires nothing so much as men able to create their own reality.14

Claiming the power to adopt such a “radical subjectivism” was a point of pride for Kissinger.15 Yet it’s unclear that this delusional position is objectively much worse than that of the other 1960s-era National Security advisers who preceded him, at least some of whom were more of the mundane manipulate-, ignore-, and suppress-the-facts type: McGeorge Bundy; Robert McNamara; Walt Rostow; Arthur Schlesinger, Jr.; and so on. When the will to power combined so casually with the industrialized means of violence and unbounded hubris, it’s no surprise that the US government’s “best and brightest” perpetrated such catastrophe across Southeast Asia.16 Delusions on this global scale — war-making, or climate change denial — may be the purview of only the highest-level bureaucrats, but it’s notable that ‘suppression of the evidence’ is so routine at even the local level that it hardly garners surprise when it happens: from police departments to college athletics departments, officials’ first impulse is often a delusional drive to avoid consequences by managing public perception or, speaking more plainly, lying.

Advanced Incompetence

What can we say about expert incompetence? The organizational tendencies to dysfunction and the play of power that Marx identified are still major factors in modern institutions, and the rise of a stratum of educated professionals has not changed that. To the contrary, the simultaneous rise and on-going refinement of professional administration has undermined the public goods stemming from technical-scientific expertise, to put it mildly. Administrative madness has regularly driven the whole industrial-technological enterprise towards nefarious ends, whenever powerful institutions, insulated from anything you might call thinking, fall under the sway of actors’ will-to-power and hubris.

Weber’s error — replicated over and over again by later techno-utopians — was to believe that bureaucrats really aimed at forms of efficiency, productivity, and precision that could approximate an abstract and generalizable “rationality.” As Ernest Mandel suggests, each particular bureaucracy entails at best a “partial rationality,” and the sum of all those partial rationalities is a “global irrationality.” In our times, when institutions have shown themselves unwilling or unable to address planetary-scale catastrophe, this global irrationality appears as institutionalized madness.17

There are two principles for devising remedies to these tendencies. The first is the principle of professional autonomy, the need for experts working in the public service to be insulated from the political maneuvers of powerful people. Academic freedom is the strongest model of professional autonomy that still operates today, although medicine, law, social work, civil service, and other professions have their own versions in varying levels of robustness. (Notably, the principle is almost completely alien to corporate practice.) Autonomy is a basic precondition for reining in excesses of authority.

The second principle is accountability. Bureaucracies are fundamentally hierarchical, and it is the higher-ups who bend organizations to their whims. The challenge is to make bureaucracies accountable to a public interest, not to the power plays of administrators. Transparency is one element of accountability, but it’s not sufficient on its own. We must remake institutions and organizations in more democratic ways and grow the capacity of the public to exercise sovereignty over these institutions.18 One dimension of that capacity is promoting expert knowledge and practice that is genuinely public-spirited and democratically inclined, since technical training alone is insufficient for guiding engagement with public problems, no matter how autonomous one’s work is.

☙

  1. Alfred Chandler, The Visible Hand: The Managerial Revolution in American Business. Cambridge, MA: 2002 [1977]; Giovanni Arrighi, Kenneth Barr, and Shuji Hisaeda, “The Transformation of Business Enterprise” in Chaos and Governance in the Modern World System, edited by Giovanni Arrighi and Beverly J Silver, 97–150, Minneapolis: 1999. ↩

  2. For government, see Sabeel Rahman, Democracy Against Domination. Oxford: 2016; for the professions generally, see Steven Brint, In an Age of Experts: The Changing Role of Professionals in Politics and Public Life, Princeton: 1996; for social work and philanthropy, see Michael Katz, In the Shadow of the Poorhouse: A Social History of Welfare in America, New York: 1996. ↩

  3. Max Weber, “Bureaucracy,” p. 214. In From Max Weber: Essays in Sociology, edited by Hans Gerth and C. Wright Mills, 196–244. New York: 1946 [1922]. ↩

  4. Ludwig Von Mises, Bureaucracy, trans. Bettina Bien Greaves. Indianapolis: 2007 [1944]. ↩

  5. Claude Lefort, “What Is Bureaucracy?” p. 100. In The Political Forms of Modern Society: Bureaucracy, Democracy, Totalitarianism, ed. John B Thompson, 89–121. Cambridge, MA: 1986 [1971]. ↩

  6. Weber describes multiple forms of rationality, and I’m glossing a bit here for the sake of space. Donald Levine outlines at least eight types of rationality in Weber’s work. See The Flight from Ambiguity: Essays in Social and Cultural Theory, Chicago: 1985. ↩

  7. A parallel distinction is Randall Collins’s, between “technologies of production” and “technologies of administration and communication”: the latter are the means for manipulating and maneuvering in “organizational politics”; pp. 34-35 in The Credential Society: An Historical Sociology of Education and Stratification, New York: 2019 [1979]. Brint makes a different set of claims, seeing a historical splintering of professional practice between a principled and public-spirited “social trustee” professionalism and a growing, socially agnostic, applied expertise for hire, a moral decline accelerated especially by the creation of business schools in universities and the post-WWII rise of management consulting. He nevertheless acknowledges some of the practical shortcomings in the older ‘trustee’ professionalism, despite its idealism. ↩

  8. Lefort, “What is Bureaucracy?” p. 106. ↩

  9. Collins provides ample empirical evidence of this phenomenon as well as stated justifications by managers. ↩

  10. “What is Bureaucracy?” p. 109. ↩

  11. The history of the professions that Brint describes highlights the declining principle of autonomy of professionals, eroding the priority of high-quality knowledge and a common good over administrative and profit-oriented imperatives. Academia has the strongest protections for professional autonomy enshrined as “academic freedom,” but even this principle has always been under constant threat (see “Declaration of Principles on Academic Freedom and Academic Tenure.” American Association of University Professors, 1915). ↩

  12. Richard Peet, Geography of Power: The Making of Global Economic Policy. London: 2007. On the carefully cultivated ignorance of experts and their institutions more generally, see Linsey McGoey, The Unknowers: How Strategic Ignorance Rules the World. London: 2019. ↩

  13. Stuart Ewen, PR!: A Social History of Spin. New York: 2003. ↩

  14. “Strains on the Alliance,” Foreign Affairs, Jan. 1963. ↩

  15. Greg Grandin, Kissinger’s Shadow: The Long Reach of America’s Most Controversial Statesman, New York: 2016. ↩

  16. David Halberstam, The Best and the Brightest. New York: 2001 [1972]. ↩

  17. Ernest Mandel, Power and Money: A Marxist Theory of Bureaucracy, London: 2010, p. 182. ↩

  18. See Katharine Jackson. “What Makes An Administrative Agency ‘Democratic’?” LPE Project, November 11, 2020. ↩