Yesterday, I posted a list of the failings of Scrum and the “Agile” fad, and the reviews have been mixed. To be honest, I find the topic of Agile rather boring. I recoil when I encounter it, because it saddles engineers with a bunch of nonsense that has nothing to do with computer science or software engineering, but the more central topic is the fact of an industry that has become really bad at management. “User stories” are a symptom, but the root problem is much deeper.
It’s easy to complain about incompetent managers or “management culture” and make fun of foolish executives when their egos cause them to flush millions of dollars in value down the toilet, but that’s fundamentally an immature person’s game. It’s much more fruitful to look into the soul of a craft or an industry, such as computer programming, beset with open-plan offices and user stories and ask, “How the fuck did this shit come into being?” It didn’t happen in a day, and it wasn’t by accident.
So why is software management typically so bad? What about our industry causes it to be poorly managed? And what can we learn from it, in order to do better?
“Everyone hates” middle management, but it’s important
Middle managers take a lot of flak from above and below, and the stock image of a middle manager isn’t a pleasant one. Whether it’s the horrendous Bill Lumbergh of Office Space, the bumbling Michael Scott in the U.S. version of The Office, or his nastier U.K. counterpart, David Brent; the image of middle management is a deeply negative one: a petty tyrant without vision, or an incompetent lackey with the ruthlessess, but not the charisma, of an executive. This, I think, exists because of a perverse need for the low to identify with the high (royalism) through a shared contempt for the “bourgeois” middle. Often, middle managers get the brunt of the negativity and even blame, for example, for terrible decisions made by executives. Some dickhead higher-up decides impose stack-ranking, but it’s middle managers who get stuck having to fire people, and who end up being the most hated people in the whole row. It’s much easier to get the low to hate the one-rank-up less-low than to overcome their desire to identify with the high.
In truth, the executive/manager distinction is something that upper-tier professional managers (i.e. “executives”) invented for their own benefit, as a way of differentiating themselves from their lower-tier counterparts. Ultimately, the job title of manager isn’t very sexy. Traditionally, a manager is someone who makes decisions pertaining to an asset that someone else owns: a financial manager allocates a wealthy person’s funds, an actor’s manager is a subordinate who manages his reputation, and a corporate manager oversees the deployment of its labor and capital. As managers (or, to use a more icy term, “handlers”) make decisions over someone else’s assets, they’re often distrusted, because the bad ones do a lot of harm to the owners of those assets. A few are unethical, using their superior knowledge over what is being managed to further their own aims rather than the interests of the owners of the resources. Other managers are abandoned when politics turns against them. At any rate, the manager of a resource is officially subordinate to the person owning that resource, and those who choose to be insubordinate are (often rightly) viewed as unethical or even as crooks.
Upper-tier professional managers began to identify themselves as executives in order to get away from the image of a subordinate, claiming a special knowledge of how to run businesses and inventing demand for it. When the manager/executive distinction formed, and to a degree even now, it wasn’t intended to be one of hierarchical rank or pay grade, but of social status. Within a group, social status gives a person the right to define how he or she is evaluated. (In fact, one of my issues with Scrum is that, while it attempts to equalize, it does so by imposing the low-status treatment– frequent requests for estimates, mandatory visibility into one’s work progress, low allowance for self-directed work, emphasis on measurability over quality– on the entire team.) A manager is responsible for putting a defined set of resources (including, and most often in the corporate setting, people) to defined tasks. It’s measurable, and a measurable job is almost always of less status than an intangible one. The job of an executive is… well, unless you’re within that high-status group yourself, you wouldn’t understand it.
Managers have hiring and firing authority but don’t get to decide how they, themselves, are evaluated. Executives, in general, do have that freedom, because their jobs aren’t as rigidly defined. While an executive will often have people (a mix of managers, assistants, and possibly other executives) reporting to him, his job isn’t to impose certain rules or processes over those people. Rather, those people are provided to assist him, not as a “company-owned resource” that he must formally manage, but toward whatever assignment he has devised for himself. Executives can fail just as they may succeed, but they’re afforded the luxury of succeeding or failing on terms that they have set. That’s the perk (and, some would argue, the definition) of being an executive.
With all of this said, being a middle manager (i.e. a manager who does not have the social status of an executive) is a decidedly unsexy job. It’s a description by exclusion: it means that a person has the responsibility of organizing other people (and is therefore exposed to any risk in their performance, in addition to her own fluctuation) but not the social status or true power that would allow her to define her own job and process of evaluation. The fact that middle managers take bumps from below and above shouldn’t be surprising. Executives are only accountable for the upkeep of their own social status in order to remain in the in-crowd, and workers are only accountable for their own performance, but managers are accountable for the performance of many people including themselves. An executive can toss blame for a fuck-up (his or someone else’s) to someone else in the organization, but a manager is stuck with responsibility for her own fuck-ups and those that occur below her (in addition to any, from above, that are thrown onto her). In many organizations, middle managers are forced into being the hardest workers, having to clean up messes made by the minimum-effort players below them and the self-interested, aggressive, and often narcissistic power players above them.
The software industry has, over the decades, de-emphasized middle management. Recognizing it as a job that few people want, it’s factored it out into roles like the “product manager”, who may or may not have reports, the “software architect” (an important role but a dangerous title) and, in some cases, ill-advised pseudo-managerial positions like “scrotum master”. To a large degree, this change has been troubling, because the disappearance of middle management capability within software engineering has made a mess of the industry. Jobs, such as creating an inclusive culture rather than a “brogrammer” culture based on AMOGing, that were traditionally the province of middle management, go undone. Typically, there is no one given the authority to do them, and doing that kind of cultural or managerial work on one’s own initiative (i.e. without formal authority) is extremely dangerous, so people avoid it.
Against “flat hierarchy”
It may surprise people that, while I champion open allocation, I’m against so-called “flat hierarchy”. The two concepts, I think, are orthogonal even if often linked. Open allocation is the idea that programmers (and, perhaps, creative workers in general) ought to be rewarded and encouraged to find more profitable uses of their time and energy. While “engineers get to work on whatever they want” isn’t an effective management strategy, I support removing authoritarian obstacles (e.g. headcount limitations, rigid job descriptions, time-in-grade mandates like Google’s “18-month rule”) that prevent them from taking initiative and enhancing their value to the company by working on something more important than stated executive priorities. It’s not that I think engineers should be able to work on whatever they want; only that they should have the right to allocate their time to existing corporate needs without being required to appeal to a specific person first. Open allocation is about equality of opportunity (i.e. you won’t be told that you can’t work on X because bullshit headcount limitations that are wielded against the politically less empowered) rather than anarchy. What I don’t think can work is “flat hierarchy” or “no middle managers”. It goes against human nature. While persistent hierarchies of people can be (and often are) toxic, we think in hierarchical terms. We group like things with like, we form taxonomies, and we understand complex systems by composing them into simpler ones… and that’s hierarchy. Once there is a certain amount of complexity in anything, humans will demand or impose a hierarchical model over it. This creates a need for people with the power and the social skills to ensure that the conceptual hierarchy is sound, and that any hierarchical relationships among people are congruent with it, and do not outlive their need without two-party consent to their continuance.
Sociologically, this means that most companies with “flat hierarchy” end up with an emergent hierarchy in spite of themselves. Plenty of self-fashioned benevolent executives wish to see a flat hierarchy below them, because it saves them from the onerous task of choosing (or asking the group to choose) formally titled middle managers, who might prove themselves untrustworthy or dangerous once given power. To her credit, the “benevolent executive” might listen equally to the “flat” array of people below her when there are four or six or possibly even fifteen. At fifty people, though, it’s almost certain that some people have a shorter, hotter line to her and her hiring, firing, allocating, and promoting authority. This means, in effect, that they’re now bosses. This is problematic, because the de facto middle managers who emerge, having no formal title or stability in position, have to compete with the rest of the group to hold status. A formally entitled manager, at least on paper, isn’t supposed to compete with his subordinates, because he’s evaluated on group achievement rather than personal glory. An informal manager, held in that position by a perception (not always reality!) of superior individual performance, is required to use that informal power to maintain said superiority, to the detriment of the less powerful.
Furthermore, as I’ve already addressed, there are jobs that only middle managers will do, because either no one wants to do them, or because it is dangerous to do them without formal authority. Conflict resolution is a big one, but there are subtler cultural jobs that emerge. Who tells the young “rock star” with a horrid personality that he needs to stop calling co-workers “pussies”, because even if the young men in his group venerate him and don’t mind him, the women who overhear it find it offensive? (The “benevolent despot” CEO is likely not around when this guy acts up.) Who decides which technical disputes (e.g. Haskell vs. Java) are important and which (tabs vs. spaces) are a waste of time? Most importantly, who mentors incoming talent and informs people of the existing political landscape (as one always exists) in order to prevent people from needlessly damaging their careers and reputations? Software engineers don’t like middle management because they don’t see what middle managers do when they do their jobs well, which is to remove politics. They’d rather pretend that they work in “politics-free” (or “meritocratic”) environments. This means that they are oblivious, because even when organizations run well, politics exists.
As a final side note on this topic, the term “political” (as a negative) is one I find irritating. When someone loses his driver’s license and is fined because he was driving drunk, that’s political, even if it’s what should happen. It’s an exercise of state power for group benefit, at individual expense. It’s politics working well. Complaints about office “politics” or past decisions being “political” are a half-handed way of saying that the decisions or environment are unfair and corrupt. I would prefer that people use those words to describe bad behavior. Don’t say that the decision was “political”; say that it was wrong and unfair.
This common conflation– of all forms of “getting political” with ill-intended political behaviors– causes harm to those who are political toward beneficial ends. One of the most time-honored way for the elite to exercise control over the middle class is to encourage in them a general aversion to “being political”. Thus you hear statements like, “I support equality for women, but I’d never become feminist or get political about it” from well-intended, if somewhat oblivious, middle-class professionals. The corporate upper class can’t say what it means– “We don’t want unions, and even professional guilds we find irritating”– so they say, “Don’t get political”, which sounds enough like the middle-middle-class “Don’t discuss politics or religion at the dinner table” to get a pass. This aversion to “being political” has become so ingrained in software engineering culture that we’ve lost sight of our need for people who are skilled at navigating and mitigating the politics that emerges naturally in groups of people. As a result, we’re becoming one of the most sexist, ageist, and classist industries in the white-collar world. Moreover, we’re still as “managed” as we would be in a more traditional environment. We still face business-driven engineering (as Agile is an attempt to “patch” business-driven engineering, despite the latter being intractably broken) and low social status and widespread incompetence in management. The fact that there are well-studied systemic reasons for technology executives to be, on average, low-quality people doesn’t help either.
Between 1997 and 2015, we’ve seen a lack of desire to fix these problems, because technology has been such a high-margin industry that it’s been able to cover up egregious mismanagement. You don’t see open-plan offices and Scrum in high-profit companies because those practices work; you see them because, if you learned how to make great phones 10 years ago or a leading search engine 18 years ago, you can have terrible management practices and still be so profitable that open-plan offices won’t outright kill you… for a while. I prefer to see the proliferation of mismanagement in software as a positive sign. If our industry is this profitable despite having emasculated the concept of middle management, and further in spite of having drooling, non-technical morons (“failed in private equity, try bossing nerds around”) leading its upper ranks, then what might it be able to do if it were properly run? I’ll answer this: a lot more.
Protect, direct, and eject
So what makes a good manager? To make it clear, the notion of being an executive and of being a manager are orthogonal ones. There are managers, there are executives, and there are managers of executives. The principles that I’m getting into here apply both inside and outside of the executive in-crowd.
Hence we can focus the core job of any manager: protect, direct, and eject. The best subordinates will need to spread their wings in order to remain engaged in the work, but they need to be protected from the political minefields that overperformers often unknowingly enter, and they need to be insulated from executives– in particular, given guidance about which higher-ups are friendly and which ones are toxic malefactors on power trips. With the strongest subordinates, it’s almost a guarantee that they’ll want to take on bigger challenges, and the manager’s job is to protect them politically while they do so. The middling subordinates, who tend to show less initiative and confidence, need to be directed: assigned useful work in order to earn their keep. Finally, if there are negative-impact members of the team, and as a last resort if they cannot be improved, they must be removed from the team (“eject”). There aren’t static percentages that apply to these three jobs, and people can move from one category into another. Ideally, a manager would seek to reform low performers before ejecting them, and “direct” the middling performers toward work that improves their skills and engages them more, thus bringing them into the “protect” group. In the ideal world, the manager’s job is 100% “protect”, because I don’t believe that people are intrinsically disengaged and mediocre (i.e. need to be directed). In the real world and on an average team, it’s probably 35% protect, 63% direct, and 2% eject.
Which of these three jobs do managers prefer? Where is their bias? Is it toward directing or protecting? I may be going out on a limb here, but I think that almost no one enjoys the “eject” job. It’s horrible to have to fire someone. It deserves to be done as a last resort, and while some bemoan that the decision to fire someone is often procrastinated, I prefer for it to be that way than for firing (especially if one cannot afford a generous severance) to be taken lightly. Where I think there is confusion about the manager’s role is between the other two jobs: direct versus protect. People who excel at one of these jobs tend to be bad at the other. Mediocre managers tend to manage up and to the short term, which favors the “direct” job: get the executives what they want, quickly and with no hiccups. Good managers tend to favor the long term and recognize the value of rapport with their best subordinates, so they’re willing to take on the “protect” job: expending effort and political capital to protect the good.
It’s probably not surprising that, over time, the most talented managers end up with the most talented subordinates, and vice versa. In the “protect, direct, eject” framework, this makes sense. The best managers generally want to be protectors and mentors, and they get their pick of people reporting to them, and they get people who don’t need to be told what to do so much as protected from unintended consequences of over-performance. Mediocre managers tend to be “direct”-heavy, and end up with people who need to directed. Finally, the worst managers tend to be “ejectors” (either because they lack the political clout to keep their employees’ jobs, or because they toss blame for their own mistakes, or even because they enjoy firing people) and would be predicted to end up with the worst subordinates, where their job seems to be implementing their terminations (a job that no one wants to do). This seems like a efficient arrangement, though. Shouldn’t talented subordinates be mapped to protectors and mediocre ones be mapped to directors. So what goes wrong?
The first issue is the lack of self-correction. Every system that must evaluate people makes errors, especially if it does so early on (as in the case of most companies, which assign managers on the first day). Moreover, people change over time. I don’t believe that there are people who are inflexibly of the “needs to be directed” or “needs to be ejected” type; people are mostly context, and impressively capable at improving themselves given the right opportunity and encouragement. Managers who are oriented toward directing (i.e. they see their job as telling people what to do) are likely to end up in conflict with strong, independent subordinates. That doesn’t end well. It also goes poorly when capable people are placed under ejectors, as can happen early in their careers. This is what’s behind the Welch Effect: the people most likely to be fired under stack ranking are junior members of underperforming teams (i.e. teams run by ejectors) and it is pathological because, being junior, they typically had the least to do with the team’s underperformance; they’re essentially fired at random.
Reading my assessment, it probably seems obvious that most people are going to consider themselves (as reports into a manager) as being in the “protect” category. Few will admit that they belong in the “direct” category as if it were a native trait of them, and I agree that it’s not a native trait. More often, it comes down to relative priorities. Some people want to take on bigger challenges and need the political support (and protection) that will enable them to do so. Others are happy being told what to do, so long as they can go home at 5:00 and their job duties don’t change too frequently or in an adverse way. Not everyone values autonomy, and thats OK. There’s nothing wrong with either approach to work, and those who prioritize comfort over achievement (which is completely reasonable, especially in an environment where achievement won’t be rewarded commensurate with the compromise of comfort) are often valuable people to have, so long as they’re properly directed. It’s not that such people are inflexibly in a “mediocre worker” category that requires them to be directed more than protected. It’s that their needs and capabilities at a certain time put them there, although a good manager will try to direct them, if they wish to go toward it, up to a higher level of competence; i.e. the “protect” group.
There is, however, a numerical mismatch between the subordinates who are better off protected than directed, and the inclinations of middle managers as a category: there are more talented subordinates (the “protect” category) than managers who view themselves primarily as protectors, and fewer mediocre subordinates (the “direct” category) than managers inclined to direct. Because managers are often rewarded for managing up, and because most corporate executives are in positions with no accountability, those who are picked for the management track are often those who are inclined to direct rather than they are those who would protect. This, I think, is why middle management gets such a bad name: it’s associated with those who value control over achievement. As I recounted in the last essay, there’s a selection process that favors negative traits. Middle management is often defective because the traits that make a person good at managing up are a readiness to control others and to furtively oppose the interests of one’s subordinates. This results in a category of people who are unduly inclined to distrust (and “direct”) while offering little or none of the protection or opportunity that would enable them to manage high performers. In fact, to many of them, the idea that they should protect their subordinates is a foreign concept: as they see it, their subordinates exist to serve them. Of course, it’s not just the incompetence of many in the role that lead to middle management’s negative reputation. As I discussed, front-line managers are often “fall guys” for executive malfeasance and incompetence, as their lack of executive-level social status makes them easy to scapegoat. Peoples’ desire to identify with power (which middle managers often don’t have, while the executives do) leaves them more than ready to dislike their immediate bosses over failures that are actually the fault of higher-ranking people.
Scrum and software management
The software industry has been trying to disintermediate middle management for decades, to mostly negative results. I’ll readily agree that middle management is often a weak link in organizations, and that the quality of people tapped for it is sometimes low (but not as low as that of the people who fail out of private equity and into investor- or executive-level positions in tech). Even still, middle management is often necessary. The job is important. Someone has to protect new talent from the sharp knives of executives and other managers, direct the middling performers so the company functions, and eject the rare person whose behavior is so toxic as to threaten the functioning of the organization. These jobs can’t be delegated to a “self-managing” (or, just as often, emergently managed) group. Protecting is a job that no one in a “flat” organization has the authority to take on, except for the executives (you know, the people who keeps the hierarchy flat and signs the paychecks) who are often too far removed from the bottom to do it. Directing, on the other hand, becomes a job that many people try to do; without clear leaders of the group, you get many who think they’re the leaders and will try to tell others what to do. The long term result of this is incoherence and tension, until the pushiest people gain credibility (usually by taking credit for others’ work) and win favor from above, and become de facto managers. Finally, ejecting is a job that either no one does (because it’s undesirable, except for psychopaths) or that attracts the worst kinds of people, and is then done in a toxic way.
Where do Scrum and Agile fit into all of this? Naively, they appear like a mechanism to remove middle managers from the equation, or push “people managers” off to the side. I’ll certainly agree that there’s a noxious, deep conflict of interest between people management and project (or product) management, because what is best for those being “people-managed” might be bad for the project (i.e. for a talented subordinate to transfer to a team more in line with his interests is something that a middle manager, held accountable for delivery on that team’s project rather than excellent people-managing, might be averse to letting happen). Many middle managers abuse their power, as a single-point-of-failure (SPOF) in a report’s career at the company, in order to get project-related goals (because, typically, they’re evaluated based on visible deliverables rather than effective people managing) done through intimidation, and I think that has led a couple generations of software engineers to conclude that most middle managers are worthless parasites who only manage up. The problem, however, has more to do with how those managers are evaluated (i.e. their need to “manage up”) and that it forces them to favor directing over protecting.
Despite the flaws of the former, when you replace the institution of middle management with “Agile” or Scrum and the illusion of “flat hierarchy”, you rarely get an improvement. Instead, you get emergent middle management and unexpected, unstable power centers. Agile and Scrum ignore, outright, the goal of protecting subordinates (or, sorry, “Scrum team members”). In fact, the often-stated point of the “user stories” and violent transparency and superficial accountability is to make it impossible for middle managers to protect their reports. Agile is, foremost, about directing and ejecting, but it replaces the single higher-up tyrant with a hundred mildly tyrannical neighbors. The formal police state vanishes in favor of a fuck-your-buddy system where instead of having one career-SPOF in a middle manager, you have tens of career SPOFs. The actual effect of this is to delegate the job that ruined middle management– that of”managing up” into executives– to the whole team.
The big problem with this change is that many executives are narcissistic assholes: probably 30 to 50 percent do more harm than good. It comes with the job. As I mentioned earlier, managers have a job (to manage) while executives are effectively unaccountable, because an executive’s real job is to maintain the social status that places them within the corporation’s nobility. So, companies get a mix of good and bad executives and are rarely very swift in removing the bad ones. A good manager with institutional knowledge will know which executives are friendly and which ones are toxic and best avoided. She’ll steer her reports toward interactions with the good executives, and thereby improve her ability and the ability of her team to get things done, while shooing them away from the bad executives who’ll meddle or do worse. Take away that middle manager and replace her with over-eager, politically clueless young developers on a “Scrum team”, and you get chaos. You get no one looking out for that team, because no one can look out for that team. From above, the team is exposed, not protected.
It’s superficially appealing to throw middle managers overboard. It’s a tough job and a thankless one and there are a huge number of people out there doing it badly. The whole “startup” craze (in which young people have been led to overvalue jobs at companies whose middling tier is comprised of “founders” managing up into VCs, rather than traditional middle managers) is based on this appeal: why work at Goldman Sachs and report to “some faceless middle manager” when you can join a startup with a “flat hierarchy” and report directly to the CTO (…and, in truth, have your career dictated by a 27-year-old product manager whom the CTO has known for 8 years and whom he trusts more than he will ever trust you)? I also think that the reflexive rejection of the idea of middle management– why it exists, why it is important, and why it deserves so much more respect than it is given when it is done well– needs to be tossed aside. We haven’t figured out a way to replace it, and the odds are that we won’t do so in this generation. What we do know is that these asinine “methodologies” that trick a team into middle-managing itself (poorly) have got to go.
Conclusion
Middle management is, perhaps surprisingly, both the solution and the problem. It exists. It always will exist. Executives are disinclined to “protect the good” within the organization, and too removed from day-to-day problems to evaluate individual people, in any case. Even at modest size for an organization, the necessary jobs of management– protect, direct, and (as a last resort) eject– cannot be handled by a single person or an executive in-crowd. Pretending that management is an outmoded job is something we do at our peril. Yes, we must acknowledge that it has mostly been done poorly in software for the past 20 years (and possibly for much longer). However, rather than declaring the whole concept obsolete, we have to acknowledge that it is a necessary function– and figure out how to get good at it.
