I may not be able to save the world, but I think I’ve come up with the fundamental idea that will be necessary in order to restore (and, afterward, preserve) the integrity of, at the very least, the software industry.

Software may or may not be “eating the world”, but we’re learning that the class of people who can rigorously solve problems using precise thought is one that ought to be held in high demand. Such people need not necessarily be writing software programs per se, but we’re in a world that cannot grow based on ego and bluster (the traditional traits used to select business leaders) alone. That approach has been played out, and we need competence to get any further. There are a variety of literacies that a leader needs in order to do her job well, and that set of needed capacities is too large for me to tackle right now, so let’s zero in on one important job, which is software engineering. As a society, we need it. We need it, done well. Unfortunately, as a society, we seem to be getting worse at it.

The problem is that most of the people who employ programmers have seen fit, over the past two decades, to diminish this kind of work as much as possible. There have been so many efforts to replace professional programmers with interchangeable, unqualified peons that it is, frankly, sickening. This is, of course, the real purpose of the “Agile Scrum” fad that has infested our industry, as well as the fetishism for open-plan offices. Dan Lyons’s new book, Disrupted, describes the ageist culture that emerges when we allow the industry to drive software, rather than the other way around. We’ve seen competent engineers driven out in favor of cheap, young, and (most importantly) obedient commodity-grade programmers hired to inflate headcount numbers and appease investors. It needs to stop, and now, before this industry gets so thoroughly infested with incompetence and anti-intellectualism that a generation’s worth of work and experience will need to be discarded. So how do we stop it?

Although it shouldn’t be this way, software engineering is pyramidal in shape. For every engineer who gets to work on a real project, there seem to be fifteen grunts only trusted to churn through Jira tickets. We need to reform our professional structure to be more like an obelisk, instead.

Architecturally, a pyramid is the most basic, inefficient way of achieving prominence (or, less pedantically, height). It’s imposing, it’s gigantic, and it historically required massive amounts of slave labor as well as natural resources. Ancient Egyptians built pyramids because they didn’t have the architectural know-how (namely, arches) to build much else at scale. In economics, one invokes the pyramidal shape when describing arrangements that deliver massive yield to the few (at the top of the pyramid) while exploiting the larger lower classes (at the bottom). Hence, the term “pyramid scheme”. One could argue that pyramids (of both kinds) are entropic. A pile of sand or stones will assume an approximately conical shape, and it’s likely that this inspired the (architecturally unsophisticated) pyramids of Ancient Egypt. Likewise, an ill-organized or disorganized set of people will devolve into an oligarchic arrangement, with many supporting the few.

Professional pyramids mean that positional scarcity will exist at every level in the profession (the term “profession”, here, used loosely) as the number of positions declines as an exponential function of experience. It’ll be as hard to stay in the profession as to get in. Except for sociopaths who enjoy social competition for its own sake, most people find this undesirable. Doctors and attorneys wouldn’t be happy if they had to continue fighting for fewer spots with each year of age, even 30 years after graduating. Not only would it be bad for individuals to have that degree of positional competition, but it would also be a waste for society. To push out 50-year-old doctors simply because they “failed” to arrogate managerial positions would be to discard hard-won expertise, and it would generate a culture of inexperience.

Left to their own devices, management hierarchies become pyramidal. If a company of 10,000 people decides on a 12:1 reporting ratio (as is typical, in software) then it’s going to have 9,167 grunts, 764 first-level managers, 64 directors (managers of managers), and only 5 people at the executive level as traditionally defined. Why can’t there be 100 directors instead of 64? There could be, but few companies would want to pay for them if given the choice. That pyramid seems painful, and it would be, if that’s how companies worked. At the top, they often don’t, and I’ve written before about the superior collective intelligence of MBAs as opposed to software programmers. People with elite MBAs aren’t stupid. They realize that they’re going to get screwed under the traditional pyramidal regime. Not only does the pyramid system make it unlikely for people to advance, but they also leave the people in high positions unprotected, because there are too few of them. CEOs promote loyalists not because the companies need so many executives, but because the CEOs are smart enough to realize that sharing rank and prominence is essential for their own survival.

One could argue that business executives, as a group, “conspire” against their companies by working together in order to convince their organizations that they’re more needed than they really are, and that more of them are needed than really are, and that they themselves as a class (executives) can recognize talent in a way that no one else (not shareholders, not workers) can. By doing this, they’ve managed to create massive demand for themselves and their skill set (which is perceived to be more rare than it actually is). On this topic, I’m hesitant to use the word “conspire” because I don’t consider it morally wrong. Corporate life has become a culture of resource extraction, and the executives have simply been the best players at that game. Ultimately, there are very few people who would hesitate, if offered a magical device that would imbue in their employers an extreme and superstitious overvaluation of their work, to use it for personal benefit. Executives have simply managed to do exactly that as a group, rather than as individuals, the latter being far more difficult.

Corporate executives, based on manufactured demand, as well as doctors and attorneys based on actual demand, have taken their leverage and replaced the pyramidal structure with a professional obelisk shape. An obelisk is upright, rising far higher than a pyramid of comparable mass could, until the pinnacle. With a professional obelisk, there’s plenty of room to grow until one encounters positional scarcity. Doctors don’t need to beat down other doctors and climb management hierarchies just to advance; they just need to gain more skills and deliver more value. Likewise, MBA-toting corporate executives might face stiff competition if they want to be Fortune-500 CEOs (because there are only 500 such spots) but they deal with minimal competition if they just want 7-figure incomes and fancy titles by age 50.

An obelisk structure shows that a value is placed on expertise: as long as people continue to improve themselves, there’ll be room for them at a level higher than where they are. Progress will be recognized. Why does the medical profession intentionally create an obelisk-shaped, guild structure? It doesn’t want “Agile” 17-year-olds performing surgery and driving out competent surgeons who didn’t pursue management jobs. It also doesn’t want people taking obscene risks due to an up-or-out culture that persists into one’s 40s and 50s. Finally, it doesn’t want a culture where the median practitioner has less than a decade of experience.

The pyramid is the entropic formation. Without a deliberate professional structure, it will emerge. Why so? An unprotected “profession” (note the oxymoron) will expand until it absorbs the most unqualified people, who draw barely more compensation than they’d get anywhere else, but who are able enough to convince people (unqualified buyers) to buy their services that they can stay afloat. The unqualified and minimally invested (the “quacks”) will, by sheer numbers, outvote the genuine practitioners (who’ve often dedicated their lives to becoming good at something) and dominate the culture– until their incompetence starts hurting people. In medicine and law, this outcome is considered so unacceptable that it’s not controversial for these industries to have professional structures that function as labor cartels. We need to start thinking the same way about software. We need to start driving out the charlatan brogrammers for whom the infantilizing open-plan startup culture was created.

So how do we get an obelisk-shaped, protected profession for software engineers?

Let’s focus on something that hasn’t worked, and look at why it has failed. Many companies have created “engineering ladders” that purport to measure professional development for individual contributors and allow people to grow without becoming managers. There might be a manager-equivalent title (Staff Engineer) and a Director-equivalent one (Principal Engineer, or Architect) and a VP-equivalent one (Fellow). What goes wrong? To start, the disparity in difficulty makes the whole thing extremely self-congratulatory on the part of the managers.

To become a Director- or VP-equivalent engineer, you have to compete against some of the smartest people in the company. It’s legitimately difficult. There are smart, qualified people who don’t make it. If anything, these dual-track systems serve the interests of management by allowing the executives to overstate their value and capability relative to engineers; if one has to be a genuinely great engineer to become Director-equivalent, the thinking is, then engineers will hold Directors and VPs on the much-easier-to-climb managerial tracks in higher regard.

Ultimately, these systems don’t work. Rather, they serve to document and justify an existing pyramidal scarcity. The organization will only allow a small number of engineers to climb into positions with the credibility and compensation afforded to senior managers, because engineers lack the collective organizational skill to thwart the penny-pinching game being played against them. Thus, a pyramid forms, and little is solved, because positional scarcity persists at each level.

I’ve been around for long enough to know that trusting employers is not a strategy. In the long term, it always fails. Don’t get me wrong: there are good companies out there, and there are managerial strategies (e.g. open allocation) that can improve the odds of creating one. Some individual employers probably can be trusted, but this doesn’t scale, because most organizations become pathological and self-serving at some point. So, if we can’t trust employers, then we can’t put stock in employer-assigned titles as a way to insure the credibility of the individual programmer. That means that we need an employer-independent system for doing so. We need a source of credibility for the individual that no employer can touch.

Doctors and attorneys have employer-independent credibility, being vouched-for by their professional associations (in the U.S., the AMA and the ABA). That’s necessary because the core of a profession is that one’s ethical obligations supersede managerial authority. That is, a doctor can’t use the superior orders defense to justify killing a patient. Of course, that only works if the professional has some sort of employer-independent credibility– in which case, losing a job is a transient financial annoyance rather than a possible career killer. If the individual lives entirely on his reputation as made by prior managers, his lack of leverage becomes a lack of moral agency. That is, of course, the problem that most programmers face. We lack moral or even creative autonomy, because we lack employer-independent credibility; once we leave school, our value is derived from the titles and recognitions given to us by profit-seeking corporations.

Some programmers hoped that the open-source world would fill this gap. It can’t. That’s not its job. Evaluating a programmer’s Github profile is a subjective task, and the only objective way of measuring one’s open-source contributions is based on how many people use the software. Here’s the issue, though. Few engineers have the time and resources available not only to build usable open-source projects, but to market their work to the global programming community enough that it actually used, and then to support it once it is. The positional scarcity and power-law distribution of open-source prominence emerge not because the open-source community is full of mean people (it’s not) but because (a) there’s no barrier to entry when it comes to just putting code out there, while (b) most of this code will never be used, because people have an entirely understandable (and, in fact, prudent) aversion to using unproven software.

How do we create an employer-independent credibility for software engineers? As far as I can tell, there are two solutions. The first is one that I hate: to require formal, institutional education. I hate it because it’s hostile to career-switchers, astronomically expensive, and can generate a culture of anti-intellectual credentialism. This leaves the second, which is to create an exam-based system, similar to what the actuarial sciences use. As programmers pass more exams, their professional credibility can grow in an employer-independent way. This would not only give us job security, but the improvement in our general leverage would enable us to negotiate for better working conditions and launch better projects, to the benefit of society as a whole.

Are there problems with exams? Of course there are. For one thing, I happen to believe that “bad test taker” is a real thing. (I was always a good test taker, but I’ve met intelligent people who aren’t.) We’d have to adapt our system to legitimate learning disabilities and to allow project-based alternatives for great programmers who struggle with traditional exams. We’d need to decide what to charge for the exams (as well as the presumably more expensive project evaluation) and how to offer financial aid. We’d have to decide what would be on the exams, and how many there would be. (I, personally, doubt that I’d be qualified to design such a curriculum.) What is an Associate-level programmer, and what’s a Fellow-level engineer, and what do we expect each to know? What are the continuing education requirements going to look like? These aren’t easy problems to solve. On the contrary, they’re quite difficult and organizationally complex.

Here’s the thing, about that. Designing a curriculum and exam track for software engineers is a massive amount of work, much of it ugly. I doubt I’m up to it, personally, because I’d rather have it built by experts in their respective fields of computer science. However, the core idea is here is one that will actually work. Blindly trusting employers hasn’t worked, and fetishizing startups has been a hilarious failure. Over the past twenty years, the programming profession has lost ground. We’ve been infantilized with open-plan offices and “Agile Scrum” practices that senior engineers don’t need. We’ve been left without any form of employer-independent credibility, which is what we’ll need if we want to become a true profession. We’re in a state of squalid failure. Our professional shape is a pyramid rather than an obelisk, making the job painfully political at every level. I’d rather see an imperfect solution (such as a professional curriculum and an employer-independent, exam-based advancement system) that fixes this than no solution at all. This problem won’t be solved by one person and it certainly won’t be solved within a year, but I’m confident that I know, at least, where to start.