Something I’ve observed in a variety of human organizations, including almost all businesses, is that the wrong people are making major decisions. I’m not talking about second-best players or even mediocrities becoming leaders; I’m talking about the rise of people who shouldn’t even be trusted with a bag of rock salt. White-collar social climbers with no more integrity than common con artists are the ones to rise through the ranks, while the most honest people (some deserving, most not) are the ones to stagnate or be pushed out. Why is this happening? It’s not that all successful and powerful people are bad. Some are; most aren’t. The problem is more subtle: it’s that the wrong people are trusted. Good people are probably slightly more likely to succeed than bad people at forming companies, but bad people rise through the ranks and take them over nonetheless. To understand why this happens, it’s important to understand trust, and why it is so easy for a class of people to earn trust they don’t deserve, and to retain that trust in spite of bad actions.

As I work my way through George R. R. Martin’s A Song of Ice and Fire, I’m starting to get a sense of just how well this author understands human nature. Unlike many fantasy novels with clear heroes and cosmic villains, the moral topology of Martin’s world is approached from several dog’s eye views, without omniscient or prescriptive narration. It’s not clear who the heroes and villains are. Charming characters can be treacherous, while those hardest to love are the most interesting. Martin writes using limited third-person narration, but each chapter from a different character’s point of view. What is most interesting is how the perception of a character changes once his or her intentions are revealed. In a novel, you actually can understand the motivations of characters– even dangerous and disliked ones like Jaime Lannister and Theon Greyjoy. You can get the whole story. In real life, people only get their own.

Something emerges as I relate the moral questions posed by narrative to the murkier world of human interaction, and it’s why people (myself included) are generally so awful at judging character. I’ve come to the conclusion that, subconsciously, most of us model the questions of peoples’ trustworthiness with a three-tiered approach. The superficial tier is that person’s speech and social skill. What does he say? The middle tier is the person’s actions. What does he do? The deepest tier is that person’s intention. What does he want? For better or worse, our tendency to separate people into “good” and “evil” relies on our assessment of a person’s true intention, rather than that person’s action.

A person who does seemingly bad things for good purposes is a dark hero, like Severus Snape in the Harry Potter series. A person who does good things for bad intentions (consider the Manhattan charity scene, a theater for social climbing more than service) is a disliked phony. This attitude would make a lot of sense, if we could reliably read peoples’ intentions. We develop first-degree trust in a person if we find that person to be socially pleasant. At this level, we’d invite that person to a party, but not share our deepest secrets. We develop second-degree trust in people who do things we like, and who refrain from doing things we dislike. Most people would call a mutual relationship of second-degree trust a friendship, although friendship involves other axes than trust alone. Third-degree trust is reserved for people we believe have the best intentions: people who might commit actions we dislike (potentially having information we don’t) but who we believe will do the right thing.

If the exploit isn’t visible, I’ll spell it out cleanly. In the real world, one really never knows what another person’s intentions are. That’s pure guesswork. Unlike in fiction, we only know our own intentions, and sometimes not even that. We have a desperate desire to know others’ intentions, but we never will. The quality of evidence available to us, even for the most perceptive and socially skilled people, is poor. So, this admits a hack. What tends to happen when knowledge is impossible to have but people desperately want it? People come up with explanations, and those with the most pleasing ones profit. Many religious organizations and movements exist on this principle alone. That which is said in the right way can appear to betray intentions. In other words, a first-level interaction (what the person says) is dressed up as carrying third-degree knowledge (of intimate intention).

This is how con artists work, but it also explains the operation of white-collar social climbers and the shenanigans that corporations use, in the guise of corporate “culture” and “changing the world”, to encourage naive young people to work three times as hard as they need to, for half the reward. They create a ruse of transparency about their intentions, earning some measure of third-degree trust from the naive. What this allows them to do is be malevolent on the second degree (i.e. perform bad actions, including those harming the finances and careers of their victims) and have a surprising number of loyal acolytes (including victims) making excuses for this behavior.

Essentially, this is the first tier of interaction and trust (the superficial one) overriding the second (of actions) by masquerading as the third (of intentions). It’s an exploit that exists because people don’t want to admit to the true nature of the world they live in, which is one where another person’s intentions are almost always opaque. This doesn’t mean most people are “bad” (not true) or have “hidden agendas” (true but irrelevant, in that all “agendas” are equally hidden)– it’s just the structural nature of a world where minds are very difficult, and sometimes impossible, to read. People have a hard time accepting this limitation, especially because the most socially confident seem not to have it, even though all people do. They compensate by developing the notion that they can read others’ intentions, a foolish confidence in their own social skill.

Some people are easy to read. For example, infants usually cry because they’re cold, hot, hungry, thirsty, or in pain. Children are, likewise, often relatively easy to read. The least socially skilled third of adults are generally easy to understand, at least partially, in this way. Moreover, assessments of motivation are often made as a sort of social punishment for undesirable actions: it’s bad enough for this person to be caught, but the insult is the assessment of his motivation. It’s a paternalistic way of calling someone a child. I know what you’re up to. It’s an assertion of confidence that often has no basis, but it gives a certain class of people confidence in their paternalistic superiority. People with this attitude tend to grow in their foolish confidence as they become more successful and powerful, and to their detriment. As they rise, they need lackeys and lieutenants and advisors. They need to trust people; most of all, they need to believe they can trust peoples’ intentions. Of course, they’ve also been shaped by experience into a person with supreme confidence in their own ability to judge others’ character…

Enter the psychopath. Contrary to popular depiction, most psychopaths are not murderers, rapists, or torturers. The majority of them are not violent, and those with violent intentions are usually able to have others do their dirty work. Most eschew violence, which is dangerous, illegal, and almost never confers any benefit (financial or social) in modern times. They’d rather rob people than kill them– it’s easier, and the rewards are greater. Also, it’s an open question whether psychopathy is “mental illness”, but there is no connection between psychopathy and psychosis, the latter rarely being associated with mental effectiveness or social skill. Instead, psychopaths’ minds tend to be as clear as anyone else’s. What characterizes the psychopath is a lack of conscience and an infinitely deep selfishness. Also, most of them are exceptionally skilled actors. Although their emotional growth tends to be stunted in childhood or adolescence, they can mimic as wide a range of emotions as anyone else. In fact, they are superior to typical people at having the “right” emotions for various circumstances. Psychopaths have no tell-tale signs, and they don’t seem like “mean” people. They are effectively invisible. Among the upper management of most companies, they are surprisingly common, yet never detected until after they’ve done their damage.

Psychopaths could not be more at home than they are in the white-collar social climbing theater of the typical corporation. The outsized rewards for corporate officers feed their narcissism, the intrigues enable their cutthroat tendencies, and their superficial charm enables their effortless rise. They acquire (misplaced) trust quickly, on account of their unusually high skill at emotional mimicry. They are not supernatural, so they cannot read the intentions of those they intend to please. Instead, they dress their intentions in such a way that the people in power will read whatever they want to see. Like “psychics”, they hedge what they say with the purpose of being right by those in power on account of flexible interpretation. They seem to have “vision” and character because they can exploit the “just like me” fallacy of their superiors. In reality, they are the worst kind of mercenary turncoat. Their “vision” is of themselves on top of something, but that could be a mountain of gold or of bones. They don’t care, as long as they win and others lose.

After a psychopath has run his course, the company where he worked is usually damaged immensely. Million- or billion-dollar losses can occur, top executives can be jailed, and thousands of jobs can be cut. Psychopaths burn whatever is no longer useful to them. After this, people tend to back-reason their interactions with that person. “I knew he was up to something.” “I never liked him.” In most cases, that’s not accurate. What really happened is this: it was obvious that this person’s actions (second level) were risky, harmful, or even criminal, but the person was so effective at making it seem that he had the right intentions (third level) that people ignored the obvious warning signs. They made excuses. They misinterpreted the person’s superficial charm as a sign of good intentions, and they were burned. Or, perhaps this word is better: they were hacked.

Functional programming is a ghetto.

Before any flamewars can start, let me explain exactly what I mean. I don’t mean “functional programming sucks”. Far from it. The opposite, actually. Not all ghettos are poor, crime-ridden, and miserable. Jewish ghettos existed for centuries in Europe, from the Renaissance to World War II, and many were intellectual centers of the world. Some were quite prosperous. Harlem was, at one time, an upper-middle-class African-American community at the center of some of America’s most important artistic contributions. The same is true of functional programming. It’s the underappreciated intellectual capital of the programming world, in that its ideas (eventually) trickle down into the rest of the industry, but it’s still a ghetto. A ghetto is the urban analog of a geographic enclave: it’s included in the metropolis, but culturally isolated and usually a lot smaller than the surrounding city. It often harbors those who’ve struggled outside of it. Those who become too used to its comforts view the outside world with suspicion, while those on the outside have a similar attitude of distrust toward those within. Ghettos usually imply that there’s something involuntary about being there, but that’s often not the case. Chinatowns are voluntary ghettos, in the non-pejorative sense, as are some religious communities like monasteries. “Functional programming” is, likewise, a voluntary ghetto. We’ve carved out an elite niche in the software industry, and many of us refuse to work outside of it, but we’re all here by choice.

What is functional programming? Oddly enough, what I’m about to talk about is not functional programming in the purist sense, because most “functional programmers” are not averse to using side effects. The cultural issues surrounding functional programming are not about some abstract dislike of computational effects, but rather an understanding of the necessity of managing the complexity they create, and using tools (especially languages) that make sane development possible. Common Lisp, Scala, and Ocaml are not purely functional languages, but they provide native support for the abstractions that make functional programming possible. What real functional programmers do is “multi-paradigm”– mostly functional, but with imperative techniques used when appropriate. What the debate comes down to is the question of what should be the primary, default “building block” of a program. To a functional programmer, it’s a referentially-transparent (i.e. returning the same output every time per input, like a mathematical function) function. In imperative programming, it’s a stateful action. In object-oriented programming, it’s an object, a more general construct that might be a referentially-transparent function, might represent an action in a hand-rolled domain-specific language (DSL) or might be something else entirely. Of course, most “object-oriented programming” becomes a sloppy mix of multiple styles and ad-hoc DSLs, especially as more than one developer comes to work on an object-oriented project. That’s a rant for later.

In general, functional programming is right. The functional approach is not right for every problem, but there is a right answer regarding the default abstractions for building most high-level programs: immutable data, and referentially transparent functions should be the default, except in special cases where something else is clearly more appropriate. Why? A computational action is, without more knowledge, impossible to test or reason about, because one cannot control the environment in which it exists. One needs to be able to know (and usually, to control) the environment in which the action occurs in order to know if it’s being done right. Usually, the tester wants to be able to cover all special cases of this action, in which case knowing the environment isn’t enough; controlling it is also necessary. But at this point, the state of the environment in which the test happens is an implicit parameter to the action, and making it explicit would make it a referentially-transparent function. In many cases, it’s better to do so– when possible. It might not be. For example, the environment (e.g. the state in a computer cluster database) might be too large, complex, or volatile to explicitly thread into a function. Much of real-world functional programming is not about eliminating all state, but about managing what state is intrinsic, necessary or appropriate.

For a concrete example, let’s say I have a blackboard, face down, with a number (state) on it, and I ask someone to read that (call it n) and erase the number, then write n+1 on the blackboard. I’m assuming he won’t lie to me, and that he’s physically capable of lifting the board; I want to determine if he can carry out this operation. If I don’t know n, and only see what is written after he is done, I have no hope of knowing whether the person carried out my order correctly. Of course, I could control the testing enviroment and write 5 on the blackboard before asking him to do this. If he writes 6, then I know he did what I asked him to do. At that point, though, the blackboard isn’t necessary. It’s more lightweight to just ask him, “what is 5 + 1?” I’ve moved from an imperative style of testing to a functional one: I’m determining whether his model of the addition function gives the right answer, rather than putting him through an exercise (action) and checking the state after it is done. The functional alternative is a unit test. I’m not trying to assess whether he knows how to turn over a blackboard, read it, erase it, and write a new number on it, because I only care about whether he can add. If I want to assess all of those as well, then I need to make an integration test of it. Both types of test are necessary in real-world software engineering, but the advantage of unit tests is that they make it easy to determine exactly what went wrong, facilitating faster debugging.

Testing, debugging, and maintenance are a major component of real-world software engineering, and functional programming gives us the tools to tackle these problems in a tractable way. Functions should be referentially transparent and, ideally, small (under 20 lines when reasonable). Large functions should be broken up, hierarchically, into smaller ones, noting that often these small components can be used in other systems. Why is this desirable? Because modularity makes code reuse easier, it makes debugging and refactoring much simpler (fixes only need to be made in one place) and, in the long term, it makes code more comprehensible to those who will have to modify and maintain it. People simply can’t hold a 500-line object method in their heads at one time, so why write these if we can avoid doing so?

The reality, for those of us who call ourselves functional programmers, is that we don’t always write stateless programs, but we aim for referential transparency or for obvious state effects in interfaces that other programmers (including ourselves, months later) will have to use. When we write C programs, for example, we write imperative code because it’s an imperative language, but we aim to make the behavior of that program as predictable and reasonable as we possibly can.

Functional programming, in the real world, doesn’t eschew mutable state outright. It requires mindfulness about it. So why is functional programming, despite its virtues, a ghetto? The answer is that we tend to insist on good design, to such a degree that we avoid taking jobs where we’re at risk of having to deal with bad designs. This isn’t a vice on our part; it’s a learned necessity not to waste one’s time or risk one’s career trying to “fix” hopeless systems or collapsing companies. Generally, we’ve come to know the signs of necrosis. We like the JVM languages Clojure and Scala, and we might use Java-the-language when needed, but we hate “Java shops” (i.e. Java-the-culture) with a passion, because we know that they generate intractable legacy messes in spite of their best efforts. Say “POJO” or “Visitor pattern”, and you’ve lost us. This seems like arrogance, but for a person with a long-term view, it’s necessary. There are a million “new new things” being developed at any given time, and 995,000 of them are reincarnations of failed old things that are going to crash and burn. If I could characterize the mindset of a functional programmer, it’s that we’re conservative. We don’t trust methodologies or “design patterns” or all-purpose frameworks promising to save the world, we don’t believe in “silver bullets” because we know that software is intrinsically difficult, and we generally believe that the shiniest IDE provides us enough to compensate for its shortfalls and false comforts. For example, an IDE is useless for resolving a production crisis occurring on a server 3,000 miles away. We’d rather use vim or emacs, and the command line, because we know they work pretty much everywhere, and because they give us enough power in editing to be productive.

From a functional programmer’s perspective, it’s easy to mistake the rest of the software industry for “the ghetto” (especially considering the pejorative association, which I am trying to disavow, with that word). Our constructions are stable and attractive, and we do such a good job of cleaning up after ourselves that there’s not much horseshit on our streets. Outside our walls are slums with rickety, fifteen-story tenements that are already starting to lean. The city without is sloppy and disease-ridden and everything built in out there will be burned down, to kill the plague rats, in ten years. We don’t like to go there, but sometimes there are advantages of doing so– for one thing, it’s fifty times larger. If we lose awareness of size and scale and what this means, we can forget that we are in the ghetto. That’s not to say we shouldn’t live in one, for it’s a prosperous and intellectually rich ghetto we inhabit, but a ghetto it is.

I think most functional programmers only get a full awareness of this when we’re job searching, and thanks to most of us being in the top 5% of programmers, our job searches tend to be short. Still, I’ve lost count of the number of times over the past five years that I’ve found a job listing that looked interesting, except for its choice of language. “5 years of experience in Java, including knowledge of design-pattern best practices.” Nope. It might be a good company writing bad copy, but its technical choices look exactly the same as those of the bad firms, so how can I be sure? The process quickly becomes depressing. It’s not that Java or C++ are “dirty” languages that I would never use. It’s that any job that involves using these languages full-time is so likely to suck that it’s hardly worth investigating. Occasionally, C++ and Java are the right tools for the job, but no one should try to build a company on these languages. Not in 2012. Java isn’t a language that people choose to use, not for primary development. Not if they’ve used three or four languages in their career. It’s a language that people make other people use. Usually, it’s risk-averse and non-technical managers making that call. A Java Shop is almost always a company in which non-engineers call the shots.

What we call functional programming is somewhat of a shibboleth for good-taste programming. We prefer the best programming languages, like Ocaml and Clojure, but we don’t actually restrict ourselves to writing functional programs. Do we use C when it’s the right tool for the job? Hell yeah. Do we put mutable state into a program when it makes it simpler (as is sometimes the case)? Hell yeah. On the other hand, we trust the aesthetic and architectural decisions made by brilliant, experienced, gray-bearded engineers far more than we trust business fads. We have a conservative faith in simplicity and ease-of-use over the shifting tastes of mainstream managerial types and the superficial attractiveness of silver bullets and “methodologies”. We roll our eyes when some fresh-faced MBA tells us that structuring our calendar around two-week “iterations” will solve every software problem known to humankind. Unfortunately, this insistence (often in the face of managerial authority) on good taste makes us somewhat unusual. It stands out, it can be unpopular, and it’s not always good for one’s career. Few stand with us. Most leave our camp, either to become managers (in which case, even a Java Shop is a plausible employer) or to accept defeat and let bad taste win. It’s hard to live in a ghetto.

Now, I have little faith in the stereotypical average programmer, the one who never thinks a technical thought after 5:01 pm, and who doesn’t mind using Java full-time because the inevitable slop is the problem of some “maintenance guy”. That person probably shouldn’t be programming. On the other hand, we’re about 2 percent of the software industry, if that, right now. We can reach out. We can do better. We’re not so brilliant that the other 98% of programmers have no hope of joining us. Not even close. There are many IDE-using, Java-hacking, semi-bored developers who are just as smart as we are but haven’t seen the light yet. It’s our job to show it to them, and if we fail to convince them that they could become 2 to 10 times more productive than they ever dreamed of being, and that programming can become fun again, then we’re the ones to blame. We must reach out, and we can probably bring 10, 20, maybe even 30 percent of programmers over to the light side, bringing about dramatic changes in the software industry.

I think the integrity of our industry depends on our ability and willingness to figure out how to do this.

Once, I was at a company that was considering (and eventually did so) moving its infrastructure over to Java, and there was a discussion about the danger of “Java Shop Politics”. It would seem strange to any non-programmer that a company’s choice of programming language would alter the political environment– these languages are just tools, right? Well, no. In this case, almost all of us knew exactly what was being talked about. Most software engineers have direct experience with Java Shop Politics, and it has a distinct and unpleasant flavor.

When Unix and C were developed, they were designed by people who had already experienced firsthand the evils of Big Software, or monolithic systems comprising of hundreds of thousands of lines of code without attention paid to modularity, that had often swelled to the point where no one understood the whole system. (In the 1970s, these monoliths were often written in assembly language, in which macroscopically incomprehensible code is not hard to create.) The idea behind the Unix environment, as a reaction to this, was to encourage people to write small programs and build larger systems using simple communication structures like pipes and files. Although C is a strictly compiled language and no Lisp-style REPL existed for it, C programs were intended to be small enough that the Unix operating system was an acceptable REPL. This was not so far from the functional programming vision, and far more practical in its time. The idea behind both is to write small programs (functional “building blocks”) that are easy to reason about, and build more complex systems out of them, while retaining the ability to piecewise debug simple components in event of failure. This style of development works extremely well, because it encourages people to build tools for general use, rather than massive projects that, if they fail, render almost all of the effort put into the project useless. As more code is written, this leads to the growth of generally useful libraries and executable utilities. In the big-program model of development, for a contrast, it leads to increasing complexity within one program, which can easily make whole-program comprehension impossible, and make decay (in the absence of forward refactoring) inevitable, regardless of programming language or engineer talent.

I adhere to this small-program mentality. I’m not saying “don’t be ambitious”. Be ambitious, but build large systems while keeping individual modules small. If a file is too big to read (for full comprehension) in one sitting, break it up. If “a program” is too big to read in a week, then it should be respected (even if it runs as one executable) as a system, and systems are harder to manage than single modules. While it is harder, up front, to develop in a modular style, the quality of the product is substantially higher, and this saves time in the long run.

Many software managers, unfortunately, like Big Projects. They like huge systems with names like cool-sounding names like “Rain Man” and “Dexter” that swell to a hundred thousand lines of code and provide features no one asked for. They like having the name of something Big, something their bosses might have heard of, in their story. Big Projects also provide a lot more in the way of managerial control. A manager can’t control the “chaotic”, often as-needed, growth of small-program development, whereas directing a Big Project is relatively straightforward. Make it Object-Oriented, use this set of analytic tools, complete at least 20 of the 34 feature-request tickets in the queue (it doesn’t matter which 20, just do 20) and have it done by Friday.

Beginning around 1990 was a pernicious attempt by software managers to commoditize programming talent. Enter C++, designed with unrelated intent, but a language that made big-program development seem palatable for C. C’s not high-level enough for most application development in 2012, but C++ is not the solution (except in very specific cases from numeric programming where templates allow very-fast code to be written once for a range of numeric types, with no runtime performance overhead) either. For over 90 percent of the applications that use it, it’s the wrong language. Here’s the metaphor I like to use: assembly coding is infantry: as fine-grained as one could want to be, but slow (to write), lumbering along at 3.1 miles per hour. C is a tank. It’s robust, it’s powerful, and it’s extremely impressive and well-designed, but it moves at ground speeds: about 50 miles per hour. That’s what it’s designed to do. Languages like Python and Ocaml are airplanes– very fast, from a development perspective, but not fine-grained at all. C++ exists because someone had a fever dream in which these two classes of vehicles got mixed up and thought, “I’m going to put wings on a fucking tank”. The drag and awkwardness imposed by the wings made it terrible as a tank, but it doesn’t fly well either. Java was invented after a few horrible tank-plane crashes, the realization being that the things are too powerful and fly too fast. It’s a similarly ridiculous “tank-icopter”. It’s not as fast as the tank-plane, and few people enjoy flying them, but it’s less likely to kill people.

It’s not that C++ or Java, as languages, are evil. They’re not. Languages are just tools. Java was designed to run in embedded systems like automatic coffee pots and cable-TV “set top boxes”, so closures were cut for time in the first release, because these use cases don’t require high-level programming features. I haven’t map-reduced a toaster cluster for years. Contrary to popular history, Java wasn’t designed with an ideological, “enterprise” distrust of the programmer. That trend, in the Java community, came later, when the 1990s attempt to commoditize programming talent (a dismal FailureFactory) co-opted the language.

Java and C++ became languages in which “object-oriented programming” was the norm. The problem is that what is currently called OOP is nothing like Alan Kay’s vision. His inspiration was the cell, which hides (encapsulates) immense mechanical complexity behind a simpler interface of chemical and electrical signals. The idea was that, when one needs complexity, simpler interfaces are invaluable, and that complex systems generally should have comprehensible interfaces. Kay was not saying, “go out there and create giant objects” or “use object-oriented programming everywhere”. He was attempting to provide tools for dealing with complexity when it becomes inevitable. Unfortunately, a generation of software managers took “object-oriented” magic and immodularity as virtues. This is similar to the “waterfall” software methodology, named by a person clearly stating it was the worst idea, and yet taken by suits as having been “recommended by some smart guy” once it was given a name.

What’s wrong with Big Project development? First, it encourages reliance on internal vaporware. Important work can be delayed “until Magneto is done”. When Magneto is done, that will solve all our problems. This works for managers seeking to allocate blame for slow progress– that fucking Magneto team can never get their shit done on time– but it’s a really bad way to structure software. In the small-program model, useful software is being continually released, and even an initiative that fails will provide useful tools. In the big-program arena, the project is either delivered in toto or not at all. What if half the Magneto team quits? What if the project fails for other reasons? Then Magneto est perdu. It’s a huge technical risk.

[ETA: when I wrote this essay, I was unaware that a company named Magneto existed. “Magneto”, in this essay, was a potential name for a large project in a hypothetical software company. There is absolutely no connection between these project names used here and real companies. Consider names like “Cindarella” and “Visigoth” to be gensyms.]

Second, under a big-program regime, people are trackable, because most programmers are “on” a single program. This is also something that managerial dinosaurs love, because it provides implicit time-tracking. Mark is on Cindarella, which has a headcount of 3. Sally is on Visigoth, which has a headcount of 5. Alan is on the project that used to be called 4:20 until Corporate said that name wasn’t okay and that now can’t decide if it wants to be called 4:21 or BikeShed.

What this kills, however, is extra-hierarchical collaboration– the lifeblood of a decent company, despite managerial objections. In a small-program software environment, people frequently help out other teams and contribute to a number of efforts. A person might be involved in more than 20 programs, and take ownership of quite a few, in a year. Those programs end up having general company-wide use, and that creates a lot of cross-hierarchical relationships. MBA dinosaurs, alas, hate cross-hierarchical, unmetered, work. That kind of work is impossible to measure, and a tightly-connected company makes it hard to fire people. On the other hand, if a programmer only works on one Big Project at a time, and has no other interaction with other teams, it’s much easier to “reduce headcount”.

The third problem with big-program methodology is that it’s inefficient. Six months can be spent “on-boarding” an engineer into the complexities of the massive Big Project he’s been assigned to work on. In light of the average job lasting two to three years, that’s just intolerable. The on-boarding problem also restricts internal mobility. If someone is a bad fit for his first Big Project, moving him to another means he could spend up to a year just on-boarding, accomplishing zilch. Transfers become unacceptable, from a business perspective, so people who don’t fit well with their first projects are Just Fucked.

Why is the on-boarding problem so severe? Big Projects, like large Java programs in general, tend to turn into shitty, ad-hoc domain-specific languages (DSLs). Greenspun’s Tenth Rule sets in, because programmers tend to compensate for underpowered tools by adding power, but in hasty ways, to the ones they have. They end up developing a terminology that no one outside of them understands. They become systems where to understand any of it requires understanding all of it. This means that people hired to modify or expand these Big Projects have to spend months understanding the existing system, whose intricacies are only known to a few people in the company, before they can accomplish anything.

The fourth problem with the big-program methodology is the titular Java Shop Politics. In a small-program development environment, engineers write programs. Plural. Several. An engineer can be judged based on whether he or she writes high-quality software that is useful to other people in the organization, and this knowledge (talent discovery) is redundant throughout the organization because the engineer is continually writing good code for a wide array of people. What this means is that technology companies can have the lightweight political environment to which they claim to aspire, in which a person’s clout is a product of (visible) contribution.

On the other hand, in a big-program shop, an engineer only works on one Project, and that project is often a full-time effort for many people. Most people in the company– especially not managers– have no idea whether an individual engineer is contributing appropriately. If John is chugging away at 5 LoC per day on Lorax, is that because he sucks, because the team failed to on-board him, or because Lorax is a badly structured project? In a small-program environment where John could establish himself, such a question could be answered. The good programmers and bad projects (and vice versa) could be objectively discovered. In a big-program world, none of that will ever be known. The person with the most clout on Lorax, usually the technical lead, gets to make that assessment. Naturally, he’s going to choose the theory that benefits him, not John. He’ll never admit that Lorax is badly designed or, worse yet, was a mistake in the first place. So under the bus John goes.

In general, it is this fog of war that creates “office politics”. When no one knows who the good and bad contributors are, there are major incentives toward social manipulation. Eventually, these manipulations take more of peoples’ emotional energy than the actual work, and the quality of the latter declines. This is not limited to technology; it’s the norm in white-collar environments. Small-program development is an antidote. Large-program development accelerates (but does not necessarily cause) the poison.

The solution to this is simple: Don’t become a Java Shop. I’m not saying that Java or the Java Virtual Machine (JVM) is evil. Far from it, I think Clojure and Scala (which run on the JVM) are excellent languages, and the JVM itself is a great piece of software. Writing an occasional (small) Java or C++ program won’t destroy a company, obviously. On the other hand, fully becoming a Java Shop (where the vast majority of development is done on large Java programs, and where success relies on understanding defective “design patterns” and flawed “best practices” instead of being able to code) will. There is no avoidance of this; politically speaking, Java Shops go straight down. The quality of engineering, predictably, follows.

The company I discussed earlier, which was one of this country’s most promising startups before this happened, did become a Java Shop, despite furious protest from existing talent. Within weeks, the politics of the organization became toxic. There was an “old team” that adhered to the Unix philosophy, and a “new team” that was all-Java. (No Scala or Clojure; Scala was flirted with and had serious managerial support at first, but they killed their Scala efforts for being “too close” to the old team.) This old/new cleavage ruptured the company, and led to an epic talent bleed– a small company, it lost several engineers in a month. Java Shop Politics had arrived, and there was no turning back.

Here’s what that looks like. First, it’s not that Java (or C++) code is inherently evil. Any decent software engineer will have a passing competency in half a dozen languages (or more) by my age, and these languages are worth knowing. Some Clojure and Scala programs require classes to be written in Java for performance. That’s fine. Where a company starts to slide is when it becomes clear that “the real code” is to be written in Java (or, as at Google, C++) and when, around the same time, big-program methodologies become the norm. This makes the company “feel” managerially simpler, because headcount and project efforts can be tracked, but it also means that the company has lost touch with the ability to assess or direct individual contribution. Worse yet, because big-program methodologies and immodular projects are usually defective, the usual result is that people with bad tastes thrive. Then the company becomes less like a software enterprise and more like a typical corporation, in which all the important decisions are made by the wrong people and decline becomes inevitable.

I wrote before on technology’s ethical crisis, a behavioral devolution that’s left me rather disgusted with the society and culture of venture-funded technology startups, also known as “VC-istan”. There are a lot of problems with the venture-funded technology industry, and I only covered a few of them in that post. Barely addressed was that so much of what we do is socially worthless bubble bullshit, like Zynga– which, in my mind, only proves that a company can be taken seriously even when its name sounds like 4th-grade anatomical slang. Most of us in venture-funded technology are merely bankers, except for the distinction that we buy and sell internet ads instead of securities. This world of crappy imitations and bad ideas exists because there’s a class of entrepreneurs (who are well-liked by venture capitalists) who’ve become convinced that “the idea doesn’t matter”. That’s ridiculous! It’s good to pivot, and sometimes one has to change or abandon an idea to survive, but ideas and purposes do matter. When this fast-and-loose attitude is taken toward ideas, the result is that stupid ideas get lots of funding. That’s unpleasant to look at, but it doesn’t have the moral weight of some of VC-istan’s deeper problems, which I’ve already addressed. To pore into those, I think we have to look at a two-word good idea taken too far, and in horribly wrong directions: fail fast.

As a systems engineering term, “fail-fast” is the principle that a failing component should report failure, and stop operation, immediately, rather than attempting to continue in spite of its malfunction. The diametric opposite of this is “silent failure”, which is almost always undesirable. In software engineering, it’s generally understood that an average runtime bug is 10 times as costly as one found in the compilation process, and that a “do-the-wrong-thing” silent bug can be 10 to 1000 times more costly than one that throws a visible error at runtime. In software engineering, redundant systems are usually preferable because components can fail (and they will, for causes ranging from programming errors to hardware defects to data corruption caused by cosmic rays) without bringing the whole system down and, in these, for dysfunctional components to halt fast is usually a desirable behavior.

In the systems case, it’s important to look at what “fail” and “fast” mean. Fail means to stop operation once there is a detected possibility of erroneous behavior. Fast means to report the failure as soon as possible. Whether it’s a bug in software or a defect in a manufacturing process, it’s always astronomically cheaper to fix it earlier rather than later. The idea isn’t to glorify failure. It’s an acknowledgment that failure happens, and it’s a strategy for addressing it. Fail fast doesn’t mean “make things unreliable”. It means “be prepared for unexpected wrongness, and ready to fix it immediately”.

In VC-istan, “fail fast” is an attitude taken toward business, in which failure becomes almost a badge of honor. I believe this is intended as an antidote for the far more typical and pernicious attitude toward business failure, which is to personalize and stigmatize it, as seen in “middle America” and most of Europe. I’ll agree that I prefer the fail-fast attitude over the paralyzing risk aversion of most of the world. The reason Silicon Valley is able to generate technological innovation at a rate faster than any other place is this lack of stigma against good-faith failure. On the other hand, I find the cavalier attitude toward failure to often veer into frank irresponsibility, and that’s what I want to address.

The typical VC startup founder is rich. Without inherited connections, it takes about twelve months worth of work without a salary to produce something that VCs will even look at. (With such connections, VC mentoring comes immediately and a fundable product can be built within about half that time.) Even for the rich and well-connected, it’s dicey. VC acceptance rates are typically below 1 percent, so a lot of good ideas are being rejected, even coming from well-positioned people. Raising money is always hard, but for people who aren’t wealthy, the risk is generally intolerable: twelve months without salary and a high likelihood that it will amount to zilch. Why’s this relevant? Because rich people can afford a cavalier attitude toward failure. Losing a job just means moving vacations around. If one company dies, another can be built.

In an ideal world, everyone would be rich, by which I mean that material limits wouldn’t dominate peoples’ lives and their work in the way they do now. This would be a world of such abundance as to implicitly provide the safety associated with socialism, without the drawbacks, and in which poverty would be eliminated as thoroughly as smallpox. I believe humanity will reach a state like this, but probably not until the end of my lifetime, if not some time after I’m dead. In this “post-scarcity” world, libertarian capitalism would actually be a great system (and so it’s easy to see why out-of-touch rich people like it so much). Business failure would just be the impersonal death of bad ideas, resources would quickly be allocated to the good ones, and people would rise into and fall out of leadership positions as appropriate but could gracefully decline when not needed, rather than having to fire their help, pull their kids out of college, or move halfway across the country when this happens. If everyone were rich, libertarian capitalism would be a wonderful economic system. However, we don’t live in an ideal world. We have to make do with what we have.

In the real world, failure hurts people, and most of those people aren’t 23-year-olds with $5-million trust funds. Investors (not all of whom are rich) lose large amounts of money, and employees get fired, often without notice or severance. Careers of innocent people can be damaged. This doesn’t mean that failure is morally unacceptable. Good-faith failure must be accepted, because if failure leads to broad-based social rejection, you end up with a society where no one takes real risk and no advancement occurs. This isn’t an abstract danger. It’s something that most people see every single fucking day in the typical corporate workplace: a bland, risk-intolerant environment where people are so afraid of social rejection that people torture themselves in order to seem busy and important, but no one is taking creative risks, and real work isn’t getting done. So my attitude toward those who take risk and fail in good faith is one of empathy and, sometimes, admiration. I’ve been there. It happens to almost everyone who wants to accomplish something in this world.

My issue with “fail fast”, and the more general cavalier attitude toward business failure observed in VC-istan, is that people who espouse this mantra generally step outside the bounds of good-faith failure, responsible risk-taking, and ethical behavior. When you take millions of dollars of someone else’s money, you should try really fucking hard not to fail. It’s a basic ethical responsibility not to let others depend on you unless you will do your best not to let them down. You should put your all into the fight. If you give it your best and don’t make it, you’ve learned a lot on someone else’s dime. That’s fine. The problem with “fail fast” is that it sounds to me a lot like “give up early, when shit gets hard”. People with that attitude will never achieve anything.

Usually, the worst “fail fast” ethical transgressions are against employees rather than investors. Investors have rights. Dilute their equity in an unfair way, and a lawsuit ensues. Throw the business away recklessly, and end up in court– possibly in jail. One can’t easily fire an investor either; at the least, one has to give the money back. On the other hand, a remnant of the flat-out elitist, aristocratic mindset that we have to kill the shit out of every couple hundred years (cf. French Revolution) is the concept that investors, socially speaking, deserve to outrank employees. This is absurd and disgusting because employees are the most important actual investors, by far, in a technology company. Money investors are just putting in funds (and, in the case of VC, money that belongs to other people). They deserve basic respect of their interests for this, but it shouldn’t qualify them (as it does) to make most of the important decisions. Employees, for contrast, are investing their time, careers, creative energy, and raw effort, often for pay that is a small fraction of the value they add. Morally speaking, it means they’re putting a lot more into the venture.

I’ve seen too many sociopaths using “fail fast” rhetoric to justify their irresponsible risk-taking. One example of a fail-fast acolyte is someone in his mid-20s whom I once saw manage the technical organization of an important company. I won’t get into too many details, but it’s an ongoing and catastrophic failure, and although it’s evident to me at least (because I’ve seen this shit before) that he is personally headed toward disaster, it’s not clear whether the company will follow him down the drain. (That company is in serious danger of failing an important deliverable because of decisions he made.) I hope it doesn’t. First, he took a scorched earth policy toward the existing code, which was written under tight deadline pressure. (Despite this twerp’s claims to the contrary about the “old team”, the engineers who wrote it were excellent, and the code quality problems were a direct result of the deadline pressure.) I don’t consider that decision an unusual moral failure on his part. Give a 25-year-old programmer the authority to burn a bunch of difficult legacy code and he usually will. At that age, I probably would have done so as well. That’s one very good reason not to give snot-nosed kids the reins to important companies without close supervision. I remember being 18 and thinking I knew everything. A decade later… turns out I really didn’t. Taken too far, the “fail fast” mentality appeals to impulsive young males who enjoy waving a gun around and shooting at things they can’t see and don’t understand.

My second encounter with this person’s “fail fast” sociopathy was in a discussion of hiring strategy, in which he discussed building “30/60/90 plans” for new hires, which would entail milestones that new employees would be expected to meet. As a way of setting guidelines, this is not a bad idea. Technology workplaces are a bit too dynamic for people to actually know what a person’s priorities should be three months in advance, but it’s always good to have a default plan and baseline expectations. New hires typically come on board, in a chaotic environment, not knowing what’s expected or how to “on-board”, and a bit of structure is a useful. This little sociopath wanted to take things a bit further. He thought it would be a good idea to fire people immediately if they missed the targets. New hire takes 35 days to meet the 30-day goal? Gone, after one month. No chance to move to another part of the organization, no opportunity to improve, no notice, no severance, and it’s all made “fair” by putting all new hires on a PIP from the outset. I’m pretty sure, I’ll note, that this young twerp has never been fired himself– and my money’s on him being three to 12 months away from his first experience with it, depending on how fast he can learn that primary executive skill of shifting blame, and how long he can run it. These sorts of terrible ideas emerge when people are permitted to take irresponsible risks with others’ careers. Most of the damaging HR “innovations” companies invent (which become tomorrow’s morale-damaging bureaucratic cruft) occur not because they’re good ideas for the company, but because people within these companies want to propose wacky ideas that affect other people, in the hope that some “greater fool” in upper-management will see the half-baked concept as “visionary” and promote the person who invented it, regardless of the idea’s lack of merit. That’s how Google’s douche-tsunami (douchenami?) system of stack-ranking and “calibration scores”, for just one example, was born.

I don’t like people who are cavalier about failure when they haven’t been on the other side of it, either as an investor who lost a large sum of money, or as a laid-off or unjustly-fired employee. To put it plainly and simply: “failing fast” with other peoples’ risk is not courage. I say this as someone who has taken a lot of risks and failed a few times, who has always accepted the consequences of what he has started, and who has always done everything possible to make sure that anyone taking a risk with me knows what he or she is getting into.

I’m going to advise something altogether different from “fail fast”, because the term “fast” has chronological implications that I don’t find useful. Protracted failures driven by denial are bad, sure. I agree with that aspect of “fast”, but people should try to avoid failure if they can, rather than jumping immediately to declare defeat and move on to a sexier prospect. Fail safely or, at least, smartly. Know what the risks are, disclose them to those who are taking them, and be prepared to address failures that occur. There are cases where chronologically fast failure are appropriate, and there are times when it is not. Largely, the ethics of this come down to what risks the involved parties have agreed to take. People who invest in a startup accept the risk of losing the entire investment in a good-faith business failure, but they don’t accept the risk that the founder will just give up or do something overtly unethical with the money (bad-faith failure). Employees in startups accept the risk of losing their jobs immediately, without severance, if the company goes out of business; but if they’re misled about how much runway the company has, they’ve been wronged.

The ethics of “fail fast” depend largely on the explicit and implicit contracts surrounding failure: how failure is defined, and how it is to be handled. These are conversations people don’t like having, but they’re extremely important. Failures happen. Often these contracts are left implicit. For example, a person who joins a five-person company accepts that if he doesn’t fit well with the project (because a startup of that size only has one project) his employment must end. More on that, being a founder means that one will be (and should be) fired immediately if one doesn’t work well with the rest of the team, just as being a elected official means one accepts the risk of being fired for being unpopular. On the other hand, a person who joins a more stable, large company, does so with the expectation of risk mitigation. Specifically, people join large companies with the understanding that being a poor fit for one’s initial project doesn’t mean leaving the company. The additional robustness of career is a primary incentive for people to join huge companies. Therefore, large companies that impede internal mobility, usually under pretenses of false objectivity in the performance review process, are deeply unethical and their reputations should be tarnished gleefully and often, in order to prevent others in the future from being blown up by undisclosed risks.

The “fail fast” mantra implies that failure is hard, and that it takes a certain fortitude to look failure in the eye and accept the risk. Alone, that’s not hard. Lying down is easy. Quitting on someone else’s risk and dime is not hard. Letting people down is not hard. The hard part is communicating risks as they actually are to people before they get involved, finding people willing to take those risks, working as hard as possible not to let people down, and working even harder to help everyone recover from the loss should failure occur.