Once, I was at a company that was considering (and eventually did so) moving its infrastructure over to Java, and there was a discussion about the danger of “Java Shop Politics”. It would seem strange to any non-programmer that a company’s choice of programming language would alter the political environment– these languages are just tools, right? Well, no. In this case, almost all of us knew exactly what was being talked about. Most software engineers have direct experience with Java Shop Politics, and it has a distinct and unpleasant flavor.
When Unix and C were developed, they were designed by people who had already experienced firsthand the evils of Big Software, or monolithic systems comprising of hundreds of thousands of lines of code without attention paid to modularity, that had often swelled to the point where no one understood the whole system. (In the 1970s, these monoliths were often written in assembly language, in which macroscopically incomprehensible code is not hard to create.) The idea behind the Unix environment, as a reaction to this, was to encourage people to write small programs and build larger systems using simple communication structures like pipes and files. Although C is a strictly compiled language and no Lisp-style REPL existed for it, C programs were intended to be small enough that the Unix operating system was an acceptable REPL. This was not so far from the functional programming vision, and far more practical in its time. The idea behind both is to write small programs (functional “building blocks”) that are easy to reason about, and build more complex systems out of them, while retaining the ability to piecewise debug simple components in event of failure. This style of development works extremely well, because it encourages people to build tools for general use, rather than massive projects that, if they fail, render almost all of the effort put into the project useless. As more code is written, this leads to the growth of generally useful libraries and executable utilities. In the big-program model of development, for a contrast, it leads to increasing complexity within one program, which can easily make whole-program comprehension impossible, and make decay (in the absence of forward refactoring) inevitable, regardless of programming language or engineer talent.
I adhere to this small-program mentality. I’m not saying “don’t be ambitious”. Be ambitious, but build large systems while keeping individual modules small. If a file is too big to read (for full comprehension) in one sitting, break it up. If “a program” is too big to read in a week, then it should be respected (even if it runs as one executable) as a system, and systems are harder to manage than single modules. While it is harder, up front, to develop in a modular style, the quality of the product is substantially higher, and this saves time in the long run.
Many software managers, unfortunately, like Big Projects. They like huge systems with names like cool-sounding names like “Rain Man” and “Dexter” that swell to a hundred thousand lines of code and provide features no one asked for. They like having the name of something Big, something their bosses might have heard of, in their story. Big Projects also provide a lot more in the way of managerial control. A manager can’t control the “chaotic”, often as-needed, growth of small-program development, whereas directing a Big Project is relatively straightforward. Make it Object-Oriented, use this set of analytic tools, complete at least 20 of the 34 feature-request tickets in the queue (it doesn’t matter which 20, just do 20) and have it done by Friday.
Beginning around 1990 was a pernicious attempt by software managers to commoditize programming talent. Enter C++, designed with unrelated intent, but a language that made big-program development seem palatable for C. C’s not high-level enough for most application development in 2012, but C++ is not the solution (except in very specific cases from numeric programming where templates allow very-fast code to be written once for a range of numeric types, with no runtime performance overhead) either. For over 90 percent of the applications that use it, it’s the wrong language. Here’s the metaphor I like to use: assembly coding is infantry: as fine-grained as one could want to be, but slow (to write), lumbering along at 3.1 miles per hour. C is a tank. It’s robust, it’s powerful, and it’s extremely impressive and well-designed, but it moves at ground speeds: about 50 miles per hour. That’s what it’s designed to do. Languages like Python and Ocaml are airplanes– very fast, from a development perspective, but not fine-grained at all. C++ exists because someone had a fever dream in which these two classes of vehicles got mixed up and thought, “I’m going to put wings on a fucking tank”. The drag and awkwardness imposed by the wings made it terrible as a tank, but it doesn’t fly well either. Java was invented after a few horrible tank-plane crashes, the realization being that the things are too powerful and fly too fast. It’s a similarly ridiculous “tank-icopter”. It’s not as fast as the tank-plane, and few people enjoy flying them, but it’s less likely to kill people.
It’s not that C++ or Java, as languages, are evil. They’re not. Languages are just tools. Java was designed to run in embedded systems like automatic coffee pots and cable-TV “set top boxes”, so closures were cut for time in the first release, because these use cases don’t require high-level programming features. I haven’t map-reduced a toaster cluster for years. Contrary to popular history, Java wasn’t designed with an ideological, “enterprise” distrust of the programmer. That trend, in the Java community, came later, when the 1990s attempt to commoditize programming talent (a dismal FailureFactory) co-opted the language.
Java and C++ became languages in which “object-oriented programming” was the norm. The problem is that what is currently called OOP is nothing like Alan Kay’s vision. His inspiration was the cell, which hides (encapsulates) immense mechanical complexity behind a simpler interface of chemical and electrical signals. The idea was that, when one needs complexity, simpler interfaces are invaluable, and that complex systems generally should have comprehensible interfaces. Kay was not saying, “go out there and create giant objects” or “use object-oriented programming everywhere”. He was attempting to provide tools for dealing with complexity when it becomes inevitable. Unfortunately, a generation of software managers took “object-oriented” magic and immodularity as virtues. This is similar to the “waterfall” software methodology, named by a person clearly stating it was the worst idea, and yet taken by suits as having been “recommended by some smart guy” once it was given a name.
What’s wrong with Big Project development? First, it encourages reliance on internal vaporware. Important work can be delayed “until Magneto is done”. When Magneto is done, that will solve all our problems. This works for managers seeking to allocate blame for slow progress– that fucking Magneto team can never get their shit done on time– but it’s a really bad way to structure software. In the small-program model, useful software is being continually released, and even an initiative that fails will provide useful tools. In the big-program arena, the project is either delivered in toto or not at all. What if half the Magneto team quits? What if the project fails for other reasons? Then Magneto est perdu. It’s a huge technical risk.
[ETA: when I wrote this essay, I was unaware that a company named Magneto existed. “Magneto”, in this essay, was a potential name for a large project in a hypothetical software company. There is absolutely no connection between these project names used here and real companies. Consider names like “Cindarella” and “Visigoth” to be gensyms.]
Second, under a big-program regime, people are trackable, because most programmers are “on” a single program. This is also something that managerial dinosaurs love, because it provides implicit time-tracking. Mark is on Cindarella, which has a headcount of 3. Sally is on Visigoth, which has a headcount of 5. Alan is on the project that used to be called 4:20 until Corporate said that name wasn’t okay and that now can’t decide if it wants to be called 4:21 or BikeShed.
What this kills, however, is extra-hierarchical collaboration– the lifeblood of a decent company, despite managerial objections. In a small-program software environment, people frequently help out other teams and contribute to a number of efforts. A person might be involved in more than 20 programs, and take ownership of quite a few, in a year. Those programs end up having general company-wide use, and that creates a lot of cross-hierarchical relationships. MBA dinosaurs, alas, hate cross-hierarchical, unmetered, work. That kind of work is impossible to measure, and a tightly-connected company makes it hard to fire people. On the other hand, if a programmer only works on one Big Project at a time, and has no other interaction with other teams, it’s much easier to “reduce headcount”.
The third problem with big-program methodology is that it’s inefficient. Six months can be spent “on-boarding” an engineer into the complexities of the massive Big Project he’s been assigned to work on. In light of the average job lasting two to three years, that’s just intolerable. The on-boarding problem also restricts internal mobility. If someone is a bad fit for his first Big Project, moving him to another means he could spend up to a year just on-boarding, accomplishing zilch. Transfers become unacceptable, from a business perspective, so people who don’t fit well with their first projects are Just Fucked.
Why is the on-boarding problem so severe? Big Projects, like large Java programs in general, tend to turn into shitty, ad-hoc domain-specific languages (DSLs). Greenspun’s Tenth Rule sets in, because programmers tend to compensate for underpowered tools by adding power, but in hasty ways, to the ones they have. They end up developing a terminology that no one outside of them understands. They become systems where to understand any of it requires understanding all of it. This means that people hired to modify or expand these Big Projects have to spend months understanding the existing system, whose intricacies are only known to a few people in the company, before they can accomplish anything.
The fourth problem with the big-program methodology is the titular Java Shop Politics. In a small-program development environment, engineers write programs. Plural. Several. An engineer can be judged based on whether he or she writes high-quality software that is useful to other people in the organization, and this knowledge (talent discovery) is redundant throughout the organization because the engineer is continually writing good code for a wide array of people. What this means is that technology companies can have the lightweight political environment to which they claim to aspire, in which a person’s clout is a product of (visible) contribution.
On the other hand, in a big-program shop, an engineer only works on one Project, and that project is often a full-time effort for many people. Most people in the company– especially not managers– have no idea whether an individual engineer is contributing appropriately. If John is chugging away at 5 LoC per day on Lorax, is that because he sucks, because the team failed to on-board him, or because Lorax is a badly structured project? In a small-program environment where John could establish himself, such a question could be answered. The good programmers and bad projects (and vice versa) could be objectively discovered. In a big-program world, none of that will ever be known. The person with the most clout on Lorax, usually the technical lead, gets to make that assessment. Naturally, he’s going to choose the theory that benefits him, not John. He’ll never admit that Lorax is badly designed or, worse yet, was a mistake in the first place. So under the bus John goes.
In general, it is this fog of war that creates “office politics”. When no one knows who the good and bad contributors are, there are major incentives toward social manipulation. Eventually, these manipulations take more of peoples’ emotional energy than the actual work, and the quality of the latter declines. This is not limited to technology; it’s the norm in white-collar environments. Small-program development is an antidote. Large-program development accelerates (but does not necessarily cause) the poison.
The solution to this is simple: Don’t become a Java Shop. I’m not saying that Java or the Java Virtual Machine (JVM) is evil. Far from it, I think Clojure and Scala (which run on the JVM) are excellent languages, and the JVM itself is a great piece of software. Writing an occasional (small) Java or C++ program won’t destroy a company, obviously. On the other hand, fully becoming a Java Shop (where the vast majority of development is done on large Java programs, and where success relies on understanding defective “design patterns” and flawed “best practices” instead of being able to code) will. There is no avoidance of this; politically speaking, Java Shops go straight down. The quality of engineering, predictably, follows.
The company I discussed earlier, which was one of this country’s most promising startups before this happened, did become a Java Shop, despite furious protest from existing talent. Within weeks, the politics of the organization became toxic. There was an “old team” that adhered to the Unix philosophy, and a “new team” that was all-Java. (No Scala or Clojure; Scala was flirted with and had serious managerial support at first, but they killed their Scala efforts for being “too close” to the old team.) This old/new cleavage ruptured the company, and led to an epic talent bleed– a small company, it lost several engineers in a month. Java Shop Politics had arrived, and there was no turning back.
Here’s what that looks like. First, it’s not that Java (or C++) code is inherently evil. Any decent software engineer will have a passing competency in half a dozen languages (or more) by my age, and these languages are worth knowing. Some Clojure and Scala programs require classes to be written in Java for performance. That’s fine. Where a company starts to slide is when it becomes clear that “the real code” is to be written in Java (or, as at Google, C++) and when, around the same time, big-program methodologies become the norm. This makes the company “feel” managerially simpler, because headcount and project efforts can be tracked, but it also means that the company has lost touch with the ability to assess or direct individual contribution. Worse yet, because big-program methodologies and immodular projects are usually defective, the usual result is that people with bad tastes thrive. Then the company becomes less like a software enterprise and more like a typical corporation, in which all the important decisions are made by the wrong people and decline becomes inevitable.
