Babak Bagheri

Computer programming, “software development,” has been a large enough field to qualify as an “industry” for some time now. Most programmers spend their time figuring out and writing out, in excruciating detail, procedures for machines to follow. To do this effectively, they have to know, in fairly excruciating detail, what procedures other programmers have coded so that their procedures can interact correctly. Can this field continue into the future along the same lines? I don’t believe so.

The reason is actually fairly simple: piling detailed procedures on top of each other is not a scalable activity. This is so even if we use the latest and greatest techniques for organizing these procedures using objects, aspects, patterns, and the like. As the industry has found out, large piles of code become unmanageable, or at least very difficult to manage, whatever organizational techniques are used to create them. In fact, the most successful software projects have managed to contain their most important procedures within a small “kernel.”

Cube Farm awaiting a "Bank of Programmers"

Unfortunately, the current bank-of-programmers approach to software development does not encourage this kind of small-footprint development. There is a deep reason for this. We have been missing an important class of computational kernel: machine learning. As long as the systems we create have to be programmed for every procedural variation, we’re stuck with hordes of programmers and brittle software.

The usual story goes something like this: a beautiful small system is developed; users interact with it and generate wish lists of what changes they’d like, sometimes very loudly; programmers are put to work piling code on top of the original system; the system gains more fans but gradually becomes more and more difficult to maintain; at some point, someone decides that the system needs to be “re-architected” and the cycle begins all over.

Some have argued that the solution is to embrace this cycle, speed it up, and monitor it to make sure the system evolves in “the right direction.” That’s an improvement over the traditional rudderless approach, but it misses the main point: if systems could learn, then much of the piles of code would be unnecessary. In the next few decades, either we take machine learning more seriously, or the software crisis will get worse and worse. And no, “the cloud” isn’t going to help with this in and of itself.

Yes, we are now officially in the oversharing age. You would think this would improve the quality of information we get; that the massive, and increasing, amounts of shared info would be useful. But it’s looking like the opposite is more likely. And again it’s because we’re all too human.

Old-School TV Infomercial

The coined term “infomercial” labels the issues well. The oversharing we see all around us can, and is, very easily subverted to become large-scale exchange of infomercials. Recently, I’ve found myself mistrusting almost all reviews that pose as “independent.” In the past, except in rare cases, advertising was easy to spot. But consider this: you’ll now have to wonder if your “friends” are actually giving you their honest opinions or are they just walking, talking ads. It can get even worse, your friend may just be parroting his friend who was sold by her friend.

There is no conspiracy here. This kind of subversion happens because we are human beings looking after our own positions in the attention-driven economy.

My guess is that in the not too distant future this insidious aspect of “sharing” will push folks to moderate or at least modulate the way they see easily shared information. The novelty will wear off and we’ll become social-info-savvy the same way we became ad-savvy. As this process gets under way, we’ll have to find new ways to get authentic information across. It’ll be harder because the level of trust will be lower.

I’ve already heard folks say that they only trust the very few people they have listed on their well-guarded FOAF lists. Let’s hope the backlash isn’t that dramatic, and the level of general trust doesn’t fall that low.

The value of a good guess can hardly be overestimated. Many have tried to make a science of guessing. The fields of probability and statistics are the results of these attempts. Two men’s ideas in these areas have gained currency in recent years. The two are the Reverend Thomas Bayes and Andrey Andreyevich Markov. As with all ideas that gain popularity, there is a danger that these will become dogma, or at least second-nature, and replace critical thinking. Their approaches are similar in that they both take an axiomatic view of likelihood and its measurement. It’s important to know their assumptions to avoid mis-applications.

The British Reverend’s axiom is also known as “Bayes’ Theorem” or “Bayes’ Rule”. It can be considered a theorem because it can be derived from other “laws” or rules. It says that new observations should affect our probability estimates for guesses in a simple multiplicative way: the probability that a guess is true after a new observation is the simple product of the likelihood of the observation, with the guess taken for granted, and the ratio of the prior probabilities of the guess and the observation. This provides a very simple way to “update our state of knowledge” based on new observations. As you can guess, it is extremely difficult, maybe even impossible, to verify this general rule empirically in any reasonable way. Is the universe really such that uncertainties propagate in this simple way? There is evidence that it may be, but that’s a far cry from taking the rule for granted.

A Markov Model of Congestive Heart Failure Treatment (National Library of Medicine)

The Russian mathematician’s axiom is easier to state. It simply says that the probability of one set of observations immediately following another set is independent of history. At first blush this seems counterintuitive. After all, how can we ignore the past and hope to make a good guess? Markov, of course, never claimed that we could. His axiom just generalizes properties of simple chance events: the probability of winning the lottery does not depend on what the winning numbers have been in the past (in the absence of cheating). His generalization simply says that the right set of observations gives a snapshot that summarizes the history that led to it. In the case of the honest lotto numbers, observations of previous drawings are irrelevant. Markov’s theory then builds on this axiom to consider chains and networks of these history-independent observations (or “states”); systems like these are said to have the “Markov Property.” One can then find interesting conclusions about these Markov chains and networks in the aggregate. It is important to keep in mind that the Markov property is a hypothesis and should not be used indiscriminately because it simplifies the math, or, worse yet, just because it sounds sophisticated.

Again, what does this all have to do with you and me? Calculations based on Bayes’ and Markov’s theories are fairly common now. Some of these calculations are in very critical policy and engineering areas. Just intoning their names gives a claim or a theory a certain level of respectability. We should guard against accepting such derivative claims and theories uncritically and consider whether the underlying assumptions are really applicable, at least in our own work.

Many forces and people have shaped the world of trade and business. In American business, three figures have had, in current jargon, outsized roles. They are Fred Taylor, Al Sloan, and John Patterson. Fred and Al are better known by their full formal names: Frederick Winslow Taylor and Alfred Pritchard Sloan Jr.; John Henry Patterson is not as well known as the others, but his influence has arguably been greater. Why should we care about these historical figures? Well, keep reading.

Time and Motion Study (© NMPFT/Walter Nurnberg/Science & Society Picture Library )

Frederick Taylor has been called “the father of scientific management.” His main idea was, to put it bluntly, that workers don’t know what they’re doing, and unless properly supervised, they’ll slack off. The original efficiency expert, Fred was the fellow who came up with the infamous time-and-motion studies. The upshot of his ideas was, essentially, to treat workers as if they were machines. The extreme forms of these views have been, at least publicly, in disfavor; presumably because we’re in a “knowledge economy.” But Fred Taylor’s basic notions linger and run very deep. The next fellow on our list took the next steps following Fred.

Alfred P. Sloan was the organizer of General Motors. He, almost single-handedly, defined the quintessential American corporation. His approach dovetailed Fred Taylors’ notions very nicely. The bedrock of Alfred Sloan’s General Motors was “professional management.” The idea was that decisions should be made by professionals who, being the best and brightest, “run the numbers” to make sure all will go well. The Sloan approach has been so thoroughly woven into the American business community, that it has almost achieved the status of being self-evident. The main challenges to these views have come from overseas, mainly Japan.

John Patterson defined the American way of selling. During his years at the National Cash Register company, he created a well-regimented army of salesmen. He gave his sales army clear and detailed marching orders that codified the suspect-to-prospect-to-customer conversion process. His techniques were picked up by most other large corporations, chief among them IBM. If you’ve had to deal with a slick salesman, you’ve been exposed to the John Patterson push.

So what does all this have to do with you and me at the beginning of the 21st century? The ideas of these three gentlemen have become so pervasive that they color our thinking without our overt awareness. We should be aware that many of the “truths” we consider self-evident were invented by these three men (and others to a lesser extent) at a specific time and place. The world has changed and will continue to change at a quickening pace. We need to find more appropriate ways to orgranize and sell, otherwise we’ll live in the past with Fred, Al and John. The first step towards exorcising their ghosts is to take automation and interaction more seriously, more seriously than another GM luminary Roger Smith.

Why do we consider some products to have high quality? This is a simple-sounding but actually complex question. The common answer is to repeat the legalism about obscenity: I can’t define good quality, but I know it when I see it. That answer, of course, just begs the question, but there’s a kernel of truth in it. High quality is in large part assigned by a community, sometimes in fairly arbitrary ways.

During the 1990’s Toyota was in the process of establishing Lexus as a high-quality luxury brand. They had noticed that fit and finish (as the Detroit auto makers would have put it) were central to the American perception of quality in an automobile. They had in fact designed the Lexus to have just about the best fit and finish possible at the time. A famous TV commercial touted this attention to fit. In the commercial, a ball bearing rolls over the gaps between the car’s body parts to show how well and consistently they fit together.

Clearly, Toyota was right about the car-buying community’s views on quality; they had made other decisions about quality based on their observations that also hit the spot.

This social nature of judgments about quality means that abstract “six sigma” approaches to quality will likely be ineffective in isolation. One has to focus on what the target audience considers to be of high quality. Statistical measures should be based on these judgment, if it makes sense to use them at all. This is where the interactive web can help. The challenge there is, as mentioned here before, to cut through the large number of voices and be heard.