depth first search

[The visualization of the week is here.]

I’ve had the opinion that climate scientists should provide expositions of the science of climate change at various levels of sophistication so that interested people from different backgrounds can educate themselves about this issue. Instead we seem to get roughly two types of output from climate scientists: useless press releases and impenetrable scholarly articles.

Ordinarily, this would not be an issue, but climate science is in the unfortunate position of attempting to inform policy, and policy decisions are political, so it seems best to try to influence the debate across many levels of understanding. I’ve said this here before:

… I think it is incumbent on scientists to provide arguments with different levels of justification, so that individuals with different levels of sophistication can access and evaluate the claims given varying backgrounds. But ACRC type claims tend to bucket themselves into two categories: black box arguments (really press releases), and the relatively inaccessible scholarly work. The fact that there are no categories in between is somewhat troubling, especially considering how the policy debate would benefit from those kinds of expositions.

Now that I know about the blog Real Climate, and have at least skimmed the latest from IPCC, I should probably admit that climate scientists are in fact providing more kinds of access to the science than just press releases. However, after stumbling across something like this excellent effort to understand and visualize the debate,  I realize that climate scientists could still be doing so much more.

My wife is one of those ferociously intelligent people who asks questions until she fully understands something. That really minimizes my opportunities for bullshit around the apartment. On the plus side, I end up understanding what I claim to understand just by virtue of being forced to think clearly enough to explain it. This is, as you might imagine, sometimes a stressful process. I consider it a rule that if you are not walking a tightrope right at the edge of your own understanding (and hence in real risk of falling off, should you meet a true skeptic along the way), you aren’t really doing your job as an intellectual or academic (if that is your job).

If you understand everything you are doing, you are a clerk. Nothing wrong with that. Clerks run most things.

[ Aside: I should note that I'm not protectionist with regards to titles like "intellectual." You can clerk by day and intellectualize by night, or on weekends, or when fishing. It's not the kind of club that has membership dues. If you struggle to understand new things, even old things that are new to you, than you are in. If you don't you're not. ]

Anyway, I was pressed into service to explain the meaning behind quotes from the other day.

[ Aside: Here I should pause to mention that my wife sometimes forces me to explain things she already understands, or that she doesn't have time to read herself. ]

I thought I did an okay job of it, so I thought I’d share with you what I understand so far of Karl Popper’s The Logic of Scientific Discovery. The first thing you should know about this book is that it basically is the foundation of our modern understanding of science. If you ever hear about falsifiability as a criterion for scientific theory, you have Popper to thank. The ideas of TLSD are so pervasive in how scientists now view their own work, that one wonders if Popper hasn’t done the unthinkable and settled a philosophical question for good. Since I’m not plugged into the larger philosophy of science community I don’t really know if that is a correct characterization of TLSD or not, but it certainly is an influential text.

So what was Karl Popper’s project? His goal was to characterize science. To do this he tried to identify the formal logic (or rules) that govern the scientific process. Anything that follows the logic of science can rightly be called science, and anything that does not cannot. The logic of science, once identified, characterizes science. This probably seems like a daunting task, but TLSD is written in a way that when read it is at once both obvious and airtight. Part of the trick on Popper’s part is to conveniently avoid some of the harder questions, like where scientific theories come from (e.g. creativity?). In doing so, he is able to assume that there are things called scientific theories, and explore what properties we expect them to have.

One of the first schools of thought that Popper has to deal with when exploring the logic of science, is the idea that science, rather than being entirely logical, contains a psychological or interpretive component (beyond those that Popper has already conceded). If this is true it is problematic for the project, because then any logic of science would also have to include a logic of psychology, thus enlarging the original problem and removing all hope of tractability. Popper sidesteps this problem by noting, as in the first quote, that while observations and perceptions are psychological, observability is not. Observability is a property of the world. The trick here, is that when describing the logic of science, and just the logic, observability is enough. Observability is one (conveniently non-psychological) property of basic statements about the world needed to falsify theories.

[ Aside: I'm being somewhat vague about basic statements. Popper is very careful defining the concept, and to do so with any rigour requires the many pages Popper spends on the topic. For our purposes,  you can think of basic statements as things that are observable, and that if observed, may falsify a theory. Interestingly, there is a kind of dual relationship between basic statements and theories, since theories have to be constructed in order to have falsifying basic statements. There's no circularity in Popper's definition, however, since basic statements exist independent of theories. Theories just partition an already existing set of basic statements into falsifying and non-falsifying ones. ]

By setting up this framework of basic statements and the partitioning power of theories, Popper is then able to argue that theories are an essential part of science, and in particular, just collecting facts about the world is not enough. That is the meaning I draw from the second quote. It is here, however, where I think Popper gets away with too much. He ignores the interesting question of where theories come from. They certainly don’t spring fully formed from the mind. It seems entirely plausible that theories form through some process running in our heads as we are “collecting and arranging our experiences.” But once we have a theory, we are definitely able to explore the world with a purpose.

At the beginning of the summer, I always plan on reading all the things I did not quite get to during other seasons of the year. Though I am often too ambitious, I am making some progress on a number of fronts. The first is Karl Popper’s landmark The Logic of Scientific Discovery. As I read through Popper’s ideas on the logic of science I am struck by how pervasive his ideas have become among the scientific establishment. No, that’s not quite right, his ideas are a pervasive part of our cultural view, not the establishment, of science. I’m not sure if Popper was the first to formulate falsifiability as a rigorous philosophical criterion demarcating scientific hypotheses, but reading his clear exposition of the concept certainly makes his ideas ring true.

I will note that one question Popper does not seem to consider is: “What makes a particular scientific pursuit interesting?” I’ve been struggling with this question in my own research, as I sift through a number of silos of work in machine learning and robotics, looking for both the big picture and the motivations behind each community effort. I do think it is possible to pose scientific questions that meet all the criteria of demarcation that Popper spells out, which fail as scientific questions simply because nobody else cares. Indeed, I think such “trivial” science actually comprises the near totality of posable scientific hypotheses. We don’t notice this because of a combination of our own bias and the natural selection bias that any peer reviewed scientific community uses as an organizing social and meritocratic engine.

Hilzoy on Will:

Where I come from, when someone writes something of the form: “P is not evidence for Q, and here’s why”, it is dishonest to quote that person saying P and use that quote as evidence for Q. If one of my students did this, I would grade her down considerably, and would drag her into my office for an unpleasant talk about basic scholarly standards. If she misused quotes in this way repeatedly, I might flunk her.

I don’t normally agree with George Will, and in fact I don’t in this case. I’m also not particularly interested in entering the climate debate on any side, but I find this piece of reasoning incomplete, and entirely indicative of why “scholarly standards” arguments always end up failing. This is a recapitulation of argument by authority, though in a unique form where the writer blithely assumes the authority of the original text.

“One important detail about the article in the Daily Tech is that the author is comparing the GLOBAL sea ice area from December 31, 2008 to same variable for December 31, 1979. In the context of climate change, GLOBAL sea ice area may not be the most relevant indicator. Almost all global climate models project a decrease in the Northern Hemisphere sea ice area over the next several decades under increasing greenhouse gas scenarios. But, the same model responses of the Southern Hemisphere sea ice are less certain. In fact, there have been some recent studies suggesting the amount of sea ice in the Southern Hemisphere may initially increase as a response to atmospheric warming through increased evaporation and subsequent snowfall onto the sea ice. (Details: http://www.sciencedaily.com/releases/2005/06/050630064726.htm )

Observed global sea ice area, defined here as a sum of N. Hemisphere and S. Hemisphere sea ice areas, is near or slightly lower than those observed in late 1979, as noted in the Daily Tech article. However, observed N. Hemisphere sea ice area is almost one million sq. km below values seen in late 1979 and S. Hemisphere sea ice area is about 0.5 million sq. km above that seen in late 1979, partly offsetting the N.Hemisphere reduction.”

Maybe in Hilzoy’s profession the authority of original texts are inviolate. In many disciplines this is probably a good default assumption. It is, however, not an appropriate view in science. This text attempts to make the argument that P is not evidence for Q, but it does so by referencing black box models of climate change. I personally think Hilzoy should flunk students who draw any conclusions about the relationship between P and Q without knowledge of the referenced climate models and their justification, including apparently, Hilzoy.

What I see happening, is that George Will and the Arctic Climate Research Center are essentially committing the same error by drawing conclusions about P (ice sheet levels) and Q (global warming/cooling) without any complete defense of the models and assumptions involved. By defending one side and not the other, Hilzoy is not applying an equal criterion of evidence to both parties in the debate.

Now you could argue that George Will is not a climate scientist, and that the Artic Climate Research Center is full of them, so we should believe ACRC’s claims without details. To an extent this is true, but I think it is incumbent on scientists to provide arguments with different levels of justification, so that individuals with different levels of sophistication can access and evaluate the claims given varying backgrounds. But ACRC type claims tend to bucket themselves into two categories: black box arguments (really press releases), and the relatively inaccessible scholarly work. The fact that there are no categories in between is somewhat troubling, especially considering how the policy debate would benefit from those kinds of expositions.

When I look at the statistics for this site, I’m often startled by the number of page views I see in a given month. I’ve had months with 20,000 page views, certainly not a lot by any kind of commercial metric, but for a back water blog with approximately 5 regular readers (judging by the number of Google Reader subscribers), the number of page requests seems way too large.

So where do they come from? Well, I can think of a couple of sources. One, using the dashboard may be generating quite a few web server requests that my stats provider is misinterpreting as legitimate traffic. Alternatively, maybe those five Google Readers are really really interested in what I have to say. Google may also be scraping my site quite a bit.

Or maybe it’s spam.

I could probably use a service that will calculate better statistics than my host provides, but I want to consider this problem as if I only had access to the messy data that I have, and can’t change it. What to do? This is actually a fairly common dilemma in science. We’d like to have data X (the number of real live individual fans of the site) but instead we have data Y (the number of server requests). How do we go from Y to X?

Now I could try to build a complicated model of how Y and X are related, plug in the Y data (which I have) and then observe the model estimates of the X data. A sufficiently accurate model would provide precisely what I want. I don’t have such a model, but what I do have is a proxy for measuring the amount of spam traffic to the site — the number of real comments divided by the number of spam comments. Of course this actually underestimates the legitimate traffic, because not every visitor comments, but this is a useful ratio for producing a lower bound.

You’ll be happy to know that 99.9% of comments on this site are spam. Conclude what you will (my spam overlords).

We are moving to a new apartment this weekend. That combined with some rather stressful events this week have delayed part three in my series on science. I wanted to hoist the following from the comments (from commenter Dad):

Scientists do more than just try to falsify theories, they also formulate new ones when they have falsified an old one. (In the process of falsification they may not succeed, essentially that’s verification.) They also spend a lot of time making sure their data, instruments, data analysis methods (statistics most importantly) etc. are collecting valid and correctly interpreting information so that falsification, verification (or new theory formulation) is based on true empirical evidence and sound methodology.

At least that’s how its supposed to work. Agree that its a human enterprise and often the actual doesn’t live up to the ideal, but that’s no reason to stop insisting on the ideal, is it? Once you start down that path its a slippery slope to no standards at all, and I don’t think we (humans) really want to go there.

I basically agree with this. Commenter Dad happens to be an engineer, which is really just a scientist who rolls up his or her sleeves and does some real work. The point of my series on science, is that Michael Crichton’s argument is subtly but dangerously different from this, and is ultimately an argument that is destructive to the difficult process of bringing scientific knowledge into the world.

Part two in my response to this.

The section of the speech on nuclear winter is the most convincing part of the argument. It is also involves a particular piece of history that I don’t know very well, so I am forced to take Michael Crichton’s retelling of the facts at face value. Even in this section though, I think we can draw out an important distinction that this speech blurs.

First, you should know that I don’t view science as a kind of pure Aristotelean endeavor. Science is a human process whose end result has very special properties. The core property is falsifiability. Michael Crichton observes:

Science … requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world.

To restate: A scientific result is verifiable in some way that is independent of who does the verifying. This is true but incomplete. I’ll ask a simple question, where do these verifiable results come from?

Certainly they don’t appear fully fledged from the minds of scientists. They arise organically out of a process that’s designed to search for these kinds of verifiable truths. Even Michael Crichton references the messy process of science, if only to further blur the distinction between the ends and the means:

The greatest scientists in history are great precisely because they broke with the consensus.

So what is consensus to science? Certainly a verifiable result provides (trivial) consensus. But the kind of consensus referenced here applies to the process of doing science more than the product of doing science. Consensus forms naturally around hypotheses. Scientists tend to be interested in particular hypotheses as groups. Maybe the hypotheses being explored are relevant to some problem people as a whole are facing. Maybe the problem is a bottleneck in the theory that is preventing an entire field from moving forward. Maybe the problem isn’t any of those things, maybe it’s just a fad.

The point is that consensus forms around hypotheses as a part of the process of science, science as a human endeavor. This has always been the case (as the quote above from Michael Crichton backhandedly admits). It isn’t even a particularly grievous problem, because the end result of science has the verification property, so we often do not have to worry about whatever messy process actually produced a result. We can check it.

So when Michael Crichton says something like:

Let’s be clear: the work of science has nothing whatever to do with consensus.

Keep in mind that the process of science often has everything to do with consensus, even if the end result has nothing to do with consensus.

The first part of my response to this.

When did “skeptic” become a dirty word in science? When did a skeptic require quotation marks around it?

I’m going to interpret this rhetorical question as an invitation to skepticism. Let’s first consider the argument against SETI.

I’m going to cut through directly to the punchline.

The Drake equation cannot be tested and therefore SETI is not science. SETI is unquestionably a religion. Faith is defined as the firm belief in something for which there is no proof.

In fairness, I need to summarize the argument preceding these statements. Fortunately, XKCD has already done so:

Now let’s decompose the punchline into its component parts. The first sentence:

The Drake equation cannot be tested and therefore SETI is not science.

For those of you who don’t know, SETI stands for the Search for Extra-Terrestrial Intelligence. The project involves listening to radio signals on radio telescopes, though the project has broadened somewhat to describe a constellation of different approaches, each of which is attempting to find evidence of extra terrestrial life. The “listening” usually involves pattern analysis over large swathes of radio frequency, usually with computers. Though this hasn’t led to the discovery of intelligence, it has led to some novel advancements in distributed computing technology.

Michael Crichton’s argument against the status of SETI as science makes sense if the hypothesis that motivates the SETI program, that noisy intelligent life exists in radio range of Earth, had anything more than a cursory relationship to the Drake equation. Fortunately for science, SETI, and us, but unfortunately for Michael Crichton’s argument, the relationship between the Drake equation and the SETI hypothesis is far more benign.

Actually, I should be more precise on this point, because it is an important one. Let’s examine two possible outcomes from SETI work. In the first, SETI discovers patterns in radio frequencies that exhibit, with little doubt, the existence of intelligence elsewhere in the universe. What are the implications for the Drake equation? Well, N > 2 instead of N > 1. And … everything else remains unchanged.

The discovery of intelligent life would be an astounding scientific discovery. It would also tell us next to nothing we don’t already know about the Drake equation. This should indicate just how useless the Drake equation is, even if SETI turns out to be spectacularly successful.

In the second case, SETI finds nothing. This is a more difficult case, because it could happen that intelligent life does exist in the universe, even if it is not noisy life within range and frequency of SETI telescopes and pattern recognition software. Then the narrow hypothesis under which SETI operates turns out to be false, but it is false in a very unsatisfying way. It is disproved by induction.

Disproved by induction.

I want to dwell on this point a bit, because I think it lies at the heart of a number of popular criticisms of science. The problem is that induction is not logically valid. Plato would not approve. Induction does not come to us from the world of perfect forms. In fact, induction is the cause of our separation from this world, living as we do beyond the Humian divide.

Let me make this clear: If you don’t allow for induction in science than you have to throw away everything except math.

So let’s accept that exhaustive search is sufficient to disprove a sufficiently narrow claim. What does a false SETI hypothesis say about the Drake equation? Nothing.

So, with this in mind, let’s return to the point of contention:

The Drake equation cannot be tested and therefore SETI is not science.

This statement has the logical form A (the Drake equation cannot be tested) then B (SETI is not science). As I’ve argued, A (the Drake equation cannot be tested) happens to be true, but B (SETI is not science) is not something we can conclude from the truth of A.

I’m close to running over some kind of self-imposed word limit. So I’ll stop here for now.

My Dad sent me the following link to a speech by Michael Crichton on the commingling of science and politics: http://www.michaelcrichton.net/speech-alienscauseglobalwarming.html. It is a well crafted speech that contains many arguments worth considering.

I’m going to spend a couple of posts eviscerating those arguments. Stay tuned.

Ever wonder what it’s like to visit Antarctica? A colleague of mine is doing just that. You can follow along on her blog.

Check out the research project she’s working on here. There’s even a TED talk.