Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

Consider this succinct counterpoint:

In answer to the question posed by the title of Lehrer’s article, my answer is Yes, there is something wrong with the scientific method, if this method is defined as running experiments and doing data analysis in a patternless way and then reporting, as true, results that pass a statistical significance threshold.

And corrections for multiple comparisons will not solve the problem: such adjustments merely shift the threshold without resolving the problem of overestimation of small effects.

In the interim I’ve learned quite a bit more about science than I knew before, and I suspect that identifying papers as “bad” is often a subjective judgement based on personal biases and community norms. Papers can include mistakes, such as a faulty proof, that would render them objectively bad. But consider this, are papers before a paradigm shift bad? Certainly, in the context in which the papers were published and read, they were considered “good”, and yet a retroactive evaluations of these papers (post-shift) would probably evaluate them in the context of a changed understanding of the relevant scientific theory. In that sense, pre-shift papers could be then considered “bad.”

[And so, even in the hardest of the hard sciences, the quality of a paper can shift with time even though the paper itself stays the same.]

[As a personal example, a professor in my program brought up the topic of a paper that he thought was terrible. The paper demonstrated a process by which programs can be jointly compiled so that when run together in some form of multitasking environment the programs would use fewer resources. At the time, this did not seem like an important problem. Yet now we live in the era of the iPhone. Last I heard, this professor is now at Microsoft. I wonder if he took his narrow vision of computing with him. ]

Maybe you don’t buy Kuhn’s ideas on science, but you would still need some kind of objective basis for forming conclusions about “good” and “bad” papers. And you’d have to recognize at least the possibility that subjective norms could seep into whatever standard you set.

Heffernan’s critique, whatever its merits, has little to do with postmodernism and science, and everything to do with calling out a particular community for bad behavior.

I love science. I love blogs. But I hate ScienceBlogs. I can’t help but think that part of the reason I feel this way is the prevalence of the same high-octane bullshit quoted above on ScienceBlogs and other “science” sites.

I’ve had the opinion that climate scientists should provide expositions of the science of climate change at various levels of sophistication so that interested people from different backgrounds can educate themselves about this issue. Instead we seem to get roughly two types of output from climate scientists: useless press releases and impenetrable scholarly articles.

Ordinarily, this would not be an issue, but climate science is in the unfortunate position of attempting to inform policy, and policy decisions are political, so it seems best to try to influence the debate across many levels of understanding. I’ve said this here before:

… I think it is incumbent on scientists to provide arguments with different levels of justification, so that individuals with different levels of sophistication can access and evaluate the claims given varying backgrounds. But ACRC type claims tend to bucket themselves into two categories: black box arguments (really press releases), and the relatively inaccessible scholarly work. The fact that there are no categories in between is somewhat troubling, especially considering how the policy debate would benefit from those kinds of expositions.

Now that I know about the blog Real Climate, and have at least skimmed the latest from IPCC, I should probably admit that climate scientists are in fact providing more kinds of access to the science than just press releases. However, after stumbling across something like this excellent effort to understand and visualize the debate,  I realize that climate scientists could still be doing so much more.

If you understand everything you are doing, you are a clerk. Nothing wrong with that. Clerks run most things.

[ Aside: I should note that I'm not protectionist with regards to titles like "intellectual." You can clerk by day and intellectualize by night, or on weekends, or when fishing. It's not the kind of club that has membership dues. If you struggle to understand new things, even old things that are new to you, than you are in. If you don't you're not. ]

Anyway, I was pressed into service to explain the meaning behind quotes from the other day.

[ Aside: Here I should pause to mention that my wife sometimes forces me to explain things she already understands, or that she doesn't have time to read herself. ]

I thought I did an okay job of it, so I thought I’d share with you what I understand so far of Karl Popper’s The Logic of Scientific Discovery. The first thing you should know about this book is that it basically is the foundation of our modern understanding of science. If you ever hear about falsifiability as a criterion for scientific theory, you have Popper to thank. The ideas of TLSD are so pervasive in how scientists now view their own work, that one wonders if Popper hasn’t done the unthinkable and settled a philosophical question for good. Since I’m not plugged into the larger philosophy of science community I don’t really know if that is a correct characterization of TLSD or not, but it certainly is an influential text.

So what was Karl Popper’s project? His goal was to characterize science. To do this he tried to identify the formal logic (or rules) that govern the scientific process. Anything that follows the logic of science can rightly be called science, and anything that does not cannot. The logic of science, once identified, characterizes science. This probably seems like a daunting task, but TLSD is written in a way that when read it is at once both obvious and airtight. Part of the trick on Popper’s part is to conveniently avoid some of the harder questions, like where scientific theories come from (e.g. creativity?). In doing so, he is able to assume that there are things called scientific theories, and explore what properties we expect them to have.

One of the first schools of thought that Popper has to deal with when exploring the logic of science, is the idea that science, rather than being entirely logical, contains a psychological or interpretive component (beyond those that Popper has already conceded). If this is true it is problematic for the project, because then any logic of science would also have to include a logic of psychology, thus enlarging the original problem and removing all hope of tractability. Popper sidesteps this problem by noting, as in the first quote, that while observations and perceptions are psychological, observability is not. Observability is a property of the world. The trick here, is that when describing the logic of science, and just the logic, observability is enough. Observability is one (conveniently non-psychological) property of basic statements about the world needed to falsify theories.

[ Aside: I'm being somewhat vague about basic statements. Popper is very careful defining the concept, and to do so with any rigour requires the many pages Popper spends on the topic. For our purposes,  you can think of basic statements as things that are observable, and that if observed, may falsify a theory. Interestingly, there is a kind of dual relationship between basic statements and theories, since theories have to be constructed in order to have falsifying basic statements. There's no circularity in Popper's definition, however, since basic statements exist independent of theories. Theories just partition an already existing set of basic statements into falsifying and non-falsifying ones. ]

By setting up this framework of basic statements and the partitioning power of theories, Popper is then able to argue that theories are an essential part of science, and in particular, just collecting facts about the world is not enough. That is the meaning I draw from the second quote. It is here, however, where I think Popper gets away with too much. He ignores the interesting question of where theories come from. They certainly don’t spring fully formed from the mind. It seems entirely plausible that theories form through some process running in our heads as we are “collecting and arranging our experiences.” But once we have a theory, we are definitely able to explore the world with a purpose.

I will note that one question Popper does not seem to consider is: “What makes a particular scientific pursuit interesting?” I’ve been struggling with this question in my own research, as I sift through a number of silos of work in machine learning and robotics, looking for both the big picture and the motivations behind each community effort. I do think it is possible to pose scientific questions that meet all the criteria of demarcation that Popper spells out, which fail as scientific questions simply because nobody else cares. Indeed, I think such “trivial” science actually comprises the near totality of posable scientific hypotheses. We don’t notice this because of a combination of our own bias and the natural selection bias that any peer reviewed scientific community uses as an organizing social and meritocratic engine.

Where I come from, when someone writes something of the form: “P is not evidence for Q, and here’s why”, it is dishonest to quote that person saying P and use that quote as evidence for Q. If one of my students did this, I would grade her down considerably, and would drag her into my office for an unpleasant talk about basic scholarly standards. If she misused quotes in this way repeatedly, I might flunk her.

I don’t normally agree with George Will, and in fact I don’t in this case. I’m also not particularly interested in entering the climate debate on any side, but I find this piece of reasoning incomplete, and entirely indicative of why “scholarly standards” arguments always end up failing. This is a recapitulation of argument by authority, though in a unique form where the writer blithely assumes the authority of the original text.

“One important detail about the article in the Daily Tech is that the author is comparing the GLOBAL sea ice area from December 31, 2008 to same variable for December 31, 1979. In the context of climate change, GLOBAL sea ice area may not be the most relevant indicator. Almost all global climate models project a decrease in the Northern Hemisphere sea ice area over the next several decades under increasing greenhouse gas scenarios. But, the same model responses of the Southern Hemisphere sea ice are less certain. In fact, there have been some recent studies suggesting the amount of sea ice in the Southern Hemisphere may initially increase as a response to atmospheric warming through increased evaporation and subsequent snowfall onto the sea ice. (Details: http://www.sciencedaily.com/releases/2005/06/050630064726.htm )

Observed global sea ice area, defined here as a sum of N. Hemisphere and S. Hemisphere sea ice areas, is near or slightly lower than those observed in late 1979, as noted in the Daily Tech article. However, observed N. Hemisphere sea ice area is almost one million sq. km below values seen in late 1979 and S. Hemisphere sea ice area is about 0.5 million sq. km above that seen in late 1979, partly offsetting the N.Hemisphere reduction.”

Maybe in Hilzoy’s profession the authority of original texts are inviolate. In many disciplines this is probably a good default assumption. It is, however, not an appropriate view in science. This text attempts to make the argument that P is not evidence for Q, but it does so by referencing black box models of climate change. I personally think Hilzoy should flunk students who draw any conclusions about the relationship between P and Q without knowledge of the referenced climate models and their justification, including apparently, Hilzoy.

What I see happening, is that George Will and the Arctic Climate Research Center are essentially committing the same error by drawing conclusions about P (ice sheet levels) and Q (global warming/cooling) without any complete defense of the models and assumptions involved. By defending one side and not the other, Hilzoy is not applying an equal criterion of evidence to both parties in the debate.

Now you could argue that George Will is not a climate scientist, and that the Artic Climate Research Center is full of them, so we should believe ACRC’s claims without details. To an extent this is true, but I think it is incumbent on scientists to provide arguments with different levels of justification, so that individuals with different levels of sophistication can access and evaluate the claims given varying backgrounds. But ACRC type claims tend to bucket themselves into two categories: black box arguments (really press releases), and the relatively inaccessible scholarly work. The fact that there are no categories in between is somewhat troubling, especially considering how the policy debate would benefit from those kinds of expositions.

So where do they come from? Well, I can think of a couple of sources. One, using the dashboard may be generating quite a few web server requests that my stats provider is misinterpreting as legitimate traffic. Alternatively, maybe those five Google Readers are really really interested in what I have to say. Google may also be scraping my site quite a bit.

Or maybe it’s spam.

I could probably use a service that will calculate better statistics than my host provides, but I want to consider this problem as if I only had access to the messy data that I have, and can’t change it. What to do? This is actually a fairly common dilemma in science. We’d like to have data X (the number of real live individual fans of the site) but instead we have data Y (the number of server requests). How do we go from Y to X?

Now I could try to build a complicated model of how Y and X are related, plug in the Y data (which I have) and then observe the model estimates of the X data. A sufficiently accurate model would provide precisely what I want. I don’t have such a model, but what I do have is a proxy for measuring the amount of spam traffic to the site — the number of real comments divided by the number of spam comments. Of course this actually underestimates the legitimate traffic, because not every visitor comments, but this is a useful ratio for producing a lower bound.

You’ll be happy to know that 99.9% of comments on this site are spam. Conclude what you will (my spam overlords).

Scientists do more than just try to falsify theories, they also formulate new ones when they have falsified an old one. (In the process of falsification they may not succeed, essentially that’s verification.) They also spend a lot of time making sure their data, instruments, data analysis methods (statistics most importantly) etc. are collecting valid and correctly interpreting information so that falsification, verification (or new theory formulation) is based on true empirical evidence and sound methodology.

At least that’s how its supposed to work. Agree that its a human enterprise and often the actual doesn’t live up to the ideal, but that’s no reason to stop insisting on the ideal, is it? Once you start down that path its a slippery slope to no standards at all, and I don’t think we (humans) really want to go there.

I basically agree with this. Commenter Dad happens to be an engineer, which is really just a scientist who rolls up his or her sleeves and does some real work. The point of my series on science, is that Michael Crichton’s argument is subtly but dangerously different from this, and is ultimately an argument that is destructive to the difficult process of bringing scientific knowledge into the world.