- Keep it simple. Stick to changing one thing at a time, and make it easy to follow.
- Be specific. Don’t test how “fruit” affects you post-workout. Test how “bananas” or “mangos” or “blueberries” affect you.
- Kill multiple birds with one stone, if possible. If you’re testing how post-workout bananas affect fat loss or gain, consider observing other possible effects, like improved performance or blood sugar readings. You don’t have to, but keep an open mind.
- Start with a sensible premise drawn from reliable sources. Like your own experience, another’s, or results from clinical studies. An example: you forget to eat breakfast and feel stronger during the afternoon lifting session. Was it the lack of breakfast? Maybe; test it.
- Eliminate outlandish premises. I wouldn’t advise testing whether the trans-fats in Crisco are actually harmful by eating a tablespoon every morning, and breatharianism almost assuredly won’t work; the literature is quite consistent.
- Be prepared to discard a hypothesis. Things won’t always work. And things that seemed to work for a while might suddenly stop working. Or, things that seem to work won’t actually work, or they’ll be working for entirely different reasons than you supposed. In other words, you’re probably never going to know with absolute certainty that something is working for the reason you thought it was. Just be ready to ditch failed hypotheses and change them on the fly.
- Don’t extrapolate to others. Just as your experiences didn’t jibe with results from randomized controlled trials, the solutions you discovered from your own experiments may not always work for other people. They’ll have to test it on themselves.
- The beauty of the self experiment is that it acknowledges the persuasive power of personal experience. Ultimately, we are all interested in leading healthy, happy, and productive lives, so we want whatever works. And while studies on other people are valuable, your own reaction to grass-fed butter always trumps what your cardiologist says the studies about other people’s reactions to butter are. The working mother of three, for example, isn’t going to truly know what works for her until she actually tries some stuff out. Conducting a self-experiment, even if it’s totally casual and would never pass the muster of peer review, gives that mother valuable insight into what works and what does not. And it’s easy, free, and gets the mind working.
The possibilities for experiments are virtually endless…
- The effect of sleep duration on next day sprint/lifting/running performance.
- Tolerance of Primal starch source (choose one).
- The effect of evening laptop abstinence on various markers (sexual performance, sleep duration, next day mood).
- The effect of a daily “forest bathing” hour-long session.
1. Easier to learn useful stuff than I expected. In contrast to the rest of life, where things turn out harder than expected, learning useful stuff by self-experimentation was always easier than I expected, in the sense that the benefit/cost ratio was unexpectedly high. I learned useful things I never expected to learn. An example is acne. When I was a grad student I had acne. My dermatologist had prescribed two drugs, tetracycline and benzoyl peroxide. I believed that the tetracycline worked and the benzoyl peroxide did not work. My results showed the opposite. It hadn’t occurred to me that I could be so wrong, nor that my dermatologist could be wrong (he believed both worked), nor that the establishment view (treat acne with tetracycline) could so easily be shown to be wrong.
2. Don’t be afraid of subjective measurements. By subjective measurements I mean non-physical measurements, such as ratings of mood or how rested I felt — what professional researchers call “self-report”. They routinely say self-report is misleading. At first, I wondered if my expectations and hopes would distort the measurements. As far as I can tell, that didn’t happen. Instead, I found such measurements helped me learn plenty of useful stuff I couldn’t have learned without it. For example, I learned how to improve my mood and how to wake up more rested.
3. Complex experimental designs were rarely worth the extra effort. Now and then I tried relatively complex experimental designs (e.g., randomization, a factorial experiment). Usually they were too hard.
4. Run conditions until you get 5-40 days of flat results (flat = what you are measuring is not going up or down). Ideal is 10-20 days. Suppose I want to compare Treatments A and B (e.g., different amounts of butter). I decide to make one measurement/day. The first step would be to do A for several days. I keep doing A until whatever I am measuring (e.g., sleep) stops steadily increasing or decreasing and then run several more days — ideally, 10-20. Then I do B for several days. I keep doing B until my measurement stops changing, then I do 10-20 more days of B. If the B measurements looked different from the A measurements, I would then return to Treatment A. It’s always a good idea to run a treatment until your central measurement stops changing, and then run it longer. How much longer? I’ve found that less than 5 days makes me nervous. Whereas running a condition for more than 40 days of flat results is a wasted opportunity to learn more by trying a different treatment.
5. Data analysis is easy. The most important thing is to plot measurement versus day. It will tell you most of what you want to know. For example, most of the graphs in this paper show whatever I was measuring (sleep, weight, etc.) as a function of day.
6. When you add data, look again at all the data. Each time I collect new data, I plot all of the data, or at least a large chunk of it. This helps spot unexpected changes. For example, each time I measure my weight I look at a plot of my weight over the last year or so. Recently I found that cold showers caused me to gain weight, which I hadn’t expected. If I hadn’t looked at a year of data every time I weighed myself, it would have taken longer to notice this.
7. Don’t adjust your set. My conclusions often contradicted expert opinion. Again and again, other data suggested my self-experimental conclusions were correct, in spite of what the experts said. Acne is one example. Later research supported my conclusion that tetracycline didn’t work. Another example is breakfast. Experts say breakfast is “the most important meal of the day.” I found it caused me to wake up too early. When I stopped eating it, my sleep got better. Other data supported my conclusion. The Shangri-La Diet is a third example. According to experts, it should never work. Hundreds of stories show that it does.
The most useful lesson I learned was also the most basic. You will be tempted to do something complicated. Don’t. Do the simplest easiest thing that will tell you something. The world was always more complicated than I realized. Gradually it sank in: Complicated (experiment) plus complicated (world) = confusion. Simple (experiment) plus complicated (world) = progress.