Designing N of 1 Experiments [N of 1, Part 2]
Designing N of 1 Experiments [N of 1, Part 2]
This post lays out a step by step process to help ensure you get the most value out of your health and performance experiments.
This is the second article in my series on n of 1 experiments. If you’d like to read the first article on why they’re essential for taking control of your health and performance, you can find it here.
Even if you last encountered experiments in high school science class, designing them to improve your health and performance is a relatively simple process. In the sections below, I lay out the step-by-step process we use at Fount for designing n of 1 experiments and discuss areas of nuance that can make them more effective. Following this process will help ensure you get the most out of the time, effort, and money you put into your health. If you want a cheat sheet version of this, you’ll find it near the bottom of the article.
Core Elements
The key to designing effective experiments is to lay out what you’re going to test, how you’re going to do it practically, how you’re going to measure the results, and any safety considerations. This ensures you keep your approach consistent enough and track the data you need to have trustworthy results you can use to make decisions.
Intervention
The first step is to identify what you’re going to be testing, how to do it, and what you will need to run the experiment. You can ask these 3 questions:
What tool or technique do you want to test? Maybe a friend recently found a supplement helpful, you heard about getting sunlight exposure in the morning on the Huberman Lab podcast, or you’ve been doing research into what the best options are for optimizing focus.
How does it work? Regardless of the source, you want to ensure you know enough details to create an action plan. Sometimes, an intervention is as simple as getting sun exposure in the morning, but do you need to be outside? Can you get light through a window? Does it need to be open? You don’t want to spend a couple of weeks trying something only to discover you haven’t been doing it right and need to start again.
What resources do I need for this intervention? Some interventions may require media, like a guided meditation track, or things to buy, like glasses or supplements. Just as it’s important to know any nuances of using the tool, it’s key to understand if there are any in what you need to buy. For example, lots of people tell me they’ve already tried blue light blocking glasses to improve sleep. When I ask what color the lenses were, they often tell me they’re clear. This means that the lenses block less than 35% of blue light and won’t be very effective in helping sleep at night (these are typically lenses for reducing eye strain). If someone intends to use blue light blockers to help sleep, they want to get lenses that block 99% or more of blue light, which means they will be orange or red.
Method
Once you’ve chosen the intervention and understand what you need for it, it’s time to lay out the details of how you’re going to put it into practice. This helps to ensure uniformity across the experiment, which is key to generating useful data. There are 3 key questions to answer here:
When will you use the tool or technique? Will you use it at a specific time of day or based on a trigger? For example, most people could take Vitamin B12 for a deficiency either at a set time in the morning or with breakfast because it doesn’t matter if you take it with food, just that you take it in the morning. However, if you’re using it to shift your circadian rhythm, you want to take it when you can get bright light exposure for the next 90 minutes to generate the strongest effects, so identifying the optimal time that also works for your schedule is key.
What dose will you use? For an action like wearing blue-light blocking glasses, the dose is how long you are going to wear them (for the 60 minutes before bed is a great starting point). For exercise, the dose might be the length of an exercise (running for 4 miles or 30 minutes) or the amount of sets, reps, and recovery time. For supplements, it’s typically how many milligrams you are taking. Especially when time’s involved, it’s important to start with a dose that doesn’t overwhelm you.
With what frequency will you use it? Are you going to be using the tool or technique daily, 4 days per week, only on weekdays, or at some other frequency? Some tools, like meditation, gain in effectiveness as you use them more, while it’s best not to use others too often or they will lose some of their benefits. For example, it’s best not to supplement daily with the amino acid L-tyrosine, which can increase dopamine levels to improve focus, because your body can become used to it over time, and it may lose its effect.
How long will you test the intervention? The longer you test the intervention, the more reliable the data will be because any one outlier day will affect the average less. Reaching the level of statistical significance used in scientific papers (p<0.05) might take a month or more, but most of us don’t need this level of certainty. Having a p value of less than 0.05 means that there is a greater than 95% chance the effects seen were due to the intervention, not random chance. I’m happy to make decisions, especially ones with small risks, if there’s a 70% or 80% chance that the effect is real, as opposed to a statistical anomaly.
There’s also a cost to running long experiments. People are busy, and the longer it goes, the easier it is to drop off, forget to use the intervention, or for some other life event to intervene that can interfere with the data. So, find a nice balance. I recommend testing a new intervention for at least a week and ideally 2 weeks if you can. That seems to be a nice middle ground for most people, except for diet and exercise experiments which can take longer to show results.
When are you going to start the experiment? It’s critical to ensure you aren’t making other changes or running other experiments that may affect the data you’re collecting. You can run multiple experiments simultaneously if the impact of each is disparate, but otherwise you may want to run them at different times. For example, if you’re running a sleep experiment and it’s working, the better sleep may improve your focus, which could make it seem like a focus intervention is working.
Beyond other experiments, some key confounding factors are if you are traveling, sick, have a uniquely stressful time at work, are on vacation, or something else major is out of the ordinary. If one or more of these is planned or pops up during the intervention phase, you may want to remove those days from the data set, extend the data collection, or both, even if it’s annoying to extend the experiment.
Metrics
Tracking the results from the experiments is critical, but you also need to have a baseline to compare the results against. Ask yourself 2 questions to get your metrics right:
How are you going to measure the effects of the intervention? Different goals will call for different metrics, from lab testing to sleep and exercise wearables and self-report questions. While an objective measure like a lab test or wearable may feel like the most valuable, don’t underrate using how you feel as a marker. This is one of the key mistakes I see people make - there are very few interventions that make you feel consistently better over the course of a week or longer that aren’t good for you long term.Even for areas like sleep, how you feel in the morning and how you think you slept are often just as good or better than a wearable for assessing sleep quality. Of course, if you can combine objective and subjective metrics, that’s ideal.
Regardless, the most important thing you can do is to use metrics that measure what you care about as directly as possible. I could use a cognitive psychology test to measure my sustained attention, but when evaluating new interventions for focus, I’m much more interested in how well I can get things done during the deep work blocks in my work schedule.
What will you compare the experiment results against to see if the intervention worked? For experiments, we need a control to compare against the experimental data. For n of 1 experiments, since there is no separate control group of people, you need to provide the experiment and the control data yourself. This means you want to use a control ‘period of time’ when you did not use the intervention to compare against the time when you used it. The most straightforward approach is to collect a baseline data set. If you wear a sleep wearable, you may be collecting a baseline data set in the background, so you can use the week or weeks before starting the intervention for comparison. For something like energy or focus that you probably don’t track regularly, you will want to take a period of time before starting the intervention to ask yourself the same questions and collect the same data as you will be during the intervention phase.
This is key: you want to collect the same data in the same way at the same time during the baseline phase as during the intervention phase, or you can add noise into your data that makes it harder to accurately discern how well the intervention worked. The baseline and intervention data sets don't have to be the same length, but just like the intervention phase, you need to ensure the baseline isn’t so short that an anomalous day can throw off the results. Similarly to the experimental phase, try to capture at least a week of data in the baseline phase and 2 weeks if you can, and consider extending the baseline phase and possibly throwing out some of the data if something out-of-the-ordinary pops up, like travel, being sick, or major stresses that don’t happen regularly.
Red Flags
Most non-prescription interventions don’t have serious side effects, but it’s worth considering any potential risks during the design process. Ask these two questions to put smart guardrails in place:
Are there any safety considerations with this intervention? It’s smart to do some research into potential safety issues before using an intervention. This is tricky because many medical sources don’t understand how non-prescription interventions work, but looking at what a variety of sources across different fields say can be helpful to find commonalities. Doing this research, you may also occasionally find that this experiment is contraindicated for you. For example, people taking psychiatric medications should be careful or avoid taking certain supplements, like neurotransmitter precursors. If you’re not sure, talk to an open minded health professional to ensure you aren’t putting yourself at serious risk.
What symptoms will tell me to stop? If there are potentially serious side effects, lay out what your red lines are before starting. If you begin a new exercise program, and you’re experiencing serious joint pain, then it’s probably smart to stop. If your muscles are sore, but joints feel fine, that’s probably not a reason to stop. Except in serious cases, this is usually a judgment call. If your stomach is off, do you keep it up for another day or two to see if that goes away? If you have a mild headache, is it worth continuing? Setting out your own guidelines beforehand can help ensure you don’t keep things going past what’s smart.
Before closing, I’ll add a note on side effects and nutrition interventions. One of the very few times when interventions make people feel much better over long periods of time but may have hidden risks are when they elevate lipid levels and cardiovascular risk. We regularly see people run experiments that lead to fantastic results, less inflammation, less stress, more energy, better sleep — and a higher ApoB level, which is a strong predictor of heart disease risk (related to your LDL-C level, your “bad cholesterol”). Some scientists argue that ApoB levels aren’t a problem without the inflammation and oxidative stress of a poor diet, but the majority of the evidence still suggests that higher levels of ApoB are a driver of increased cardiovascular risk. So, especially with diet and also supplement interventions, I recommend doing baseline and post-experiment blood testing to see how changes are affecting your lipid levels.
Optional Additions
While the above areas are core to running effective experiments, there are two additional areas you may consider adding to get extra value from your experiments.
Mechanisms of Action
For each intervention, it can be useful to understand the physiological or psychological mechanisms of action behind how it works. This is valuable because it can help ensure you’re selecting interventions that make sense for you and because it can set you up to notice “off-target” (or unintended) effects.
For example, returning to the amino acid L-tyrosine, if you know that the body uses it to make the neurotransmitter dopamine and that dopamine is involved in motivation, then you may be more likely to make the connection if you feel a stronger drive to work out on the days you take it. Alternately, if you know that dopamine is involved in reward signaling and addiction, you would also probably be more likely to tie the supplement to behavior changes like feeling even more glued to your phone than usual.
Tracking Off-Target Effects
While you may happen to notice off-target effects, especially if you understand the intervention well, you can also choose to track potentially related areas. While trying to track too many things is a risk because you can feel overwhelmed and stop tracking anything, intentionally collecting data on related factors can help you identify unexpected effects. I regularly track the results of experiments on my sleep, even if sleep isn’t the goal. Once, when I was running an experiment with a supplement for workout recovery, I noticed that I regularly had 15-20% more deep sleep when I used it late in the day. This led to a discovery that the supplement dramatically improves sleep for about 1/3 of people. While I did feel better the next day, I don’t think I would have noticed the effect had I not been tracking my sleep.
Cheat Sheet
Here is a distillation of the questions that will set you up for an effective n of 1 experiment.
Intervention
What tool or technique do you want to test?
How does it work in practice?
What resources do I need for it?
Method
When will you use the tool or technique?
What dose will you use?
When and with what frequency will you use it?
How long will you test the intervention for?
When are you going to start the experiment?
Metrics
How are you going to measure the effects of the intervention?
What will you compare the experimental results against to see if the intervention worked? What period of time will you use as baseline data?
Red Flags
What safety considerations are there, if any, with this intervention?
What symptoms will tell you to stop?
What It Looks Like in Practice
You don’t need to formally write up your experiments, but here is an example of all the information described above for an experiment.
Intervention: L-Tyrosine for Focus
Method: Take 500 mg of L-tyrosine on an empty stomach 30 minutes before starting the most important deep work block of the day for 2 weeks on Monday, Tuesday, Thursday, and Friday.
Metrics: Answer the following questions on a 1-5 scale
Do you notice any difference trying to start high-focus work?
Do you notice any difference in sustaining high focus work?
Collect baseline data Monday, Tuesday, Thursday, and Friday of the prior week.
Red Flags: Do not run this experiment without talking to your doctor if you are taking psychiatric medications or have a mental health disorder.
During the experiment, if you feel serious anxiety or get a bad headache, stop the experiment.
Reasoning
Some people find it difficult to drop into deep work. One potential cause for this is that in most modern jobs, your brain gets dopamine hits when it switches from one task to another. Unfortunately, this teaches your brain to favor task switching and makes it harder to drop into and maintain focused work sessions. One approach that can aid this is boosting background dopamine levels, which can cause the brain to see the reward in the current task and enable sustained focus. The amino acid l-tyrosine is a precursor to dopamine and can help to increase background levels. Since dopamine can also be converted to adrenaline and noradrenaline, increasing dopamine levels has also been shown to improve stress resilience in some individuals. Conversely though, it may also accentuate the stress response, potentially causing deleterious effects.
Tracking Off Target Effects: Answer these questions daily
Do you notice any difference in how you feel overall?
Do you feel calmer, more excited, and/or more stressed?
Do you notice any differences in your energy levels?
Do you notice any differences in how you’re reacting to what would normally be stressful events?
Conclusion
The approach I’ve laid out above will help to ensure you won’t waste your time when running n of 1 experiments. You will have everything setup to collect the data that will tell you whether an intervention works and if it fits into your life. In my next 2 posts, I’m going to describe some great ancillary benefits of running n of 1 experiments and how to design a more holistic program of experimentation, what we call an Experiment Journey at Fount, to give guidance on questions like which experiments you should run first and how many you should run at a time.