Curious how different posters here think about these terms. They are notoriously vague. Andrew Gelman offers five definitions of these terms in this paper (which I only know of by way of this video):
Fixed Effects are constant across individuals, and random effects vary
Effects are fixed if they are interesting in themselves or random if there is an interest in the underlying population
Fixed effects are estimated from large samples, random effects from small samples
Random effects are realizations of random variables
Fixed effects are estimated using maximum likelihood, random effects exhibit shrinkage.
I was trained in econometrics, where (helpfully!) none of these definitions apply. There, fixed effects are group specific intercepts, and random effects are something like a clustered standard error adjustment. I will spare you the specifics.
So when people ask questions about these terms, I never really understand what is being asked. I’m curious what everyone understands these terms to mean. It’s like a community survey, but nerdier!
I’m gonna be a contrarian and say I very much like the fixed / random terminology.
Random effect: Synonymous with a hierarchical model. Partial pooling is happening. The word “random” is because each member of the group is a sample from some broader population. The sigma parameter you infer that controls the partial pooling tells you about the broader population.
For example say you have some medical study run at 5 hospitals. The hospital effect is a random effect because the 5 hospitals in the data is a random sample from an infinite number of hospitals.
Fixed effect: The effect is “fixed” by the experimenter. In your medical study say you’re testing three drugs vs. a placebo. This is a fixed effect because you want to measure the effect of drug A, drug B and drug C independently. You don’t use a hierarchical model / partial pooling because it would be a bad idea to think about the three drugs as some random sample from an infinite number of different drugs. You also don’t want to have any notion of, if drug A and B work well and C doesn’t, then that means drug C should get regularized up to the mean of all three drugs, and A and B probably aren’t as good as they seem because of C. You want to treat each drug effect (that the experimenter fixed in advance) independently.
I honestly think that those 5 definitions fit nicely into the definition above.
Definition 5: fixed effects can be estimated with maximum likelihood just fine, while random effects do exhibit shrinkage, so that’s true.
Definition 4: Yes, random effects (the observed groups), are realizations of random variables (the population), so I think that’s true too.
Definition 3: Maybe that’s usually true?
Definition 2: I think that’s often true. You don’t really care about the hospital effect, it’s a nuisance variable. You care about which drug is best.
Definition 1: The same drugs A, B, C are used across all 5 hospitals. The hospital used across each drug varies, necessarily. So I’d agree with that too.
I think the confusion comes from non-Bayesians having a hard time fitting hierarchical models? Also possible that I have retconned the fixed and random effects terminology into something that makes sense only to me?
Random = hierarchical, partial pooling is happening, information sharing. Probably don’t have control over the selection, hence random.
Fixed = not hierarchical, no pooling, effects are independent. Might control these up front.
I’m struggling with this. Can’t fixed effects also be inherently hierarchical. If you have strong reason to believe that drugs A, B, and C are similar (e.g., chemical composition) can’t A, B, and C can all have a fixed effects that are hierarchical in nature, no?
To give an example, is the intercept in the radon example a fixed effect or random effect? Prior to this thread I would have called it a fixed effect, but based on the definitions here it seems like it should be considered a random effect.
I would also call it a “hierarchical fixed effect”, because I was taught that group-level intercepts are fixed effects. Then we slap on a hierarchy to do shrinkage.
In the radon example both the intercepts and the slopes are random effects, because they’re hierarchical over county. There’s an overall global (literally) radon level and the counties are a random sampling of discrete areas, so partial pooling makes sense. I’ll note that the PyMC radon example calls both the intercept and slope random effects.
I think it doesn’t matter to fixed vs random whether it’s an “intercept” (not multipled by some other covariate) or a “slope” (multiplied by 0 or 1 depending on basement or not). I think the distinction is only:
@jaharvey8 with the drug example, I think that’s like definition 2 it depends on what your goal is. I think a fixed effect is better because personally I wouldn’t want to take a drug that was tested once, killed someone, but got it’s effect pooled up towards the other drugs that were considered similar but happened to be trialed more. I think you’d really want no information sharing, so a fixed effect.
But, if your goal is to figure out whether this drug company is a good company and if you should buy stock in it, then I think a random effect / hierarchical model / shrinkage term makes sense for the drug effect. Like the definition 2 says, your goal is now to learn something about the underlying population of their drugs, and you don’t necessarily care about the precise effect of drug A, B or C.
What would a hierarchical effect that’s fixed mean? What would be “fixed” about it?
Each unit in the hierarchy is different, sure. But the effect is still “fixed” in the sense that it’s a constant mean offset for that unit that shouldn’t change. It isn’t a slope coefficient for a value that could be intervened on.