Data modelling · Regression

Multicollinearity: when predictors tell the same story

Expect it as "what is multicollinearity, how do you detect it, and how do you fix it?" That's three parts, and this lesson gives you all three.

Your win todayDefine it in one sentence, compute a VIF in your head, and list three fixes.

What it is

Multicollinearity means two or more predictor (independent) variables in a regression are highly correlated with each other. Example: predicting salary from both age and years of experience. They move together, so the model can't tell which one deserves the credit.[Penn State STAT 462]

Salary (Y) Age (X₁) Experience (X₂) ? The overlap of X₁ and X₂ explains part of Y, but it's shared. The model can't split that credit, so the coefficients swing wildly.
The bigger the overlap between predictors, the less unique information each one carries, and the less you can trust any single coefficient.

Why it hurts (and what it doesn't hurt)

❌ Interpretation breaks

Coefficients become unstable: small data changes flip their size or even their sign. Standard errors inflate, so truly important variables can show high p-values ("not significant").

✅ Prediction often survives

Overall fit (R²) and predictions can still be fine. It's mainly a problem when you need to explain which driver matters, which is exactly what analysts get asked.

Detect it: correlation matrix, then VIF

  1. Correlation matrix of the predictors. A pairwise |r| above about 0.8 is a red flag. But it misses cases where three variables combine.
  2. Variance Inflation Factor for each predictor j: regress Xj on all the other predictors, take that R²j, and compute:

VIFj = 1 ÷ (1 − R²j)

1 · no correlation> 4 · investigate> 10 · serious, fix it R²ⱼ = 0 → VIF 1R²ⱼ = 0.75 → VIF 4R²ⱼ = 0.9 → VIF 10 Thresholds are rules of thumb. Penn State uses 4 / 10. Many practitioners use 5 / 10.
VIF says how much a coefficient's variance is inflated by overlap with the other predictors. Formula and thresholds from Penn State STAT 462 §10.7.

Fix it

  1. Drop one of the correlated variables (keep the one that's easier to explain or measure).
  2. Combine them into one feature: a ratio, an index, or principal components (PCA).
  3. Regularise with ridge regression, which shrinks and stabilises the coefficients.
  4. More data helps a little. Centre variables before creating interaction or squared terms.
Special case: the dummy-variable trapA "City" column with 3 values (Delhi, Mumbai, Pune) becomes dummy columns. If you include all 3 dummies plus an intercept, they always sum to 1. That's perfect multicollinearity, and the model can't be fitted. Use k − 1 = 2 dummies.
Say it in the interview"Multicollinearity is when independent variables in a regression are highly correlated with each other, like age and years of experience. It doesn't necessarily hurt prediction, but it makes coefficients unstable and inflates their standard errors, so you can't trust which variable drives the outcome. I'd detect it with a correlation matrix and VIF, where above about 5 to 10 is a concern. To fix it, I'd drop or combine variables, or use ridge regression."

Retrieval check

Multicollinearity is a high correlation between …

What is its main harm?

Regressing X₁ on the other predictors gives R² = 0.9. What is VIF₁?

Which of these is not a fix for multicollinearity?

"City" has 3 categories and the model has an intercept. How many dummy columns?

Type it

R²j = 0.75. VIF = ?

What R²j gives a VIF of exactly 5?

Your turn

Go deeper

Primary source: Penn State STAT 462 §10.7: Detecting Multicollinearity Using Variance Inflation Factors. It's university course notes with a worked example.

Related: Lesson 7: p-values (why inflated standard errors hide significance) · Stats cheat sheet · Statistics question bank

💬 Ask your teacher: "explain ridge regression simply" or "multicollinearity vs correlation vs causation?"