Isye 6501 Midterm 2 Cheat Sheet

ISYE 6501 Midterm 2 Strategic Review: Core Concepts & Essential Formulas

Success in ISYE 6501: Introduction to Analytics Modeling hinges on a deep, intuitive understanding of statistical modeling principles, not just memorizing equations. Midterm 2 typically builds upon the foundation of simple linear regression, diving into the complexities of multiple regression, model selection, diagnostics, and validation. This strategic review is designed as a comprehensive study guide to consolidate your knowledge, clarify common points of confusion, and connect the mathematical formalism to practical analytical thinking. Think of it not as a shortcut, but as a map to navigate the key terrain you will be tested on.

Foundational Pillars: From Simple to Multiple Regression

The leap from simple linear regression (one predictor) to multiple linear regression (multiple predictors) is the cornerstone of this exam. The core model equation expands from Y = β₀ + β₁X + ε to Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε.

The interpretation of coefficients changes critically. In a simple model, β₁ is the effect of X₁ on Y. In a multiple model, β₁ represents the effect of X₁ on Y, holding all other predictor variables constant. This "ceteris paribus" condition is the heart of multivariable analysis. A common trap is to interpret a multiple regression coefficient as a marginal, total effect without considering potential confounding or mediation by other variables in the model.

Hypothesis testing retains its structure but gains layers. The global F-test (H₀: β₁ = β₂ = ... = βₖ = 0) assesses if at least one predictor contributes to the model. If rejected, you proceed to individual t-tests for each βⱼ (H₀: βⱼ = 0). Remember, a significant global F-test with no significant individual t-tests often signals multicollinearity—predictors are correlated, making it hard to isolate their unique effects. The Variance Inflation Factor (VIF) is your diagnostic tool here; a VIF > 5-10 indicates problematic multicollinearity.

Model Selection: Navigating the Trade-off Between Fit and Complexity

With dozens of potential predictors, how do you choose the "best" model? This is a central theme. The goal is to find a model that fits the data well (low error) but is also parsimonious (simple and generalizable). Overfitting—capturing noise instead of signal—is the primary enemy.

You must be fluent in the main selection criteria:

• Adjusted R-squared (R²_adj): Penalizes the addition of useless predictors. It increases only if a new term improves the model more than expected by chance. It's a good, intuitive measure for comparing models on the same dataset.
• Akaike Information Criterion (AIC): Based on information theory. It balances model likelihood (fit) with the number of parameters (complexity). AIC = -2*log(Likelihood) + 2k. Lower AIC is better. It tends to select more complex models than BIC.
• Bayesian Information Criterion (BIC): Similar to AIC but imposes a harsher penalty for complexity (BIC = -2*log(Likelihood) + k*log(n)). Lower BIC is better. It strongly favors simpler models and is consistent (will select the true model if it exists in the candidate set as n grows).
• Mallows' Cp: Aims to estimate the total prediction error. A good model has Cp ≈ p (where p is the number of parameters including the intercept). Cp much larger than p indicates high bias (underfitting); Cp much smaller than p suggests overfitting.

Key Strategy: There is no single "best" criterion. Use them in concert. A model that is top-ranked by R²_adj, AIC, and BIC is a strong candidate. Always perform nested F-tests when comparing models where one is a subset of the other. This provides a formal statistical test for whether the additional predictors provide significant explanatory power.

Categorical Predictors & Interactions: Expanding the Model

Real-world data is messy. You must know how to incorporate categorical (factor) variables. A categorical variable with k levels requires k-1 dummy variables (indicator variables). The dropped level becomes the reference category. The coefficient for a dummy variable (β_dummy) represents the estimated difference in the mean response Y between that category and the reference category, holding other variables constant.

Interactions are crucial for capturing effect modification. An interaction term (X₁ * X₂) allows the effect of X₁ on Y to depend on the level of X₂. The model becomes Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁*X₂) + ε. Here, the effect of X₁ is β₁ + β₃X₂. If β₃ is significant, the relationship is not parallel. Always center continuous variables (subtract the mean) before creating interaction terms to reduce multicollinearity and make the main effects (β₁, β₂) more interpretable (they now represent the effect when the other variable is at its mean).

Model Diagnostics & Assumption Checking: The Non-Negotiable Step

A model with a high R² is useless if its assumptions are violated. You must be able to diagnose and suggest remedies for the four classical linear regression assumptions:

1. Linearity: The relationship between predictors and response is linear.

   • Diagnosis: Residuals vs. Fitted Values plot. Look for random scatter around zero. A curved pattern indicates missing nonlinearity.
   • Remedy: Add polynomial terms (e.g., X²), use transformations (log, sqrt), or consider a different model family.

2. Constant Variance (Homoscedasticity): The error variance is constant across all fitted values.

   • Diagnosis: Residuals vs. Fitted Values plot. Look for a "funnel" shape (variance increasing or decreasing with fitted value).
   • Remedy: Transform the response variable Y (e.g., log(Y) often stabilizes variance), or use Weighted Least Squares.

3. Normality of Errors: The errors are normally distributed.

   • Diagnosis: Normal Q-Q plot of residuals. Points should follow a straight line. Histogram of residuals (less reliable).
   • Remedy: Transform Y. For large samples, the Central Limit Theorem often mitigates this concern for inference on coefficients.

4. Independence: Errors are uncorrelated.

   • Diagnosis: Plot residuals in chronological order (if time series) or against a spatial variable. Look for