To estimate our population parameters, we need to take a random sample from the population and use an estimator (a rule that combines data to output a numerical value for a population parameter) to calculate the estimate (the numerical value taken on by an estimator for a particular sample of data). Note that every time you calculate an estimator's value, you are effectively drawing one value for it from its underlying sampling distribution. Using the same estimator on two different samples will, in general, result in two different estimates.

Ideally, the estimator that we use should be unbiased and should have a low sampling distribution. An unbiased estimator is the one that does not consistently overestimate or underestimate the population parameter (bias means systematic as opposed to random error). For an estimator to be unbiased, the difference between the estimator's expected value for a parameter (i.e., the sampling mean of the estimator) and the actual value of the parameter has to be zero, regardless of the sample size. An asymptotically unbiased estimator is one whose bias decreases to zero as the sample size increases. Note that being biased or unbiased is the estimator's property and not the estimate, which is just a fixed number obtained from a particular sample using an estimator. If an estimator is not unbiased, it should be at least consistent for us to use it. A consistent estimator is asymptotically unbiased, and the variance of its sampling distribution converges to zero as the sample size increases. In other words, an estimator is consistent if its sampling distribution becomes narrower and narrower around the true population parameter as the sample size increases.

Ordinary Least Squares (OLS) is an estimator that chooses the estimates such that the sum of squared residuals is minimized. Note the difference between errors and residuals: error is a population value that we will never know, so we have to rely on the next best thing we have: residuals, which are the sample estimate of the errors for each observation. According to a theory known as Gauss-Markov, under the following four conditions, OLS estimators are unbiased. These conditions are: 1) the population model is linear in the parameters and error term, the sample is random in which the independent variables do not have 2) the same values and 3) a perfectly linear relationship, and 4) the expected value of the error term conditional on the independent variables is zero (i.e., ). A violation in the second or third condition results in us not being able to calculate the OLS estimates. To understand why a breach in the last condition makes the OLS estimator biased, note that the error term explains variability in the target variable that is not explainable by the independent variables. When the last condition is violated, OLS incorrectly attributes such variability to the independent variables, which causes the estimates to deviate from their actual population value. This also influences any ceteris paribus (i.e., other things equal) conclusion that we may want to draw about the effect of one independent variable on the target variable. Finally, if, instead of assumption 4, we make the less restrictive assumption of , (note that this condition is less restrictive than assumption 4 because it requires that only each be uncorrelated with u, whereas assumption 4 requires any function of s be uncorrelated with u) then OLS estimators will be only consistent instead of unbiased.

Covers: theory of Estimators
Estimated time needed to finish: 10 minutes
Questions this item addresses:
  • What is Ordinary Least Squares?
0 comment

Linear Regression

Total time needed: ~37 minutes
you learn the assumptions behind linear regression and, more importantly, what will occur if those assumptions are violated.
Potential Use Cases
refreshing knowledge on linear regression, preparing for job interview questions
Who is This For ?
BEGINNERpeople entering the field of data science and data scientists who think stat is their weak point
Click on each of the following annotated items to see details.
WRITEUP 1. What is linear regression
  • What is linear regression?
10 minutes
WRITEUP 2. Ordinary Least Squares
  • What is Ordinary Least Squares?
10 minutes
WRITEUP 3. Homoscedasticity and error normality
  • Why do we need homoscedasticity and error normality assumptions?
9 minutes
ARTICLE 4. Potential business consequences of violation in GM assumptinos
  • What are some of the consequences of violation in GM assumptinos?
8 minutes

Concepts Covered

0 comment