This page was created using Mplus 5.1.
Below is an example of ordinary least squares (OLS) regression with footnotes explaining the output. To summarize the output, both predictors in this model, x1 and x3, are significantly related to the outcome variable, y1.
Here is the same example illustrated in Mplus based on the ex3.1.dat data file.
TITLE:
this is an example of a simple linear
regression for a continuous observed
dependent variable with two covariates
DATA:
FILE IS ex3.1.dat;
VARIABLE:
NAMES ARE y1 x1 x3;
MODEL:
y1 ON x1 x3;
SUMMARY OF ANALYSIS
Number of groups 1
Number of observations 500
Number of dependent variables 1
Number of independent variables 2
Number of continuous latent variables 0
<output omitted>
TESTS OF MODEL FIT
Chi-Square Test of Model Fita
Value 0.000
Degrees of Freedom 0
P-Value 0.0000
Chi-Square Test of Model Fit for the Baseline Modelb
Value 469.585
Degrees of Freedom 2
P-Value 0.0000
CFI/TLIa
CFI 1.000
TLI 1.000
Loglikelihoodc
H0 Value -2124.388
H1 Value -2124.388
Information Criteriad
Number of Free Parameters 4
Akaike (AIC) 4256.776
Bayesian (BIC) 4273.634
Sample-Size Adjusted BIC 4260.938
(n* = (n + 2) / 24)
RMSEA (Root Mean Square Error Of Approximation)a
Estimate 0.000
90 Percent C.I. 0.000 0.000
Probability RMSEA <= .05 0.000
SRMR (Standardized Root Mean Square Residual)a
Value 0.000
- Chi-Square Test of Model Fit, CFI/TLI, RMSEA, and the
SRMR. – Because the current model (like all ols style regression models) is “saturated” (meaning all possible coefficients are being estimated), many fit indices will show perfect fit (either zero or 1.0 for most fit indices). This does not mean that the model actually fits the data perfectly, nor does it indicate a problem with the model, it is just a result of the model having zero degrees of freedom. What the “perfect” fit statistics do mean is that these statistics cannot be used to help you determine how well your model fits. - Chi-Square Test of Model Fit for the Baseline Model – The baseline model is the model to which the current model is compared for many fit statistics. For regression models in Mplus, the baseline model is typically a “null” models, that is a model which assume no relationship among the variables. In this case, the baseline model is a model in which the regression coefficients (paths from x1 to y1, and from
x3 to y1) are constrained to zero. (If you’d like to test this yourself, change the model statement to read y1 ON x1@0 x3@0 and rerun the model, the “Chi-Square test of Model Fit” and the “Chi-Square Test of Model Fit for the Baseline Model” will match each other, because in that case, the model specified will be the baseline model.) - Loglikelihood – This section of the output gives the log likelihood for the model, which can be used to compare the fit of nested models, using a likelihood ratio test. Note that unless you have used the difftest
command, the H0 value and the H1 value should be the same for this type of model. - Information Criteria – Information criteria can also be used to compare competing models, a particularly useful feature is that they can be used to compare non-nested models. The first thing Mplus lists in the information criteria section is the number of free parameters, that is, the number of parameters estimated in the model. Next the Akaike information criteria (AIC), the Bayesian information criteria (BIC), and the sample-size adjusted BIC are each listed. The three criteria each have their advantages and disadvantages, but all three are used to compare the fit of models. To do this one runs the models one wishes to compare, and then compares the two models in terms of one or more of the information criteria (among other things). The interpretation of the AIC and the BIC varies somewhat based on which formulae are being used to calculate the statistic (and different statistical software uses slightly different formulae). In the case of Mplus, higher values indicate worse fit, (and lower values indicate better fit). The larger the difference in criterion statistics, the greater the difference in fit between the two models. In other words, large differences in the information criterion of choice indicates one model fits much better than the other, while small differences indicate that one model fits the data about as well as the other. There are no hypothesis (significance) tests for differences in information criteria, but various sets of guidelines for interpreting them exist.
Model Results
MODEL RESULTS
Two-Tailed
Estimatef S.E.g Est./S.E.h P-Valuei
Y1e ON
X1 0.969 0.042 23.357 0.000
X3 0.649 0.044 14.626 0.000
Interceptsj
Y1 0.511 0.043 11.765 0.000
Residual Variancesk
Y1 0.941 0.060 15.811 0.000
- ON – The dependent variable is y1. The “ON” following it indicates that the coefficients listed are for the regression of y1
on the variables listed below (x1 and x3). - Estimate – The estimate column gives the estimate of the parameter in question. In the case of regression (denoted with “ON”), the parameters being estimated are the regression coefficients. These estimates tell you about the relationship between the independent variables and the dependent variable. More specifically, the estimates of the regression coefficients indicate the amount of increase in y1
that would be predicted by a 1 unit increase in the predictor (i.e. x1
or x3).Note: For the independent variables which are not significant, the coefficients are not significantly different from 0, which should be taken into account when interpreting the coefficients. (See items h and i for a discussion of testing whether the coefficients are significantly different from zero.)We can interpret the regression coefficients shown in the following way:
x1 – The coefficient (parameter estimate) is 0.969, so for every one unit increase in x1 the expected (predicted) value of y1 increases by 0.969, holding all other variables in the model constant.
x3 – The coefficient for x3 is 0.649, meaning that for every one unit increase in x3 the predicted value of y1 increases by 0.649, holding all other variables in the model constant.
We can use the estimates of the regression coefficients along with the intercept (see j below) to predict values of y1. One way of writing the equation looks like this:
y1_predicted = b0 + b1*x1 + b2*x3
Using the values from this section of the output, as well as the intercept (discussed below in item j), we get the following equation:
y1_predicted = 0.511 + 0.969*x1 + 0.649*x3
- S.E. – The column labeled S.E. contains the standard errors associated with the coefficients. The standard error is used for testing whether the parameter is significantly different from 0 by dividing the parameter estimate by the standard error. (See items h and i for a discussion of testing whether the coefficients are significantly different from zero.) The standard errors can also be used to form a confidence interval for the parameter.
- Est./S.E – The column labeled “Est./S.E.” contains exactly that, the coefficient estimate divided by the standard error. This value is used compute the p-value, that is, to perform a significance test.
- Two-Tailed P-Value – The column labeled “Two-Tailed P-Value” contains the p-values for a two-tailed test, testing the null hypothesis that the coefficient (estimated parameter) is 0. If you use a 2-tailed test, then you would compare each p-value to your preselected value of alpha. Coefficients having p-values less than alpha are statistically significant. For example, if you chose alpha to be 0.05, coefficients having a p-value of 0.05 or less would be statistically significant (i.e., you can reject the null hypothesis and say that the coefficient is significantly different from 0). If you use a 1-tailed test (i.e., you predict that the parameter will go in a particular direction), then you can divide the p-value by 2 before comparing it to your preselected alpha level. Looking at the example, we can say the following. The coefficient for x1 (0.969) is significantly different from 0 using an alpha of 0.05 because its p-value is listed as 0.000, which is smaller than 0.05. The coefficient for x3 (0.649) is significantly different from 0 using an alpha of 0.05 because its p-value is listed as 0.000*, which is smaller than 0.05.Note that a p-value listed as 0.000 does not mean that the actual p-value is zero, instead it indicates that the p-value is less than 0.0005 (0.0004 being approximately the largest number which rounds to 0.000). Values less than 0.0005 are indeed very close to zero in many practical applications, but this is conceptually distinct from an actual value of zero.
- Intercepts – Intuitively, the intercepts for the model appear under the heading “Intercepts” in the case of an OLS regression, only the intercept of the dependent variable (i.e. y1) is part of the model. The intercept is the expected value of
y1 when both x1 and x3 are equal to zero. The actual estimate of the intercept is in the first column (i.e. under “Estimates” see item f), followed by the standard error (listed under S.E., see item g). The estimate and the standard error are followed by the estimate over the standard error (listed under “Est./S.E.” see item h), and finally the p-value listed under “Two-Tailed P-Value” (see item i). In this case the intercept of 0.511 is (statistically) significantly different from zero using an alpha of 0.05 because its p-value is listed as 0.000 (see item i for a discussion of p-values listed as 0.000), which is smaller than 0.05. - Residual Variances – The section labeled “Residual Variances” lists the estimated variances for the dependent variable. This is an estimate of the variance of the residuals (also known as errors) from the regression.
Data were collected on 200 high schools students and the dataset hsbdemo contains scores on various tests, including science, math, reading and social studies (socst).
# Import data into R
d <- read.csv("https://stats.oarc.ucla.edu/stat/data/hsbdemo.csv")
# Inspect the structure of the dataset
str(d)
## 'data.frame': 200 obs. of 13 variables:
## $ id : int 45 108 15 67 153 51 164 133 2 53 ...
## $ female : chr "female" "male" "male" "male" ...
## $ ses : chr "low" "middle" "high" "low" ...
## $ schtyp : chr "public" "public" "public" "public" ...
## $ prog : chr "vocation" "general" "vocation" "vocation" ...
## $ read : int 34 34 39 37 39 42 31 50 39 34 ...
## $ write : int 35 33 39 37 31 36 36 31 41 37 ...
## $ math : int 41 41 44 42 40 42 46 40 33 46 ...
## $ science: int 29 36 26 33 39 31 39 34 42 39 ...
## $ socst : int 26 36 42 32 51 39 46 31 41 31 ...
## $ honors : chr "not enrolled" "not enrolled" "not enrolled" "not enrolled" ...
## $ awards : int 0 0 0 0 0 0 0 0 0 0 ...
## $ cid : int 1 1 1 1 1 1 1 1 1 1 ...
The “female” variable is a character variable with two possible values: “female” and “male”.
unique(d$female)
## [1] "female" "male"
We are going to change this variable into a numeric variable coded 1 if the student was female and 0 if male.
d$female <- ifelse(d$female=="female",1,0)
Run regression
For an example of a regression model output, we will model science score as a function of math, reading and social studies (socst) scores and student’s gender.
m1 <- lm(science ~ math + female + socst + read, data = d)
summary(m1)
##
## Call:
## lm(formula = science ~ math + female + socst + read, data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -18.6706 -4.5764 -0.3237 4.5006 21.8563
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 12.32529 3.19356 3.859 0.000154 ***
## math 0.38931 0.07412 5.252 3.92e-07 ***
## female -2.00976 1.02272 -1.965 0.050820 .
## socst 0.04984 0.06223 0.801 0.424139
## read 0.33530 0.07278 4.607 7.36e-06 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 7.148 on 195 degrees of freedom
## Multiple R-squared: 0.4892, Adjusted R-squared: 0.4788
## F-statistic: 46.69 on 4 and 195 DF, p-value: < 2.2e-16
Annotated output
## [...]
## Call:
## lm(formula = science ~ math + female + socst + read, data = d)
## [...]
The first two lines show the formula that R used to fit the regression model.
## [...]
## Residuals:
## Min 1Q Median 3Q Max
## -18.6706 -4.5764 -0.3237 4.5006 21.8563
## [...]
This is a summary of the distribution of the residuals from the model. Residuals are differences between actual observed values and the values that the model predicts. In addition to minimum, median and maximum, the summary provides lower quartile (1Q) under which 25% of the data points are found and upper quartile (3Q) under which 75% of the data points are found in the distribution of the residuals.
## [...]
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 12.32529 3.19356 3.859 0.000154 ***
## math 0.38931 0.07412 5.252 3.92e-07 ***
## female -2.00976 1.02272 -1.965 0.050820 .
## socst 0.04984 0.06223 0.801 0.424139
## read 0.33530 0.07278 4.607 7.36e-06 ***
## ---
## [...]
The first column lists the names of predictors that were included in the model formula: math, female, socst, read. The (Intercept) is a constant that represents predicted value of science score when all other predictors are 0.
Estimate
The column “Estimate” provides estimated coefficients (or parameter estimates) for predicting dependent variable (science scores) from independent variables specified in the model formula (math, social studies (socst), reading scores and student’s gender). Regression equation for this dataset and this model can be represented:
Predicted science score \(= 12.32529 + 0.38931\cdot math + (-2.00976)\cdot female + 0.04984\cdot socst + 0.33530\cdot read\)
These coefficients indicate the change in predicted science scores associated with a one-unit increase in the predictor variable.
An estimate of 0.38931 for math means that for every unit (i.e., point, since this is the metric in which the tests are measured) increase in math, a 0.38931 unit increase in science score is predicted, holding all other variables constant. (It does not matter at what value you hold the other variables constant, because it is a linear model.) Or, for every increase of 1 point on the math test, a student’s science score is predicted to be higher by 0.38931 points.
The interpretation for other continuous variables is the same. An estimate 0.04984 for socst means that for every increase in 1 point on the social studies score, student’s science score is predicted to be higher by 0.04984 points. An estimate 0.33530 for read means that for every increase in 1 point on the reading scores, student’s science score is predicted to be higher by 0.33530 points.
The estimate for the categorical variable representing student’s gender (0=male, 1=female) indicates that switching from the reference category (in this case male) to females, the science score is predicted to decrease by an amount of -2.00976 points, holding all other predictors constant. So, science scores for females are predicted to be about 2.00976 points lower than for males, all else being equal.
Std. Error
These are the standard errors associated with the coefficients. The standard error is used for testing whether the parameter estimate is significantly different from 0 by dividing the estimate by its standard error to obtain a t-value. The standard errors can also be used to form a confidence interval for the parameter estimate.
t value and Pr(>|t|)
These two columns provide the t-value and 2-tailed p-value used in testing the null hypothesis that the estimate of a parameter is 0.
t-value is a test statistic obtained dividing the coefficient by its the standard error. P-values represented in column Pr(>|t|) are probabilities of getting a t-value equal or greater than the one computed for this sample if the null hypothesis were true.
If you use a 2-tailed test, then you would compare each p-value to your pre-selected value of alpha. Coefficients having p-values less than alpha are statistically significant. For example, if you chose alpha to be 0.05, coefficients having a p-value of 0.05 or less would be statistically significant (i.e., you can reject the null hypothesis). If you use a 1-tailed test (i.e., you hypothesize that the parameter estimate will go in a particular direction), then you can divide the p-value by 2 before comparing it to your pre-selected alpha level.
## [...]
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## [...]
The asterisks provide a quick visual shortcut to p-values. A p-value that is a very small number is indicated by three asterisks. In this model, p-values of 3.92e-07, 0.000154, and 7.36e-06 have three asterisks. A p-value that is between a very small number and 0.001 is indicated by two asterisks. A p-value that is between 0.001 and 0.01 is indicated by one asterisk. A p-value that is at the alpha level set before the analysis is indicated by a dot. In this case, the p-value with a dot is 0.051 when rounded. All p-values that are above 0.05 do not receive any asterisks.
## [...]
## Residual standard error: 7.148 on 195 degrees of freedom
## [...]
The residual standard error is an estimate of the standard deviation of the residuals, which are the differences between the observed values and the predicted values of the response variable. The residual standard error measures the average amount by which the observed responses deviate from the true regression line, and it is expressed in the same units as the response variable (i.e., points in science scores in this case).
The residual standard error was calculated with 195 degrees of freedom (df). Degrees of freedom represent the number of independent observations in the data that are available to estimate a parameter. In the case of linear regression, the number of parameters is equal to the number of predictors plus one (for the intercept term), and the number of degrees of freedom is equal to the total number of observations minus the number of parameters. In this model, there are 200 observations and 5 parameters (4 predictors plus the intercept), so the number of degrees of freedom for the residual error is 195.
## [...]
## Multiple R-squared: 0.4892, Adjusted R-squared: 0.4788
## [...]
Multiple R-squared is the proportion of variance in the dependent variable which can be predicted from the independent variables. The Multiple R-squared is also known as the coefficient of determination. The value of R-squared ranges from 0 to 1. A value of 0 means that the model explains none of the variability in the dependent variable, while a value of 1 means that the model explains all of the variability in the dependent variable. It is important to note that a high R-squared does not necessarily mean that the model is a good fit for the data or that the predictors are good at predicting the outcome variable. A model with high R-squared may still have a poor fit to the data if the residuals are not normally distributed. Or, it does not indicate anything about predictive power of the model in the future. It should be used along other mesures of model fit to fully evaluate the model quality.
The R-squared value of 48.92% of the variance in science scores can be predicted from the variables math, social studies (socst), reading scores and student’s gender. Note that this is an overall measure of the strength of association, and does not reflect the extent to which any particular independent variable is associated with the dependent variable.
In multiple regression, R-squared will always increase as more variables are included in the model, even if the model with more variables is overfitting the sample. The adjusted R-squared takes into account the number of predictors in the model and is always lower than R-squared. It is computed using the formula \(1 – \left( (1 – R{\text -squared}) \cdot \frac{N – 1}{ N – k – 1} \right)\).
When the number of observations (N) is small and the number of predictors (k) is large, there will be a much greater difference between R-squared and adjusted R-squared (because the ratio of \(\frac{N – 1}{N – k – 1}\) will be much greater than 1). By contrast, when the number of observations is very large compared to the number of predictors, the value of R-squared and adjusted R-squared will be much closer because the ratio of \(\frac{N – 1}{N – k – 1}\) will approach 1.
F-statistic
## [...]
## F-statistic: 46.69 on 4 and 195 DF, p-value: < 2.2e-16
## [...]
The F-statistic and associated p-value indicate whether the overall model is statistically significant and whether there is evidence that at least one independent variable is related to the dependent variable. However, they do not provide information on the specific contributions of each individual independent variable in predicting the dependent variable.
This page shows an example regression analysis with footnotes explaining the output. These data (hsb2demo) were collected on 200 high schools students and are scores on various tests, including science, math, reading and social studies (socst). The variable female is a dichotomous variable coded 1 if the student was female and 0 if male.
In the code below, the data = option on the proc reg statement tells SAS where to find the SAS data set to be used in the analysis. On the model statement, we specify the regression model that we want to run, with the dependent variable (in this case, science) on the left of the equals sign, and the independent variables on the right-hand side. We use the clb option after the slash on the model statement to get the 95% confidence limits of the parameter estimates. The quit statement is included because proc reg is an interactive procedure, and quit tells SAS that not to expect another proc reg immediately.
%let path=C:\temp; libname idre "&path"; proc reg data = idre.hsb2demo; model science = math female socst read / clb; run; quit;
The REG Procedure
Model: MODEL1
Dependent Variable: science science score
| Number of Observations Read | 200 |
|---|---|
| Number of Observations Used | 200 |
| Analysis of Variance | |||||
|---|---|---|---|---|---|
| Source | DF | Sum of Squares |
Mean Square |
F Value | Pr > F |
| Model | 4 | 9543.72074 | 2385.93019 | 46.69 | <.0001 |
| Error | 195 | 9963.77926 | 51.09630 | ||
| Corrected Total | 199 | 19507 | |||
| Root MSE | 7.14817 | R-Square | 0.4892 |
|---|---|---|---|
| Dependent Mean | 51.85000 | Adj R-Sq | 0.4788 |
| Coeff Var | 13.78624 |
| Parameter Estimates | ||||||||
|---|---|---|---|---|---|---|---|---|
| Variable | Label | DF | Parameter Estimate |
Standard Error |
t Value | Pr > |t| | 95% Confidence Limits | |
| Intercept | Intercept | 1 | 12.32529 | 3.19356 | 3.86 | 0.0002 | 6.02694 | 18.62364 |
| math | math score | 1 | 0.38931 | 0.07412 | 5.25 | <.0001 | 0.24312 | 0.53550 |
| female | 1 | -2.00976 | 1.02272 | -1.97 | 0.0508 | -4.02677 | 0.00724 | |
| socst | social studies score | 1 | 0.04984 | 0.06223 | 0.80 | 0.4241 | -0.07289 | 0.17258 |
| read | reading score | 1 | 0.33530 | 0.07278 | 4.61 | <.0001 | 0.19177 | 0.47883 |
Anova Table
| Analysis of Variance | |||||
|---|---|---|---|---|---|
| Source | DF | Sum of Squares |
Mean Square |
F Value | Pr > F |
| Model | 4 | 9543.72074 | 2385.93019 | 46.69 | <.0001 |
| Error | 195 | 9963.77926 | 51.09630 | ||
| Corrected Total | 199 | 19507 | |||
Source: This is the source of variance, Model, Residual, and Total. The Total variance is partitioned into the variance which can be explained by the independent variables (Model) and the variance which is not explained by the independent variables (Residual, sometimes called Error). Note that the Sums of Squares for the Model and Residual add up to the Total Variance, reflecting the fact that the Total Variance is partitioned into Model and Residual variance.
DF: These are the degrees of freedom associated with the sources of variance. The total variance has N-1 degrees of freedom. In this case, there were N=200 students, so the DF for total is 199. The model degrees of freedom corresponds to the number of predictors minus 1 (K-1). You may think this would be 4-1 (since there were 4 independent variables in the model, math, female, socst and read). But, the intercept is automatically included in the model (unless you explicitly omit the intercept). Including the intercept, there are 5 predictors, so the model has 5-1=4 degrees of freedom. The Residual degrees of freedom is the DF total minus the DF model, 199 – 4 is 195.
Sum of Squares: These are the Sum of Squares associated with the three sources of variance, Total, Model and Residual. These can be computed in many ways. Conceptually, these formulas can be expressed as: SSTotal The total variability around the mean. S(Y – Ybar)2. SSResidual The sum of squared errors in prediction. S(Y – Ypredicted)2. SSModel The improvement in prediction by using the predicted value of Y over just using the mean of Y. Hence, this would be the squared differences between the predicted value of Y and the mean of Y, S(Ypredicted – Ybar)2. Another way to think of this is the SSModel is SSTotal – SSResidual. Note that the SSTotal = SSModel + SSResidual. Note that SSModel / SSTotal is equal to .4892, the value of R-Square. This is because R-Square is the proportion of the variance explained by the independent variables, hence can be computed by SSModel / SSTotal.
Mean Square: These are the Mean Squares, the Sum of Squares divided by their respective DF. For the Model, 9543.72074 / 4 = 2385.93019. For the Residual, 9963.77926 / 195 = 51.0963039. These are computed so you can compute the F ratio, dividing the Mean Square Model by the Mean Square Residual to test the significance of the predictors in the model.
F Value and Pr > F – The F-value is the Mean Square Model (2385.93019) divided by the Mean Square Residual (51.0963039), yielding F=46.69. The p-value associated with this F value is very small (0.0000). These values are used to answer the question “Do the independent variables reliably predict the dependent variable?”. The p-value is compared to your alpha level (typically 0.05) and, if smaller, you can conclude “Yes, the independent variables reliably predict the dependent variable”. You could say that the group of variables math, female, socst and read can be used to reliably predict science (the dependent variable). If the p-value were greater than 0.05, you would say that the group of independent variables does not show a statistically significant relationship with the dependent variable, or that the group of independent variables does not reliably predict the dependent variable. Note that this is an overall significance test assessing whether the group of independent variables when used together reliably predict the dependent variable, and does not address the ability of any of the particular independent variables to predict the dependent variable. The ability of each individual independent variable to predict the dependent variable is addressed in the table below where each of the individual variables are listed.
Overall Model Fit
| Root MSE | 7.14817 | R-Square | 0.4892 |
|---|---|---|---|
| Dependent Mean | 51.85000 | Adj R-Sq | 0.4788 |
| Coeff Var | 13.78624 |
Root MSE: Root MSE is the standard deviation of the error term, and is the square root of the Mean Square Residual (or Error).
Dependent Mean: This is the mean of the dependent variable.
Coeff Var: This is the coefficient of variation, which is a unit-less measure of variation in the data. It is the root MSE divided by the mean of the dependent variable, multiplied by 100: (100*(7.15/51.85) = 13.79).
R-Square: R-Square is the proportion of variance in the dependent variable (science) which can be predicted from the independent variables (math, female, socst and read). This value indicates that 48.92% of the variance in science scores can be predicted from the variables math, female, socst and read. Note that this is an overall measure of the strength of association, and does not reflect the extent to which any particular independent variable is associated with the dependent variable.
Adj R-Sq: Adjusted R-square. As predictors are added to the model, each predictor will explain some of the variance in the dependent variable simply due to chance. One could continue to add predictors to the model which would continue to improve the ability of the predictors to explain the dependent variable, although some of this increase in R-square would be simply due to chance variation in that particular sample. The adjusted R-square attempts to yield a more honest value to estimate the R-squared for the population. The value of R-square was .4892, while the value of Adjusted R-square was .4788 Adjusted R-squared is computed using the formula 1 – ((1 – Rsq)((N – 1) / (N – k – 1)). From this formula, you can see that when the number of observations is small and the number of predictors is large, there will be a much greater difference between R-square and adjusted R-square (because the ratio of (N – 1) / N – k – 1) will be much less than 1). By contrast, when the number of observations is very large compared to the number of predictors, the value of R-square and adjusted R-square will be much closer because the ratio of (N – 1)/(N – k – 1) will approach 1.
Parameter Estimates
| Parameter Estimates | ||||||||
|---|---|---|---|---|---|---|---|---|
| Variable | Label | DF | Parameter Estimate |
Standard Error |
t Value | Pr > |t| | 95% Confidence Limits | |
| Intercept | Intercept | 1 | 12.32529 | 3.19356 | 3.86 | 0.0002 | 6.02694 | 18.62364 |
| math | math score | 1 | 0.38931 | 0.07412 | 5.25 | <.0001 | 0.24312 | 0.53550 |
| female | 1 | -2.00976 | 1.02272 | -1.97 | 0.0508 | -4.02677 | 0.00724 | |
| socst | social studies score | 1 | 0.04984 | 0.06223 | 0.80 | 0.4241 | -0.07289 | 0.17258 |
| read | reading score | 1 | 0.33530 | 0.07278 | 4.61 | <.0001 | 0.19177 | 0.47883 |
Variable: This column shows the predictor variables (constant, math, female, socst, read). The first variable (constant) represents the constant, also referred to in textbooks as the Y intercept, the height of the regression line when it crosses the Y axis. In other words, this is the predicted value of science when all other variables are 0.
Label: This column gives the label for the variable. Usually, variable labels are added when the data set is created so that it is clear what the variable is (as the name of the variable can sometimes be ambiguous). SAS has labeled the variable Intercept for us by default. Note that this variable is not added to the data set.
DF: This column give the degrees of freedom associated with each independent variable. All continuous variables have one degree of freedom, as do binary variables (such as female).
Parameter Estimates: These are the values for the regression equation for predicting the dependent variable from the independent variable. The regression equation is presented in many different ways, for example:
The column of estimates (coefficients or parameter estimates, from here on labeled coefficients) provides the values for b0, b1, b2, b3 and b4 for this equation. Expressed in terms of the variables used in this example, the regression equation is
These estimates tell you about the relationship between the independent variables and the dependent variable. These estimates tell the amount of increase in science scores that would be predicted by a 1 unit increase in the predictor. Note: For the independent variables which are not significant, the coefficients are not significantly different from 0, which should be taken into account when interpreting the coefficients. (See the columns with the t-value and p-value about testing whether the coefficients are significant). math – The coefficient (parameter estimate) is .3893102. So, for every unit (i.e., point, since this is the metric in which the tests are measured) increase in math, a .3893102 unit increase in science is predicted, holding all other variables constant. (It does not matter at what value you hold the other variables constant, because it is a linear model.) Or, for every increase of one point on the math test, your science score is predicted to be higher by .3893102 points. This is significantly different from 0. female – For every unit increase in female, there is a -2.009765 unit decrease in the predicted science score, holding all other variables constant. Since female is coded 0/1 (0=male, 1=female) the interpretation can be put more simply. For females the predicted science score would be 2 points lower than for males. The variable female is technically not statistically significantly different from 0, because the p-value is greater than .05. However, .051 is so close to .05 that some researchers would still consider it to be statistically significant. socst – The coefficient for socst is .0498443. This means that for a 1-unit increase in the social studies score, we expect an approximately .05 point increase in the science score. This is not statistically significant; in other words, .0498443 is not different from 0. read – The coefficient for read is .3352998. Hence, for every unit increase in reading score we expect a .34 point increase in the science score. This is statistically significant.
Standard Error: These are the standard errors associated with the coefficients. The standard error is used for testing whether the parameter is significantly different from 0 by dividing the parameter estimate by the standard error to obtain a t-value (see the column with t-values and p-values). The standard errors can also be used to form a confidence interval for the parameter, as shown in the last two columns of this table.
p. t Value and Pr > |t|– These columns provide the t-value and 2 tailed p-value used in testing the null hypothesis that the coefficient/parameter is 0. If you use a 2 tailed test, then you would compare each p-value to your preselected value of alpha. Coefficients having p-values less than alpha are statistically significant. For example, if you chose alpha to be 0.05, coefficients having a p-value of 0.05 or less would be statistically significant (i.e., you can reject the null hypothesis and say that the coefficient is significantly different from 0). If you use a 1 tailed test (i.e., you predict that the parameter will go in a particular direction), then you can divide the p-value by 2 before comparing it to your preselected alpha level. With a 2-tailed test and alpha of 0.05, you may reject the null hypothesis that the coefficient for female is equal to 0. The coefficient of -2.009765 is significantly greater than 0. However, if you used a 2-tailed test and alpha of 0.01, the p-value of .0255 is greater than 0.01 and the coefficient for female would not be significant at the 0.01 level. Had you predicted that this coefficient would be positive (i.e., a one tail test), you would be able to divide the p-value by 2 before comparing it to alpha. This would yield a one-tailed p-value of 0.00945, which is less than 0.01 and then you could conclude that this coefficient is greater than 0 with a one tailed alpha of 0.01. The coefficient for math is significantly different from 0 using alpha of 0.05 because its p-value is 0.000, which is smaller than 0.05. The coefficient for socst (0.0498443) is not statistically significantly different from 0 because its p-value is definitely larger than 0.05. The coefficient for read (0.3353) is statistically significant because its p-value of 0.000 is less than .05. The constant (_cons) is significantly different from 0 at the 0.05 alpha level. However, having a significant intercept is seldom interesting.
95% Confidence Limits: This shows a 95% confidence interval for the coefficient. This is very useful as it helps you understand how high and how low the actual population value of the parameter might be. The confidence intervals are related to the p-values such that the coefficient will not be statistically significant if the confidence interval includes 0. If you look at the confidence interval for female, you will see that it just includes 0 (-4 to .007). Because .007 is so close to 0, the p-value is close to .05. If the upper confidence level had been a little smaller, such that it did not include 0, the coefficient for female would have been statistically significant. Also, consider the coefficients for female (-2) and read (.34). Immediately you see that the estimate for female is so much bigger, but examine the confidence interval for it (-4 to .007). Now examine the confidence interval for read (.19 to .48). Even though female has a bigger coefficient (in absolute terms) it could be as small as -4. By contrast, the lower confidence level for read is .19, which is still above 0. So, even though female has a bigger coefficient, read is significant and even the smallest value in the confidence interval is still higher than 0. The same cannot be said about the coefficient for socst. Such confidence intervals help you to put the estimate from the coefficient into perspective by seeing how much the value could vary.
This page shows an example regression analysis with footnotes explaining the output. These data (hsb2) were collected on 200 high schools students and are scores on various tests, including science, math, reading and social studies (socst). The variable female is a dichotomous variable coded 1 if the student was female and 0 if male.
In the syntax below, the get file command is used to load the data into SPSS. In quotes, you need to specify where the data file is located on your computer. Remember that you need to use the .sav extension and that you need to end the command with a period. In the regression command, the statistics subcommand must come before the dependent subcommand. You can shorten dependent to dep. You list the independent variables after the equals sign on the method subcommand. The statistics subcommand is not needed to run the regression, but on it we can specify options that we would like to have included in the output. Here, we have specified ci, which is short for confidence intervals. These are very useful for interpreting the output, as we will see. There are four tables given in the output. SPSS has provided some superscripts (a, b, etc.) to assist you in understanding the output.
Please note that SPSS sometimes includes footnotes as part of the output. We have left those intact and have started ours with the next letter of the alphabet.
get file "c:\data\hsb2.sav". regression /statistics coeff outs r anova ci /dependent science /method = enter math female socst read.
| Model | Variables Entered | Variables Removed | Method |
|---|---|---|---|
| 1 | reading score, female, social studies score, math scoreb | . | Enter |
| a. Dependent Variable: science score | |||
| b. All requested variables entered. | |||
Model: SPSS allows you to specify multiple models in a single regression command. This tells you the number of the model being reported.
Variables Entered: SPSS allows you to enter variables into a regression in blocks, and it allows stepwise regression. Hence, you need to know which variables were entered into the current regression. If you did not block your independent variables or use stepwise regression, this column should list all of the independent variables that you specified.
Variables Removed: This column listed the variables that were removed from the current regression. Usually, this column will be empty unless you did a stepwise regression.
Method: This column tells you the method that SPSS used to run the regression. “Enter” means that each independent variable was entered in usual fashion. If you did a stepwise regression, the entry in this column would tell you that.
| Model | R | R Square | Adjusted R Square | Std. Error of the Estimate |
|---|---|---|---|---|
| 1 | .699a | .489 | .479 | 7.14817 |
| a. Predictors: (Constant), reading score, female, social studies score, math score | ||||
Model: SPSS allows you to specify multiple models in a single regression command. This tells you the number of the model being reported.
R: R is the square root of R-Squared and is the correlation between the observed and predicted values of dependent variable.
R-Square: R-Square is the proportion of variance in the dependent variable (science) which can be predicted from the independent variables (math, female, socst and read). This value indicates that 48.9% of the variance in science scores can be predicted from the variables math, female, socst and read. Note that this is an overall measure of the strength of association, and does not reflect the extent to which any particular independent variable is associated with the dependent variable. R-Square is also called the coefficient of determination.
Adjusted R-square: As predictors are added to the model, each predictor will explain some of the variance in the dependent variable simply due to chance. One could continue to add predictors to the model which would continue to improve the ability of the predictors to explain the dependent variable, although some of this increase in R-square would be simply due to chance variation in that particular sample. The adjusted R-square attempts to yield a more honest value to estimate the R-squared for the population. The value of R-square was .489, while the value of Adjusted R-square was .479 Adjusted R-squared is computed using the formula 1 – ((1 – Rsq)(N – 1 )/ (N – k – 1)). From this formula, you can see that when the number of observations is small and the number of predictors is large, there will be a much greater difference between R-square and adjusted R-square (because the ratio of (N – 1) / (N – k – 1) will be much greater than 1). By contrast, when the number of observations is very large compared to the number of predictors, the value of R-square and adjusted R-square will be much closer because the ratio of (N – 1)/(N – k – 1) will approach 1.
Std. Error of the Estimate: The standard error of the estimate, also called the root mean square error, is the standard deviation of the error term, and is the square root of the Mean Square Residual (or Error).
| Model | Sum of Squares | df | Mean Square | F | Sig. | |
|---|---|---|---|---|---|---|
| 1 | Regression | 9543.721 | 4 | 2385.930 | 46.695 | <.001b |
| Residual | 9963.779 | 195 | 51.096 | |||
| Total | 19507.500 | 199 | ||||
| a. Dependent Variable: science score | ||||||
| b. Predictors: (Constant), reading score, female, social studies score, math score | ||||||
Model: SPSS allows you to specify multiple models in a single regression command. This tells you the number of the model being reported.
The column under Model reports the source of variance, Regression, Residual and Total. The Total variance is partitioned into the variance which can be explained by the independent variables (Regression) and the variance which is not explained by the independent variables (Residual, sometimes called Error). Note that the Sums of Squares for the Regression and Residual add up to the Total, reflecting the fact that the Total is partitioned into Regression and Residual variance.
Sum of Squares: These are the Sum of Squares associated with the three sources of variance, Total, Model and Residual. These can be computed in many ways. Conceptually, these formulas can be expressed as:
SSTotal: The total variability around the mean. S(Y – Ybar)2.
SSResidual: The sum of squared errors in prediction. S(Y – Ypredicted)2.
SSRegression: The improvement in prediction by using the predicted value of Y over just using the mean of Y. Hence, this would be the squared differences between the predicted value of Y and the mean of Y, S(Ypredicted – Ybar)2. Another way to think of this is the SSRegression is SSTotal – SSResidual. Note that the SSTotal = SSRegression + SSResidual. Note that SSRegression / SSTotal is equal to .489, the value of R-Square. This is because R-Square is the proportion of the variance explained by the independent variables, hence can be computed by SSRegression / SSTotal.
df: These are the degrees of freedom associated with the sources of variance. The total variance has N-1 degrees of freedom. In this case, there were N=200 students, so the DF for total is 199. The model degrees of freedom corresponds to the number of predictors minus 1 (K-1). You may think this would be 4-1 (since there were 4 independent variables in the model, math, female, socst and read). But, the intercept is automatically included in the model (unless you explicitly omit the intercept). Including the intercept, there are 5 predictors, so the model has 5-1=4 degrees of freedom. The Residual degrees of freedom is the DF total minus the DF model, 199 – 4 is 195.
Mean Square: These are the Mean Squares, the Sum of Squares divided by their respective DF. For the Regression, 9543.72074 / 4 = 2385.93019. For the Residual, 9963.77926 / 195 =51.0963039. These are computed so you can compute the F ratio, dividing the Mean Square Regression by the Mean Square Residual to test the significance of the predictors in the model.
F and Sig.: The F-value is the Mean Square Regression (2385.93019) divided by the Mean Square Residual (51.0963039), yielding F=46.69. The p-value associated with this F value is very small (0.0000). These values are used to answer the question “Do the independent variables reliably predict the dependent variable?”. The p-value is compared to your alpha level (typically 0.05) and, if smaller, you can conclude “Yes, the independent variables reliably predict the dependent variable”. You could say that the group of variables math, and female, socst and read can be used to reliably predict science (the dependent variable). If the p-value were greater than 0.05, you would say that the group of independent variables does not show a statistically significant relationship with the dependent variable, or that the group of independent variables does not reliably predict the dependent variable. Note that this is an overall significance test assessing whether the group of independent variables when used together reliably predict the dependent variable, and does not address the ability of any of the particular independent variables to predict the dependent variable. The ability of each individual independent variable to predict the dependent variable is addressed in the table below where each of the individual variables are listed.
| Model | Unstandardized Coefficients | Standardized Coefficients | t | Sig. | 95.0% Confidence Interval for B | |||
|---|---|---|---|---|---|---|---|---|
| B | Std. Error | Beta | Lower Bound | Upper Bound | ||||
| 1 | (Constant) | 12.325 | 3.194 | 3.859 | <.001 | 6.027 | 18.624 | |
| math score | .389 | .074 | .368 | 5.252 | <.001 | .243 | .535 | |
| female | -2.010 | 1.023 | -.101 | -1.965 | .051 | -4.027 | .007 | |
| social studies score | .050 | .062 | .054 | .801 | .424 | -.073 | .173 | |
| reading score | .335 | .073 | .347 | 4.607 | <.001 | .192 | .479 | |
| a. Dependent Variable: science score | ||||||||
Model: SPSS allows you to specify multiple models in a single regression command. This tells you the number of the model being reported.
The column under Model shows the predictor variables (constant, math, female, socst, read). The first variable (constant) represents the constant, also referred to in textbooks as the Y intercept, the height of the regression line when it crosses the Y axis. In other words, this is the predicted value of science when all other variables are 0.
B: These are the values for the regression equation for predicting the dependent variable from the independent variable. These are called unstandardized coefficients because they are measured in their natural units. As such, the coefficients cannot be compared with one another to determine which one is more influential in the model, because they can be measured on different scales. For example, how can you compare the values for gender with the values for reading scores? The regression equation can be presented in many different ways, for example:
Ypredicted = b0 + b1*x1 + b2*x2 + b3*x3 + b3*x3 + b4*x4
The column of estimates (coefficients or parameter estimates, from here on labeled coefficients) provides the values for b0, b1, b2, b3 and b4 for this equation. Expressed in terms of the variables used in this example, the regression equation is
sciencePredicted = 12.325 +.389*math + -2.010*female+.050*socst+.335*read
These estimates tell you about the relationship between the independent variables and the dependent variable. These estimates tell the amount of increase in science scores that would be predicted by a 1 unit increase in the predictor. Note: For the independent variables which are not significant, the coefficients are not significantly different from 0, which should be taken into account when interpreting the coefficients. (See the columns with the t-value and p-value about testing whether the coefficients are significant).
math – The coefficient (parameter estimate) is .389. So, for every unit (i.e., point, since this is the metric in which the tests are measured) increase in math, a .389 unit increase in science is predicted, holding all other variables constant. (It does not matter at what value you hold the other variables constant, because it is a linear model.) Or, for every increase of one point on the math test, your science score is predicted to be higher by .389 points. This is significantly different from 0.
female: For every unit increase in female, there is a -2.010 unit decrease in the predicted science score, holding all other variables constant. Since female is coded 0/1 (0=male, 1=female) the interpretation can be put more simply. For females the predicted science score would be 2 points lower than for males. The variable female is technically not statistically significantly different from 0, because the p-value is greater than .05. However, .051 is so close to .05 that some researchers would still consider it to be statistically significant.
socst: The coefficient for socst is .050. This means that for a 1-unit increase in the social studies score, we expect an approximately .05 point increase in the science score. This is not statistically significant; in other words, .050 is not different from 0.
read: The coefficient for read is .335. Hence, for every unit increase in reading score we expect a .335 point increase in the science score. This is statistically significant.
Std. Error: These are the standard errors associated with the coefficients. The standard error is used for testing whether the parameter is significantly different from 0 by dividing the parameter estimate by the standard error to obtain a t-value (see the column with t-values and p-values). The standard errors can also be used to form a confidence interval for the parameter, as shown in the last two columns of this table.
Beta: These are the standardized coefficients. These are the coefficients that you would obtain if you standardized all of the variables in the regression, including the dependent and all of the independent variables, and ran the regression. By standardizing the variables before running the regression, you have put all of the variables on the same scale, and you can compare the magnitude of the coefficients to see which one has more of an effect. You will also notice that the larger betas are associated with the larger t-values.
t and Sig.: These columns provide the t-value and 2 tailed p-value used in testing the null hypothesis that the coefficient/parameter is 0. If you use a 2 tailed test, then you would compare each p-value to your preselected value of alpha. Coefficients having p-values less than alpha are statistically significant. For example, if you chose alpha to be 0.05, coefficients having a p-value of 0.05 or less would be statistically significant (i.e., you can reject the null hypothesis and say that the coefficient is significantly different from 0). If you use a 1 tailed test (i.e., you predict that the parameter will go in a particular direction), then you can divide the p-value by 2 before comparing it to your preselected alpha level. With a 2-tailed test and alpha of 0.05, you should not reject the null hypothesis that the coefficient for female is equal to 0, because p-value = 0.051 > 0.05. The coefficient of -2.009765 is not significantly different from 0. However, if you hypothesized specifically that males had higher scores than females (a 1-tailed test) and used an alpha of 0.05, the p-value of .0255 is less than 0.05 and the coefficient for female would be significant at the 0.05 level. In this case, we could say that the female coefficient is significantly greater than 0. Neither a 1-tailed nor 2-tailed test would be significant at alpha of 0.01.
The constant is significantly different from 0 at the 0.05 alpha level. However, having a significant intercept is seldom interesting.
The coefficient for math (.389) is statistically significantly different from 0 using alpha of 0.05 because its p-value is 0.000, which is smaller than 0.05.
The coefficient for female (-2.01) is not statistically significant at the 0.05 level since the p-value is greater than .05.
The coefficient for socst (.05) is not statistically significantly different from 0 because its p-value is definitely larger than 0.05.
The coefficient for read (.335) is statistically significant because its p-value of 0.000 is less than .05.
[95% Conf. Interval]: These are the 95% confidence intervals for the coefficients. The confidence interval gives values of the coefficient that are reasonably consistent with the data. Assuming the model is correct, if we were to repeatedly sample from this population and construct 95% confidence intervals for each sample, 95% of those confidence intervals should contain the true parameter value.
This page shows an example regression analysis with footnotes explaining the output. These data were collected on 200 high schools students and are scores on various tests, including science, math, reading and social studies (socst). The variable female is a dichotomous variable coded 1 if the student was female and 0 if male.
use https://stats.idre.ucla.edu/stat/stata/notes/hsb2
(highschool and beyond (200 cases))
regress science math female socst read
Source | SS df MS Number of obs = 200
-------------+------------------------------ F( 4, 195) = 46.69
Model | 9543.72074 4 2385.93019 Prob > F = 0.0000
Residual | 9963.77926 195 51.0963039 R-squared = 0.4892
-------------+------------------------------ Adj R-squared = 0.4788
Total | 19507.5 199 98.0276382 Root MSE = 7.1482
------------------------------------------------------------------------------
science | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
math | .3893102 .0741243 5.25 0.000 .243122 .5354983
female | -2.009765 1.022717 -1.97 0.051 -4.026772 .0072428
socst | .0498443 .062232 0.80 0.424 -.0728899 .1725784
read | .3352998 .0727788 4.61 0.000 .1917651 .4788345
_cons | 12.32529 3.193557 3.86 0.000 6.026943 18.62364
------------------------------------------------------------------------------
Anova Table
Sourcea | SSb dfc MSd
-------------+------------------------------
Model | 9543.72074 4 2385.93019
Residual | 9963.77926 195 51.0963039
-------------+------------------------------
Total | 19507.5 199 98.0276382
a. Source – This is the source of variance, Model, Residual, and Total. The Total variance is partitioned into the variance which can be explained by the independent variables (Model) and the variance which is not explained by the independent variables (Residual, sometimes called Error). Note that the Sums of Squares for the Model and Residual add up to the Total Variance, reflecting the fact that the Total Variance is partitioned into Model and Residual variance.
b. SS – These are the Sum of Squares associated with the three sources of variance, Total, Model and Residual. These can be computed in many ways. Conceptually, these formulas can be expressed as: SSTotal The total variability around the mean. S(Y – Ybar)2. SSResidual The sum of squared errors in prediction. S(Y – Ypredicted)2. SSModel The improvement in prediction by using the predicted value of Y over just using the mean of Y. Hence, this would be the squared differences between the predicted value of Y and the mean of Y, S(Ypredicted – Ybar)2. Another way to think of this is the SSModel is SSTotal – SSResidual. Note that the SSTotal = SSModel + SSResidual. Note that SSModel / SSTotal is equal to .4892, the value of R-Square. This is because R-Square is the proportion of the variance explained by the independent variables, hence can be computed by SSModel / SSTotal.
c. df – These are the degrees of freedom associated with the sources of variance. The total variance has N-1 degrees of freedom. In this case, there were N=200 students, so the DF for total is 199. The model degrees of freedom corresponds to the number of predictors minus 1 (K-1). You may think this would be 4-1 (since there were 4 independent variables in the model, math, female, socst and read). But, the intercept is automatically included in the model (unless you explicitly omit the intercept). Including the intercept, there are 5 predictors, so the model has 5-1=4 degrees of freedom. The Residual degrees of freedom is the DF total minus the DF model, 199 – 4 is 195.
d. MS – These are the Mean Squares, the Sum of Squares divided by their respective DF. For the Model,
9543.72074 / 4 = 2385.93019. For the Residual, 9963.77926 / 195 =
51.0963039. These are computed so you can compute the F ratio, dividing the Mean Square Model by the Mean Square Residual to test the significance of the predictors in the model.
Overall Model Fit
Number of obse = 200 F( 4, 195)f = 46.69 Prob > Ff = 0.0000 R-squaredg = 0.4892 Adj R-squaredh = 0.4788 Root MSEi = 7.1482
e. Number of obs – This is the number of observations used in the regression analysis.
f. F and Prob > F – The F-value is the Mean Square Model (2385.93019) divided by the Mean Square Residual (51.0963039), yielding F=46.69. The p-value associated with this F value is very small (0.0000). These values are used to answer the question “Do the independent variables reliably predict the dependent variable?”. The p-value is compared to your alpha level (typically 0.05) and, if smaller, you can conclude “Yes, the independent variables reliably predict the dependent variable”. You could say that the group of variables math and female can be used to reliably predict science (the dependent variable). If the p-value were greater than 0.05, you would say that the group of independent variables does not show a statistically significant relationship with the dependent variable, or that the group of independent variables does not reliably predict the dependent variable. Note that this is an overall significance test assessing whether the group of independent variables when used together reliably predict the dependent variable, and does not address the ability of any of the particular independent variables to predict the dependent variable. The ability of each individual independent variable to predict the dependent variable is addressed in the table below where each of the individual variables are listed.
g. R-squared – R-Squared is the proportion of variance in the dependent variable (science) which can be predicted from the independent variables (math, female, socst and read). This value indicates that 48.92% of the variance in science scores can be predicted from the variables math, female, socst and read. Note that this is an overall measure of the strength of association, and does not reflect the extent to which any particular independent variable is associated with the dependent variable.
h. Adj R-squared – Adjusted R-square. As predictors are added to the model, each predictor will explain some of the variance in the dependent variable simply due to chance. One could continue to add predictors to the model which would continue to improve the ability of the predictors to explain the dependent variable, although some of this increase in R-square would be simply due to chance variation in that particular sample. The adjusted R-square attempts to yield a more honest value to estimate the R-squared for the population. The value of R-square was .4892, while the value of Adjusted R-square was .4788 Adjusted R-squared is computed using the formula 1 – ((1 – Rsq)((N – 1) /( N – k – 1)). From this formula, you can see that when the number of observations is small and the number of predictors is large, there will be a much greater difference between R-square and adjusted R-square (because the ratio of (N – 1) / (N – k – 1) will be much greater than 1). By contrast, when the number of observations is very large compared to the number of predictors, the value of R-square and adjusted R-square will be much closer because the ratio of (N – 1)/(N – k – 1) will approach 1.
i. Root MSE – Root MSE is the standard deviation of the error term, and is the square root of the Mean Square Residual (or Error).
Parameter Estimates
------------------------------------------------------------------------------
sciencej | Coef.k Std. Err.l tm P>|t|m [95% Conf. Interval]n
-------------+----------------------------------------------------------------
math | .3893102 .0741243 5.25 0.000 .243122 .5354983
female | -2.009765 1.022717 -1.97 0.051 -4.026772 .0072428
socst | .0498443 .062232 0.80 0.424 -.0728899 .1725784
read | .3352998 .0727788 4.61 0.000 .1917651 .4788345
_cons | 12.32529 3.193557 3.86 0.000 6.026943 18.62364
------------------------------------------------------------------------------
j. science – This column shows the dependent variable at the top (science) with the predictor variables below it (math, female, socst, read and _cons). The last variable (_cons) represents the constant, also referred to in textbooks as the Y intercept, the height of the regression line when it crosses the Y axis. In other words, this is the predicted value of science when all other variables are 0.
k. Coef. – These are the values for the regression equation for predicting the dependent variable from the independent variable. The regression equation is presented in many different ways, for example:
Ypredicted = b0 + b1*x1 + b2*x2 + b3*x3 + b4*x4
The column of estimates (coefficients or parameter estimates, from here on labeled coefficients) provides the values for b0, b1, b2, b3 and b4 for this equation. Expressed in terms of the variables used in this example, the regression equation is
sciencePredicted = 12.32529 + .3893102*math + -2.009765*female+.0498443*socst+.3352998*read
These estimates tell you about the relationship between the independent variables and the dependent variable. These estimates tell the amount of increase in science scores that would be predicted by a 1 unit increase in the predictor. Note: For the independent variables which are not significant, the coefficients are not significantly different from 0, which should be taken into account when interpreting the coefficients. (See the columns with the t-value and p-value about testing whether the coefficients are significant). math – The coefficient (parameter estimate) is
.3893102. So, for every unit (i.e., point, since this is the metric in which the tests are measured) increase in math, a .3893102 unit increase in science is predicted, holding all other variables constant. (It does not matter at what value you hold the other variables constant, because it is a linear model.) Or, for every increase of one point on the math test, your science score is predicted to be higher by .3893102 points. This is significantly different from 0. female – For every unit increase in female, there is a
-2.009765 unit decrease in the predicted science score, holding all other variables constant. Since female is coded 0/1 (0=male, 1=female) the interpretation can be put more simply. For females the predicted science score would be 2 points lower than for males. The variable female is technically not statistically significantly different from 0, because the p-value is greater than .05. However, .051 is so close to .05 that some researchers would still consider it to be statistically significant. socst – The coefficient for socst is .0498443. This means that for a 1-unit increase in the social studies score, we expect an approximately .05 point increase in the science score. This is not statistically significant; in other words, .0498443 is not different from 0. read – The coefficient for read is .3352998. Hence, for every unit increase in reading score we expect a .34 point increase in the science score. This is statistically significant.
l. Std. Err. – These are the standard errors associated with the coefficients. The standard error is used for testing whether the parameter is significantly different from 0 by dividing the parameter estimate by the standard error to obtain a t-value (see the column with t-values and p-values). The standard errors can also be used to form a confidence interval for the parameter, as shown in the last two columns of this table.
m. t and P>|t| – These columns provide the t-value and 2-tailed p-value used in testing the null hypothesis that the coefficient (parameter) is 0. If you use a 2-tailed test, then you would compare each p-value to your pre-selected value of alpha. Coefficients having p-values less than alpha are statistically significant. For example, if you chose alpha to be 0.05, coefficients having a p-value of 0.05 or less would be statistically significant (i.e., you can reject the null hypothesis and say that the coefficient is significantly different from 0). If you use a 1-tailed test (i.e., you hypothesize that the parameter will go in a particular direction), then you can divide the p-value by 2 before comparing it to your pre-selected alpha level.
The coefficient for female (-2.009765) is technically not significantly different from 0 because with a 2-tailed test and alpha of 0.05, the p-value of 0.051 is greater than 0.05. However, if you used a 1-tailed test, the p-value is now (0.051/2=.0255), which is less than 0.05 and then you could conclude that this coefficient is less than 0. CAUTION: We do not recommend changing from a two-tailed test to a one-tailed test after running your regression. This would be statistical cheating! You must know the direction of your hypothesis before running your regression.
The coefficient for math (3893102) is significantly different from 0 using alpha of 0.05 because its p-value is 0.000, which is smaller than 0.05.
The coefficient for socst (.0498443) is not statistically significantly different from 0 because its p-value is definitely larger than 0.05.
The coefficient for read (.3352998) is statistically significant because its p-value of 0.000 is less than .05.
The constant (_cons) is significantly different from 0 at the 0.05 alpha level. However, having a significant intercept is seldom interesting.
n. [95% Conf. Interval] – This shows a 95% confidence interval for the coefficient. This is very useful as it helps you understand how high and how low the actual population value of the parameter might be. The confidence intervals are related to the p-values such that the coefficient will not be statistically significant if the confidence interval includes 0. If you look at the confidence interval for female, you will see that it just includes 0 (-4 to .007). Because .007 is so close to 0, the p-value is close to .05. If the upper confidence level had been a little smaller, such that it did not include 0, the coefficient for female would have been statistically significant. Also, consider the coefficients for female (-2) and read (.34). Immediately you see that the estimate for female is so much bigger, but examine the confidence interval for it (-4 to .007). Now examine the confidence interval for read (.19 to .48). Even though female has a bigger coefficient (in absolute terms) it could be as small as -4. By contrast, the lower confidence level for read is .19, which is still above 0. So, even though female has a bigger coefficient, read is significant and even the smallest value in the confidence interval is still higher than 0. The same cannot be said about the coefficient for socst. Such confidence intervals help you to put the estimate from the coefficient into perspective by seeing how much the value could vary.