Fit the model using the statsmodels OLS library
This module covers the following steps:
➤ 1. Preparing Data for Regression Analysis
➤ 2. Importing statsmodels Library
➤ 3. OLS Regression results
➤ 4. Model Summary
➤ 5. Coefficients Table
➤ 6. Potential Issues
➤ 7. Residual Analysis
➤ 8. Predict and test the model
Fit the model using the statsmodels OLS library
↪ 1. Preparing Data for Regression Analysis
Import the pre-processed data for analysis. Subsequently, the data will be partitioned into training and test sets to facilitate the analysis.
# Import required libraries
import pandas as pd # Matplotlib is the fundamental plotting library
import matplotlib.pyplot as plt # Seaborn builds upon Matplotlib, offering a
import seaborn as sns # higher-level interface for statistical visualization.
import numpy as np
# Import data
data = pd.read_csv('https://raw.githubusercontent.com/csxplore/data/main/andromeda-cleaned.csv',
header=0)
data = data[['Prev Close','Close Price']]
x=data[['Prev Close']]
y=data[['Close Price']]
# Set default style and color scheme for Seaborn plots
sns.set(style="ticks", color_codes=True)
# Split the data into training and test sets
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x,y, test_size=0.2,random_state=0)
Fit the model using the statsmodels OLS library
↪ 2. Importing statsmodels Library for OLS Regression
The statsmodels library, by default, fits a line through the origin. To include an intercept, manually add a constant to the x_train dataset using the add_constant attribute of statsmodels.
After adding the constant, fit a regression line using the OLS (Ordinary Least Squares) attribute of statsmodels.
import statsmodels.api as sm x_train_sm = sm.add_constant(x_train) # Add a constant to get an intercept lr = sm.OLS(y_train, x_train_sm).fit() # Fit the regression line using 'OLS'
A fitted model's params property provides the slope and intercept values.
print(lr.params)
---Output--- # const 16.587365 # Prev Close 0.990746 # dtype: float64
In the output, 'const' represents the intercept, and 'Prev Close' represents the slope for 'Close Price'.
Fit the model using the statsmodels OLS library
↪ 3. OLS Regression results
The summary() method summarizes the regression results.
print(lr.summary())
---Output--- # OLS Regression Results # ============================================================================== # Dep. Variable: Close Price R-squared: 0.977 # Model: OLS Adj. R-squared: 0.977 # Method: Least Squares F-statistic: 8279. # Date: Wed, 25 Oct 2023 Prob (F-statistic): 1.71e-163 # Time: 20:19:48 Log-Likelihood: -854.93 # No. Observations: 200 AIC: 1714. # Df Residuals: 198 BIC: 1720. # Df Model: 1 # Covariance Type: nonrobust # ============================================================================== # coef std err t P>|t| [0.025 0.975] # ------------------------------------------------------------------------------ # const 16.5874 16.749 0.990 0.323 -16.442 49.617 # Prev Close 0.9907 0.011 90.987 0.000 0.969 1.012 # ============================================================================== # Omnibus: 26.996 Durbin-Watson: 1.953 # Prob(Omnibus): 0.000 Jarque-Bera (JB): 66.459 # Skew: 0.577 Prob(JB): 3.70e-15 # Kurtosis: 5.578 Cond. No. 2.09e+04 # ==============================================================================
Fit the model using the statsmodels OLS library
↪ 4. Model Summary
Dep. variable The 'Close Price' is a dependent variable dependent on the variable 'Prev Close'.
Model/Method The Ordinary Least Squares (OLS) method provides the best approximation of the true population regression line by minimizing the sum of squared errors.
Degrees of freedom Degrees of freedom determine the shape of the t-distribution, which is used in t-tests to calculate p-values.
Sets with lower degrees of freedom have a higher probability of extreme values. Conversely, sets with higher degrees of freedom, such as a sample size of 30 or more, approach a normal distribution. Smaller sample sizes correspond to fewer degrees of freedom and result in wider (fatter) t-distribution tails.
From the summary table above, Number of observations (it is the size of data) = 200 Degrees of freedom (Df) (it is Df of residuals) = 198 Df of model = 1
⚠ HOTS
Degree of freedom(df) of residuals:
Degrees of freedom represent the number of independent observations used to calculate the sum of squares.
# K = (number of variables + 1) = (1 + 1) = 2 # Df Residuals = No. of observations - K # = 200 – (2) = 198 # # Df of model = (K - 1) # Df of model = (2 – 1) = 1
Fit the model using the statsmodels OLS library
↪ 4. Model Summary
Covariance Type The “Covariance Type” in OLS regression refers to the method used to estimate the covariance matrix of the regression coefficients. This matrix is crucial for calculating standard errors, t-statistics, and confidence intervals.
The nonrobust method assumes that the errors in your model are homoscedastic (have constant variance) and uncorrelated.
Other Covariance Types are:
Fit the model using the statsmodels OLS library
↪ 4. Model Summary
R-squared and Adj. R-squared The coefficient of determination, R-squared (R²), measures how well a statistical model predicts an outcome.
Because the model includes only one independent variable, the adjusted R-squared is identical to the regular R-squared. The adjusted R-squared is particularly useful when comparing models with different numbers of independent variables or multiple regression models with several predictors.
F-statistic and Prob (F-statistic) While R-squared tells about the proportion of variance explained by the model, F-statistic tests the statistical significance of the overall model, indicating if the independent variables as a whole have a meaningful effect on the dependent variable.
The F-statistic tests the hypothesis of a linear regression model's existence. This is done by setting up a null hypothesis. The following example illustrates the hypothesis definition for a linear regression model.
# y = β0 + β1.x1, where # y is the dependent variable, # x1 is an independent variable, and # β1 is a coefficient to be estimated for x1 # # Then, the null and alternate hypotheses can be written as: # H0: β1 = 0, Regression model does not exist # Ha: β1 ≠ 0, Regression model exists
A higher F-value indicates stronger evidence against the null hypothesis. This means that the independent variables collectively explain a significant portion of the variation in the dependent variable. A value of 8279 is high, suggesting strong evidence against the null hypothesis.
A low p-value (typically less than 0.05) suggests that there is sufficient evidence to reject the null hypothesis and to conclude that the model as a whole is statistically significant. The p-value of 1.71e-163 is much less than 0.05, providing strong evidence to reject the null hypothesis.
Fit the model using the statsmodels OLS library
↪ 4. Model Summary
AIC and BIC Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are two model selection criteria used in statistical modeling. AIC and BIC are calculated for different models by varying the sets of independent variables included.
A model with more independent variables is generally more complex. While a more complex model can often achieve a higher R-squared, it may overfit the data.
R-squared measures the goodness of fit, indicating the proportion of variance explained by the model. Meanwhile, AIC and BIC consider model complexity. By comparing R-squared, AIC, and BIC across different models, one can determine the optimal balance between goodness of fit and model complexity.
Log-Likelihood Log-likelihood measures how well a model fits the data, taking into account the model's complexity. A higher likelihood indicates that the model is more likely to have produced the observed data. Log-likelihood is simply the natural logarithm of the likelihood.
Log-likelihood values from different models can be directly compared when the models have the same number of observations. However, log-likelihood values are often used in conjunction with AIC and BIC.
A higher log-likelihood value generally indicates a better fit.
Fit the model using the statsmodels OLS library
↪ 5. Coefficients Table
coef From the summary table, Intercept: The values of const coef = 16.5874 Slope: The values of 'Prev Close' coef = 0.9907 So, the linear equation of the fitted line is y = 0.9907x + 16.5874
std err The standard error of the regression slope, often called the standard error of regression (SER), provides a measure of the uncertainty in the estimated slope of a linear regression model.
The SER, which is 0.011, measures the variability around the estimated regression slope. It is used to calculate the t-statistic for the predictor variable “Prev Close”.
Fit the model using the statsmodels OLS library
↪ 5. Coefficients Table
t-distribution and p-value The t-distribution is a probability distribution resembling the normal distribution with its bell shape, but it has heavier tails. This means t-distributions have a greater chance for extreme values than normal distributions, resulting in fatter tails and higher kurtosis.
The t-statistic is used in a t-test to determine whether to support or reject the null hypothesis.
If the null hypothesis is H0 : Prev Close = 0 ( variable Prev Close does not influence Close Price) Ha : Prev Close ≠ 0 ( Prev Close has a significant impact on Close Price)
From the summary table, the t-score of “Prev Close” or slope variable is 90.987.
⚠ HOTS
Recall that the linear equation defined earlier, which is E(y|x) = ax + b i.e. y = 0.9907x + 16.5874, and from the summary table Std error (SER) of the slope parameter = 0.011, and A standard normal distribution has a mean (M) of 0, then A t-score of the slope term is calculated as t = (a – M)/SER(a) = (0.9907 – 0) / 0.011 = 90.987
In the summary table above, the P>|t| value for the slope parameter is 0.000. This indicates that the probability of obtaining a t-value of 90.987 or greater, assuming the null hypothesis is true, is extremely low. Therefore, we reject the null hypothesis and conclude that the alternative hypothesis – that “Prev Close” has a significant impact on “Close Price” – is supported by the data.
⚠ HOTS
A p-value is the probability of getting a result equal to or greater than what was observed, assuming that the Null hypothesis is true. In simpler terms, the p-value indicates the likelihood that the observed results occurred due to random chance.
A p-value measures the strength of the evidence against H0:
Fit the model using the statsmodels OLS library
↪ 5. Coefficients Table
Confidence Interval The column labeled “[0.025 0.975]” represents the 95% confidence interval for each regression coefficient.
The confidence interval in the above table is 0.969 and 1.012, which means 95% confidence that the true population coefficient for the impact of 'Prev Close' on 'Close Price' lies between 0.969 and 1.012.
Since the confidence interval does not include zero, the coefficient is statistically significant at the 95% level i.e. 'Prev Close' has a statistically significant impact on 'Close Price'.
Fit the model using the statsmodels OLS library
↪ 6. Potential Issues
Omnibus and Prob(Omnibus) The Omnibus test assesses the normality of the residuals: the difference between the actual values and the predicted values in a regression model are normally distributed. The test utilizes a combination of skewness and kurtosis: skewness measures the asymmetry of the distribution, while kurtosis measures its “tailedness.”
The null and alternate hypotheses can be written as: H0: The residuals are normally distributed. Ha: The residuals are not normally distributed.
A low p-value (typically less than 0.05) indicates that you can reject the null hypothesis, meaning there's evidence that the residuals are not normally distributed.
The p-value 0.000 indicates that the model's residuals are not normally distributed.
The next step is to examine Residual Plots: histograms and other diagnostic plots of the residuals to visually assess their distribution.
Fit the model using the statsmodels OLS library
↪ 6. Potential Issues
Skew and Kurtosis Skewness measures the asymmetry of the distribution. A symmetrical distribution (like a normal distribution) has a skewness of 0.
Kurtosis measures the “tailedness” or “peakedness” of a distribution. A normal distribution has a kurtosis of 3.
The model has a positive skew and kurtosis > 3. The next step is to visualize the residuals with histograms and other diagnostic plots. These plots can help confirm the presence of skewness or kurtosis and provide more insight into the distribution.
Fit the model using the statsmodels OLS library
↪ 6. Potential Issues
Durbin-Watson The Durbin-Watson statistic shows the presence of autocorrelation in the residuals of an OLS regression model. The consequences of the existence of autocorrelation are
Autocorrelation refers to the lack of independence between values. The value of the Durbin-Watson statistic will lie between 0 and 4.
From the summary table, the Durbin-Watson statistic is 1.953. This means the data has no autocorrelation i.e. all observations are independent.
Fit the model using the statsmodels OLS library
↪ 6. Potential Issues
Jarque-Bera (JB) and Prob(JB) Similar to the Omnibus, the JB test checks if the residuals from the OLS model are normally distributed. The JB test calculates a statistic based on the skewness and kurtosis of the residuals.
The null and alternate hypotheses can be written as: H0: The residuals are normally distributed. Ha: The residuals are not normally distributed.
The p-value, Prob(JB), represents the probability of observing a JB statistic as extreme as the one calculated if the null hypothesis were true. A low p-value (usually less than 0.05) indicates strong evidence to reject the null hypothesis, suggesting that the residuals are not normally distributed.
The JB statistic of 66.459 indicates a large deviation from normality. The Prob(JB) of 3.70e-15 is less than 0.05, meaning reject the null hypothesis. Non-normal residuals might indicate issues like:
Fit the model using the statsmodels OLS library
↪ 6. Potential Issues
Cond. No. The “Cond. No.” (Condition Number) in OLS regression output measures a model's sensitivity to small changes in the data.
A low condition number (below 10) indicates a model that is less sensitive to small changes in the data. Conversely, a high condition number (generally above 10) suggests that even minor variations in the data can lead to significant changes in the estimated coefficients and other model outputs.
A high condition number often indicates multicollinearity, a situation where independent variables are highly correlated with each other. This can make it difficult for the model to accurately estimate the coefficients for individual predictors.
The condition number in this case is large, 2.09e+04. This might indicate that there is strong multicollinearity or other numerical problems.
Fit the model using the statsmodels OLS library
↪ 7. Residual Analysis
If a linear model makes sense, the
Fit the model using the statsmodels OLS library
↪ 7. Residual Analysis
Normal Distribution
The value of high kurtosis ( > 3 ) and Jarque-Bera (JB) number in the model indicates that the data has no normal distribution, i.e.and statistical analysis may perform poorly on the data.
A histogram of residuals and a normal probability plot of residuals can be used to evaluate whether the residuals are approximately normally distributed or not.
from scipy import stats mu, std = stats.norm.fit(lr.resid) fig, ax = plt.subplots() sns.histplot(x=lr.resid, ax=ax, stat="density", linewidth=0, kde=True) ax.set(title="Distribution of residuals", xlabel="residual") # # plot corresponding normal curve xmin, xmax = plt.xlim() # the maximum x values from the histogram above x = np.linspace(xmin, xmax, 100) # generate some x values p = stats.norm.pdf(x, mu, std) # calculate the y values for the normal curve sns.lineplot(x=x, y=p, color="orange", ax=ax) plt.show()
While the curve is slightly skewed, it has not deviated significantly from a normal distribution. Unless the residuals are substantially non-normal or exhibit a clear pattern, normality is not a major concern.
Fit the model using the statsmodels OLS library
↪ 7. Residual Analysis
Constant Variance
Homoscedasticity, also known as homogeneity of variances, describes a situation where the error term (or variance) in a regression model is constant across all values of the independent variables.
Heteroscedasticity (the violation of Homoscedasticity) is present when the size of the error term differs across values of an independent variable.
Breusch-Pagan test checks whether Heteroscedasticity exists in the regression model. The test follows the hypotheses below:
The null hypothesis is H0 : Homoscedasticity is present (the residuals are distributed with equal variance) Ha : Heteroscedasticity is present (the residuals are not distributed with equal variance)
from statsmodels.compat import lzip import statsmodels.stats.api as sms names = ['Lagrange multiplier statistic', 'p-value', 'f-value', 'f p-value'] test_result = sms.het_breuschpagan(lr.resid, lr.model.exog) print(lzip(names, test_result))
---Output--- # [('Lagrange multiplier statistic', 0.010023637585443446), # ('p-value', 0.9202505513848309), # ('f-value', 0.009923898577395387), # ('f p-value', 0.9207477833851023)]
The Lagrange multiplier statistic for the test is 0.01, with a corresponding p-value of 0.92. Since the p-value is greater than 0.05, we fail to reject the null hypothesis. This indicates that there is insufficient evidence to suggest heteroscedasticity in the regression model.
Heteroscedasticity can also be visually assessed by examining a scatterplot of residuals versus fitted values. In a homoscedastic model, the variance of the residuals should be constant across all values of the x-axis. This same scatterplot can also help to identify autocorrelation, which is discussed in the next section.
Fit the model using the statsmodels OLS library
↪ 7. Residual Analysis
Independence
Autocorrelation refers to the lack of independence between values. Durbin Watson statistic checks for the presence of autocorrelation in lag 1 of the residuals. The value of the statistic will lie between 0 to 4.
From the summary table, the Durbin-Watson statistic is 1.953. This means the data has no autocorrelation i.e. all observations are independent.
An existence of autocorrelation can be visualized with a scatter plot with residuals on one axis and the time component on the other axis. If the residuals are randomly distributed, there is no autocorrelation. If a specific pattern is observed, it indicates the presence of autocorrelation.
plt.scatter(y_train, lr.resid, c = "g", s = 60)
plt.hlines(y=0, xmin=1250, xmax=1750)
plt.title("Residual Plot")
plt.ylabel("Residual")
plt.show()
The points are randomly scattered, and it does not appear that there is a relationship.
A final note: Detecting residual patterns can improve the model.
Fit the model using the statsmodels OLS library
↪ 8. Predict and test the model
y_pred_ols = lr.predict(sm.add_constant(x_test)) y_pred = pd.DataFrame(y_pred_ols, columns=['Pred']) dframe = pd.concat([y_test.reset_index(drop=True).astype(float), y_pred.reset_index(drop=True).astype(float)], axis=1) dframe.columns = ['Actual','Predicted'] graph = dframe.head(10) print(graph)
---Output--- # - Actual Predicted # 0 1671.80 1665.338077 # 1 1617.65 1626.698965 # 2 1409.80 1404.276386 # 3 1600.65 1601.187244 # 4 1568.30 1599.304826 # 5 1644.10 1638.736535 # 6 1564.35 1570.622716 # 7 1389.55 1417.354240 # 8 1486.10 1442.667811 # 9 1590.90 1570.375029
Plot Graph
graph.plot(kind='bar')
plt.title('Actual vs Predicted')
plt.ylabel('Closing price')
plt.show()
Fit the model using the statsmodels OLS library
↪ 8. Predict and test the model
Predicting Close Price
Predict_Close_Price= lr.predict(np.array([1,1508.80]))
print("Predicted Value: ", Predict_Close_Price)
---Output---
# Predicted Value: [1511.42561533]