Forward stagewise sparse regression estimation implemented for gretl.
Package for computing forward-stagewise shrinkage and selection regressions.
So called shrinkage and/ or selection estimators such as Ridge or Lasso among others are known to handle such issues by imposing an additional restriction to an otherwise ordinary least square setting. Another alternative estimation approach is the so called forward-stagewise regression approach (fsboost henceforth).
fsboost is a simple strategy for constructing a sequence of sparse regression estimates: Initially set all coefficients to zero, and iteratively update the coefficient (by a small amount, depending on the learning rate) of the variable that achieves (under quadratic loss) the maximal absolute correlation with the current residual.
Learning from the residuals has some connection to an approach known as boosting in the machine-learning community.
References
A detailed help file can be found here: https://github.com/atecon/fsboost/blob/master/docs/fsboost.pdf
Get the package from the gretl package server and install it:
pkg install fsboost
Once the package is installed, the user can access the GUI interface via the “Model —> Other linear models —> Forward Stagewise” menu. The interface will look like this:
Here is a sample script on how to use it (see also: https://raw.githubusercontent.com/atecon/fsboost/master/src/fsboost_sample.inp):
First, load the package and open a the well-known cross-sectional data set mroz87.gdt (723 observations). In this example, we ‘model’ the hourly wages of women, WW, by means of 17 features (exogenous variables). The fsreg() function calles the the linear regression computation. All relevant output is stored in the returned bundle (a kind of struct data type) named B here:
clearset verbose offinclude fsboost.gfnlist RHS = const datasetRHS -= LHS WW # drop endogenous variablebundle B = fsreg(WW, RHS, opts) # Run estimationprint B # Print content of the returned bundle
Estimation does not even take half a second.
Once the computation is finished, the user can print the summary results by means of the print_fsboost_results() function:
print_fsboost_results(B)
The printed table looks like this:
Forward-stagewise regression results (no inference)-------------------------------------------------------coefficient std. error z p-value---------------------------------------------------const -1.24238 NA NA NALFP 2.60587 NA NA NAWHRS -0.000284255 NA NA NAWE 0.132787 NA NA NARPWG 0.494871 NA NA NAFAMINC 1.02652e-05 NA NA NAMTR -0.644518 NA NA NALearning rate = 0.0002Number of iterations = 4964Correl. w. residuals = -0.0578633S.E. of regression = 2.18792R-squared = 0.544504R-squared alt. = 0.547703
The list of the active set (variable’s with non-zero coefficients) can be retrieved from the resulting model bundle:
list X_final = B.X_final # Retrieve list of selected regressorseval varnames(X_final) # Print names of selected regressors
The estimated coefficient paths can be plotted through the function plot_coefficient_paths():
plot_coefficient_paths(B)
The resulting plot looks similar to the following one:
For more details on information available read the pdf help.
The gretl script including unit-tests can be found under “./tests/run_tests.inp”. The coverage is already quite high (probably > 75%). The script can be executed through the shell script “./run_tests.sh”.