machine_learning _ lecture 4_2

used in many algorithms where it is required to find

the extremum of a function

It is a first-order optimization algorithm. This means it only

takes into account the first derivative when performing the updates on the parameters. On each iteration, we update the parameters in the opposite direction of the gradient of the objective function J(w) w.r.t the parameters where the gradient gives the direction of the steepest ascent. The size of the step we take on each iteration to reach the local minimum is determined by the learning rate α. Therefore, we follow the direction of the slope downhill until we reach a local minimum.

SGDRegressor
# Create a linear regression object

regr = linear_model.SGDRegressor(max_iter=10000, tol

=0.001)

features or a low number of observations, a linear model

tends to overfit and variable selection is tricky.

and Ridge can improve the prediction accuracy as they reduce

the estimation variance while providing an interpretable final model.
In this tutorial, we will examine Ridge and Lasso regressions, compare it to the classical linear regression and apply it to a dataset in Python. Ridge and Lasso build on the linear model, but their fundamental peculiarity is regularization. The goal of these methods is to improve the loss function so that it depends not only on the sum of the squared differences but also on the regression coefficients.
 One of the main problems in the construction of such models is the correct selection of the regularization parameter. Сomparing to linear regression, Ridge and Lasso models are more resistant to outliers and the spread of data. Overall, their main purpose is to prevent overfitting.
The main difference between Ridge regression and Lasso is how they assign a penalty term to the coefficients.

process of introducing additional information in order to prevent overfitting),

i.e. adds penalty equivalent to absolute value of the magnitude of coefficients.

In particular, the minimization objective does not only include the residual sum of squares (RSS) - like in the OLS regression setting - but also the sum of the absolute value of coefficients.

upper bound on the sum of the absolute values of

the model parameters. The lasso estimate is defined by the solution to the L1 optimization problem:

of the penalty assumes great importance. Indeed, when α is

sufficiently large, coefficients are forced to be exactly equal to zero. This way, dimensionality can be reduced. T
The larger the parameter , the more the number of coefficients are shrunk to zero. On the other hand, if = 0, we have just an OLS (Ordinary Least Squares) regression.
Alpha simply defines regularization strength and is usually chosen by cross-validation.

α

α

do not explain a sufficient amount of variance in the

data. It also has a tendency to set the coefficients of the bad predictors mentioned above 0.
This makes Lasso useful in feature selection.
Lasso however struggles with some types of data. If the number of predictors (p) is greater than the number of observations (n), Lasso will pick at most n predictors as non-zero, even if all predictors are relevant. Lasso will also struggle with colinear features (they’re related/correlated strongly), in which it will select only one predictor to represent the full suite of correlated predictors. This selection will also be done in a random way, which is bad for reproducibility and interpretation.

cost function, but instead sums the squares of coefficient values

(the L-2 norm) and multiplies it by some constant lambda.












Compared to Lasso, this regularization term will decrease the values of coefficients, but is unable to force a coefficient to exactly 0. This makes ridge regression’s use limited with regards to feature selection. However, when p > n, it is capable of selecting more than n relevant predictors if necessary unlike Lasso. It will also select groups of colinear features, which its inventors dubbed the ‘grouping effect.’
Much like with Lasso, we can vary lambda to get models with different levels of regularization with lambda=0 corresponding to OLS and lambda approaching infinity corresponding to a constant function.

terms.
This gives us the benefits of both Lasso and

Ridge regression.
It has been found to have predictive power better than Lasso, while still performing feature selection.
We therefore get the best of both worlds, performing feature selection of Lasso with the feature-group selection of Ridge.

penalty, which when used alone is a ridge regression (L2). The estimates

from the elastic net method are defined by