Hyperparameter Tuning in Linear Regression.

Before that let us understand why do we tune the model.

We tune the model to maximize model performances without overfitting and reduce the variance error in our model. We have to apply the appropriate Hyperparameter technique for our model.

It refers to the amount that the predicted value would change if different training data were used.

It is the error due to the model assumptions that are made to simplify the model.

It reduces the overfitting nature of the model. Even if the model works well, this is done in order to prevent the problem from occurring in the future. This is done by introducing more errors and making the model learn more. This will help the model to learn more. And as a result, even if more data is added in the later stage, the model will be able to process those without any issues. Now the model performance will increase and will be better than the unregularized model.

Coefficient shrinks whenever we do regularization. We need to make sure that our model doesn’t get under-fitted by tuning too much in alpha as well. Alpha is a penalty factor. Error is introduced in the system by drawing a line that’s doesn’t touch the majority of the points. These regularization models will shrink the coefficient as they build models that reduce the slope which will not change much for new data. Shrinkage in coeff totally depends on the variables. If the feature is significant then the shrinkage will be less but if the feature is not significant then shrinkage will more. if the feature is highly insignificant then the coeff will become 0. The advantage of this regularizes models is that even if the assumptions are not checked, the model will do all the work.

When your model learns all complex and noise from training data and performs well in training data but while coming to validation data it does not work well then our data is overfitting.

When our data is underfitting then our model does learn the underlying trend data. It occurs when we have fewer data to build the model or when we try to build the linear model with non-linear data.

Cross-Validation is essentially a technique used to assess how well a model performs on a new independent dataset.

The simplest example of cross-validation is when you split your data into three groups: training data, validation data, and testing data, where you see the training data to build the model, the validation data to tune the hyperparameters, and the testing data to evaluate your final model.

It adds the “Squared magnitude” of coefficient as a penalty term to the loss function. It is called an L2 penalty

sse = np.sum ((y-b1x1-b2x2-…-bo) **2) + (alpha * (b1**2+b2**2+b3**2+…+bo**2))

The (least absolute shrinkage and selection operator) adds the “Absolute value of magnitude” of coefficient as a penalty term to the loss function. It is called an L1 penalty.

sse = np.sum ((y-b1x1-b2x2-…-bo) **2) + (alpha * (|b1|+|b2|+|b3|+…+|bo|))

It is the combination of both Ridge and Lasso regularization.

We can find the best alpha value and best regularization using this method. Gradient descent is an iterative optimization algorithm used in machine learning to minimize a loss function. The loss function describes how well the model will perform given the current set of parameters (weights and biases), and gradient descent is used to find the best set of parameters.

Thanks for reading :)

Related posts:

This is a complete list of all the meditations that I recommend based on guided sessions that I have enjoyed. As you try each of the guided meditations you may enjoy them get the same understandings…

Retiring can be a difficult decision to make, but it is an important one that requires careful consideration. When is the right time to retire? Is passive yielding a viable option for retirement…

About two months ago we made a roadmap section where everyone in the application can suggest improvements! We received a lot of feedback. Thank you so much for your commitment to the project! Now the…