Optimization-SteepestDescent vs GradientDescent

Problem Statement :-

Different Approaches:

1.Steepest Descent:

a) Base problem

Steepest Descent Method also known as saddle method. In this we move in a direction untill the cost function stops decreasing [i.e. ∇f(dx1)=0, where dx1 is that direction. Cost function may still decrease in other direction]. Then we pick another direction let say dx2. In this each next direction is perpendicular to last direction ( dx1 ⊥ dx2 ). The saddle point is point of contact to the contour line where that direction is tangent.
#iterations: 13

b) Steepest Descent with Newton's Method

In newton's method we approximate our cost function with a similar quadratic equation which behaves similar to our original function in some small neigbourhood. It then tries to find the optiamal pt of that quadratic function. It repeats this process untill stopping criteria is met. [Note: here since our original function is quadratic function itself so the approximation is same function and thus we reach it's optimal point in one iteration.]
#iterations: 1

2.Gradient Descent:

a) Gradient Descent with learning rate `0.1`

Since leanring rate is much less than the 0.3771609692315777, it will converge nicely.
#iterations: 86

b) Gradient Descent with learning rate `0.3771609692315777 - 0.00002`

Since leanring rate is smaller but very close to 0.3771609692315777 it will converge after much oscillation, thus will take much more iterations.
#iterations: 163093

c) Gradient Descent with learning rate `0.3771609692315777 + 0.00002`

Since leanring rate is larger than 0.3771609692315777 it will keep oscillating and will never converge.

Let's put all our findings in one table