Evaluating Regression Lines Lab
Introduction
In the previous lesson, we learned to evaluate how well a regression line estimated our actual data. In this lab, we will turn these formulas into code. In doing so, we'll build lots of useful functions for both calculating and displaying our errors for a given regression line and dataset. We'll have access to our graph library and our linear equations library as we work through these tasks.
Determining Quality
In the file, movie_data.py
you will find movie data written as a python list of dictionaries, with each dictionary representing a movie. The movies are derived from the first 30 entries from the 538 dataset provided here.
from movie_data import movies
len(movies)
30
movies[0]
{'budget': 13000000, 'domgross': 25682380.0, 'title': '21 & Over'}
Note that like in previous lessons, we are still referencing a budget as our x_values
or explanatory value and our revenue as our ouput. Here revenue is represented as the key domgross
. First, convert the explanatory budget values to x_values
, and convert the domgross values to y_values
.
def build_x_values(movies):
pass
x_values = build_x_values(movies) or ['implement']
x_values[0] # 13000000
'implement'
def build_y_values(movies):
pass
y_values = build_y_values(movies) or ['implement']
y_values[0] # 25682380.0
'implement'
Assign a variable called titles
equal to the titles of the movies.
def build_titles(movies):
pass
titles = build_titles(movies) or ['implement']
titles[0] # '21 & Over'
'implement'
For the dataset, let's use the following regression formula: regression_formula
that calculates our
def regression_formula(x):
pass
regression_formula(100000) # 270000.0
regression_formula(250000) # 525000.0
525000.0
Let's plot the data as well as the regression line to get a sense of what we are looking at. We can make use of the graph.py library, that we wrote earlier.
from plotly.offline import iplot, init_notebook_mode
init_notebook_mode(connected=True)
from graph import trace_values, m_b_trace, plot
# movies_trace = trace_values(x_values, y_values, text=titles, name='movie data')
# regression_trace = m_b_trace(1.7, 100000, x_values, name='estimated revenue')
# plot([movies_trace, regression_trace])
Calculating errors of a regression Line
Now that we have our regression formula, we can move towards calculating the error. Let's start by writing a function named y_actual
, that given a value, x_values
and a list of y_values
to represent our dataset, returns the value of
def y_actual(x, x_values, y_values):
pass
y_actual(13000000, x_values, y_values) # 25682380.0
25682380.0
Now write a function called error
, that given a list of x_values
, and a list of y_values
, the values m
and b
of a regression line, and a value of x
, returns the error at that x value. Remember
def error(x_values, y_values, m, b, x):
pass
# error(x_values, y_values, 1.7, 100000, 13000000) # 3482380.0
Now that we have a formula to calculate our errors, write a function that returns a trace of an error at a given point. The function is called error_line
and it takes our dataset of x_values
as the first argument, and y_values
as the second argument. It also takes in values of
The return value is a dictionary that represents a trace. The x values of the trace are two x values, and the y values of the trace are the actual value and the expected value. The mode of the trace should equal line
.
def error_line_trace(x_values, y_values, m, b, x):
pass
error_at_120m = error_line_trace(x_values, y_values, 1.7, 10000, 120000000)
# {'marker': {'color': 'red'},
# 'mode': 'line',
# 'name': 'error at 120000000',
# 'x': [120000000, 120000000],
# 'y': [93050117.0, 204010000.0]}
error_at_120m
{'marker': {'color': 'red'},
'mode': 'line',
'name': 'error at 120000000',
'x': [120000000, 120000000],
'y': [93050117.0, 204010000.0]}
We just ran the our function to draw a trace of the error for the movie Elysium. Let's see how it looks.
movies[19]
{'budget': 120000000, 'domgross': 93050117.0, 'title': 'Elysium'}
We can use a function from our linear_equations library that we previously wrote.
from linear_equations import expected_value_for_line
# y_actual(13000000, x_values, y_values) # 25682380.0
# expected_value_for_line(1.7, 10000, 13000000) # 22110000.0
from plotly.offline import iplot, init_notebook_mode
init_notebook_mode(connected=True)
from graph import trace_values, m_b_trace, plot
# movies_trace = trace_values(x_values, y_values, text=titles, name='movie data')
# regression_trace = m_b_trace(1.7, 100000, x_values, name='estimated revenue')
# plot([movies_trace, regression_trace, error_at_120m])
From there, we can write a function called error_lines
, that takes in a list of x_values
as an argument, y_values
as an argument, and returns a list of traces for every x_value provided.
def error_line_traces(x_values, y_values, m, b):
pass
# errors_for_regression = error_line_traces(x_values[0:5], y_values[0:5], 1.7, 100000)
# len(errors_for_regression) # 5
# errors_for_regression[-1]
# {'marker': {'color': 'red'},
# 'mode': 'line',
# 'name': 'error at 40000000',
# 'x': [40000000, 40000000],
# 'y': [95020213.0, 68100000.0]}
from plotly.offline import iplot, init_notebook_mode
init_notebook_mode(connected=True)
from graph import trace_values, m_b_trace, plot
# movies_trace = trace_values(x_values, y_values, text=titles, name='movie data')
# regression_trace = m_b_trace(1.7, 100000, x_values, name='estimated revenue')
# plot([movies_trace, regression_trace, *errors_for_regression])
Calculating RSS
Now write a function called squared_error
, that given a value of x, returns the squared error at that x value.
def squared_error(x_values, y_values, m, b, x):
pass
# squared_error(x_values, y_values, 1.7, 100000, x_values[0]) # 12126970464400.0
12126970464400.0
Next, write a function called residual_sum_squares
that provided a list of points returns the total squared error for the movies in our dataset.
def squared_errors(x_values, y_values, m, b):
pass
# squared_errors(x_values, y_values, 1.7, 100000)
def residual_sum_squares(x_values, y_values, m, b):
pass
residual_sum_squares(x_values, y_values, 1.7, 100000)
Now we'll provide a couple functions for you. a function called trace_rss
, that build a bar chart displaying the value of the RSS. The return value is a dictionary with keys of x
and y
, both which point to lists. The
import plotly.graph_objs as go
def trace_rss(x_values, y_values, m, b):
pass
trace_rss(x_values, y_values, 1.7, 100000)
# {'type': 'bar', 'x': ['RSS'], 'y': [3.0025171100327725e+17]}
Once this is built, we can turn build a subplot showing the regression line, as well as the related RSS for the regression line.
import plotly
from plotly import tools
import plotly.graph_objs as go
def plot_regression_and_rss(scatter_trace, regression_trace, rss_calc_trace):
fig = tools.make_subplots(rows=1, cols=2)
fig.append_trace(scatter_trace, 1, 1)
fig.append_trace(regression_trace, 1, 1)
fig.append_trace(rss_calc_trace, 1, 2)
plotly.offline.iplot(fig)
# m = 1.7
# b = 100000
# scatter_trace = trace_values(x_values, y_values, text=titles, name='movie data')
# regression_trace = m_b_trace(m, b, x_values, name='estimated revenue')
# rss_calc_trace = trace_rss(x_values, y_values, m, b)
# plot_regression_and_rss(scatter_trace, regression_trace, rss_calc_trace)