LAB 01.02 - Metrics

!wget --no-cache -O init.py -q https://raw.githubusercontent.com/rramosp/ai4eng.v1/main/content/init.py
import init, inspect; init.init(force_download=False); init.get_weblink()

from local.lib.rlxmoocapi import submit, session
session.LoginSequence(endpoint=init.endpoint, course_id=init.course_id, lab_id="L01.02", varname="student");

General remark

You do not need to use Python to solve the problems in this notebook, you can use any tool of your choice (Excel, etc.), including pen and paper. But

If you want to try out in Python

  • numpy is the Python library used for vectors

  • there are operations that take a vector and produce another vector (i.e. np.log)

  • there are operations that take a vector and procude a number (i.e. np.mean)

  • there are operations that take two vectors and produce a number (see the HINTs below)

  • etc.

For instance

import numpy as np

v1 = np.array([1,2,3,4])

# the log of each element of the vector
print ( "the log   =", np.log(v1)  )

# the mean of all elements of the vector
print ( "the mean  =",  np.mean(v1)  )

# multiply all elements of a vector with a scalar
print ("times two =", 2*v1)

you can always check the type of any variable

a = 2.0
type(v1), type(a)

Task 01. Accuracy

Compute the percentage of correct predictions accuracy (see here) for the following model output (predicted) and ground truth (actual).

Execute the following cell to generate the data from which you must compute the metric. You may compute the metric implementing python code, or manually, or copy/pasting the actual and predicted data in Excel, etc.

CHALLENGE: use Python with sklearn.metrics.accuracy_score

Observe that every time you execute the following cell, a different set of values is generated. You will have to compute the metric for the values that you see. If you run the cell again you will have to compute your metric value again.

import numpy as np
t1_actual    = np.random.randint(2, size=20)
t1_predicted = np.abs(t1_actual*(np.random.random(size=20)>(np.random.random()*.9+.05)).astype(int))
print ("actual   ", ", ".join([str(i) for i in t1_actual]))
print ("predicted", ", ".join([str(i) for i in t1_predicted]))

Assign the value of your computation to the accuracy variable, with three decimal places

accuracy =

submit your answer

student.submit_task(globals(), task_id="task_01");

Task 2: Sensitivity

Compute the sensitivity metric [aka the True Positive Rate or Recall see Sensitivity on Wikipedia] for the following model output (predicted) and ground truth (actual)

Execute the following cell to generate the data from which you must compute the metric. You may compute the metric implementing python code, or manually, or copy/pasting the actual and predicted data in Excel, etc.

Challenge: Use Python sklearn.metrics.recall_score

Observe that every time you execute the following cell, a different set of values is generated. You will have to compute the metric for the values that you see. If you run the cell again you will have to compute your metric value again.

import numpy as np
t2_predicted = np.random.randint(2, size=20)
t2_actual = np.random.randint(2, size=20)
t2_predicted[np.argwhere(t2_actual==1)[0][0]]=0
print ("actual   ", ", ".join([str(i) for i in t2_actual]))
print ("predicted", ", ".join([str(i) for i in t2_predicted]))

Assign the value of your computation to the tpr variable with three decimal places

tpr =

submit your answer

student.submit_task(globals(), task_id="task_02");

Task 3: Evaluation in New York City Taxi Trip Duration Kaggle Competition

Understand the data and the evaluation metric (Root Mean Squared Logarithmic Error, RMSLE) of the following Kaggle competition

Observe that this competition is a regression task as we are measuring the difference in prediction with respect to the actual.

For instance, the following model predictions and ground truth:

actual    [66 37 22]
predicted [79 51 67]

produce a RMSLE of 0.66 aprox.

Execute the following cell to generate the data from which you must compute the metric. You may compute the metric implementing python code, or manually, or copy/pasting the actual and predicted data in Excel, etc.

Challenge: For python use numpy function np.log or np.log1p

Observe that every time you execute the following cell, a different set of values is generated. You will have to compute the metric for the values that you see. If you run the cell again you will have to compute your metric value again.

t3_actual    = np.random.randint(80,size=15)+20
t3_predicted = np.random.randint(80,size=15)+20
print ("actual   ", t3_actual)
print ("predicted", t3_predicted)

Assign the value of your computation to the rmsle variable with three decimal places

rmsle =

submit your answer

student.submit_task(globals(), task_id="task_03");

Task 4: Evaluation in Shelter Animal Outcomes Kaggle Competition

Understand the data and the evaluation metric (Multiclass Logaritmic Loss, logloss) of the following Kaggle competition

Observe that this competition is a classification task with 5 classes and, for each item, the model produces a probability for each class. Classes are numbered from 0 to 4.

For instance, the following represents the model output for three items

[[0.17 0.27 0.03 0.31 0.21]
 [0.09 0.44 0.02 0.15 0.3 ]
 [0.26 0.18 0.25 0.2  0.11]]

Where the classes with gretest probability assigned by the model are

  • class 3 for the first item (with 0.31 probability)

  • class 1 for the second item (with 0.44 probability)

  • class 0 for the third item (with 0.26 probability)

The class labels are expressed as a similar matrix, but with 0/1 For instance, the ground truth for the corresponding three items above, could be:

[[0 0 0 1 0]
 [0 0 1 0 0]
 [1 0 0 0 0]]

and will produce a logloss of approx 2.14

Execute the following cell to generate the data from which you must compute the metric. You may compute the metric implementing python code, or manually, or copy/pasting the actual and predicted data in Excel, etc.

import numpy as np

t4_predicted = np.random.random(size=(7,5)).T+0.5
t4_predicted = np.round((t4_predicted/np.sum(t4_predicted,axis=0)),2).T

t4_actual = np.eye(5)[np.random.randint(5,size=len(t4_predicted))].astype(int)

print ("actual")
print (t4_actual)
print ("\npredicted")
print (t4_predicted)

Assign the value of your computation to the logloss variable with three decimal places

logloss =

submit your answer

student.submit_task(globals(), task_id="task_04");