Assignment 11: Inference and Applications

Learning Goals

  • Learn how to export a trained model.
  • Learn how to import a trained model.
  • Learn how to use a model for inference.
  • Understand the general process for building an AI application.

Grading Walk-Throughs

This assignment will be graded as “Nailed It” / “Not Yet” by a TA. To complete (“Nailed It”) the assignment, you must

  1. Complete the assignment and submit your work to gradescope.
  2. Meet with a TA during their mentor session hours.
  3. Complete the walk-through with all group members. I prefer all partners to be present during the walk-through, but you can each meet with the TA separately if needed.
  4. Walk the TA through your answers. Do not expect to make corrections during the walk-through.
  5. The TA will then either
    • mark your assignment as 100% on gradescope, or
    • inform you that you have some corrections to make.
  6. If corrections are needed, then you will need to complete them and conduct a new walk-through with a TA.

If you have concerns about the grading walk-through, you can meet with me after you have first met with a TA.

Overview

In this assignment you will create a gradio application for an already-trained model from either PyTorch (torchvision) or Hugging Face.

Image Collection for Lecture Demo

The deadline for uploading images is Monday, April 14 at 11:59 PM.

Prior to the assignment, I will demonstrate the process using the image classification task. To do so, we need to create out own dataset. We will be training a model to classify pictures as buildings from Pomona, Scripps, or CMC.

Everyone must:

  1. Take 16 pictures of buildings on Pomona’s campus
  2. Take 16 pictures of buildings on Scripps’s campus
  3. Take 16 pictures of buildings on Claremont McKenna’s campus
  4. Put them on Box in the folder (you’ll receive an email to your college email address)
    • Pomona pictures go in the “Pomona” sub-folder
    • Scripps pictures go in the “Scripps” sub-folder
    • CMC pictures go in the “CMC” sub-folder

(You can add more than 48 total images if you’d like!)

Tips for taking pictures:

  • Take pictures in different locations
  • Use different angles
  • Don’t always have the item in the center
  • Use different lighting
  • Don’t always take the image with the same background
  • Take pictures at different meal times

File naming conventions:

For the sake of anonymity, you can use whatever random identifier you’d like in your filenames. You do not have to, but I also recommend removing the meta-data from your image files (if you do not do so, I will before class–and I’ll show you how using jpegtran -copy none).

Name your files using this scheme: <identifier>-<class>-<0#>.<ext>

Where

  • <identifier> is any random identifier that you choose that helps me know that they are your images (so that you get credit)

  • <class> is one of “Pomona” or “Scripps” or “CMC”

  • <0#> denotes a zero padded number (“01”, “02”, … “16”)

  • <ext> is your file extension (probably “.jpg” or the like)

For example, this is what I’ll name my files:

## In the Pomona folder
profclark-Pomona-01.jpg
profclark-Pomona-02.jpg
profclark-Pomona-03.jpg
profclark-Pomona-04.jpg
profclark-Pomona-05.jpg
profclark-Pomona-06.jpg
profclark-Pomona-07.jpg
profclark-Pomona-08.jpg
profclark-Pomona-09.jpg
profclark-Pomona-10.jpg
profclark-Pomona-11.jpg
profclark-Pomona-12.jpg
profclark-Pomona-13.jpg
profclark-Pomona-14.jpg
profclark-Pomona-15.jpg
profclark-Pomona-16.jpg

## In the Scripps folder
profclark-Scripps-01.jpg
profclark-Scripps-02.jpg
profclark-Scripps-03.jpg
profclark-Scripps-04.jpg
profclark-Scripps-05.jpg
profclark-Scripps-06.jpg
profclark-Scripps-07.jpg
profclark-Scripps-08.jpg
profclark-Scripps-09.jpg
profclark-Scripps-10.jpg
profclark-Scripps-11.jpg
profclark-Scripps-12.jpg
profclark-Scripps-13.jpg
profclark-Scripps-14.jpg
profclark-Scripps-15.jpg
profclark-Scripps-16.jpg

## In the CMC folder
profclark-CMC-01.jpg
profclark-CMC-02.jpg
profclark-CMC-03.jpg
profclark-CMC-04.jpg
profclark-CMC-05.jpg
profclark-CMC-06.jpg
profclark-CMC-07.jpg
profclark-CMC-08.jpg
profclark-CMC-09.jpg
profclark-CMC-10.jpg
profclark-CMC-11.jpg
profclark-CMC-12.jpg
profclark-CMC-13.jpg
profclark-CMC-14.jpg
profclark-CMC-15.jpg
profclark-CMC-16.jpg

If you make a mistake, you should be able to upload a file with the same name to overwrite the original upload.

You will put your images in the folders at the box link you receive by email.

Assignment Tasks

For this assignment you will

  1. Select a pre-trained model from one of these sources:
    • torchvision: these are just for vision problems, including classification, multi-class classification, segmentation, object detection, and video classification
    • Hugging Face
  2. Build an application that runs on the server. You do not need to (and probably should not) deploy your application to any cloud service.

It is possible to complete this assignment with pretty minimal effort, but I highly encourage you to spend some time and create an app that you’d find interesting. For example, if object detection sounds interesting, then take a look at this documentation and grab a pretrained Faster R-CNN model (I also have a demo of using this model listed below).

Here are some examples and documentation that you will find useful:

  • Hugging Face demos: I show some basic usage of Hugging Face models for sentiment analysis, text generation, translation, and a few image tasks
  • torchvision demos: Here is some code I demonstrated last semester that shows how to use pretrained torchvision models for object detection and segmentation
  • Pipelines for inference: This documentation steps you through how you can use existing Hugging Face models for inference, they have examples for
  • List of pipelines: This lists all existing pipelines (some for audio, text, tables, and images)
  • Getting started with gradio: Once you’ve selected a model, you’ll use gradio to build an app
  • Hugging face tutorial: Here is a more detailed tutorial for Hugging Face (maybe something you could look at for your projects)

Hugging Face with gradio

gradio makes it (almost too) easy to use Hugging Face models. Here is a two-line python script for image classification. I’d like you to put in more effort than just taking this existing example.

Using the Jupyter Server

To access your gradio app in your browser you will need to:

  1. Choose a port number and announce it on Slack so that nobody else uses the same port (choose something in the range 8000-9999)

  2. Start the app on the server. Something like this:

    GRADIO_SERVER_PORT=PORT python app.py

  3. Open a terminal and forward a local port to the server

    ssh -L PORT:localhost:PORT USERNAME@jupyter.pomona.edu

Where PORT is the port number you chose in step 1, and USERNAME is your username on the server.

Submitting Your Assignment

You will submit your responses on gradescope. Only one partner should submit. The submitter will add the other partner through the gradescope interface.

Additional details for using gradescope can be found here: