Analysis of Algorithms: Big-O Notation

Joseph C. Osborn

CSCI 051a PO

November 26, 2019

Algorithms

  • Today’s topics: Algorithms and worst case (“big O”) analysis
  • What is an algorithm?
  • A method for solving a problem
  • Which is implementable on a computer

Examples

  • Sorting a list of numbers
  • Finding a route from one place to another
  • Solving Sudoku
  • Encrypting a message
  • Wiring a microchip
  • Compressing a video file
  • Playing back a compressed video file
  • You’ve been implementing algorithms all semester!
  • …

Algorithms vs Code

  • Functions like next_states implement algorithms
    • In this case, finding successors of a Sudoku board
  • Depth-first search is an algorithm
    • def dfs(...) is some code implementing it
  • For the purposes of 62, 101, 140, … we use “algorithm” to mean:
    • Programming language independent…
    • Specifications of a problem solving method
  • Algorithms are solution methods for problems

Developing algorithms

  • Understand a problem really well
  • Think about the steps involved in solving it
  • Get something that works
  • Analyze it
  • Improve it

How do we evaluate an algorithm?

  • Imagine we have five algorithms for solving a problem
  • How do we choose between them?
  • Ideas?
  • Correct results
  • … (within some approximation bounds)
  • Speed
  • Memory use
  • Easy to read/understand/debug

“Speed”, “Memory use”, …

  • Some of these characteristics have an important problem…
  • They’re realy machine dependent!
  • Not just faster/slower computers, but more/less RAM, faster/slower interconnects between components, 32/64-bit, etc…
  • Different implementations might have different characteristics too! E.g. Python vs C
  • We would like a way to analyze algorithms in a machine-independent way
    • And hopefully a programming language-independent way

Analysis of Algorithms

  • So we use this tool called worst case analysis
    • Or “Big O” analysis
  • Maybe you’ve seen Big O before

Big O

2019-11-26_09-17-03_.jpeg

Big O

2019-11-26_09-19-50_Big_o.jpg

Big O


Big O

\[O(n)\]


Example

  • Let’s implement this algorithm now:

“To find an element in some list, compare the element against each element of the list. If it compares equal to any element return true, otherwise return false.” 

def member(element, some_list):
  pass

Example

  • We want to evaluate this algorithm’s performance
  • In a way that doesn’t depend on:
    • Length of the list
    • Speed of my computer
    • Amount of RAM in my computer
    • Types of items being compared
    • …

Example

  • What’s the worst situation for this algorithm, in terms of performance?
  • The item isn’t in the list!
  • So let’s assume it’s not in the list
  • How many trips through the loop do I make if the list has one item?
  • Two items? Four items? Eight?
  • What’s the pattern here?
  • What’s the y-axis?
  • “Number of times through the loop”; or even better, “Number of comparisons between items”
def member(element, some_list):
  for elt in some_list:
    if elt == element:
      return True
  return False

Asymptotic Complexity

  • As our member function is called with longer and longer lists…
  • the number of comparisons we make grows linearly!
  • We write functions like this as \(O(n)\)
  • “We can upper-bound the worst case performance with some line”
  • If I had a graph with an upward curve, could I find such a line that bounds the worst case?
  • Out of curiosity: What’s the best case for member?

Exercise

  • Prove we can’t beat the simple implementation of member for worst-case performance
  • Or find a counter-example (it might be a more specialized algorithm!)
  • What if I only care about numbers 1-100 and only care whether the item is present, not where?
  • What if the data are numbers sorted in ascending order?

Exercise: Map and Filter

  • Big-O of map and filter?
    • How many calls of the given function?
  • Whoa, doesn’t depend on anyfn at all!!!
  • Hopefully this motivates the use of map and filter a little bit more
  • We can know properties about them and automatically know those properties for any application of them.
  • Cool!
def map(a_fn, a_list):
  ret = []
  for elt in a_list:
    ret.append(a_fn(elt))
  return ret

def filter(a_fn, a_list);
  ret = []
  for elt in a_list:
    if a_fn(elt):
      ret.append(elt)
  return ret

Exam Notes

  • Since not much partial credit available, boosted total to 30 points and every exam by 3 points
  • Mean: 81.99%
  • Median: 82.5%
  • Std. Dev: 3.53

Exam Notes

  • Don’t sweat it too much
    • Homeworks have been going great
  • Areas to review:
    • Naive Bayes
    • Aliasing
    • Search algorithms and node visit order
    • Recursion

Sorting


Big O