Complexity in Algorithms

Although complexity of algorithms is a theoretical subject that is 99.9% of the time not used in real world, it is worth to refresh knowledge on it. Classifying the behaviour of an implemented algorithm can come in hand when analysing performance issues or designing more efficient algorithms.

Basic concepts

Input

The input to an algorithm, typically defined by . The size of the input is defined based on the type of data: number of items for arrays, number of bits for files, or characterisation of graphs or trees and so on.

Running time

The running time or time complexity of an algorithm estimates the time taken to run the algorithm based on the size of a particular input. It is common to define the running time as a function of the size of its input, or .

Examples of the formula defined by the running time can be or .

To better identify the implications of an algorithm, it is common to analyse how it behaves under best, average and worst case scenarios (all of them quite dependant on the algorithm):

  • Best-case running time: behaviour under optimal conditions. E.g., a linear search algorithm finds the element on the first position
  • Average running time: typical behaviour. This can be difficult to measure, but could be something like a pseudo-randomly generated array generated by a *nix distribution
  • Worst-case running time: behaviour under non-convenient conditions. E.g., insertion sort finds the input reversed sorted

For some algorithms, the worst case occurs fairly often. In other cases, it will be more interesting to focus on the average case. It is then possible that under some constraints it is preferable to pick an algorithm with better performance on average time than other, even at the cost of a worst performance on the worst-case scenario. Such would be the case of quicksort when compared to others such as a mergesort or heapsort.

Some of the most common running times, in increasing order of time, are described in the table below.

Name Running time
Logarithmic
Constant
Linear
Quadratic
Cubic
Exponential
Factorial
Rate of growth

The order/rate of growth of the running time performs a simple estimation of the algorithm’s efficiency. This is obtained by considering the leading terms of the formula from the running time and ignoring lower-order terms and constants, which are negligible in comparison.

Examples of the formula defined by the rate of growth of the running time can be or .

Space complexity

The space complexity of an algorithm estimates the usage of the memory space based on the size of a particular input.

Though it does not seem as popular as the time complexity analysis, identifying the memory space used by the computer can be useful to troubleshoot and improve performance of algorithms dealing with large sets of data to be delivered by some client.

Asymptotic notation

The asymptotic efficiency of an algorithm analyses the trend of the running time when as the size of the input increases without bond, to a limit. This concept also operates over mathematical functions and is depicted plotting different functions over a bi-dimensional graph (X=size of input, Y=running time).

These asymptotic notation is used to formalise the running time of an algorithm. Each notation provides different bounds to a given function , so analysis is performed for different scenarios and input sizes. The upper and lower bound represent the lowest (best) and highest (worst) running times, specifically. More details here and here.

  • -notation: provides upper and lower asymptotically tight bounds; so that
  • O-notation: provides a non-necessarily asymptotically tight upper bound; so that
  • -notation: provides a non-necessarily asymptotically tight lower bound; so that
  • o-notation: similar to O-notation, providing a non-asymptotically tight upper bound
  • -notation: similar to -notation, providing a non-asymptotically tight lower bound

The Big-O cheat sheet provides a useful reference to check best, average and worst cases for time complexity and worst case for space complexity, based on common algorithms.

-notation

This notation has been already used to define the rate of growth for an algorithm, based on its running time.

O-notation

Also called “Big-O”. Commonly used to define the worst-case running time. Some basics about Big-O can complement with examples.

-notation

Commonly used to define the best-case running time; formalised by its lower bound.

Complexity analysis

Identifying the running time(s) of an algorithm can be done by looking at its behaviour (number of iterations, conditions on loops, specifics of the algorithm, etc) and computing its rate of growth; or by detailedly evaluating each step. The following examples aim to perform the latter procedure.

Selection sort

The selection sort is an inefficient sorting in-place algorithm. It works by iterating each time from a position to in an array , finding the minimum element contained at and exchanging it with .

The “Introduction to Algorithms, 3rd edition” book provides exercise 2.2-2, asking for the analysis of the running time in best and worst case for such algorithm. The answer is based on the analysis provided on the same book for insertion-sort (check here for information on conditional steps).

A Python3 implementation would be as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def swap(A, x, y):
    aux = A[x]
    A[x] = A[y]
    A[y] = aux
    return A 

def insertion_sort(A):
    for i in range(len(A)):
        min = i
        for j in range(i+1, len(A)):
            if A[j] < A[min]:
                min = j
        if A[i] != A[min]:
            A = swap(A, i, min)
    return A

The pseudo-code can be used to analyse the running time in both worst and best cases:

1
2
3
4
5
6
7
8
SELECTION-SORT(A)
  for i=1 to A.length-1
    min = A[i]
    for j=i+1 to A.length-1
      if A[j] < min
          min = A[j]
    if A[i] != min
      swap(min, A[i])
Line Instruction Cost Times Explanation
2 for i=1 to A.length-1 1 iteration per item in the outer loop + last 1 to get out of it
3     min = A[i] Runs every time in the outer loop
4     for j=i+1 to A.length-1 Subset of items in the outer loop + last 1 to get out of inner loop
5         if A[j] < min Worst-case: 1 iteration per item in the inner loop
6             min = A[j] Same as L5 Same as L5
7     if A[i] != min Worst-case: 1 iteration per item in the outer loop
8         swap(min, A[i]) Same as L7 Same as L7

The worst-case scenario assumes the input array is not sorted or pseudo-random. The algorithm would perform all steps.

The best-case scenario assumes the input array is already sorted. The algorithm would still run both loops and conditional checks, yet it should not enter the latter.

Note that the constants , numeric constants and variables can be ignored to simplify the equation; as these do not lead the running time. Also, the summations have been replaced using the formula to sum elements in an arithmetic progression:

The running time is quadratic in both cases. Even if some instructions are skipped in the best-case scenario, the cost-related constants are negligible when compared to the order of the terms formalising the iterations.

Merge sort

The merge sort is an efficient recursive sorting algorithm. It works by diving the array in two halves , sorting each of them and merge the previous results into a new array .

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
MERGE(l, r)
  var list result
  while l.length > 0 and r.length > 0
      if first(l) &le; first(r)
          append first(l) to result
          left = rest(l)
      else
          append first(r) to result
          right = rest(r)
  if l.length > 0
      append rest(l) to result
  if r.length > 0
      append rest(r) to result
  return result

MERGESORT(A)
  var list l, r, result
  if A.length <= 1
      return A
  else
      m = floor(A.length / 2)
      for x=l to m-1
          add A[x] to l
      for x=m to r
          add A[x] to r
      l = MERGESORT(l)
      r = MERGESORT(r)
      if last(l) &le; first(r) 
         append r to l
         return l
      result = MERGE(l, r)
      return result