Algorithm Analysis

Measuring Complexity

Nathan Tenney

WSU Tri-Cities

Algorithm Analysis

What is an Algorithm?
Are algorithms useful?
What we will discuss
- How to estimate time required for a program
- How to reduce running time of a program
- Recursion done wrong, and it's effects

Algorithms

An algorithm is a set of instructions that leads to the solution to a particular problem
Goals in writing computer programs
- Easy to understand, code, and debug
- Makes efficient use of computing resources
Often the choice in algorithm means that one of these is sacrificed to accomplish the other

Analysis steps

Analyze the problem
Enumerate the resource constraints identified by the analysis
Identify the data structure that best meets the requirements identified by the resource constraints

Data Structures

Given an infinite amount of memory and time, any data structure is sufficient for a given solution to a problem
A program is considered to be efficient if it solves the problem within a set of resource constraints
- What are some resource constraints?
- What is commonly considered to be the cost of running a program?
To quantify cost, we need some way to measure it
- Often this is done as a function of the amount of data being processed
- This gives us an understanding of how the algorithm performs as larger amounts of data are fed through it
Which algorithm or data structure is "best"

Review

Data Type vs Abstract Data Type
Classes and terminology
- Encapsulation
- Has-a (composition), is-a (inheritance)
- Class
- Object

Properties of Algorithms

Must be correct
Steps must be well understood
No ambiguity at any point in the process
Fixed number of steps
Must end (no infinite loops)

Math Review: Sets

A set is a unique collection of items, either a primitive, or another set
- Elements not ordered
Set Definitions $P={1, 2, 3}, Q={5,10}$
X is a member of set P ($x \in P$)
X is not a member of set P ($x \notin P$)

Math Review: Sets

P is a subset of Q ($P \subseteq Q $)
Q is a superset of P ($Q \supseteq P $)
Union ($P \cup Q$)
Intersection ($P \cap Q$)

Math Review: Sequences

A sequence is an ordered group of elements that may contain duplicates
Shown as $\langle1,2,3,4,2,4\rangle$
Elements are numbered, starting at 0.
May also be known as a tuple

Math Review: Relations

A relation over set S consists of a set of ordered pairs from the the set S.
Given a set $S={a, b, c}$, a relation R over S might consist of ${\langle a, c\rangle, \langle b, c\rangle,\langle c, b\rangle}$
If a tuple $\langle x,y \rangle$ exists in relation R, it may be written xRy

Math Review: Relations

A relation is reflexive if aRa for all $a \in S$
A relation is symmetric if when aRb, then bRa for all $a,b \in S$
A relation is antisymmetric if when aRb and bRa, then $a = b$ for all $a,b \in S$
A relation is transitive if when aRb and bRc, then aRc for all $a,b,c \in S$

Math Review: Logarithms

Unless otherwise specified, in this class, logarithms are base 2.
For any positive values of m, n, and r, and any positive integers a and b
- $\log_{2}(nm) = \log_{2} n + \log_{2} m$
- $\log_{2}(n/m) = \log_{2} n - \log_{2} m$
- $\log_{2}(n^r) = r\log_{2} n $
- $\log_a n = \log_b n /\log_b a $ (change of base formula)

Definitions

$T\left(N\right) = O\left(f\left(N\right)\right)$ iff positive constants c and $n_{0}$ exist S.T. $T\left(N\right) \le cf\left(N\right)$ when $N \ge n_{0}$.
- This is known as "Big O"
$T\left(N\right) = \Omega\left(g\left(N\right)\right)$ if positive constants c and $n_{0}$ exist S.T. $T\left(N\right) \ge cg\left(N\right)$ when $N \ge n_{0}$.
- This is known as "Big Omega"

Definitions

$T\left(N\right) = \Theta\left(h\left(N\right)\right)$ IFF $T\left(N\right) = O\left(h\left(N\right)\right)$ and $\Omega\left(h\left(N\right)\right)$.
- Big Theta
$T\left(N\right) = o\left(p\left(N\right)\right)$ if for all constants c T.E. an $n_{0}$ S.T. $T\left(N\right) \lt cf\left(N\right)$ when $N \gt n_{0}$.
- Little o

Things to note

Looking at rates of growth, not absolute values
Not appropriate to say that $T\left(N\right) \le O\left(N\right)$
Makes no sense to say that $T\left(N\right) \ge O\left(N\right)$

Proof

Disregard constants and terms that grow at slower rates

Claim: if $T\left(n\right) = a_{k}N^{k} + … + a_{1}N + a_{0}$ then $T\left(N\right) = O\left(N^{k}\right)$

Proof: choose $n_0 = 1$ and $ c = |a_{k}| + |a_{k-1}| + … + |a_{1}| + |a_{0}| $

Need to show that $\forall N \ge 1$, $T\left(N\right) \le c \cdot N^{k}$

Properties

If $T_{1}\left(N\right) = O\left(F\left(N\right)\right)$ and $T_{2}\left(N\right) = O\left(g\left(N\right)\right)$ then
1. $T_{1}\left(N\right) + T_{2}\left(N\right) = O\left(f\left(N\right) + g\left(N\right)\right)$
2. $T_{1}\left(N\right) * T_{2}\left(N\right) = O\left(f\left(N\right) * g\left(N\right)\right)$
If $T\left(N\right)$ is a polynomial of degree k, then $T\left(N\right) = \Theta\left(N^{k}\right)$
$\log^{k} N = O\left(N\right)$ for any constant k (logarithms grow slowly)

Typical Growth Rates

Function	Name
c	Constant
$\log N$	Logarithmic
$\log^2 N$	Log-squared
N	Linear
$N \log N$
$N^{2}$	Quadratic
$N^{3}$	Cubic
$2^N$	Exponential

The Model

Our "computer" has a simple instruction set (add, sub, mul, div, comparison, assignment)
It takes 1 unit of time to perform any simple operation (including disk access)
Assume infinite memory
Focus on the "running time" of the algorithm
Generally looking at the worst case performance

Analysis of a Program

What counts as an instruction
- Assignment
- Mathematical operations
- Comparisons
- increment/decrement
Do not count declaration
Different books count slightly different instruction sets

Rules

For Loops
- The running time of the items in the loop times the number of iterations
Nested loops
- Analyze inside out
Consecutive statements
- Sum of the consecutive statements
if/else
- Running time of the test plus the maximum of the two statements.

Simple Example

int sum( int n )
{
    int partialSum;
    
    partialSum = 0;
    for( int i = 1; i <= n; i++ )
        partialSum += i * i * i;
        
    return partialSum;
}

What is the running time of this function?

Maximum Subsequence Sum Problem

Given a sequence of numbers $A_{1},A_{2},…,A_{N}$ find the maximum value of $\sum_{k=i}^j A_{k}$
Example: given input of -2, 11, -4, 13, -5, and -2 the answer is 20 ($A_{2}$ through $A_{4}$)
We will examine 4 algorithms for performing this sum.

Maximum Subsequence Sum

Input Size	$$ O(N^3) $$	$$ O(N^2) $$	$$ O(N log N) $$	$$O(N)$$
N=10	0.000009	0.000004	0.000006	0.000003
N=100	0.002580	0.000109	0.000045	0.000006
N=1000	2.281013	0.010203	0.000485	0.000031
N=10000	NA	1.2329	0.005712	0.000317
N=100000	NA	135	0.064618	0.003206

Which is "the best?"

Cubic Solution

/	*
 * Cubic maximum contiguous subsequence sum algorithm.
 */
int maxSubSum1( const vector<int> & a )
{
    int maxSum = 0;
    for( int i = 0; i < a.size( ); i++ )
        for( int j = i; j < a.size( ); j++ )
        {
            int thisSum = 0;
            for( int k = i; k <= j; k++ )
                thisSum += a[ k ];
            if( thisSum > maxSum )
                maxSum   = thisSum;
        }
    return maxSum;
}

Mathematical Evaluation

We can see that this is an $O\left(N^3\right)$ algorithm
A more rigorous evaluation would reveal that this is a $\Theta\left(N^3\right)$ algorithm
To do this evaluation, we need a couple of useful formulas.

Useful Formulas

$\begin{eqnarray}\displaystyle\sum_{i=M}^N 1 &=& N - M + 1 \\\\ \displaystyle\sum_{i=M}^N i &=& \left(N+M\right)\cdot\frac{\left(N-M+1\right)}{2} \\\\ \displaystyle\sum_{i=1}^N i^2 &=& \frac{N\left(N+1\right)\left(2N+1\right)}{6}\end{eqnarray}$

Mathematical Evaluation

We can see that essentially, this algorithm breaks down like this: $\displaystyle\sum_{i=0}^{N-1} \sum_{j=i}^{N-1} \sum_{k=i}^j 1$
Looking at the last sum, we have $\displaystyle\sum_{k=i}^j 1 = j - i + 1$
Next we evaluate the middle sum $\displaystyle\sum_{j=i}^{N-1} \left(j-i+1\right) = \frac{\left(N-i+1\right)\left(N-i\right)}{2}$
we got here by using the Gaussaian method for summation of a series.

Mathematical Evaluation

\[\begin{eqnarray}\displaystyle\sum_{j=i}^{N-1} j - i + 1 &\rightarrow& \frac{\left(N-M+1\right)\left(N+M\right)}{2} \\ \left(n+m\right) &=& \left(N-1-i+1\right) + \left(i - i + 1\right) = \left(N-i+1\right) \\ \left(n-m+1\right) &=& \left(N - 1 - i + 1\right) - \left(i - i + 1\right) + 1 \\ &=& N - i - 1 + 1 = N - i \\ \displaystyle\sum_{j=i}^{N-1} j - i + 1 &=& \frac{\left(N-i+1\right)\left(N-i\right)}{2}\end{eqnarray}\]

Mathematical Evaluation

Next we have $\displaystyle\sum_{i=0}^{N-1} \frac{\left(N-i+1\right)\left(N-i\right)}{2} = \sum_{i=1}^N \frac{\left(N-i+1\right)\left(N-i+2\right)}{2}$
Break this down by hand…

Mathematical Evaluation

\[\displaystyle\sum_{i=0}^{N-1} \frac{\left(N-i+1\right)\left(N-i\right)}{2}\] \[\begin{eqnarray}&=&\sum_{i=1}^N \frac{\left(N-i+1\right)\left(N-i+2\right)}{2}\\ &=&\sum_{i=1}^N \frac{N^2-Ni+2N-Ni+i^2-2i+n-i+2}{2} \\ &=&\sum_{i=1}^N \frac{i^2-2Ni-3i+N^2+3N+2}{2}\end{eqnarray}\]

Mathematical Evaluation

\[\begin{eqnarray}&=&\sum_{i=1}^N \frac{i^2-\left(2Ni+3i\right)+\left(N^2+3N+2\right)}{2}\\ &=&\frac{1}{2}\sum_{i=1}^N i^2 - \left( N + \frac{3}{2} \right) \sum_{i=1}^N i + \frac{1}{2} \left(N^2+3N+2\right) \sum_{i=1}^N 1 \\ &=&\left(\frac{1}{2}\frac{N\left(N+1\right)\left(2N+1\right)}{6}\right) - \left(N+\frac{3}{2}\right)\frac{N\left(N+1\right)}{2} + \frac{N^2+3N+2}{2}N \end{eqnarray}\]

Mathematical Evaluation

\[\begin{eqnarray}&=&\left(\frac{1}{2}\frac{N\left(N+1\right)\left(2N+1\right)}{6}\right) - \left(N+\frac{3}{2}\right)\frac{N\left(N+1\right)}{2} + \frac{N^2+3N+2}{2}N \\ &=&\frac{N^3 + 3N^2 + 2N}{6} \end{eqnarray} \]

Finally, we end up with $\frac{N^3 + 3N^2 + 2N}{6}$
So, we can see that this algorithm is bounded on either side by $N^3$

Quadratic Solution

int maxSubSum2( const vector<int> & a )
{
    int maxSum = 0;
    for( int i = 0; i < a.size( ); i++ )
    {
        int thisSum = 0;
        for( int j = i; j < a.size( ); j++ )
        {
            thisSum += a[ j ];
            if( thisSum > maxSum )
                maxSum = thisSum;
        }
    }
    return maxSum;
}

Quadratic Solution

This solution is found by realizing that we have some mathematically extraneous calculations.
This may not be possible with many algorithms, but is possible here.

Recursive Solution

int maxSumRec( const vector<int> & a, int left, int right )
{
    if( left == right )  // Base case
        if( a[ left ] > 0 )
            return a[ left ];
        else
            return 0;

    int center = ( left + right ) / 2;
    int maxLeftSum  = maxSumRec( a, left, center );
    int maxRightSum = maxSumRec( a, center + 1, right );

    int maxLeftBorderSum = 0, leftBorderSum = 0;
    for( int i = center; i >= left; i-- )
    {
        leftBorderSum += a[ i ];
        if( leftBorderSum > maxLeftBorderSum )
            maxLeftBorderSum = leftBorderSum;
    }
    int maxRightBorderSum = 0, rightBorderSum = 0;
    for( int j = center + 1; j <= right; j++ )
    {
        rightBorderSum += a[ j ];
        if( rightBorderSum > maxRightBorderSum )
            maxRightBorderSum = rightBorderSum;
    }
    return max3( maxLeftSum, maxRightSum,
                    maxLeftBorderSum + maxRightBorderSum );
}

Recursive Solution

Solution is $O\left(N \log N\right)$
We'll discuss logarithmic algorithms later

Linear Solution

int maxSubSum4( const vector<int> & a )
{
    int maxSum = 0, thisSum = 0;
    for( int j = 0; j < a.size( ); j++ )
    {
        thisSum += a[ j ];
        if( thisSum > maxSum )
            maxSum = thisSum;
        else if( thisSum < 0 )
            thisSum = 0;
    }
    return maxSum;
}

Logarithmic Algorithms

Certain classes of problems are logarithmic
Defined by the following rule: An Algorithim is $O\left(\log N\right)$ if it takes constant time to cut the problem size by a fraction.
Consider an algorithm that requires input to be read.
- Algorithm must be $\Omega\left(N\right)$ simply to read in the data
- When discussing algorithms that have this requirement that are $O\left(\log N\right)$, we consider the input to be pre-read.

Binary Search

Given an integer X and a presorted set of integers $A_{0},A_{1},…,A_{N-1}$, Find $i \ni A_{i} \equiv X$ or return $i \equiv -1$ if X is not in the input.
Obvious solution is to scan from one side to the other, but this ignores advantages presented by sorting

Binary Search

int binarySearch( const vector<Comparable> & a, 
    const Comparable & x )
{
    int low = 0, high = a.size( ) - 1;
    while( low <= high )
    {
        int mid = ( low + high ) / 2;
        if( a[ mid ] < x )
            low = mid + 1;
        else if( a[ mid ] > x )
            high = mid - 1;
        else
            return mid;   // Found
    }
    return NOT_FOUND;     // NOT_FOUND is defined as -1
}

Euclid's Algorithm

long gcd( long m, long n )
{
    while( n != 0 )
    {
        long rem = m % n;
        m = n;
        n = rem;
    }
    return m;
}

Exponentiation

bool isEven( int n )
{
    return n % 2 == 0;
}

long pow( long x, int n )
{
    if( n == 0 )
        return 1;
    if( n == 1 )
        return x;
    if( isEven( n ) )
        return pow( x * x, n / 2 );
    else
        return pow( x * x, n / 2 ) * x;
}

Checking Analysis

Once an algorithm has been analyzed, you should check to verify the results of your analysis.
Simple way to do this is to code up the algorithm and create a simple test case of it.
- Note that in contradiction to our model, all operations are not performed in the same amount of time
- Looking for differences in time based on problem size (i.e. how does the algorithm perform if N doubles)

Function	Name
c	Constant
\(\log N\)	Logarithmic
\(\log^2 N\)	Log-squared
N	Linear
\(N \log N\)
\(N^{2}\)	Quadratic
\(N^{3}\)	Cubic
\(2^N\)	Exponential

CptS 223 Fall 2020