Math Toolkit 2: Sums, Symbols & Reading Formulas

#Sets and Symbols

When a later lesson says an event is a set of outcomes, "A or B" will be the union and "A and B" will be the intersection. Python has the same ideas built in, so you can check each row.

#Sigma and Pi: Sums and Products in One Symbol

Start with a small sum worked by hand: add the first four square numbers, 1 squared, 2 squared, 3 squared and 4 squared. Keep a running total as you go.

The answer is 30. Sigma is the short way to write that whole table. Here is the plainest case, adding four numbers x_1, x_2, x_3 and x_4:

Here is the same sum of i squared as a plain Python loop. The total starts at 0, and each pass through the loop adds the next term:

total = 0
for i in range(1, 5):      # range(1, 5) counts 1, 2, 3, 4 and stops before 5
    total = total + i ** 2
print(total)               # 30

The stepper below does the same table for several expressions.

Pick an expression, press the step button, and watch each term join the running total, then switch to a pi expression and see the running product.

#Four sigma expressions you will meet constantly

Each of these is a sum in disguise. Work through them once and the later lessons will read easily.

#Pi: the same idea for products

The capital Greek letter pi, written with its big product symbol, means "multiply these" in exactly the same way sigma means "add these". The index, bounds and subscripts work identically.

Now the bridge from Part 1. Take the log of a pi expression and the product rule from the logs section turns it into a sigma expression:

That one line is the whole reason maximum likelihood methods are written as sums of log-probabilities.

#Reading a Formula Word by Word

A formula is a sentence. It has a handful of symbols, and once you know the alphabet you can read it left to right. Here is the alphabet that ML uses most.

Two reading habits help. First, the same letter can mean different things in different books, so check the line near the formula that says what each symbol stands for. Second, brackets and bars group things. Work the inside first.

#Reading softmax

Softmax is the function that turns a list of raw scores z into probabilities. Before the formula, build it by hand for just two scores, z = [1, 0].

1. Make each score positive with exp: exp(1) = 2.718 and exp(0) = 1.
2. Add them to get the total: 2.718 + 1 = 3.718.
3. Divide each by the total: 2.718 / 3.718 = 0.731 and 1 / 3.718 = 0.269.

The two answers add to 1, and the larger score got the larger share. The formula is that recipe written for any number K of scores (K is just how many classes there are).

Read it word by word:

1. softmax(z) with subscript i means "the answer for class number i".
2. The top, e to the z_i, takes class i's score and makes it positive.
3. The fraction bar says: divide.
4. The bottom is a sigma. Its index j runs from 1 to K, meaning "all K classes". It adds up e to the z_j for every class.
5. Together: class i's share of the total of all the positive scores.

#Reading the Gaussian density

Stack those five pieces together and you have the formula.

Word by word, working from the middle outward:

1. x minus mu is how far the point is from the centre.
2. Squared, so left and right count the same.
3. Divided by two sigma squared, so a wide curve (big sigma) treats the same distance as less surprising.
4. A minus sign, then exp. At the centre the exponent is 0 and exp(0) = 1. Away from the centre it goes negative, so exp gives a number between 0 and 1.
5. The front factor rescales everything so the total area under the curve is exactly 1. The area under a curve is the amount of space between the curve and the flat axis beneath it. For a probability curve that area is the total probability of every possible outcome, so it must come to exactly 1.

#Sigmoid and Softmax: Exp Plus Normalise

The biggest score gets the biggest share, but every class keeps some probability. Because exp grows quickly, a score gap of one point already gives A nearly three times B's share.

Change one score in the widget above and watch all the probabilities move together, then add 100 to every score and see that nothing changes.

That add-100 switch also exposes a practical trap. The probabilities do not change, but the naive exp of a large score can overflow. A 32-bit float, common on GPUs, tops out near e^88.7, so exp(102) is already inf there, and inf divided by inf is NaN. Plain Python floats are 64-bit and reach much further: math.exp(709) is about 8.2e307, and math.exp(710) raises an OverflowError (NumPy returns inf instead). So the widget's overflow demo is a 32-bit story, and in Python the same trouble starts only near exp(710). The standard fix works for both. Subtract the largest score from every score before applying exp. The largest exponent becomes 0, so nothing overflows, and the probabilities are unchanged because exp(z - m) = exp(z) / exp(m) and the common factor cancels in the division.

Notice the same σ letter again. Here, sigma(t) is the sigmoid function. Context tells you which sigma you are looking at: capital Σ is a sum, lowercase σ is usually a standard deviation, and σ followed by a bracket is often the sigmoid.

Why ML cares: softmax and sigmoid sit at the end of nearly every classifier, and cross-entropy then takes the log of the probability they produce. Exp and log are two halves of one pipeline.

#A Slope Is Rise Over Run

For a straight line like f(x) = 2x + 1 the slope is the same everywhere: 2. For a curve it changes. Take f(x) = x squared and look at the point x = 3 with a shrinking run:

As the run shrinks, the slope settles toward 6. That settled value is the slope of the curve exactly at x = 3. Chasing it as the run shrinks to nothing is the idea behind derivatives, and it is where the next lessons pick up. In ML, the slope of the loss tells a model which way to nudge its weights to improve.

#Try It Yourself

When you run the solution you should see the naive product print as 0.0, so the "product is zero" check prints True. The log-likelihood is -794.38, the average per event is -1.9859, and the cross-entropy is 0.3567.

#Key Takeaways

• Sigma means sum and pi means product. The log of a pi is a sigma, so a likelihood becomes a log-likelihood.
• Mean, dot product, variance and cross-entropy are all sigma expressions, and each one has a one-line Python sum() version.
• Softmax and sigmoid are exp followed by a normalising division. A slope is rise over run, and a derivative is what that slope settles to when the run shrinks.
• Python floats are 64-bit and overflow near exp(710), while a 32-bit float overflows near exp(88.7). Subtracting the largest score before exp avoids both and leaves the softmax probabilities unchanged.

#Quick Check