Math Toolkit: Functions, Exponents, Logs & Sums

#Why a Maths Toolkit First

Open any ML paper or tutorial and you will see things like exp, log, a tall sigma, a product symbol, and little letters attached above and below. Most courses use these symbols on page one and never explain them.

This lesson explains them on purpose. It is not about getting good at algebra. It is about being able to look at a formula and know what to do with each piece.

Before the main ideas there are two short warm-ups. One is a six-question algebra self-check, so you can find out whether you are ready. The other is a table of set symbols, because the probability lessons use them.

Then you will meet seven small ideas, in an order where each one uses the one before:

1. Functions, the machines that turn inputs into outputs.
2. Exponents and the special number e.
3. Logarithms, the undo button for exponents.
4. Why ML lives on logs.
5. Sigma and pi, the notation for sums and products.
6. A symbol table and two formulas read word by word.
7. Sigmoid and softmax, and then a first look at slopes.

Every Python cell on this page runs in your browser. Run them. Reading about a number and seeing your own machine print it are different experiences.

#Algebra Self-Check

Try these six questions on paper before you read the answers. They use only school arithmetic, and they are the skills the rest of this lesson leans on.

1. Solve 4x = 20. What is x?
2. What is 1/2 + 1/3?
3. What are 3 squared and 10 cubed?
4. What are -5 + 8 and (-3) x (-2)?
5. What is 2 + 3 x 4? What is (2 + 3) x 4?
6. Solve 3x - 4 = 11. What is x?

If you got at least five of the six, you are ready. If you missed several, do not stop. Read this lesson slowly, run every Python cell, and redo any question you missed when its idea appears below: powers return in the exponents section, and negatives, brackets and fractions return all through the lesson.

#Sets and Symbols

When a later lesson says an event is a set of outcomes, "A or B" will be the union and "A and B" will be the intersection. Python has the same ideas built in, so you can check each row.

#Functions: Machines That Turn Inputs into Outputs

Take the rule "double it and add one". In symbols:

Feed in a few inputs and write down what comes out:

#A graph is every (x, f(x)) pair

A neural network is also a function. It takes an input (the pixels of a photo) and returns an output (the probability of "cat"). The only difference is that its rule has millions of numbers inside it, and training is the process of choosing those numbers.

#Composition: one machine feeding another

Work it for x = 3, from the inside out:

1. g(3) = 3 squared = 9.
2. f(9) = 2 x 9 + 1 = 19.

So f(g(3)) = 19. Order matters. Reverse it: f(3) = 7, then g(7) = 49. So g(f(3)) = 49, a different answer.

A neural network is a long composition. Layer 1 feeds layer 2, which feeds layer 3, so the whole model is f3(f2(f1(x))). Later, when you learn how a small change in the input travels through a stack of layers, you will use exactly this idea of nesting. That rule is called the chain rule, and composition is its raw material.

#Exponents and the Number e

Doubling is the easiest way to feel how fast this grows. After 10 doublings you have 2 to the 10 = 1024. After 20 doublings you have 1,048,576. A power is shorthand for growth that compounds.

#Zero and negative exponents

Two extensions look odd at first, but each follows from one idea: an exponent counts how many times you multiply, and subtracting one from the exponent means dividing by the base once.

Read the pattern going down: every step down in the exponent halves the value. A negative exponent means "one over". Any number to the power 0 is 1.

A note on negatives, so that 2 to the -3 is not confusing. The minus sign in an exponent is not a negative number: 2 to the -3 is 0.125, which is positive. Separately, multiplying two negative numbers gives a positive one. You can see why from a pattern: 3 x (-3) = -9, 2 x (-3) = -6, 1 x (-3) = -3, 0 x (-3) = 0. Each step down in the first number adds 3, so the next row must be (-1) x (-3) = 3. That is why (-2) x (-3) = 6.

#Two rules for powers, and fractional exponents

Two handy rules follow from "an exponent counts copies". Multiplying powers of the same base adds the exponents. Count the copies: 2 to the 3 is three 2s, 2 to the 2 is two 2s, and together there are five 2s. In numbers, 8 x 4 = 32, which is 2 to the 5.

Raising a power to a power multiplies the exponents. Take (2 to the 3) to the 2. That means two copies of 2 to the 3 multiplied together, which is (2 x 2 x 2) x (2 x 2 x 2), six 2s in all. In numbers, 8 x 8 = 64, which is 2 to the 6, and 3 x 2 = 6.

#The number e

Suppose a bank pays 100% interest per year on 1 unit of money. Paid once at year end, you hold 2. Paid twice, as 50% every half-year, you hold (1 + 1/2) squared = 2.25. Split into more and more slices, the total creeps up but never explodes:

For exp, the steepness at any spot equals the height of the curve there. At x = 2 the height is 7.389 and the steepness is also about 7.39 (between x = 2 and x = 2.001 the rise over run is 7.3928). At x = 3 both are about 20.09. No other base does this, and that is why e is the natural base. The last section of this lesson takes slopes further, and the derivatives lesson explains why exp behaves this way.

Why ML cares: exp turns any real number, positive or negative, into a positive one. Probabilities must be positive, so exp is how a model turns raw scores into valid ones. You will see that in the softmax section below.

#Logarithms: The Undo Button for Exponents

Here is the picture. Start at 1 and keep doubling. Each doubling is one step up a staircase, and the log base 2 of a number is which step you are standing on.

Reading the table sideways: to get from the value 32 back to the step number, take log2, and you get 5. The exponent walks up the stairs and the log walks back down.

Because a log is the undo of an exponent, they cancel in either order: ln(exp(x)) = x and exp(ln(x)) = x. Feed a number in and then undo it, and you are back where you started. Also note that logs only exist for positive numbers. There is no power of 2 that equals 0 or a negative number.

The log of a number below 1 is negative. Check it with base 2: log2(0.5) = -1, because 0.5 is 1 over 2, and "one over" is what a negative exponent means, so 2 to the -1 = 0.5. The natural log works the same way: ln(0.1) = -2.3026, because 0.1 is 1 over 10, so the power of e that gives 0.1 must be negative (e to the -2.3026 is 0.1). Remember this, because every probability sits between 0 and 1, so the log of a probability is always negative or zero.

#The three rules

These three rules carry almost everything. Each one is just an exponent rule turned inside out.

The first rule is the one to remember. It comes straight from the exponent rule you already saw: multiplying powers adds their exponents. A log reads off exponents, so multiplying the numbers adds their logs.

The third rule has a reason too. Let x be log2(10), which means 2 to the x is 10. Take ln of both sides. The power rule moves the exponent to the front, so x times ln(2) equals ln(10). Divide by ln(2) and you get x = ln(10) / ln(2) = 2.3026 / 0.6931 = 3.3219. As a check, 2 to the 3.3219 is 9.9998, which is 10 up to rounding.

#Why Machine Learning Lives on Logs

Here is the practical reason logs matter so much, as a short story.

Three heads in a row: 0.5 x 0.5 x 0.5 = 0.125. Ten heads in a row: 0.5 multiplied ten times is 0.000977. A hundred heads in a row is 7.9e-31, where "e-31" means the decimal point moves 31 places to the left. Four hundred heads in a row is 3.9e-121. Every extra event multiplies in another number below 1, so the total keeps shrinking toward zero.

Make it concrete with a smaller probability. Say 400 independent events each have probability 0.1. The likelihood of all 400 is 0.1 multiplied by itself 400 times, which mathematically is 10 to the power -400. Computers write that as 1e-400, meaning 1 divided by a 1 followed by 400 zeros.

A 64-bit float holds numbers down to about 1e-308 at full precision and, with reduced precision, down to about 5e-324. Below that it silently rounds to zero. 0.1 multiplied by itself 100 times is 1e-100, which still fits. At 300 copies you reach 1e-300, still fine. At 320 copies the value is about 1e-320 and has lost most of its precision. At 400 copies it is gone, and the program says 0.0.

Real models are far worse than this. A language model scoring a long sentence, or a classifier scoring thousands of training rows, multiplies far more than 400 factors.

Even with probability 0.5, which is far larger than 0.1, the plain product underflows to 0.0 after 1075 events. Python gives no warning at all.

Now use the product rule. The log of a product is the sum of the logs, and adding numbers does not shrink anything:

The answer, -921.03, loses nothing. You can still compare two models by it: the one with the larger (less negative) total log-likelihood explained the data better. Since log is increasing, "larger likelihood" and "larger log-likelihood" always pick the same winner.

Run it yourself. The cell prints the naive product, the sum of logs, and where each one breaks.

Next, try the explorer below. It draws exp and log as mirror images and shows the product-to-sum rule on numbers you choose. Its third panel repeats the underflow demo from the cell above as a second look, so spend your time on the first two.

Move the sliders and watch the exp and log curves reflect across the diagonal, then enter two numbers in the product panel and compare the log of their product with the sum of their logs.

#Cross-entropy is the same idea

#Sigma and Pi: Sums and Products in One Symbol

Start with a small sum worked by hand: add the first four square numbers, 1 squared, 2 squared, 3 squared and 4 squared. Keep a running total as you go.

The answer is 30. Sigma is the short way to write that whole table. Here is the plainest case, adding four numbers x_1, x_2, x_3 and x_4:

Here is the same sum of i squared as a plain Python loop. The total starts at 0, and each pass through the loop adds the next term:

total = 0
for i in range(1, 5):      # range(1, 5) counts 1, 2, 3, 4 and stops before 5
    total = total + i ** 2
print(total)               # 30

The stepper below does the same table for several expressions.

Pick an expression, press the step button, and watch each term join the running total, then switch to a pi expression and see the running product.

#Four sigma expressions you will meet constantly

Each of these is a sum in disguise. Work through them once and the later lessons will read easily.

#Pi: the same idea for products

The capital Greek letter pi, written with its big product symbol, means "multiply these" in exactly the same way sigma means "add these". The index, bounds and subscripts work identically.

Now the bridge from the previous section. Take the log of a pi expression and the product rule turns it into a sigma expression:

That one line is the whole reason maximum likelihood methods are written as sums of log-probabilities.

#Reading a Formula Word by Word

A formula is a sentence. It has a handful of symbols, and once you know the alphabet you can read it left to right. Here is the alphabet that ML uses most.

Two reading habits help. First, the same letter can mean different things in different books, so check the line near the formula that says what each symbol stands for. Second, brackets and bars group things. Work the inside first.

#Reading softmax

Softmax is the function that turns a list of raw scores z into probabilities. Before the formula, build it by hand for just two scores, z = [1, 0].

1. Make each score positive with exp: exp(1) = 2.718 and exp(0) = 1.
2. Add them to get the total: 2.718 + 1 = 3.718.
3. Divide each by the total: 2.718 / 3.718 = 0.731 and 1 / 3.718 = 0.269.

The two answers add to 1, and the larger score got the larger share. The formula is that recipe written for any number K of scores (K is just how many classes there are).

Read it word by word:

1. softmax(z) with subscript i means "the answer for class number i".
2. The top, e to the z_i, takes class i's score and makes it positive.
3. The fraction bar says: divide.
4. The bottom is a sigma. Its index j runs from 1 to K, meaning "all K classes". It adds up e to the z_j for every class.
5. Together: class i's share of the total of all the positive scores.

#Reading the Gaussian density

Stack those five pieces together and you have the formula.

Word by word, working from the middle outward:

1. x minus mu is how far the point is from the centre.
2. Squared, so left and right count the same.
3. Divided by two sigma squared, so a wide curve (big sigma) treats the same distance as less surprising.
4. A minus sign, then exp. At the centre the exponent is 0 and exp(0) = 1. Away from the centre it goes negative, so exp gives a number between 0 and 1.
5. The front factor rescales everything so the total area under the curve is exactly 1. The area under a curve is the amount of space between the curve and the flat axis beneath it. For a probability curve that area is the total probability of every possible outcome, so it must come to exactly 1.

#Sigmoid and Softmax: Exp Plus Normalise

The biggest score gets the biggest share, but every class keeps some probability. Because exp grows quickly, a score gap of one point already gives A nearly three times B's share.

Change one score in the widget above and watch all the probabilities move together, then add 100 to every score and see that nothing changes.

Notice the same σ letter again. Here, sigma(t) is the sigmoid function. Context tells you which sigma you are looking at: capital Σ is a sum, lowercase σ is usually a standard deviation, and σ followed by a bracket is often the sigmoid.

Why ML cares: softmax and sigmoid sit at the end of nearly every classifier, and cross-entropy then takes the log of the probability they produce. Exp and log are two halves of one pipeline.

#A Slope Is Rise Over Run

For a straight line like f(x) = 2x + 1 the slope is the same everywhere: 2. For a curve it changes. Take f(x) = x squared and look at the point x = 3 with a shrinking run:

As the run shrinks, the slope settles toward 6. That settled value is the slope of the curve exactly at x = 3. Chasing it as the run shrinks to nothing is the idea behind derivatives, and it is where the next lessons pick up. In ML, the slope of the loss tells a model which way to nudge its weights to improve.

#Try It Yourself

When you run the solution you should see the naive product print as 0.0, so the "product is zero" check prints True. The log-likelihood is -794.38, the average per event is -1.9859, and the cross-entropy is 0.3567.

#Key Takeaways

• A function is a rule with one output per input, and f(g(x)) means do g first, then f. A neural network is a long composition of simple functions.
• Exponents count repeated multiplication. A negative exponent means one over, a fractional exponent means a root, and e = 2.71828 is the base where exp(x) has a slope equal to its own value.
• A logarithm is the undo of an exponent. Its three rules are product to sum, power to multiplier, and change of base.
• Multiplying 400 probabilities of 0.1 gives exactly 0.0 in 64-bit floats, while the sum of their logs is a normal number, -921.03. This is why ML works with log-probabilities.
• Sigma means sum and pi means product. The log of a pi is a sigma, so a likelihood becomes a log-likelihood.
• Mean, dot product, variance and cross-entropy are all sigma expressions, and each one has a one-line Python sum() version.
• Softmax and sigmoid are exp followed by a normalising division. A slope is rise over run, and a derivative is what that slope settles to when the run shrinks.

#Quick Check