RugvAI Labs

NumPy & Pandas: The Data Science Toolkit

NumPy underlies every tensor library — PyTorch tensors and JAX arrays speak NumPy's array protocol. Pandas is the table-wrangling layer sitting under most data work in Python, from Kaggle notebooks to production feature pipelines. These two libraries are why Python won data science, and learning them well separates "I know Python" from "I can ship ML."

Learning Objectives

After this lesson, you will be able to:

Create and manipulate NumPy arrays with slicing, reshaping, and broadcasting
Use vectorized operations instead of loops for massive speed gains
Build and query Pandas DataFrames -- the workhorse of data analysis
Filter, group, and aggregate data with Pandas
Load CSV files and perform basic exploratory data analysis

Prerequisites

Before You Start

Review

#NumPy Arrays

NumPy's core object is the ndarray (n-dimensional array). Think of it as a supercharged Python list that can do math blazingly fast.

pythonrunnable cell

import numpy as np

# Creating arrays
a = np.array([1, 2, 3, 4, 5])           # from a Python list
b = np.zeros(5)                           # [0. 0. 0. 0. 0.]
c = np.ones(5)                            # [1. 1. 1. 1. 1.]
d = np.arange(0, 10, 2)                   # [0, 2, 4, 6, 8]
e = np.linspace(0, 1, 5)                  # [0.  0.25 0.5  0.75 1. ]
f = np.random.randn(5)                    # 5 random numbers (normal distribution)

print(f"Array: {a}")
print(f"Shape: {a.shape}")    # (5,)
print(f"Data type: {a.dtype}") # int64
print(f"Dimensions: {a.ndim}") # 1

#2D Arrays (Matrices)

pythonreference · read-only

# 2D array -- a matrix
matrix = np.array([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
])

print(f"Shape: {matrix.shape}")    # (3, 3)
print(f"Dimensions: {matrix.ndim}") # 2
print(f"Total elements: {matrix.size}")  # 9

# Zeros and ones matrices
zeros = np.zeros((3, 4))    # 3 rows, 4 columns of zeros
ones = np.ones((2, 3))      # 2 rows, 3 columns of ones
identity = np.eye(3)        # 3x3 identity matrix
random_matrix = np.random.randn(3, 3)  # 3x3 random values

#Indexing and Slicing

NumPy slicing is like Python list slicing but extended to multiple dimensions:

pythonreference · read-only

arr = np.array([10, 20, 30, 40, 50, 60, 70, 80, 90])

# Basic indexing (same as lists)
print(arr[0])      # 10
print(arr[-1])     # 90
print(arr[2:5])    # [30, 40, 50]

# 2D indexing
matrix = np.array([
    [1,  2,  3,  4],
    [5,  6,  7,  8],
    [9,  10, 11, 12]
])

print(matrix[0, 0])    # 1  (row 0, column 0)
print(matrix[1, 2])    # 7  (row 1, column 2)
print(matrix[0])       # [1, 2, 3, 4]  (entire row 0)
print(matrix[:, 0])    # [1, 5, 9]  (entire column 0)
print(matrix[0:2, 1:3])  # [[2, 3], [6, 7]] (submatrix)

# Boolean indexing (filtering)
scores = np.array([85, 92, 78, 95, 88, 76, 91])
high_scores = scores[scores >= 90]  # [92, 95, 91]
print(f"Scores >= 90: {high_scores}")

# Fancy indexing
indices = np.array([0, 2, 4])
selected = scores[indices]  # [85, 78, 88]
print(f"Selected: {selected}")

#Vectorized Operations and Broadcasting

This is where NumPy's real power shines. Vectorized operations apply to every element at once without Python loops:

pythonreference · read-only

a = np.array([1, 2, 3, 4, 5])
b = np.array([10, 20, 30, 40, 50])

# Element-wise operations (no loops needed!)
print(a + b)       # [11, 22, 33, 44, 55]
print(a * b)       # [10, 40, 90, 160, 250]
print(a ** 2)      # [1, 4, 9, 16, 25]
print(np.sqrt(a))  # [1.  1.41 1.73 2.  2.24]

# Broadcasting -- operations between different shapes
scores = np.array([85, 92, 78, 95, 88])

# Add 5 bonus points to every score (scalar + array)
curved = scores + 5
print(f"Curved: {curved}")  # [90, 97, 83, 100, 93]

# Normalize to 0-1 range
normalized = (scores - scores.min()) / (scores.max() - scores.min())
print(f"Normalized: {normalized}")
# [0.41 0.82 0.   1.   0.59]

What Do You Think?

What's faster: a Python for loop or NumPy vectorized operation?

The answer is NumPy vectorized operation, typically 10-100x faster. NumPy operations are implemented in C and operate on contiguous blocks of memory. A Python for loop has overhead for every single iteration (type checking, object creation, etc.). For a million elements, a Python loop takes tens of milliseconds while NumPy takes well under one — on a current laptop, roughly 40 ms versus 0.5 ms.

#Speed Comparison

pythonreference · read-only

import time

size = 1_000_000
python_list = list(range(size))
numpy_array = np.arange(size)

# Python loop
start = time.time()
result_loop = [x * 2 + 1 for x in python_list]
loop_time = time.time() - start

# NumPy vectorized
start = time.time()
result_numpy = numpy_array * 2 + 1
numpy_time = time.time() - start

print(f"Python loop: {loop_time:.4f}s")
print(f"NumPy:       {numpy_time:.4f}s")
print(f"NumPy is {loop_time / numpy_time:.0f}x faster!")
# Typical output: NumPy is 50-100x faster!

#Reshaping

pythonreference · read-only

a = np.arange(12)  # [0, 1, 2, ..., 11]

# Reshape to 3 rows, 4 columns
reshaped = a.reshape(3, 4)
print(reshaped)
# [[ 0,  1,  2,  3],
#  [ 4,  5,  6,  7],
#  [ 8,  9, 10, 11]]

# Reshape to 2 rows, -1 means "figure out the other dimension"
reshaped2 = a.reshape(2, -1)
print(reshaped2.shape)  # (2, 6)

# Flatten back to 1D
flat = reshaped.flatten()
print(flat)  # [0, 1, 2, ..., 11]

# Transpose
print(reshaped.T)       # swap rows and columns
print(reshaped.T.shape) # (4, 3)

See How Array Shapes WorkInteractive

Loading visualization...

#Common Aggregation Functions

pythonreference · read-only

data = np.array([85, 92, 78, 95, 88, 76, 91, 83, 97, 80])

print(f"Mean:   {np.mean(data):.1f}")     # 86.5
print(f"Median: {np.median(data):.1f}")   # 86.5
print(f"Std:    {np.std(data):.1f}")      # 6.9
print(f"Min:    {np.min(data)}")          # 76
print(f"Max:    {np.max(data)}")          # 97
print(f"Sum:    {np.sum(data)}")          # 865
print(f"Argmax: {np.argmax(data)}")       # 8 (index of max value)

# Aggregation along axes (for 2D arrays)
grades = np.array([
    [85, 90, 78],  # Student 1: math, science, english
    [92, 88, 95],  # Student 2
    [76, 82, 89],  # Student 3
])

print(f"Average per student: {np.mean(grades, axis=1)}")  # [84.3, 91.7, 82.3]
print(f"Average per subject: {np.mean(grades, axis=0)}")  # [84.3, 86.7, 87.3]

Intuition

Build Your Intuition

Broadcasting rules: align shapes from the right, then stretch dimensions of size 1.

When NumPy sees grades - row_means, it follows three rules:

Compare shapes right-to-left: (3, 3) vs (3,) — the trailing dimensions match (3 == 3).
If a dimension is 1 (or missing), NumPy stretches it to match the other array.
Shapes must either match or be 1 after alignment — otherwise it raises an error.

pythonreference · read-only

grades = np.array([[85, 90, 78], [92, 88, 95], [76, 82, 89]])  # shape (3, 3)
row_means = np.mean(grades, axis=1)      # shape (3,) — one mean per student
# WRONG: (3, 3) - (3,) tries to subtract along columns, not rows
# col_centered = grades - row_means   # raises error or gives wrong result

# RIGHT: reshape row_means to (3, 1) so broadcasting subtracts row-wise
row_means_col = row_means.reshape(-1, 1)  # shape (3, 1)
row_centered = grades - row_means_col     # (3, 3) - (3, 1) → (3, 3) ✓

#Pandas DataFrames

Pandas is built on top of NumPy and adds labeled rows/columns, making data manipulation intuitive:

pythonrunnable cell

import pandas as pd

# Create a DataFrame from a dictionary
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
    "age": [17, 16, 18, 17, 16],
    "math": [95, 87, 92, 78, 99],
    "science": [88, 91, 85, 82, 94],
    "english": [92, 78, 88, 90, 85],
})

print(df)
#       name  age  math  science  english
# 0    Alice   17    95       88       92
# 1      Bob   16    87       91       78
# 2  Charlie   18    92       85       88
# 3    Diana   17    78       82       90
# 4      Eve   16    99       94       85

# Basic info
print(f"\nShape: {df.shape}")        # (5, 5)
print(f"Columns: {list(df.columns)}") # ['name', 'age', 'math', ...]
print(f"\n{df.dtypes}")              # data types per column
print(f"\n{df.describe()}")          # summary statistics

#Selecting Data

pythonreference · read-only

# Select a single column (returns a Series)
names = df["name"]
print(names)

# Select multiple columns (returns a DataFrame)
scores = df[["math", "science", "english"]]
print(scores)

# Select rows by index
print(df.iloc[0])     # first row (by integer position)
print(df.iloc[0:3])   # first 3 rows

# Select by label
df_indexed = df.set_index("name")
print(df_indexed.loc["Alice"])   # Alice's row
print(df_indexed.loc["Alice", "math"])  # Alice's math score: 95

Watch Out

Watch out

Fix: Use .iloc[row_int, col_int] for positional selection (like Python list indexing) and .loc[row_label, col_label] for label-based selection. When the index is the default 0, 1, 2 ... range, .loc[0] and .iloc[0] return the same row — but as soon as you reset the index to a column (like 'name'), .loc['Alice'] works while .iloc['Alice'] raises a TypeError.

Confusing .iloc (integer-location based) with .loc (label-based) — using integer indices with .loc or string labels with .iloc

This is one of the most common sources of subtle Pandas bugs. When you filter a DataFrame (df[df['math'] > 90]), the resulting DataFrame keeps the original integer index (e.g., rows 0, 2, 4). Now .iloc[1] returns the second row of the filtered result (original index 2), while .loc[1] returns original row 1 — which may not even exist in the filtered result and raises a KeyError. For safe assignment, always use .loc: df.loc[condition, 'column'] = value.

#Filtering

pythonreference · read-only

# Boolean filtering -- like NumPy but with labeled data
high_math = df[df["math"] >= 90]
print("Math >= 90:")
print(high_math)

# Multiple conditions (use & for AND, | for OR, ~ for NOT)
smart_young = df[(df["math"] >= 90) & (df["age"] <= 17)]
print("\nMath >= 90 AND age <= 17:")
print(smart_young)

# Filter by string
alice = df[df["name"] == "Alice"]
print(f"\nAlice's data:\n{alice}")

#Data Manipulation with Pandas

#Adding Columns

pythonreference · read-only

# Calculate average score
df["average"] = (df["math"] + df["science"] + df["english"]) / 3
df["average"] = df["average"].round(1)

# Categorize students
df["grade_letter"] = pd.cut(
    df["average"],
    bins=[0, 60, 70, 80, 90, 100],
    labels=["F", "D", "C", "B", "A"]
)

print(df[["name", "average", "grade_letter"]])

#Sorting

pythonreference · read-only

# Sort by a column
df_sorted = df.sort_values("average", ascending=False)
print("Ranked by average:")
print(df_sorted[["name", "average"]])

# Sort by multiple columns
df_multi = df.sort_values(["age", "average"], ascending=[True, False])

#GroupBy -- Split, Apply, Combine

pythonreference · read-only

# Group by age and calculate statistics
age_groups = df.groupby("age")["average"].agg(["mean", "count", "max"])
print("Stats by age:")
print(age_groups)

# More complex groupby
report = df.groupby("age").agg({
    "math": "mean",
    "science": "mean",
    "english": "mean",
    "name": "count"
}).rename(columns={"name": "student_count"})
print("\nDetailed report by age:")
print(report.round(1))

#Handling Missing Data

pythonreference · read-only

# Create data with missing values
df_messy = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie", "Diana"],
    "score": [95, None, 88, None],
    "grade": ["A", "B", None, "C"],
})

# Check for missing values
print(df_messy.isnull().sum())  # count NaN per column

# Fill missing values
df_filled = df_messy.fillna({"score": df_messy["score"].mean(), "grade": "N/A"})
print(df_filled)

# Drop rows with any missing values
df_clean = df_messy.dropna()
print(f"Rows after dropping NaN: {len(df_clean)}")

#Reading CSV Files

pythonrunnable cell

# Read a CSV file into a DataFrame
# df = pd.read_csv("students.csv")

# Common options
# df = pd.read_csv("data.csv", sep=";", encoding="utf-8", index_col=0)
# df = pd.read_csv("big_file.csv", nrows=1000)  # read first 1000 rows only

# Quick exploration workflow
# print(df.head())         # first 5 rows
# print(df.info())         # column types and non-null counts
# print(df.describe())     # summary statistics
# print(df.value_counts("column_name"))  # frequency counts

#Putting It Together -- Analyzing a Dataset

pythonrunnable cell

import numpy as np
import pandas as pd

# Create a mock student grades dataset
np.random.seed(42)
n_students = 100

data = {
    "student_id": [f"STU_{i:03d}" for i in range(n_students)],
    "class": np.random.choice(["9A", "9B", "9C"], n_students),
    "math": np.random.normal(75, 12, n_students).clip(0, 100).astype(int),
    "science": np.random.normal(78, 10, n_students).clip(0, 100).astype(int),
    "english": np.random.normal(80, 11, n_students).clip(0, 100).astype(int),
    "attendance_pct": np.random.normal(90, 8, n_students).clip(50, 100).round(1),
}
df = pd.DataFrame(data)

# Calculate average score
df["avg_score"] = df[["math", "science", "english"]].mean(axis=1).round(1)

# Assign letter grades
df["letter_grade"] = pd.cut(
    df["avg_score"],
    bins=[0, 60, 70, 80, 90, 100],
    labels=["F", "D", "C", "B", "A"]
)

# Analysis 1: Class-level performance
print("=== CLASS PERFORMANCE ===")
class_report = df.groupby("class").agg({
    "avg_score": ["mean", "std"],
    "student_id": "count"
}).round(1)
print(class_report)

# Analysis 2: Correlation between attendance and scores
print("\n=== ATTENDANCE vs PERFORMANCE ===")
high_attendance = df[df["attendance_pct"] >= 90]
low_attendance = df[df["attendance_pct"] < 90]
print(f"Avg score (attendance >= 90%): {high_attendance['avg_score'].mean():.1f}")
print(f"Avg score (attendance < 90%):  {low_attendance['avg_score'].mean():.1f}")

# Analysis 3: Top and bottom students
print("\n=== TOP 5 STUDENTS ===")
top5 = df.nlargest(5, "avg_score")[["student_id", "class", "avg_score", "letter_grade"]]
print(top5.to_string(index=False))

print("\n=== STUDENTS NEEDING HELP (avg < 65) ===")
struggling = df[df["avg_score"] < 65][["student_id", "class", "avg_score"]]
print(f"Count: {len(struggling)}")
print(struggling.to_string(index=False))

# Analysis 4: Subject difficulty
print("\n=== SUBJECT DIFFICULTY ===")
for subject in ["math", "science", "english"]:
    mean = df[subject].mean()
    fail_rate = (df[subject] < 60).mean() * 100
    print(f"  {subject:>7}: avg = {mean:.1f}, fail rate = {fail_rate:.1f}%")

#Code Playground

pythonplayground.py · Pyodide

# NumPy & Pandas concepts in Python
# Vectorized operations, DataFrames, filtering, groupby

import numpy as np

# --- NUMPY OPERATIONS ---
scores = np.array([85, 92, 78, 95, 88, 76, 91, 83, 97, 80])
print("Scores:", scores)
print(f"Mean: {np.mean(scores):.1f}")
print(f"Std: {np.std(scores):.1f}")
normalized = (scores - scores.min()) / (scores.max() - scores.min())
print("Normalized:", [f"{v:.2f}" for v in normalized])
print("Curved (+5):", scores + 5)

# --- PANDAS-LIKE OPERATIONS (pure Python for Pyodide) ---

students = [
  {"name": "Alice",   "cls": "9A", "math": 95, "science": 88, "english": 92},
  {"name": "Bob",     "cls": "9A", "math": 87, "science": 91, "english": 78},
  {"name": "Charlie", "cls": "9B", "math": 92, "science": 85, "english": 88},
  {"name": "Diana",   "cls": "9B", "math": 78, "science": 82, "english": 90},
  {"name": "Eve",     "cls": "9A", "math": 99, "science": 94, "english": 85},
  {"name": "Frank",   "cls": "9B", "math": 71, "science": 75, "english": 68},
  {"name": "Grace",   "cls": "9A", "math": 88, "science": 90, "english": 91},
  {"name": "Hank",    "cls": "9B", "math": 65, "science": 70, "english": 72},
]

# Add average column
for s in students:
  s["average"] = round((s["math"] + s["science"] + s["english"]) / 3, 1)

# Filter: honor roll (avg >= 90)
honor = [s for s in students if s["average"] >= 90]
print("\nHonor Roll:", ", ".join(f"{s['name']} ({s['average']})" for s in honor))

# GroupBy: average score per class
from collections import defaultdict
classes: dict[str, list[float]] = defaultdict(list)
for s in students:
  classes[s["cls"]].append(s["average"])

print("\nClass averages:")
for cls, avgs in classes.items():
  print(f"  {cls}: {sum(avgs)/len(avgs):.1f} (n={len(avgs)})")

# Sort by average descending
ranked = sorted(students, key=lambda s: s["average"], reverse=True)
print("\nRankings:")
for i, s in enumerate(ranked, 1):
  print(f"  #{i} {s['name']:<8} {s['cls']} avg={s['average']}")

# Subject analysis
print("\nSubject stats:")
for subj in ["math", "science", "english"]:
  vals = np.array([s[subj] for s in students])
  print(f"  {subj:<8}: mean={np.mean(vals):.1f}, std={np.std(vals):.1f}")

# CHALLENGES:
# 1. Find the student with the highest math score
# 2. Calculate the correlation between math and science scores
# 3. Add a "letter_grade" based on average (A >= 90, B >= 80, C >= 70, D >= 60, F)
# 4. Find which class has the best average english score
# 5. Create a "needs_help" list of students scoring below 75 in any subject

Tests · Analyze the student dataset, calculate statistics, and verify your results!

#NumPy for ML: Images, Batches & Embeddings

In machine learning, your data almost always arrives as multi-dimensional arrays. Here's the most common shapes you'll encounter:

pythonrunnable cell

import numpy as np

# Images are 3D arrays: (height, width, channels)
image = np.zeros((28, 28))        # Grayscale MNIST digit (28×28 pixels)
rgb   = np.zeros((224, 224, 3))   # RGB image (224×224, 3 colour channels)

# A batch of 32 images for training:
batch = np.zeros((32, 28, 28))    # 32 grayscale images at once
print(batch.shape)                # (32, 28, 28)
print(batch[0].shape)             # (28, 28) — first image

# Embeddings are 1D or 2D vectors
word_vec     = np.random.randn(512)         # One word embedding (512 dims)
sentence_mat = np.random.randn(100, 512)   # 100 words × 512 dims

# Reshaping — fundamental in ML pipelines
flat = image.reshape(-1)          # (784,) — flatten for a dense layer
back = flat.reshape(28, 28)       # (28, 28) — back to image

Why this matters: When you load a dataset into a neural network, it arrives as a 4D array (batch, height, width, channels). Understanding array shapes is the #1 skill for debugging ML model errors like RuntimeError: Expected input size.

Interactive Lab

Visualize how NumPy arrays transform and combine — the math behind every ML model

Vectors & Matrices

#Key Takeaways

NumPy arrays are faster than Python lists -- vectorized operations run in optimized C code, giving 10-100x speedups over Python loops. Always use NumPy for numerical computation
Broadcasting lets you mix array shapes -- adding a scalar to an array, or a 1D array to a 2D array, works automatically. NumPy "broadcasts" the smaller shape to match the larger one
DataFrames are labeled 2D tables -- unlike NumPy arrays, Pandas DataFrames have named columns and can hold mixed types (strings, numbers, dates). They are the primary data structure for tabular data
Filtering is powerful and concise -- df[df["score"] >= 90] instantly filters to matching rows. Combine conditions with & (AND) and | (OR) for complex queries
GroupBy is the key to data analysis -- df.groupby("category")["value"].mean() splits data into groups, applies a function to each group, and combines the results. This pattern solves most analysis questions

Quick Check1 / 5

Why are NumPy operations faster than Python for loops?

Your Reflection

Continue Learning

Discussion

NumPy & Pandas: The Data Science Toolkit

After this lesson, you will be able to:

Before You Start

#NumPy Arrays

#2D Arrays (Matrices)

#Indexing and Slicing

#Vectorized Operations and Broadcasting

#Speed Comparison

#Reshaping

#Common Aggregation Functions

#Pandas DataFrames

#Selecting Data

#Filtering

#Data Manipulation with Pandas

#Adding Columns

#Sorting

#GroupBy -- Split, Apply, Combine

#Handling Missing Data

#Reading CSV Files

#Putting It Together -- Analyzing a Dataset

#Code Playground

Views vs Copies -- A Subtle NumPy Trap

#NumPy for ML: Images, Batches & Embeddings

#Key Takeaways