RugvAI Labs

Regular Expressions: Pattern Matching Mastery

Every LLM tokenizer starts with a regex (GPT-2's tokenizer is famously ~30 lines of regex). Email validators, log parsers, SQL injection filters — regex is the universal text-pattern language. Master it once in Python, and you've mastered it in JavaScript, Go, and grep too.

Learning Objectives

After this lesson, you will be able to:

Understand why regular expressions exist and when to use them over plain string methods
Write patterns using character classes, quantifiers, anchors, and alternation
Use capturing groups, named groups, and backreferences to extract structured data
Apply re.match(), re.search(), re.findall(), re.sub(), and re.split() to real text processing tasks
Write lookahead and lookbehind assertions for context-sensitive matching
Compile patterns with re.compile() for performance in loops and production code
Recognize and avoid catastrophic backtracking (ReDoS) in your patterns
Build real-world regex solutions: email validation, log parsing, tokenization, and data cleaning

Why

Why This Matters

NLP pipelines start with regex -- before any model touches your text, regular expressions handle sentence splitting, tokenization, whitespace normalization, and entity extraction. NLTK's word_tokenize, spaCy's tokenizer, and even GPT's BPE tokenizer all rely on regex patterns internally
Data cleaning is 80% of ML work, and regex handles 80% of data cleaning -- extracting phone numbers from free-text fields, normalizing date formats, stripping HTML tags, parsing log files into structured records. These are everyday tasks in data science
Log analysis and monitoring depend on regex -- every production ML system generates logs. Parsing timestamps, error codes, request IDs, and stack traces from log lines is fundamentally a regex problem. Tools like Elasticsearch, Splunk, and Grafana all use regex for log parsing
Web scraping and API response parsing use regex -- while BeautifulSoup handles HTML structure, regex handles the fine-grained text extraction within elements: pulling prices from strings like "$1,234.56", extracting version numbers, parsing structured text
Regex is universal across languages -- the same patterns work in Python, JavaScript, Java, Go, and even command-line tools like grep, sed, and awk. Learning regex once pays dividends everywhere

Prerequisites

Before You Start

Review

#Why Regex?

#The Limits of Plain String Methods

Python's built-in string methods are great for simple cases, but they hit a wall quickly:

pythonrunnable cell

# Simple case: string methods work fine
text = "Hello, World!"
print("World" in text)          # True
print(text.replace("World", "Python"))  # "Hello, Python!"
print(text.startswith("Hello"))  # True

But what about these tasks?

pythonrunnable cell

# Task 1: Find all phone numbers in text (any format)
text = "Call me at 415-555-1234 or (555) 867-5309 or 555.123.4567"
# str.find()? You'd need multiple calls for each format...

# Task 2: Validate an email address
email = "user@example.com"
# How many string checks would you need? @, dot, no spaces, valid chars...

# Task 3: Extract all prices from a product listing
listing = "Widget A: $12.99, Widget B: $1,234.56, Widget C: $0.99"
# str.split("$")? What about the comma in 1,234.56?

# Task 4: Replace multiple date formats with a single standard
log = "2024-01-15 error, 01/15/2024 warning, Jan 15 2024 info"
# You'd need a separate replace() for every format...

These tasks share a common pattern: you need to match shapes of text, not exact strings. That is exactly what regular expressions do.

#Enter re -- Python's Regex Module

pythonrunnable cell

import re

# Task 1: Find ALL phone numbers, regardless of format
text = "Call me at 415-555-1234 or (555) 867-5309 or 555.123.4567"
phones = re.findall(r'\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}', text)
print(phones)
# ['415-555-1234', '(555) 867-5309', '555.123.4567']

# Task 2: Validate an email (simplified)
email = "user@example.com"
if re.match(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', email):
    print("Valid email")

# Task 3: Extract all prices
listing = "Widget A: $12.99, Widget B: $1,234.56, Widget C: $0.99"
prices = re.findall(r'\$[\d,]+\.\d{2}', listing)
print(prices)  # ['$12.99', '$1,234.56', '$0.99']

# Task 4: Find dates in multiple formats
log = "2024-01-15 error, 01/15/2024 warning, Jan 15 2024 info"
dates = re.findall(r'\d{4}-\d{2}-\d{2}|\d{2}/\d{2}/\d{4}|[A-Z][a-z]{2} \d{1,2} \d{4}', log)
print(dates)  # ['2024-01-15', '01/15/2024', 'Jan 15 2024']

Intuition

Build Your Intuition

Regex is like a tiny search-and-replace programming language embedded inside a string.

Think of it like this: "hello" in Python matches the exact text "hello". But what if you want to match "any word that starts with h and ends with o"? Plain strings cannot express that. Regex gives you a mini-language to describe patterns:

h\w+o means "h, then one or more word characters, then o"
\d{3}-\d{4} means "3 digits, a dash, 4 digits" (a phone number)
[A-Z][a-z]+ means "a capital letter followed by one or more lowercase letters" (a capitalized word)

The r prefix before the string (r'\d{3}') makes it a raw string -- backslashes are treated literally instead of as escape characters. Without it, \d would be interpreted as an escape sequence by Python before regex even sees it. Always use raw strings for regex patterns.

#Basic Patterns

#Literal Characters

The simplest regex is a literal string. It matches exactly that text:

pythonrunnable cell

import re

text = "The cat sat on the mat"

# Literal match -- finds the exact substring
print(re.findall(r'cat', text))   # ['cat']
print(re.findall(r'at', text))    # ['at', 'at', 'at'] -- found in cat, sat, mat
print(re.findall(r'dog', text))   # [] -- not found

#Special Characters (Metacharacters)

These characters have special meaning in regex. To match them literally, escape with \:

pythonreference · read-only

# . (dot) matches ANY character except newline
print(re.findall(r'c.t', text))    # ['cat'] -- c, any char, t

# Special characters that need escaping: . ^ $ * + ? { } [ ] \ | ( )
price = "The price is $9.99"
print(re.findall(r'\$\d+\.\d+', price))  # ['$9.99']
#                  \$ = literal $
#                  \d+ = one or more digits
#                  \. = literal dot
#                  \d+ = one or more digits

#Character Classes

Character classes match any single character from a set:

pythonreference · read-only

# [abc] matches a, b, or c
print(re.findall(r'[cm]at', text))   # ['cat', 'mat']

# [a-z] matches any lowercase letter (range)
print(re.findall(r'[a-z]+', "Hello World 123"))  # ['ello', 'orld']

# [A-Za-z] matches any letter
print(re.findall(r'[A-Za-z]+', "Hello World 123"))  # ['Hello', 'World']

# [0-9] matches any digit (same as \d)
print(re.findall(r'[0-9]+', "Room 237, Floor 13"))  # ['237', '13']

# [^abc] matches any character EXCEPT a, b, c (negation)
print(re.findall(r'[^aeiou ]+', "hello world"))  # ['h', 'll', 'w', 'rld']

# Combine ranges
print(re.findall(r'[A-Za-z0-9_]+', "user_name = 42"))  # ['user_name', '42']

#Shorthand Character Classes

pythonreference · read-only

# \d = digit       [0-9]
# \D = non-digit   [^0-9]
# \w = word char   [a-zA-Z0-9_]
# \W = non-word    [^a-zA-Z0-9_]
# \s = whitespace  [ \t\n\r\f\v]
# \S = non-space   [^ \t\n\r\f\v]
# .  = any char    (except \n by default)

text = "Order #12345 placed on 2024-01-15"
print(re.findall(r'\d+', text))      # ['12345', '2024', '01', '15']
print(re.findall(r'\w+', text))      # ['Order', '12345', 'placed', 'on', '2024', '01', '15']
print(re.findall(r'\S+', text))      # ['Order', '#12345', 'placed', 'on', '2024-01-15']

#Quantifiers: How Many?

pythonreference · read-only

# *  = 0 or more
# +  = 1 or more
# ?  = 0 or 1 (optional)
# {n} = exactly n
# {n,} = n or more
# {n,m} = between n and m (inclusive)

text = "aaa ab abbb ac a"

print(re.findall(r'ab*', text))     # ['a', 'a', 'a', 'ab', 'abbb', 'a', 'a']
#                                      a=0 b's, ab=1 b, abbb=3 b's
print(re.findall(r'ab+', text))     # ['ab', 'abbb']  -- must have at least 1 b
print(re.findall(r'ab?', text))     # ['a', 'a', 'a', 'ab', 'ab', 'a', 'a']
#                                      ? means 0 or 1 b

# Specific counts
phone = "Call 555-1234 or 555-12345678"
print(re.findall(r'\d{3}-\d{4}', phone))    # ['555-1234'] -- exactly 3, dash, exactly 4
print(re.findall(r'\d{3}-\d{4,8}', phone))  # ['555-1234', '555-12345678']

#Anchors: Where to Match

pythonreference · read-only

# ^  = start of string (or start of line with re.MULTILINE)
# $  = end of string (or end of line with re.MULTILINE)
# \b = word boundary (between \w and \W)

text = "cat caterpillar concatenate scat"

# \b ensures we match whole words
print(re.findall(r'cat', text))       # ['cat', 'cat', 'cat', 'cat'] -- 4 matches
print(re.findall(r'\bcat\b', text))   # ['cat'] -- only the standalone word

# ^ and $ for start/end
lines = "error: disk full\nwarning: low memory\nerror: timeout"
print(re.findall(r'^error', lines))                    # ['error'] -- only first line
print(re.findall(r'^error', lines, re.MULTILINE))      # ['error', 'error'] -- both lines

# \b is critical for NLP tokenization
sentence = "I can't believe it's not butter"
words = re.findall(r"\b[\w']+\b", sentence)
print(words)  # ["I", "can't", "believe", "it's", "not", "butter"]

What Do You Think?

What does the pattern r'\b\d{3}\b' match in the string '12 123 1234 12345'?

#Groups and Capturing

#Capturing Groups with ()

Parentheses create capturing groups -- they extract specific parts of a match:

pythonrunnable cell

import re

# Without groups: findall returns the full match
text = "2024-01-15 and 2023-12-25"
print(re.findall(r'\d{4}-\d{2}-\d{2}', text))
# ['2024-01-15', '2023-12-25']

# With groups: findall returns only the captured parts
print(re.findall(r'(\d{4})-(\d{2})-(\d{2})', text))
# [('2024', '01', '15'), ('2023', '12', '25')]

# Access groups via match object
match = re.search(r'(\d{4})-(\d{2})-(\d{2})', text)
if match:
    print(match.group(0))   # '2024-01-15'  -- full match
    print(match.group(1))   # '2024'        -- first group
    print(match.group(2))   # '01'          -- second group
    print(match.group(3))   # '15'          -- third group
    print(match.groups())   # ('2024', '01', '15') -- all groups as tuple

#Named Groups with (?P<name>...)

Named groups make your regex self-documenting:

pythonreference · read-only

# Named groups -- much more readable than group(1), group(2)
pattern = r'(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})'

match = re.search(pattern, "Event on 2024-03-15")
if match:
    print(match.group('year'))    # '2024'
    print(match.group('month'))   # '03'
    print(match.group('day'))     # '15'
    print(match.groupdict())      # {'year': '2024', 'month': '03', 'day': '15'}

# Named groups in finditer (the professional way to iterate matches)
log = "2024-01-15 ERROR disk full | 2024-01-16 WARNING low memory"
for m in re.finditer(pattern, log):
    print(f"{m.group('year')}/{m.group('month')}/{m.group('day')}")
# 2024/01/15
# 2024/01/16

#Non-Capturing Groups with (?:...)

Sometimes you need grouping for alternation or quantifiers but do not want to capture:

pythonreference · read-only

# Without non-capturing group: both parts are captured
text = "I like cats and dogs"
print(re.findall(r'(cat|dog)s', text))
# ['cat', 'dog'] -- captures the group content

# With non-capturing group: only the full match is returned
print(re.findall(r'(?:cat|dog)s', text))
# ['cats', 'dogs'] -- no capture, returns full match

# Useful when you need alternation inside a larger pattern
urls = "Visit http://example.com or https://secure.example.com"
print(re.findall(r'https?://[\w.]+', urls))
# ['http://example.com', 'https://secure.example.com']

# Same thing with explicit non-capturing group
print(re.findall(r'(?:https?://)([\w.]+)', urls))
# ['example.com', 'secure.example.com'] -- captures only the domain

#Backreferences

Backreferences match the same text that was previously captured by a group:

pythonreference · read-only

# \1 refers to whatever group 1 matched
# Find repeated words (common typo)
text = "The the quick brown fox fox jumped over the the lazy dog"
print(re.findall(r'\b(\w+)\s+\1\b', text, re.IGNORECASE))
# ['The', 'fox', 'the']

# Find matching HTML tags
html = "<b>bold</b> and <i>italic</i>"
print(re.findall(r'<(\w+)>(.*?)</\1>', html))
# [('b', 'bold'), ('i', 'italic')]

#The re Module API

pythonrunnable cell

import re

text = "Hello World, hello python, HELLO REGEX"

# re.search() -- find FIRST match anywhere in string
match = re.search(r'hello', text, re.IGNORECASE)
print(match.group())      # 'Hello'
print(match.start())      # 0
print(match.end())        # 5
print(match.span())       # (0, 5)

# re.match() -- match only at the START of string
match = re.match(r'Hello', text)
print(match.group())      # 'Hello'
match = re.match(r'World', text)
print(match)              # None -- 'World' is not at the start

# re.findall() -- find ALL non-overlapping matches
print(re.findall(r'hello', text, re.IGNORECASE))
# ['Hello', 'hello', 'HELLO']

# re.finditer() -- iterator of match objects (use for large texts)
for m in re.finditer(r'hello', text, re.IGNORECASE):
    print(f"Found '{m.group()}' at position {m.start()}-{m.end()}")
# Found 'Hello' at position 0-5
# Found 'hello' at position 13-18
# Found 'HELLO' at position 27-32

# re.sub() -- search and replace
print(re.sub(r'hello', 'HI', text, flags=re.IGNORECASE))
# 'HI World, HI python, HI REGEX'

# re.sub() with a function -- dynamic replacements
def censor(match):
    return '*' * len(match.group())

print(re.sub(r'\b\w{5,}\b', censor, "The quick brown fox"))
# 'The ***** ***** fox'  -- censors words with 5+ characters

# re.split() -- split by pattern (more powerful than str.split)
text = "one,two; three  four\tfive"
print(re.split(r'[,;\s]+', text))
# ['one', 'two', 'three', 'four', 'five']

What Do You Think?

What is the difference between re.match() and re.search()?

#Advanced Patterns

#Greedy vs Lazy Matching

By default, quantifiers are greedy -- they match as much text as possible. Adding ? after a quantifier makes it lazy -- it matches as little as possible:

pythonreference · read-only

html = "<b>bold</b> and <i>italic</i>"

# Greedy: .* matches as MUCH as possible
print(re.findall(r'<.*>', html))
# ['<b>bold</b> and <i>italic</i>']  -- matched from first < to LAST >

# Lazy: .*? matches as LITTLE as possible
print(re.findall(r'<.*?>', html))
# ['<b>', '</b>', '<i>', '</i>']  -- matched each tag individually

# Another example: extracting quoted strings
text = 'She said "hello" and he said "goodbye"'

# Greedy: captures from first " to last "
print(re.findall(r'".*"', text))
# ['"hello" and he said "goodbye"']

# Lazy: captures each quoted string
print(re.findall(r'".*?"', text))
# ['"hello"', '"goodbye"']

#Lookahead and Lookbehind Assertions

Lookaheads and lookbehinds check for context without including it in the match:

pythonreference · read-only

# (?=...) -- positive lookahead: "followed by"
# (?!...) -- negative lookahead: "NOT followed by"
# (?<=...) -- positive lookbehind: "preceded by"
# (?<!...) -- negative lookbehind: "NOT preceded by"

text = "100 USD, 200 EUR, 300 GBP, 400 USD"

# Find amounts followed by USD (without capturing USD)
print(re.findall(r'\d+(?= USD)', text))
# ['100', '400'] -- gets the numbers, not "USD"

# Find amounts NOT followed by USD
print(re.findall(r'\d+(?! USD)', text))
# ['10', '200', '300', '40']
# Note: '10' and '40' appear because regex finds partial matches
# Better version with word boundary:
print(re.findall(r'\b\d+\b(?! USD)', text))
# ['200', '300'] -- only complete numbers not followed by USD

# Lookbehind: find amounts preceded by $
prices = "Items: $50, $100, 200, $75"
print(re.findall(r'(?<=\$)\d+', prices))
# ['50', '100', '75'] -- gets numbers after $, without the $

# Negative lookbehind: find numbers NOT preceded by $
print(re.findall(r'(?<!\$)\b\d+\b', prices))
# ['200'] -- only the number without a $

#Practical Lookahead: Password Validation

pythonreference · read-only

def validate_password(password: str) -> bool:
    """
    Password must have:
    - At least 8 characters
    - At least one uppercase letter
    - At least one lowercase letter
    - At least one digit
    - At least one special character
    """
    pattern = r'^(?=.*[A-Z])(?=.*[a-z])(?=.*\d)(?=.*[!@#$%^&*]).{8,}$'
    #           (?=.*[A-Z])  -- lookahead: has uppercase somewhere
    #           (?=.*[a-z])  -- lookahead: has lowercase somewhere
    #           (?=.*\d)     -- lookahead: has digit somewhere
    #           (?=.*[!@#$%^&*]) -- lookahead: has special char somewhere
    #           .{8,}        -- actual match: 8+ of any characters
    return bool(re.match(pattern, password))

print(validate_password("Abc12345!"))   # True
print(validate_password("abc12345"))    # False -- no uppercase or special
print(validate_password("SHORT1!"))     # False -- less than 8 chars

#re.compile() for Performance

When you use the same pattern multiple times, compile it first:

pythonrunnable cell

import re
import time

# Approach 1: Compile once, use many times (fast)
email_pattern = re.compile(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$')

emails = [
    "alice@example.com",
    "bob@company.co.uk",
    "invalid@",
    "no-at-sign.com",
    "valid.email+tag@gmail.com",
    "bad email@space.com",
    "good_one@domain.org",
]

# Compiled pattern -- call .match(), .search(), .findall() on the pattern object
valid = [e for e in emails if email_pattern.match(e)]
print(f"Valid emails: {valid}")
# ['alice@example.com', 'bob@company.co.uk', 'valid.email+tag@gmail.com', 'good_one@domain.org']

# Approach 2: Using re.match() each time (slower -- recompiles pattern each call)
# Python does cache recently used patterns, but re.compile() is explicit and clear
valid2 = [e for e in emails if re.match(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', e)]

pythonreference · read-only

# Compiled patterns are especially useful with multiple patterns
log_patterns = {
    'error': re.compile(r'ERROR\s+(.+)'),
    'warning': re.compile(r'WARNING\s+(.+)'),
    'timestamp': re.compile(r'(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})'),
    'ip_address': re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})'),
}

def parse_log_line(line: str) -> dict:
    """Parse a log line into structured data."""
    result = {}
    for name, pattern in log_patterns.items():
        match = pattern.search(line)
        if match:
            result[name] = match.group(1)
    return result

log_line = "2024-01-15 14:30:22 ERROR Connection refused from 192.168.1.100"
print(parse_log_line(log_line))
# {'error': 'Connection refused from 192.168.1.100',
#  'timestamp': '2024-01-15 14:30:22',
#  'ip_address': '192.168.1.100'}

Intuition

Build Your Intuition

Regex is a tiny programming language -- and like any language, readability matters.

A regex like (?:(?<=\s)|^)(?:[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)(?=\s|[,.]|$) is technically correct, but nearly unreadable. Python provides re.VERBOSE (or re.X) mode, which lets you write regex with comments and whitespace:

pythonreference · read-only

# Unreadable:
pattern = r'(?:(?<=\s)|^)(?:[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)(?=\s|[,.]|$)'

# Readable (re.VERBOSE lets you add comments and whitespace):
pattern = re.compile(r'''
    (?:(?<=\s)|^)          # preceded by whitespace or start of string
    (?:                    # proper noun group (non-capturing)
        [A-Z][a-z]+        # capitalized word
        (?:\s+[A-Z][a-z]+)*  # optionally followed by more capitalized words
    )
    (?=\s|[,.]|$)         # followed by whitespace, punctuation, or end
''', re.VERBOSE)

For any pattern longer than ~30 characters, use re.VERBOSE. Your future self will thank you. This is the same principle as writing readable code: working code that nobody can understand is a liability.

#Flags (Modifiers)

pythonrunnable cell

import re

# re.IGNORECASE (or re.I) -- case-insensitive matching
print(re.findall(r'python', 'Python PYTHON python', re.IGNORECASE))
# ['Python', 'PYTHON', 'python']

# re.MULTILINE (or re.M) -- ^ and $ match at line boundaries
text = "error: disk full\nwarning: low memory\nerror: timeout"
print(re.findall(r'^error: .+', text, re.MULTILINE))
# ['error: disk full', 'error: timeout']

# re.DOTALL (or re.S) -- . matches newline too
html = "<div>\n  <p>Hello</p>\n</div>"
print(re.findall(r'<div>.*</div>', html))           # [] -- . doesn't match \n
print(re.findall(r'<div>.*</div>', html, re.DOTALL)) # ['<div>\n  <p>Hello</p>\n</div>']

# Combine flags with | (bitwise OR)
print(re.findall(r'^error', text, re.IGNORECASE | re.MULTILINE))
# ['error', 'error']

#Real-World Applications

#Application 1: Email Validation

pythonrunnable cell

import re

def validate_email(email: str) -> bool:
    """
    Validate an email address.
    
    This is a simplified pattern. RFC 5322 compliant email regex
    is thousands of characters long -- in production, use a library
    like email-validator instead.
    """
    pattern = re.compile(r'''
        ^                       # start of string
        [a-zA-Z0-9]             # must start with alphanumeric
        [a-zA-Z0-9._%+-]*      # followed by valid chars
        @                       # the @ symbol
        [a-zA-Z0-9]             # domain starts with alphanumeric
        [a-zA-Z0-9.-]*         # domain body
        \.                      # dot before TLD
        [a-zA-Z]{2,}           # TLD: at least 2 letters
        $                       # end of string
    ''', re.VERBOSE)
    return bool(pattern.match(email))

# Test cases
test_emails = [
    ("alice@example.com", True),
    ("bob.smith+tag@company.co.uk", True),
    ("user@domain.org", True),
    ("@missing-local.com", False),
    ("no-at-sign.com", False),
    ("spaces in@email.com", False),
    ("missing@tld.", False),
    (".starts-with-dot@example.com", False),
]

for email, expected in test_emails:
    result = validate_email(email)
    status = "PASS" if result == expected else "FAIL"
    print(f"  {status}: {email!r:40s} -> {result}")

#Application 2: Phone Number Extraction

pythonreference · read-only

def extract_phone_numbers(text: str) -> list[dict]:
    """
    Extract phone numbers in various US formats.
    Returns structured data with area code, exchange, and number.
    """
    pattern = re.compile(r'''
        (?:                       # optional country code
            \+?1[\s.-]?           # +1 or 1
        )?
        \(?                       # optional opening paren
        (?P<area>\d{3})           # area code (3 digits)
        \)?                       # optional closing paren
        [\s.-]?                   # optional separator
        (?P<exchange>\d{3})       # exchange (3 digits)
        [\s.-]?                   # optional separator
        (?P<number>\d{4})         # subscriber number (4 digits)
    ''', re.VERBOSE)
    
    results = []
    for match in pattern.finditer(text):
        results.append({
            'full': match.group(0).strip(),
            'area_code': match.group('area'),
            'exchange': match.group('exchange'),
            'number': match.group('number'),
        })
    return results

text = """
Contact us:
  Main office: (555) 123-4567
  Support: 555.987.6543
  Toll-free: +1-800-555-0199
  Direct: 5551234567
"""

for phone in extract_phone_numbers(text):
    print(f"  {phone['full']:20s} -> ({phone['area_code']}) {phone['exchange']}-{phone['number']}")
# (555) 123-4567       -> (555) 123-4567
# 555.987.6543         -> (555) 987-6543
# +1-800-555-0199      -> (800) 555-0199
# 5551234567           -> (555) 123-4567

#Application 3: Log Parsing

pythonreference · read-only

def parse_web_logs(log_text: str) -> list[dict]:
    """
    Parse Apache-style web server log entries.
    
    Format: IP - - [timestamp] "METHOD /path HTTP/x.x" status bytes
    """
    pattern = re.compile(r'''
        (?P<ip>\d{1,3}(?:\.\d{1,3}){3})    # IP address
        \s+-\s+-\s+                          # two dashes
        \[(?P<timestamp>[^\]]+)\]            # timestamp in brackets
        \s+"                                 # opening quote
        (?P<method>\w+)                      # HTTP method
        \s+(?P<path>\S+)                     # request path
        \s+HTTP/[\d.]+"                      # HTTP version
        \s+(?P<status>\d{3})                 # status code
        \s+(?P<bytes>\d+|-)                  # response bytes
    ''', re.VERBOSE)
    
    entries = []
    for match in pattern.finditer(log_text):
        entries.append(match.groupdict())
    return entries

logs = """192.168.1.100 - - [15/Jan/2024:14:30:22 +0000] "GET /api/users HTTP/1.1" 200 1234
10.0.0.50 - - [15/Jan/2024:14:30:23 +0000] "POST /api/login HTTP/1.1" 401 89
192.168.1.100 - - [15/Jan/2024:14:30:24 +0000] "GET /static/style.css HTTP/1.1" 304 0"""

for entry in parse_web_logs(logs):
    print(f"  {entry['ip']:16s} {entry['method']:5s} {entry['path']:25s} -> {entry['status']}")
# 192.168.1.100    GET   /api/users                -> 200
# 10.0.0.50        POST  /api/login                -> 401
# 192.168.1.100    GET   /static/style.css         -> 304

#Application 4: NLP -- Sentence Splitting and Tokenization

pythonreference · read-only

def split_sentences(text: str) -> list[str]:
    """
    Split text into sentences.

    Handles common abbreviations (Mr., Dr., Prof.) by protecting them
    before splitting. This is a simplified version -- production NLP
    uses spaCy or NLTK.

    Why not use a lookbehind with alternation?
    Python's `re` module requires fixed-width lookbehinds, so
    `(?<!(?:Mr|Mrs|Prof))` raises an error. The protect/restore trick
    below sidesteps that limitation cleanly.
    """
    abbreviations = ['Mr', 'Mrs', 'Ms', 'Dr', 'Prof', 'Jr', 'Sr', 'vs', 'etc']
    sentinel = ''  # an unlikely character

    # 1. Protect "Mr.", "Dr.", etc. by replacing the period with a sentinel
    protected = text
    for abbr in abbreviations:
        protected = re.sub(rf'\b{abbr}\.', f'{abbr}{sentinel}', protected)

    # 2. Split on sentence-ending punctuation followed by whitespace + uppercase
    parts = re.split(r'(?<=[.!?])\s+(?=[A-Z])', protected)

    # 3. Restore the sentinel back to a period
    return [p.replace(sentinel, '.') for p in parts]

text = """Dr. Smith went to Washington. He met with Prof. Jones at 3:30 p.m. They discussed the U.S. economy. Was it productive? Yes! The meeting went well."""

for i, sentence in enumerate(split_sentences(text), 1):
    print(f"  {i}. {sentence}")


def tokenize_simple(text: str) -> list[str]:
    """
    Simple regex tokenizer for NLP preprocessing.
    
    Splits on whitespace and punctuation while keeping
    contractions and hyphenated words together.
    """
    pattern = r"""
        \w+(?:'\w+)*     |  # words with optional contractions (don't, it's)
        \$?\d+(?:\.\d+)? |  # numbers, optionally with decimal and dollar sign
        [^\w\s]              # individual punctuation characters
    """
    return re.findall(pattern, text, re.VERBOSE)

sample = "I can't believe it's only $9.99! That's a 50% discount."
tokens = tokenize_simple(sample)
print(f"\nTokens: {tokens}")
# ["I", "can't", "believe", "it's", "only", "$9.99", "!", "That's", "a", "50", "%", "discount", "."]

#Application 5: HTML Tag Stripping

pythonreference · read-only

def strip_html_tags(html: str) -> str:
    """
    Remove HTML tags from text, preserving content.
    
    For production HTML parsing, use BeautifulSoup or lxml.
    Regex cannot correctly parse nested/malformed HTML (see DeepDive below),
    but it works well for simple cases and data cleaning.
    """
    # Remove HTML tags
    text = re.sub(r'<[^>]+>', '', html)
    # Collapse multiple whitespace into single space
    text = re.sub(r'\s+', ' ', text)
    # Decode common HTML entities
    entities = {'&amp;': '&', '&lt;': '<', '&gt;': '>', '&nbsp;': ' ', '&quot;': '"'}
    for entity, char in entities.items():
        text = text.replace(entity, char)
    return text.strip()

html = """
<div class="article">
  <h1>Machine Learning &amp; AI</h1>
  <p>This is a <b>bold</b> statement about <i>deep learning</i>.</p>
  <ul>
    <li>Neural networks</li>
    <li>Transformers</li>
  </ul>
</div>
"""

print(strip_html_tags(html))
# 'Machine Learning & AI This is a bold statement about deep learning. Neural networks Transformers'

Watch Out

Watch out

Fix: Use a proper HTML parser like BeautifulSoup or lxml. Regex cannot handle nested tags, malformed HTML, or context-dependent structures. For simple tag stripping or extracting from well-structured text, regex is fine.

Using regex to fully parse HTML or XML documents

HTML is not a regular language -- it has recursive nesting that regex fundamentally cannot handle. Consider: <div><div>inner</div></div>. A regex trying to match the outer div with <div>.*?</div> would match <div><div>inner</div> -- stopping at the first closing tag, not the correct one.

Additionally, catastrophic backtracking (ReDoS) is a real security concern. A pattern like <.*>.*<.*> on maliciously crafted input can take exponential time, freezing your application. This is not theoretical -- ReDoS attacks have taken down production services at major companies.

The rule of thumb: use regex for flat text patterns (phone numbers, emails, log lines, simple extraction). Use a parser for structured documents (HTML, XML, JSON). And NEVER use user-supplied regex patterns without timeout protection.

Watch Out

Watch out

Fix: Always prefix regex pattern strings with r. Write r'\\\\d+' not '\\\\d+'. Inside a raw string, \\\\d is two characters (backslash + d); without it, Python converts \\\\d to a tab-like escape which the regex engine does not understand as 'digit'.

Forgetting to use raw strings (r prefix) for regex patterns, causing backslashes to be interpreted by Python before the regex engine sees them.

Python resolves string escapes before the regex engine ever sees the pattern. For escapes Python does not recognise, such as \d and \w, nothing changes: '\d' is still two characters, so re.findall('\d+', "abc 123") returns ['123'] exactly like the raw version. Since Python 3.12 it does raise SyntaxWarning: invalid escape sequence, and that warning is scheduled to become a SyntaxError. The escapes that genuinely break are the ones Python does recognise -- \b above all, because it means "word boundary" to regex but "backspace" to Python.

pythonrunnable cell

import re

# WRONG: Python turns \b into a backspace character before regex sees it
print(re.findall('\bcat\b', "the cat sat"))   # []

# RIGHT: the raw string preserves both backslashes
print(re.findall(r'\bcat\b', "the cat sat"))  # ['cat']

# \d survives either way -- but the non-raw form now emits a SyntaxWarning,
# so prefix every pattern with r and stop having to think about which is which.
print(re.findall(r'\d+', "abc 123"))           # ['123']

The same applies to \s (whitespace), \w (word char), \b (word boundary), \A (string start), etc. Make r'' your default habit for ALL regex patterns.

#Code Playground

pythonplayground.py · Pyodide

import re

# ══════════════════════════════════════════════════════════════════════════════
# REGEX CHALLENGES -- Test your pattern-matching skills
# ══════════════════════════════════════════════════════════════════════════════

# ── Challenge 1: Parse CSV Without the csv Module ────────────────────────────
# Parse this CSV text into a list of dictionaries.
# Handle quoted fields that may contain commas.

csv_text = """name,age,city
Alice,30,"New York, NY"
Bob,25,Chicago
"Carol, Jr.",35,"San Francisco, CA"
"""

def parse_csv(text: str) -> list[dict]:
  """Parse CSV text into a list of dictionaries."""
  lines = text.strip().split('\n')
  headers = re.findall(r'"([^"]*?)"|([^,]+)', lines[0])
  headers = [h[0] or h[1] for h in headers]
  
  rows = []
  for line in lines[1:]:
      # TODO: Use regex to extract fields (handle quoted fields with commas)
      # Pattern hint: match either "quoted,field" or unquoted_field
      fields = []  # Your regex here
      if fields:
          rows.append(dict(zip(headers, fields)))
  return rows

# Test it:
# result = parse_csv(csv_text)
# for row in result:
#     print(row)
# Expected:
# {'name': 'Alice', 'age': '30', 'city': 'New York, NY'}
# {'name': 'Bob', 'age': '25', 'city': 'Chicago'}
# {'name': 'Carol, Jr.', 'age': '35', 'city': 'San Francisco, CA'}

# ── Challenge 2: Extract URLs From Text ──────────────────────────────────────
# Find all URLs (http and https) in this messy text.

messy_text = """
Check out https://example.com/path?q=hello&lang=en for info.
Also see http://blog.example.org/2024/regex-guide (great article).
Not a URL: ftp://files.example.com (we only want http/https).
Link: https://api.example.com/v2/users/123#section-2
Plain text with no URL here.
Another: http://localhost:8080/test
"""

def extract_urls(text: str) -> list[str]:
  """Extract all http/https URLs from text."""
  # TODO: Write a pattern that matches URLs starting with http:// or https://
  # URLs can contain: letters, digits, /, ?, &, =, -, _, ., #, :, %
  pattern = r''  # Your pattern here
  return re.findall(pattern, text)

# Test it:
# urls = extract_urls(messy_text)
# for url in urls:
#     print(f"  {url}")

# ── Challenge 3: Clean Messy Data ────────────────────────────────────────────
# Normalize this messy user input data.

messy_records = [
  "  John   Smith  , age:  32 ,  email: john.smith@email.com  ",
  "Jane    DOE,    AGE: 28,EMAIL:   jane@company.org",
  "  bob jones ,age :45,   email:bob.jones@mail.co.uk   ",
]

def clean_record(record: str) -> dict:
  """
  Clean and parse a messy record string into structured data.
  
  Should handle:
  - Extra whitespace everywhere
  - Inconsistent casing
  - Inconsistent separators around colons
  """
  # TODO: Extract name, age, and email using regex
  # Hint: use named groups (?P<name>...)
  result = {}  # Your code here
  return result

# Test it:
# for record in messy_records:
#     cleaned = clean_record(record)
#     print(cleaned)
# Expected:
# {'name': 'John Smith', 'age': 32, 'email': 'john.smith@email.com'}
# {'name': 'Jane Doe', 'age': 28, 'email': 'jane@company.org'}
# {'name': 'Bob Jones', 'age': 45, 'email': 'bob.jones@mail.co.uk'}

Tests · Test CSV parsing with quoted fields containing commas. Test URL extraction excludes ftp:// URLs. Test data cleaning normalizes casing and whitespace.

#Quick Reference -- Regex Cheat Sheet

Here is a compact reference for the most common regex syntax. Bookmark this section.

pythonrunnable cell

# ╔══════════════════════════════════════════════════════════════════╗
# ║  REGEX QUICK REFERENCE                                         ║
# ╠══════════════════════════════════════════════════════════════════╣
# ║                                                                ║
# ║  CHARACTER CLASSES                                              ║
# ║  .        any character (except \n)                            ║
# ║  \d       digit [0-9]                                          ║
# ║  \D       non-digit [^0-9]                                     ║
# ║  \w       word char [a-zA-Z0-9_]                               ║
# ║  \W       non-word [^a-zA-Z0-9_]                               ║
# ║  \s       whitespace [ \t\n\r\f\v]                             ║
# ║  \S       non-whitespace                                       ║
# ║  [abc]    any of a, b, c                                       ║
# ║  [^abc]   none of a, b, c                                      ║
# ║  [a-z]    range a to z                                         ║
# ║                                                                ║
# ║  QUANTIFIERS                                                    ║
# ║  *        0 or more (greedy)                                   ║
# ║  +        1 or more (greedy)                                   ║
# ║  ?        0 or 1 (optional)                                    ║
# ║  {n}      exactly n                                            ║
# ║  {n,}     n or more                                            ║
# ║  {n,m}    between n and m                                      ║
# ║  *?  +?   lazy versions (match as little as possible)          ║
# ║                                                                ║
# ║  ANCHORS                                                        ║
# ║  ^        start of string/line                                 ║
# ║  $        end of string/line                                   ║
# ║  \b       word boundary                                        ║
# ║                                                                ║
# ║  GROUPS                                                         ║
# ║  (...)           capturing group                               ║
# ║  (?P<name>...)   named group                                   ║
# ║  (?:...)         non-capturing group                           ║
# ║  \1, \2          backreference to group 1, 2                   ║
# ║                                                                ║
# ║  LOOKAROUND                                                     ║
# ║  (?=...)   positive lookahead                                  ║
# ║  (?!...)   negative lookahead                                  ║
# ║  (?<=...)  positive lookbehind                                 ║
# ║  (?<!...)  negative lookbehind                                 ║
# ║                                                                ║
# ║  FLAGS                                                          ║
# ║  re.IGNORECASE (re.I)    case-insensitive                     ║
# ║  re.MULTILINE (re.M)     ^ $ match line boundaries            ║
# ║  re.DOTALL (re.S)        . matches \n                         ║
# ║  re.VERBOSE (re.X)       allow comments and whitespace        ║
# ║                                                                ║
# ║  re MODULE FUNCTIONS                                            ║
# ║  re.search(pat, text)    first match anywhere                  ║
# ║  re.match(pat, text)     match at start only                   ║
# ║  re.fullmatch(pat, text) match entire string                   ║
# ║  re.findall(pat, text)   all non-overlapping matches           ║
# ║  re.finditer(pat, text)  iterator of match objects             ║
# ║  re.sub(pat, repl, text) search and replace                    ║
# ║  re.split(pat, text)     split by pattern                      ║
# ║  re.compile(pat)         compile for reuse                     ║
# ╚══════════════════════════════════════════════════════════════════╝

Interactive Lab

See how text gets split into tokens — the same pattern-matching logic that powers regex

Text Tokenization

#Key Takeaways

Regex describes shapes of text, not exact strings -- use regex when str.find() and str.replace() cannot express the pattern you need. Character classes (\d, \w, [a-z]), quantifiers (+, *, {n,m}), and anchors (^, $, \b) are the fundamental building blocks
Capturing groups extract structured data from unstructured text -- use () to capture parts of a match, (?P<name>...) for readable named groups, and (?:...) when you need grouping without capturing. re.finditer() with named groups is the professional way to parse text
-- this distinction trips up beginners constantly. When in doubt, use . Use for validation (must match the entire string)

Quick Check1 / 5

What does the regex pattern r'\b\w+ing\b' match?

Regular expressions are one of the most practical tools in a programmer's toolkit. The patterns you learned here -- character classes, groups, lookaheads, and the re module API -- appear in every data pipeline, every log parser, and every NLP preprocessing step. Master them, and messy text data stops being a problem. Next up: explore Python's data model and the dunder methods that power the language's most elegant syntax.

re.search()

re.fullmatch()

Your Reflection

Continue Learning

Discussion

Regular Expressions: Pattern Matching Mastery

After this lesson, you will be able to:

Before You Start

#Why Regex?

#The Limits of Plain String Methods

#Enter re -- Python's Regex Module

#Basic Patterns

#Literal Characters

#Special Characters (Metacharacters)

#Character Classes

#Shorthand Character Classes

#Quantifiers: How Many?

#Anchors: Where to Match

#Groups and Capturing

#Capturing Groups with ()

#Named Groups with (?P<name>...)

#Non-Capturing Groups with (?:...)

#Backreferences

#The re Module API

#Advanced Patterns

#Greedy vs Lazy Matching

#Lookahead and Lookbehind Assertions

#Practical Lookahead: Password Validation

#re.compile() for Performance

#Flags (Modifiers)

#Real-World Applications

#Application 1: Email Validation

#Application 2: Phone Number Extraction

#Application 3: Log Parsing

#Application 4: NLP -- Sentence Splitting and Tokenization

#Application 5: HTML Tag Stripping

Deep Dive

#Code Playground

#Quick Reference -- Regex Cheat Sheet

#Key Takeaways