Python Playwright Web Automation Guide

If you’re looking to supercharge your web testing or scraping projects with a modern, reliable tool, Python Playwright is the answer. Built by the creators of Microsoft Edge, Playwright offers cross‑browser automation that’s faster, more stable, and easier to maintain than many legacy solutions. In this guide, we’ll walk you through everything you need to know—from installation to advanced techniques—so you can start automating the web with confidence.

What Is Playwright and Why Choose It for Python?

Playwright is an open‑source automation library that supports Chromium, Firefox, and WebKit with a single API. Its key advantages include:

• Cross‑browser consistency: Write one script and run it on Chrome, Edge, Safari, and Firefox.
• Auto‑wait mechanisms: No more flaky time.sleep() calls; Playwright waits for elements to become actionable.
• Built‑in network interception: Mock responses, block ads, or capture traffic for debugging.
• Headless and headed modes: Perfect for CI pipelines and local debugging alike.

Because it’s designed for modern web apps, Playwright handles single‑page applications (SPAs), shadow DOM, and iframes out of the box—making it a top choice for Python developers.

Getting Started: Installation and Project Setup

1. Install Playwright with pip

pip install playwright
# Install the browsers (Chromium, Firefox, WebKit)
python -m playwright install

This single command downloads the necessary browser binaries, ensuring version compatibility.

2. Create a Minimal Project Structure

• tests/ – Store all automation scripts.
• requirements.txt – List dependencies (e.g., playwright, pytest).
• conftest.py – Optional fixtures for shared setup/teardown.

3. Verify the Installation

from playwright.sync_api import sync_playwright

def run():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()
        page.goto("https://example.com")
        print(page.title())
        browser.close()

if __name__ == "__main__":
    run()

Running this script should open a visible browser, navigate to example.com, print “Example Domain,” and close gracefully.

Core Concepts and Basic Automation

Launching Browsers and Contexts

A browser represents the executable (Chromium, Firefox, WebKit). A browser context isolates cookies, storage, and permissions, allowing parallel sessions without interference.

with sync_playwright() as p:
    # Launch Chromium in headless mode
    browser = p.chromium.launch(headless=True)

    # Create two independent contexts
    context1 = browser.new_context()
    context2 = browser.new_context()

Page Interaction Basics

• page.goto(url) – Navigate to a page.
• page.click(selector) – Click an element.
• page.fill(selector, text) – Type into input fields.
• page.wait_for_selector(selector) – Explicit wait for an element.
• page.screenshot(path="shot.png") – Capture a screenshot.

Sample Login Flow

def login(page, username, password):
    page.goto("https://demo-site.com/login")
    page.fill("#email", username)
    page.fill("#password", password)
    page.click("button[type='submit']")
    page.wait_for_selector("text=Dashboard")

Advanced Playwright Features

Network Interception and Mocking

Intercepting requests lets you test edge cases without relying on external services.

def mock_api(route, request):
    if "api/v1/data" in request.url:
        route.fulfill(
            status=200,
            content_type="application/json",
            body='{"mocked": true}'
        )
    else:
        route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    page = context.new_page()
    page.route("**/*", mock_api)
    page.goto("https://myapp.com")

Handling Dialogs, Frames, and Shadow DOM

• Dialogs: page.on("dialog", lambda d: d.accept())
• Frames: frame = page.frame(name="my-iframe")
• Shadow DOM: Use page.locator("css=shadow-host >> css=inner-element")

Taking Screenshots and Recording Videos

These artifacts are invaluable for debugging CI failures.

# Screenshot of a specific element
page.locator("#profile-pic").screenshot(path="profile.png")

# Video recording for an entire test
context = browser.new_context(record_video_dir="videos/")
page = context.new_page()
page.goto("https://example.com")
page.wait_for_timeout(2000)  # Let the page settle
page.close()
# Retrieve video path
video_path = page.video.path()

Parallel Execution with Pytest

Combine Playwright with pytest and pytest-xdist for scalable test suites.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="session")
def browser():
    with sync_playwright() as p:
        browser = p.chromium.launch()
        yield browser
        browser.close()

# test_login.py
def test_successful_login(browser):
    context = browser.new_context()
    page = context.new_page()
    # Reuse the login helper from earlier
    login(page, "user@example.com", "Secret123")
    assert page.is_visible("text=Welcome")

Run with pytest -n auto to distribute tests across CPU cores.

Integrating Playwright into CI/CD Pipelines

Most CI platforms (GitHub Actions, GitLab CI, Azure Pipelines) support Docker, making Playwright integration straightforward.

GitHub Actions Example

name: Playwright Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          python -m playwright install --with-deps
      - name: Run tests
        run: pytest -n auto --headless

Best Practices for Reliable Automation

• Prefer locators over raw selectors: page.get_by_role("button", name="Submit") is more resilient to UI changes.
• Use explicit waits: Rely on page.wait_for_selector or built‑in auto‑wait instead of fixed sleeps.
• Isolate tests with fresh contexts: Prevent state leakage between scenarios.
• Store credentials securely: Use environment variables or secret managers, never hard‑code passwords.
• Capture artifacts on failure: Screenshots, logs, and videos help pinpoint flaky behavior.

Common Pitfalls and How to Avoid Them

1. Ignoring Browser Version Mismatches

Playwright bundles its own browsers, but if you manually install a system browser, version drift can cause unexpected errors. Stick to the bundled binaries unless you have a specific need.

2. Overusing time.sleep()

Static sleeps increase test runtime and cause flakiness. Replace them with page.wait_for_load_state() or element‑specific waits.

3. Not Cleaning Up Resources

Always close pages, contexts, and browsers. Using with sync_playwright() as p: ensures proper teardown even when exceptions occur.

4. Missing Headless Configuration in CI

CI environments often lack a display server. Explicitly launch browsers in headless mode or use playwright install-deps to add required libraries.

Where to Learn More

• Official documentation: playwright.dev/python
• Playwright GitHub repository for community examples.
• Video tutorials on YouTube (search “Python Playwright tutorial”).
• Open‑source projects on GitHub that showcase real‑world test suites.

Conclusion

Python Playwright combines the power of modern browsers with a clean, Pythonic API, making it an ideal choice for anyone