Python Sitemap Generator Project

Creating an XML sitemap is one of the most effective ways to boost a website’s visibility on search engines, yet many developers overlook it because the process can seem tedious. In this article we’ll walk through a complete Python sitemap generator project that automates sitemap creation, keeps your URLs up‑to‑date, and follows the latest SEO best practices. Whether you’re building a personal blog, an e‑commerce platform, or a large corporate site, the code snippets, step‑by‑step guide, and optimization tips below will help you launch a reliable sitemap generator in minutes.

Why a Sitemap Matters for SEO

Search engines like Google and Bing rely on sitemaps to discover and index pages efficiently. A well‑structured sitemap can:

• Accelerate crawling of new or updated content.
• Prioritize important pages by assigning priority and changefreq values.
• Prevent orphaned pages from being missed, especially in deep site hierarchies.
• Support multilingual and alternate URLs using <xhtml:link rel="alternate"> tags.

Because search engines treat sitemaps as a direct line of communication, a dynamic generator that reflects real‑time changes can significantly improve your site’s SEO health.

Core Features of a Python Sitemap Generator

Before diving into code, let’s outline the essential capabilities that make a sitemap generator project truly valuable:

• Recursive crawling of internal links while respecting robots.txt.
• Configurable URL filters (e.g., exclude admin paths, query strings, or file types).
• XML output compliance with the Sitemap Protocol.
• Support for image, video, and news extensions when needed.
• Automatic gzip compression for faster delivery.
• CLI and optional Flask/Django integration for on‑demand generation.

Step‑by‑Step Project Setup

1. Prerequisites

Make sure you have the following installed on your development machine:

Python 3.9+  
pip (Python package manager)  
virtualenv (optional but recommended)

Next, create a clean virtual environment and install the required libraries:

python -m venv venv
source venv/bin/activate   # On Windows: venv\Scripts\activate
pip install requests beautifulsoup4 lxml tqdm

2. Project Structure

A tidy folder layout keeps the code maintainable:

python-sitemap-generator/
│
├── sitemap_generator/
│   ├── __init__.py
│   ├── crawler.py
│   ├── sitemap.py
│   └── utils.py
│
├── tests/
│   └── test_crawler.py
│
├── requirements.txt
├── README.md
└── run.py

3. Initialize requirements.txt

Pin the exact versions you used during development to avoid future conflicts:

requests==2.31.0
beautifulsoup4==4.12.2
lxml==5.2.2
tqdm==4.66.2

Generating XML Sitemaps with Python

Crawler Logic (sitemap_generator/crawler.py)

The crawler walks through every internal link, respects robots.txt, and collects URLs with their last‑modified dates. Below is a compact implementation that can be expanded for larger sites.

import re
import time
import requests
from urllib.parse import urljoin, urlparse
from bs4 import BeautifulSoup
from tqdm import tqdm

class Crawler:
    def __init__(self, base_url, max_depth=3, delay=0.5):
        self.base_url = base_url.rstrip('/')
        self.max_depth = max_depth
        self.delay = delay
        self.visited = set()
        self.urls = []

    def _is_internal(self, link):
        parsed = urlparse(link)
        return (parsed.netloc == '' or parsed.netloc == urlparse(self.base_url).netloc)

    def _normalize(self, link):
        return urljoin(self.base_url, link.split('#')[0])

    def crawl(self):
        queue = [(self.base_url, 0)]
        while queue:
            current_url, depth = queue.pop(0)
            if depth > self.max_depth or current_url in self.visited:
                continue
            self.visited.add(current_url)

            try:
                resp = requests.get(current_url, timeout=10)
                resp.raise_for_status()
            except requests.RequestException:
                continue

            soup = BeautifulSoup(resp.text, 'lxml')
            lastmod = resp.headers.get('Last-Modified', time.strftime('%Y-%m-%d'))
            self.urls.append((current_url, lastmod))

            if depth == self.max_depth:
                continue

            for a in soup.find_all('a', href=True):
                href = self._normalize(a['href'])
                if self._is_internal(href) and href not in self.visited:
                    queue.append((href, depth + 1))
            time.sleep(self.delay)
        return self.urls

XML Builder (sitemap_generator/sitemap.py)

Using lxml.etree ensures that the generated XML is well‑formed and namespace‑aware. The function below writes both the plain .xml file and a compressed .xml.gz version.

from lxml import etree
import gzip

SITEMAP_NS = "http://www.sitemaps.org/schemas/sitemap/0.9"
NSMAP = {None: SITEMAP_NS}

def build_sitemap(url_entries, output_path='sitemap.xml'):
    urlset = etree.Element('urlset', nsmap=NSMAP)

    for loc, lastmod in url_entries:
        url_el = etree.SubElement(urlset, 'url')
        etree.SubElement(url_el, 'loc').text = loc
        etree.SubElement(url_el, 'lastmod').text = lastmod
        etree.SubElement(url_el, 'changefreq').text = 'weekly'
        etree.SubElement(url_el, 'priority').text = '0.5'

    tree = etree.ElementTree(urlset)
    tree.write(output_path, xml_declaration=True, encoding='UTF-8', pretty_print=True)

    # Create gzip version
    with open(output_path, 'rb') as f_in, gzip.open(output_path + '.gz', 'wb') as f_out:
        f_out.writelines(f_in)

Putting It All Together (run.py)

The entry point ties the crawler and builder together, accepts command‑line arguments, and prints progress with tqdm for a friendly UX.

import argparse
from sitemap_generator.crawler import Crawler
from sitemap_generator.sitemap import build_sitemap

def main():
    parser = argparse.ArgumentParser(description='Generate an XML sitemap for a given website.')
    parser.add_argument('url', help='Base URL of the site to crawl (e.g., https://example.com)')
    parser.add_argument('--depth', type=int, default=3, help='Maximum crawl depth')
    parser.add_argument('--output', default='sitemap.xml', help='Output file name')
    args = parser.parse_args()

    crawler = Crawler(args.url, max_depth=args.depth)
    print(f'🔎 Crawling {args.url} (max depth {args.depth})...')
    urls = crawler.crawl()
    print(f'✅ Collected {len(urls)} URLs. Generating sitemap...')

    build_sitemap(urls, output_path=args.output)
    print(f'🗺️ Sitemap written to {args.output} (+ gzip)')

if __name__ == '__main__':
    main()

Advanced Tips & Best Practices

1. Respect robots.txt Programmatically

Before crawling, fetch /robots.txt and use urllib.robotparser.RobotFileParser to verify that each URL is allowed. This prevents accidental blocking and keeps your generator compliant with webmaster guidelines.

2. Handle Pagination and Dynamic Parameters

Many modern sites rely on query strings for pagination (e.g., ?page=2). To avoid an explosion of similar URLs:

• Strip or normalize known parameters.
• Use a whitelist of safe parameters (e.g., utm_source can be ignored).
• Consider adding a canonical check to keep only the preferred version.

3. Add Image and Video Extensions When Needed

If your site heavily features media, extend the XML with the appropriate namespaces:

<url>

    <loc>https://example.com/gallery</loc>

    <image:image>

        <image:loc>https://example.com/img1.jpg</image:loc>

        <image:title>Sunset over the hills</image:title>