ArchiveBox Logo
  • Contents
  • Overview
    • ArchiveBox Documentation
    • Key Features
    • 🤝 Professional Integration
    • Quickstart
      • ✳️  Easy Setup
      • 🛠  Package Manager Setup
      • 🎗  Other Options
      • ➡️  Next Steps
      • Usage
        • ⚡️  CLI Usage
        • ArchiveBox Subcommands
    • Overview
      • Input Formats: How to pass URLs into ArchiveBox for saving
      • Output Formats: What ArchiveBox saves for each URL
      • Configuration
      • Dependencies
      • Archive Layout
      • Static Archive Exporting
      • Caveats
        • Archiving Private Content
        • Security Risks of Viewing Archived JS
        • Working Around Sites that Block Archiving
        • Saving Multiple Snapshots of a Single URL
        • Storage Requirements
      • Screenshots
    • Background & Motivation
      • Comparison to Other Projects
      • Internet Archiving Ecosystem
    • Documentation
      • Getting Started
      • Advanced
      • Developers
      • More Info
    • ArchiveBox Development
      • Setup the dev environment
        • 1. Setup the monorepo
        • 2. Option A: Install the Python, JS, and system dependencies directly on your machine
        • 2. Option B: Build the docker container and use that for development instead
      • Common development tasks
        • Run in DEBUG mode
        • Install and run a specific GitHub branch
          • Use a Pre-Built Image
          • Build Branch from Source
        • Run the linters / tests
        • Make DB migrations, enter Django shell, other dev helper commands
        • Contributing a new extractor
        • Build the docs, pip package, and docker image
        • Roll a release
      • Further Reading
  • Getting Started
    • Quickstart
      • 1. Set up ArchiveBox
      • 2. Get your list of URLs to archive
      • 3. Add your URLs to the archive
      • ✅ Done!
    • Install
      • Supported Systems
      • Option A. Docker / Docker Compose Setup ⭐️
      • Option B. Automatic Setup Script
      • Option C. Bare Metal Setup
        • 1. Install base system dependencies needed for your OS
          • macOS
          • Ubuntu/Debian-based Systems
          • FreeBSD
          • OpenBSD
          • Arch Linux / Nix / Guix / etc. Other OSs
        • 2. Install the Python dependencies using pip
        • 3. Install the JS dependencies using archivebox setup
        • Troubleshooting
        • Next Steps: Add some URLs to archive and try out CLI / Web UI
        • Next Steps: Upgrading Archivebox to a new version
        • Further Reading
    • Docker
      • Overview
      • Docker Compose
        • Setup
        • Upgrading
        • Usage
        • Accessing the data
        • Configuration
      • Docker
        • Setup
        • Upgrading
        • Usage
        • Accessing the data
        • Configuration
    • Configuration
      • General Settings
        • ONLY_NEW
        • TIMEOUT
        • RESOLUTION
        • CHECK_SSL_VALIDITY
        • USER_AGENT
        • COOKIES_FILE
        • DEFAULT_PERSONA
        • URL_DENYLIST
        • URL_ALLOWLIST
        • TAG_SEPARATOR_PATTERN
        • CRAWL_MAX_URLS
        • CRAWL_MAX_SIZE
        • CRAWL_TIMEOUT
        • CRAWL_MAX_CONCURRENT_SNAPSHOTS
        • SNAPSHOT_MAX_SIZE
        • DELETE_AFTER
        • PERMISSIONS
        • PLUGINS
      • Server Settings
        • ADMIN_USERNAME / ADMIN_PASSWORD
        • PUBLIC_INDEX / PUBLIC_ADD_VIEW
        • SECRET_KEY
        • BIND_ADDR
        • BASE_URL
        • SERVER_SECURITY_MODE
        • SNAPSHOTS_PER_PAGE
        • FOOTER_INFO
        • CUSTOM_TEMPLATES_DIR
        • REVERSE_PROXY_USER_HEADER
        • REVERSE_PROXY_WHITELIST
        • LOGOUT_REDIRECT_URL
        • LDAP Settings
          • LDAP_ENABLED
          • LDAP_SERVER_URI
          • LDAP_BIND_DN
          • LDAP_BIND_PASSWORD
          • LDAP_USER_BASE
          • LDAP_USER_FILTER
          • LDAP_USERNAME_ATTR
          • LDAP_FIRSTNAME_ATTR
          • LDAP_LASTNAME_ATTR
          • LDAP_EMAIL_ATTR
          • LDAP_CREATE_SUPERUSER
      • Storage Settings
        • OUTPUT_PERMISSIONS
        • PUID / PGID
        • ENFORCE_ATOMIC_WRITES
        • TMP_DIR
        • LIB_DIR
        • ALLOW_NO_UNIX_SOCKETS
      • Database Settings
        • SQLITE_JOURNAL_MODE
        • SQLITE_MMAP_SIZE
        • SQLITE_BUSY_TIMEOUT
        • SQLITE_LOCK_RETRY_TIMEOUT
        • SQLITE_LOCK_RETRY_INTERVAL
      • Search Settings
        • SEARCH_BACKEND_ENGINE
      • Shell Options
        • DEBUG
        • USE_COLOR
        • SHOW_PROGRESS
      • Plugin Configuration
        • Shared core options that plugins fall back to
        • Listing & setting plugin options
        • Why is plugin config documented separately?
    • Security Overview
      • Web UI Permissions
      • ArchiveBox Use-Cases
        • Archiving Public Content Only ⭐️ [Default, recommended for most people]
        • Archiving Content Behind Log-Ins 🚨 [Advanced users only]
        • ⚠️ Things to watch out for: ⚠️
        • Publishing
      • Do not run as root
      • Output Folder
        • Database
        • Filesystem
          • Purging entries
          • Permissions
    • Usage
      • CLI Usage
        • Run ArchiveBox with configuration options
        • Import a single URL
        • Import a list of URLs from a text file
        • Import list of links from browser history
        • Import browser cookies into a persona
      • UI Usage
        • Explanation of buttons in the web UI - admin snapshots list
      • Browser Extension Usage
        • More Info
      • Disk Layout
        • Large Archives
      • SQL Shell Usage
      • Python Shell Usage
      • Python API Usage
  • Guides
    • Setting Up Storage
      • Supported Local Filesystems
        • EXT4 (default on Linux), APFS (default on macOS)
        • ZFS (recommended for best experience on Linux/BSD) ⭐️
        • NTFS, HFS+, BTRFS
        • EXT2, EXT3, FAT32, exFAT
      • Supported Remote Filesystems
        • NFS (Docker Driver)
        • SMB / Ceph (Docker CIFS Driver)
        • Amazon S3 / Backblaze B2 / Google Drive / etc. (RClone)
          • RClone Config Examples
          • Option A: Running RClone on Bare Metal host
          • Option B: Running RClone with Docker Storage Plugin
        • More Docker Storage Plugins
    • Setting Up Authentication
      • Set Up Admin Web UI Permissions
      • Admin Web UI Authentication Methods
        • Username & Password (the default)
        • Reverse Proxy Authentication
        • LDAP Authentication
        • Not Yet Supported: SAML / OAuth2 / OpenID Authentication
      • REST API
        • API Bearer Token Authentication
        • API Request Header Authentication
        • API Query Parameter Authentication
        • API Session Cookie Authentication
        • API HTTP Basic Authentication
          • Further Reading
    • Setting Up Search
      • How to Search in ArchiveBox
      • How Search Works
      • ArchiveBox Search Backends
        • ripgrep (the default)
          • Pros
          • Cons
        • ripgrep-all (aka rga)
        • ugrep
          • Pros
          • Cons
        • sonic ⭐️ (the recommended upgrade path for most people)
          • Pros
          • Cons
        • SQLite FTS5
          • Pros
          • Cons
        • Further Reading
    • Publishing Your Archive
      • 1. Use the built-in web server
      • 2. Export and host it as static HTML
      • Security Concerns
        • Protecting the Admin Dashboard
      • Copyright Concerns
        • Further Reading: USA Copyright Law & Fair Use Exemptions
    • Scheduled Archiving
      • How It Works
      • CLI Usage
      • Docker Compose
      • Examples
    • Chrome / Chromium Setup
      • Installing Chromium
        • ⭐️ Any OS (recommended)
        • macOS
        • Ubuntu/Debian
      • Installing Google Chrome
        • macOS
        • Ubuntu/Debian
      • Troubleshooting Chromium Install
    • Setting Up a Chromium User Profile
      • Docker VNC Setup
      • Non-Docker Setup (Local Host)
      • Non-Docker Setup (Remote Host)
      • More Info & Troubleshooting
    • Upgrading Versions
      • Upgrading with Docker Compose ⭐️
      • Upgrading with plain Docker
      • Upgrading with a package manager
      • Merge two or more existing archives
      • Related Documents
    • Upgrading or Merging Archives
    • Merging Collections
      • Modify the ArchiveBox SQLite3 DB directly
        • Example: Modifying an existing user’s email
        • Example: Adding a new user with a hashed password
      • Database Troubleshooting
      • Related Documents
    • Troubleshooting
      • Installing
        • macOS
        • Python
        • Chromium/Google Chrome
        • Wget & Curl
        • NPM Dependencies
      • Archiving
        • No links parsed from export file
        • Lots of skipped sites
        • Lots of errors
        • Lots of broken links from the index
        • Removing unwanted links from the index
      • Hosting the Archive
        • Other database or filesystem issues
          • Docker Permissions issues
      • Database
        • Filesystem doesn’t support FSYNC (e.g. network mounts)
        • Database and filesystem contention issues when running multiple ArchiveBox processes
        • Database migrations errors or upgrade issues
        • Repairing a corrupted SQLite3 database file
  • Architecture
    • ArchiveBox Architecture Diagrams
      • High-Level System Execution Flow
      • State Diagrams for Main Models
        • Crawl
      • Snapshot
        • ArchiveResult
  • API Reference
    • Filesystem
    • SQL API
    • REST API
    • Python API
      • archivebox
        • Subpackages
          • archivebox.machine
          • archivebox.progressmonitor
          • archivebox.ldap
          • archivebox.misc
          • archivebox.cli
          • archivebox.api
          • archivebox.workers
          • archivebox.core
          • archivebox.search
          • archivebox.mcp
          • archivebox.personas
          • archivebox.crawls
          • archivebox.config
          • archivebox.base_models
          • archivebox.services
          • archivebox.plugins
        • Submodules
          • archivebox.uuid_compat
          • archivebox.__main__
          • archivebox.manage
        • Package Contents
          • Classes
          • Functions
          • Data
          • API
  • Meta
    • Roadmap
      • Planned Specification
        • v0.7: Schema improvements
        • v0.8:  Security
        • v0.9:  Performance
        • v1.0: Full headless browser control
        • v2.0 Federated or distributed archiving + paid hosted service offering
        • Major long-term changes
        • Smaller planned features
      • Past Releases
      • UI / UX Improvements Planned
      • New Extractors Planned
        • Social Media
        • Video/Streams
        • Audio/Music
        • Photos/Images/Comics
        • Text/Forums
        • MOOC/Educational Content
        • Re-Archiving / WARC Creation
        • Other
    • Changelog
    • Supporting Development
    • Web Archiving Community
      • The Master Lists
      • Web Archiving Projects
        • Bookmarking Services
        • From the Archive.org & Archive-It teams
        • From Webrecorder
        • From Rhizome.org (Conifer)
        • From the Old Dominion University: Web Science Team
        • From the Archives Unleashed Team
        • From the IIPC team
        • Other Public Archiving Services
        • Other ArchiveBox Alternatives
        • Smaller Utilities
      • Reading List
        • Blogs Friends of ArchiveBox
        • Articles We Like About Internet Archiving
        • ArchiveBox-Specific Posts, Tutorials, and Guides
        • ArchiveBox Discussions in News & Social Media
      • Communities
        • Most Active Communities
        • Web Archiving Communities
        • General Archiving Foundations, Coalitions, Initiatives, and Institutes
      • ArchiveBox Community Resources
        • ArchiveBox Chat Rooms
        • ArchiveBox on Social Media
        • ArchiveBox on Package Distribution Platforms
ArchiveBox
  • Contents
  • API Reference
  • archivebox
  • archivebox.misc
  • Edit on GitHub

archivebox.misc

Submodules

  • archivebox.misc.paginators
  • archivebox.misc.toml_util
  • archivebox.misc.system
  • archivebox.misc.shell_welcome_message
  • archivebox.misc.logging
  • archivebox.misc.logging_util
  • archivebox.misc.checks
  • archivebox.misc.db
  • archivebox.misc.serve_static
  • archivebox.misc.jsonl
  • archivebox.misc.monkey_patches
  • archivebox.misc.hashing
  • archivebox.misc.util
Previous Next

© Copyright 2026 ArchiveBox.