archivebox.misc.db

Database utility functions for ArchiveBox.

Post-bootstrap: requires archivebox.config constants and uses Django lazily (from django.db import ... inside functions). Not safe to import pre-bootstrap.

Module Contents

Functions

is_postgres	True if DATABASE_ENGINE selects postgres (sqlite is the default).
postgres_db_params	Postgres connection params from config (NAME/USER/PASSWORD/HOST/PORT).
_psycopg_connect
database_exists	True if this collection’s database has been initialized.
database_display_location	Human-readable location of the database (file path or postgres DSN).
ensure_database_ready	Make sure a database server is reachable before running migrations.
approximate_row_counts	Cheap per-table approximate row counts from the backend’s optimizer stats.
truncate_overlong_charfields	Clamp a model instance’s CharField values to their declared max_length.
rebuild_models_from_migration_state	(non-sqlite only) Drop and recreate the given models’ tables from the current migration state.
drop_models_on_postgres	Reverse companion to rebuild_models_from_migration_state.
run_db_analyze_batch	Advance one step of a batched SQLite ANALYZE sweep.
compact_command
sqlite_lock_holders
log_sqlite_lock_holders
sqlite_lock_error
retry_sqlite_locks
migration_lock
migration_state	Cheaply compare migration files to django_migrations without invoking migrate.
pending_migrations	Return migration files on disk that have not been applied yet.
apply_migrations	Apply pending Django migrations

Data

HISTORICAL_GHOST_MIGRATIONS

API

archivebox.misc.db.is_postgres() → bool

True if DATABASE_ENGINE selects postgres (sqlite is the default).

archivebox.misc.db.postgres_db_params() → dict[str, str]

Postgres connection params from config (NAME/USER/PASSWORD/HOST/PORT).

archivebox.misc.db._psycopg_connect(dbname: str | None = None, connect_timeout: int = 5)

archivebox.misc.db.database_exists() → bool

True if this collection’s database has been initialized.

sqlite: the index.sqlite3 file exists on disk. postgres: the configured database is reachable and contains the django_migrations table. Safe to call before Django is set up.

archivebox.misc.db.database_display_location() → str

Human-readable location of the database (file path or postgres DSN).

archivebox.misc.db.ensure_database_ready() → None

Make sure a database server is reachable before running migrations.

sqlite: no-op (the file is created on first connection). postgres: verify the server accepts connections and create the configured database if it does not exist yet. Raises SystemExit with a helpful message if the server is unreachable.

archivebox.misc.db.approximate_row_counts(connection) → dict[str, int]

Cheap per-table approximate row counts from the backend’s optimizer stats.

sqlite: reads sqlite_stat1 (populated by ANALYZE). postgres: reads pg_class.reltuples (maintained by autovacuum/ANALYZE). Returns {} on any failure; tables never analyzed may be absent.

archivebox.misc.db.truncate_overlong_charfields(instance=None, **kwargs) → None

Clamp a model instance’s CharField values to their declared max_length.

SQLite never enforces VARCHAR(n) limits, so ArchiveBox has always stored overlong values (e.g. long crawl labels or page titles) untruncated. PostgreSQL enforces them and would raise DataError on save instead. Truncating keeps writes succeeding identically on both backends.

Dual-use: works as a pre_save receiver (Django passes instance= and sender= as kwargs; registered in CoreConfig.ready()) and as a plain truncate_overlong_charfields(obj) call for bulk_create paths, which bypass signals.

archivebox.misc.db.rebuild_models_from_migration_state(apps, schema_editor, app_label: str, model_names: list[str]) → None

(non-sqlite only) Drop and recreate the given models’ tables from the current migration state.

ArchiveBox’s historical sqlite migrations rebuild tables with raw SQL that intentionally diverges from Django migration state (state-only AddFields reconciled by later sqlite rebuilds). Postgres support postdates all of them, so a non-sqlite database can never contain legacy data at these points in history: every affected table is empty, and dropping + recreating it from state is always equivalent, keeping the real schema in lockstep with migration state at each divergence point. No-op on sqlite.

archivebox.misc.db.drop_models_on_postgres(apps, schema_editor, app_label: str, model_names: list[str]) → None

Reverse companion to rebuild_models_from_migration_state.

Drops the given models’ tables on non-sqlite backends (in the order given, so callers pass reverse-dependency order). No-op on sqlite, whose reverse is handled by the gated RunSQL reverse_sql instead.

archivebox.misc.db.run_db_analyze_batch(remaining: list[str] | None, *, max_seconds_per_table: float = 120.0) → list[str]

Advance one step of a batched SQLite ANALYZE sweep.

Without periodic ANALYZE the optimizer’s table stats go stale as snapshot/archiveresult tables grow, causing it to start large joins from auth_user instead of using the indexed url column and blowing snapshot detail page render time from ~50ms to ~500ms+.

The whole sweep is spread across many calls instead of running as one blocking ANALYZE: pass None to start a fresh sweep (this call enumerates user tables and runs ANALYZE on the first one); pass the returned list to advance one more table on each subsequent call. An empty return value means the sweep is complete (or has been aborted) and the next caller should pass None again. Caller is responsible for throttling new sweeps (orchestrator starts at most one per 24hr while idle) and enforcing a hard upper bound on total sweep wall time.

Safety guarantees:

• Never raises: every database call is wrapped; on any failure the function returns [] (abandoning the rest of the sweep) so the orchestrator never crashes on maintenance errors.

• Bounded per-call wall time: a SQLite progress handler aborts the current ANALYZE statement once max_seconds_per_table is exceeded, so a single pathological table cannot wedge the call.

• Never leaves the db locked: each ANALYZE runs as a single statement transaction that auto-commits (or rolls back on abort/error). The cursor and progress handler are always cleaned up in finally blocks even if Python raises mid-call.

• Silent no-op on non-SQLite backends.

WAL journal mode (set in Django settings) keeps readers fully unblocked throughout; the writer lock is only held for the brief sqlite_stat* flush after each table completes.

archivebox.misc.db.compact_command(cmdline: list[str] | None, fallback: str = '') → str

archivebox.misc.db.sqlite_lock_holders(db_path: pathlib.Path = CONSTANTS.DATABASE_FILE) → list[str]

archivebox.misc.db.log_sqlite_lock_holders(console: Any, *, db_path: pathlib.Path = CONSTANTS.DATABASE_FILE, limit: int = 8) → None

archivebox.misc.db.sqlite_lock_error(error: BaseException) → bool

archivebox.misc.db.retry_sqlite_locks(action: collections.abc.Callable[[], Any], *, label: str, stderr: TextIO | None = None) → Any

archivebox.misc.db.migration_lock(stdout: TextIO | None = None)

archivebox.misc.db.HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]]

‘frozenset(…)’

archivebox.misc.db.migration_state(out_dir: pathlib.Path = CONSTANTS.DATA_DIR) → tuple[list[str], list[str], dict[str, str]]

Cheaply compare migration files to django_migrations without invoking migrate.

archivebox.misc.db.pending_migrations(out_dir: pathlib.Path = CONSTANTS.DATA_DIR) → list[str]

Return migration files on disk that have not been applied yet.

archivebox.misc.db.apply_migrations(out_dir: pathlib.Path = CONSTANTS.DATA_DIR, stdout: TextIO | None = None, stderr: TextIO | None = None, verbosity: int = 1) → list[str]

Apply pending Django migrations