archivebox.misc.db

Database utility functions for ArchiveBox.

Post-bootstrap: requires archivebox.config constants and uses Django lazily (from django.db import ... inside functions). Not safe to import pre-bootstrap.

Module Contents

Functions

is_postgres	True if this process started with the PostgreSQL database backend.
postgres_db_params	Postgres connection params from config (NAME/USER/PASSWORD/HOST/PORT).
_psycopg_connect
database_exists	True if this collection’s database has been initialized.
database_display_location	Human-readable location of the database (file path or postgres DSN).
ensure_database_ready	Make sure a database server is reachable before running migrations.
approximate_row_counts	Cheap per-table approximate row counts from the backend’s optimizer stats.
truncate_overlong_charfields	Clamp a model instance’s CharField values to their declared max_length.
rebuild_models_from_migration_state	(non-sqlite only) Drop and recreate the given models’ tables from the current migration state.
drop_models_on_postgres	Reverse companion to rebuild_models_from_migration_state.
run_db_analyze_batch	Refresh approximate SQLite planner stats for one table per idle tick.
compact_command
sqlite_lock_holders
log_sqlite_lock_holders
sqlite_lock_error
retry_sqlite_locks
migration_lock
migration_state	Cheaply compare migration files to django_migrations without invoking migrate.
pending_migrations	Return migration files on disk that have not been applied yet.
apply_migrations	Apply pending Django migrations

Data

_IS_POSTGRES
HISTORICAL_GHOST_MIGRATIONS

API

archivebox.misc.db._IS_POSTGRES: bool | None

None

archivebox.misc.db.is_postgres(config: Any | None = None) → bool

True if this process started with the PostgreSQL database backend.

archivebox.misc.db.postgres_db_params() → dict[str, str]

Postgres connection params from config (NAME/USER/PASSWORD/HOST/PORT).

archivebox.misc.db._psycopg_connect(dbname: str | None = None, connect_timeout: int = 5)

archivebox.misc.db.database_exists() → bool

True if this collection’s database has been initialized.

sqlite: the index.sqlite3 file exists on disk. postgres: the configured database is reachable and contains the django_migrations table. Safe to call before Django is set up.

archivebox.misc.db.database_display_location() → str

Human-readable location of the database (file path or postgres DSN).

archivebox.misc.db.ensure_database_ready() → None

Make sure a database server is reachable before running migrations.

sqlite: no-op (the file is created on first connection). postgres: verify the server accepts connections and create the configured database if it does not exist yet. Raises SystemExit with a helpful message if the server is unreachable.

archivebox.misc.db.approximate_row_counts(connection) → dict[str, int]

Cheap per-table approximate row counts from the backend’s optimizer stats.

sqlite: reads sqlite_stat1 (populated by ANALYZE). postgres: reads pg_class.reltuples (maintained by autovacuum/ANALYZE). Returns {} on any failure; tables never analyzed may be absent.

archivebox.misc.db.truncate_overlong_charfields(instance=None, **kwargs) → None

Clamp a model instance’s CharField values to their declared max_length.

SQLite never enforces VARCHAR(n) limits, so ArchiveBox has always stored overlong values (e.g. long crawl labels or page titles) untruncated. PostgreSQL enforces them and would raise DataError on save instead. Truncating keeps writes succeeding identically on both backends.

Dual-use: works as a pre_save receiver (Django passes instance= and sender= as kwargs; registered in CoreConfig.ready()) and as a plain truncate_overlong_charfields(obj) call for bulk_create paths, which bypass signals.

archivebox.misc.db.rebuild_models_from_migration_state(apps, schema_editor, app_label: str, model_names: list[str]) → None

(non-sqlite only) Drop and recreate the given models’ tables from the current migration state.

ArchiveBox’s historical sqlite migrations rebuild tables with raw SQL that intentionally diverges from Django migration state (state-only AddFields reconciled by later sqlite rebuilds). Postgres support postdates all of them, so a non-sqlite database can never contain legacy data at these points in history: every affected table is empty, and dropping + recreating it from state is always equivalent, keeping the real schema in lockstep with migration state at each divergence point. No-op on sqlite.

archivebox.misc.db.drop_models_on_postgres(apps, schema_editor, app_label: str, model_names: list[str]) → None

Reverse companion to rebuild_models_from_migration_state.

Drops the given models’ tables on non-sqlite backends (in the order given, so callers pass reverse-dependency order). No-op on sqlite, whose reverse is handled by the gated RunSQL reverse_sql instead.

archivebox.misc.db.run_db_analyze_batch(remaining: list[str] | None) → list[str]

Refresh approximate SQLite planner stats for one table per idle tick.

A full ANALYZE scans every index while holding a write transaction; moving it into the runner did not prevent it from blocking submissions. Sampling at most about 1000 entries per index keeps maintenance independent of collection size. Approximate stats are sufficient for choosing indexes. PostgreSQL owns its statistics through autovacuum/autoanalyze.

archivebox.misc.db.compact_command(cmdline: list[str] | None, fallback: str = '') → str

archivebox.misc.db.sqlite_lock_holders(db_path: pathlib.Path = CONSTANTS.DATABASE_FILE) → list[str]

archivebox.misc.db.log_sqlite_lock_holders(console: Any, *, db_path: pathlib.Path = CONSTANTS.DATABASE_FILE, limit: int = 8) → None

archivebox.misc.db.sqlite_lock_error(error: BaseException) → bool

archivebox.misc.db.retry_sqlite_locks(action: collections.abc.Callable[[], Any], *, label: str, stderr: TextIO | None = None) → Any

archivebox.misc.db.migration_lock(stdout: TextIO | None = None)

archivebox.misc.db.HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]]

‘frozenset(…)’

archivebox.misc.db.migration_state(out_dir: pathlib.Path = CONSTANTS.DATA_DIR) → tuple[list[str], list[str], dict[str, str]]

Cheaply compare migration files to django_migrations without invoking migrate.

archivebox.misc.db.pending_migrations(out_dir: pathlib.Path = CONSTANTS.DATA_DIR) → list[str]

Return migration files on disk that have not been applied yet.

archivebox.misc.db.apply_migrations(out_dir: pathlib.Path = CONSTANTS.DATA_DIR, stdout: TextIO | None = None, stderr: TextIO | None = None, verbosity: int = 1) → list[str]

Apply pending Django migrations