archivebox.cli.archivebox_crawl

archivebox crawl [args…] [–filters]

Manage Crawl records.

Actions: create - Create Crawl jobs from URLs list - List Crawls as JSONL (with optional filters) update - Update Crawls from stdin JSONL delete - Delete Crawls from stdin JSONL

Examples: # Create archivebox crawl create https://example.com https://foo.com –depth=1 archivebox crawl create –tag=news https://example.com

# List with filters
archivebox crawl list --status=queued
archivebox crawl list --urls__icontains=example.com

# Update
archivebox crawl list --status=started | archivebox crawl update --status=queued

# Delete
archivebox crawl list --urls__icontains=spam.com | archivebox crawl delete --yes

# Full pipeline
archivebox crawl create https://example.com | archivebox snapshot create | archivebox run

Module Contents

Functions

create_crawl	Create a Crawl job from URLs.
list_crawls	List Crawls as JSONL with optional filters.
update_crawls	Update Crawls from stdin JSONL.
delete_crawls	Delete Crawls from stdin JSONL.
main	Manage Crawl records.
create_cmd	Create a Crawl job from URLs or stdin.
list_cmd	List Crawls as JSONL.
update_cmd	Update Crawls from stdin JSONL.
delete_cmd	Delete Crawls from stdin JSONL.

Data

__command__

API

archivebox.cli.archivebox_crawl.__command__

‘archivebox crawl’

archivebox.cli.archivebox_crawl.create_crawl(urls: collections.abc.Iterable[str], depth: int = 0, tag: str = '', status: str = 'queued', created_by_id: int | None = None) → int

Create a Crawl job from URLs.

Takes URLs as args or stdin, creates one Crawl with all URLs, outputs JSONL. Pass-through: Records that are not URLs are output unchanged (for piping).

Exit codes: 0: Success 1: Failure

archivebox.cli.archivebox_crawl.list_crawls(status: str | None = None, urls__icontains: str | None = None, max_depth: int | None = None, limit: int | None = None) → int

List Crawls as JSONL with optional filters.

Exit codes: 0: Success (even if no results)

archivebox.cli.archivebox_crawl.update_crawls(status: str | None = None, max_depth: int | None = None) → int

Update Crawls from stdin JSONL.

Reads Crawl records from stdin and applies updates. Uses PATCH semantics - only specified fields are updated.

Exit codes: 0: Success 1: No input or error

archivebox.cli.archivebox_crawl.delete_crawls(yes: bool = False, dry_run: bool = False) → int

Delete Crawls from stdin JSONL.

Requires –yes flag to confirm deletion.

Exit codes: 0: Success 1: No input or missing –yes flag

archivebox.cli.archivebox_crawl.main()

Manage Crawl records.

archivebox.cli.archivebox_crawl.create_cmd(urls: tuple, depth: int, tag: str, status: str)

Create a Crawl job from URLs or stdin.

archivebox.cli.archivebox_crawl.list_cmd(status: str | None, urls__icontains: str | None, max_depth: int | None, limit: int | None)

List Crawls as JSONL.

archivebox.cli.archivebox_crawl.update_cmd(status: str | None, max_depth: int | None)

Update Crawls from stdin JSONL.

archivebox.cli.archivebox_crawl.delete_cmd(yes: bool, dry_run: bool)

Delete Crawls from stdin JSONL.