archivebox.index

Submodules

• archivebox.index.schema
• archivebox.index.csv
• archivebox.index.json
• archivebox.index.html
• archivebox.index.sql

Package Contents

Functions

merge_links	deterministially merge two links, favoring longer field values over shorter, and “cleaner” values over worse ones.
validate_links
archivable_links	remove chrome://, about:// or other schemed links that cant be archived
fix_duplicate_links	ensures that all non-duplicate links have monotonically increasing timestamps
sorted_links
links_after_timestamp
lowest_uniq_timestamp	resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2
timed_index_update
write_main_index	Writes links to sqlite3 file for a given list of links
load_main_index	parse and load existing index with any new links from import_path merged in
load_main_index_meta
parse_links_from_source
fix_duplicate_links_in_index	Given a list of in-memory Links, dedupe and merge them with any conflicting Snapshots in the DB.
dedupe_links	The validation of links happened at a different stage. This method will focus on actual deduplication and timestamp fixing.
write_link_details
load_link_details	check for an existing link archive in the given directory, and load+merge it into the given link dict
q_filter
search_filter
snapshot_filter
get_indexed_folders	indexed links without checking archive status or data directory validity
get_archived_folders	indexed links that are archived with a valid data directory
get_unarchived_folders	indexed links that are unarchived with no data directory or an empty data directory
get_present_folders	dirs that actually exist in the archive/ folder
get_valid_folders	dirs with a valid index matched to the main index and archived content
get_invalid_folders	dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized
get_duplicate_folders	dirs that conflict with other directories that have the same link URL or timestamp
get_orphaned_folders	dirs that contain a valid index but aren’t listed in the main index
get_corrupted_folders	dirs that don’t contain a valid index and aren’t listed in the main index
get_unrecognized_folders	dirs that don’t contain recognizable archive data and aren’t listed in the main index
is_valid
is_corrupt
is_archived
is_unarchived
fix_invalid_folder_locations

Data

LINK_FILTERS

API

archivebox.index.merge_links(a: archivebox.index.schema.Link, b: archivebox.index.schema.Link) → archivebox.index.schema.Link

deterministially merge two links, favoring longer field values over shorter, and “cleaner” values over worse ones.

archivebox.index.validate_links(links: Iterable[archivebox.index.schema.Link]) → List[archivebox.index.schema.Link]

archivebox.index.archivable_links(links: Iterable[archivebox.index.schema.Link]) → Iterable[archivebox.index.schema.Link]

remove chrome://, about:// or other schemed links that cant be archived

archivebox.index.fix_duplicate_links(sorted_links: Iterable[archivebox.index.schema.Link]) → Iterable[archivebox.index.schema.Link]

ensures that all non-duplicate links have monotonically increasing timestamps

archivebox.index.sorted_links(links: Iterable[archivebox.index.schema.Link]) → Iterable[archivebox.index.schema.Link]

archivebox.index.links_after_timestamp(links: Iterable[archivebox.index.schema.Link], resume: Optional[float] = None) → Iterable[archivebox.index.schema.Link]

archivebox.index.lowest_uniq_timestamp(used_timestamps: collections.OrderedDict, timestamp: str) → str

resolve duplicate timestamps by appending a decimal 1234, 1234 -> 1234.1, 1234.2

archivebox.index.timed_index_update(out_path: pathlib.Path)

archivebox.index.write_main_index(links: List[archivebox.index.schema.Link], out_dir: pathlib.Path = DATA_DIR, created_by_id: int | None = None) → None

Writes links to sqlite3 file for a given list of links

archivebox.index.load_main_index(out_dir: pathlib.Path | str = DATA_DIR, warn: bool = True) → List[archivebox.index.schema.Link]

parse and load existing index with any new links from import_path merged in

archivebox.index.load_main_index_meta(out_dir: pathlib.Path = DATA_DIR) → Optional[dict]

archivebox.index.parse_links_from_source(source_path: str, root_url: Optional[str] = None, parser: str = 'auto') → List[archivebox.index.schema.Link]

archivebox.index.fix_duplicate_links_in_index(snapshots: django.db.models.QuerySet, links: Iterable[archivebox.index.schema.Link]) → Iterable[archivebox.index.schema.Link]

Given a list of in-memory Links, dedupe and merge them with any conflicting Snapshots in the DB.

archivebox.index.dedupe_links(snapshots: django.db.models.QuerySet, new_links: List[archivebox.index.schema.Link]) → List[archivebox.index.schema.Link]

The validation of links happened at a different stage. This method will focus on actual deduplication and timestamp fixing.

archivebox.index.write_link_details(link: archivebox.index.schema.Link, out_dir: Optional[str] = None, skip_sql_index: bool = False) → None

archivebox.index.load_link_details(link: archivebox.index.schema.Link, out_dir: Optional[str] = None) → archivebox.index.schema.Link

check for an existing link archive in the given directory, and load+merge it into the given link dict

archivebox.index.LINK_FILTERS

None

archivebox.index.q_filter(snapshots: django.db.models.QuerySet, filter_patterns: List[str], filter_type: str = 'exact') → django.db.models.QuerySet

archivebox.index.search_filter(snapshots: django.db.models.QuerySet, filter_patterns: List[str], filter_type: str = 'search') → django.db.models.QuerySet

archivebox.index.snapshot_filter(snapshots: django.db.models.QuerySet, filter_patterns: List[str], filter_type: str = 'exact') → django.db.models.QuerySet

archivebox.index.get_indexed_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

indexed links without checking archive status or data directory validity

archivebox.index.get_archived_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

indexed links that are archived with a valid data directory

archivebox.index.get_unarchived_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

indexed links that are unarchived with no data directory or an empty data directory

archivebox.index.get_present_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

dirs that actually exist in the archive/ folder

archivebox.index.get_valid_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

dirs with a valid index matched to the main index and archived content

archivebox.index.get_invalid_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

dirs that are invalid for any reason: corrupted/duplicate/orphaned/unrecognized

archivebox.index.get_duplicate_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

dirs that conflict with other directories that have the same link URL or timestamp

archivebox.index.get_orphaned_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

dirs that contain a valid index but aren’t listed in the main index

archivebox.index.get_corrupted_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

dirs that don’t contain a valid index and aren’t listed in the main index

archivebox.index.get_unrecognized_folders(snapshots, out_dir: pathlib.Path = DATA_DIR) → Dict[str, Optional[archivebox.index.schema.Link]]

dirs that don’t contain recognizable archive data and aren’t listed in the main index

archivebox.index.is_valid(link: archivebox.index.schema.Link) → bool

archivebox.index.is_corrupt(link: archivebox.index.schema.Link) → bool

archivebox.index.is_archived(link: archivebox.index.schema.Link) → bool

archivebox.index.is_unarchived(link: archivebox.index.schema.Link) → bool

archivebox.index.fix_invalid_folder_locations(out_dir: pathlib.Path = DATA_DIR) → Tuple[List[str], List[str]]