archivebox.base_models.models

This file provides the Django ABIDField and ABIDModel base model to inherit from.

Module Contents

Classes

AutoDateTimeField
ModelWithReadOnlyFields	Base class for models that have some read-only fields enforced by .save().
ModelWithUUID
ModelWithSerializers
ABIDModel	Abstract Base Model for other models to depend on. Provides ArchiveBox ID (ABID) interface and other helper methods.
ModelWithNotes	Very simple Model that adds a notes field to any model.
ModelWithHealthStats
ModelWithConfig	Base Model that adds a config property to any ABIDModel. This config is retrieved by abx.pm.hook.get_scope_config(…) later whenever this model is used.
ModelWithOutputDir	Base Model that adds an output_dir property to any ABIDModel.

Functions

get_or_create_system_user_pk	Get or create a system user with is_superuser=True to be the default owner for new DB rows
find_all_abid_prefixes	Return the mapping of all ABID prefixes to their models. e.g. {‘tag_’: core.models.Tag, ‘snp_’: core.models.Snapshot, …}
find_prefix_for_abid	Find the correct prefix for a given ABID that may have be missing a prefix (slow). e.g. ABID(‘obj_01BJQMF54D093DXEAWZ6JYRPAQ’) -> ‘snp_’
find_model_from_abid_prefix	Return the Django Model that corresponds to a given ABID prefix. e.g. ‘tag_’ -> core.models.Tag
find_model_from_abid	Shortcut for find_model_from_abid_prefix(abid.prefix)
find_obj_from_abid_rand	This is a huge hack and should only be used for debugging, never use this in real code / expose this to users.
find_obj_from_abid	Find an object with a given ABID by filtering possible models for a matching abid/uuid/id (fast). e.g. ‘snp_01BJQMF54D093DXEAWZ6JYRPAQ’ -> Snapshot(‘snp_01BJQMF54D093DXEAWZ6JYRPAQ’)

Data

DEFAULT_ICON
ABIDField

API

archivebox.base_models.models.DEFAULT_ICON

‘<img src=”data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABoAAAAgCAYAAAAMq2gFAAAAAXNSR0IArs4c6QAAAIRl…’

archivebox.base_models.models.ABIDField

‘partial(…)’

archivebox.base_models.models.get_or_create_system_user_pk(username='system')

Get or create a system user with is_superuser=True to be the default owner for new DB rows

class archivebox.base_models.models.AutoDateTimeField

Bases: django.db.models.DateTimeField

exception archivebox.base_models.models.ABIDError

Bases: Exception

class archivebox.base_models.models.ModelWithReadOnlyFields

Bases: django.db.models.Model

Base class for models that have some read-only fields enforced by .save().

read_only_fields: ClassVar[tuple[str, ...]]

()

class Meta

abstract

True

_fresh_from_db()

diff_from_db(keys: Iterable[str] = ()) → dict[str, tuple[Any, Any]]

Get a dictionary of the fields that have changed from the values in the database

save(*args, **kwargs) → None

class archivebox.base_models.models.ModelWithUUID

Bases: archivebox.base_models.models.ModelWithReadOnlyFields, tags.models.ModelWithKVTags

read_only_fields

(‘id’, ‘created_at’)

id

‘UUIDField(…)’

created_at

‘AutoDateTimeField(…)’

class Meta

Bases: django_stubs_ext.db.models.TypedModelMeta

abstract

True

default_json_keys: ClassVar[tuple[str, ...]]

(‘TYPE’, ‘id’, ‘abid’, ‘str’, ‘modified_at’, ‘created_at’, ‘created_by_id’, ‘status’, ‘retry_at’, ‘n…

classmethod from_dict(fields: dict[str, Any]) → Self

update(**kwargs) → None

Update the object’s properties from a dict

as_json(keys: Iterable[str] = ()) → dict

Get the object’s properties as a dict

TYPE() → str

Get the full Python dotted-import path for this model, e.g. ‘core.models.Snapshot’

property admin_change_url: str

get the admin URL e.g. /admin/core/snapshot/abcd-1234-1234-asdfjkl23jsdf4/change/

class archivebox.base_models.models.ModelWithSerializers

Bases: archivebox.base_models.models.ModelWithUUID

as_csv_row(keys: Iterable[str] = (), separator: str = ',') → str

Get the object’s properties as a csv string

as_jsonl_row(keys: Iterable[str] = (), **json_kwargs) → str

Get the object’s properties as a jsonl string

as_html_icon() → str

Get a representation of this object as a simple html tag or emoji

as_html_row() → str

Get a representation of this object as a static html table … string

as_html_embed() → str

Get a representation of this object suitable for embedding inside a roughly 400x300px iframe

as_html_fullpage() → str

Get a static html page representation of this object

class archivebox.base_models.models.ABIDModel(*args: Any, **kwargs: Any)

Bases: archivebox.base_models.models.ModelWithReadOnlyFields, archivebox.base_models.models.ModelWithUUID

Abstract Base Model for other models to depend on. Provides ArchiveBox ID (ABID) interface and other helper methods.

Initialization

Overriden init method ensures we have a stable creation timestamp that fields can use within initialization code pre-saving to DB.

abid_prefix: str

None

abid_ts_src

‘self.created_at’

abid_uri_src

‘None’

abid_subtype_src

‘self.class.name’

abid_rand_src

‘self.id’

abid_drift_allowed: bool

False

abid_salt: str

None

read_only_fields

(‘id’, ‘abid’, ‘created_at’, ‘created_by’)

id

‘UUIDField(…)’

abid

‘ABIDField(…)’

created_at

‘AutoDateTimeField(…)’

created_by

‘ForeignKey(…)’

modified_at

‘DateTimeField(…)’

_prefetched_objects_cache: Dict[str, Any]

None

class Meta

Bases: django_stubs_ext.db.models.TypedModelMeta

abstract

True

__str__() → str

classmethod check(**kwargs)

clean(abid_drift_allowed: bool | None = None) → None

save(*args: Any, abid_drift_allowed: bool | None = None, **kwargs: Any) → None

Overriden save method ensures new ABID is generated while a new object is first saving.

classmethod id_from_abid(abid: str) → str

property ABID_SOURCES: Dict[str, str]

“Get the dict of fresh ABID component values based on the live object’s properties.

property ABID_FRESH_VALUES: Dict[str, Any]

“Get the dict of fresh ABID component values based on the live object’s properties.

property ABID_FRESH_HASHES: Dict[str, str]

“Get the dict of fresh ABID component hashes based on the live object’s properties.

property ABID_FRESH_DIFFS: Dict[str, Dict[str, Any]]

Get the dict of discrepancies between the existing saved ABID and a new fresh ABID computed based on the live object.

issue_new_abid(overwrite=False) → archivebox.base_models.abid.ABID

Issue a new ABID based on the current object’s properties, can only be called once on new objects (before they are saved to DB). TODO: eventually we should move this to a separate service that makes sure they’re all unique and monotonic perhaps it could be moved to a KVTag as well, and we could just use the KVTag service + Events to issue new ABIDs

property ABID: archivebox.base_models.abid.ABID

Get the object’s existing ABID (from self.abid if it’s already saved to DB, otherwise generated fresh) e.g. -> ABID(ts=’01HX9FPYTR’, uri=’E4A5CCD9’, subtype=’00’, rand=’ZYEBQE’)

property api_url: str

Compute the REST API URL to access this object. e.g. /api/v1/core/snapshot/snp_01BJQMF54D093DXEAWZ6JYRP

property api_docs_url: str

Compute the REST API Documentation URL to learn about accessing this object. e.g. /api/v1/docs#/Core%20Models/api_v1_core_get_snapshots

class archivebox.base_models.models.ModelWithNotes

Bases: django.db.models.Model

Very simple Model that adds a notes field to any model.

notes

‘TextField(…)’

class Meta

abstract

True

class archivebox.base_models.models.ModelWithHealthStats

Bases: django.db.models.Model

num_uses_failed

‘PositiveIntegerField(…)’

num_uses_succeeded

‘PositiveIntegerField(…)’

class Meta

abstract

True

increment_num_uses_failed() → None

increment_num_uses_succeeded() → None

reset_health_counts() → None

property health: int

class archivebox.base_models.models.ModelWithConfig

Bases: django.db.models.Model

Base Model that adds a config property to any ABIDModel. This config is retrieved by abx.pm.hook.get_scope_config(…) later whenever this model is used.

config

‘JSONField(…)’

class Meta

abstract

True

class archivebox.base_models.models.ModelWithOutputDir(*args: Any, **kwargs: Any)

Bases: ModelsWithSerializers, archivebox.base_models.models.ModelWithUUID, archivebox.base_models.models.ABIDModel

Base Model that adds an output_dir property to any ABIDModel.

It creates the directory on .save(with_indexes=True), automatically migrating any old data if needed. It then writes the indexes to the output_dir on .save(write_indexes=True). It also makes sure the output_dir is in sync with the model.

Initialization

Overriden init method ensures we have a stable creation timestamp that fields can use within initialization code pre-saving to DB.

class Meta

abstract

True

save(*args, write_indexes=False, **kwargs) → None

property output_dir_parent: str

Get the model type parent directory name that holds this object’s data e.g. ‘archiveresults’

property output_dir_name: str

Get the subdirectory name for the filesystem directory that holds this object’s data e.g. ‘snp_2342353k2jn3j32l4324’

property output_dir_str: str

Get relateive the filesystem directory Path that holds that data for this object e.g. ‘snapshots/snp_2342353k2jn3j32l4324’

property OUTPUT_DIR: pathlib.Path

Get absolute filesystem directory Path that holds that data for this object e.g. Path(‘/data/snapshots/snp_2342353k2jn3j32l4324’)

write_indexes()

Write the Snapshot json, html, and merkle indexes to its output dir

save_merkle_index(**kwargs) → None

Write the ./.index.merkle file to the output dir

save_html_index(**kwargs) → None

save_json_index(**kwargs) → None

Save a JSON dump of the object to the output dir

save_symlinks_index() → None

Set up the symlink farm pointing to this object’s data

as_json(*keys) → dict

Get the object’s properties as a dict

as_html() → str

Get the object’s properties as a html string

archivebox.base_models.models.find_all_abid_prefixes() → Dict[str, type[django.db.models.Model]]

Return the mapping of all ABID prefixes to their models. e.g. {‘tag_’: core.models.Tag, ‘snp_’: core.models.Snapshot, …}

archivebox.base_models.models.find_prefix_for_abid(abid: archivebox.base_models.abid.ABID) → str

Find the correct prefix for a given ABID that may have be missing a prefix (slow). e.g. ABID(‘obj_01BJQMF54D093DXEAWZ6JYRPAQ’) -> ‘snp_’

archivebox.base_models.models.find_model_from_abid_prefix(prefix: str) → type[archivebox.base_models.models.ABIDModel] | None

Return the Django Model that corresponds to a given ABID prefix. e.g. ‘tag_’ -> core.models.Tag

archivebox.base_models.models.find_model_from_abid(abid: archivebox.base_models.abid.ABID) → type[django.db.models.Model] | None

Shortcut for find_model_from_abid_prefix(abid.prefix)

archivebox.base_models.models.find_obj_from_abid_rand(rand: Union[archivebox.base_models.abid.ABID, str], model=None) → List[archivebox.base_models.models.ABIDModel]

This is a huge hack and should only be used for debugging, never use this in real code / expose this to users.

Find an object corresponding to an ABID by exhaustively searching using its random suffix (slow). e.g. ‘obj_………………..JYRPAQ’ -> Snapshot(‘snp_01BJQMF54D093DXEAWZ6JYRPAQ’)

archivebox.base_models.models.find_obj_from_abid(abid: archivebox.base_models.abid.ABID, model=None, fuzzy=False) → Any

Find an object with a given ABID by filtering possible models for a matching abid/uuid/id (fast). e.g. ‘snp_01BJQMF54D093DXEAWZ6JYRPAQ’ -> Snapshot(‘snp_01BJQMF54D093DXEAWZ6JYRPAQ’)