archivebox.misc.utilο
Module Contentsο
Classesο
Extended json serializer that supports serializing several model fields and objects |
Functionsο
Filter a queryset to UUID-column matches by prefix or suffix (case-insensitive). |
|
Trim quote garbage and dangling prose punctuation from an extracted URL candidate. |
|
check that all parentheses in a string are balanced and nested properly |
|
cleanup a regex-parsed url that may contain dangling trailing parens from markdown link syntax helpful to fix URLs parsed from markdown e.g. input: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def).somemoretext result: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def |
|
Parse a byte count from an integer or human-readable string like 45mb or 2 GB. |
|
Enforce function arg and kwarg types at runtime using its python3 type hints Simpler version of pydantic @validate_call decorator |
|
attach the given docstring to the decorated function |
|
Parse unix timestamps, iso format, and human-readable strings |
|
Download the contents of a remote url and return the text |
|
Based on: https://stackoverflow.com/questions/19212665/python-converting-ansi-color-codes-to-html Simple way to render colored CLI stdout/stderr in HTML properly, Textual/rich is probably better though. |
|
Serialize object to JSON string with extended type support |
Dataο
APIο
- archivebox.misc.util.filter_queryset_by_uuid_substring(queryset, slug: str, field: str = 'id')[source]ο
Filter a queryset to UUID-column matches by prefix or suffix (case-insensitive).
Avoids
id__icontains(an unindexed full-table scan over the UUID column) by stripping non-hex chars fromslugand matching withistartswith/iendswith. Returns an empty queryset for inputs with fewer than 8 hex chars to avoid overly broad matches. A full 32-char hex string falls back to an exact-equality lookup.
- archivebox.misc.util.QUOTE_DELIMITERS[source]ο
(βββ, βββ, β`β, βββ, βββ, βββ, βββ)
- archivebox.misc.util.QUOTE_ENTITY_DELIMITERS[source]ο
(βββ, βββ, βββ, βββ, βββ, βββ)
- archivebox.misc.util.URL_ENTITY_REPLACEMENTS[source]ο
((β&β, β&β), (β&β, β&β), (β&β, β&β))
- archivebox.misc.util.sanitize_extracted_url(url: str) str[source]ο
Trim quote garbage and dangling prose punctuation from an extracted URL candidate.
- archivebox.misc.util.parens_are_matched(string: str, open_char='(', close_char=')')[source]ο
check that all parentheses in a string are balanced and nested properly
- archivebox.misc.util.fix_url_from_markdown(url_str: str) str[source]ο
cleanup a regex-parsed url that may contain dangling trailing parens from markdown link syntax helpful to fix URLs parsed from markdown e.g. input: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def).somemoretext result: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def
IMPORTANT ASSUMPTION: valid urls wont have unbalanced or incorrectly nested parentheses e.g. this will fail the user actually wants to ingest a url like βhttps://example.com/some_wei)(rd_urlβ in that case it will return https://example.com/some_wei (truncated up to the first unbalanced paren) This assumption is true 99.9999% of the time, and for the rare edge case the user can use url_list parser.
- archivebox.misc.util.parse_filesize_to_bytes(value: str | int | float | None) int[source]ο
Parse a byte count from an integer or human-readable string like 45mb or 2 GB.
- archivebox.misc.util.enforce_types(func)[source]ο
Enforce function arg and kwarg types at runtime using its python3 type hints Simpler version of pydantic @validate_call decorator
- archivebox.misc.util.docstring(text: str | None)[source]ο
attach the given docstring to the decorated function
- archivebox.misc.util.parse_date(date: Any) datetime.datetime | None[source]ο
Parse unix timestamps, iso format, and human-readable strings
- archivebox.misc.util.download_url(url: str, timeout: int | None = None, config=None, **config_kwargs) str[source]ο
Download the contents of a remote url and return the text
- archivebox.misc.util.ansi_to_html(text: str) str[source]ο
Based on: https://stackoverflow.com/questions/19212665/python-converting-ansi-color-codes-to-html Simple way to render colored CLI stdout/stderr in HTML properly, Textual/rich is probably better though.
- class archivebox.misc.util.ExtendedEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]ο
Bases:
json.JSONEncoderExtended json serializer that supports serializing several model fields and objects
Initialization
Constructor for JSONEncoder, with sensible defaults.
If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float, bool or None. If skipkeys is True, such items are simply skipped.
If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.
If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an RecursionError). Otherwise, no such check takes place.
If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.
If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.
If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.
If specified, separators should be an (item_separator, key_separator) tuple. The default is (β, β, β: β) if indent is
Noneand (β,β, β: β) otherwise. To get the most compact JSON representation, you should specify (β,β, β:β) to eliminate whitespace.If specified, default is a function that gets called for objects that canβt otherwise be serialized. It should return a JSON encodable version of the object or raise a
TypeError.