Web Archiving Community

🔢 Just getting started and want to learn more about why Web Archiving is important?
Check out this article: On the Importance of Web Archiving.


The internet archiving community is surprisingly far-reaching and almost universally friendly!

Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open source tool for your web archiving need, or just want to see where archivists hang out online, this is my attempt at an index of the entire web archiving community.


The Master Lists

Indexes of archiving institutions and software maintained by other people. If there’s anything archivists love doing, it’s making lists.


Web Archiving Projects

           

Bookmarking Services

  • Pocket Premium Bookmarking tool that provides an archiving service in their paid version, run by Mozilla
  • Pinboard Bookmarking tool that provides archiving in a paid version, run by a single independent developer
  • Instapaper Bookmarking alternative to Pocket/Pinboard (with no archiving)
  • Wallabag / Wallabag.it Self-hostable web archiving server that can import via RSS
  • Shaarli Self-hostable bookmark tagging, archiving, and sharing service
  • ReadWise A paid Pocket/Pinboard alternative that includes article snippet and highlight saving

From the Archive.org & Archive-It teams

  • Archive.org The O.G. wayback machine provided publicly by the Internet Archive (Archive.org)
  • Archive.it commercial Wayback-Machine solution
  • Heretrix The king of internet archiving crawlers, powers the Wayback Machine
  • Brozzler chrome headless crawler + WARC archiver maintained by Archive.org
  • WarcProx warc proxy recording and playback utility
  • WarcTools utilities for dealing with WARCs
  • Grab-Site An easy preconfigured web crawler designed for backing up websites
  • WPull A pure python implementation of wget with WARC saving
  • More on their Github…

From the Rhizome.org/WebRecorder.io/Conifer team


From the Old Dominion University: Web Science Team

  • ipwb A distributed web archiving solution using pywb with ipfs for storage
  • archivenow tool that pushes urls into all the online archive services like Archive.is and Archive.org
  • node-warc Parse And Create Web ARChive (WARC) files with node.js
  • WAIL Web archiver GUI using Heritrix and OpenWayback
  • Squidwarc User-scriptable, archival crawler using Chrome
  • WAIL (Electron) Electron app version of the original wail for creating and interacting with web archives
  • warcreate a Chrome extension for creating WARCs from any webpage
  • More on their Github…

From the Archives Unleashed Team


From the IIPC team


Other Public Archiving Services

  • https://perma.cc
  • https://www.pagefreezer.com
  • https://www.smarsh.com
  • https://www.stillio.com
  • https://archive.is / https://archive.today
  • https://archive.st
  • http://theoldnet.com
  • https://timetravel.mementoweb.org/
  • https://freezepage.com/
  • https://webcitation.org/archive
  • https://archiveofourown.org/
  • https://megalodon.jp/
  • https://www.webarchive.org.uk/ukwa/
  • https://github.com/HelloZeroNet/ZeroNet (super cool project)
  • Google, Bing, DuckDuckGo, and other search engine caches

Other ArchiveBox Alternatives

  • browsertrix-crawler / ArchiveWeb.page + ReplayWeb.page + pywb Webrecorder.io’s archiving suite has the highest fidelity, and can flawlessly archive YouTube, Twitter, FB and other complex, JS-heavy SPAs
  • SingleFile Web Extension / CLI util for Firefox and Chrome to save a web page as a single HTML file
  • Memex by Worldbrain.io a beautiful, user-friendly browser extension that archives all history with full-text search, annotation support, and more
  • Hypothes.is a web/pdf/ebook annotation tool that also archives content
  • Reminiscence extremely similar to ArchiveBox, uses a Django backend + UI and provides auto-tagging and summary features with NLTK
  • Shaarchiver very similar project that archives Firefox, Shaarli, or Delicious bookmarks and all linked media, generating a markdown/HTML index
  • Archivy Python-based self-hosted knowledge base embedded into your filesystem
  • Polarized a desktop application for bookmarking, annotating, and archiving articles offline
  • 22120 Archiving tool that uses the Chrome debugger protocol to save each page as-loaded in the browser
  • Photon a fast crawler with archiving and asset extraction support
  • LinkAce A self-hosted bookmark management tool that saves snapshots to archive.org
  • Trilium Personal web UI based knowledge-base with web clipping and note-taking
  • Herodotus Django-based web archiving tool with a focus on collecting text-based content
  • Buku Browser-independent bookmark manager CLI written in Python3 and SQLite3
  • ReadableWebProxy A proxying archiver that downloads content from sites and can snapshot multiple versions of sites over time
  • Perkeep “Perkeep lets you permanently keep your stuff, for life.”
  • Fetching.io A personal search engine/archiver that lets you search through all archived websites that you’ve bookmarked
  • Fossilo A commercial archiving solution that appears to be very similar to ArchiveBox
  • Archivematica web GUI for institutional long-term archiving of web and other content
  • Headless Chrome Crawler distributed web crawler built on puppeteer with screenshots
  • WWWofle old proxying recorder software similar to ArchiveBox
  • Erised Super simple CLI utility to bookmark and archive webpages
  • Zotero collect, organize, cite, and share research (mainly for technical/scientific papers & citations)
  • TiddlyWiki Non-linear bookmark and note-taking tool with archiving support
  • Joplin Desktop + mobile app for knowledge-base-style info collection and notes (w/ optional plugin for archiving)
  • Hunchly A paid web archiving / session recording tool design for OSINT

Smaller Utilities

Random helpful utilities for web archiving, WARC creation and replay, and more…

  • https://github.com/karlicoss/promnesia A browser extension that collects and collates all the URLs you visit into a hierarchical/graph structure with metadata
  • https://github.com/vrtdev/save-page-state A Chrome extension for saving the state of a page in multiple formats
  • https://github.com/jsvine/waybackpack command-line tool that lets you download the entire Wayback Machine archive for a given URL
  • https://github.com/hartator/wayback-machine-downloader Download an entire website from the Internet Archive Wayback Machine.
  • https://github.com/Lifesgood123/prevent-link-rot Replace any broken URLs in some content with Wayback machine URL equivalents
  • https://en.archivarix.com download an archived page or entire site from the Wayback Machine
  • https://proofofexistence.com prove that a certain file existed at a given time using the blockchain
  • https://github.com/chfoo/warcat for merging, extracting, and verifying WARC files
  • https://github.com/mozilla/readability tool for extracting article contents and text
  • https://github.com/mholt/timeliner All your digital life on a single timeline, stored locally
  • https://github.com/wkhtmltopdf/wkhtmltopdf Webkit HTML to PDF archiver/saver
  • Sheetsee-Pocket project that provides a pretty auto-updating index of your Pocket links (without archiving them)
  • Pocket -> IFTTT -> Dropbox Post by Christopher Su on his Pocket saving IFTTT recipe
  • http://squidman.net/squidman/index.html
  • https://wordpress.org/plugins/broken-link-checker/
  • https://github.com/ArchiveTeam/wpull
  • http://freedup.org/
  • https://en.wikipedia.org/wiki/Furl
  • https://preservica.com/digital-archive-software-1/active-digital-preservation For-profit company offering a digital preservation software suite
  • https://github.com/karlicoss/grasp capture webpages from Firefox and Chrome into Org-mode documents
  • And many more on the other lists…

Reading List

A collection of blog posts and articles about internet archiving, contact me / open an issue if you want to add a link here!


Blogs

  • https://blog.archive.org
  • https://netpreserveblog.wordpress.com
  • https://blog.conifer.rhizome.org/ (formerly https://blog.webrecorder.io/)
  • https://ws-dl.blogspot.com
  • https://siarchives.si.edu/blog
  • https://parameters.ssrc.org
  • https://sr.ithaka.org/publications
  • https://ait.blog.archive.org
  • https://brewster.kahle.org
  • https://ianmilligan.ca
  • https://medium.com/@giovannidamiola

Articles

  • https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/
  • https://theconversation.com/your-internet-data-is-rotting-115891
  • https://www.bbc.com/future/story/20190401-why-theres-so-little-left-of-the-early-internet
  • https://sr.ithaka.org/publications/the-state-of-digital-preservation-in-2018/
  • https://gizmodo.com/delete-never-the-digital-hoarders-who-collect-tumblrs-1832900423
  • https://siarchives.si.edu/blog/we-are-not-alone-progress-digital-preservation-community
  • https://www.gwern.net/Archiving-URLs
  • http://brewster.kahle.org/2015/08/11/locking-the-web-open-a-call-for-a-distributed-web-2/
  • https://lwn.net/Articles/766374/
  • https://en.wikipedia.org/wiki/List_of_Web_archiving_initiatives
  • https://medium.com/@giovannidamiola/making-the-internet-archives-full-text-search-faster-30fb11574ea9
  • https://xkcd.com/1909/
  • https://samsaffron.com/archive/2012/06/07/testing-3-million-hyperlinks-lessons-learned#comment-31366
  • https://www.gwern.net/docs/linkrot/2011-muflax-backup.pdf
  • https://thoughtstreams.io/higgins/permalinking-vs-transience/
  • http://ait.blog.archive.org/files/2014/04/archiveit_life_cycle_model.pdf
  • https://blog.archive.org/2016/05/26/web-archiving-with-national-libraries/
  • https://blog.archive.org/2014/10/28/building-libraries-together/
  • https://ianmilligan.ca/2018/03/27/ethics-and-the-archived-web-presentation-the-ethics-of-studying-geocities/
  • https://ianmilligan.ca/2018/05/22/new-article-if-these-crawls-could-talk-studying-and-documenting-web-archives-provenance/
  • https://ws-dl.blogspot.com/2019/02/2019-02-08-google-is-being-shuttered.html

If any of these links are dead, you can find an archived version on https://archive.sweeting.me.


ArchiveBox-Specific Posts, Tutorials, and Guides

  • https://www.cyberpunks.com/preserve-the-internet-with-archivebox/
  • https://nixintel.info/osint-tools/make-your-own-internet-archive-with-archive-box/
  • “How to install ArchiveBox to preserve websites you care about” https://blog.sleeplessbeastie.eu/2019/06/19/how-to-install-archivebox-to-preserve-websites-you-care-about/
  • “How to remotely archive websites using ArchiveBox” https://blog.sleeplessbeastie.eu/2019/06/26/how-to-remotely-archive-websites-using-archivebox/
  • “How to use CutyCapt inside ArchiveBox” https://blog.sleeplessbeastie.eu/2019/07/10/how-to-use-cutycapt-inside-archivebox/
  • “Automate ArchiveBox with Google Spreadsheet to Backup your internet” https://manfred.life/archivebox
  • “【デモ有♪】ConoHaのArchiveBoxアプリケーションを使ってみたよ” https://qiita.com/CloudRemix/items/691caf91efa3ef19a7ad
  • “WEB-ARCHIV TEIL 8: WALLABAG UND ARCHIVEBOX” http://webermartin.net/blog/web-archiv-teil-8-wallabag-und-archivebox/
  • https://metaxyntax.neocities.org/entries/7.html

Communities

Most Active Communities



General Archiving Foundations, Coalitions, Initiatives, and Institutes

Find your local archiving group in the list and see how you can contribute!

You can find more organizations and initiatives on these other lists: