Getting Your Data Out of History Book
History Book stores all the saved pages with Core Data, which means there’s an SQLite database somewhere on your Mac. If you want to export those pages somewhere, you can read that database directly.
Disclaimer: This is all undocumented and unsupported by Apple. Only follow this tutorial if you’re comfortable with the terminal.
Where is the database
~/Library/Group Containers/group.com.andadinosaur.HistoryBook/
We only care about 3 files: History.sqlite, History.sqlite-wal, and History.sqlite-shm. SQLite runs this database in write-ahead logging mode, meaning SQLite will write the changes to the -wal file first, then add them to History.sqlite later. So we need all 3 files. Let’s copy them to somewhere safe. Before you copy, quit History Book so all the transactions are committed:
mkdir -p ~/history-book-export
cd ~/history-book-export
cp ~/Library/Group\ Containers/group.com.andadinosaur.HistoryBook/History.sqlite* .
Export to JSON
If you know your SQL well, you can stop reading here. If you don’t, I recommend you export the data into a JSON file first:
sqlite3 -json History.sqlite "
SELECT
Z_PK AS id,
ZTITLE AS title,
ZURL AS url,
ZHOSTNAME AS hostname,
ZSITENAME AS site_name,
ZBYLINE AS byline,
ZEXCERPT AS excerpt,
ZLANG AS lang,
ZDIR AS dir,
datetime(ZSAVEDAT + 978307200, 'unixepoch') AS saved_at,
ZCONTENT AS content
FROM ZPAGE
ORDER BY ZSAVEDAT DESC;
" > pages.json
(I don’t know why the columns are named like that. Core Data chose the column names, not me. The Z prefix has nothing to do with me being Zhenyi.)
I think exporting the data to JSON first is worth it, because almost all programming languages have some nice JSON library. Also because some of you might ignore my advice and won’t copy the SQLite file out first.
Sidenote: WTF is 978307200? Well, Core Data stores dates as seconds since 2001-01-01, and we need to convert it to seconds since 1970-01-01 for unixepoch. I’d rather not do date math manually so here’s a one-liner to calculate that:
DateTime.parse('2001-01-01').to_time - DateTime.parse('1970-01-01').to_time
=> 978307200
Sidenote 2: The command above dumps all the saved pages into a JSON file. If you only want pages in a folder, do this:
sqlite3 History.sqlite "SELECT Z_PK, ZNAME FROM ZFOLDER;"
That gives you the folder’s primary key (Z_PK) for your WHERE ZFOLDER = ? clause.
Reference implementation
Here’s a Ruby script that turns pages.json into HTML files. You should be able to read it and write your own with your favorite language. It’s about 90 lines of code, and most of it is just the inline HTML template.
# run it like this:
# ruby export.rb pages.json export/
require "json"
require "cgi"
require "set"
require "fileutils"
input = ARGV[0] || "pages.json"
output = ARGV[1] || "export"
# keep the filenames short
MAX_SLUG_BYTES = 180
# turn-the-titles-into-something-like-this
def slugify(title)
slug = title.to_s
.unicode_normalize(:nfc)
.downcase
.gsub(/[^[:alnum:]]+/, "-")
.delete_prefix("-")
.delete_suffix("-")
truncate_bytes(slug, MAX_SLUG_BYTES).delete_suffix("-")
end
# titles are utf-8 so don't accidentally split a character
def truncate_bytes(string, limit)
return string if string.bytesize <= limit
string.each_char.with_object(+"") do |char, kept|
break kept if kept.bytesize + char.bytesize > limit
kept << char
end
end
# in case 2 articles have the same title
def unique_filename(page, taken)
base = slugify(page["title"])
base = slugify(page["hostname"]) if base.empty?
base = "untitled" if base.empty?
candidate = base
suffix = 1
while taken.include?(candidate)
suffix += 1
candidate = "#{base}-#{suffix}"
end
taken << candidate
candidate
end
def document(page)
# how we do things in rails
h = ->(value) { CGI.escapeHTML(value.to_s) }
<<~HTML
<!doctype html>
<html lang="#{h[page["lang"] || "en"]}" dir="#{h[page["dir"] || "auto"]}">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<base href="#{h[page["url"]]}">
<title>#{h[page["title"]]}</title>
<style>
body { max-width: 42rem; margin: 4rem auto; padding: 0 1.5rem;
font: 1rem/1.6 system-ui, sans-serif; }
img, video, figure { max-width: 100%; height: auto; }
pre { overflow-x: auto; }
.meta { color: #666; font-size: 0.875rem; }
</style>
</head>
<body>
<article>
<h1>#{h[page["title"]]}</h1>
<p class="meta">
#{h[[page["byline"], page["site_name"], page["saved_at"]].compact.join(" · ")]}<br>
<a href="#{h[page["url"]]}">#{h[page["url"]]}</a>
</p>
#{page["content"]}
</article>
</body>
</html>
HTML
end
pages = JSON.parse(File.read(input))
FileUtils.mkdir_p(output)
taken = Set.new
pages.each do |page|
name = unique_filename(page, taken)
File.write(File.join(output, "#{name}.html"), document(page))
end
puts "Wrote #{pages.size} files to #{output}"