Start doc/ directory, for high-level intra-Signal docs

.github/workflows/build_and_test.yml

@@ -86,6 +86,7 @@ jobs:
           - '.gitattributes'
           - '.editorconfig'
           - 'justfile'
+          - 'doc/**'
 
     - name: Check pattern completeness
       run: echo "::error file=.github/workflows/build_and_test.yml::File not included in any filter" && false

.github/workflows/docs.yml

@@ -0,0 +1,31 @@
+name: Docs
+
+env:
+  MDBOOK_VERSION: "0.4.43"
+
+on:
+  push:
+    branches: [ main ]
+    paths: ['doc/**', '.github/workflows/docs.yml']
+  pull_request:
+    paths: ['doc/**', '.github/workflows/docs.yml']
+
+jobs:
+  docs:
+    name: Check docs
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+      with:
+        submodules: recursive
+
+    - name: Download mdbook ${{ env.MDBOOK_VERSION }}
+      run: mkdir ~/bin && curl -sSL https://github.com/rust-lang/mdBook/releases/download/v${{ env.MDBOOK_VERSION }}/mdbook-v${{ env.MDBOOK_VERSION }}-x86_64-unknown-linux-gnu.tar.gz | tar -xz --directory ~/bin
+
+    - run: ~/bin/mdbook build
+      working-directory: doc
+
+    - run: ~/bin/mdbook test
+      working-directory: doc

doc/.gitignore

@@ -0,0 +1 @@
+book

doc/book.toml

@@ -0,0 +1,4 @@
+[book]
+language = "en"
+multilingual = false
+title = "libsignal"

doc/src/README.md

@@ -0,0 +1,24 @@
+# Introduction
+
+libsignal contains platform-agnostic APIs used by the official Signal clients and servers, exposed as a Java, Swift, or TypeScript library. The underlying implementations are written in Rust.
+
+This documentation is meant primarily for developers working at Signal who will be calling the libsignal APIs, and secondarily for those who maintain libsignal itself. It's meant to be a high-level guide to what's available, and generally won't contain implementation details or detailed API-by-API descriptions. It's *not* meant to be any sort of promise or commitment across versions of the library.
+
+That is, if you're outside of Signal, please don't read too much into these.
+
+
+## Viewing the book
+
+First, [install mdBook](https://rust-lang.github.io/mdBook/guide/installation.html). Then, from the `doc` directory:
+
+```console
+% mdbook serve
+2025-01-21 18:05:27 [INFO] (mdbook::book): Book building has started
+2025-01-21 18:05:27 [INFO] (mdbook::book): Running the html backend
+2025-01-21 18:05:27 [INFO] (mdbook::cmd::serve): Serving on: http://localhost:3000
+2025-01-21 18:05:27 [INFO] (mdbook::cmd::watch::poller): Watching for changes...
+2025-01-21 18:05:27 [INFO] (warp::server): Server::run; addr=[::1]:3000
+2025-01-21 18:05:27 [INFO] (warp::server): listening on http://[::1]:3000
+```
+
+Now you can open the URL listed (probably <http://localhost:3000>) and view the rendered book. This is also a "watch" mode, which is convenient when editing the book---just save and watch the page reload.

doc/src/SUMMARY.md

@@ -0,0 +1,5 @@
+# Summary
+
+[Introduction](README.md)
+
+- [Backups](backups/README.md)

doc/src/backups/README.md

@@ -0,0 +1,73 @@
+# Backups
+
+libsignal has a handful of APIs related to backups:
+
+- [Account keys](#account-keys)
+- [Backup validation](#backup-validation)
+- [BackupAuthCredential](#backupauthcredential)
+
+## Account keys
+
+Going forward, a number of account keys, including backup keys, will be derived from an *account entropy pool,* a 64-character alphanumeric ASCII string. While the AEP is represented as a plain String in libsignal APIs, methods to work with it can be found on `AccountEntropyPool`, including generation, validation, and derivation of other keys.
+
+Derived from the account entropy pool is the `BackupKey`, a strongly-typed 32-byte blob that is used for all aspects of backups. There are many keys and identifiers that are derivable from a BackupKey.
+
+Finally, the key specifically used to encrypt backup files is the `MessageBackupKey`, another strongly-typed object that consists of an HMAC key and an AES key for signing and encrypting the backup, respectively.
+
+![](account-keys.svg)
+
+
+## Backup validation
+
+There are a few different APIs that work with backup files:
+
+### MessageBackup: Bulk validation of an encrypted backup file stream
+
+Takes in an encrypted input stream and produces validation results. Hard errors are thrown as exceptions, soft errors (unknown fields) are returned for manual checking or logging.
+
+Validation makes two passes over the stream to verify its contained MAC both before and after parsing, so the relevant APIs take a callback to *produce* streams rather than a single stream object. There is no guarantee that the first stream has been fully consumed before the second is produced, so do not reuse the same stream object.
+
+Provided by:
+
+- `MessageBackup` in Java
+- `validateMessageBackup` in Swift
+- `validate` in the `MessageBackup` module in TypeScript
+
+
+### OnlineBackupValidator: Validation of a backup as it's being made
+
+Feed frames into the validator to check them one by one, but don't forget to finalize the backup (`close` in Java, `finalize` in Swift and TypeScript) to run the end-of-file checks!
+
+This is usually going to be faster than the bulk validation because it skips the decryption and decompression steps, but of course this also means the encryption and compression of the backup being created aren't tested.
+
+Only logs soft errors rather than returning them in a manipulatable form.
+
+
+### ComparableBackup: "Canonicalization" of a backup for testing purposes
+
+Takes an **unencrypted** backup as input and produces a JSON string as output, which can be formatted and diffed as JSON or just line-by-line. Don't forget to check for any unknown fields that get reported as well; they won't be included in the JSON and could invalidate any comparison.
+
+Note that this format should not be considered stable (i.e. don't persist it or try to parse it). It's only intended for comparing two backups to each other; the usual way this is used is for an app to import Backup A and then immediately export its data as Backup B, then verify that the results are identical.
+
+
+## BackupAuthCredential
+
+The BackupAuthCredential types follow the usual zkgroup construction from the client's perspective, similar to the other AuthCredential variants:
+
+```pseudocode
+let requestContext = BackupAuthCredentialRequestContext.create(backupKey, aci)
+let httpResponse = goRequestBackupAuthCredentialsFromServer(requestContext.getRequest())
+for each response in httpResponse {
+    let expectedRedemptionTime = start of each day
+    let credential = requestContext.receive(response, expectedRedemptionTime, serverParams)
+    log(credential.backupLevel, credential.type)
+    goSaveCredential(credential, expectedRedemptionTime)
+}
+```
+
+```pseudocode
+let presentation = credential.present(serverParams)
+goDoSomeBackupOperation(presentation)
+```
+
+More information on these credentials in the client/server docs.