Introduction

What is `mdbook-utils`? What is it used for?

The mdbook-utils command-line tool manages links, reference definitions, and code blocks in large collections of Markdown files, especially mdbook source directories. It is the companion tool for the Rust How-to book (github).

mdbook-utils is useful for the following:

centralize all reference definitions in one file to make Markdown files more readable and ease link maintenance,
replace simple Markdown links by badges,
identify duplicate or broken links,
generate a sitemap file for your book or website,
extract fenced code bocks embedded into the Markdown to separate files for easier formatting, debugging and testing,
replace code examples by mdbook #include statements,
conversely replace includes by the file contents.

mdbook-utils' underlying library also exposes a public API that may be used from your code.

Key Links

mdbook-utils (GitHub repo) (docs.rs) (crates.io) (user guide - this book)

Installation

Install the command-line tool using cargo:

cargo install mdbook-utils

then invoke mdbook-utils at a shell prompt.

For the bleeding-edge development version, use:

cargo install --git https://github.com/john-cd/mdbook-utils

To uninstall the tool, enter the following in a shell:

cargo uninstall mdbook-utils

Definitions

mdbook is a command-line tool to create books with Markdown. It is commonly used for Rust user guides, such as the Rust book and the Rust How-to book.

Markdown links and reference definitions

Markdown is a lightweight, readable markup language for writing structured documents.

A Markdown link can be an autolink, e.g. <https://example.com>, an inline link like [The user will see this](https://example.com), or a reference-style link: [The user will see this][thisisthelabel].

A reference-style link requires a reference definition with a matching label thisisthelabel: https://example.com/ on a separate line.

Images and badges

Images can be inserted using ![Image alternative text](link/to/image.png) or, reference-style, ![Image][1] followed by a reference definition [1]: <http://url/b.jpg>.

More details may be found in the CommonMark documentation.

A status badge is a small image that provides at-a-glance information, for example the build status of a code repository. Badges are commonly displayed on GitHub READMEs and inserted in mdbook documentation as links to a crate's docs.rs documentation, GitHub repo, or crates.io page. More information about badges may be found in the awesome-badges repo and in the shields.io documentation.

There is no "badge" concept in the Markdown specification. Badges are simply clickable images e.g. [ ![image-alt-text](link-to-image) ](link-to-webpage).

Code blocks and includes

Markdown fenced code blocks (we will call them code examples as well) are inserted between two code fences (e.g. sets of triple backticks), with an optional info string (a.k.a. attributes ) after the first backtick group:

```rust
fn main() {}
```

mdbook allows including files into your book via include statements. mdbook interprets included files as Markdown. Since the include syntax is usually used for inserting code snippets and examples, it is often wrapped between two sets of backticks.

```rust
{{# include file.rs}}
```

Markdown links and reference definitions

Markdown is a lightweight, readable markup language for writing structured documents.

A reference-style link requires a reference definition with a matching label thisisthelabel: https://example.com/ on a separate line.

Images and badges

Images can be inserted using ![Image alternative text](link/to/image.png) or, reference-style, ![Image][1] followed by a reference definition [1]: <http://url/b.jpg>.

More details may be found in the CommonMark documentation.

There is no "badge" concept in the Markdown specification. Badges are simply clickable images e.g. [ ![image-alt-text](link-to-image) ](link-to-webpage).

Code blocks and includes

```rust
fn main() {}
```

```rust
{{# include file.rs}}
```

Usage

Run the tool without arguments to display the the list of commands:

Tools to manage links, reference definitions, and code examples in Markdown files, especially `mdbook` source directories.

Usage: mdbook-utils [OPTIONS] <COMMAND>

Commands:
  refdefs   Manage reference definitions
  links     Manage links
  markdown  Manage code blocks (embedded examples) and includes
  sitemap   Generate a sitemap.xml file from the list of Markdown files in a source directory
  debug     Parse the entire Markdown code as events and write them to a file
  help      Print this message or the help of the given subcommand(s)

Options:
  -y, --yes      Automatically answer `yes` to any user confirmation request
  -h, --help     Print help
  -V, --version  Print version

In turn, most commands offer a menu of subcommands.

Reference Definitions

mdbook-utils refdefs offers the following subcommands:

Manage reference definitions

Usage: mdbook-utils refdefs [OPTIONS] <COMMAND>

Commands:
  write   Write existing reference definitions to a file
  badges  Generate badges (reference definitions) for e.g. Github links
  help    Print this message or the help of the given subcommand(s)

Options:
  -y, --yes   Automatically answer `yes` to any user confirmation request
  -h, --help  Print help

Links

mdbook-utils links currently offers two main subcommands:

Manage links

Usage: mdbook-utils links [OPTIONS] <COMMAND>

Commands:
  write-all     Write all existing links to a Markdown file
  write-inline  Write all existing inline / autolinks (i.e., not written as reference-style links) to a Markdown file
  help          Print this message or the help of the given subcommand(s)

Options:
  -y, --yes   Automatically answer `yes` to any user confirmation request
  -h, --help  Print help

Markdown

mdbook-utils markdown deals with fenced code blocks and includes:

Manage code blocks (embedded examples) and includes

Usage: mdbook-utils markdown [OPTIONS] <COMMAND>

Commands:
  extract-code-examples              Copy Rust code examples from the Markdown into .rs files
  replace-code-examples-by-includes  Replace Rust code examples from the Markdown by #include statements
  replace-includes-by-contents       Replace #include statements by the file contents
  remove-includes                    Remove #include statements (and replace them by a hard-coded string)
  help                               Print this message or the help of the given subcommand(s)

Options:
  -y, --yes   Automatically answer `yes` to any user confirmation request
  -h, --help  Print help

mdbook-utils sitemap and mdbook-utils debug do not have subcommands.

Command-line options

Command-line options vary by subcommand and include -o to set the path of the output file; -m to set the path of the source Markdown directory (./src or ./drafts by default, depending on the subcommand); -c to set the path to the directory containing the Cargo.toml that declares the dependencies (Rust crates) used in your book; and -t to set the path to the destination directory.

-y is a global option that skips confirmation dialogs and is useful when calling mdbook-utils from a script.

Use mdbook-utils <command> <subcommand> --help or help <command> <subcommand> for more details.

The following illustrates options for mdbook-utils sitemap:

Generate a sitemap.xml file from the list of Markdown files in a source directory

Usage: mdbook-utils sitemap [OPTIONS]

Options:
  -m, --markdown-dir <DIR>  Source directory containing the source Markdown files
  -b, --base-url <URL>
  -o, --output <FILE>       Path of the file to create
  -y, --yes                 Automatically answer `yes` to any user confirmation request
  -h, --help                Print help

Reference Definitions

mdbook-utils refdefs offers the following subcommands:

Manage reference definitions

Usage: mdbook-utils refdefs [OPTIONS] <COMMAND>

Commands:
  write   Write existing reference definitions to a file
  badges  Generate badges (reference definitions) for e.g. Github links
  help    Print this message or the help of the given subcommand(s)

Options:
  -y, --yes   Automatically answer `yes` to any user confirmation request
  -h, --help  Print help

Links

mdbook-utils links currently offers two main subcommands:

Manage links

Usage: mdbook-utils links [OPTIONS] <COMMAND>

Commands:
  write-all     Write all existing links to a Markdown file
  write-inline  Write all existing inline / autolinks (i.e., not written as reference-style links) to a Markdown file
  help          Print this message or the help of the given subcommand(s)

Options:
  -y, --yes   Automatically answer `yes` to any user confirmation request
  -h, --help  Print help

Markdown

mdbook-utils markdown deals with fenced code blocks and includes:

Manage code blocks (embedded examples) and includes

Usage: mdbook-utils markdown [OPTIONS] <COMMAND>

Commands:
  extract-code-examples              Copy Rust code examples from the Markdown into .rs files
  replace-code-examples-by-includes  Replace Rust code examples from the Markdown by #include statements
  replace-includes-by-contents       Replace #include statements by the file contents
  remove-includes                    Remove #include statements (and replace them by a hard-coded string)
  help                               Print this message or the help of the given subcommand(s)

Options:
  -y, --yes   Automatically answer `yes` to any user confirmation request
  -h, --help  Print help

Command-line options

-y is a global option that skips confirmation dialogs and is useful when calling mdbook-utils from a script.

Use mdbook-utils <command> <subcommand> --help or help <command> <subcommand> for more details.

The following illustrates options for mdbook-utils sitemap:

Generate a sitemap.xml file from the list of Markdown files in a source directory

Usage: mdbook-utils sitemap [OPTIONS]

Options:
  -m, --markdown-dir <DIR>  Source directory containing the source Markdown files
  -b, --base-url <URL>
  -o, --output <FILE>       Path of the file to create
  -y, --yes                 Automatically answer `yes` to any user confirmation request
  -h, --help                Print help

Configuration

Each subcommand uses defaults that are overwritten by values in book.toml (if present), by environment variables (if set), or command-line options (the latter trumps the former).

You may export environment variables manually or store them in a .env file, which will be read automatically:

# Root directory of the book
# `book.toml` is looked up in BOOK_ROOT_DIR_PATH, if set,
# in the current working directory otherwise.
export BOOK_ROOT_DIR_PATH=./test_book/

# Markdown source directory
export MARKDOWN_DIR_PATH=./test_book/src/

# Directory where mdbook outputs the book's HTML and JS;
# typically ./book/ or ./book/html/
export BOOK_HTML_BUILD_DIR_PATH=./test_book/book/

# Directory where `mdbook` outputs the book's fully expanded Markdown,
# i.e. with all includes resolved, when `[output.markdown]` is added to `book.toml`.
# It is typically ./book/markdown/.
export BOOK_MARKDOWN_BUILD_DIR_PATH=./test_book/book/markdown/

# Directory where `Cargo.toml` may be found
export CARGO_TOML_DIR_PATH=./test_book/book/code/

# Default destination directory for mdbook-utils outputs.
export DEFAULT_DEST_DIR_PATH=./test_book/temp/

# Base url of the website where the book will be deployed
# (used to build sitemaps)
export BASE_URL=http://myexample.com/some_book/

You may also set the RUST_LOG environment variable to display the logs.

See cli/config.rs in the GitHub repo for more details.

Note: mdbook-utils is not a mdbook mdbook-preprocessor or mdbook-backend at this point.

Public API

To use the library in your code, add the crate to your Cargo.toml as usual:

cargo add mdbook-utils

and peruse its documentation.

Note that cargo changes the dash into an underscore, thus insert use mdbook_utils::*; or similar into your code.

Contributing

Pull requests, comments, and issue submissions are actively encouraged!

Repo structure

mdbook-utils is written in Rust and follows the typical cargo package layout.

The source code is in the src folder. The main executable is in main.rs and the cli module. It calls the API in lib.rs.
A simple test mdbook book is found in test_book.
The user guide' sources are in user_guide.
The Dev Container and Docker (Compose) configuration files are found in .devcontainer.
- devcontainer.json uses Docker Compose (configured in compose.yaml and compose.override.yaml), which in turn runs a container from Dockerfile.
.github contains the continuous integration (GitHub Actions) workflow.

Development Setup

Using VS Code

Clone the repo and open the folder in VS Code. Edit .devcontainer/.env if needed. VS Code should prompt you to open the code in a Docker container, which installs Rust tooling automatically. Make sure you have previously installed the following:

Dev Container extension
Docker Desktop (or at least the Docker engine).

Note that opening the code folder in VS Code using Dev Containers may take a little while the first time around.

Other

If you are not using VS Code, install the Dev Container CLI, use docker compose directly (see below), or simply install the required tools on your local machine.

The following works with Ubuntu (including WSL):

sudo apt-get update
rustup update
rustup component add clippy
rustup toolchain install nightly
rustup component add rustfmt --toolchain nightly
cargo install just
cargo install mdbook

Review .devcontainer/Dockerfile for other optional dependencies.

Build and test the code and user guide

The just command runner is configured to simplify compilation and testing.

Type just at a shell prompt for a list of commands:

just clean  # Clean Cargo's `target` and mdbook's `book` folders

just fmt    # Format all code

just check  # Check whether the code can compile

just build  # Build all code and books

just clippy # Scan all code for common mistakes

just test   # Test all code and books

just run <command>  # Run the tool

just doc    # Build and display the `cargo doc` documentation

just serve  # Display the user guide

just prep   # Run all the steps required before pushing code to GitHub

just update # Update Cargo.lock dependencies

Docker Compose

Test the Docker Compose setup used during development (which Dev Containers runs) with:

cd ./.devcontainer
docker compose build   # uses compose.yaml and compose.override.yaml by default
docker compose up -d
# or simply
docker compose up --build -d

Use the following commands to build and test the code and user guide using the Continuous Integration configuration:

docker compose -f .devcontainer/compose.yaml -f .devcontainer/compose-ci.yaml run \
--build --rm mdbook-utils

Publish to crates.io

Manual method

Go to crates.io, sign in, and create an API token in Account Settings > API Tokens.
Use cargo login <token> to save the token in $CARGO_HOME/credentials.toml.
just build; just clippy; just run; just doc; cargo package --locked
Review the packaging output in target/mdbook-utils/package or use cargo package --list.
When ready, cargo publish --locked --dry-run; cargo publish --locked

Docker Compose method

Pass the publish.sh script (and required argument -y) as a command to docker compose run.
Pass the CRATES_TOKEN env. variable (which is used by publish.sh) to Docker Compose using --env.

export CRATES_TOKEN="<token from crates.io>"
docker compose -f .devcontainer/compose.yaml -f .devcontainer/compose-ci.yaml run \
       --rm --env CRATES_TOKEN mdbook-utils .devcontainer/publish.sh -y