Validate README Commands with Innovation Engine

by Manny Silva

Technical writer by day and engineer by night, Manny Silva is Head of Documentation at Skyflow, the creator of Docs as Tests, and the creator of Doc Detective. He's passionate about intuitive and scalable developer experiences and likes diving into the deep end as the 0th developer.

3 min read July 5, 2024

Prerequisites

This tutorial is slightly more advanced than other recent ones. Let’s make sure you’re ready to go. You need

A bash shell.
Go installed on your machine.

Set up Innovation Engine

To install and build Innovation Engine, you need to download or build the ie binary.

On Linux, you can download the binary and make it executable with the following commands:

VERSION="latest"
wget -q -O ie https://github.com/Azure/InnovationEngine/releases/download/$VERSION/ie
chmod +x ie

On macOS (or if you like to go the hard road on Linux), you need to clone the repo and run the build command:

git clone https://github.com/Azure/InnovationEngine;
cd InnovationEngine;
make build-ie;
cd ./bin

Sample document and formatting

Here’s a simple tutorial in Markdown that uses ReqRes, a testing API that uses fake data. The tutorial focuses on two common API requests: fetching a list of users and creating a new user.

Each command is in a bash code block, and the expected output is in a JSON code block below it. By default, Innovation Engine runs commands in bash code blocks to make sure they run as expected. For example:

    ```bash
    curl -X POST https://reqres.in/api/users \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Jane Doe",
        "job": "Developer"
      }'
    ```

To validate responses, you need to add an HTML-style comment before the immediately following code block and include the expected_similarity key and a value between 0 and 1. This value represents the similarity between the expected output and the actual output, allowing for fuzzy matching.

    <!--expected_similarity=0.8-->
    ```
    {"name":"Jane Doe","job":"Developer","id":"557","createdAt":"2024-07-04T21:58:42.684Z"}
    ```

Note: Response code blocks can’t have language tags, so you can’t use json or bash here. If there’s a language tag, Innovation Engine won’t consider it a response block.

Testing the commands

Using Innovation Engine, we can test the commands in the code blocks to make sure they run correctly.

Download reqres_tutorial.md into the same directory at the ie binary.

Run the following command to test the commands in the tutorial:

 ./ie test reqres_tutorial.md

This command reads the Markdown file, executes the curl commands, and validates the output.

The output should look similar to this:

 $ ./ie test reqres_tutorial.md
 bash:$ curl https://reqres.in/api/users

 {"page":1,"per_page":6,"total":12,"total_pages":2,"data":[{"id":1,"email":"[email protected]","first_name":"George","last_name":"Bluth","avatar":"https://reqres.in/img/faces/1-image.jpg"},{"id":2,"email":"[email protected]","first_name":"Janet","last_name":"Weaver","avatar":"https://reqres.in/img/faces/2-image.jpg"},{"id":3,"email":"[email protected]","first_name":"Emma","last_name":"Wong","avatar":"https://reqres.in/img/faces/3-image.jpg"},{"id":4,"email":"[email protected]","first_name":"Eve","last_name":"Holt","avatar":"https://reqres.in/img/faces/4-image.jpg"},{"id":5,"email":"[email protected]","first_name":"Charles","last_name":"Morris","avatar":"https://reqres.in/img/faces/5-image.jpg"},{"id":6,"email":"[email protected]","first_name":"Tracey","last_name":"Ramos","avatar":"https://reqres.in/img/faces/6-image.jpg"}],"support":{"url":"https://reqres.in/#support-heading","text":"To keep ReqRes free, contributions towards server costs are appreciated!"}}
 bash:$ curl -X POST https://reqres.in/api/users \
   -H "Content-Type: application/json" \
   -d '{
     "name": "Jane Doe",
     "job": "Developer"
   }'

 {"name":"Jane Doe","job":"Developer","id":"334","createdAt":"2024-07-04T22:17:31.936Z"}

The output shows the commands being executed and the expected output. If the actual output matches the expected output within the similarity threshold, the test passes. If the output isn’t similar enough to the expected output, the test fails.

Wrapping up

By validating your documentation with Innovation Engine, you can make sure that your code examples are always accurate and functional. While Innovation Engine is limited to testing CLI docs, it can be integrated into your CI/CD pipeline to automate your testing and make sure that your most important CLI-based docs, like installation instructions, are always up-to-date and accurate.

About the tool

Azure’s Innovation Engine is a CLI tool that transforms markdown documentation into executable scripts. This allows for interactive tutorials, automated tests, and execution of CLI commands, making sure your documentation remains accurate and up-to-date. For more information, see the Innovation Engine GitHub repository.

Thanks for reading. If you like what you read here, sign up for the newsletter to stay up to date on Docs as Tests discussions, share this article with those you think would appreciate it, or sound off in the comments below.

Categories