We’ve covered how to validate UI procedures and API procedures with Doc Detective, but what about shell commands and scripts? After all, not every procedure is navigating through UIs or making API calls.
Today, I’ll walk you through validating both shell commands and scripts, using the commands from the UI validation tutorial as a starting point. We’ll use Doc Detective to validate that the commands work as expected and show how you can run scripts to validate code or commands written in other languages.
(Disclosure: I’m the creator of Doc Detective, but I’ll try to keep this as objective as possible. If you have questions, feel free to reach out in the comments, on social media, or on Discord.)
This is going to get a little meta!
The setup
In the UI validation tutorial, we validated a procedure for searching for American Shorthair kittens on Google, but if you look at all the steps involved, a lot was going on:
- Install Doc Detective.
- Save the procedure as a Markdown file.
- Write a test in JSON.
- Run the test.
- Validate the results.
The secret to validating shell commands and scripts is Doc Detective’s runShell action. This action lets you run a shell command or script and validate the results.
We’ll do this a few ways: first, we’ll test individual commands and validate them one at a time, then we’ll create and run a script to perform the whole UI validation procedure in one go, and finally I’ll show you how to validate scripts in whatever language you prefer. Before that though, let’s get Doc Detective ready on your machine.
Install Doc Detective
If you already have Doc Detective installed, skip to the next section.
Before you can get Doc Detective running, you need the prerequisites:
- Node.js 18 or higher. I like using Node Version Manager, or you can use the official installers.
- Git
Once you have those installed, you can clone Doc Detective to your machine and install dependencies:
git clone https://github.com/doc-detective/doc-detective.git
cd doc-detective
npm install
Validate individual commands
Running shell commands and scripts with Doc Detective is straightforward. The command or script runs in your device’s native shell (cmd on Windows and bash on macOS and Linux), and if the command or script returns a non-zero exit code, the test fails.
We can test this out with a single test that has two steps. The first step runs a command that should pass, and the second step runs a command that should fail. Let’s save this test as command-validation.spec.json:
{
"tests": [
{
"id": "single-command-validation",
"description": "The first step should pass. The second step should fail.",
"steps": [
{
"action": "runShell",
"command": "echo 'Hello, world!'"
},
{
"action": "runShell",
"command": "exit 1"
}
]
}
]
}
Now, let’s run the test:
npm run runTests -- -i command-validation.spec.json
Great! We see that the first command passed and the second command failed. This is exactly what we expected because the second command exits with a non-zero exit code. But while these steps test individual commands, you’ll often want to run a script to test multiple commands at once. Let’s look at how to do that.
Validate a bash script
Note: The following commands are for a bash shell. If you’re using Windows, make sure bash is available from cmd, such as through Windows Subsystem for Linux or Git Bash.
Let’s look at each of the steps in the UI validation procedure and identify commands that would complete them:
-
Install Doc Detective:
git clone https://github.com/doc-detective/doc-detective.git cd doc-detective npm install -
Save the procedure as a Markdown file:
echo -e 'To search for American Shorthair kittens,\n\n1. Go to [Google](https://www.google.com).\n\n2. In the search bar, enter \"American Shorthair kittens\", then press Enter.\n\n' > kitten-search.md -
Write a test in JSON:
echo -e '{ "tests": [ { "steps": [ { "action": "goTo", "url": "https://www.google.com" }, { "action": "find", "selector": "[title=Search]", "click": true }, { "action": "typeKeys", "keys": ["American Shorthair kittens", "$ENTER$"] }, { "action": "wait", "duration": 5000 }, { "action": "saveScreenshot", "path": "search-results.png" } ] } ]}' > kitten-search.spec.json -
Run the test:
npm run runTests -- -i kitten-search.spec.json -
Validate the results, in this case by checking that the screenshot and test results files exist:
FILE=search-results.png; if [ -f "$FILE" ]; then echo "File $FILE exists."; else false; fi for file in testResults-*.json; do [ -e "$file" ] && echo "File $file exists." || false; break; done
If you wanted to validate each of these commands individually, you could, but that’s a lot of work. Instead, let’s create a bash script. Save the following script as ui-validation.sh:
# ui-validation.sh
#
# !/bin/bash
git clone https://github.com/doc-detective/doc-detective.git
cd doc-detective
npm install
echo -e 'To search for American Shorthair kittens,\n\n1. Go to [Google](https://www.google.com).\n\n2. In the search bar, enter \"American Shorthair kittens\", then press Enter.\n\n' > kitten-search.md
echo -e '{ "tests": [ { "steps": [ { "action": "goTo", "url": "https://www.google.com" }, { "action": "find", "selector": "[title=Search]", "click": true }, { "action": "typeKeys", "keys": ["American Shorthair kittens", "$ENTER$"] }, { "action": "wait", "duration": 5000 }, { "action": "saveScreenshot", "path": "search-results.png" } ] } ]}' > kitten-search.spec.json
npm run runTests -- -i kitten-search.spec.json
FILE=search-results.png; if [ -f "$FILE" ]; then echo "File $FILE exists."; else exit 1; fi
for file in testResults-*.json; do [ -e "$file" ] && echo "File $file exists." || exit 1; break; done
Now, we can validate the whole procedure with a single command. Let’s save this test as ui-validation.spec.json:
{
"tests": [
{
"id": "ui-validation",
"description": "Validate the UI procedure for searching for American Shorthair kittens on Google.",
"steps": [
{
"action": "runShell",
"command": "bash ui-validation.sh"
}
]
}
]
}
Now that we have our test and script, let’s run it. Note that while the script is running, we won’t see any output in the terminal. Please be patient.
npm run runTests -- -i ui-validation.spec.json
This script is a little more complex than the others, but it’s a great way to validate a whole procedure in one go.
Validate other kinds of scripts
Script validation isn’t limited to bash. As long as your script can return a non-zero exit code, in can be in any language you like, whether that’s JavaScript, Python, Ruby, or whatever else. Any command you can run from the command line, you can validate with Doc Detective. Need to validate an SDK? Write a script and go for it.
For example, here’s a script that validates only the outputs of our UI procedure. Save this script as validate-kitten-search-outputs.js:
// validate-kitten-search-outputs.js
const fs = require('fs');
const FILE = 'search-results.png';
if (fs.existsSync(FILE)) {
console.log(`File ${FILE} exists.`);
} else {
process.exit(1);
}
const files = fs.readdirSync(process.cwd());
const testResults = files.filter(file => file.startsWith('testResults-'));
if (testResults.length > 0) {
console.log(`File ${testResults[0]} exists.`);
} else {
process.exit(1);
}
We can test this just as easily as our other script. Save the following test as js-validation.spec.json:
{
"tests": [
{
"id": "js-validation",
"description": "Validate outputs for searching for American Shorthair kittens on Google.",
"steps": [
{
"action": "runShell",
"command": "node validate-kitten-search-outputs.js"
}
]
}
]
}
And finally, run the test:
npm run runTests -- -i js-validation.spec.json
Just like the bash script and the individual commands before it, this script exits cleanly. You’re welcome to try deleting the output files and watching it fail too!
One more thing!
If you’re not running Doc Detective as part of a script, you can use runShell to help set up your test environment. For example, you can use runShell to install dependencies, spin up Docker containers, or perform other setup tasks before running your tests. This is a great way to automate your tests and keep them self-contained.
Conclusion
If you want to validate individual commands, to validate scripts in whichever language you require, or to automate your testing environment, Doc Detective is a powerful tool at your disposal. With Doc Detective, you can validate UIs, APIs, and everything in between to make sure your content works the way you and your users expect.
About the tool
Doc Detective is an open-source documentation testing framework. Doc Detective has multiple components to integrate with your workflows as you need it to:
- Doc Detective: A standalone tool to test docs against product UIs and APIs.
- Doc Detective Companion: A browser extension for Chrome and Firefox to help you build tests and find CSS selectors.
- Doc Detective Core: An NPM package that provides Doc Detective’s testing functionality.
Doc Detective is open to contributions, and you can join the chat on Discord. If you like what you see, you can ⭐ star the repositories on GitHub or ❤️ sponsor development via Open Collective or GitHub Sponsors.
Thanks for reading. If you like what you read here, sign up for the newsletter to stay up to date on Docs as Tests discussions, share this article with those you think would appreciate it, or sound off in the comments below.