Building a custom WebdriverIO reporter using documented and undocumented WebdriverIO features

WebdriverIO (or Wdio) is a great open-source tool for end-to-end UI testing. It provides Selenium bindings for NodeJS, along with a set of different services and reporters.

But what if none of them serves your purpose? E.g., reporter doesn’t show test duration time or doesn’t attach failure screenshots? You have at least two options: make a pull request and ask the package’s maintainer to review it and merge it, or develop your own reporter (or service).

In this post, we will show you step-by-step how to create a custom Wdio reporter using TypeScript and NodeJS. We’ll be using documented Wdio features as well as undocumented Wdio features that we discovered along the way.

Stack

We will be using the following stack:

  • Typescript 2.4.2
  • NodeJS 8.9.1 with yarn as package manager

Development

Let’s start developing wdio-simple-reporter.

  1. Create a new directory and name it wdio-simple-reporter.
  2. Create a package.json file in the root of the wdio-simple-reporter folder. To do this, you can either (1) run yarn init command, or (2) set it up manually. We recommend the first option because it’s easier, and you can always edit package.json manually in the future.
  3. Install devDependencies.
     yarn add typescript @types/node @types/mkdirp -D
    
  4. Install dependencies.
     yarn add mkdirp -S
    
  5. Add npm scripts to package.json.
     "scripts": {
       "build": "rm -rf ./lib && tsc"
     }
    

    build - to compile ts files into js and store them in the lib folder (see outDir options in the tsconfig.json file).

    Your final package.json file should look something like this:

     {
       "name": "wdio-simple-reporter",
       "version": "0.0.1",
       "description": "A WebdriverIO plugin. Reports results in 'json' format.",
       "main": "lib/reporter.js",
       "scripts": {
         "build": "rm -rf ./lib && tsc"
       },
       "keywords": [
         "json",
         "reporter",
         "webdriverio",
         "wdio",
         "wdio-plugin",
         "wdio-reporter"
       ],
       "license": "MIT",
       "devDependencies": {
         "@types/mkdirp": "0.3.29",
         "@types/node": "7.0.11",
         "typescript": "2.4.2"
       },
       "dependencies": {
         "mkdirp": "0.5.1"
       }
     }
    
  6. Create tsconfig.json file and place it in the root of the wdio-simple-reporter folder.

    Why do we need tsconfig.json file?

    Each TypeScript project’s root folder should contain a tsconfig.json. Its presence indicates that the directory is the root of a TypeScript project. Also, the tsconfig.json file specifies options for project compilation.

    In our case, we need only basic tsconfig.json in order to set up and configure the project.

     {
       "compilerOptions": {
         "moduleResolution": "node",
         "target": "es5",
         "noImplicitAny": true,
         "outDir": "lib",
         "lib": [
           "es6"
         ]
       },
       "exclude": [
         "node_modules"
       ],
       "include": [
         "src/**/*.ts"
       ],
       "rootDirs": [
         "."
       ]
     }
    

    Let’s take a closer look at the compiler options:

    "moduleResolution": "node" - use Node.js modules style resolution. See documentation for more details.

    "target": "es5" - compile source code in ES5.

    "noImplicitAny": true - warn on expressions and declarations with an implied ‘any’ type.

    "outDir": "lib" - output directory where compiled files will be stored.

    "lib": ["es6"] - libs to be included during compilation. In our case, we only need es6 lib to be able to use Array.prototype.find method (see getRunner helper function).

    "exclude": ["node_modules"] - files to be excluded from compilation.

    "include": ["src/**/*.ts"] - files to be included in compilation.

    "rootDirs": ["."] - folder whose content represents the structure of the project at runtime.

    More information about tsconfig.json can be found here.

    Detailed information about compiler options can be found here.

  7. Now we are ready to start developing the custom Wdio reporter.

    If you take a look at the WebdriverIO custom reporter documentation, you will find that it says:

    You can register event handler for several events which get triggered during the test. All these handlers will receive payloads with useful information about the current state and progress. The structure of these payload objects depend on the event and are unified across the frameworks (Mocha, Jasmine and Cucumber). Once you implemented your custom reporter it should work for all frameworks.

    It also provides a list of all possible events you can register an event handler on:

       'start'
       'suite:start'
       'hook:start'
       'hook:end'
       'test:start'
       'test:end'
       'test:pass'
       'test:fail'
       'test:pending'
       'suite:end'
       'end'
     

    However, this list is outdated. Half of the events are missing. If you take a look at the Wdio BaseReporter, you will find that there are a lot more events.

    So, here is the list of all possible events you can register an event handler on:

       'start'
       'runner:start'
       'runner:init'
       'runner:beforecommand'
       'runner:command'
       'runner:aftercommand'
       'runner:result'
       'runner:screenshot'
       'runner:log'
       'runner:end'
       'suite:start'
       'hook:start'
       'hook:end'
       'test:start'
       'test:end'
       'test:pass'
       'test:fail'
       'test:pending'
       'suite:end'
       'error'
       'end'
     

    This isn’t the only way to obtain runner-, suite-, or test-related data. You can use the baseReporter object that contains data about all tests run. We’ll show you how to do that later.

    Here is another useful Wdio feature that is not mentioned in the documentation: starting with Wdio v4.7.0, you can send events to runner’s reporters. For example, let’s say you want to log the name of the feature that is covered by your test.

    In your test:

       process.send({
         event: 'log-feature-name',
         data: 'User login'
       })
    

    In reporter:

       this.on('log-feature-name', (data: Data) => console.log(data))
    
  8. Let’s create reporter.ts file inside wdio-simple-reporter/src folder and import the following:

       import * as mkdirp from 'mkdirp'
       import * as path from 'path'
       import * as fs from 'fs'
       import * as events from 'events'
       import * as Utils from './utils'
    

    Add types.

       // 'runner: start' event
       type RunnerStart = {
         readonly cid: string
         readonly specs: string[]
         readonly capabilities: Capabilities
         readonly specHash: string
       }
    
       // 'runner: end' event
       type RunnerEnd = {
         readonly cid: string
         readonly specHash: string
       }
    
       // 'test: pass | fail | pending' event
       type Test = {
         readonly cid: string
       }
    
       // 'runner:screenshot' event
       type Screenshot = {
         readonly filename: string
         readonly uid: string
         absolutePath: string
       }
    
       type SpecStats = {
         readonly suites: { [key: string]: SuiteStats }
       }
    
       type SuiteStats = {
         readonly tests: { [key: string]: TestStats }
       }
    
       type TestStats = {
         readonly uid: string
         failureScreenshot: Screenshot
       }
    
       type RunnerResult = {
         readonly cid: string
         readonly capabilities: Capabilities
         readonly specFilePath: string[]
         readonly specFileHash: string
         readonly runnerTestsNumber: TestsNumber
         suites: SuiteStats[]
       }
    
       type TestsNumber = {
         passing: number
         pending: number
         failing: number
       }
    
       type Capabilities = {
         readonly browserName: string
         readonly version: string | number
         readonly platform: string
         readonly platformName: string
         readonly platformVersion: string | number
       }
    
       type Config = {
         readonly screenshotPath: string
       }
    
       type ConfigOptions = {
         readonly resultsDir: string
         readonly resultsFile: string
       }
    
       type BaseReporter = {
         readonly stats: ReporterStats
       }
    
       type ReporterStats = {
         readonly runners: { [key: string]: RunnerStats }
       }
    
       type RunnerStats = {
         readonly specs: { [key: string]: SpecStats }
       }
    

    Now, create SimpleReporter class and extend it from events.EventEmitter.

       export default class SimpleReporter extends events.EventEmitter { }
    

    Add (1) a default reporter name, (2) the path to default results directory, and (3) default results file name.

     /**
      * Initialize a new `Simple` test reporter.
      *
      * @param {Runner} runner
      * @api public
      */
     export default class SimpleReporter extends events.EventEmitter {
       public static reporterName = 'wdio-simple-reporter'
       private resultsDir = '../reports'
       private resultsFile = 'report.json'
     }
    

    Wdio launcher checks whether reporterName is a function or a string; otherwise, it will throw an error.

    Add objects that will store runners’ results and screenshots for failed tests.

     export default class SimpleReporter extends events.EventEmitter {
       public static reporterName = 'wdio-simple-reporter'
       private resultsDir = '../reports'
       private resultsFile = 'report.json'
       private runnerResults: RunnerResult[] = []
       private failureScreenshots: Screenshot[] = []
     }
    

    Wdio runner creates a separate runner for each spec file. We can use RunnerResult type and store all runners’ results in an array of elements of this type. The same approach is used for failure screenshots.

    Create a class constructor which will take baseReporter and configOptions as its parameters.

    baseReporter — wdio uses BaseReporter as its core reporter. So, this object contains runners, suites, and test results. With access to the baseReporter object we can easily get runners, suites and test results without registering event handlers.

    config — object that stores Wdio config (wdio.conf.js) options. We will use it for only one purpose: to get screenshotPath property, which contains path (relative or absolute) to failure screenshots folder.

    configOptions — object that represents reporterOptions (also from wdio.conf.js). In our case, it will be storing resultsDir and resultsFile values.

    Now SimpleReporter class should look like

     ...
     export default class SimpleReporter extends events.EventEmitter {
       public static reporterName = 'wdio-simple-reporter'
       private resultsDir = '../reports'
       private resultsFile = 'report.json'
       private runnerResults: RunnerResult[] = []
       private failureScreenshots: Screenshot[] = []
       constructor(
         private baseReporter: BaseReporter,
         private config: Config,
         private options: ConfigOptions
       ) {
         super()
       }
     }
    

    Note. Don’t forget to call super() constructor. Constructors for derived classes must contain a super call.

    Now we need to get access to base reporter stats (the object that contains run results for each runner).

     export default class SimpleReporter extends events.EventEmitter {
       ...
       constructor(
         private baseReporter: BaseReporter,
         private config: Config,
         private options: ConfigOptions
       ) {
         super()
    
         const baseReporterStats = this.baseReporter.stats
       }
     }
    

    Save path to screenshots folder. We will use it later to obtain the absolute path for each screenshot (see runner:screenshot event handler).

       export default class SimpleReporter extends events.EventEmitter {
       ...
       constructor(
         private baseReporter: BaseReporter,
         private config: Config,
         private options: ConfigOptions
       ) {
         super()
       }
         const screenshotPath = this.config.screenshotPath
         const screenshotsFolder = path.isAbsolute(screenshotPath)
           ? screenshotPath
           : path.join(process.cwd(), screenshotPath)
         const baseReporterStats = this.baseReporter.stats
       }
    

    Now it’s time to register the event handlers. We only need runner:start, runner:end, end, runner:screenshot, test:pass, test:fail, test:pending events.

    Register event handler for runner:start event.

    It will create currentRunnerResult object to store runner cid (unique identifier for a capability, also can be used to identify runner), capabilities (browser, platform, etc.), test counters (failed, passed and skipped), spec file hash, spec file path, and spec suites. Then currentRunnerResult will be stored in the runnerResults array.

     this.on('runner:start', (runner: RunnerStart) => {
       const cid = runner.cid
    
       const currentRunnerResult: RunnerResult = {
         cid,
         capabilities: runner.capabilities,
         runnerTestsNumber: {
           failing: 0,
           passing: 0,
           pending: 0
         },
         specFileHash: runner.specHash,
         specFilePath: runner.specs,
         suites: []
       }
    
       this.runnerResults.push(currentRunnerResult)
     })
    

    Register event handler for runner:end event.

    runner:end event will get all current runner’s suites. Using the helper function getAllTestSuites, select corresponding runner from runners storage (using helper function getRunner) and assign suites to runner’s suites property.

     this.on('runner:end', (runner: RunnerEnd) => {
       const cid = runner.cid
       const specHash = runner.specHash
       const suites: SuiteStats[] = this
         .getAllTestSuites(baseReporterStats.runners[cid].specs[specHash])
       this.getRunner(cid).suites = suites
     })
    

    Register event handler for end event.

    On end event we need to attach the failure screenshots to the corresponding tests (using failureScreenshot property) and save the results to a json file.

     this.on('end', () => {
       // attach screenshots to corresponding tests
       this.failureScreenshots.map(screenshot => this.runnerResults.map(runnerResult => {
         runnerResult.suites.map(suite => Object.keys(suite.tests).map(test => {
           if (suite.tests[test].uid === screenshot.uid)
             suite.tests[test].failureScreenshot = screenshot
         }))
       }))
    
       // save final results to json file
       const finalResultsDir = options.resultsDir
         ? options.resultsDir
         : path.join(__dirname, this.resultsDir)
       const finalResultsFile = options.resultsFile
         ? options.resultsFile
         : this.resultsFile
       const fullPath = path.join(finalResultsDir, finalResultsFile)
    
       try {
         if (!fs.existsSync(finalResultsDir)) mkdirp.sync(finalResultsDir)
         Utils.saveObjectToJson(this.runnerResults, fullPath)
       } catch (e) {
         console.error(`Failed to save report file: ${finalResultsFile}`, e)
       }
     })
    

    Register event handler for runner:screenshot event.

    If runner:screenshot is triggered, the screenshot will be saved in the screenshots storage.

     this.on('runner:screenshot', (screenshot: Screenshot) => {
       screenshot.absolutePath = path.join(screenshotsFolder, screenshot.filename)
       this.failureScreenshots.push(screenshot)
     })
    

    Register event handler for test:pass event.

    test:pass only increments the counter of successful tests for current runner.

     this.on('test:pass', (test: Test) =>
       this.getRunner(test.cid).runnerTestsNumber.passing++
     )
    

    Register event handler for test:fail event.

    test:fail only increments the counter of failed tests for current runner.

     this.on('test:fail', (test: Test) =>
       this.getRunner(test.cid).runnerTestsNumber.failing++
     )
    

    Register event handler for test:pending event.

    test:pending only increments the counter of skipped tests for current runner.

     this.on('test:pending', (test: Test) =>
       this.getRunner(test.cid).runnerTestsNumber.pending++
     )
    

    Add helper functions.

    getAllTestSuites - returns all test suites for current runner. Except ones with empty tests property.

     private getAllTestSuites(spec: SpecStats) {
       let suites: SuiteStats[] = []
    
       Object.keys(spec.suites)
         .map(suiteName => {
           const specSuites = spec.suites[suiteName]
           if (Object.keys(specSuites.tests).length !== 0)
             suites.push(specSuites)
         })
    
       return suites
     }
    

    getRunner - returns runner by its cid or throws an error if runner not found.

     private getRunner(cid: string) {
       const res = this.runnerResults.find(r => r.cid === cid)
       if (!res) {
         throw new Error(`Results with runner CID ${cid} not found.`)
       }
    
       return res
     }
    
  9. Create utils.ts file.

    We only need one util function to save results to json file.

     import * as fs from 'fs'
    
     export function saveObjectToJson(object: any, path: string) {
       try {
         fs.writeFileSync(path, JSON.stringify(object))
       } catch (e) {
         console.error(`Can not save results to file: ${path}`, e)
       }
     }
    

Usage and configuration

Just import reporter, add to the reporters array and configure resulting directory and file name (or skip this and default resultsDir and resultsFile will be used).

    // wdio.conf.js
    var simpleReporter = require('wdio-simple-reporter/lib/reporter').SimpleReporter
    ...
    module.exports = {
      ...
      reporters: ['dot', simpleReporter],
      reporterOptions: {
        resultsDir: './reports'
        resultsFile: 'report.json'
      },
      ...
    }

Or

    // wdio.conf.ts
    import SimpleReporter from 'wdio-simple-reporter/lib/reporter'
    ...
    module.exports = {
      ...
      reporters: ['dot', SimpleReporter],
      reporterOptions: {
        resultsDir: './reports'
        resultsFile: 'report.json'
      },
      ...
    }

Sample output

Here is an example of a generated report.json file.

[
    {
        "cid": "0-0",
        "capabilities": {
            "maxInstances": 1,
            "browserName": "chrome",
            "chromeOptions": {
                "args": [
                    "--disable-notifications",
                    "--enable-automation"
                ]
            }
        },
        "runnerTestsNumber": {
            "failing": 1,
            "passing": 1,
            "pending": 1
        },
        "specFileHash": "04fa2bf3e5eebf36b320ff69c18fde72",
        "specFilePath": [
            "../src/test/test.js"
        ],
        "suites": [
            {
                "type": "suite",
                "start": "2017-03-23T11:51:53.707Z",
                "_duration": 8316,
                "uid": "Suite #13",
                "title": "Suite #1",
                "tests": {
                    "Test #1: success5": {
                        "type": "test",
                        "start": "2017-03-23T11:51:53.709Z",
                        "_duration": 1281,
                        "uid": "Test #1: success5",
                        "title": "Test #1: success",
                        "state": "pass",
                        "screenshots": [],
                        "output": [...],
                        "end": "2017-03-23T11:51:54.990Z"
                    },
                    "Test #2: failes8": {
                        "type": "test",
                        "start": "2017-03-23T11:51:54.990Z",
                        "_duration": 7032,
                        "uid": "Test #2: failes8",
                        "title": "Test #2: failes",
                        "state": "fail",
                        "screenshots": [],
                        "output": [...],
                        "error": {
                            "message": "element (#id1) still not visible after 3000ms",
                            "stack": "Error: element (#id1) still not visible after 3000ms\n    at Context.<anonymous> (src/test/test.js:18:17)\n    at new Promise (node_modules/core-js/library/modules/es6.promise.js:191:7)\n    at elements(\"#id1\") - isVisible.js:54:17\n    at isVisible(\"#id1\") - waitForVisible.js:37:22",
                            "type": "WaitUntilTimeoutError"
                        },
                        "end": "2017-03-23T11:52:02.022Z",
                        "failureScreenshot": {
                            "event": "runner:screenshot",
                            "cid": "0-0",
                            "specs": [
                                "../src/test/test.js"
                            ],
                            "filename": "ERROR_chrome_2017-03-23T11-52-01.602Z.png",
                            "data": <base64 encoded screenshot>,
                            "title": "Test #2: failes",
                            "uid": "Test #2: failes8",
                            "parent": "Suite #1",
                            "parentUid": "Suite #13",
                            "time": "2017-03-23T11:52:02.021Z",
                            "specHash": "04fa2bf3e5eebf36b320ff69c18fde72",
                            "absolutePath": "/Users/andrii/Projects/GitHub/wdio-simple-reporter/errorShots/ERROR_firefox_2017-05-08T01-48-10.666Z.png"
                        }
                    },
                    "Test #3: skipped11": {
                        "type": "test",
                        "start": "2017-03-23T11:52:02.022Z",
                        "_duration": 1,
                        "uid": "Test #3: skipped11",
                        "title": "Test #3: skipped",
                        "state": "pending",
                        "screenshots": [],
                        "output": [],
                        "end": "2017-03-23T11:52:02.023Z"
                    }
                },
                "hooks": {
                    "\"before all\" hook4": {
                        "type": "hook",
                        "start": "2017-03-23T11:51:53.707Z",
                        "_duration": 1,
                        "uid": "\"before all\" hook4",
                        "title": "\"before all\" hook",
                        "parent": "Suite #1",
                        "parenUid": "Suite #13",
                        "end": "2017-03-23T11:51:53.708Z"
                    }
                },
                "end": "2017-03-23T11:52:02.023Z"
            }
        ]
    }
]

Conclusion

We could achieve the same results just by registering event handlers and processing the results. However, we recommend you learn how the tools and libs you use work from the top level (documentation) to the implementation (source code). You’ll always benefit from a better understanding of when and why you can use them.