Create screenshots on failure for TypeScript / Protractor based projects


#1

If things go wrong in your tests, it is certainly helpful to have logs telling you what it was about. But there is the saying that pictures say more than a thousand words. So wouldn’t it be great if your Protractor project could automatically create screenshots on failing expectations? Read on for how to configure this feature.

Prepare the project setup and dependencies

We’ll start with a fresh new TypeScript/Protractor project called auto-screenshot-ts.
Once scaffolded navigate to the created package.json file and add the following to the dependencies section:

"jasmine2-protractor-utils": "1.3.0"

After saving the file, Webtestit will automatically install the added dependency.

Configure protractor

With that done our next stop is the protractor.conf.js configuration file.

We’re going to configure the installed dependency by adding a new plugins property. Simply place the plugin inside of the const=configuration{}, by putting a comma after the last entry

plugins: [{
    package: 'jasmine2-protractor-utils',
    disableHTMLReport: true,
    disableScreenshot: false,
    screenshotPath: './reports/screenshots',
    screenshotOnExpectFailure: true,
    screenshotOnSpecFailure: true,
    htmlReportDir: './reports/htmlReports'
}]

For all available configurations take a look at the plugins configuration options

Also make sure that the noGlobals property is set to false, as most plugins expect some way of accessing variables like the protractor browser global

{
 ...,
  // You could set no globals to true to avoid jQuery '$' and protractor '$'
  // collisions on the global namespace.
  noGlobals: false,
}

Last but not least we want to make sure that the generated screenshot is also part of the JUnit report so that each testcase has a reference to the generated image. Therefore we need to modify the JUnitXmlReporter configuration - located inside the protractor onPrepare hook, by adding the systemOut property:

const junitReporter = new jasmineReporters.JUnitXmlReporter({
   ...
      consolidateAll: true,
      filePrefix: process.env.TEST_REPORT_FILENAME,

      // WE'RE ADDING THIS
      systemOut: (spec, suiteFileName) => {
        return `screenshots/${process.env.RX_ENDPOINT_BROWSER}-${spec.fullName}-expect failure-${spec.id.replace("spec", "")}.png`;        
      }
     // END OF ADD
    });
    jasmine.getEnv().addReporter(junitReporter);

The path constructed contains the screenshots subfolder we’ve defined in our plugin configuration ('./reports/screenshots',) and follows the naming convention used by the jasmine2-protractor-utils plugin.

Create a sample page object and test

Now to verify that our setup works let’s create a page object named main-po.ts with the following file content

// Ranorex Webtestit Page Object File

import { browser, element, by, ExpectedConditions } from 'protractor';

export class MainPo {
  private get MainTitle() { return by.css('h1'); }
  
  public async open(url: string): Promise<MainPo> {
    await browser.get(url);

    return this;
  }

  public async GetMainTitle(): Promise<string> {
    await browser.wait(ExpectedConditions.visibilityOf(element(this.MainTitle)), browser.allScriptsTimeout, this.MainTitle.toString());
    const MainTitleText = await element(this.MainTitle).getText();
    
    return MainTitleText;
  }
}

and a main-test.ts test file with the following content, where we will expect the wrong title from the Ranorex homepage.

// Ranorex Webtestit Test File

import { browser } from 'protractor';
import { MainPo } from '../pageobjects/main-po';

describe('the Ranorex homepage', () => {
  beforeEach(() => {
    //Make sure to set the ignoreSynchronization for every testrun for non Angular applications
    browser.ignoreSynchronization = true;
  });

  it('should have a proper browser title', async () => {
    const po = new MainPo();
    await po.open("https://www.ranorex.com");

    expect( await po.GetMainTitle() ).toBe("Not the expected title");
  });
});

Execute tests and inspect the screenshots

With that setup, lets run our tests and see what happens. Add as an example a local Chrome browser endpoint and Run all test files.

After the test has ended and failed, you will see a subfolder screenshots created in reports containing a new png file. Right click on that and click open in containing folder. Now you can inspect the image and see the mismatch from our assertion.

The resulting report xml file should now look like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="the Ranorex homepage" timestamp="2018-07-31T10:17:35.475Z" hostname="localhost" time="3.266" errors="0" tests="1" skipped="0" disabled="0" failures="1">
  <testcase classname="the Ranorex homepage" name="should have a proper browser title" time="3.265">
   <failure type="toBe" message="Expected 'Test Automation for All' to be 'Not the expected title'."><![CDATA[Error: Failed expectation
    at Object.<anonymous> (/Users/vsoftic/work/demo/auto-screenshot-ts/test/tests/main-test.ts:16:39)
    at Generator.next (<anonymous>)
    at fulfilled (/Users/vsoftic/work/demo/auto-screenshot-ts/dist/test/tests/main-test.js:5:58)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)]]>
   </failure>
   <system-out>screenshots/chrome-the Ranorex homepage should have a proper browser title-expect failure-0.png</system-out>
  </testcase>
 <properties><property name="endpointName" value="New Endpoint"/><property name="endpointBrowserName" value="chrome"/><property name="endpointHeadless" value="false"/><property name="endpointType" value="local"/><property name="projectName" value="auto-screenshot-ts"/></properties></testsuite>
</testsuites>

Note how we now have the <system-out> element, pointing to the created screenshot, inside the testcase, which can be useful if you plan to integrate your reports with a continuous integration system.

Adjusting the Webtestit Report

Now, this was all fun, but actually, it would be great to see those images also directly inside the reports which are generated after each run.

We’ve got you covered here since Webtestit Reports are HTML templates (handlebars), which can be fully customized by yourself. Take a look at this in depth tutorial.

So in order to start, we’re going to create inside our root project a folder templates/report/partials.
Open the templates folder via the menu -> templates -> open templates folder which will open your finder/explorer in the location of the original templates.

23

Copy the file report/partials/test-case.hbs over there into your newly created partials folder.
This will override the default template for the test-case rendered repeater of the report.

Now we will adjust the test-case template.
In line 42 change the following line with the new one shown here

// line 42: change this
<td colspan="3" class="test-report__message">{{#sanitize_html failure.message}}{{/sanitize_html}}</td>

// to this:
<td colspan="3" class="test-report__message">{{#sanitize_html failure.message}}{{/sanitize_html}}
  <br />
  <img style="max-width: 500px" src="{{@root.projectRoot}}/reports/{{extraElementValue extraElements 'system-out'}}" />
</td>

After saving the file and going back to your report you should now see the image inlined below your error.

Conclusion

With these simple steps, you’re able to record screenshots of all your failed expectations and use them to analyze the visual difference. This is especially useful when running with a remote endpoint, where you can’t easily inspect the actual representation.

Want to learn more about remote endpoints? Take a look at this introduction

We’ve also learned how the report templates can be easily changed to pretty much show whatever you like. There are tons of other possibilities which you can learn about here.

The used plugin jasmine2-protractor-utils is just one of several other existing approaches. You might want to check out Protractor-screenshot-reporter or Protractor-Screenshoter-plugin as well.

If you’d like to have something similar up and running for your Java project take a look here


Create screenshots on failure for Java / TestNG based projects
Screenshot on failure :Online documentation need to be improved
Screenshots on failures shall be included
Release notes v0.15.0
Screenshot on failure :Online documentation need to be improved
How to access endpoint information from within tests