Pass capabilities to Sauce Labs executions


#1

I am using the data driven approach (with Java) and want to shift the test execution to Sauce Labs. Execution basically runs fine so far. Unfortunately the job list in Sauce Labs is not useful:
image
I need to pass a job name with test parameter information to generate such an output for the job name:

Something like that should do the job:
https://wiki.saucelabs.com/display/DOCS/Best+Practice%3A+Use+Build+IDs%2C+Tags%2C+and+Names+to+Identify+Your+Tests

But how can I acces that capabilities object in Webtestit test method and pass it to the webdriver?


#2

You can customize the capabilities in the endpoint configuration panel. In the Desired capabilities add yours, in case of the Saucelabs testname the prop is called name:

image


Now this will use the name statically though since it’s configured for the whole session of the remote driver.

In order to update the name based on iterations, the only way would be to consume the SauceLabs API and change the name after the run on SauceLabs. We have something like that already as queued Item in the backlog, as it’s especially interesting in a data driven example, like yours.

EDIT: hold on, there seems to be a manual workaround meanwhile though.

Give this page a look https://wiki.saucelabs.com/display/DOCS/Annotating+Tests+with+Selenium's+JavaScript+Executor

As noted there, you can run the following command to setup a dynamic name

((JavascriptExecutor)driver).executeScript("sauce:job-name=My test");

In there you can pass any attributes from the iteration you’d like by concatenating them into the string of executeScript. We’d love to hear back how that worked out for you.


#3

Using capabilites in endpoint configuration works, but does not solve my problem. My methods are attached to data providers and process a lot of datasets. Each single execution is listed in Sauce Labs as single job and displayed with some sort of hash as display name.
Trying the JavaScriptExecutor example is not working. Placing the command in the test method (right at the start or at the end) has no effect on the jobname in Sauce Labs dashboard. Placing the code into @AfterMethod code results in an exception “No WebDriver instance was created for this test method”.
So I am a little bit lost here. Using Sauce Labs with useless job names makes no sense. Setting up one test name via endpoint for all jobs is also not an option. The only solution to use Sauce Labs in my scenario would currently be to not use Webtestit :frowning:


#4

yes, as mentioned updating the capabilities is a static approach for the whole run, including all iterations.

I wonder though why it didn’t work for you with the JavaScriptExecutor sample. I’ve just tried it and it works with the following sample:

// newtestfile.java
class newtestfile extends TestNgTestBase {
    @Test
    public void SampleTestCase() {
        WebDriver driver = getDriver();

        newpofile newpofileInstance = new newpofile(driver);
        newpofileInstance.open("https://www.ranorex.com");
        newpofileInstance.setName();
    }
}

// newpofile.java
public class newpofile {

    ... // default methods of a PO skipped to keep the sample short

    public newpofile setName() {
        ((JavascriptExecutor) driver).executeScript("sauce:job-name=My sample test");
        return this;
    }
}

Once I run this test this is how it looks like in SauceLabs’ dashboard

As you can see, the hard-coded testname from setName is used. Perhaps you have some escaping going on with your custom built string for the job-name?

As mentioned, this is already in our Backlog and it should be addressed in the future. Data-bound tests are a delicate case which have very specific requirements per project, so we want to make sure to offer something that works for most cases vs one very narrowed down approach.


#5

Thanks for the example. You are pointing to the right direction.
There is a colon in my custom built job name string. That silently fails when using the JavaScriptExecutor example. I got things working as expected when replacing special characters in the string.

No I am facing two more challenges:

  1. Setting the job name is just required when using Sauce Labs as an endpoint. Can I access endpoint information in my code to set the job name only when executing the tests against Sauce Labs?
  2. There is no information about success/failed in Sauce Labs for executed tests. The status is always “complete”. Do I have to push the test result manually? In the local reports I can see the correct status but there is not longer a screenshot attached.

#6

Good to hear that.

With regards to 1. there is no direct way to tell you what type of remote execution it is. This is something also planned to add for future (at least for Java). For now, depending on where you’re at, e.g inside tests you can make use of the base class TestNgTestBase.gridHubUrl and check whether that contains saucelabs.

// newtestfile.java
public void SampleTestCase() {
    WebDriver driver = getDriver();

    newpofile newpofileInstance = new newpofile(driver);
    newpofileInstance.open("https://www.ranorex.com");
    newpofileInstance.setName();

    if (this.gridHubUrl.getHost().contains("saucelabs")) {
        // this is running on SauceLabs
    }
}

if you require the info in your PageObject there are multiple ways to do it. Either pass the info from the test or just read the properties by yourself. Get the config by instantiating the SuiteConfiguraiton in the constructor and setup a boolean. Use that wherever you need to check.

// newpofile.java
public class newpofile {
    protected WebDriver driver;
    protected WebDriverWait wait;
    protected boolean isSauceLabs;

    public newpofile(WebDriver driver) throws IOException {
        this.driver = driver;
        this.wait = new WebDriverWait(driver, 10);

        SuiteConfiguration config = new SuiteConfiguration();
        this.isSauceLabs = config.getProperty("grid.url").contains("saucelabs");
    }

    public newpofile setName() {
        if (isSauceLabs) {
            // saucelabs running
        }
        
        ((JavascriptExecutor) driver).executeScript("sauce:job-name=My sample test");
        return this;
    }
}

About question 2. I’d recommend creating a new AfterMethod hook which you place either in your testfile or in the TestNgTestBase.java which looks like the following:

@AfterMethod
public void reportToSauceLabs(ITestResult result) {
    if (this.gridHubUrl.getHost().contains("saucelabs")) {
        ((JavascriptExecutor) driver).executeScript("sauce:job-result=" + (result.isSuccess() ? "true" : "false"));
    }
}

Just make sure to add it as a new method if you decide so in your TestNgTestBase. The reason is that in future updates the auto-migration feature will have an easier time to detect changes and automatically merging your changes of that base file. Otherwise after a new update of Webtestit in worst case you’d just have to do the merge by-yourself, which shouldn’t be a big deal though.


While all this sounds pretty heavy, and I agree it can feel so, this is exactly the reason why we’d like to improve on these things in future with the planned work, so that essentially most things are prepared for you and ready to use. Feedback like yours helps us to gather valuable use-cases in order to make sure to have a properly working generic solution. So thanks again


#7

Thanks again for your great support.

Adding the method reportToSauceLabs with @AfterMethod annotation to the test class runs into “No WebDriver instance was created for this test method”. But it works fine in TestNgTestBase class.
So I see the test result in the Sauce Labs job list.

What I miss in the local reporting is the screenshot on errors. Yes, there is a video on Sauce Labs for each excution. But find the matching remote job from local reporting is unpractical.

A little info about our intensions: to make coding efficient we selected Webtestit. To reach device and platform coverage we decided for Sauce Labs. Currently there are two major things missing:

  1. Reporting (outside of developers Webtestit IDE)
  2. Debugging of Java code (without breakpoints and watches an awkward thing)

#8

Hello @HAR!
We have not forgotten your requests! :wink: , and we are proud to inform you that we introduced the debugger, and the report server features with the new 1.11.0 release. Make sure to check them out!
Regards