Execute Test-Groups


#1

I want to group test methods according to this TestNG example:


Trying to use the described testng.xml example from that site was not working with Webtestit. Maybe that has something to do with this isse:

It would be great to run test-groups from Webtestit execution tab (and/or endpoint confuguration parameter). Keep in mind to show the group information in reporting.


#2

Hello @HRA
Could you tell me what error are you getting when you try to edit the testng.xml file that way (to include some test groups)? Have you edited the testng.xml file like in this example (lets say you have the group1 marked test in TC2 class)

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Tests" verbose="5">
    <test name="Test1">
        <groups>
            <run>
                <exclude name="group2" />
                <include name="group1" />
            </run>
        </groups>
        <classes>
            <class name="uitest.tests.TC2" />
        </classes>
    </test>
</suite>

For showing information about groups in the report, I would have to discuss with the team, but an interesting proposal.
Regards.

Update @HRA:
The issue here is that the driver does not get instantiated in the specific group provided trough testng.xml file. Most probably due to a known TestNG bug, and if you want to find out more details, please visit this topic.
The solution here would be to place the (alwaysRun=true) parameter on all @BeforeMethod, @BeforeSuite and @AfterMethod, @AfterSuite in the TestNGTestBase.java file and you should be fine! Kind Regards,
Sasa


#3

Hi @smatijas
My testng.xml file is like your example and seems to be ok for me.
Adding (alwaysRun=true) to @BeforeMethod annotation is not solving the issue. Error message is:
java.lang.Error: No WebDriver instance was created for this test method
Some more backgroud to my use case: I have several test methods for testing payment workflows. With groups I could group the “payment test” execution. Grouping the execution of that test cases and reporting the results with group information would be very helpful.
Thanks again for your help and best Regards,
Hans


#4

@HRA Hello again!
Have you added the (alwaysRun=true) to all of the before and after methods in the TestNGTestBase.java? Could you perhaps do it like that? I could reproduce your issue with the same error, and after adding this fixed the issue:

@BeforeSuite(alwaysRun=true)
@BeforeMethod(alwaysRun=true)
@AfterMethod(alwaysRun=true)
@AfterSuite(alwaysRun = true)

Alternatively, I can send you the modified TestNGTestBase.java file if needed.
Regards,
Sasa


#5

Hi @smatijas
I added (alwaysRun=true) to all that annotations in TestNGTestBase,java. But I also use @Before method in my test classes to get the test parameters passed to the report output. So I also had to add (alwaysRun=true) to the @Before methods in my test classes. Now its working as expected.
It might be helpful for others how to retrieve the group name for reporting. Thats a working example:

@BeforeMethod(alwaysRun = true)
    public void BeforeMethod(Method method, Object[] testData) {
        // Parameter output to report
        testName.set(String.format("%s|%s]", method.getName(), testData[0]));

        // Log test-group name
        Test t = method.getAnnotation(Test.class);
        LOGGER.debug("TEST-GROUP: " + t.groups()[0]);
    }```

#6

Let me add some words regarding reporting. For me thats an important topic to make QA information transparent in the organization and support manual processes with required data. The report integration in Webtestit itself is straight forward and useful for develeopment time and/or basic needs. But when things getting beyond that basics I miss painfully:

  • A clean way to output test parameter to the report (currently I use the @Before method in my test classes to add that information to the method name)

  • A way to add test result data from code to te report (e.g. add order number after test run of a purchase workflow)

  • Bring the above mentioned test-group information to the report (this information is also in reporting a grouping property)

  • Make reports available outside Webtestit (at least drop a PDF; HTML would be much better to make the reports accessible via browser)

I know, Webtestit makes use of a standard XML reporting format which can be used in other tools to make it suitable for your needs. But in scenarios where a CI/CD tool chain would be oversized the above mentioned functionality would be very helpful.


#7

@HRA!
Glad it worked out for you with the group test execution, and thank you for digging in and spreading the knowledge!
Regarding the reporting, we will definitely discuss those points of yours and see what could be done. For the last point (Make reports available outside Webtestit) we already have a cool new feature in place, and we expect it to be available with upcoming releases.
I will let the team know about the other requirements, and keep you updated about the outcome.
Kind regards