Page tree
Skip to end of metadata
Go to start of metadata

Integrating with testing frameworks is achieved by processing the results contained in the reports test runners create (e.g. in XML, JSON). The results are mapped to the proper Test issues and, if possible, if they don't exist then they will be created. This flow is explained in Using Generic Tests for Automation, although Xray also supports Cucumber tests natively.     

Besides Cucumber, there are currently many testing frameworks available for every language you may think of.

Many of these frameworks provide test runners that are able to output reports in the JUnit XML format. Therefore, in the "worst case" your automated tests can be imported to JIRA and mapped to "Generic" Test issues, by importing JUnit XML test result reports as detailed here. Note that JUnit report format is rather limited and sometimes it's not supported by some testing frameworks.

Another way of integrating with testing frameworks is by using the test runner Nunit's XML report format. Nunit report format has more features than JUnit 4.0 XML report, which allows Xray to do things such as automatic linking to issues (e.g. requirements) or assignment of labels to the newly created Test entities.

Besides this, Xray also provides specific integrations for some frameworks, such as Cucumber, Behave, Robot, Xamarin, and other.


xray_integration_with_testing_frameworks

Summary of features per framework

The following table presents the available features, whenever importing automated test results.



Robot framework

Junit 4

Nunit 2.6/3.x

Cucumber

Abstract automated test as a Test

(map an automated test to a Test issue in JIRA)

Yes

(as a Generic Test)

Yes

(as a Generic Test)

Yes

(as a Generic Test)

Yes

(as a Cucumber Test)

Make Test specification in JIRA

(specify the Test itself in JIRA)


NoNoNoYes

Create Tests from results

(create Tests whenever importing results)

Yes

Yes

Yes

No*

(however, tehre is and endpoint for importing Cucumber features, which will create Tests for the respective Scenarios/Scenario Outlines)

Uniquely identify Tests

(identify existing Tests whenever importing results, avoiding duplication of Test issues)

Yes

Yes

Yes

Yes

Import results

(importing results by REST API or UI)

Yes

Yes

Yes

Yes

Import “labels”

(create labels in the Test issues)

Yes

(labels may be specified in the test's source code)

No

Yes

(labels may be specified in the test's source code)

Yes*

(this is only available whenever using the endpoint for importing Cucumber features; it's not possible whenever importing results)

Automatic linking to requirements

(create links to requirements)

Yes

(requirement's issue key may be specified in the test's source code)

No

Yes

(requirement's issue key may be specified in the test's source code)

Yes*

(this is only available whenever using the endpoint for importing Cucumber features; it's not possible whenever importing results)

Semantic on the results

(present detailed execution details in the execution screen)

Yes

(keywords “steps”)

-

Yes

(Test Suites & parameterized Tests)

Yes

(steps for scenario/scenario outline and background)

Integration by platform

 

 

Integration by testing framework report

 

 

Testing by target environment

 

 

 

 

 

 

 

 

 

  • No labels