Page tree

This is the documentation of Xray Server. If you are looking for Xray Cloud, visit this page. Documentation of older versions of Xray Server is available in this page.

Skip to end of metadata
Go to start of metadata


Cucumber is a framework for behavior-driven development (BDD), where the test specification is done in natural language, while the implementation of each specification's phrase (i.e., step) is done in code (e.g., Java, Ruby or some other language).

Xray provides the tool to create the specification inside Jira, although the implementation code itself is outside of Xray's scope.  

One key thing that you must decide is: where do you want to manage the Cucumber scenarios? Do you want to use Jira as the master of information (recommended)? Or do you want to use Git, for example, as the master for your features and corresponding scenarios?


Please note

Having both Jira and the VCS (e.g. Git, SVN) as master of information for Cucumber Scenarios/Scenario Outlines can lead to synchronization issues because the scenarios edited in JIRA won't be commited automatically to the VCS to the original feature files used.

Thus, you can use a mix of the Standard and VCS workflows below but have in mind that the scenarios can be out of synch between what you have in Jira and what you have in the VCS.


Standard Cucumber workflow

In this workflow, Tests are created and managed in Jira, thus Jira will be the master for the Cucumber scenarios.

cucumber_tests_workflow


  1. Specify Cucumber tests using natural language, in Jira.
  2. Export Cucumber features from Jira to the CI environment, using the REST API.
  3. Implement tests in code and commit them to the source code versioning system.
  4. Execute tests in the CI environment.
  5. Report results to Xray, using the REST API.

Note that Cucumber steps and related code may, in fact, be implemented before the first step, or may not even be needed to be implemented for a given Test if all the sentences are already implemented as code.


Learn more

See Automated Tests (Import/Export) for more information on how Cucumber feature files are generated and how to import back results to Xray.

Pure VCS based workflow


Please note

The support for this workflow has been added in v3.1, in experimental mode.


In this workflow, the features are being "managed" in the source code VCS (versioning control system). This means that users will be editing features elsewhere, other than Jira, and they will want to synchronize the scenarios contained within those features to Jira. Therefore the master of information will be Git, SVN or whatever VCS is being used.


cucumber_tests_legacy_vcs_workflow2


  1. Create Cucumber tests (e.g. Scenario/Scenario Outlines) and their respective step code in some other tool.
    1. add an unique tag to each Scenario/Scenario outline, in the form of "@id:xxx", in which xxx is a number (e.g. 1, 2, 3, ...)
  2. Import existing Cucumber tests using the REST API (e.g., from an existing code base which may have tests, and their corresponding step implementation) 
  3. Export Cucumber features from Jira to the CI environment, using the REST API or the Jenkins/Bamboo plugins. It is crucial to use these exported features instead of the ones from the VCS, since the features exported from JIRA will contain additional tags that will allow several things, including the ability to correctly report the results to back to the correct entities in Jira, afterwards.
  4. Execute tests in the CI environment.
  5. Report results to Xray, using the REST API or the Jenkins/Bamboo plugins.
  6. Repeat from step 1 onwards.


Please note

This flow will work fine as long as:

  • An unique tag "@id:xxx" is added to each Scenario/Scenario Outline within the file. The id just needs to be unique (e.g. sequential) within the feature file
  • The path of the feature file is not changed

Is any of the previous conditions is not satisfied then new and duplicated Tests, and Pre-Conditions, may be created.

See an example in Importing Cucumber Tests - REST.


Legacy Cucumber migraton workflow

The workflow below applies to legacy/existing projects that already have Cucumber tests in some version control system (e.g., GIT, SVN), possibly alongside the software's implementation.


Existing Cucumber tests are initially imported to Jira. Editing the tests can then proceed on the Jira side. While test editing can also occur simultaneously in Jira and in some other tool, it is highly recommended to keep it only on one side, preferably Jira.


cucumber_tests_legacy_workflow


  1. Create Cucumber tests (e.g., Scenario/Scenario Outlines) and their respective step code in some other tool.
  2. Import existing Cucumber tests using the REST API (e.g., from an existing code base which may have tests, and their corresponding step implementation) 
  3. Optionally, and in parallel, 
    1. Specify additional or edit existing Cucumber tests in natural language, in Jira.
    2. Implement remaining tests in code and commit them to the source code versioning system.
  4. Export Cucumber features from Jira to the CI environment, using the REST API or the Jenkins/Bamboo plugins.
  5. Execute tests in the CI environment.
  6. Report results to Xray, using the REST API or the Jenkins/Bamboo plugins.
  7. Repeat from step 3 onwards.


Please note

As of Xray v2.3.x, step 2 is not idempotent, i.e., if you try to import Cucumber Tests multiple times using the endpoint Importing Cucumber Tests - REST, you may end up with repeated entities in Jira. This has been addressed in v3.1.


Common problems

Submission of Cucumber results says "No tests found in execution result"

Most probably you're trying to submit a Cucumber JSON results file without having previously created the Cucumber Tests (e.g., Scenario/Scenario Outline) in Jira.

You need to first have these Test issues created in Jira, with the correct specification. Then, you need to export those Tests to one or more Cucumber .feature files, so you're able to run them in your CI environment. 

Following this workflow is important because whenever you export the features from Jira, along with the corresponding Cucumber Scenarios/Scenario Outlines, the Scenarios will be tagged with labels that will allow Xray to map the results back to the proper entities in Jira.

Is it possible to create of Cucumber tests automatically when submitting Cucumber results?

No. It's not possible to properly recreate the Cucumber Scenario/Scenario Outline/Background from a Cucumber result file in JSON format.

Where is the master of information for Cucumber Tests?

Our recommendation is to manage it in Jira and always edit the Cucumber Scenarios/Scenario Outlines/Backgrounds there. You can use your VCS (e.g. Git, SVN) as the master but don't try to use it and at the same time also use Jira as master.


  • No labels