XRAY: explaining the (complex) relations between Test Plans, Tests, and Test Executions

Have you heard about the synch between Test Plans (TP) and Test Executions (TE)?

Don’t feel bad if you didn’t – there isn’t any. Not full synch in any case.

This can be very confusing and important to understand in regards to data integrity.

What to do to avoid data integrity issues?

Verify that all the tests in the test section of the TP are also embedded in TE in the same TP.
If all TE tests run (Pass, Failed etc., not in TODO) you will see unlinked tests in the TP as TODO. In the TP choose Create Test Execution=>with Status=>TODO and run them in the new TE.
Verify that Tests appear only in one TE associated to the same TP.
If you remove TE from a TP because it is not relevant, you need to manually remove the tests in the TP.
Also important: Test executions are in context to TP: only results from the embedded TE will be seen in the TP.
Test X Failed, run again later and pass. If the Fail is n a TE linked to a TP and same test passed in a different TE, the TP will show: Fail.
If you run the same test in different TE in the same TP – last result is seen.

Tests result in the TP example:

Tests in a TP:

TE in a TP:

State	Action	Result
Add tests to a TP and add all the test to TE.	Remove tests from the TE (the test doesn’t exist in other TE in the TP).	In the TP: TE remain w/o the Test. All the tests remain in the TP. Tests that had results in the removed TE are set to TODO.
Add test to a TP and then add all the test to TE.	Remove tests from the TP	The tests remain in the TE.
		The result of running the tests in the TE will NOT be seen in the TP.
		In the XRAY TP report, only the results from the tests in the TP are presented – not from the TE! TP results = only TP tests TP Report - only tests from the TP:
Add a new TE to a TP with tests	*and one of the tests failed	In the TP you see the tests from the TE. In the TP you see the failed test as failed (as in the reports).