Skip to content

[feat] Add support for marking tests as expected failures#3481

Merged
vkarak merged 4 commits intoreframe-hpc:developfrom
vkarak:feat/expected-failures
Jun 27, 2025
Merged

[feat] Add support for marking tests as expected failures#3481
vkarak merged 4 commits intoreframe-hpc:developfrom
vkarak:feat/expected-failures

Conversation

@vkarak
Copy link
Copy Markdown
Contributor

@vkarak vkarak commented May 15, 2025

The main elements introduced by this PR are:

  1. The @xfail decorator to mark a test as an expected sanity failure.
  2. The xfail() builtin to mark reference tuples as expected performance failures.

Two new test states are introduced which are also logged: XFAIL for expected failures and XPASS for unexpected passes.

Design-wise the concept is the same as the one explained in #2378.

Implementation-wise there are two major changes:

  1. The execution policies code is refactored by eliminating code duplication between the serial and asynchronous policies.
  2. A new utility function reference_bounds() is introduced which is used to validate and calculate the absolute lower/upper bounds. This used to be internal to assert_reference() and now it's made public (maybe the docs there need a slight reorganization).
  3. The treatment of performance failures is rewritten as now we have to handle much more complex cases, were the performance variables of a test can simultaneously be pass, fail, xpass or xfail.

For more details on how feature works macroscopically check the tutorial and the reference docs.

Closes #2378.

@vkarak vkarak added this to the ReFrame 4.9 milestone May 15, 2025
@vkarak vkarak requested review from ekouts, teojgo and victorusu May 15, 2025 14:08
@vkarak vkarak self-assigned this May 15, 2025
@vkarak vkarak force-pushed the feat/expected-failures branch from 7bbb18a to 0d5b5e4 Compare May 15, 2025 14:10
@vkarak vkarak requested a review from jack-morrison June 6, 2025 08:32
Copy link
Copy Markdown
Contributor

@victorusu victorusu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

@github-project-automation github-project-automation bot moved this from Todo to In Progress in ReFrame Backlog Jun 18, 2025
Comment thread reframe/core/pipeline.py
@vkarak vkarak force-pushed the feat/expected-failures branch from 8cb57af to 489e3b0 Compare June 18, 2025 19:17
@vkarak vkarak requested a review from teojgo June 18, 2025 19:18
@codecov
Copy link
Copy Markdown

codecov bot commented Jun 18, 2025

Codecov Report

Attention: Patch coverage is 97.77778% with 5 lines in your changes missing coverage. Please review.

Project coverage is 91.16%. Comparing base (2a2dd82) to head (17d6122).
Report is 5 commits behind head on develop.

Files with missing lines Patch % Lines
reframe/frontend/executors/__init__.py 93.02% 3 Missing ⚠️
reframe/core/decorators.py 90.90% 1 Missing ⚠️
reframe/frontend/executors/policies.py 98.61% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3481      +/-   ##
===========================================
+ Coverage    91.13%   91.16%   +0.02%     
===========================================
  Files           62       62              
  Lines        13132    13253     +121     
===========================================
+ Hits         11968    12082     +114     
- Misses        1164     1171       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vkarak vkarak force-pushed the feat/expected-failures branch from 5615f79 to 17d6122 Compare June 26, 2025 16:18
@vkarak vkarak merged commit b57f957 into reframe-hpc:develop Jun 27, 2025
39 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in ReFrame Backlog Jun 27, 2025
@vkarak vkarak deleted the feat/expected-failures branch June 27, 2025 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Support marking a test as an expected failure

3 participants