Skip to content

fix: Ignore payload-less attachments during validation - #432

Closed
mvanhorn wants to merge 1 commit into
XCTestHTMLReport:mainfrom
mvanhorn:fix/387-ignore-payloadless-attachments
Closed

mvanhorn wants to merge 1 commit into
XCTestHTMLReport:mainfrom
mvanhorn:fix/387-ignore-payloadless-attachments

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Aug 11, 2026

Copy link
Copy Markdown

Summary by CodeRabbit

Tighten the post-condition in Summary.validate() so .none is fault-worthy only when the attachment also has a payload identifier, preserving the existing idempotent fault deduplication and genuine unresolved-payload behavior. Add a passing sample UI test that records an attachment with XCTest's payload-removing lifetime so prepareTestResults.sh produces a real attachment entry without a payloadRef; keep fixture generation and production parsing unchanged. Strengthen FaultReportingTests by first proving the generated fixture contains that payload-less attachment, then asserting validation does not report it as unresolved, so the regression test cannot pass vacuously.

Verification

  • Generate TestResults.xcresult from the sample project and verify the fixture contains at least one attachment whose payloadId is absent and whose rendered content is .none.
  • Validate that fixture and verify the payload-less attachment produces no unresolvedAttachment fault; running the CLI against the same fixture must continue to exit 0 without a degradation warning.
  • Verify attachments with a non-nil payload reference and successfully exported content remain fault-free in both linking and downsized fixture paths.
  • Preserve the existing failure rule and idempotency: an attachment that has a payload reference but resolves to .none is still reported as unresolved, and repeated validation does not duplicate that fault or disturb previously collected faults.

Why

Summary.validate() currently records an unresolvedAttachment fault whenever an attachment's rendering content is .none. Attachment also uses .none for attachments that never had a payloadRef, so a valid payload-less attachment is misclassified as failed resolution and can make the CLI exit 3. The model already retains Attachment.payloadId, which distinguishes an attempted payload resolution from the absence of a payload without adding a new state or test-only abstraction. The current generated fixtures contain no payload-less attachment, leaving this false-positive path untested.

Closes #387

Summary by CodeRabbit

  • Bug Fixes

    • Improved attachment validation so attachments without payload references are ignored rather than incorrectly reported as unresolved.
    • Clarified diagnostics for attachments whose payloads cannot be resolved.
  • Tests

    • Added coverage for attachments configured to be deleted after successful test completion.
    • Updated test result expectations and validation checks to reflect the expanded sample test coverage.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Summary.validate() now ignores attachments without payload identifiers and continues to report unresolved payloads. Tests cover payload-less attachments and add a delete-on-success attachment sample.

Changes

Attachment validation correction

Layer / File(s) Summary
Payload-aware attachment validation
Sources/XCTestHTMLReportCore/Classes/Models/Summary.swift
Validation now reports unresolved-attachment faults only when an attachment has a payloadId and no resolved content.
Attachment behavior coverage
Tests/XCTestHTMLReportTests/FaultReportingTests.swift, Tests/XCTestHTMLReportTests/CoreTests.swift, XCTestHTMLReportSampleApp/SampleAppUITests/FirstSuite.swift
Tests verify that payload-less attachments are ignored. The sample adds an attachment with .deleteOnSuccess, and the expected test count increases to 18.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: tylervick

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The new testAttachDeletedOnSuccess test covers attachment deletion semantics, which are unrelated to issue #387. Move the attachment deletion test to a separate pull request or link an issue that requires deletion-semantics coverage.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the primary change: ignoring payload-less attachments during validation.
Linked Issues check ✅ Passed The changes prevent false positives for payload-less attachments, preserve unresolved-payload faults, and add fixture coverage for issue #387.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
Tests/XCTestHTMLReportTests/FaultReportingTests.swift (1)

53-72: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add coverage for payload-bearing unresolved attachments.

This test verifies only the payload-less case. It does not verify that a non-nil payloadId with .none content still creates an .unresolvedAttachment fault. The test would pass if Summary.validate() skipped all .none attachments. Add or retain a fixture case for the required positive behavior.

This matches the PR objective to continue reporting attachments with payload references that resolve to .none.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Tests/XCTestHTMLReportTests/FaultReportingTests.swift` around lines 53 - 72,
Extend the test around summary.validate() to include an attachment with a
non-nil payloadId and .none content, then assert that validation reports it as
an .unresolvedAttachment fault. Preserve the existing assertion that
payload-less attachments are not reported, and ensure the fixture covers both
behaviors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@Tests/XCTestHTMLReportTests/FaultReportingTests.swift`:
- Around line 53-72: Extend the test around summary.validate() to include an
attachment with a non-nil payloadId and .none content, then assert that
validation reports it as an .unresolvedAttachment fault. Preserve the existing
assertion that payload-less attachments are not reported, and ensure the fixture
covers both behaviors.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a7afa8f5-29c5-4a69-9297-72476560133a

📥 Commits

Reviewing files that changed from the base of the PR and between f717a1d and eb5fd0a.

📒 Files selected for processing (4)
  • Sources/XCTestHTMLReportCore/Classes/Models/Summary.swift
  • Tests/XCTestHTMLReportTests/CoreTests.swift
  • Tests/XCTestHTMLReportTests/FaultReportingTests.swift
  • XCTestHTMLReportSampleApp/SampleAppUITests/FirstSuite.swift

@tylervick

Copy link
Copy Markdown
Member

Closing as superseded: #438 landed the same fix (only fault attachments whose payload existed but failed to resolve) with tests that go through the production decode path and pin that a genuinely unexportable payload still faults. Thanks for surfacing the issue-side of this — the fixture-based test idea from this PR is noted as a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

validate() false-positives on attachments with no payloadRef

2 participants