feat: optimize count(*) for pk/append table by lucasfang · Pull Request #317 · alibaba/paimon-cpp

lucasfang · 2026-05-29T02:49:18Z

Purpose

Linked issue: close #321

This PR adds an optimized count(*) path for both PK and Append tables by introducing a dedicated CountReader API and table-specific implementations.

Main changes:

Add CountReader API and TableRead::CreateCountReader(splits).
Implement PK count path via PKCountReader (metadata path + MOR merge path).
Implement Append count path via AppendCountReader (metadata-first).
Keep current unsupported boundaries explicit:
- predicate pushdown is not supported in CountReader path
- force_keep_delete is not supported in PK CountReader path

Tests

UT:

Add src/paimon/core/table/source/pk_count_reader_test.cpp
- MOR snapshot count
- DV snapshot count
- empty splits
- invalid split type
Add src/paimon/core/table/source/append_count_reader_test.cpp
- append snapshot count
- empty splits
- invalid split type
- predicate-not-supported behavior

IT:

Update test/inte/scan_and_read_inte_test.cpp
- merge count assertions into corresponding original PK scan/read tests
- keep consistency check between CreateReader row count and CreateCountReader result
- keep unsupported-option tests

API and Format

API impact:

Add include/paimon/reader/count_reader.h
Add TableRead::CreateCountReader(splits) in include/paimon/table/source/table_read.h

Storage format / protocol impact:

No storage format change
No protocol compatibility change

Documentation

No additional user-facing documentation is required for this PR.

Generative AI tooling

Generated-by: GitHub Copilot (GPT-5.3-Codex)

Copilot

Pull request overview

Adds an optimized CountReader API for PK tables. The default TableRead::CreateCountReader returns NotImplemented, while KeyValueTableRead provides a real implementation built on a new CountSplitRead that picks between a metadata-only count for raw-convertible/DV splits and a merge + drop-delete + count path (via a new RowCountAccumulator) for MOR splits, using a column-pruned read schema. Predicate pushdown and force_keep_delete are rejected for now. Integration tests verify counts against existing full-read results, plus empty-splits and unsupported-option cases.

Changes:

New CountReader abstraction (include/paimon/reader/count_reader.h) and TableRead::CreateCountReader with PK implementation via CountSplitRead + RowCountAccumulator.
Metadata-based counting reuses DataSplitImpl::PartialMergedRowCount; MOR path reuses MergeFileSplitRead::CreateSortMergeReaderForSection(..., drop_delete=true) with a trimmed PK-only schema (added InternalReadContext::CreateWithSchema and MergeFileSplitRead::GetKeyComparator).
Adds count integration tests and an unrelated NormalizePath scheme test.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
include/paimon/reader/count_reader.h	New `CountReader` interface with `CountRows()`.
include/paimon/table/source/table_read.h	Adds virtual `CreateCountReader(splits)` on `TableRead`.
src/paimon/core/table/source/table_read.cpp	Default `CreateCountReader` returns `NotImplemented`.
src/paimon/core/table/source/key_value_table_read.{h,cpp}	PK `CreateCountReader` implementation; rejects predicate and `force_keep_delete`; stores path factory/context/executor.
src/paimon/core/operation/count_split_read.{h,cpp}	New `CountSplitRead` with metadata-count + merge-count strategies and PK-only read schema.
src/paimon/core/mergetree/row_count_accumulator.{h,cpp}	New helper that iterates `SortMergeReader` output and counts rows.
src/paimon/core/operation/internal_read_context.{h,cpp}	Adds `CreateWithSchema` factory to clone context with a new read schema.
src/paimon/core/operation/merge_file_split_read.h	Exposes `GetKeyComparator()` for use by count path.
src/paimon/CMakeLists.txt	Registers the two new source files.
test/inte/scan_and_read_inte_test.cpp	Adds count IT cases (MOR/DV/empty/consistency/predicate/force-keep-delete).
src/paimon/common/utils/path_util_test.cpp	Unrelated test for `NormalizePath` with URI scheme.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

lxy-9602 · 2026-05-29T07:16:18Z

Please add a issue for this pr.

lxy-9602 · 2026-06-01T05:24:14Z

+    }
+
+    return split->RowCount();
+}


DataFileMeta may not contain deletion information (e.g., data_deletion_files_.cardinality == nullopt), and falling back to split->RowCount() would ignore deletions, which is not correct.

lxy-9602 · 2026-06-01T05:37:11Z

+    ASSERT_OK_AND_ASSIGN(int64_t count, count_reader->CountRows());
+    ASSERT_EQ(count, 0);
+}
+


I’d like to understand whether these tests actually cover the case where the delete info in older versions is nullopt, so the code falls back to the merge read path. Also, could you please add an end-to-end test for append tables as well?

lxy-9602 · 2026-06-01T05:40:12Z

+/// This replaces the entire KeyValueProjectionReader + FieldMappingReader + PredicateBatchReader
+/// chain in COUNT(*) queries. Instead of materializing Arrow batches, it directly iterates
+/// merged KeyValue objects and counts them.
+///


I’m not quite following this comment. Why would entire replace FieldMappingReader? If merging is needed, aren’t those still necessary? Please double check all the comments @dalingmeng

Also PredicateBatchReader is only optional, and in most cases, there is no PredicateBatchReader.

Copilot AI review requested due to automatic review settings May 29, 2026 02:49

Copilot started reviewing on behalf of lucasfang May 29, 2026 02:49 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

Comment thread src/paimon/common/utils/path_util_test.cpp Outdated

Comment thread include/paimon/reader/count_reader.h

Comment thread src/paimon/CMakeLists.txt Outdated

Comment thread src/paimon/core/operation/count_split_read.cpp Outdated

lucasfang force-pushed the count_optimize branch from b9b5998 to efc89c0 Compare May 29, 2026 03:48

zjw1111 reviewed May 29, 2026

View reviewed changes

Comment thread src/paimon/core/operation/count_split_read.cpp Outdated

lucasfang force-pushed the count_optimize branch 3 times, most recently from 469ae2d to 9ffcc24 Compare May 29, 2026 06:47

lxy-9602 reviewed May 29, 2026

View reviewed changes

Comment thread include/paimon/table/source/table_read.h

lxy-9602 reviewed May 29, 2026

View reviewed changes

Comment thread src/paimon/common/utils/path_util_test.cpp

lxy-9602 reviewed May 29, 2026

View reviewed changes

lucasfang force-pushed the count_optimize branch from 47be84b to 56d887a Compare June 1, 2026 03:16

lucasfang changed the title ~~feat: optimize count(*) for pk table~~ feat: optimize count(*) for pk/append table Jun 1, 2026

lxy-9602 reviewed Jun 1, 2026

View reviewed changes

lucasfang added 11 commits June 1, 2026 17:57

feat: optimize count(*) for pk table

468426a

fix

dc336c4

fix

b79c520

fix

d2505d0

fix

2d380a8

fix

318b7c7

fix

92cf492

fix

1d988ab

fix

39beb2b

fix

0359892

update merge row count & support data evolution merge row count

22b9416

lucasfang force-pushed the count_optimize branch from b60c8b7 to 22b9416 Compare June 1, 2026 09:57

fix

1b5312c

Conversation

lucasfang commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lxy-9602 commented May 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lxy-9602 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lxy-9602 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

lxy-9602 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

lxy-9602 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lucasfang commented May 29, 2026 •

edited

Loading