refact: Reuse RowGroupPageIndexReader across columns to improve page-level predicate pushdown performance regression on wide tables#316
Conversation
…edRowGroupReader::ReadFilteredRowGroup to avoid performance drop on wide tables
|
|
|
cc @liangjie3138 . This PR optimizes the RowGroupPageIndexReader usage introduced in #232. Please take a look when you have a chance. I'm wondering if you have any plans to push bitmap filters down to the page level. If not, I'd like to pick this up — it would be a natural next step and could further improve point-query performance on wide tables. |
zhf999
changed the title
lxy-9602
changed the title
LGTM 👍 Go ahead with the page-level bitmap pushdown. |
4 participants
Purpose
Linked issue: close #137
In
PageFilteredRowGroupReader::ReadFilteredRowGroup, the original implementation passed PageIndexReader into ReadFilteredColumn, where each column calledpage_index_reader->RowGroup(row_group_index)to reconstruct aRowGroupPageIndexReader. TheRowGroup()call reads and parses all ColumnIndex and OffsetIndex metadata for every column in that row group. For wide tables (hundreds or even thousands of columns), the same metadata was redundantly read N times (N = number of columns being read), causing page index read time to scale linearly and resulting in a significant performance drop for point-query scenarios where page-level predicate pushdown was supposed to help.Fix: Simply Hoist the RowGroupPageIndexReader creation out of the per-column loop in ReadFilteredRowGroup — construct it once and reuse it across all columns. Change the ReadFilteredColumn parameter type from PageIndexReader to RowGroupPageIndexReader, eliminating redundant metadata reads.
Tests
On a test case with a wide table containing over 1,000 columns, the original code took significantly longer to execute queries. The flamegraph clearly shows that RowGroup() consumed a substantial portion of the total execution time:
After applying the changes in this PR, range query performance on the wide table shows a significant improvement:
The redundant RowGroup() calls are no longer visible in the flamegraph:
API and Format
Not API and format changes.
Documentation
No new features were introduced.
Generative AI tooling
Claude code (Opus 4.6)