Feature/annotation parser improvments by lwasylow · Pull Request #1312 · utPLSQL/utPLSQL

lwasylow · 2026-03-15T02:07:06Z

This is attempt to speed up the annotation parsing for large volume of packages.
It replaces costly lob being open up and closed up with a line scanner and set of short circuits conditions.

On the 1000 auto generated set of sample unit tests packages executing both ut_annotation_parser.parse_object_annotations old vs new we seeing large improvments.

Packages    : 1000
Procs/pkg   : 20
Contexts/pkg: 4
Iterations  : 1
=================================================

-- Phase 1: Performance --
package              | baseline_ms | new_ms | ratio | status
----------------------------------------------------------------------
TST_GEN_PKG_0001     |      14.285 |  1.689 |  .118 | PASS (88.2% faster)
TST_GEN_PKG_0100     |       7.382 |  2.318 |  .314 | PASS (68.6% faster)
TST_GEN_PKG_0200     |       7.538 |   1.64 |  .218 | PASS (78.2% faster)
TST_GEN_PKG_0300     |       7.504 |  1.713 |  .228 | PASS (77.2% faster)
TST_GEN_PKG_0400     |       8.112 |  2.659 |  .328 | PASS (67.2% faster)
TST_GEN_PKG_0500     |       7.696 |  1.666 |  .216 | PASS (78.4% faster)
TST_GEN_PKG_0600     |       7.832 |   2.65 |  .338 | PASS (66.2% faster)
TST_GEN_PKG_0700     |       7.371 |  1.653 |  .224 | PASS (77.6% faster)
TST_GEN_PKG_0800     |       6.988 |  1.585 |  .227 | PASS (77.3% faster)
TST_GEN_PKG_0900     |       7.267 |  1.636 |  .225 | PASS (77.5% faster)
TST_GEN_PKG_1000     |       7.981 |  1.773 |  .222 | PASS (77.8% faster)
----------------------------------------------------------------------
Baseline total ms : 7762.393
New total ms      : 1783.356
Overall ratio     : .23
Improvement       : 77%
Performance check : PASS — new code is at least 20% faster

-- Phase 2: Cache presence --
Expected in cache : 1000
Found in cache    : 1000

-- Phase 3: Annotation count correctness --
Packages checked  : 1000
Count mismatches  : 0

…ture/annotation_parser

github-actions · 2026-03-15T02:07:44Z

Test Results

1 files 102 suites 2m 39s ⏱️
1 109 tests 1 105 ✅ 4 💤 0 ❌
1 113 runs 1 105 ✅ 8 💤 0 ❌

Results for commit aa3bfe7.

♻️ This comment has been updated with latest results.

codecov · 2026-03-15T02:08:54Z

Codecov Report

❌ Patch coverage is 95.62842% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.02%. Comparing base (c243e4e) to head (aa3bfe7).
⚠️ Report is 13 commits behind head on develop.

Files with missing lines	Patch %	Lines
source/core/ut_utils.pkb	93.63%	7 Missing ⚠️
source/core/annotations/ut_annotation_parser.pkb	98.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1312      +/-   ##
===========================================
- Coverage    95.04%   95.02%   -0.03%     
===========================================
  Files          111      111              
  Lines         5369     5444      +75     
  Branches        33       30       -3     
===========================================
+ Hits          5103     5173      +70     
- Misses         266      271       +5

Flag	Coverage Δ
19se	`95.06% <95.58%> (-0.03%)`	⬇️
21XE	`95.21% <95.58%> (-0.03%)`	⬇️
23-free	`94.96% <95.62%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jgebal · 2026-03-15T11:21:14Z

Can you also include the code you have used for large-scale testing of scanning?
Perhaps you could put it inside the examples/

Is the change related only to how the comment removal is done for lines scanning?

jgebal · 2026-03-15T11:33:09Z

Please assure that the coverage on new functionality is higher.
Many code blocks are not covered.
https://app.codecov.io/gh/utPLSQL/utPLSQL/pull/1312?src=pr&el=tree

Added example

…ctions

lwasylow · 2026-03-15T14:45:43Z

Can you also include the code you have used for large-scale testing of scanning? Perhaps you could put it inside the examples/

Is the change related only to how the comment removal is done for lines scanning?

No the change is related to replace_multiline_comments and switching parse_object_annotations to use a lines rather than clob and its associated procedures.

    l_source := ut_utils.replace_multiline_comments(l_source);
    l_comments := extract_and_replace_comments(l_source);
    add_procedure_annotations(l_annotations, l_source, l_comments);

Added the file to example.

lwasylow · 2026-03-15T14:46:20Z

Please assure that the coverage on new functionality is higher. Many code blocks are not covered. https://app.codecov.io/gh/utPLSQL/utPLSQL/pull/1312?src=pr&el=tree

I will update tests.
There are also some sonnar scanners coming up so will update these too to resolve them where valid/sensible.

…tation parser

…results in annotation processing

jgebal

Nice improvement.

Some fixes are required and old code needs to be removed.

I wonder where did the need for this improvement came from?

Do you actually see this as a problem in day-to-day use of utPLSQL?

source/core/annotations/ut_annotation_parser.pks

source/core/ut_utils.pkb

source/core/annotations/ut_annotation_parser.pks

source/core/annotations/ut_annotation_parser.pkb

source/core/ut_utils.pkb

source/core/ut_utils.pks

source/core/ut_utils.pkb

lwasylow · 2026-03-15T18:02:33Z

Nice improvement.

Some fixes are required and old code needs to be removed.

I wonder where did the need for this improvement came from?

Do you actually see this as a problem in day-to-day use of utPLSQL?

No issues.
Honestly I wanted to go back to do some fun coding other than work stuff and looked for the areas of improvement and I knew from experience that parsing was one of things that took time other than suit building. Felt like a easy win to get back into groove you know :)

Next on table is to add the finally the boolean matcher from 23c :)

- Removed unused test procedures from test_annotation_parser package. - Updated replace_multiline_comments procedure in test_ut_utils to use dbms_preprocessor.source_lines_t for actual and expected results. - Added lines_to_str and make_source functions in main_helper to facilitate conversion between source lines and string formats.

…te documentation

source/core/annotations/ut_annotation_parser.pkb

…ng header stripping functionality

lwasylow · 2026-03-17T22:27:11Z

I'm still working on it.
Getting odd error on my local docker:
on cursor:

java.sql.SQLException: ORA-01000: maximum open cursors for session exceeded
ORA-06512: at "UT3.UT_RUNNER", line 151
ORA-06512: at "UT3.UT_EXECUTABLE", line 155
ORA-06512: at "SYS.DBMS_SQL", line 1838
ORA-06512: at "UT3.UT_EXECUTABLE", line 142
ORA-06512: at "UT3.UT_EXECUTABLE", line 44
ORA-06512: at "UT3.UT_EXECUTABLE_TEST", line 178
ORA-06512: at "UT3.UT_EXECUTABLE_TEST", line 38
ORA-06512: at "UT3.UT_TEST", line 80
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_RUN", line 59
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_RUNNER", line 144
ORA-06512: at line 1

Weird it run ok in docker in the CICD pipeline ....

lwasylow · 2026-03-17T22:45:59Z

I'm still working on it. Getting odd error on my local docker: on cursor:

java.sql.SQLException: ORA-01000: maximum open cursors for session exceeded
ORA-06512: at "UT3.UT_RUNNER", line 151
ORA-06512: at "UT3.UT_EXECUTABLE", line 155
ORA-06512: at "SYS.DBMS_SQL", line 1838
ORA-06512: at "UT3.UT_EXECUTABLE", line 142
ORA-06512: at "UT3.UT_EXECUTABLE", line 44
ORA-06512: at "UT3.UT_EXECUTABLE_TEST", line 178
ORA-06512: at "UT3.UT_EXECUTABLE_TEST", line 38
ORA-06512: at "UT3.UT_TEST", line 80
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_SUITE", line 66
ORA-06512: at "UT3.UT_RUN", line 59
ORA-06512: at "UT3.UT_SUITE_ITEM", line 50
ORA-06512: at "UT3.UT_RUNNER", line 144
ORA-06512: at line 1

Weird it run ok in docker in the CICD pipeline ....

Must been something with my image...
Downloaded one we are using and worked fine.
Oh well few hours wasted tracking cursor leak ....

I will fix rest of the things later. Managed to remove dead code will have to now refactor some of the new one.

…y in ut_utils

…procedure

…d constants

sonarqubecloud · 2026-03-18T22:20:28Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

jgebal · 2026-04-13T15:20:06Z

source/core/annotations/ut_annotation_parser.pkb

+    end loop;
+  end add_procedure_annotations;
+
  function extract_and_replace_comments(a_source in out nocopy clob) return tt_comment_list is


@lwasylow
Can you remove that function? It's not used anymore.

lwasylow added 22 commits March 9, 2026 22:47

Testing something

38e02ab

Another test

461ad5d

Test

44f5495

Thats funny:)

b5f2dc7

that is rubbish

0dd7dba

revert test

4ea0fcb

Small tweaks

17604d8

one at the time

5869f32

Move out outside

3f8d2cf

fix invalid locator

37fae24

revert

5189955

Changing a code to validate by lines instead of clobs for performance.

5a21774

Fixing loop syntax

39f5d64

Merge branch 'develop' of https://github.com/utPLSQL/utPLSQL into fea…

93ffb72

…ture/annotation_parser

Update code

393cfec

Update block

8d794f0

Small fixes

a32e09d

Update comment

b2e914f

Introduce global variable

66de806

Cleanup

f82a3b5

Optimization fiurther

934df67

Trim spaces

65145c2

lwasylow added 2 commits March 15, 2026 14:17

Added extra tests.

8105179

Added example

Enhance annotation parser with new tests and source line handling fun…

cc491cf

…ctions

lwasylow added 2 commits March 15, 2026 14:46

Add tests for Windows-style newlines and long procedure names in anno…

6717ad3

…tation parser

Fix regex extraction for procedure/function names and ensure ordered …

6c513be

…results in annotation processing

jgebal requested changes Mar 15, 2026

View reviewed changes

lwasylow added 4 commits March 15, 2026 21:20

Update tests

46f3d12

Remove redundant parse_object_annotations function overloads and upda…

fecbcc1

…te documentation

Update code to avoid

2547b40

jgebal reviewed Mar 17, 2026

View reviewed changes

source/core/annotations/ut_annotation_parser.pkb Show resolved Hide resolved

Enhance annotation processing by improving SQL text handling and addi…

40eb166

…ng header stripping functionality

lwasylow added 3 commits March 18, 2026 21:44

Refactor annotation processing and enhance line scanning functionalit…

1cdeb9c

…y in ut_utils

Remove redundant exception handling in build_annot_cache_for_sources …

c4d0ab0

…procedure

Refactor annotation patterns in ut_annotation_parser and remove unuse…

aa3bfe7

…d constants

lwasylow requested a review from jgebal March 24, 2026 21:01

jgebal requested changes Apr 13, 2026

View reviewed changes

Conversation

lwasylow commented Mar 15, 2026

Uh oh!

github-actions bot commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

codecov bot commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jgebal commented Mar 15, 2026

Uh oh!

jgebal commented Mar 15, 2026

Uh oh!

lwasylow commented Mar 15, 2026

Uh oh!

lwasylow commented Mar 15, 2026

Uh oh!

jgebal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lwasylow commented Mar 15, 2026

Uh oh!

Uh oh!

lwasylow commented Mar 17, 2026

Uh oh!

lwasylow commented Mar 17, 2026

Uh oh!

sonarqubecloud bot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Quality Gate passed

Uh oh!

jgebal Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Mar 15, 2026 •

edited

Loading

codecov bot commented Mar 15, 2026 •

edited

Loading

sonarqubecloud bot commented Mar 18, 2026 •

edited

Loading