-
-
Notifications
You must be signed in to change notification settings - Fork 34.5k
bpo-37966: Fully implement the UAX #15 quick-check algorithm. #15558
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
4025110
2a222da
26892d3
27e8122
3762787
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
is_normalized helper.
This restores a small optimization that the original version of this
code had for the `unicodedata.normalize` use case.
With this, that case is actually faster than in master!
$ build.base/python -m timeit -s 'import unicodedata; s = "\u0338"*500000' \
-- 'unicodedata.normalize("NFD", s)'
500 loops, best of 5: 561 usec per loop
$ build.dev/python -m timeit -s 'import unicodedata; s = "\u0338"*500000' \
-- 'unicodedata.normalize("NFD", s)'
500 loops, best of 5: 512 usec per loop- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -783,9 +783,14 @@ typedef enum {YES = 0, MAYBE = 1, NO = 2} QuickcheckResult; | |
| * | ||
| * Return YES or NO if quickcheck determines the input is certainly | ||
| * normalized or certainly not, and MAYBE if quickcheck is unable to | ||
| * tell. */ | ||
| * tell. | ||
| * | ||
| * If `yes_only` is true, then return MAYBE as soon as we determine | ||
| * the answer is not YES. | ||
| */ | ||
| static QuickcheckResult | ||
| is_normalized_quickcheck(PyObject *self, PyObject *input, int nfc, int k) | ||
| is_normalized_quickcheck(PyObject *self, PyObject *input, | ||
| int nfc, int k, int yes_only) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These "boolean int" parameters could be actual
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh wow it's the future! 🙂 I guess that's a C99 feature, and one of those we now accept using. Good to know. I'm not sure I want to convert the existing two parameters within this PR; it seems like that could muddy the diff a bit with changes that are orthogonal to what it's really doing. But I guess I can see how it looks.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Pushed a change to I was a bit surprised to find that I needed to add an
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I made a quick further commit converting I'm ambivalent about adding it here. It expands the diff a bit -- was:
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What makes the diffstat larger? Aren't you already touching all callers, since you added a parameter?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we want to avoid polluting public headers with a
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Ah, fair enough. I think that's consistent with putting it in a generic internal-only header, though. Within the tree, the very few existing uses are either stdbool, or (in
Hmm true. I guess IWYU doesn't feel like how this codebase is mostly written... but maybe it's best to make it more so.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Here's the extra commit: gnprice@896c970f5
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Okay, we can fix up the rest in a sequel. |
||
| { | ||
| /* This is an implementation of the following algorithm: | ||
| https://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms | ||
|
|
@@ -820,12 +825,17 @@ is_normalized_quickcheck(PyObject *self, PyObject *input, int nfc, int k) | |
| return NO; /* non-canonical sort order, not normalized */ | ||
| prev_combining = combining; | ||
|
|
||
| unsigned char quickcheck = record->normalization_quick_check; | ||
| switch ((quickcheck >> quickcheck_shift) & 3) { | ||
| case NO: | ||
| return NO; | ||
| case MAYBE: | ||
| result = MAYBE; /* this string might need normalization */ | ||
| unsigned char quickcheck_whole = record->normalization_quick_check; | ||
| if (yes_only) { | ||
| if (quickcheck_whole & (3 << quickcheck_shift)) | ||
| return MAYBE; | ||
| } else { | ||
| switch ((quickcheck_whole >> quickcheck_shift) & 3) { | ||
| case NO: | ||
| return NO; | ||
| case MAYBE: | ||
| result = MAYBE; /* this string might need normalization */ | ||
| } | ||
| } | ||
| } | ||
| return result; | ||
|
|
@@ -884,7 +894,7 @@ unicodedata_UCD_is_normalized_impl(PyObject *self, PyObject *form, | |
| return NULL; | ||
| } | ||
|
|
||
| m = is_normalized_quickcheck(self, input, nfc, k); | ||
| m = is_normalized_quickcheck(self, input, nfc, k, 0); | ||
|
|
||
| if (m == MAYBE) { | ||
| cmp = (nfc ? nfc_nfkc : nfd_nfkd)(self, input, k); | ||
|
|
@@ -930,28 +940,28 @@ unicodedata_UCD_normalize_impl(PyObject *self, PyObject *form, | |
| } | ||
|
|
||
| if (_PyUnicode_EqualToASCIIId(form, &PyId_NFC)) { | ||
| if (is_normalized_quickcheck(self, input, 1, 0) == YES) { | ||
| if (is_normalized_quickcheck(self, input, 1, 0, 1) == YES) { | ||
| Py_INCREF(input); | ||
| return input; | ||
| } | ||
| return nfc_nfkc(self, input, 0); | ||
| } | ||
| if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKC)) { | ||
| if (is_normalized_quickcheck(self, input, 1, 1) == YES) { | ||
| if (is_normalized_quickcheck(self, input, 1, 1, 1) == YES) { | ||
| Py_INCREF(input); | ||
| return input; | ||
| } | ||
| return nfc_nfkc(self, input, 1); | ||
| } | ||
| if (_PyUnicode_EqualToASCIIId(form, &PyId_NFD)) { | ||
| if (is_normalized_quickcheck(self, input, 0, 0) == YES) { | ||
| if (is_normalized_quickcheck(self, input, 0, 0, 1) == YES) { | ||
| Py_INCREF(input); | ||
| return input; | ||
| } | ||
| return nfd_nfkd(self, input, 0); | ||
| } | ||
| if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKD)) { | ||
| if (is_normalized_quickcheck(self, input, 0, 1) == YES) { | ||
| if (is_normalized_quickcheck(self, input, 0, 1, 1) == YES) { | ||
| Py_INCREF(input); | ||
| return input; | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes_onlyis a bizarre name for a parameter that causes the function to returnMAYBE.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'm not really a fan of this name; I'd be glad for a better one.
The idea is that this flag says the caller doesn't care about NO vs. MAYBE; it only cares whether the answer is YES.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe invert the sense and call it
want_definitemeaning "process the string until we're sure it's not normalized or we reach the end"?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The algorithm in the standard is the
yes_only = false(orwant_definite = true) case, so I think things are simplest to describe if that's the baseline and the other case is described by contrast with it. Could certainly handle that with some added words, though.Another aspect is that there is an asymmetry between a definite YES and a definite NO. In the
yes_only/!want_definitecase, what the caller really most wants is a definite YES... it's just a definite NO that is no more helpful than a MAYBE. Or another angle on that: if the caller really didn't care about getting a definite answer, it could just assume MAYBE and not bother calling the helper 😉 It's in the hope of getting a YES that it makes the call.More ideas just now: perhaps
maybe_is_no, to mean "a MAYBE is no worse than a NO"?Or
yes_or_maybe, for "there are only two distinct answers, YES and MAYBE". That's basically what I was going for withyes_only, but probably a less paradoxical way of putting it 🙂There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The API also surprised me. I suggest tho rename is_normalized_quickcheck() to is_normalized_impl() and add 2 functions: is_normalized() (yes_only=false) and is_normalized_quickcheck() (yes_only=true).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I like the idea of covering over some of the multiplication of flags with wrapper functions!
I don't like quite these names, though. First, I don't think
is_normalizedis a good name, for either value of the flag, because the function doesn't actually take full responsibility for finding out if the string is normalized -- it can return MAYBE, and then it's up to the caller to go and normalize the string and compare.What the function does do is implement a quick but partial check to try to see if the string is normalized. Specifically it implements one that's defined in the standard, which the standard calls
quickCheckin its sample code, and which uses a property in the UCD calledQuick_Check.More precisely, the yes_only=false case implements that standard quick-check algorithm; the yes_only=true (or yes_or_maybe=true, etc.) case (which is the version in master) is a variation on it. I think the name "quickcheck" or
is_normalized_quickcheck, with no further qualifications, sounds like it will mean the standard's algorithm and it's confusing if it means a different algorithm instead.Here's another possible set of names to use with the same idea:
is_normalized_quickcheck_implis_normalized_quickcheckis the standard's quickcheck (yes_only/yes_or_maybe=false)is_normalized_quickercheckis the variant that's more impatient toreturnand hand responsibility back to the caller (yes_or_maybe=true)is_normalized_quickcheck_quickerOr if those names feel too long:
is_normalized_qc_impl,is_normalized_qc,is_normalized_qc_quicker.How does that sound?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another idea for the flag name (though its name becomes less important if we use Victor's idea of a pair of wrapper functions):
yes_vs_maybe.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't seem like anything is standing out as dramatically better. At least the semantics are clearly documented and easily read from the function.
So, let's go with the current state.