grep - Bugs: bug #28275, grep -P should use PCRE_UTF8
You are not allowed to post comments on this tracker with your current authentication level.
bug #28275: grep -P should use PCRE_UTF8
Submitter: | D.O.A. <tkzv> | ||
Submitted: | Sun 13 Dec 2009 02:06:05 PM UTC | ||
Category: | None | Severity: | 3 - Normal |
Item Group: | None | Status: | Confirmed |
Privacy: | Public | Assigned to: | None |
Open/Closed: | Open |
Jump to the original submission
Mon 07 Jan 2013 02:28:23 AM UTC, comment #9: |
NIIBE Yutaka <gniibe> |
Tue 22 Dec 2009 08:46:10 AM UTC, comment #8: That would require conversion to UTF-8 before searching (for non-single-byte, non-UTF-8 character sets) but it is a good idea.
|
Paolo Bonzini <bonzini> |
Mon 21 Dec 2009 02:49:29 PM UTC, comment #7:
|
D.O.A. <tkzv> |
Thu 17 Dec 2009 01:30:21 AM UTC, comment #6: The testcase gave me the following results.
|
Norihirio Tanaka <noritnk> |
Mon 14 Dec 2009 11:38:16 PM UTC, comment #5:
|
Paolo Bonzini <bonzini> |
Mon 14 Dec 2009 11:10:11 PM UTC, comment #4: Does [a-z] mean any Latin-based symbol? Does [а-я] mean any extended Cyrillic? Is this behavior in glibc by design? Thanks for the explanation.
|
D.O.A. <tkzv> |
Mon 14 Dec 2009 09:01:07 PM UTC, comment #3: As long as it doesn't match cyrillic or arabic or East Asian scripts, it's the intended behavior. To filter letters make sure to set the environment variable LC_ALL=C when running grep. |
Paolo Bonzini <bonzini> |
Mon 14 Dec 2009 08:57:33 PM UTC, comment #2: No. It matches various non-ASCII symbols. Like
|
D.O.A. <tkzv> |
Mon 14 Dec 2009 09:38:54 AM UTC, comment #1: Do you mean that [a-z] also matches for example uppercase? If so, that is expected and is a result of how collating sequences are defined by glibc. For example, try "ls -1" and "LANG=C ls -1" on a directory with both uppecase and lowercase file names. |
Paolo Bonzini <bonzini> |
Sun 13 Dec 2009 02:06:05 PM UTC, original submission:
In UTF-8 locale if basic or extended regular expressions are selected, ranges like [a-z] or [а-я] seem to match much more symbols, than they should. Simply enumerating all the symbols, e.g. [abcdefghijklmnopqrstuvwxyz] or [абвгдеёжзийклмнопрстуфхцчшщъыьэюя] works fine.
|
D.O.A. <tkzv> |
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follow 7 latest changes.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2009-12-22 | bonzini | Status | Invalid | Confirmed | |
Open/Closed | Closed | Open | |||
Summary | Ranges like [a-z] incorrectly match in UTF systems | grep -P should use PCRE_UTF8 | |||
2009-12-14 | bonzini | Open/Closed | Open | Closed | |
Status | None | Invalid | |||
2009-12-14 | tkzv | Attached File | - | Added random [a-z].txt, #19266 | |
Attached File | - | Added random-symbols.txt, #19265 |
Powered by Savane 3.13-758e.
Corresponding source code
I believe that this problem was fixed by the commit:
67436786c110bbb565f0c1b96f1ca1d4586e1c8d
pcresearch: set UTF-8 flag correctly for UTF-8 locales