buggrep - Bugs: bug #28275, grep -P should use PCRE_UTF8

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #28275: grep -P should use PCRE_UTF8

Submitter:  D.O.A. <tkzv>
Submitted:  Sun 13 Dec 2009 02:06:05 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Confirmed
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Jump to the original submission

Mon 07 Jan 2013 02:28:23 AM UTC, comment #9: 

I believe that this problem was fixed by the commit:

67436786c110bbb565f0c1b96f1ca1d4586e1c8d

pcresearch: set UTF-8 flag correctly for UTF-8 locales

NIIBE Yutaka <gniibe>
Tue 22 Dec 2009 08:46:10 AM UTC, comment #8: 

That would require conversion to UTF-8 before searching (for non-single-byte, non-UTF-8 character sets) but it is a good idea.

Yet another thing to rewrite. :-)

Paolo Bonzini <bonzini>
Mon 21 Dec 2009 02:49:29 PM UTC, comment #7: 


> Regarding Perl mode, that's unfortunately a bug in PCRE, not grep.


pcregrep that comes with PCRE works fine if the -u switch is specified. Philip Hazel suggested calling pcre_compile() with PCRE_UTF8 flag set, when UTF-8 is involved. I tried patching grep, but ran into some Gentoo-specific difficulties.

D.O.A. <tkzv>
Thu 17 Dec 2009 01:30:21 AM UTC, comment #6: 

The testcase gave me the following results.

% dd if=/dev/urandom bs=1024 count=1024 \
| iconv -c -f ucs-2 -t utf-8 \
| LANG=en_US.UTF8 grep -oha '[a-z]' \
| hexdump -C \
| sed -e 's/^[^ ]*//; s/|.*//; s/ 0a/\
/g' \
| sed -e 's/^ //; s/  / /g; /^$/d'

c5 a3
c5 b7
c5 ad
77
c4 81
c4 89
c5 9b
68
c2 aa
c5 b5
c4 a7
c3
a8
c5 a9
c3 a6
c4 b8
c3 ae
78
c4 ab
c3 a4
c3 a3
c5 9b
c3 bd

Though I don't know what character "c5 a3", "c5 b7", "c5 ad",
etc show, this behavior is designed in glibc.

Norihirio Tanaka <noritnk>
Mon 14 Dec 2009 11:38:16 PM UTC, comment #5: 


> Is this behavior in glibc by design?


In POSIX actually, but yes.

Regarding Perl mode, that's unfortunately a bug in PCRE, not grep.  (Personally I consider grep -P a bad idea, but maybe that's just me).

Paolo Bonzini <bonzini>
Mon 14 Dec 2009 11:10:11 PM UTC, comment #4: 

Does [a-z] mean any Latin-based symbol? Does [а-я] mean any extended Cyrillic? Is this behavior in glibc by design? Thanks for the explanation.

Still, in Perl mode bracket expressions work incorrectly with non-ASCII symbols.

grep -Poha '[бв]' random-symbols.txt

treats the expression in brackets as 4 1-byte synbols: '\xd0\xb1\xd0\xb2'. It gives a column of 1-byte symbols \xb1 \xb2 and \xd0.

D.O.A. <tkzv>
Mon 14 Dec 2009 09:01:07 PM UTC, comment #3: 

As long as it doesn't match cyrillic or arabic or East Asian scripts, it's the intended behavior.  To filter letters make sure to set the environment variable LC_ALL=C when running grep.

Paolo Bonzini <bonzini>
Mon 14 Dec 2009 08:57:33 PM UTC, comment #2: 

No. It matches various non-ASCII symbols. Like
ǹûṣőṏŭṋṽẚęčẉįļẹèĕểöâǩǝŏä

For example type (on UTF-8 system):

dd if=/dev/urandom bs=1024000 count=1 |iconv -c -f ucs-2 -t utf-8 > random-symbols.txt

grep -oha '[a-z]' random-symbols.txt > 'random [a-z].txt'

and you'll see what I mean.

(file #19265, file #19266)

D.O.A. <tkzv>
Mon 14 Dec 2009 09:38:54 AM UTC, comment #1: 

Do you mean that [a-z] also matches for example uppercase?  If so, that is expected and is a result of how collating sequences are defined by glibc.  For example, try "ls -1" and "LANG=C ls -1" on a directory with both uppecase and lowercase file names.

Paolo Bonzini <bonzini>
Sun 13 Dec 2009 02:06:05 PM UTC, original submission:  

In UTF-8 locale if basic or extended regular expressions are selected, ranges like [a-z] or [а-я] seem to match much more symbols, than they should. Simply enumerating all the symbols, e.g. [abcdefghijklmnopqrstuvwxyz] or [абвгдеёжзийклмнопрстуфхцчшщъыьэюя] works fine.

If perl regular expressions are selected (-P switch), ranges with ASCII-only symbols like [a-z] work correctly, but multibyte (both ranges and enumeration) symbols are interpreted as several 1-byte symbols.

D.O.A. <tkzv>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #19266:  random [a-z].txt added by tkzv (835B - text/plain - Random text in UTF and what grep matches to '[a-z]')
file #19265:  random-symbols.txt added by tkzv (169KiB - text/plain - Random text in UTF and what grep matches to '[a-z]')

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gniibe (Posted a comment)
  • -email is unavailable- added by noritnk (Posted a comment)
  • -email is unavailable- added by bonzini (Posted a comment)
  • -email is unavailable- added by tkzv (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-12-22 bonzini StatusInvalid Confirmed
        Open/ClosedClosed Open
        SummaryRanges like [a-z] incorrectly match in UTF systems grep -P should use PCRE_UTF8
    2009-12-14 bonzini Open/ClosedOpen Closed
        StatusNone Invalid
    2009-12-14 tkzv Attached File- Added random [a-z].txt, #19266
        Attached File- Added random-symbols.txt, #19265

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code