펄 문자 인코딩에 대한 간략한 가이드

We will normally want to read and write UTF-8 encoded data. Therefore you should make sure that your terminal can handle it. While we will occasionally have to deal with other encodings, and will often want to look at the byte sequences that we are reading and writing and not just the characters they represent, your life will still be much easier if you have a UTF-8 capable terminal. You can test your terminal thus:

$ perl -E 'binmode(STDOUT, ":encoding(UTF-8)"); say "\N{GREEK SMALL LETTER LAMDA}"'

That should print λ , a letter that looks a bit like a lower-case y mirrored through the horizontal axis.

And if you pipe the output from that into hexdump -C you should see the byte sequence 0xce 0xbb 0x0a .

Ideally, your code will only have to care about any of this at the edges - that is, where data enters and leaves the application. That could be when reading or writing a file, sending/receiving data across the network, making system calls, or talking to a database. And in many of these cases - especially talking to a database - you will be using a library which already handles everything for you. In a brand new code-base which doesn’t have to deal with any legacy baggage you should, in theory, only have to read this first section of this document.

Alas, most real programming is a habitation of devils, who will beset you from all around and make you have to care about the rest of it.

Perl can work with strings containing any character in Unicode. Characters are written in source code either as a literal character such as "m" or in several other ways. These are all equivalent:

"m"
chr(0x6d) # or chr(109), of course
"\x{6d}"
"\N{U+6d}"
"\N{LATIN SMALL LETTER M}"

As are these:

chr(0x3bb)
"\x{3bb}"
"\N{U+3bb}"
"\N{GREEK SMALL LETTER LAMDA}"

Non-ASCII characters can also appear as literals in your code, for example "λ" , but this is not recommended - see the discussion of the utf8 pragma below. You can also use octal - "\154" - but this too is not recommended as hexadecimal encodings are marginally more familiar and easier to read.

Internally, characters have a representation, a sequence of bytes that is unique for a particular combination of character and encoding. Most modern languages default to using UTF-8 for that representation, but perl is old enough to pre-date UTF-8 - and indeed to pre-date any concern for most character sets. For backward-compatibility reasons, and for compatibility with the many C libraries for which perl bindings exist, it was decided when perl sprouted its Unicode tentacle that the default representation should be ISO-Latin-1. This is a single-byte character set that covers most characters used in most modern Western European languages, and is a strict superset of ASCII.

Any string consisting solely of characters in ISO-Latin-1 will by default be represented internally in ISO-Latin-1. Consider these strings:

Release the raccoon! - consists solely of ASCII characters. ASCII is a subset of ISO-Latin-1, so the string’s internal representation is an ISO-Latin-1-encoded string of bytes.

Libérez le raton laveur! - consists solely of characters that exist in ISO-Latin-1, so the string’s internal representation is an ISO-Latin-1-encoded string of bytes. The "é" character has code point 0xe9 and is represented as the byte 0xe9 internally.

Rhyddhewch y racŵn! - the "ŵ" does not exist in ISO-Latin-1. But it does exist in Unicode, with code point 0x175. As soon as perl sees a non-ISO-Latin-1 character in a string, it switches to using something UTF-8-ish, so code point 0x175 is represented by byte sequence 0xc5 0xb5. Note that while valid characters’ internal representations are valid UTF-8 byte sequences, this can also encode invalid characters.

Libérez le raton laveur! Rhyddhewch y racŵn! - this contains both an "é" (which is in ISO-Latin-1) and a "ŵ" (which is not), so the whole string is UTF-8 encoded. The "ŵ" is as before encoded as byte sequence 0xc5 0xb5, but the "é" must also be UTF-8 encoded instead of ISO-Latin-1-encoded, so becomes byte sequence 0xc3 0xa9.

But notice that ISO-Latin-1 not only contains ASCII, and characters like "é" (at code point 0xe9, remember), it also contains characters "Ã" (capital A with a tilde, code point 0xc3) and "©" (copyright symbol, code point 0xa9). So how do we tell the difference between the ISO-Latin-1 byte sequence 0xc3 0xa9 representing "é" and the UTF-8 byte sequence 0xc3 0xa9 representing "é"? Remember that a representation is "a sequence of bytes that is unique for a particular combination of character and encoding". So perl stores the encoding as well as the byte sequence. It is stored as a single bit flag. If the flag is unset then the sequence is ISO-Latin-1, if it is set then it is UTF-8.

It is possible to put non-ASCII characters into your source code. For example, consider this file:

my $string = "é";

print "$string contains ".length($string)." characters\n";

from which some problems arise. First, if the file is encoded in UTF-8, how can perl tell when it comes across the byte sequence 0xc3 0xa9 what encoding that is? Is it ISO-Latin-1? Well, it could be. Is it UTF-8? Again, it could be. In general, it isn’t possible to tell from a sequence of bytes what encoding is in use. For backward-compatibility reasons, perl assumes ISO-Latin-1.

If you save that file encoded in UTF-8, and have a UTF-8-savvy terminal, that code will output:

é contains 2 characters

which is quite clearly wrong. It interpreted the 0xc3 0xa9 as two characters, but then when it spat those two characters out your terminal treated them as one.

We can tell perl that the file contains UTF-8-encoded source code by adding a use utf8 . We also need to fix the output encoding - use utf8 doesn’t do that for you, it only asserts that the source file is UTF-8 encoded:

use utf8;
binmode(STDOUT, ":encoding(UTF-8)");

my $string = "é";

print "$string contains ".length($string)." character\n";

(For more on output encoding see the next section)

And now we get this:

é contains 1 character

Hurrah!

At this point a second problem arises. Some editors aren’t very clever about encodings and even if they correctly read a file that is encoded in UTF-8, they will save it in ISO-Latin-1. VSCode for example is known to do this at least some of the time. If that happens, you’re still asserting via use utf8 that the file is UTF-8, but the "é" in the sample file will be encoded as byte 0xe9, and the following double-quote and semicolon as 0x22 0x3b. This results in a fatal error:

Malformed UTF-8 character: \xe9\x22\x3b (unexpected non-continuation byte 0x22,
immediately after start byte 0xe9; need 3 bytes, got 1) at ...

So given that you’re basically screwed if you have non-ASCII source code no matter whether you use utf8 or not, I recommend that you just don’t do it. If you need a non-ASCII character in your code, use any of the many other ways of specifying it, and if necessary put a comment nearby so that whoever next has to fiddle with the code knows what it is:

chr(0xe9);   # e-acute

Strings aren’t the only things that have encodings. File handles do too. Just like how perl defaults to assuming that your source code is encoded in ISO-Latin-1, it assumes unless told otherwise that file handles similarly are ISO-Latin-1, and so if you try to print "é" to a a handle, what actually gets written is the byte 0xe9.

Even if your source code has the use utf8 pragma, and your code contains the byte sequence 0xc3 0xa9, which will internally by decoded as the character "é", your handles are still ISO-Latin-1 and you'll get a single byte for that character. For how this happens see "PerlIO layers" below.

Things get a bit more interesting if you try to send a non-ISO-Latin-1 character to an ISO-Latin-1 handle. Perl does the best it can and sends the internal representation - which is UTF-8, remember - to the handle and emits a warning "Wide character in print". Pay attention to the warnings!

This behaviour is another common source of bugs. If you send the two strings "Libérez le raton laveur!" followed by "Rhyddhewch y racŵn!" to an ISO-Latin-1 handle, then the first one will sail through, correctly encoded, but the second will also go through. You’ve now got two different character encodings in your output stream and no matter what encoding is expected at the other end you’ll get mojibake.

We’ve seen how by default input and output is assumed to be in ISO-Latin-1. But that can be changed. Perl has supported different encodings for I/O since the dawn of time - since at least perl 3.016. That’s when it started to automatically convert "\n" into "\r\n" and vice versa on MSDOS, and the binmode() function was introduced in case you wanted to open a file on DOS without any translation.

The Encode::encode() function takes a string of characters and returns a string of bytes that represent that string in your desired encoding. For example:

my $string = "Libérez le raton laveur!";
encode("UTF-8", $string, Encode::FB_CROAK|Encode::LEAVE_SRC);

will return a string where the character "é" has been replaced by the two bytes 0xc3 0xa9. If the original string was encoded in UTF-8 then the underlying representation of the input and output strings will be the same, but their encodings (as stored in the single bit flag we mentioned earlier) will be different, and the output will be reported as being one character longer by the length() function.

Encode::encode can sometimes for Complicated Internals Optimisation Reasons modify its input. To avoid this set the Encode::LEAVE_SRC bit in its third argument.

It is quite common for us to receive data, either from a network connection or from a library, which is a UTF-8-encoded byte stream. Naively treating this as ISO-Latin-1 characters will lead to doom and disaster, as the byte sequence 0xc3 0xa9 will, as already explained, be interpreted as the characters "Ã" and "©". Encode::decode() takes a bunch of bytes and turns them into characters assuming that they are in a specified encoding. For example, this will return a "é" character:

decode("UTF-8", chr(0xc3).chr(0xa9), Encode::FB_CROAK)

You should consider how to handle a byte stream that turns out to not be valid in your desired encoding and again I recommend use of Encode::FB_CROAK .

The "Encode" module provides some other functions that, on the surface, look useful. They are, mostly, not.

Remember how waaaay back I briefly mentioned that perl’s internal representation for non-ISO-Latin-1 characters was UTF-8-ish and how they could contain invalid characters? That’s why you shouldn’t use encode_utf8 or decode_utf8 . You may be tempted to use Encode::is_utf8() to check a string's encoding. Don't, for the same reason.

You will generally not be calling encode() with a string literal as its input, but with a variable as its input. However, any errors like "Modification of a read-only value attempted" are your fault, you should have told it to Encode::LEAVE_SRC .

Don't even think about using the _utf8_on and _utf8_off functions. They are only useful for deliberately breaking things at a lower level than you should care about.

The UTF8 flag is a reliable indicator that the underlying representation uses multiple bytes per non-ASCII character, but that’s about it. It is not a reliable indicator whether a string’s underlying representation is valid UTF-8 or that the string is valid Unicode.

The result of this:

Encode::encode("UTF-8", chr(0xe9), 8)

is a string whose underlying representation is valid UTF-8 but the flag is off.

This, on the other hand has the flag on but the underlying representation is not valid UTF-8 because the character is out of range:

chr(2097153)

This is an invalid character in Unicode, but perl encodes it (it has to encode it so it can store it) and turns the UTF8 flag on (so that it knows how the underlying representation is encoded):

chr(0xfff8)

And finally, this variable that someone else’s broken code might pass to you contains an invalid encoding of a valid character:

my $str = chr(0xf0).chr(0x82).chr(0x82).chr(0x1c);
Encode::_utf8_on($str);

"\N{GREEK SMALL LETTER LAMDA}"

This may sometimes be preferable to providing the (hexa)decimal codepoint with an associated comment, but it gets awful wordy awful fast. By default the name must correspond exactly to that in the Unicode standard. Shorter aliases are available if you ask for them, via the charnames pragma. The documentation only mentions this for the Greek and Cyrillic scripts, but they are available for all scripts which have letters. For example, these are equivalent:

"\x{5d0}"

\N{HEBREW LETTER ALEF}"

use charnames qw(hebrew);
"\N{ALEF}"                  # א

Be careful if you ask for character-set-specific aliases as there may be name clashes. Both Arabic and Hebrew have a letter called "alef", for example:

use charnames qw(arabic);
"\N{ALEF}"                  # ا

use charnames qw(arabic hebrew);
"\N{ALEF}"                  # Always Hebrew, no matter the order of the imports!

A happy medium ground is to ask for :short aliases:

use charnames qw(:short);
"\N{ALEF}"                           # error
"\N{hebrew:alef} \N{arabic:alef}"    # does what it says on the tin

"\N{U+3bb}"

This notation looks a little bit more like the U-ish hexadecimal notations used in other languages while also being a bit like the \P notation for codepoint names. Unless you want to mix hexadecimal along with codepoint names you should probably not use this, and prefer \X which is more familiar to perl programmers.