Unicode
Unicode is an industry standard allowing computers to consistently represent and manipulate text expressed in most of the world's writing systems.
Encoding
Unicode can be encoded in several ways. In Windows (and thus Visual Prolog) the most interesting formats are UTF-8 and UTF-16 (Little Endian). UTF is an acronym for Unicode Transfer Format. UTF-8 is an 8-bit character set encoding where standard ANSII characters are kept as they are and more rare (in western languages) characters are encoded using two, three or four byte character sequences.
In UTF-16 Little Endian Unicode is encoded using sequences of 1 or 2 16-bit units. UTF-16 is used internally in Windows API calls. Little Endian means that the least significant byte of each 16-bit unit is stored before the most significant byte.
Byte-order mark
A byte-order mark (BOM) is the Unicode character with hex value 0xFEFF.
It is optionally written in the start of Unicode files to clarify the byte order of the file.
| Encoding | Representation (hexadecimal) |
|---|---|
| UTF-16 Big Endian | FE FF |
| UTF-16 Little Endian | FF FE |
It is also optionally written in UTF-8 files to indicate that UTF-8 format is used. In UTF-8 the values are EF BB BF. It can also be used in other Unicode transfer.
Unicode support in Visual Prolog
In Visual Prolog the data type string represent Unicode strings. 8-bit character strings are represented by the data type string8. PFC by default targets Unicode string support, but some support is given for 8-bit character set support. Files can for example be read and written in 8-bit character sets.
UTF-8 is represented as string8 strings.
Unicode files created using PFC will get a byte order mark.
S = outputStream_file::create("file.text"), % create a Unicode file that starts with a BOM
To create a Unicode file without a byte order mark, create the file as an 8-bit file and then change the stream to Unicode format afterwards:
S = outputStream_file::create8("file.text"), S:setMode(stream::unicode()), % create a Unicode file that starts without a BOM<