Programming with Unicode Documentation, Release 2011
Multilingual Plane (BMP). There are 3 different implementations of character strings:
• array of 32 bits unsigned integers (the UCS-4 encoding): full Unicode range
• array of 16 bits unsigned integers (UCS-2): BMP only
• array of 16 bits unsigned integers with surrogate pairs (UTF-16): full Unicode range
UCS-4 use twice as much memory than UCS-2, but it supports all Unicode character. UTF-16 is a compromise
between UCS-2 and UCS-4: characters in the BMP range use one UTF-16 unit (16 bits), characters outside this range
use two UTF-16 units (a surrogate pair, 32 bits). This advantage is also the main disadvantage of this kind of character
string.
The length of a character string implemented using UTF-16 is the number of UTF-16 units, and not the number of
characters, which is confusing. For example, the U+10FFFF character is encoded as two UTF-16 units: {U+DBFF,
U+DFFF}. If the character string only contains characters of the BMP range, the length is the number of charac-
ters. Getting the n
th
character or the length in characters using UTF-16 has a complexity of 𝑂(𝑛), whereas it has a
complexity of 𝑂(1) for UCS-2 and UCS-4 strings.
The Java language, the Qt library and Windows 2000 implement character strings with UTF-16. The C and Python
languages use UTF-16 or UCS-4 depending on: the size of the wchar_t type (16 or 32 bits) for C, and the compilation
mode (narrow or wide) for Python. Windows 95 uses UCS-2 strings.
See also:
UCS-2, UCS-4 and UTF-16 encodings, and surrogate pairs.
3.6 Byte string
A byte string is a character string encoded to an encoding. It is implemented as an array of 8 bits unsigned integers.
It can be called by its encoding. For example, a byte string encoded to ASCII is called an “ASCII encoded string”, or
simply an “ASCII string”.
The character range supported by a byte string depends on its encoding, because an encoding is associated to a charset.
For example, an ASCII string can only store characters in the range U+0000—U+007F.
The encoding is not stored explicitly in a byte string. If the encoding is not documented or attached to the byte string,
the encoding has to be guessed, which is a difficult task. If a byte string is decoded from the wrong encoding, it will
not be displayed correctly, leading to a well known issue: mojibake.
The same problem occurs if two byte strings encoded to different encodings are concatenated. Never concatenate
byte strings encoded to different encodings! Use character strings, instead of byte strings, to avoid mojibake issues.
PHP5 only supports byte strings. In the C language, “strings” are usually byte strings which are implemented as the
char
*
type (or const char
*
).
See also:
The char
*
type of the C language and the mojibake issue.
3.7 UTF-8 encoded strings and UTF-16 character strings
A UTF-8 string is a particular case, because UTF-8 is able to encode all Unicode characters
1
. But a UTF-8 string is
not a Unicode string because the string unit is byte and not character: you can get an individual byte of a multibyte
character.
1
A UTF-8 encoder should not encode surrogate characters (U+D800—U+DFFF).
6 Chapter 3. Definitions
剩余68页未读，继续阅读
u011065846
- 粉丝: 0
- 资源: 3
我的内容管理
展开
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测：静态代码分析与数据流方法
- 青苔数据CEO程永：技术生态与阿里云开放创新
- 制造业转型： HyperX引领企业上云策略
- 赵维五分享：航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛：AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器：设计与实现
- 猩便利工程师仲小玉：Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享：高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会：阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
