IBM ShopIBM Support Download
HomeNewsProductsservicesSolutionsAbout IBM
Search 
IBM : developerWorks : Open source overview : IBM Classes for Unicode project : FAQ

IBM Classes for Unicode (ICU) and Internationalization - FAQ

This page contains frequently asked questions about the content provided with the IBM Classes for Unicode as well as basics on internationalization. It is organized into the following sections:

Overview

Q: What are the problems that plague the shipping international versions of my code?
A: Often, companies develop a first version of a program or system to just deal with English. When it comes time to produce a first international version, a common tactic is to just go through all the lines of code, and translate the literal strings.

While this may work once, it is not a pattern that you want to follow. Not all literal strings get translated, so this process requires human judgement, and is time-consuming. Each new version is expensive, since people have to go through the same process of identifying the strings that need to be changed. In addition, since there are multiple versions of the source code, maintenance and support becomes expensive. Moreover, there is a high risk that a translator may introduce bugs by mistakenly modifying code.

Q: What is the modern approach?
A: The general technique used now is to internationalize the programs. This means to prepare them so that the code never needs modification--separate files contain the translatable information. This involves a number of modifications to the code:

Once this process is concluded, you have an internationalized program. To localize that program then involves no changes to the source. Instead, the translatable files are typically handed off to contractors or translation agencies to modify. The initial cost of producing internationalized code is somewhat higher than localizing to a single market, but you only pay that once. The costs of simply doing a localization, once your code is internationalized, is a fraction of the previous cost--and avoids the considerable cost of maintenance and source code control for multiple code versions.

Q: How does Unicode play in internationalization?
A: Unicode is a further step in this process of internationalization. Older codepages were difficult to use, and have inconsistent definitions for characters. Internationalizing your code while using the same code base is complex, since you would have to support different character sets--with different architectures--for different markets.

But modern business requirements are even stronger; programs have to handle characters from a wide variety of languages at the same time: the EU alone requires several different older character sets to cover all its languages. Mixing older character sets together is a nightmare, since all data has to be tagged, and mixing data from different sources is nearly impossible to do reliably.

With Unicode, a single internationalization process can produce code that handles the requirements of all the world markets at the same time. Since Unicode has a single definition for each character, you don't get data corruption problems that plague mixed codeset programs. Since it handles the characters for all the world markets in a uniform way, it avoids the complexities of different character code architectures. All of the modern operating systems, from PCs to mainframes, support Unicode now or are actively developing support for it. The same is true of databases, as well.

Q: So where does ICU come in?
A: Unicode is not a magic wand; it is a standard for the storage and interchange of textual data. Somewhere there has to be code that recognizes and provides for the conventions of different languages and countries. These conventions can be quite complex, and require considerable expertise to develop code for and to produce the data formats. Changing conditions and new markets also require considerable maintenance and development.

ICU provides that code. It provides functions for formatting numbers, dates, times, currencies according to locale conventions, and parsing text in those formats. It provides flexible patterns for formatting messages, where the pattern determines the order of the variable parts of the messages, and the format for each of those variables. These patterns can be stored in resource files for translation to different languages. Similarly, ICU provides code and data to handle the complexities of native language collation, searching, and other processes. It also provides a mechanism for accessing strings from resource files, whereby common strings can be shared across countries that have the same language.

Beyond the fact that it provides tested, state-of-the-art Unicode functionality, perhaps the chief benefit is that it is fully portable cross-platform. Not only is the code portable to a wide variety of platforms, but any customized data formats that drive the code can also be shared -- at runtime -- across different platforms. This can be used to guarantee that you get precisely the same results, for example, when collating on your 390 host as you do on your client PCs.

Q: Can you summarize the alternatives?

  1. Don't internationalize.
    Drawbacks: continually mounting costs, long time-to-market
  2. Internationalize, but without Unicode
    Drawbacks: higher maintenance costs, data interchange problems, unable to handle multilingual markets such as the EU.
  3. Internationalize with Unicode, but you roll your own support.
    Drawbacks: you take on a huge development and maintenance task.
  4. Internationalize with Unicode, with someone else's package
    Drawbacks: you get less functionality, and don't have the same assurance of quality, reliability, and continued development and maintenance.
  5. Internationalize with ICU.
    You get top-notch Unicode capabilities now, across all of your platforms. IBM handles the continued development and maintenance, so you don't have to.

Return to top

Unicode and Internationalization

Q: You talked about Unicode being the right technical approach. But is the market accepting it?
A: The industry is converging on Unicode for all internationalization. For example, Microsoft NT is built on a base of Unicode; AIX, Sun, HP/UX all offer Unicode support. All the new web standards; HTML, XML, etc. are supporting Unicode. The latest versions of Netscape Navigator and Internet Explorer both support Unicode.

Q: Why is Unicode the right product direction?
A: Unicode provides a unique encoding for every character. Once your data is in Unicode, it can be all handled the same way—sorted, searched, and manipulated without fear of data corruption.

Q: What about the Far East support?
A: Unicode incorporates the characters of all the major government standards for ideographic characters from Japan, Korea, China, and Taiwan, and more. The Unicode Standard, Version 2.1 has more than 21,000 ideographic characters. The Unicode Consortium actively works with the IRG committee of ISO SC2/WG2 to define additional sets of ideographic characters for inclusion: The Unicode Standard 3.0 has an additional 6,000 ideographic characters.

Q: How about the Web?
A: Netscape Navigator and Internet Explorer both support Unicode HTML pages. The newest Internet standards, such as XML, are requiring Unicode support.

Return to top

IBM Classes for Unicode (ICU): General Questions

Q: What are some of the chief advantages of ICU?
A: First, it is an industrial strength product.

Q: What do you mean by "industrial strength"?
A: IBM has unequalled strength and depth in multi-platform support. This product has a strong future and will become part of key IBM OS environments.

IBM has recognized expertise in internationalization and Unicode. The ICU team has an unparalleled depth of experience in working with Unicode, and has been working with it since Unicode was founded. The team includes the key developers behind the international classes in the Java Development Kit.

Q: How about other advantages?
A: It has a full range of functionality over world market requirements for Unicode text. IBM has access to the broad sources of information about locale country requirements that are continually being incorporated into ICU. These include:

Q: How about customization? My customers often have special requirements.
A: ICU has superior functionality and control in all of its modules, allowing your programmers to customize output to meet customer needs. These include formatting dates, numbers, times, message, Unicode conformant collations, etc. For C++ customers, the libraries are fully object-oriented and extensible throughout.

Q: Are you Year 2000 compliant?
A: Yes, absolutely.

Return to top

Portability and Conversion

Q: Do you supply a Solaris port?
A: The code is portable to a wide variety of platforms. Reference versions are supplied for Windows, Unix (AIX, Solaris, HP-UX and Linux), and Mac. In addition, several licensees have ported the code to many different platforms including OS/2 and S/390(16 at last count), and we have already incorporated the results of those ports into the source code.

Q: What about porting to even more platforms?
A: Porting is very straightforward. We have isolated all platform-specific code to special files, so only those files may need modification.

Q: Do I need an ANSI C++ Compiler to use the code?
A: You do need a C++ compiler, but it can be almost any version of C++ compiler. Because of the wide variation in C++ compilers, we have coded extremely conservatively, to a least common denominator of language features. In particular, we make no use of templates or exceptions.

Q: Do you supply character codeset conversion?
A: Yes, we supply a full range of codeset converters. These are based on the extensive data that IBM has collected on codeset conversions, both for its own character sets and a vast array of others. The converters support two different types of operations—simple conversions for normal strings, and more powerful buffer-based conversions for efficient large-scale conversions of text data.

Return to top

Licensing and Integration

Q: Do I get access to the source code for ICU?
A: Yes, the standard license for ICU includes full source code.

Q: I need to be able to make changes in the code, and according to my own schedule. Can I do that?
A: Absolutely. The standard license for ICU allows you to make any modifications you want in the source code.

Q: But we have our own programming standards that this will conflict with. How can I handle this?
A: We provide a perl script that can be used to modify all the header and sort files to use short names. You can easily modify this script to produce names in a style of your choosing.

Return to top

How to Internationalize Software

Q: Once I have ICU, do I have to completely internationalize my program?
A: No, you don't. You may only need a subset of the features, and only certain aspects of your program may need to be internationalized. Suppose that you have a server program that is using an XML parser, like XML4C. Your internationalization might be limited to sorting certain fields within XML data based on the country or language. Even if your program does not use Unicode internally, you still often need to translate into and out of multiple character sets.

Q: But suppose that I do want a "checklist" of items to watch for?
A: The basic steps to internationalize software using the ICU are described below.

  1. Translate Strings: The first step to take in preparing your program is to enable translation of display strings (strings that get displayed to the user on the client side) by separating them from the rest of your code. In the ICU the resource bundles provide a general mechanism that allows you to access strings and other objects according to locale conventions.
  2. Remove Concatenation: The ordering of parts of a sentence is different in different languages; this difference can easily lead you into trouble. For example, if you create a string by concatenating "File deleted on " with the date, the localizer is limited to modifying only the string, and not the position of the date. If a language requires verbs to be at the end of the sentence, the localizer is stuck. You can replace concatenation by use of message formatters, which allow the localizer to position the variable information appropriately.
  3. Handle Numbers, Currencies, Dates and Times: Number and date formats can also be used separately, with similar control over their formatting. Number formatters handle general numbers and currencies; date formatters cover both dates and times. To globalize your program, you can either use the defaults for the local language, or provide a pattern that specifies the precise format you want, which is adapted to the language conventions.
  4. Fix String Comparison: Standard string comparisons only do a binary comparison. For display strings this is almost always incorrect. Wherever the ordering or equality of strings is important to the user, such as when presenting an alphabetized list, then use a Collator. Otherwise you will find that you don't equate two strings that your user thinks are equal.
  5. Use Character Properties: If your code assumes that all characters of a given type (such as letters or digits) are the ones in the ASCII range, then it will break with foreign languages. Rather than test for particular ranges of characters, you should use the Unicode character properties wherever possible.
  6. Extend Word-Break Detection: Text boundaries are detected differently for different languages. The ICU provides different break iterators to find word boundaries, line-wrap boundaries, sentence boundaries and character boundaries.
  7. Convert Non-Unicode Text: Whenever you are dealing with different character sets, you are faced with the problems of converting between them. Even if you are not using Unicode, The ICU provides character set converters for converting in and out of Unicode. These provide both high and low level support. At a low level, you have a simple interface for converting strings. At a low level, you have full control over the process, so that you can precisely control buffering, and handle edge cases as appropriate.
  8. Handle Multilingual Text: All of the formats, collators, and other classes in the ICU allow you to pass an explicit Locale as a parameter. This means that you can give the user the choice of which locales to use for display. For instance, you could generate HTML that has French currencies in one column of a table and German currencies in another.

Return to top



PrivacyLegalContact