Introducing IBM's International Classes for Unicode: Making your Java/C++/C Applications Global IBM Center for Java Technology, SV

Introduction to IBM Classes for Unicode

2.7 Extend Word-Break Detection

Word breaks in natural language are not only defined by spaces. For example, when I search in this word processor for the word "checked" with the option "Whole Words" checked, I find the last instance of "checked" even though it is not bounded by spaces (there is a comma at the end). Even if you are using more sophisticated tests for ASCII text, such as checking for various kinds of punctuation, you must now deal with the wealth of possible characters in Unicode, and how they may behave differently in different countries. By using a BreakIterator, you can avoid dealing with these complexities.

Going Word-by-Word

Java

BreakIterator boundary = BreakIterator.getWordInstance();

Boundary.setText(stringToExamine);

Int start = boundary.first();

for (int end = boundary.next();

end != BreakIterator.DONE;

start = end, end = boundary.next()) {

resultString = (source.substring(start,end));

//... Result is resultString

}

C++

BreakIterator *boundary = BreakIterator::createWordInstance();

boundary->setText(&uniString);

start = boundary->first();

for (end = boundary->next();

end != BreakIterator.DONE;

start = end, end = boundary->next()) {

uniString.extractBetween(start, end, resultString);

//... Result is resultString

}

BreakIterator *boundary = T_BreakIterator_createWordInstance(NULL);

T_BreakIterator_setText(boundary, uniString);

start = T_BreakIterator_first(boundary);

for (end = T_BreakIterator_next(boundary);

end != T_BreakIterator_DONE;

start = end, end = T_BreakIterator_next(boundary)) {

T_UnicodeString_extractBetween(uniString, start, end, resultString);

//... Result is in resultString

}

To find out whether a current index is at a word break, you can use the following code (this is in a convenience routine in JDK 1.2).

Testing Word Breaks

Java

if (currentIndex < 0 || currentIndex stringToExamine.length())

return false;

if (currentIndex == 0 || currentIndex == stringToExamine.length())

return true;

return( boundary.following(currentIndex-1) == currentIndex);

Using the convenience method offered in JDK 1.2

return boundary.isBoundary(currentIndex);

C++

return boundary->isBoundary(currentIndex);

T_BreakIterator_isBoundary( boundary, currentIndex);

You can use different break iterators to find word boundaries, line-wrap boundaries, sentence boundaries and character boundaries. The latter may seem mysterious: "character" simply means Unicode character, right? However, what native users consider a single character may not be only a single Unicode character, and user expectations may differ from country to country (e.g. in Danish a + ° is considered a single character).