Grauw’s blog

Rick DeNatale blogs about Safely dividing a UTF-8 String in Ruby. The goal is to split a string on a certain number of characters in a language with byte-oriented strings (such as Ruby 1.8 and PHP 5). As UTF-8 encodes certain characters as a sequence of bytes, one needs to make sure this split doesn’t happen mid-sequence.

While working, his method is a bit crude and not as efficient as it could be. The method described iteratively splits the string and then does a ‘valid UTF-8’ test using some built-in Ruby functionality. This creates a number of objects and loops over them only to discard them soon after. Instead, with a little better knowledge of UTF-8, this can be done more efficiently.

Basically what you need to do in order to see if you can split a UTF-8 string at a certain point is to see whether the byte after the split is not within the range 0x80-0xC0 (or 128-192 in decimal notation). If it is, move one byte back (or forward).

UTF-8 is an ASCII-compatible encoding, so bytes < 0x80 represent regular ASCII characters. Characters >= 0xC0 indicate the start of a multi-byte character. Breaking before either of these is safe. Characters in the range 0x80-0xC0 however are always the second, third or fourth byte of an encoded character, and thus you should not break before them or you will be breaking an encoded character mid-sequence. You can take a look at the UTF-8 Wikipedia page if you are interested in a little more details.

Now I am not actually familiar with Ruby, but this should be the code necessary to do a UTF-8-safe split in Ruby:

class String
  def utf8_safe_split_on_char(n)?
    self[n] < 0x80 || self[n] >= 0xC0
  end

  def utf8_safe_split(n)
    if length <= n
      [self, nil]
    else
      until self.utf8_safe_split_on_char(n)
        n = n - 1
      end
      before = self[0, n]
      after = self[n..-1]
      [before, after.empty? ? nil : after]
    end
  end
end