What is character encoding and why should I bother with it

by

(Note that I’m using some of these terms loosely/colloquially for a simpler explanation that still hits the key points.)

A byte can only have 256 distinct values, being 8 bits.

Since there are character sets with more than 256 characters in the character set one cannot in general simply say that each character is a byte.

Therefore, there must be mappings that describe how to turn each character in a character set into a sequence of bytes. Some characters might be mapped to a single byte but others will have to be mapped to multiple bytes.

Those mappings are encodings, because they are telling you how to encode characters into sequences of bytes.

As for Unicode, at a very high level, Unicode is an attempt to assign a single, unique number to every character. Obviously that number has to be something wider than a byte since there are more than 256 characters 🙂 Java uses a version of Unicode where every character is assigned a 16-bit value (and this is why Java characters are 16 bits wide and have integer values from 0 to 65535). When you get the byte representation of a Java character, you have to tell the JVM the encoding you want to use so it will know how to choose the byte sequence for the character.

More Related Contents:

  1. What’s the difference between encoding and charset?
  2. How do I correct the character encoding of a file?
  3. What is the most efficient binary to text encoding?
  4. Should we HTML-encode special characters before storing them in the database?
  5. What is the most common encoding of each language?
  6. Java : How to determine the correct charset encoding of a stream
  7. Using PowerShell to write a file in UTF-8 without the BOM
  8. What is base 64 encoding used for?
  9. Working with UTF-8 encoding in Python source [duplicate]
  10. How to convert Strings to and from UTF8 byte arrays in Java
  11. What is the difference between UTF-8 and Unicode?
  12. How to support UTF-8 encoding in Eclipse
  13. Detect encoding and make everything UTF-8
  14. How to detect the character encoding of a text file?
  15. Base64 Encode String in VBScript
  16. Setting the PHP default encoding to UTF-8
  17. Why does base64 encoding require padding if the input length is not divisible by 3?
  18. How can I transform string to UTF-8 in C#?
  19. Get a list of all the encodings Python can encode to
  20. What is “=C2=A0” in MIME encoded, quoted-printable text?
  21. How to set standard encoding in Visual Studio
  22. ASP.NET Excel export encoding problem
  23. Difference between Url Encode and HTML encode
  24. UTF-8 output from PowerShell
  25. What charset does Microsoft Excel use when saving files?
  26. Unicode value \uXXXX to Character in Javascript
  27. Difference between double and single quotation marks in fortran?
  28. PHP messing with HTML Charset Encoding
  29. Emoji value range
  30. Base64 encode with Stream_StringToBinary inserts a newline, breaking the string?

Leave a Comment