Generally, string comparisons in SQL are controlled by column or expression COLLATE rules. In Android, only three collation sequences are pre-defined: BINARY (default), LOCALIZED and UNICODE. None of them is ideal for your use case, and the C API for installing new collation functions is unfortunately not exposed in the Java API. To work around … Read more
iconv(“utf-8″,”ascii//TRANSLIT”,$input); Extended example
Reposting my post from How do I remove diacritics (accents) from a string in .NET? This method works fine in java (purely for the purpose of removing diacritical marks aka accents). It basically converts all accented characters into their deAccented counterparts followed by their combining diacritics. Now you can use a regex to strip off … Read more
I have done this recently in Java: public static final Pattern DIACRITICS_AND_FRIENDS = Pattern.compile(“[\\p{InCombiningDiacriticalMarks}\\p{IsLm}\\p{IsSk}]+”); private static String stripDiacritics(String str) { str = Normalizer.normalize(str, Normalizer.Form.NFD); str = DIACRITICS_AND_FRIENDS.matcher(str).replaceAll(“”); return str; } This will do as you specified: stripDiacritics(“Björn”) = Bjorn but it will fail on for example Białystok, because the ł character is not diacritic. If … Read more
A correctly formatted UTF8 file can have a Byte Order Mark as its first three octets. These are the hex values 0xEF, 0xBB, 0xBF. These octets serve to mark the file as UTF8 (since they are not relevant as “byte order” information).1 If this BOM does not exist, the consumer/reader is left to infer the … Read more
Use java.text.Normalizer to handle this for you. string = Normalizer.normalize(string, Normalizer.Form.NFD); // or Normalizer.Form.NFKD for a more “compatible” deconstruction This will separate all of the accent marks from the characters. Then, you just need to compare each character against being a letter and throw out the ones that aren’t. string = string.replaceAll(“[^\\p{ASCII}]”, “”); If your … Read more
With ES2015/ES6 String.prototype.normalize(), const str = “Crème Brulée” str.normalize(“NFD”).replace(/[\u0300-\u036f]/g, “”) > “Creme Brulee” Two things are happening here: normalize()ing to NFD Unicode normal form decomposes combined graphemes into the combination of simple ones. The è of Crème ends up expressed as e + ̀. Using a regex character class to match the U+0300 → U+036F … Read more
I’ve not used this method, but Michael Kaplan describes a method for doing so in his blog post (with a confusing title) that talks about stripping diacritics: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others) static string RemoveDiacritics(string text) … Read more
Unidecode is the correct answer for this. It transliterates any unicode string into the closest possible representation in ascii text. Example: accented_string = u’Málaga’ # accented_string is of type ‘unicode’ import unidecode unaccented_string = unidecode.unidecode(accented_string) # unaccented_string contains ‘Malaga’and is of type ‘str’