Facebook JSON badly encoded

I can indeed confirm that the Facebook download data is incorrectly encoded; a Mojibake. The original data is UTF-8 encoded but was decoded as Latin -1 instead. I’ll make sure to file a bug report.

In the meantime, you can repair the damage in two ways:

  1. Decode the data as JSON, then re-encode any strings as Latin-1, decode again as UTF-8:

    >>> import json
    >>> data = r'"Rados\u00c5\u0082aw"'
    >>> json.loads(data).encode('latin1').decode('utf8')
    'Radosław'
    
  2. Load the data as binary, replace all \u00hh sequences with the byte the last two hex digits represent, decode as UTF-8 and then decode as JSON:

    import re
    from functools import partial
    
    fix_mojibake_escapes = partial(
         re.compile(rb'\\u00([\da-f]{2})').sub,
         lambda m: bytes.fromhex(m.group(1).decode()))
    
    with open(os.path.join(subdir, file), 'rb') as binary_data:
        repaired = fix_mojibake_escapes(binary_data.read())
    data = json.loads(repaired.decode('utf8'))
    

    From your sample data this produces:

    {'content': 'No to trzeba ostatnie treningi zrobić xD',
     'sender_name': 'Radosław',
     'timestamp': 1524558089,
     'type': 'Generic'}
    

Leave a Comment