I can indeed confirm that the Facebook download data is incorrectly encoded; a Mojibake. The original data is UTF-8 encoded but was decoded as Latin -1 instead. I’ll make sure to file a bug report.
In the meantime, you can repair the damage in two ways:
-
Decode the data as JSON, then re-encode any strings as Latin-1, decode again as UTF-8:
>>> import json >>> data = r'"Rados\u00c5\u0082aw"' >>> json.loads(data).encode('latin1').decode('utf8') 'Radosław'
-
Load the data as binary, replace all
\u00hh
sequences with the byte the last two hex digits represent, decode as UTF-8 and then decode as JSON:import re from functools import partial fix_mojibake_escapes = partial( re.compile(rb'\\u00([\da-f]{2})').sub, lambda m: bytes.fromhex(m.group(1).decode())) with open(os.path.join(subdir, file), 'rb') as binary_data: repaired = fix_mojibake_escapes(binary_data.read()) data = json.loads(repaired.decode('utf8'))
From your sample data this produces:
{'content': 'No to trzeba ostatnie treningi zrobić xD', 'sender_name': 'Radosław', 'timestamp': 1524558089, 'type': 'Generic'}