Mcrypt js encryption value is different than that produced by PHP mcrypt / Mcrypt JS decrypt doesn’t work for UTF-8 chars

The main issue appears to be that your string_encrypt and string_decrypt PHP functions don’t have access to the $key variable, so for the encryption key mcrypt_encrypt is using \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0. See this question for an explanation. PHP should report a notice that key is undefined, have you turned off error reporting perhaps? Echo the key from inside the encrypt function to confirm this.

Another issue is a bug in the Mcrypt JS library. This library pads the encryption key with \0 if the key length is less than 32 bytes, the problem is that this is not how the PHP mcrypt_encrypt function pads the key. The mcrypt_encrypt function pads the key up to the nearest valid key length (16, 24, or 32 bytes). The issue in mcrypt.js is at lines 63 and 64, change this:

if(key.length<32)
    key+=Array(33-key.length).join(String.fromCharCode(0));

to this:

if(key.length<16)
    key+=Array(17-key.length).join(String.fromCharCode(0));
else if(key.length<24 && key.length>16)
    key+=Array(25-key.length).join(String.fromCharCode(0));
else if(key.length<32 && key.length>24)
    key+=Array(33-key.length).join(String.fromCharCode(0));

Now we can confirm the fix…

PHP:

function string_encrypt($string) {
    $crypted_text = mcrypt_encrypt(MCRYPT_RIJNDAEL_256, "", $string, MCRYPT_MODE_ECB);
    return $crypted_text;
}

$test_str = "This is test message to be encrypted.";
$enc_str = string_encrypt($test_str);
echo bin2hex($enc_str);

Output:
f98fca4ddc4c10d6cd47df56b081b78566ee4facbcf2254b46f7809d9b255529d2078f28b150e802d72818be1888536fac6219f6ce7b9d9332a24afa09288f0e

Javascript:

function toHex(str) {
    var hex = '';
    for(var i=0;i<str.length;i++) {
        var val=""+str.charCodeAt(i).toString(16);
        if(val.length == 1)
            hex += '0'+val;
        else
            hex += val;
    }
    return hex;
}

var enc_str = mcrypt.Encrypt("This is test message to be encrypted.", "", "", "rijndael-256", "ecb");
alert(toHex(enc_str));

Output:
f98fca4ddc4c10d6cd47df56b081b78566ee4facbcf2254b46f7809d9b255529d2078f28b150e802d72818be1888536fac6219f6ce7b9d9332a24afa09288f0e

Finally, all of these encryption functions produce binary as their output. Binary cannot be written as plain text in most cases without damaging the data. To solve this, either encode the binary to Hex or Base64 and then decode it before trying to decrypt.

So to get everything working…

<?php 
$key = 'testtesttesttesttesttesttesttest';

function string_encrypt($string, $key) {
    $crypted_text = mcrypt_encrypt(MCRYPT_RIJNDAEL_256, $key, $string, MCRYPT_MODE_ECB);
    return $crypted_text;
}

function string_decrypt($encrypted_string, $key) {
    $decrypted_text = mcrypt_decrypt(MCRYPT_RIJNDAEL_256, $key, $encrypted_string, MCRYPT_MODE_ECB);
    return trim($decrypted_text);
}

echo $test_str="This is test message to be encrypted.";   echo '<br />';
$enc_str = string_encrypt($test_str, $key);
echo bin2hex($enc_str);                                     echo '<br />';
echo string_decrypt($enc_str, $key);                        echo '<br />';

?>

<script src="https://stackoverflow.com/questions/18786025/rijndael.js"></script>
<script src="mcrypt.js"></script>

<script lang='javascript'>
    function toHex(str) {
        var hex = '';
        for(var i=0;i<str.length;i++) {
            var val=""+str.charCodeAt(i).toString(16);
            if(val.length == 1)
                hex += '0'+val;
            else
                hex += val;
        }
        return hex;
    }
    function hexToString (hex) {
        var str="";
        for (var i=0; i<hex.length; i+=2) {
            str += ''+String.fromCharCode(parseInt(hex.charAt(i)+hex.charAt(i+1), 16));
        }
        return str;
    }
    var enc_str = mcrypt.Encrypt('<?php echo $test_str ?>', '', 'testtesttesttesttesttesttesttest', 'rijndael-256', 'ecb');
    alert(toHex(enc_str));
    alert(mcrypt.Decrypt(hexToString('<?php echo bin2Hex($enc_str) ?>'), '', 'testtesttesttesttesttesttesttest', 'rijndael-256', 'ecb').replace(/\x00+$/g, '')); 
</script>

A few more notes…

1. You cannot trim the output of the string_encrypt function. This will cause leading or trailing zeros to be removed, which will make it so that you cannot decrypt the output.
2. ECB mode is insecure and you really shouldn’t use it. CBC is the way to go. CBC does require an IV, and the IV must be the same for both encryption and decryption.
3. Javascript encryption is not secure for various reasons, given your usage of it anyone could simply view the pages source or debug the running javascript to get the encryption key. Read the link posted by ntoskrnl in your question comments.

Update:

Your Base64 encoding issue occurs because the library you’re using doesn’t work with binary data. This is a fairly common issue for Base64 javascript libraries. I’d recommend using this library instead.

As for the trailing � characters when decrypting with javascript, you need to trim the decrypted output. You’re doing this in your PHP string_decrypt method, but not in your javascript. You can trim the decrypted output by doing a regex replace on all \0 characters at the end of the string.

Example:

mcrypt.Decrypt(dec_str,'').replace(/\x00+$/g, '')

I should have included this in my original post, but I didn’t notice the \0 characters in the output because FF’s alert box doesn’t display them. Sorry about that.

Finally, I noticed another bug in the Mcrypt JS library. Lines 41 to 47:

var ciphers={       //  block size, key size
    "rijndael-128"  :[  16,         32],
    "rijndael-192"  :[  24,         32],
    "rijndael-256"  :[  32,         32],
    "serpent"       :[  16,         32],
    "twofish"       :[  16,         32],
}

Notice the comma at the end of the “twofish” line. Firefox and Chrome don’t seem to mind this, but IE8 will report an error and fail to load the mcrypt library because of it. To fix the issue change:

"twofish"       :[  16,         32],

to:

"twofish"       :[  16,         32]