AES CTR 256 Encryption Mode of operation on OpenSSL

Usually, you will be intending to call AES_ctr128_encrypt() repeatedly to send several messages with the same key and IV, and an incrementing counter. This means you need to keep track of the ‘ivec’, ‘num’ and ‘ecount’ values between calls – so create a struct to hold these, and an initialisation function:

struct ctr_state {
    unsigned char ivec[16];  /* ivec[0..7] is the IV, ivec[8..15] is the big-endian counter */
    unsigned int num;
    unsigned char ecount[16];
};

int init_ctr(struct ctr_state *state, const unsigned char iv[8])
{
    /* aes_ctr128_encrypt requires 'num' and 'ecount' set to zero on the
     * first call. */
    state->num = 0;
    memset(state->ecount, 0, 16);

    /* Initialise counter in 'ivec' to 0 */
    memset(state->ivec + 8, 0, 8);

    /* Copy IV into 'ivec' */
    memcpy(state->ivec, iv, 8);
}

Now, when you start communicating with the destination, you’ll need to generate an IV to use and initialise the counter:

unsigned char iv[8];
struct ctr_state state;

if (!RAND_bytes(iv, 8))
    /* Handle the error */;

init_ctr(&state, iv);

You will then need to send the 8 byte IV to the destination. You’ll also need to initialise an AES_KEY from your raw key bytes:

AES_KEY aes_key;

if (AES_set_encrypt_key(key, 128, &aes_key))
    /* Handle the error */;

You can now start encrypting data and sending it to the destination, with repeated calls to AES_ctr128_encrypt() like this:

AES_ctr128_encrypt(msg_in, msg_out, msg_len, &aes_key, state->ivec, state->ecount, &state->num);

(msg_in is a pointer to a buffer containing the plaintext message, msg_out is a pointer to a buffer where the encrypted message should go, and msg_len is the message length).

Decryption is exactly the same, except that you do not generate the IV with RAND_bytes() – instead, you take the value given to you by the other side.

Important:

0. Do not call init_ctr() more than once during the encryption process. The counter and IV must be initialised once only prior to the start of encryption.

1. Under no circumstances be tempted to get the IV anywhere other than from RAND_bytes() on the encryption side. Don’t set it to a fixed value; don’t use a hash function; don’t use the recipient’s name; don’t read it from disk. Generate it with RAND_bytes() and send it to the destination. Whenever you start with a zero counter, you must start with a completely fresh IV that you have never used before.

2. If it is at all possible that you will be sending 2**64 bytes without changing the IV and/or key, you will need to test for the counter overflowing.

3. Do not omit error-checking. If a function fails and you ignore it, it’s quite possible (even likely) that your system will appear to be functioning normally, but will actually be operating completely insecurely.