Directly Read/Write Handshake data with Memory BIO

Finally I get it to work, I was in the right way:

The function SSL_set_connect_state(ssl) is needed to tell the connection to be prepared for the handshake init. Then, we call SSL_do_handshake(ssl) to start the handshake. This function will return -1 because the handshake was not finished, but we can actually read from the client ssl connection BIO writer and send the data using the protocol we want (in my case, EAP RADIUS packets over UDP).

Client

ctx = SSL_CTX_new(SSLv3_client_method());
SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);

ssl = SSL_new(ctx);
rbio = BIO_new(BIO_s_mem());
wbio = BIO_new(BIO_s_mem());

SSL_set_bio(ssl, rbio, wbio);
SSL_set_connect_state(ssl); 

SSL_do_handshake(ssl); // This will return -1 (error) as the handshake is not finished, we can ignore it.

char buf[4096];
BIO_read(wbio, buf, 4096); // Read from BIO, put data in buffer

// Then use data in buffer to send to the server

The server, in the other hand, should be configured using the credential and private key. Also, instead of SSL_set_connect_state() we should use SSL_set_accept_state() as the server will wait for the client’s handshake hello. Then, we simply write the client handshake hello data to the server BIO reader:

Server

ctx = SSL_CTX_new(SSLv3_server_method()); // This is the server!
SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);

ssl = SSL_new(ctx);
rbio = BIO_new(BIO_s_mem());
wbio = BIO_new(BIO_s_mem());

SSL_set_bio(ssl, rbio, wbio);
SSL_set_accept_state(ssl); // The server uses SSL_set_accept_state

// Here we get the data from the client suppose it's in the variable buf
// and write it to the connection reader BIO.
BIO_write(rbio, buf, strlen(buf));

if (!SSL_is_init_finished(ssl)) {
    SSL_do_handshake(ssl);
}

We can use the the SSL_is_init_finished(ssl) function to check if the handshake was done, and while it is not done we call SSL_do_handshake(ssl), and then read again from the BIO_writer to send data to the client.

This process between client and server should be done until the connection is done (i.e. SSL_is_init_finished(ssl) returns true).

Then, after the handshake is done, you can send secure data between client/server, by using the SSL_read and SSL_write functions. Hope this short explanation is useful for someone!

Leave a Comment