In my previous post I explained that we needed to encrypt a communication messages between Windows C++/VCL client and PHP based web service. We cannot use SSL and decided to use RSA encryption with the help of low-level functions provided by CryptoAPI at the client side and OpenSSL PHP extension at the server.

We also faced and resolved the key incompatibility problem. See my post about this.

In this post I will describe implementation or RSA encryption/decryption and digital signing.

Important Tips

1. RSA is slow as far as asymmetric encryption is generally slow. Consider use of symmetric encryption for large amounts of data.
2. Key exchange is a subject to possible man-in-the-middle attack. We deploy client and server applications with predefined encryption keypairs and exchange signature keys only in encrypted session.
3. Every RSA keypair is all about three numbers: modulus, private exponent and public exponent. The keypair consists of private and public key. Private key is a combination of modulus and private exponent (and some extra values usually). Public key is a combination of the modulus and public exponent (which is usually 65537).
4. Key length is an important property of the key. It is the length of key modulus in bits. You’ll actually need key length in bytes. Thus, it’s better to get byte length at the early beginning:
   ByteLength = BitLength / 8;
5. You cannot encrypt data longer than ByteLength-11. Instead, you need to split your data into chunks of  ByteLength-11 bytes and encrypt each chunk separately. Due to padding encryption functions always output chunks of ByteLength bytes regardless of input data length.
6. The values RSA operates with such as modulus, exponent, encrypted and decrypted chunks are actually very long unsigned integers. You can’t use normal integer variables to store them (with the only exception for public exponent which is usually short enough). We use PHP strings and VCL memory streams to handle long numbers. Pure C/C++ approach uses memory buffers for them.
7. If you’re going to use digital signing then you need some way to separate message body from its signature. We used a zero byte (\x00) to do so as our messages are UTF-8 encoded and can’t contain zero byte normally.
8. You need openssl_pkey_get_details function to get a key length or export public key for generated key with PHP. The function is not available prior to PHP v5.2. We developed own replacement code for the function based on DER format encoding/decoding. I will describe handling DER format later.
9. You cannot use imported private key to sign messages with CryptoAPI. It only allows usage of generated keys or imported certificate keys for signing.

Regardless of programming language or cryptography library encryption and decryption tasks are usually the same. Below I’ll explain common tasks each with C++/CryptoAPI and PHP/OpenSSL code examples.

My C++ code is based on VCL library. Look here for pure C code examples.

Handling Cryptography Subsystem

As you need to store, use and release handle values I recommend to define a class for encryption subsystem that will do encryption, decryption, signing and verification and a class for RSA key that will manage each key you use. You can use constructors and destructors of that classes for initialization and clean up routines.

I did not explained some minor tasks like getting key lengths, releasing key handles and cleaning key containers here. Please refer to CryptoAPI and OpenSSL documentation.

Initialization

To use CryptoAPI you need to initialize key container first. Then you’ll be able to import and generate keys and perform encryption operations. OpenSSL do not need any function call to initialize it.

C++/CryptioAPI:

Use CryptAcquireContext function to initialize CryptoAPI. We use Unicode version of it:

HCRYPTPROV hCryptProv;
//Clear context
CryptAcquireContextW(&hCryptProv, L"Your product name", NULL, PROV_RSA_FULL,
  CRYPT_DELETEKEYSET);
//Create new context
if (!CryptAcquireContextW(&hCryptProv, L"Your product name", NULL, PROV_RSA_FULL,
  CRYPT_NEWKEYSET)) {
	//Error handling
}

Specify your unique product name as a key container name. On success hCryptProv variable will contain cryptography context handle that you’ll need later to work with keys and perform encryption operations.
Import Keys
Well, you need some keys to perform encryption. We use predefined encryption keys and import them at startup. Notice that CryptoAPI and OpenSSL functions require keys encoded with different formats. I explain key format conversion in the next post.
C++/CryptioAPI:
Use CryptImportKey to import private or public key with CryptoAPI. This function requires a key in PRIVATEKEYBLOB or PUBLICKEYBLOB format.
HCRYPTKEY hKey;
if (!CryptImportKey(hCryptProv, (System::PByte)blob->Memory, blob->Size, 0, 0, &hKey)) {
	//Error handling
}

Here blob variable is a TMemoryStream object containing key blob data. On success hKey variable will contain the handle for imported key.
PHP/OpenSSL:
Use openssl_pkey_get_private or openssl_pkey_get_public function to import private or public key. These functions require a key in PEM format.
$hKey = $private? openssl_pkey_get_private($pem) : openssl_pkey_get_public($pem);
if (!$hKey) {
        //Error handling
}

Here pem variable must contain a string with key data in PEM format. Boolean private variable specifies if the data contains a private key. On success hKey variable will store a handle for the imported key.
Generate and Export Public Keys
You may also need to generate keys on session init. We generate signature keypairs and export public keys to exchange with another party. Once again, notice that CryptoAPI and OpenSSL export keys in different formats.
C++/CryptioAPI:
Use CryptGenKey function to generate a new key and CryptExportKey function to export it.
// Generate key
HCRYPTKEY hKey;
if (!CryptGenKey(hCryptProv, AT_SIGNATURE, length<<16, &hKey)) {
 	//Error handling
}
// Export public key
TMemoryStream* buf = new TMemoryStream;
//Get required buffer size
DWORD dwLen;
if (!CryptExportKey(hKey, NULL, PUBLICKEYBLOB, 0, NULL, &dwLen)) {
 	//Error handling
}
//Prepare buffer
buf->Size = dwLen;
//Write public key to buffer
if (!CryptExportKey(hKey, NULL, PUBLICKEYBLOB, 0, (System::PByte)buf->Memory, &dwLen)) {
	//Error handling
}

Here length variable specifies required key length in bits. On success hKey variable will store a handle for the generated key and buf stream will contain exported PUBLICKEYBLOB data for it.
PHP/OpenSSL:
Use openssl_pkey_new function to generate a new key. Notice that this function requires openssl.cnf file. We use configArgs array to specify the path to it. You can find more information here.
The export code sample uses openssl_pkey_get_details function which is not available for PHP version less than 5.2. I explained workaround for this problem in the next post.
//Generate key
$configArgs = array(
            'config' => $opensslCnfPath,
            'private_key_bits' => $length
);
$hKey = openssl_pkey_new($configArgs);
if (!$hKey) {
           //Error handling
}

//Export public key
$info = openssl_pkey_get_details($hKey);
$pem = $info["key"];

Here length variable specifies required key length in bits. On success hKey variable will store a handle for the generated key and pem string will store exported PEM data for the public key.
Encryption
To encrypt the message you must first sign it and then join the message and the signature into a single package and finally encrypt the whole package.

It’s up to you how to separate the message from its signature in the package. We use zero byte for it.
Signing
Digital signature is actually an encrypted hash value of the message. Unlike normal encryption, digital signature is encrypted with a private key.
C++/CryptioAPI:
Use CryptCreateHash, CryptHashData, CryptSignHash and CryptDestroyHash functions to sign the message. Notice that CryptSignHash function outputs signature value in little-endian order. We use own ReverseStream function to reverse it to big-endian order native to OpenSSL. You can find the function in the first part of the post.
// Create hash object
HCRYPTHASH hHash;
if(!CryptCreateHash(hCryptProv, CALG_SHA1, 0, 0, &hHash)) {
	//Error handling
}

// Calulate hash value
if(!CryptHashData(hHash, (System::PByte)message->Memory, message->Size, 0)) {
	//Error handling
}

// Prepare signature buffer
System::PByte pbSignature;
DWORD dwSigLen;
if(!CryptSignHash(hHash, AT_SIGNATURE, NULL, 0, NULL, &dwSigLen)) {
	//Error handling
}
TMemoryStream* signature = new TMemoryStream;
signature->Size = dwSigLen;

// Sign hash
if(!CryptSignHash(hHash, AT_SIGNATURE, NULL, 0, (System::PByte)signature->Memory,
  &dwSigLen)) {
	//Error handling
}

// Reverse stream
ReverseStream(Signature);  //to big-endian

// Clean up
CryptDestroyHash(hHash);

Here message variable is a TMemoryStream containing message data. On success signature stream will contain a digital signature for the message.
PHP/OpenSSL:
Signing with PHP is much easier. You just need to call openssl_sign function.
if (!openssl_sign($message, $signature, hKey)) {
        //Error handling
}

Here message variable is a string containing message data and hKey is the handle for private signature key. On success signature variable will contain a digital signature for the message.
Encrypting
To encrypt some data with RSA you must split it to chunks of ByteLength-11 bytes, encrypt each chunk and concatenate encrypted chunks.
C++/CryptioAPI:
Use CryptEncrypt function to encrypt chunks of arbitrary data with CryptoAPI. Notice that the function outputs encrypted data in little-endian order. You must reverse byte order for compatibility with OpenSSL.
//Set buffer sizes
DWORD dwOutSize = ByteLength;
DWORD dwInSize = dwOutSize-11;
DWORD dwSize = dwInSize;
//Prepare buffers
TMemoryStream* chunk = new TMemoryStream;
TMemoryStream* encrypted = new TMemoryStream;
bool final = false;
do {
	if (data->Position + dwInSize >= data->Size) {
		final = true;
		dwSize = data->Size - data->Position;
	}
	else {
		dwSize = dwInSize;
	}
	chunk->Position = 0;
	chunk->CopyFrom(data, dwSize);
	// Encrypt data
	if (!CryptEncrypt(hKey, NULL, final, 0, (System::PByte)chunk->Memory, &dwSize,
  dwOutSize)) {
		//Error handling
	}

	//Reverse bytes and add to result
	ReverseStream(chunk);
	encrypted->CopyFrom(chunk, 0);
} while (!final);
delete chunk;

Here data variable must be TMemoryStream containing source data for encryption (a package of message and signature). The hKey variable must contain a handle of the public encryption key. On success encrypted stream variable will store encrypted data for sending to the server.
PHP/OpenSSL:
With PHP you can use str_split function to split source data into chunks and openssl_public_encrypt function to encrypt the chunks.
$chunks = str_split($data, ByteLength-11);
$encrypted = '';
foreach($chunks as $chunk) {
        if (!openssl_public_encrypt($chunk, $res, hKey)) {
                //Error handling
        }
        $encrypted .= $res;
}

Here data variable must contain a string with source data for encryption (a package of message and signature). The hKey variable must contain a handle of the public encryption key. On success encrypted variable will store a string with encrypted data for sending to the client.
Decryption
Obviously, decryption process is an opposite to the encryption. First, you decrypt the incoming package. Then you split the package into the message and the signature. Finally, you verify the message aginst the signature.
Decrypting
To decrypt data you need first to split it into chunks of ByteLength bytes, decrypt each chunk and then concatenate decrypted chunks.
C++/CryptioAPI:
Use CryptDecrypt function to decrypt chunks of data with CryptoAPI. Notice that the function requires encrypted data in little-endian order. You must reverse byte order for compatibility with OpenSSL.
//Set buffer sizes
DWORD dwInSize = ByteLength;
DWORD dwSize = dwInSize;
//Prepare buffers
TMemoryStream* chunk = new TMemoryStream;
chunk->Size = ByteLength;
TMemoryStream* decrypted = new TMemoryStream;

//Loop throgh chunks of data
data->Position = 0;
bool final = false;
do {
	buf->Position = 0;
	if (chunk->Position + ByteLength >= Stream->Size) {
		final = true;
		dwSize = data->Size - data->Position;
	}
	else {
		dwSize = dwInSize;
	}
	chunk->CopyFrom(data, dwSize);
	ReverseStream(chunk); //to little-endian

	// Decrypt data
	if (!CryptDecrypt(hKey, NULL, final, 0, (System::PByte)chunk->Memory, &dwSize)) {
		//Error handling
	}
	chunk->Position = 0;
	decrypted->CopyFrom(chunk, dwSize);
} while (!final);
delete chunk;

Here data variable must be TMemoryStream containing source data for decryption. The hKey variable must contain a handle of the private encryption key. On success decrypted stream will store decrypted data received.
PHP/OpenSSL:
With PHP you can use str_split function to split source data into chunks and openssl_private_decrypt function to decrypt the chunks.
$chunks = str_split($Data, $this->ServerKey->ByteLength);
$decrypted = '';
foreach($chunks as $chunk) {
       if (!openssl_private_decrypt($chunk, $res, $hKey)) {
            //Error handling
       }
       $decrypted .= $res;
}

Verifying
To verify the message you must calculate its hash and verify if it corresponds to the value encrypted in the signature.
C++/CryptioAPI:
Use CryptCreateHash, CryptHashData, CryptVerifySignature and CryptDestroyHash functions to verify the message. Notice that CryptVerifySignature function requires signature value in little-endian order. You need to reverse it for compatibility with OpenSSL.
// Create hash object
HCRYPTHASH hHash;
if(!CryptCreateHash(hCryptProv, CALG_SHA1, 0, 0, &hHash)) {
	//Error handling
}

// Calulate hash value
if(!CryptHashData(hHash, (System::PByte)message->Memory, message->Size, 0)) {
	//Error handling
}

//Verify signature
ReverseStream(signature); //to little-endian
if (!CryptVerifySignature(hHash, (System::PByte)signature->Memory,
  signature->Size, hKey, NULL, 0)) {
	if (GetLastError() == (DWORD)NTE_BAD_SIGNATURE) {
		//Security alert
	}
	else {
		//Error handling
	}
}

// Clean up
CryptDestroyHash(hHash);

Here message and signature variables are memory streams containing the message and its signature. The hKey variable must contain a handle for signature public key. In case signature doesn’t correspond to the message NTE_BAD_SIGNATURE error will happen.
PHP/OpenSSL:
Once again, verification with PHP is much easier. You just need to call openssl_verify function and analyse return result.
$verifyRes = openssl_verify($message], $signature, $hKey);
if ($verifyRes==1) {
    //Signature OK
}
else if ($verifyRes==0){
    //Security alert
}
else {
    //Error handling
}

Here message and signature variables are strings containing message and signature data. The hKey must contain a handle for signature public key. In case signature doesn’t correspond to the message openssl_verify returns zero.
Conclusion
It appears that even with the help of CryptoAPI and OpenSSL libraries implementing RSA encryption still needs a lot of development work. Please forgive me possible omitting of some minor subjects. I did this just to ever finish this very long post. I hope it will help somebody.
Don’t hesitate to write your comments and ask any questions you have.
As the task was a part of a commercial project, I cannot post a complete working example here. You can request a project or consultation at Pumka.net if you need our help implementing encryption in your application.
Continued…
In the next post I explained how we resolved the problem of RSA key format incompatibility between CryptoAPI and OpenSSL, how you can read and write keys in PEM, DER, PUBLICKEYBLOB and PRIVATEKEYBLOB formats as well as how you can provide replacement code for openssl_pkey_get_details function which is not available for PHP versions earlier than 5.2.