OAEP uses a separate hash invocation to hash a (usually empty) label as well as a parameter to MGF1 (mask generation function), used for most of the OAEP padding.
The hash doesn’t have that much impact on the security of OAEP, and for that reason it may be left to this default. Most libraries however use the same hash algorithm for MGF-1 and the hashing of the (always empty) label. Java however defaults to MGF1.
We can easily test this by comparing the standard Java Cipher instantiated using "OAEPWITHSHA-256ANDMGF1PADDING" against one instantiated using "OAEPPadding" and OAEPParameterSpec:
// --- we need a key pair to test encryption/decryption
KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
kpg.initialize(1024); // speedy generation, but not secure anymore
KeyPair kp = kpg.generateKeyPair();
RSAPublicKey pubkey = (RSAPublicKey) kp.getPublic();
RSAPrivateKey privkey = (RSAPrivateKey) kp.getPrivate();
// --- encrypt given algorithm string
Cipher oaepFromAlgo = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING");
oaepFromAlgo.init(Cipher.ENCRYPT_MODE, pubkey);
byte[] ct = oaepFromAlgo.doFinal("owlstead".getBytes(StandardCharsets.UTF_8));
// --- decrypt given OAEPParameterSpec
Cipher oaepFromInit = Cipher.getInstance("RSA/ECB/OAEPPadding");
OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-1"), PSpecified.DEFAULT);
oaepFromInit.init(Cipher.DECRYPT_MODE, privkey, oaepParams);
byte[] pt = oaepFromInit.doFinal(ct);
System.out.println(new String(pt, StandardCharsets.UTF_8));
The code will fail with a padding related exception if you substitute "SHA-256" for the MGF1 as parameter, showing that SHA-1 is indeed the default.
The reason why the long algorithm string is needed is compatibility with other Cipher algorithms. Code written for, for instance "RSA/ECB/PKCS1Padding" doesn’t use any parameters; without the longer string OAEP can therefore not function as drop in replacement.
The mode of operation "ECB" doesn’t mean anything in this context, it should have been "None" or it should have been left out completely. You can only encrypt a single block using the RSA implementation of the SunRSA provider.
If you want to encrypt more data, create a random (AES) symmetric key and encrypt that using OAEP. Then use the AES key to encrypt your specific data. This is called a hybrid cryptosystem as it uses both asymmetric and symmetric primitives to encrypt data.
Note that OAEP is not supported in JDK 7 (1.7) or earlier. OAEP is included in the implementation requirements for Java runtimes since Java 8:
RSA/ECB/OAEPWithSHA-1AndMGF1Padding(1024, 2048)RSA/ECB/OAEPWithSHA-256AndMGF1Padding(1024, 2048)
Some protocols may require you to use SHA-256 or SHA-512 within the padding, as SHA-1 is being deprecated for most use – even if it is not directly vulnerable for this kind of purpose. Neither SHA-224 or SHA-384 make any sense as those are versions of SHA-256 and SHA-512 with a reduced output size, and therefore require more calls to create the padding, without providing any security benefit (and yes, testing this does show a performance disadvantage).
If you have an invalid OAEP ciphertext you should first make sure that the right “default” is being used for the label and MGF1.
It is impossible to wrong any library implementation for choosing their own default; in the end it is up to the protocol to define the hashes used.
Unfortunately no mandatory default exists – which is especially a problem if protocol owners forget to fully specify a configuration for the algorithms.