For brevity, I’m using a fake root key. In practice, replace Hive_Key with HKEY_LOCAL_MACHINE for system default settings, or HKEY_CURRENT_USER for per-user settings. Either of these keys are allowed to exist, or both. If they both exist, the HKCU key takes precedence.
To associate an extension with a file type, you need to set the default value of the extension key (Hive_Key\Software\Classes\.ext) with a chosen file type, by setting the default key value.
The actual program launched, as well as other file type details, are found in the file type. File types are noted by what is referred to as a ProgID (short for “Programmatic Identifier”, which is a more easily readable version of a Class Identifier). ProgID keys are found in Hive_Key\Software\Classes, and an example value for this illustration might be ext_auto_key.
The ProgID may have a default value, which will be the friendly description of the file type in Explorer (such as, “Microsoft Word Document”). It’s up to you to make sure you choose a description that’s easily understandable for users.
The ProgID may have a subkey, DefaultIcon, which is where the file type icon is stored. That icon path is the default value of that key.
The ProgID key may a subkey, shell, which will contain the context menu items on the files, and the program that that context menu item will invoke. Similar to the default value of the ProgID, the default value of the verb key is the text which will show up on the context menu. The default value of this shell key has the default verb key name, which is the verb invoked when the user double clicks a file.
These context menu items are Verbs. For our example, a verb that opens the file with Notepad would look like this:
Hive_Key\Software\Classes\ext_auto_file\shell\open\command with default value notepad.exe %1.
This is where you would put your program path. If your program is in the system PATH, as notepad.exe is, you don’t need to specify the full path. In the more likely case, you’d need to specify the path to your exe. For testing purposes, you can just set it to be your build directory.
You asked how to check this stuff, and this can be done by first inspecting the default value of the extension key to get the ProgID, then inspecting the shell subkey of the ProgID key to get default verb, then inspecting \shell\verb\command to get the path to the program launched.
It might be enlightening to open regedit.exe and browse those registry keys for other file types to get a better idea of how it all works.
Also, the above is all valid if the particular extension is not under control of a program set as default (Default Programs) in the Control Panel. You can check this status by checking of the existence of the key HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\ Explorer\FileExts\.EXT\UserChoice. If it is, you will need to revoke Default Programs control before your customizations will go into effect. This can be done by deleting that UserChoice subkey.