If you just want to keep it on top of the main window then you can use the SetWindowPos() func and put in the flags to set up the Z order so that its higher than the main window without being topmost. You can use topmost but they bug the hell out of me when using a dialog app. But sometimes they can be useful.
An RC file is a giant PITA because the format for them is very odd and there is no default editor with the free VC that you can download. So I tend not to use a visual editor for them and do it by text. The only downside to that is you have to keep track of the numbers and also drop down combo box default text items is done in hex... WTF.... anyway thats MS for you.
But all you do is take a valid RC file and compile it with the RCCompiler to a .res file and then link it just like compiling C to obj file and linking. Then within the main app you can create the dialog by passing in a value for the ID. Usually thats defined as an IDD_DialogWhatever in the resource.h file.
You should also read about the DDX or the dialog data exchange because that makes life easier passing values in and out of the dialog items. By setting up a few standard macros it will patch the windows message to a function callback and also manage to update variables set for the dialog item using the UpdateData( TRUE/FALSE ) calls.
Dialogs come in Modal and Modeless. Modal ones keep the user locked to it until they dismiss the dialog with the OK or Cancel button/widgets but a modeless one sticks around and you can carry one with the other user interface elements or other dialogs without dismissing it. Modeless ones are just a little harder to code for.
My advice is to program some easy stuff first and get used to the principles before trying to code up topmost modeless dialogs without using an RC file and by creating all the UI elements by hand and patching all the windows messages by hand. Its the super super long winded and hard method to use despite it being the more pure form of coding.
Given the direction you want to head to for your coding, you might even be better off looking into wxWidgets for your dialogs and not using MS style at all. a) its more like what your doing with no RC file and b) its cross platform so you can take the apps to unix, linux, Mac etc and they should compile a version for them as well.