Ok, i'll continue with explaining how i would tackle the second problem you encountered.

- "Fade" the ocean around the coastal regions to signify shallower waters and make the land pop. I also haven't been successful with adding different "ocean" layers and adjusting the opacity to achieve different tones as several artists have suggested in other threads.

First i want to delve into some things. The Brush Dynamics are a rather nice feature of Gimp. Krita and also PS seem to have more powerful tools for that, but you still can do a lot of awesome stuff with it in Gimp. There is a nice range of funny dynamics preinstalled and i can only recommend to take a look at them sometimes. You can even customize and tweak them to your liking (to a certain extent), although i have to admit, that i haven't used the possibility to create or customize my own brush dynamics too much in the past and many features are still pretty much unknown to me.
Yet for one thing it is quite necessary to create a new brush dynamic, that is for making nice textures that don't look too regularly patterned. You can achieve this by making your brush rotate randomly. Ofc if you have a clean round brush it won't matter too much, but if you want to work with a fuzzy/grungy brush for making textures for your maps (and that's exactly what you want to do, right?) it looks much better, i promise

= Excursus: How to create a "rotating" brush dynamic in gimp =
The steps shown here are from my tutorial about "Parchment Background for Titles, Legends, etc. in Gimp" where i covered this too.

Click on the Dynamics button in your Paintbrush Tool Options dialog and then Open the dynamics selection dialog.

Here you choose Create a new dynamics.

Name the new Dynamic "Rotation" and click in the table on the "Angle / Random" box.

Congratulations, now you have a custom made brush dynamic that is only rotating!

Ok, now finally we have all the tools ready to come to the actual task, to create some nice fading shallow waters around the coastline. Here is how i like to do them:

Create a new layer, "Shallow Waters" might be a good name.
Set the Layermode to Overlay.
Have you saved the selection of your landmass? If not just select it again and do it with Select > Save to Channel
If you did, go to the Channels Menu.
Right-click the saved Selection Mask.
Choose Channel to Selection.
Set your Foreground color to white. Grab a nice grungy brush (for the shown examples below, i used the Brush called "Acrylic 04#2 i saw in your list of brushes in your screenshot) and under Tool Options set the Opacity to 20-30%.
You got to experiment with the brushsize, because it depends on the size of your selection/landmass.
Make sure, to set the Spacing of your brush to 20-30!
Your setup should now look similar to this:
Click image for larger version. 

Name:	Step4.jpg 
Views:	42 
Size:	243.9 KB 
ID:	91299
It does? Fine!

Now it gets a bit tricky, since there are several options or paths to follow. None of them is as "clean" as i think you could do it wit Photoshop, and i use them for different things.

Basically the whole technique is about stroking with your grungy brush along your selection or your path. (Edit>Stroke Selection or Edit<Stroke Path) You can create a path from your selection by Select>To Path, but in my example here i go with just the selection for the start. In the above mentioned Tutorial the same thing is done by stroking the path.

Make sure you've selected the landmass not the water around it. If you Edit>Stroke Selection now, a window appears.
Click image for larger version. 

Name:	Step5.jpg 
Views:	27 
Size:	247.9 KB 
ID:	91304
Choose Stroke with a paint tool, and choose Paintbrush. Activate Emulate brush dynamics.
With the low opacity and a rather small brush it might look as if nothing happens at first, that's why you repeat this step for maybe 4-5 times, each time enlarge the size of your brush. In the example i started with 25, then i doubled the size to 50, then to 100, 200, 400, .... To make the shallow waters fade more gently into the deeper waters, you can also lower the opacity step by step.
Here you really have to experiment a bit with the settings until you like it.

As you can see, the brush strokes directly on the selection, so the brush is still on both sides of the selection. You can now just Edit>Clear the brush stroke inside the selection leaving only the part on the water.
Since the effect was rather subtle after going through this, i decided to Duplicate the "Shallow Waters" Layer
and set the Opacity of the Layer to 50%. You can also make every single instance of your Stroke on a seperate Layer if you like, so you can tweak the respective Layer Opacity better afterwards.
Well, this is the result and i hope it is similar to the effect you wanted to achieve.
Click image for larger version. 

Name:	Step6.jpg 
Views:	37 
Size:	247.0 KB 
ID:	91306