So, after playing with my code some more, I reached a better set of parameters. Here is my new result (blurred and contrasted to reduce the random dots):
Click image for larger version.  Name: image_three.png  Views: 574  Size: 10.2 KB  ID: 55032

Much better! Note the peninsulas, the coves, the small islands off the coast; altogether, a much better broad view of the coastline.

Now, for what I fixed:

In my previous post, you'll note these two variables: Iteration Size Factor and Random Variance Factor. These defined the following:

1. How big are the children a shape places
2. How random are the children a shape places

All of this based off the size of a parent.

Now, the size of a child is equal to the iteration size factor by the size of the parent plus a gaussian distribution around zero with a standard deviation of the variance by the size of the parent. For those of you familiar with a gaussian distribution, you'll know that 96% of the random variables it produces will be within the two standard deviation of the mean, and roughly 70% within one.

Next, looking at each iteration. If we had an iteration size factor of 1, it would mean that (all things being equally) the program would run forever. Slightly less than equal, for a long time, etc. Variance has no effect on how long it'll run (on average) since it will just as often shrink a child as it will grow a child.

As such, instead of having a high ISF and a low RVF, reverse the two; a small ISF (so the program terminates quickly and doesn't overfill) and a large RVF so the coastline varies dramatically. Hence the fairly neat result up there.

Next step towards making it more realistic: increase the size, and switch from using positions in x and y to using latitude and longitude and put a mercator projection on the whole thing.