Chris forwarded this interesting essay to the group:
The Frustrated Gene: Origins of Eukaryotic Gene Expressionby Hiten D. Madhani.
The arguments are not airtight, but how could they be anyway? A friend of mine has said that these efforts are necessarily a connect-the-dots exercise with the dots spread very far apart. It is not at all easy to figure out how these ancient devices came about. Nevertheless, the presented idea that the complexity of gene regulation was created to temporarily ward off selfish DNA makes more sense than it having arisen primarily to actually enable complexity. I think that the perspective still leaves open the possibility that we've kept all those adaptations because they do enable complexity, but I am nowhere near qualified to judge that for sure.
What is clear is that looking at every aspect of gene regulation in eukaryotes and thinking that it has to create a net benefit to the organism at the present, rather than maybe resulting from a prior bottleneck, a prior arms race with a parasite, or even the defeat at the hands of a parasite, may be a mistake. It is terrifying how natural selection is rather helpless in preventing the spread of even damaging transposons in a sexual population. Biology is supposed to make sense in light of evolution, but it may be that no organism may make sense if we require it to be in all aspects evolutionary optimal.
I recall one of Marc Kirschner's book saying that it was at the time an ongoing embarrassment to evolutionary biologists that they don't have a solid explanation for why sexual reproduction is advantageous. They give us the simple story that it enabled complexity and evolvability, but they themselves are not convinced that the numbers work out, or at least they weren't a decade or two ago. It is hard for us to even make a case for sexual reproduction, and yet some of us are losing sleep trying to explain things like apparent gene redundancy.
It is possible that us larger eukaryotes at least operate in too big a variety of circumstances, and perhaps we compete more on the basis of behavior than on the basis of physiology. Perhaps our environment changes too rapidly, and we reproduce too slowly, to allow us to reach our biochemical evolutionary pinnacle. I've been watching "The Life of Mammals" series from BBC. It seems far easier to explain the extraordinary diversity in behavior and shapes of animals on evolutionary grounds, than it is to explain even basic things like their wildly different DNA content.