Thinking vs. doing in science

Lately, I've been thinking some about whether it's always best to think about experiments before doing them.  I know, of course, you have to think about experiments you do on some level.  I guess I mean more big picture thinking about "well, is this experiment informative, what will I learn from this?" kinda stuff.  I know that on the face of it, every experiment should make sense, and that the abstract notion of the scientific method we learn in grade school, every experiment should be designed to test some scientific hypothesis.  But on the other hand, most of the interesting findings that we have come across did not really arise in this way.  If we really thought through those experiments rationally, I think we would have ended up with considerably more boring results.  I'm guessing a lot of people have had this experience, where a little side-experiment ends up being super interesting and turns into a whole line of work unto itself.  And sometimes it doesn't make sense, but you just do it and see what happens.

That said, it's important to know where the line is between interesting side-experiment and hopeless distraction.  Also, one of the most important skills to develop as a scientist is the ability to take the seed of something interesting and turn it into a coherent and logical argument through carefully planned and executed experiments.  But I guess what I'm saying is that it's important to let the mind wander sometimes.  A question: how do people find those seeds?  Is it luck?  Hard work?  A "nose" for good problems?  Or is it a skill that one can acquire and hone with experience?  I think that last one is actually more true than people think...