This can be a touchy subject, and I look forward to being corrected and amplified upon for this one. So I will merely toss out a straw-man here and hope to get the ball rolling.
1: Measuring a specific thing
A: relative
B: absolute measurement
2: Looking for the unexpected
A: known unknowns
B: unknown unknowns
Which should get things going. Ya'll chime in.
For a lot of things, 1-A gets you there. Let's say I have a fusor, and want relative neutron outputs, mainly so I can tell if this change made things better or not. All you need for that is stable relative measurements at least until you start wanting to compare your results with those of others. For that, you need 1-B, and you hope that others who think they have achieved that really have and haven't missed something important. But to have a clue at all if one of the other of you did, you have to get to where the numbers mean something (you think) and if you can't get them to agree with otherwise identical conditions, well, that should be a spur to all to find out why the discrepancy exists. There have been far too many times in science where people fudged their results to agree with some "authority" that later have been blown because some new guy insisted on getting them to agree without fudging, and found some missed assumption the authority made....which resulted in better numbers, after all too long a delay that wasted a lot of time for others working in the same field. We'd rather shuck the egos and just make faster progress here.
These are both very-extremely important for development (ie refinement), but less so for raw discovery unless you got lucky and measured the correct thing that allows the discovery. So things under class 2 are a lot more diverse and interesting, the main difficulty being "how do I set myself up for discovery of things I don't expect". After all, it's easy to say "be prepared" or "expect the unexpected" but not so easy to really do, and not blind yourself to the idea that something not in your personal theory might be happening despite what you think (or if you're intellectually honest, what you thought up till you saw the new thing).
Since I don't myself know how to tell you to actually expect the unexpected as a philosophical issue, truly and fully, a couple of examples may help here.
Let's say I have a neutron counter working in class 1-A here -- it counts neutrons per second, and is stable between experiments. That will tell me if this one is making more neutrons per second than the last one.
But since a count is a count from one of these, I can't tell if I'm getting multiple hits in bursts inside one detector response time window -- I tossed that information out the window when I thresholded the signal and just counted how many of them there were. So there will be a certain tendency for that to lose information, and only tell me what I expect it to. However, if I also send that same signal, raw, to an oscilloscope or an audio amplifier, I might take advantage of my superior in-brain processing to see things that a counter cannot report -- some pulses are wider, or taller, indicating multiple hits that the counter would have reported as just one count. The counter wouldn't notice if the counts were coming at say, a 60hz rate, which might be due either to simple EMI getting into the system, or the fact that I may have a 60hz component in what I'm driving the experiment with, that is affecting it's success -- most of the neutrons may come out during some phase of the ripple of my HV for example, and that would sure be nice to know, but a counter won't tell you things like that, or let you separate them from simple hum on the wiring either.
Here's another one:
Let's suppose (truly) that sometimes I don't have a clue where the charged particles are moving from and in what directions in my fusor. And I'm interested because I've not yet heard anyone else with a theory of why the thing doesn't simply degenerate into a static equilibrium with very little fusion, that doesn't need "armwaving" or "then a miracle occurs" at some point. So I build a pinhole camera I can move around in there and look, and I get a picture somewhat different than what I see with the naked eye, and as I move it, it changes -- obviously I've got some things entering it in straight lines (like X ray photons) and some things that seem to be coming in in beams so they get brighter or darker relatively as I move the camera around, compared to isotropic X ray emission. Well then, the obvious thing to do is fit that camera with some electromagnets so I can move the charged particles around vs the X rays, then I can learn something I couldn't learn from the thing as unmodified. If I actuate the magnets, the X ray picture won't move, but the charged particle one will, and different polarities of particle will move in opposite directions. So I began with a known unknown -- where's the radiation coming from, and discovered that there is more than one type -- an unknown unknown, then hammered that one down. I might then chase this one down by putting in a movable Faraday or Langmuir probe to learn more, but at that point, I'm back in "known unknown" and trying to nail it down -- the basic discovery has been made already.
Wash, rinse repeat -- the basics of science!
I am drawing from actuality in these examples. I thought I could calibrate my 3He neutron counter against say BTI bubble detectors and/or silver activations. Then I discovered that there was divergence between the ratios of the raw numbers there produced from run to run when there were other changes -- for example a run that didn't make the 3He make exciting counts, did make silver and one but not all of the BTI's fill with a larger amount of bubbles that would be indicated from other cross-calibrations, made before. Once I hooked up the tube raw (pre threshold) output to an audio amp the reason became obvious. The run that produced the low counts, but high readings on the other detectors was bursting in its output, so hitting into the tube dead-time -- something I've since been looking into here.
So I guess I should add a little philosophy too -- for one thing, measure everything you can more than one way, and keep track of when they track, or don't -- it could be broken tools, but it might not be.
The other thing, exemplified by Fleming, is if something unexpected, maybe even negative, happens, do go ahead and try to find out why it did and what it means -- had he tossed that contaminated petri dish in the trash, we'd not have penicillin. And most "big science" outfits would do just that -- toss it and tell the lab assistant to get it right next time, instead of making that discovery. Sad but mostly true. This is where we "small science" people have a real advantage....and we need all we can get!