Yes, that is what I figured as far as Meltdown goes.
But Spectre is a different beast, much more complicated, no NaR’s are involved at all.
“Well, I can imagine attacks along these lines that would let someone figure out which hidden addresses are in caches”
This is precisely the problem. With spectre, you can trick a program to load different areas of a cache depending on secret information. The attacker can then check which cache line was loaded and deduce the secret information based on that. The contents of the cache are not important and can remain inaccessible, what is important is which line was loaded.