Oftentimes the justification in science is that "it happens to work". In my university physics program I worked with a project for imaging the atomic structure of metal surfaces using a scanning tunneling electron microscope, and I saw how they make the probe tips. It felt like the most redneck, questionably unscientific method for making such a high precision instrument that I had ever seen. It happens to work. And you test that it works by imaging with it.
So we see that reconstructions of temperature are not just mappings of temperature change at the location of proxy. They are determinations of temperature by how the proxy behavior relates to the surrounding climate dynamics. Oftentimes there are relationships that are widely separated in space. But these aren't merely assumptions -- they are testable relationships. Usually there is a good physical understanding for why they work the way they do, but even if there isn't, they make predictions that you can check. They're also not perfect, and sometimes the nature of the relationship is "magic", but they happen to work for producing a sensible, coherent record of planetary temperature. And you can test that this works in a variety of ways.
Uncertainty in this record is based on statistical analysis of the coverage and resolutions of the different proxies, and variations in the results between different reconstructions and methods (the "spaghetti graphs"). Confidence in the record comes from broad agreement between these reconstructions, with known sources of change in radiative forcing and the fundamental physics, and our knowledge of drivers and effects of climate change throughout Earth history.
The resulting picture is quite clear and robust. Global temperature through the Holocene slowly declined, and now it is rising rapidly with the enhanced greenhouse effect. The magnitude of this forcing now dominates all other natural factors, and changes in the greenhouse forcing are key to understanding not only this period, but much of geologic history. Geology even provides us with a really good example of what happens as a result of this experiment we're now performing.