SE don't detect VRAM amount on ATI - they dropped support of this feature in drivers. You can set VideoMemorySize to your VRAM megabytes in config/main-user.cfg. But this is not related to your problem.
Why do you want to rescale SE graphics at all? You always can specify any custom window resolution in the display settings menu, and then take a screenshot.
Hotsampling is an on-the-fly way of downsampling to huge resolutions without needing to force it via driver downsampling or DSR or whatever other method. It also lets you specify a bespoke aspect ratio - so I can take a shot directly in 2:3 for example, and not crop afterwards. It's commonplace within the screenshot community. Not all games support hotsampling (it requires that the viewport be resizeable), but SE does.
It means I can explore at my native resolution and aspect ratio (16:9, 2560x1440), change the window to 2:3 or 1:1 or 2.55:1 or whatever to compose a shot, and then momentarily increase the resolution at that aspect ratio, take the shot, then change the window back to its normal size. Obviously, downsampling or hotsampling in this way produces better image quality (downsampling is an excellent method of anti-aliasing, and it also gives models more depth, deepens shadows, makes the image look a lot cleaner, etc), and gives shots that are large enough to be printed at decent sizes if desired.
I haven't tried changing the resolution in the display settings menu. How does it work? Are you suggesting it works in the same way as hotsampling, then? That despite having a 1440p 16:9 monitor I could set the resolution to, for example, 4000x4000px and I would end up with a screenshot that kept those proportions? How would I compose in that aspect ratio? Does it add black bars around the image? Does it shrink the image back down (downsampling) to fit on my monitor or does it expand it beyond the screen (hotsampling)? And in what way would it avoid the problems I'm having at the moment rendering at larger resolutions?
(I will pass on the information about ATI and vram detection to the person I was speaking to yesterday - this might help them somewhat.)