What would be the smallest plausible density of a potentially habitable rocky planet?
According to Space Engineer, SE uses the
mass-radius relationship for planets to determine how big a planet should be for a given mass and type. Figure 4 from this paper is particularly helpful:
This basically shows what you might expect. For a given mass, iron planets are the smallest (most dense), followed by "rocky" worlds with mixtures of iron and silicates, then planets with more water in them, and finally up to gas giants. We can also see that as a planet's mass increases, the size increases more slowly, and eventually flattens out. This is the compression effect you pointed out.
To make it a bit easier for us to interpret here, I used the formulas from the paper, and re-plotted for density vs. mass. I also add a few curves for different types of planets with high carbon content (carbon monoxide, graphite with iron core, and silicon-carbide with iron core).
Now we can clearly see how the density increases due to the compression effect, or read specific values. The lowest density iron planets would be around 9 or 10 g/cm
3, while silicates can be down to about 4 g/cm
3. Mars also gives us a nice benchmark of 3.93 g/cm
3 at 0.107 Earth masses.
And this is probably about as low as we can get for a "rocky" planet. To be less dense than that, we must increase the proportion of lighter materials. Most commonly this will be water, but carbon also works -- and that brings us to the interesting case of
carbon planets.
Currently carbon planets are not specifically shown in Space Engine, though they may be added later. They are thought to arise in systems with a high abundance of carbon relative to oxygen. Their structure would consist of iron, carbide, graphite (maybe even diamond?) and hydrocarbons. Depending on the mixture, a carbon planet will have a density somewhere between a silicate planet and a water planet.
So with those super-low density deserts you're finding, I'm not entirely sure what's going on. They are too low density for a purely rocky planet. Perhaps Space Engine is modelling the planet as a silicate-water mixture (like the dark blue dotted or dash-dotted curves in the first figure), but with no water on the surface and therefore it is classified as a desert instead of a terra, but this is just a guess.