You're correct that it's not explosive on its own. But being a strong oxidizer that is explosive when mixed with fuels is precisely the problem. If it is not stored properly, and especially if it's subject to much heat in a confined space with other stuff around, it can easily result in an explosion, and many of the most destructive industrial accidents have happened for this reason. And it's really the same reasoning for why any chemical storage at an industrial site or a laboratory has such strict guidelines. It's not just in case a particular hazardous chemical leaks out, it's also to maintain their purity and stability, and to limit the possibilities of having things mix in ways that could go really bad.
This advisory by EPA, OSHA and ATF gives a good run-down on the handling of ammonium nitrate and prior accidents with it:
https://permanent.fdlp.gov/gpo45474/AN_advisory.pdf