Ultimate space simulation software

  • 1
  • 2
  • 3
  • 4
  • 5
  • 10
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

10 Nov 2018 10:49

GAIA to SpaceEngine catalogs Works in 0.980, 0.980e and 0.990 What does it do? It adds a certain amount of stars from the Gaia DR2 catalog to SpaceEngine. All the stars are fetched from the Gaia archive, which means that their position relative to Sol will be an accurate representation of real life. The emergence of dust from the absence of stars is even visible with high magnitude limits in versions that do not have a distance limit. Obviously, more stars = more overall resemblance to the actual survey, and thus more resemblance to what it is like in real life.
How to Install: If you are using Steam, skip to workshop. 1. Download one of the paks from the list below
2. Choose ONE version.
3. Extract one pak file into SpaceEngine\addons folder.

GAIA designation is GAIA DR2 [######]

I suggest to install the 1M-11mag.pak version as it very clearly depicts the overall shape of the GAIA catalog without being too heavy. Disabling procedural stars when increasing mag limit is also a great idea. You can do so by clicking the gear icon on the bottom of the right toolbar and checking off procedural under stars or f4 > filters > tick off procedural under stars.

Current versions:
STEAM: >>Workshop<<
Normal Versions Image Below is a list of normal gaia catalogs:


250k-10mag.pak - Contains all the stars brighter than magnitude 10.>>DOWNLOAD<<


400k-10point5mag.pakContains all the stars brighter than magnitude 10.5.>>DOWNLOAD<<


700k-100L.pak  - Contains all the stars with a luminosity value greater than 100 Sun L.>>DOWNLOAD<<


1M-11mag.pakContains all the stars brighter than magnitude 11.>>DOWNLOAD<<


3M-10L.pakContains all the stars with a luminosity value greater than 10 Sun L.>>DOWNLOAD<<


5M-1L.pakContains all the stars with a luminosity value greater than 1 Sun L.>>DOWNLOAD<<


7M-13mag.pakContains all the stars brighter than magnitude 13.>>DOWNLOAD<<


14M-14mag.pak- Contains all the stars brighter than magnitude 14.>>DOWNLOAD<<


30M-15mag.pakContains all the stars brighter than magnitude 15.>>DOWNLOAD<<


The further down the list it is, the more it will have an impact on performance, loading time and stability. The 1M pak is a good middle ground. Anything beyond will require a significant amount of RAM to run. Use at your own risk.


Future Versions Image

Below is a list of catalogs where the position of stars have been shifted according to observations of proper motion and radial velocity made today. This assumes the stars move at constant velocity in a straight line.

700k-100ky.pak - Simulates where the stars would be 100k years from today. >>DOWNLOAD<< 700k-1My.pak Simulates where the stars would be 1M years from today.>>DOWNLOAD<< 700k-10My.pak Simulates where the stars would be 10M years from today.>>DOWNLOAD<< 700k-100My.pakSimulates where the stars would be 100M years from today.>>DOWNLOAD<<


The further down the list it is, the more the stars will have shifted, and thus the more disorganized everything will be.

OTHER Gaia's White Dwarf catalog by FastFourierTransform containing 73 221 white dwarfs. (Highly suggested)
Contains a highly detailed and accurate catalog of white dwarfs, with their 3D positions relative to Earth.
http://forum.spaceengine.org/viewtopic.php?f=3&t=465

EXTRA PREVIEWS
Video Showcase

[spoiler][youtube]-dsFHuTaALY[/youtube][/spoiler]
Comparison to real life.
[spoiler]Wide view of the milky way with only stars (galaxies disabled).
Image

3M in 0.990, dust becomes much more visible.Image

comparison to irl
Image[/spoiler]
Plan of the Milky way with procedural stars on with Gaia (3819x3717)
[spoiler]Gaia catalog to the right, with the randomized main sequence spectral classes.Image[/spoiler]

Giant Photorealistic Panorama by tRetro-Visor, post processed with photoshop.
[spoiler]Image[/spoiler]

My own version, but with 50 million stars, don't worry I downscaled it. Original can be found here: https://www.deviantart.com/phunnieb/art/Mw50mpano-798308402
[spoiler]Image[/spoiler]


Don't worry if you don't have an insane gaming computer. This barely affects loading time (below 3M) and won't have any perfomance impacts until you start turning magnitude way up. With magnitude set at default (7), it's as if the addon was not present. All the stars are very dim, and require a high magnitude limit to be able to visually see them (the pictures in the previews were taken at magnitudes 14.5 to 16)

If anyone is wondering: Yes those stars are real are present in real life. The planets around the stars, however, are not real. The planets are procedural. Also, the spectral class of the stars are wrong as I could not retrieve any spectral data from the gaia archive.

Side note: Only the stars with the required and useful information were pulled from the search query. All data was fetched from https://gea.esac.esa.int/archive/ and then converted to SE .csv.


Credits go to LucasAstro for making the base converter and saving millions of years of slavery.
Massive thanks to [/color]JackDole for pointing out an error in the conversion from GAIA to SE, as well as being very helpful. Also massive thanks to Watsisname for helping with super secret stuff.


Changelog
V3 - Fixed HIP duplicates, redid all of the versions, increased the range (now up to 30M) and added new "Future" test versions.
V2.91 - Small hotfix for missing clusters in 3M and 4M. 09/12/2018
V2.9 - Changed the algorithm of the way spectral classes are assigned to classes using the HR diagram as a model. Now has more than just the V type class (Main Sequence). Includes Ia (luminous supergiant, Ib(supergiant), II(bright giant), III (giant) and V (main sequence). IV (subgiant) line is ambiguous so I did not add them. Spectral subclass (1-9) are still randomized (does not affect anything). 08/12/2018
V2.6 - Fixed version 800k, 2k and 2M.
V2.5. - "Fixed" the clusters in 800k(missing M 11), 3M, 4M and 5M from pillars of stars to a smaller "cube" of stars.
V2. - Fixed RA, added 4M & 5M.
V1. - Added 2k, 127k, 609k, 800k, 2M, 3M.


Known Issues:
-Spectral type and class areinaccurate (all are main sequence, and the class is randomized.) Fixed according to HR diagram. Subclasses (1-9) cannot be accurately determined so they are still random. I and IV classes are not included due to ambiguity or not present in-game.
-There are some duplicate stars for Hipparcos catalog. Fixed in v3.
-Clusters are pillars of stars Fixed, but still not accurate.
-Wrong RA for all stars Fully fixed.
Last edited by Phunnie on 22 Jul 2020 14:03, edited 78 times in total.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
ettore_bilbo
Explorer
Explorer
Posts: 204
Joined: 11 Nov 2016 05:33
Location: Italy

GAIA to SpaceEngine catalogs

10 Nov 2018 11:53

Great addon, thanks!!!
 
User avatar
JackDole
Star Engineer
Star Engineer
Posts: 1874
Joined: 02 Nov 2016 18:18
Location: Terra

GAIA to SpaceEngine catalogs

10 Nov 2018 12:13

GAIA to SpaceEngine catalogs
Your stars all have a wrong position!
You are using the Gaia specification for 'RA', degree, 0 to 359.xxxxxxxxxxxxxx.
But SpaceEngine uses 'hours' for 'RA', 0 to 23.xxxxxxxx!
That's why all stars have a wrong position!
Also, your naming may lead to confusion with planets when someone uses an addon that numbers the planets. Especially the first 10 or so stars.
You should use the Gaia designations, for example 'Gaia DR2 4047792202996857344'
JackDole's Universe 0.990: http://forum.spaceengine.org/viewtopic.php?f=3&t=546
JackDole's Archive: http://forum.spaceengine.org/viewtopic.php?f=3&t=419
JackDole: Mega structures ... http://old.spaceengine.org/forum/17-3252-1 (Old forum)
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

10 Nov 2018 12:42

GAIA to SpaceEngine catalogs
Your stars all have a wrong position!
You are using the Gaia specification for 'RA', degree, 0 to 359.xxxxxxxxxxxxxx.
But SpaceEngine uses 'hours' for 'RA', 0 to 23.xxxxxxxx!
That's why all stars have a wrong position!
Also, your naming may lead to confusion with planets when someone uses an addon that numbers the planets. Especially the first 10 or so stars.
You should use the Gaia designations, for example 'Gaia DR2 4047792202996857344'
Yes. Thank you for bringing this to my attention. I am currently working on a fix.
EDIT: fix has been found, currently re-converting. Might take a while.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

10 Nov 2018 22:42

Fixed! Updated with new versions too!
4M & 5M. 
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
Hornblower
Pioneer
Pioneer
Posts: 595
Joined: 02 Nov 2016 14:30
Location: Gale Crater
Contact:

GAIA to SpaceEngine catalogs

11 Nov 2018 16:14

There are duplicate stars caused by the mod (idk how you could fix this). But since this adds stars and doesn't replace them, some stars already in space engine also have a duplicate in the gaia catalog. Example: HIP 81852 and [too long of a name to copy over]. This gives a sort of expansion effect. 
[attachment=0]scr00627.jpg[/attachment][/font]
Attachments
scr00627.jpg
"Space is big. Really big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist, but that's just peanuts to space." - Douglas Adams
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

11 Nov 2018 16:29

Fixable. Will take some time.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
HarbingerDawn
SE Team Member
SE Team Member
Posts: 694
Joined: 22 Aug 2016 15:11
Location: CT, USA
Contact:

GAIA to SpaceEngine catalogs

11 Nov 2018 23:09

There are duplicate stars caused by the mod (idk how you could fix this)
Just mod your game to disable the default hipparcos catalog
Ryzen 7 3700X, 64 GB DDR4-3200 RAM, RTX 3090 24 GB VRAM
Posts on old forum: 8717
 
User avatar
Ensabahnur
Space Pilot
Space Pilot
Posts: 110
Joined: 27 Nov 2016 16:11
Location: Romania

GAIA to SpaceEngine catalogs

12 Nov 2018 00:06

Amazing mod Phunnie!
 
User avatar
SpaceEngineer
Author of SpaceEngine
Author of SpaceEngine
Posts: 1125
Joined: 17 May 2016 22:16
Location: Saint-Petersburg
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 04:01

Star octree probably didn't configure itself optimally. For the next update I added support of a custom configuration to the stars catalog, similar to that you can find in the galaxy catalog catalogs/galaxies/LocalGroup.sc:
//////////////////////////////////////////////////////////////////
//////     Octree parameters for HIPPSRCOS.csv catalog      //////
//////////////////////////////////////////////////////////////////

OctreeSize  ( 300 300 300 )
OctreePos   ( 1 1 1 )
OctreeDepth 4
These lines now (in 0.990) should be added to any massive star catalog like these ones. SE will read them from each .sc catalog and choose the maximum numbers (so you should add to every .csv file an "empty" .sc file contains only octree configuration).

Explanation:

SE handles stars using the octree data structure. It is used to cull out stars which are outside the field of view or are too dim to be rendered. Octree splits space by a large cubes, and culls out the whole cubes and all stars that they include (so engine don't need to check every single stars, just the cubes). These cubes are called nodes (more precisely, leafs. Leafs are the last level of octree nodes, and only them contains stars in SE). You may visualize octree by pressing Y in the Debug mode. Large red cubes of different size are nodes, small green cubes are leafs:
scr00479.jpg
To work efficiently, octree must contain all stars, and its size and depth (recursion level) must be so that each leaf contain 1000-10,000 stars. But in real catalogs, stars are distributed very irregularly: 90% of stars are cluttering near the Sun, and other 10% filling up the huge volume around. So star octree in SE have a special outer "leafs" which contain all stars that are not fit into the main large "cube". There are just 24 of them. If you make octree size to small, millions of stars can fall into these special leafs, and culling will be ineffective. If you make octree too large, only small fraction of leafs will have stars: majority of outer leafs will be empty, but central ones will be filled with 100k stars each, which again will make culling ineffective. So adjusting settings is a question of performance balancing.

Default settings for HIPPSRCOS.csv:

OctreeSize  ( 300 300 300 ) - half-dimensions of the octree "central cube" in parsecs (i.e. actual width of the cube is 600 pc). Special outer leafs are stretched automatically to enclose most distant catalog stars (currently they are supermassivle blackholes in some galaxies, so located really far away).

OctreePos   ( 1 1 1 ) - position of the octree center. Zero in the Sun's position, but octree is displaced a bit to make sure that Sun is not in the exact corner of some node (this may cause clipping issues).

OctreeDepth 4 - recursion depth. The main octree cube is split into 8 smaller cubes, each one split again, and so on 4 times. So in this example we have 2^4 = 16 splits of each side; 16^3 = 4096 smallest cubes (leafs), and dimensions of each leaf 600 / 16 = 37.5 pc.

With this settings, HIPPARCOS catalog is split efficiently, it has ~1000 stars in the most central octree leafs. GAIA catalog have much greater star density, and expand to much larger volume, so we need smaller leafs and larger dimensions simultaneously. From experiments with Tycho-2 catalog (~2 millions stars) I obtained that these parameters works well:

OctreeSize  ( 1600 1600 1600 ) 
OctreeDepth 7

OctreePos may be the same - ( 1 1 1 ). With this parameters, we have width of octree leafs of 1600*2 / 2^7 = 25 pc. This leads to roughly the same star density in the central leafs - about 1000 stars per leaf.

Current beta build and v0.980 have hard-coded OctreeSize and OctreeDepth calculation, but it ends at 2 millions stars. For larger dataset, you probably need to increase numbers more than that, this is why I added ability to specify these parameters in the catalog file.
 
User avatar
SpaceEngineer
Author of SpaceEngine
Author of SpaceEngine
Posts: 1125
Joined: 17 May 2016 22:16
Location: Saint-Petersburg
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 04:05

There are duplicate stars caused by the mod (idk how you could fix this). But since this adds stars and doesn't replace them, some stars already in space engine also have a duplicate in the gaia catalog. Example: HIP 81852 and [too long of a name to copy over]. This gives a sort of expansion effect. 
This is a great problem. GAIA catalog, to be included in SE by default, must be cross-matched with existing stars. FastFourierTransform working on this currently, we had a large discussion about future integration of the GAIA dataset. It will require major changes in engine though: star catalog will be split into chunks which will be loaded from disk on-demand, similar to how planet textures works.
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 05:32

Another problem is that star clusters with great uncertainty, like 47 Tucanae are rendered as long columns of star. It looks fine from Earth, but looks extremely strange from anywhere else in the galaxy.
Image
47 Tucanae, taken by Vigo Hornblower.

Also, the gaia archive doesn't give any spectral classes, so the converter took the temperature and made an assumption on the spectral class, so all stars are main sequence.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
SpaceEngineer
Author of SpaceEngine
Author of SpaceEngine
Posts: 1125
Joined: 17 May 2016 22:16
Location: Saint-Petersburg
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 07:30

LOL. Stars with parallax error > 10% should be skipped.
Also, the gaia archive doesn't give any spectral classes, so the converter took the temperature and made an assumption on the spectral class, so all stars are main sequence.
I plan to change how star octree works. It will need only coordinates, luminosity and color to render dots. Star class will not be mandatory like now - it will be computed / generated if missing, and only when this really will be needed (by clicking, generating planetary system, searching in the star browser).
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 13:17

LOL. Stars with parallax error > 10% should be skipped.

Ah, yeah that's a good idea. I'll work on fixing that.
Also, the gaia archive doesn't give any spectral classes, so the converter took the temperature and made an assumption on the spectral class, so all stars are main sequence.
I plan to change how star octree works. It will need only coordinates, luminosity and color to render dots. Star class will not be mandatory like now - it will be computed / generated if missing, and only when this really will be needed (by clicking, generating planetary system, searching in the star browser).
Good to hear! Hopefully then it will look more natural without having to apply filters.
Also, it might be an error on my end, but it doesn't seem like the plane of the Milky Way in SpaceEngine aligns with the plane created by the Gaia catalog stars. Perhaps importing the stars also provided some more accurate data on the inclination of the plane of the Milky Way! :D
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
ChaosRobie
Observer
Observer
Posts: 10
Joined: 14 Oct 2017 23:57

GAIA to SpaceEngine catalogs

12 Nov 2018 14:25

[quote="Phunnie"]Another problem is that star clusters with great uncertainty, like 47 Tucanae are rendered as long columns of star. It looks fine from Earth, but looks extremely strange from anywhere else in the galaxy.

47 Tucanae, taken by Vigo Hornblower.

Also, the gaia archive doesn't give any spectral classes, so the converter took the temperature and made an assumption on the spectral class, so all stars are main sequence.[/quote]
[quote="SpaceEngineer"]LOL. Stars with parallax error > 10% should be skipped.[/quote]
Those unnatural column structures might only be apparent for globular clusters. If you can identify stars that are likely to be in known clusters (cluster RA/Dec/r_t available at http://gclusters.altervista.org/) then you should be able to "correct" their distances and make them globular once more.
  • 1
  • 2
  • 3
  • 4
  • 5
  • 10

Who is online

Users browsing this forum: No registered users and 24 guests