Ultimate space simulation software

  • 1
  • 2
  • 3
  • 4
  • 5
  • 7
 
User avatar
Phunnie
Astronaut
Astronaut
Topic Author
Posts: 78
Joined: 02 Aug 2017
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 13:47

ChaosRobie wrote:
Phunnie wrote:
Another problem is that star clusters with great uncertainty, like 47 Tucanae are rendered as long columns of star. It looks fine from Earth, but looks extremely strange from anywhere else in the galaxy.

47 Tucanae, taken by Vigo Hornblower.

Also, the gaia archive doesn't give any spectral classes, so the converter took the temperature and made an assumption on the spectral class, so all stars are main sequence.

SpaceEngineer wrote:
LOL. Stars with parallax error > 10% should be skipped.

Those unnatural column structures might only be apparent for globular clusters. If you can identify stars that are likely to be in known clusters (cluster RA/Dec/r_t available at http://gclusters.altervista.org/) then you should be able to "correct" their distances and make them globular once more.

They also appear for open clusters too unfortunately. NGC 869 is a perfect example. Two open clusters, two pillars of stars.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
PlutonianEmpire
Pioneer
Pioneer
Posts: 429
Joined: 02 Nov 2016
Location: MinneSNOWta
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 15:02

I think the duplicate stars happened to the White Dwarf .csv too, didn't it?

Also, when choosing a download that says "At least x Luminosity", does that mean stars below that luminosity will not be shown?
Specs: Dell Inspiron 5547 (Laptop); 8 gigabytes of RAM; Processor: Intel® Core™ i5-4210U CPU @ 1.70GHz (4 CPUs), ~2.4GHz; Operating System: Windows 7 Home Premium 64-bit; Graphics: Intel® HD Graphics 4400 (That's all there is :( )
 
User avatar
Phunnie
Astronaut
Astronaut
Topic Author
Posts: 78
Joined: 02 Aug 2017
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 15:20

PlutonianEmpire wrote:
I think the duplicate stars happened to the White Dwarf .csv too, didn't it?

Also, when choosing a download that says "At least x Luminosity", does that mean stars below that luminosity will not be shown?

Precisely. This is one of the way I used to limit the amount of stars put in a catalog. I'm thinking about making one with radius as a value, and take all stars above X sol radius to get giant stars, or below X sol radius to get white dwarfs perhaps. I'm sure there is a better way of searching for those, like using the HR diagram, but right now I'm just fixing things.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
Phunnie
Astronaut
Astronaut
Topic Author
Posts: 78
Joined: 02 Aug 2017
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

12 Nov 2018 20:53

SpaceEngineer wrote:
Star octree probably didn't configure itself optimally. For the next update I added support of a custom configuration to the stars catalog, similar to that you can find in the galaxy catalog catalogs/galaxies/LocalGroup.sc:

//////////////////////////////////////////////////////////////////
//////     Octree parameters for HIPPSRCOS.csv catalog      //////
//////////////////////////////////////////////////////////////////

OctreeSize  ( 300 300 300 )
OctreePos   ( 1 1 1 )
OctreeDepth 4


These lines now (in 0.990) should be added to any massive star catalog like these ones. SE will read them from each .sc catalog and choose the maximum numbers (so you should add to every .csv file an "empty" .sc file contains only octree configuration).

Explanation:

SE handles stars using the octree data structure. It is used to cull out stars which are outside the field of view or are too dim to be rendered. Octree splits space by a large cubes, and culls out the whole cubes and all stars that they include (so engine don't need to check every single stars, just the cubes). These cubes are called nodes (more precisely, leafs. Leafs are the last level of octree nodes, and only them contains stars in SE). You may visualize octree by pressing Y in the Debug mode. Large red cubes of different size are nodes, small green cubes are leafs:

scr00479.jpg

To work efficiently, octree must contain all stars, and its size and depth (recursion level) must be so that each leaf contain 1000-10,000 stars. But in real catalogs, stars are distributed very irregularly: 90% of stars are cluttering near the Sun, and other 10% filling up the huge volume around. So star octree in SE have a special outer "leafs" which contain all stars that are not fit into the main large "cube". There are just 24 of them. If you make octree size to small, millions of stars can fall into these special leafs, and culling will be ineffective. If you make octree too large, only small fraction of leafs will have stars: majority of outer leafs will be empty, but central ones will be filled with 100k stars each, which again will make culling ineffective. So adjusting settings is a question of performance balancing.

Default settings for HIPPSRCOS.csv:

OctreeSize  ( 300 300 300 ) - half-dimensions of the octree "central cube" in parsecs (i.e. actual width of the cube is 600 pc). Special outer leafs are stretched automatically to enclose most distant catalog stars (currently they are supermassivle blackholes in some galaxies, so located really far away).

OctreePos   ( 1 1 1 ) - position of the octree center. Zero in the Sun's position, but octree is displaced a bit to make sure that Sun is not in the exact corner of some node (this may cause clipping issues).

OctreeDepth 4 - recursion depth. The main octree cube is split into 8 smaller cubes, each one split again, and so on 4 times. So in this example we have 2^4 = 16 splits of each side; 16^3 = 4096 smallest cubes (leafs), and dimensions of each leaf 600 / 16 = 37.5 pc.

With this settings, HIPPARCOS catalog is split efficiently, it has ~1000 stars in the most central octree leafs. GAIA catalog have much greater star density, and expand to much larger volume, so we need smaller leafs and larger dimensions simultaneously. From experiments with Tycho-2 catalog (~2 millions stars) I obtained that these parameters works well:

OctreeSize  ( 1600 1600 1600 ) 
OctreeDepth 7

OctreePos may be the same - ( 1 1 1 ). With this parameters, we have width of octree leafs of 1600*2 / 2^7 = 25 pc. This leads to roughly the same star density in the central leafs - about 1000 stars per leaf.

Current beta build and v0.980 have hard-coded OctreeSize and OctreeDepth calculation, but it ends at 2 millions stars. For larger dataset, you probably need to increase numbers more than that, this is why I added ability to specify these parameters in the catalog file.

I tried to do some testing on my own with the octree settings, but it doesn't seem like I can edit the octree at all. changing the dimension of the main cube doesn't seem to be changing anything when changing the octree from a .sc file next to the catalog csv. Regardless, I do have a question since I cannot edit the octree: can the octree be rectangular?

The Gaia catalog is quite large in diameter (over 25kly) and so past the first 1000ly, the shape already stars becoming a thin flattened cylinder. At 5000pc, the sphere shape is no longer even visible and it's basically a cylindrical plane with a little bit of thickness and scattered stars above and under the plane. My concern is that wouldn't it be a massive waste of leaves and also extremely inefficient placing a cube for the Gaia catalog? 90% of the stars are on the plane of the milky way (with about 60% of that 90% being in the direct viscinity of Earth, depending on the version and min luminosity values), and the rest is scatered under and above the plane. so all the cubes are centered in the center of the cube and across, with barely nothing above and under.

zoomed out view with the square of the default octree cube visible
Image
clear example of a waste of leaves in the octree cube
Image
Image
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
SpaceEngineer
Author of SpaceEngine
Author of SpaceEngine
Posts: 800
Joined: 17 May 2016
Location: Saint-Petersburg
Contact:

GAIA to SpaceEngine catalogs

13 Nov 2018 02:23

Phunnie wrote:
Source of the post I tried to do some testing on my own with the octree settings, but it doesn't seem like I can edit the octree at all.

Lol I Phunnie, i wrote
Phunnie wrote:
Source of the post  For the next update

Wait for the update on steam!
Phunnie wrote:
Source of the post Regardless, I do have a question since I cannot edit the octree: can the octree be rectangular?

No, and it cannot be rotated to match the galactic plane. Anyway, I will change the octree completely: make caalog stars using the same octree as procedural do.
This also will be an elegant way to seamlessly merge catalog and procedural generation. First, I have to derive some statistics from GAIA data: how many stars each level of nodes have (procedural stars octree has a different structure, stars are stored in all nodes, not only in leafs). When procedural generator will take a node, look how many catalog stars already exist in this node, and generate as many procedural stars as needed to match statistics.
 
User avatar
PlutonianEmpire
Pioneer
Pioneer
Posts: 429
Joined: 02 Nov 2016
Location: MinneSNOWta
Contact:

GAIA to SpaceEngine catalogs

13 Nov 2018 02:35

SpaceEngineer wrote:
Phunnie wrote:
Source of the post I tried to do some testing on my own with the octree settings, but it doesn't seem like I can edit the octree at all.

Lol I Phunnie, i wrote
Phunnie wrote:
Source of the post  For the next update

Wait for the update on steam!
Phunnie wrote:
Source of the post Regardless, I do have a question since I cannot edit the octree: can the octree be rectangular?

No, and it cannot be rotated to match the galactic plane. Anyway, I will change the octree completely: make caalog stars using the same octree as procedural do.
This also will be an elegant way to seamlessly merge catalog and procedural generation. First, I have to derive some statistics from GAIA data: how many stars each level of nodes have (procedural stars octree has a different structure, stars are stored in all nodes, not only in leafs). When procedural generator will take a node, look how many catalog stars already exist in this node, and generate as many procedural stars as needed to match statistics.

Will this result in less dense procedural neighborhoods overall? I've heard that the Sol system's neighborhood may be less dense than the galactic average, so I don't know how much that might affect this new method
Specs: Dell Inspiron 5547 (Laptop); 8 gigabytes of RAM; Processor: Intel® Core™ i5-4210U CPU @ 1.70GHz (4 CPUs), ~2.4GHz; Operating System: Windows 7 Home Premium 64-bit; Graphics: Intel® HD Graphics 4400 (That's all there is :( )
 
User avatar
midtskogen
World Builder
World Builder
Posts: 700
Joined: 11 Dec 2016
Location: Oslo, Norway
Contact:

GAIA to SpaceEngine catalogs

13 Nov 2018 04:05

Phunnie wrote:
Source of the post Another problem is that star clusters with great uncertainty, like 47 Tucanae are rendered as long columns of star.

It should be possible to improve the catalogue, if the stars belonging to the cluster can be identified.  The distance to the cluster can be taken as the average of its stars, and all the stars can be forced inside a sphere matching the diameter as seen from Earth.  This gives a cluster with pretty randomly positioned stars within the cluster, still each star will have a much smaller parallax error.

Identification of clusters and its stars may not be trivial to automate, though.

I wonder if the distance to stars with large parallax uncertainty can be improved by making assumptions about its distance based on it's spectrum and luminosity.
NIL DIFFICILE VOLENTI
 
User avatar
Phunnie
Astronaut
Astronaut
Topic Author
Posts: 78
Joined: 02 Aug 2017
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

13 Nov 2018 09:53

midtskogen wrote:
Phunnie wrote:
Source of the post Another problem is that star clusters with great uncertainty, like 47 Tucanae are rendered as long columns of star.

It should be possible to improve the catalogue, if the stars belonging to the cluster can be identified.  The distance to the cluster can be taken as the average of its stars, and all the stars can be forced inside a sphere matching the diameter as seen from Earth.  This gives a cluster with pretty randomly positioned stars within the cluster, still each star will have a much smaller parallax error.

Identification of clusters and its stars may not be trivial to automate, though.

I wonder if the distance to stars with large parallax uncertainty can be improved by making assumptions about its distance based on it's spectrum and luminosity.


Doing an average of the distance is bad, as the uncertainty is much greater for far away stars than for closer stars (streches out very far). Automating this will most likely not be possible as I do not have much free time to spare. I've talked to Watsisname about possible options, and the most accurate ones involve looking for proper motion and radial velocity of stars(given by gaia) and comparing to surrounding stars(or to an average of the surrounding stars). That's very complicated, so for now I've opted to manually determine a cluster by using the database in Simbad (find RA, DEC, and apparent size) and let it automatically set all the distances of all the stars present within that radius to a manually inputted one. Of course this is terribly inaccurate, but it's still more accurate than a pillar of stars. Also, to reduce inacurracies, I have made a file with all stars with parallax error over 15% to try to only keep stars in clusters and change their distances.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
midtskogen
World Builder
World Builder
Posts: 700
Joined: 11 Dec 2016
Location: Oslo, Norway
Contact:

GAIA to SpaceEngine catalogs

13 Nov 2018 10:40

Phunnie wrote:
Source of the post Doing an average of the distance is bad, as the uncertainty is much greater for far away stars than for closer stars (streches out very far).

I don't understand.  The stretching is not real.  In reality the diameter of the cluster is quite small compared to the distance between our solar system and the centre of the cluster.  So the closest stars in GAIA are hardly more accurate than the furthest.  Rather, the closest are likely among the least accurate (as the furthest), assuming that the measuring error of all stars in the cluster level out.
NIL DIFFICILE VOLENTI
 
User avatar
N0B0DY
Explorer
Explorer
Posts: 170
Joined: 09 Dec 2016

GAIA to SpaceEngine catalogs

13 Nov 2018 13:00

HarbingerDawn wrote:
Hornblower wrote:
Source of the post There are duplicate stars caused by the mod (idk how you could fix this)

Just mod your game to disable the default hipparcos catalog

Aand how is it possible to do this (without touching the data folder) if I may ask? I noticed that SE (0.980) is trying to merge duplicates but displays a few impossible merges in the console. I'd prefer a console command or an exception command in a script or config file if possible.
This is a f-a-n-t-a-s-t-i-c add-on btw. Have some virtual rep Mr Phunnie.
 
User avatar
HarbingerDawn
SE Team Member
SE Team Member
Posts: 503
Joined: 22 Aug 2016
Location: CT, USA
Contact:

GAIA to SpaceEngine catalogs

13 Nov 2018 13:33

N0B0DY wrote:
Source of the post Aand how is it possible to do this (without touching the data folder)

It isn't. Just open the catalogs .pak file, add ".off" or similar to the end of the hipparcos.csv file, and you're done. To reset it, just remove that added extension.
Ryzen 7 1700 @ 3.8 GHz, 32 GB DDR4-3200 RAM, GTX 1080 Ti 11 GB VRAM
Posts on old forum: 8717
 
User avatar
ettore_bilbo
Space Pilot
Space Pilot
Posts: 144
Joined: 11 Nov 2016
Location: Italy

GAIA to SpaceEngine catalogs

13 Nov 2018 13:52

I was thinking: Putting togheter this mod, the new features of 0.990 version and some spaceship mod,  we could start thinking about some collaborative gameplay to pass the time (waiting the real game) and colonize the SE space, or at least the first thousand light years around the earth :-D
 
User avatar
SpaceEngineer
Author of SpaceEngine
Author of SpaceEngine
Posts: 800
Joined: 17 May 2016
Location: Saint-Petersburg
Contact:

GAIA to SpaceEngine catalogs

13 Nov 2018 14:43

To disable default star catalog, you may try to make an empty file on the same virtual path as the HIPPARCOS.csv, for example:

addons/catalogs/stars/HIPPARCOS.csv
 
User avatar
JackDole
World Builder
World Builder
Posts: 1142
Joined: 02 Nov 2016
Location: Terra

GAIA to SpaceEngine catalogs

13 Nov 2018 22:09

N0B0DY wrote:
Source of the post Aand how is it possible to do this (without touching the data folder) if I may ask?

SpaceEngineer wrote:
Source of the post To disable default star catalog, you may try to make an empty file on the same virtual path as the HIPPARCOS.csv, for example:

addons/catalogs/stars/HIPPARCOS.csv

It works in any case with this file:
Name, RA, Dec, Dist, AppMagn, specClass, MassSol, RadSol, Temperature

HIPPARCOS.csv
(63 Bytes) Downloaded 27 times

Put it in 'data\catalogs\stars'.
JackDole's Universe: http://forum.spaceengine.org/viewtopic.php?f=3&t=71
JackDole's Archive: http://forum.spaceengine.org/viewtopic.php?f=3&t=419
JackDole: Mega structures ... http://old.spaceengine.org/forum/17-3252-1 (Old forum)
 
User avatar
N0B0DY
Explorer
Explorer
Posts: 170
Joined: 09 Dec 2016

GAIA to SpaceEngine catalogs

14 Nov 2018 01:05

There are issues if the Hiparcos.cvs is disabled. I was orbiting Proxima b and the whole proxima system has disappeared. Space Engine is doing an awesome job in merging the dublicates so better leave it there..
I suspect (but I may be wrong) that in its final form, the GAIA catalog will complement the Hipparcos catalog. But to achieve this a lot of work must be done to filter out the dublicates.
  • 1
  • 2
  • 3
  • 4
  • 5
  • 7

Who is online

Users browsing this forum: No registered users and 1 guest