Ultimate space simulation software

 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

09 Dec 2018 01:43

Updated to V2.9. All details in changelog.
Can you publish the new program you used to create the new catalogs? And the parameters necessary for the Gaia query?
I'll need to make some changes to the code so it takes into consideration the new HR diagram based algorithm I've implemented and convert from the basic gaia ADQL query I've been using so far to SE.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
A-L-E-X
Galaxy Architect
Galaxy Architect
Posts: 3498
Joined: 06 Mar 2017 20:19

GAIA to SpaceEngine catalogs

09 Dec 2018 01:44

You could, it might take a while to load because SE will have to delete a substantial amount of duplicate stars. I'll keep that idea in mind and I might make other paks in the future when I'll have time. I do really want to make a pak with giants and above only, as well as some based on apparent magnitude and perhaps even cross sections of the milky way to hopefully get some sort of structure.
Thats understandable however on the Gaia wiki page I saw that they were also talking about discovering thousands of exoplanets and it would be useful to have sunlike stars and even red dwarfs included because planets discovered around those stars could be habitable.  They also mentioned quasars, and it would be nice to have a full catalog of quasars in SE.
I agree, having quasars would be nice. However, I have no idea what to query in the ADQL search to get the data for galaxies. Once I figure that out, I'll add galaxy paks, or include them in each version.
I wonder how it will be once Gaia starts including exoplanets.  Right now we have to add systems one at a time, it would be great if there was a way for the program to update its list of exoplanets automatically and give each a default landscape (but also randomized on smaller scales for some variety) based on type.
 
A-L-E-X
Galaxy Architect
Galaxy Architect
Posts: 3498
Joined: 06 Mar 2017 20:19

GAIA to SpaceEngine catalogs

09 Dec 2018 22:32

Just wanted to thank you Phunnie, the 59 spectral class errors in the 5M file are now gone!  I tried to run the 2M and 5M files together but SE would not load.  With the 5M file, SE takes about 1 min to load and uses up about 2.3 GB of RAM.  It also uses 50% of my GPU while the vanilla SE used about 30%.

The only errors left now are those duplicate errors, which there was 1 from the intergalactic stars pak, 4 from the white dwarfs pak and 3 from the 5M pak.  You said the program self-corrects those though so those shouldn't be an issue?
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

12 Dec 2018 16:51

Well I got super side-tracked and ended up deciding to cut a flat cross section of the plane of the milky way to maybe get a glimpse of where the nearest dust "clouds" are.
Image
and well while doing that I realized that the milky way is actually crooked
Image
so I corrected it:
Image
Note: It's still not 100% perfect.

Fix attached below:

I also managed to pull a 28M stars query lol.
Image

and converted it to a SE .csv catalog that's 2.3GB in size, then took a 20pc thick cross section of those 28M stars on the plane of the MW giving some interesting results:
Image
750k stars in that tiny cross section from a abridged (28M) catalog, Imagine the entire catalog (over 1B). Ouch.
Image
Attachments
LocalGroup.sc
(12.63 KiB) Downloaded 616 times
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
A-L-E-X
Galaxy Architect
Galaxy Architect
Posts: 3498
Joined: 06 Mar 2017 20:19

GAIA to SpaceEngine catalogs

13 Dec 2018 02:33

Thanks Phunnie, do I need to replace the files I downloaded lol (800k, 2M and 5M) or are those the same as before?  I downloaded them on the same day I made that post thanking you- 12/8 or 12/9 I think.

I see this in the changelog- also there is no 28M star file.  You can go higher with stars than 5M, Starry Night has 17M stars and runs fine as long as all the stars aren't being viewed all at once lol.  What is that Localgroup.sc file supposed to do?  Since I am using the 5M file from 12/8 (2.9) do I need to use that too or are all those stars already included in the 5M pak?

Changelog
V2.91 - Small hotfix for missing clusters in 3M and 4M. 09/12/2018
V2.9 - Changed the algorithm of the way spectral classes are assigned to classes using the HR diagram as a model. Now has more than just the V type class (Main Sequence). Includes Ia (luminous supergiant, Ib(supergiant), II(bright giant), III (giant) and V (main sequence). IV (subgiant) line is ambiguous so I did not add them. Spectral subclass (1-9) are still randomized (does not affect anything). 08/12/2018
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

13 Dec 2018 11:02

Thanks Phunnie, do I need to replace the files I downloaded lol (800k, 2M and 5M) or are those the same as before?  I downloaded them on the same day I made that post thanking you- 12/8 or 12/9 I think.

I see this in the changelog- also there is no 28M star file.  You can go higher with stars than 5M, Starry Night has 17M stars and runs fine as long as all the stars aren't being viewed all at once lol.  What is that Localgroup.sc file supposed to do?  Since I am using the 5M file from 12/8 (2.9) do I need to use that too or are all those stars already included in the 5M pak?

Changelog
V2.91 - Small hotfix for missing clusters in 3M and 4M. 09/12/2018
V2.9 - Changed the algorithm of the way spectral classes are assigned to classes using the HR diagram as a model. Now has more than just the V type class (Main Sequence). Includes Ia (luminous supergiant, Ib(supergiant), II(bright giant), III (giant) and V (main sequence). IV (subgiant) line is ambiguous so I did not add them. Spectral subclass (1-9) are still randomized (does not affect anything). 08/12/2018
No i haven't made any changes. Also really, you can go over 5M? I've always crashed upon startup past 5M. Not just me either, I ran test on other systems and it seemed SE just couldn't support more than 5M stars in one pak. Also localgroup file is a fix for galaxies. The file should go in addons/catalogs/galaxies or data/catalogs/galaxies.
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
A-L-E-X
Galaxy Architect
Galaxy Architect
Posts: 3498
Joined: 06 Mar 2017 20:19

GAIA to SpaceEngine catalogs

13 Dec 2018 17:39

Thanks Phunnie, do I need to replace the files I downloaded lol (800k, 2M and 5M) or are those the same as before?  I downloaded them on the same day I made that post thanking you- 12/8 or 12/9 I think.

I see this in the changelog- also there is no 28M star file.  You can go higher with stars than 5M, Starry Night has 17M stars and runs fine as long as all the stars aren't being viewed all at once lol.  What is that Localgroup.sc file supposed to do?  Since I am using the 5M file from 12/8 (2.9) do I need to use that too or are all those stars already included in the 5M pak?

Changelog
V2.91 - Small hotfix for missing clusters in 3M and 4M. 09/12/2018
V2.9 - Changed the algorithm of the way spectral classes are assigned to classes using the HR diagram as a model. Now has more than just the V type class (Main Sequence). Includes Ia (luminous supergiant, Ib(supergiant), II(bright giant), III (giant) and V (main sequence). IV (subgiant) line is ambiguous so I did not add them. Spectral subclass (1-9) are still randomized (does not affect anything). 08/12/2018
No i haven't made any changes. Also really, you can go over 5M? I've always crashed upon startup past 5M. Not just me either, I ran test on other systems and it seemed SE just couldn't support more than 5M stars in one pak. Also localgroup file is a fix for galaxies. The file should go in addons/catalogs/galaxies or data/catalogs/galaxies.
Thanks I will put in the galaxies folder, does it include stars or just images for galaxies?  Does it alter the numbering scheme for procedural stars in galaxies as far as you know? (I thought all galaxy add ons did that.)
I downloaded the three different ones to test them and I installed the 2M+5M pak for awhile and it did load the first time but every subsequent time I loaded it, it would hang on start up so I just use the 5M pak now. 
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

20 May 2019 18:46

Still haven't figured out how to remove duplicates. However, with the most recent 64-bit update, I have sucessfully ran a 49.3 million stars pak, albeit with a lot of suffering. I also got a 10M, 30M and 78M (the entire catalog of stars with radius values) that I will upload as addons sometime in the future. I also managed to pull out the LMC and SMC, but the radius of the stars have been randomized (because they are missing) and the distance set to an arbitrary number somewhat close to what it should be.
► Show Spoiler
bonus downscaled milky way pano with 49M
► Show Spoiler
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie
 
User avatar
SpaceEngineer
Author of SpaceEngine
Author of SpaceEngine
Posts: 1125
Joined: 17 May 2016 22:16
Location: Saint-Petersburg
Contact:

GAIA to SpaceEngine catalogs

21 May 2019 04:32

You shouldn't randomize radius if data is missing. Just leave the field blank, SE will calculate radius itself.
 
LucasAstro
Space Tourist
Space Tourist
Posts: 32
Joined: 15 Sep 2017 04:32

GAIA to SpaceEngine catalogs

27 Jul 2019 22:58

That's pretty neat
 
User avatar
Khorrah
Explorer
Explorer
Posts: 161
Joined: 11 Nov 2019 17:50
Location: Earth

GAIA to SpaceEngine catalogs

03 Dec 2019 06:51

Still haven't figured out how to remove duplicates. However, with the most recent 64-bit update, I have sucessfully ran a 49.3 million stars pak, albeit with a lot of suffering. I also got a 10M, 30M and 78M (the entire catalog of stars with radius values) that I will upload as addons sometime in the future. I also managed to pull out the LMC and SMC, but the radius of the stars have been randomized (because they are missing) and the distance set to an arbitrary number somewhat close to what it should be.
► Show Spoiler
bonus downscaled milky way pano with 49M
► Show Spoiler
Any update?, I believe that this one it's for 0.9.9.0 now, or still can be for 0.9.8.0? This may make my pc explode lol
Hello, I LOVE lasagna and Space Engine. ♥

My Specs: Intel Core I5 12600K, Corsair Vengeance Pro SL 32GB, MSI Z690-A WIFI DDR4, GIGABYTE NVIDIA GEFORCE RTX 3070 AORUS MASTER 8G.
 
A-L-E-X
Galaxy Architect
Galaxy Architect
Posts: 3498
Joined: 06 Mar 2017 20:19

GAIA to SpaceEngine catalogs

03 Dec 2019 08:19

I have his biggest catalog installed for .990 and it actually works much better there no lag.  I have the 5M catalog installed, dont know what this 49M thing is lol.  
 
A-L-E-X
Galaxy Architect
Galaxy Architect
Posts: 3498
Joined: 06 Mar 2017 20:19

GAIA to SpaceEngine catalogs

03 Dec 2019 08:22

Still haven't figured out how to remove duplicates. However, with the most recent 64-bit update, I have sucessfully ran a 49.3 million stars pak, albeit with a lot of suffering. I also got a 10M, 30M and 78M (the entire catalog of stars with radius values) that I will upload as addons sometime in the future. I also managed to pull out the LMC and SMC, but the radius of the stars have been randomized (because they are missing) and the distance set to an arbitrary number somewhat close to what it should be.
► Show Spoiler
bonus downscaled milky way pano with 49M
► Show Spoiler
Nothing new to install since last Dec right?  I am fine so far with the v2.9 5M star file you uploaded back then lol.
 
KnarzKnorp
Observer
Observer
Posts: 8
Joined: 16 Jan 2020 05:27

GAIA to SpaceEngine catalogs

16 Jan 2020 05:38

The 5M version works fine on my PC. Incredible amount of stars. Thank you!
 
User avatar
Phunnie
Space Pilot
Space Pilot
Topic Author
Posts: 137
Joined: 02 Aug 2017 22:39
Location: Canada
Contact:

GAIA to SpaceEngine catalogs

20 Jul 2020 12:59

So it's been a while since any work has been done on this, but I would like to announce that duplicate stars are no more!

Before:
The duped stars are clearly visible where it looks like there are rays of stars pointing at earth
Image

After:
Image

Now this doesn't 100% fix all duped stars (more like 99.999% of them) as I only used HIP cross-referenced to GAIA.

I will update it soonish after I redo everything and add a couple extra even more extreme versions.

GIF:
https://gfycat.com/disastrousaltruisticballoonfish
Oh. I have a youtube channel. https://www.youtube.com/c/Phunnie

Who is online

Users browsing this forum: Ahrefs [Bot] and 16 guests