Ultimate space simulation software

 
User avatar
Mosfet
Star Engineer
Star Engineer
Posts: 1770
Joined: 24 Oct 2016 11:34
Location: Italy
Contact:

The Gaia Mission

25 Apr 2018 12:08

After an exciting presentation of the release 2, whoever has a VR headset probably would like to try this application, updated just today:

Gaia Sky 2.0.0
https://zah.uni-heidelberg.de/institutes/ari/gaia/outreach/gaiasky/

Gaia Sky VR
https://github.com/langurmonkey/gaiasky/blob/vr/README.md

Small color map of measured stars - Release 2:
Image

http://sci.esa.int/gaia/60198-gaia-hertzsprung-russell-diagram/
Image
"Time is illusion. Lunchtime doubly so". Douglas N. Adams
| My mods: http://forum.spaceengine.org/viewtopic.php?f=3&t=80 | My specs: Asus x555ub - cpu i5-6200u, ram 12gb, gpu nvidia geforce 940m 2gb vram |
 
User avatar
FastFourierTransform
Pioneer
Pioneer
Topic Author
Posts: 356
Joined: 17 Nov 2016 15:09

The Gaia Mission

25 Apr 2018 18:14

Image

Today we had a very busy day here at the European Space Agency (ESAC center) in Madrid, as you can imagine. It has been a mind-bending experience for me. I'm very tired right now and tomorrow we would have to work much more, so don't expect any news from my side right now. Also, we have been probably attacked so the servers of the Gaia Archive have shut down for some time.

You would have to wait for the interesting results, but for now, in the few hours that we have been able to work with the data, some incredible things have pop-up that are evident on sight;

White Dwarfs: there is direct confirmation that they are presented in 3 different types. Those are clearly visible in the HR diagrams that we saw today. So yeah white dwarfs are not just a kind of object with small differences but they have 3 entire categories that we couldn't separate previously.

The orbits of satellite galaxies of the Milky Way (and halo globular clusters) and even their rotation are now known (some papers are coming)

Someone has come up with a U-V diagram (U and V are two components of motion in the galactic frame) of the entire dataset. We now see clumps in the diagram (impossible to see with Hipparcos or DR1) that have to be interpreted as stellar populations with different motions in the disc, we can now separate them in terms of motion and understand their history with minute precision (even if their spatial distribution looks homogeneous and there's no clear link between them). Fascinating discovery of today!!: There is a layered structure that was previously unknown (someone suspected it should be there) in the U-V diagram. It has been interpreted by some of the guys in the room as a stellar density wave in the galactic disc produced by the collision with an ancient galaxy.

There are many news guys. We have opened a window to the universe today. Our knowledge is going to expand incredibly. I'm so excited. And even If I can't write too much, since work has to continue tomorrow, and the all the rigorous research it's starting now, I can tell you that this is an historic moment. The scientific community will finish digesting this information in a century. Many many things are deep inside these nearly 2 billion row tables.

Just watch this diagram:
Image
You can see how there has been an historic trend for stellar catalogs to be more and more precise and have more and more stars in it. Since the y-axis is in logarithmic scale this trend means that we are getting exponentially better and better with precision for parallaxes. But watch where is Gaia. It is totally out of trend. We should expect Gaia in a few centuries not now. But here it is. Not only is better than the previous catalogs (as expected), not only is way better than the previous catalogs (as the historic exponential growth in resolution tells us) but it is so good that has broken the trend. This is truly magnificent.

Think about the fact that the Tycho catalogue was precise enough to allow for the first determination of an elliptical orbit (that of Mars) and with it we had the first experimental milestone of the Copernican revolution and the possibility to formulate Kepler's laws. What should we expect from 1.7 billion stars? This catalogue is like opening eyes for the first time to the history and geography of our own galaxy.

News are ahead ;)
 
User avatar
Hornblower
Pioneer
Pioneer
Posts: 595
Joined: 02 Nov 2016 14:30
Location: Gale Crater
Contact:

The Gaia Mission

25 Apr 2018 19:36

it would be sort of interesting to know what those outlier stars are like. Particularly the ones between main sequence and white dwarfs
"Space is big. Really big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist, but that's just peanuts to space." - Douglas Adams
 
User avatar
midtskogen
Star Engineer
Star Engineer
Posts: 1537
Joined: 11 Dec 2016 12:57
Location: Oslo, Norway
Contact:

The Gaia Mission

25 Apr 2018 23:54

Could Gaia shed light on the dark matter/energy questions?  I mean, it's also a mapping of the gravity of the galaxy, besides a mapping of the visible stars of the galaxy, since it has much data on the movement of stars.
NIL DIFFICILE VOLENTI
 
User avatar
Watsisname
Science Officer
Science Officer
Posts: 2318
Joined: 06 Sep 2016 02:33
Location: Bellingham, WA

The Gaia Mission

26 Apr 2018 01:30

It could definitely help with modelling the distribution of dark matter in the galaxy.  Probably not so much for dark energy, which is more relevant on a broader cosmological scale.
 
User avatar
midtskogen
Star Engineer
Star Engineer
Posts: 1537
Joined: 11 Dec 2016 12:57
Location: Oslo, Norway
Contact:

The Gaia Mission

26 Apr 2018 06:22

Is this database something SE could handle, on a computer with sufficient memory?  My home server has 64 GB, which might be tight, I guess.  But perhaps with a fast SSD and a clever database format with different detail layers it could work.
NIL DIFFICILE VOLENTI
 
User avatar
FastFourierTransform
Pioneer
Pioneer
Topic Author
Posts: 356
Joined: 17 Nov 2016 15:09

The Gaia Mission

26 Apr 2018 07:56

Could Gaia shed light on the dark matter/energy questions?  I mean, it's also a mapping of the gravity of the galaxy, besides a mapping of the visible stars of the galaxy, since it has much data on the movement of stars.
Yes, indeed. For dark matter an ESA press release said:
By watching for the large-scale motion of stars in our Galaxy, it will also probe the distribution of dark matter, the invisible substance thought to hold our Galaxy together.
Consider also that not only we are now capable of mapping the fine structure of dark matter in out galaxy but also in broad terms thanks to the accurate motion of globular clusters in the halo.
For the dark energy I don't have an answer. But considering the fact that DR2 includes a new catalog of 500.000 quasar detections (some are going to lensed sources which means we are probing vast distances) maybe someone can come with some ideas to further constrain cosmological models.
Another interesting topic in fundamental physics that is going to emerge from DR2 is gravitational wave detection (just by observing the changes in the path of light using the exquisite astrometry of the spacecraft).

Is this database something SE could handle, on a computer with sufficient memory?  My home server has 64 GB, which might be tight, I guess.  But perhaps with a fast SSD and a clever database format with different detail layers it could work.
I don't know what sizes could SE actually reach but I can tell you something about the amount of data. Currently the DR2 main catalogue occupies 587 GB of memory (All columns for the 1.692.919.135 stars). But it is true that not all the information would be necessary for SE. I've just done a reduction of the data such that we are now only considering stars that have parallax errors of around 5% (very very nice distances for SE) and those are 148.476.074 (the 8,8% of the entire DR2). This means that we are dealing with 51,5 GB or so of data. But is also true that we don't need to know all the 96 columns for each of those stars. It is sufficient with 5 parameters for SE (Source ID, ra, dec, parallax and G magnitude) even if DR2 has Temperature and Radius for the stars also (using some models) which could be useful for SE. But yeah, taking only DR2 stars with good quality parallaxes and taking only those 5 parameters for each one we are going to deal with around 2.7 GB of data. Any computer can handle this in memory but accessing the information and making it accessible from SE in terms of milliseconds is a feat that seems impossible to me. But who knows I've saw many things in SE that I though were completely impossible for one developer. The ideal would be to have some sort of SQL capability incorporated in SE for something like this.

In the next Gaia release, the DR3 there are going to be flags marking which are binary resolved systems and maybe even some orbital parameters (something interesting for SE). The parallax measurements are going to be better since the time-baseline increases and with it the number of observations for each star, which means that we are probably having many more stars with 5% accurate parallaxes. Also is very probable that DR3 might reach the 2 billion star milestone (expanding a "little" the 1.7 billion in DR2). So this will keep growing. DR3 is expected for 2020.
 
User avatar
midtskogen
Star Engineer
Star Engineer
Posts: 1537
Joined: 11 Dec 2016 12:57
Location: Oslo, Norway
Contact:

The Gaia Mission

26 Apr 2018 08:03

Pretty cool.  Considering that the database is 1.4 TB (though likely more verbose than strictly needed), I wonder what the size of the total observations that lead to the 1.4 TB summary might have been.
NIL DIFFICILE VOLENTI
 
User avatar
FastFourierTransform
Pioneer
Pioneer
Topic Author
Posts: 356
Joined: 17 Nov 2016 15:09

The Gaia Mission

26 Apr 2018 08:32

1.4 TB
No, no. It's 587 GB. I'm sorry I was working with another thing and confused the numbers XD
The post is now corrected. Sorry for that
 
User avatar
FastFourierTransform
Pioneer
Pioneer
Topic Author
Posts: 356
Joined: 17 Nov 2016 15:09

The Gaia Mission

10 May 2018 05:48

I was going to write about some Gaia DR2 news but since I'm still quite busy and since today PBS SpaceTime released a very good account for all that is happening right now I will just add the video here :D

I will explain some of these and other very interesting things that are currently going on with DR2 data analysis in the future. But for now, this is a majestic explanation.

[youtube]Jdy09y0A4t0[/youtube]



The movement of 1.3 billion stars in the next 800.000 years as seen from Earth:

[youtube]wcW1coBGETo[/youtube]


The video was produced using GaiaSky. A free and open source software that allows for quick DR2 data visualization. The program itself is very user-friendly and allows to travel in a simulated universe just like SpaceEngine (it is uglier but it has stellar movements). Give it a try! Maybe SE could borrow some code from them (since it's open source).

[youtube]LcHgvjx4nuA[/youtube]


The Large Magellanic cloud slowly rotating through 2 million years of history (just by using Gaia's proper motions). ¡You are watching the rotation of a satellite galaxy!

[youtube]ynEp7SjJ9vU[/youtube]


In this video (also made with GaiaSky) you can see the extension of the catalogue throughout the entire Milky Way. You can also see the arrows marking stellar motion.

[youtube]ed1JhrCnY-w[/youtube]


And finally a video showing the 14.099 asteroids detected in DR2 so far. This is important, I'm saying "so far" because these are just the obvious asteroids, we classified them just as the data release came out using known objects from other databases. Just a few of those are actually newly discovered asteroids. Bare in mind this; there are possibly hundreds of thousands of asteroids in DR2 data waiting for a much more clever analysis. Expect thousands of new asteroid discoveries in the next months as the data is studied with much more care.

[youtube]Fv8iq6jIfdE[/youtube]
 
User avatar
SpaceEngineer
Author of SpaceEngine
Author of SpaceEngine
Posts: 1125
Joined: 17 May 2016 22:16
Location: Saint-Petersburg
Contact:

The Gaia Mission

10 May 2018 09:11

SE can't handle more than few million stars right now. I will rewrite the catalog stars octree so it will support dynamic loading of visible nodes only (just like procedural stars octree works). This way it will support unlimited number of stars, just like terrain engine supports unlimited resolution of textures (almost, limited only by accuracy of floating point numbers).

Star motions is a complex topic, because octree is a static data structure. I can make a hack and expand the octree nodes ~2 times (so they will intersect each other), this way they could contain their stars even if they are moving. The limitation will be performance and accuracy: with GAIA data, we have only linear movement data, so extrapolating it for 100 million years have no sense. 1-2 million years could be limit.

To represent a star, SE need its 3D position, luminosity, and spectral class (SE must know what to render: a main sequence star, red giant or white dwarf). GAIA doesn't provides the class, but it provides radius and effective temperature for many stars. These data can be used to classify stars in SE. Or I may expand the star solver so it will make classification based on R-V color, or whatever GAIA provides.
 
User avatar
FastFourierTransform
Pioneer
Pioneer
Topic Author
Posts: 356
Joined: 17 Nov 2016 15:09

The Gaia Mission

10 May 2018 10:01

Those are amazing news Vladimir! Expanding the ability to load much more star would be just one of the most awesome features ever.
To represent a star, SE need its 3D position, luminosity, and spectral class (SE must know what to render: a main sequence star, red giant or white dwarf). GAIA doesn't provides the class, but it provides radius and effective temperature for many stars. These data can be used to classify stars in SE. Or I may expand the star solver so it will make classification based on R-V color, or whatever GAIA provides.
For DR2 we have the full 6D parameters (two sky coordinates, parallax, radial velocity and two components of proper motion) for 1.3 billion stars. Photometry is done with G, B and R bands (they have different precision). Under the "teff_val" column of DR2 you have a precomputed effective temperature for the star (done using photometry and extinction measurements) so, in principle this would be easier. "teff_val" is only available for 161.5 million stars (I don't know how many of those have also at the same time the 6D astrometric parameters but I estimate more than 90%). Radius and luminosity for the star is only available for 77 million cases. So maybe the best we can handle right now are at max 77 million stars for SE. Also, we should apply some strict criteria If we want to import part of DR2 to SE; there is a photometric quality flag "phot_proc_mode" that goes like 0: Gold, 1: Silver and 2: Bronze. Now with DR2 the vast majority of stars with photometry have high quality (Gold) data so we should keep that as a criterion (I've seen silver and bronze two-color diagrams and are just ugly). For 6D parameters there's also the issue I always talk about; the fact that the inverse of parallax is not an unbiased estimator for distance due to Lutz-Kelker bias and others. We should never go with parallaxes with 10% uncertainties or larger and even with 5% to 10% uncertainties we should perform a Lutz-Kelker correction if possible. If we wait for some people to perform the calculations we could have all the needed data for tens of millions of stars in SE. And if we wait for DR3 in 2020 this is just going to get bigger.


SpaceEngineer: The limitation will be performance and accuracy: with GAIA data, we have only linear movement data, so extrapolating it for 100 million years have no sense. 1-2 million years could be limit.
You are absolutely right. No more than 2 million years to the past and to the future for realism. Also true, thanks to Gaia's DR2 we are going to be able to produce high quality models of the galactic potential. I don't know how hard would it be to implement a calculator for longer time intervals. Taking into account halo, disk, bulge and spiral arms you could easily extend the time-frame to hundreds of millions of years. Even with a simple disk potential model you could make an approximation for stellar orbits around the Galaxy. But yeah this is just wild speculation right now. Your post made me so happy.



P.D: I'm very busy right now so I have to apologize to you because I never finished the corrected stellar catalogue for SE that we talked about, but I'm thinking and rethinking it (now with more precise data from DR2) so I hope to make it in the summer at worst.
 
User avatar
FastFourierTransform
Pioneer
Pioneer
Topic Author
Posts: 356
Joined: 17 Nov 2016 15:09

The Gaia Mission

26 Sep 2020 14:16

Well, as many of you might know by now, we broke the 3rd Gaia Data release in two pieces, the first of them, EDR3 is now scheduled for December 3 at 12:00 CET, and the full DR3 release will happen somewhere in 2022 (the date is still unknown).

What should we expect from the December release?
Well, for now at least 100 million additional stars that DR2 missed (that's a ~6% increase in the amount of stars). This is funny, we started the mission expecting a billion star survey and now we are slowly reaching 2 billion stars (not only Gaia is performing better than expected but also the Milky Way seems to be more populated than we previously thought).

As you might also know, not all the stars in the previous release (DR2) had parallax and proper motion measurements, in fact it was 78.7% of the sample that had all of them. But now with EDR3 many more have parallax measurements for the first time (thus, geometric distance estimates). In fact around 1.5 billion stars are expected to have the full set of astrometric parameters (that will be around the 83% of the new release). So more distances and more completeness inside the catalog.

Another beautiful thing to expect from December's release is the increase in accuracy of the parallaxes and the better disentanglement of the parallax motion with the proper motion (thanks to a longer time since the mission begun). We expect an improvement in the parallax measurements of 20% with respect to Gaia DR2, that will in turn translate to some improvements in the estimated distances to the stars. The proper motion will be a factor of 2 more precise (better understanding of the dynamics of the galaxy and the evolution of our stellar neighbourhood).

So, to sum up, the EDR3 will enhance the data of DR2 and add a little more stars to the catalog solidifying what we currently have. But the true next quantum leap (as the one that happened when DR2 was released in 2018) will happen in 2022 with the full DR3 release, when radial velocities will be calculated for a massive number of stars, low resolution spectra will be collected (so we can finally have a consistent way of estimating the classification of hundreths of millions of stars), and brightness variability data for the first time (which will show millions of new binaries, thousands of new exoplanets and hundreths of new Cepheids that in turn will help us to make the distance ladder stronger and change our knowledge on cosmological scales).

We expect funding for an extended Gaia mission (for now ESA can talk about plans only up to 2022) so we can yield a DR4 and a DR5 in the next decade. This spacecraft is one of the things I think of to make me feel proud of mankind.
 
A-L-E-X
Galaxy Architect
Galaxy Architect
Posts: 3498
Joined: 06 Mar 2017 20:19

The Gaia Mission

01 Oct 2020 07:40

Hi, whats the difference between these and what I downloaded from Phunnie?  I actually tried both his 30 million and 15 million star add-ons and while I found the 30 million add-on workable, there was some graphics flickering from time to time so I switched to the 15 million star add-on which is working fine.
 
John Done
Observer
Observer
Posts: 11
Joined: 31 Aug 2020 07:08

The Gaia Mission

02 Oct 2020 06:39

The Milky Way is not a nice neighbour. It’s constantly taking stars from other star clusters and smaller galaxies. Gaia has shown us this in great detail, and has found streams of stars torn form other galaxies, stretching through space. Some of these streams stretch for thousands of light years. Gaia observations of these streams can tell astronomers a lot about the Milky Way’s gravitational power, and its mass.Gais has also found stars travelling through the Milky Way at very high speeds.

Who is online

Users browsing this forum: No registered users and 21 guests