The 4GB limitation does not come from the Lagarith video codec or Space Engine. It comes from the container, or in this case, the AVI file format. In its standard form, the container does not support files larger than 4GB, does not properly support timecode, and doesn’t properly support pixel aspect ratios. There are non-standard extensions to the container format, like OpenDML which overcomes the 4GB file limit, but video player support is dubious. Also many video players have trouble playing ANY file types larger than 4GB.
The good news is there is no 4GB limit for MP4s. To build MP4s larger than 4GB the muxer must use 64-bit addressing. MPEG’s MP4 container is an industry standardized format based on Apple Quicktime MOV with some additional extending features. It is commonly both an acquisition (recording) and delivery (playing) format. Although it can contain different codecs, the most commonly used codecs in the MP4 container like H.264 aren’t totally suitable in an editing environment. Also H.264 is slow to seek in a player, so if you intend to browse around the video stream, either fixing stuff or looking for stuff, it's not an ideal choice. H.264 does give you the highest quality compared to file size though.
So, instead of using the Lagarith lossless video codec (faster, AVI, smaller than 4GB), you can use the lossless Ut-video-codec (slower, MP4, larger than 4GB). Once installed it will show up in your list of codecs when you press configure on the F9 video capture window in Space Engine. However there is another way that is separate from the built in video recorder with Space Engine. That is, the Dxtory movie capture program!
Dxtory by ExKode is a movie capture tool for DirectX and OpenGL Applications. Since data is directly acquired from a surface memory buffer, it is very high-speed and works with smaller CPU load. With it you can use the Ut Video Codec to create MP4 containers larger than 4GB in size.
---
Encoder - Latest Version 2.0.141 (December 07, 2016)
http://exkode.com/products-en.html
Codec - Latest Version 17.3.0 (January 15, 2017)
http://www.videohelp.com/software/Ut-Video-Codec-Suite
Remember, be sure to get the 64-bit version for making files larger than 4GB.
If you use the Ut video codec alone, without Dxtory, and select it from the F9 menu it will be in AVI format still. I select the second Ut codec on the list, "UtVideo RGB VCM" and when configuring just leave as is, leaving the frame divide count at 8 and with optimize for decoding speed selected.
If you do use the Dxtory capture program, change your video settings in it (yellow button at the bottom) to your screen resolution. Also, under the grey folder looking icon, set your folder settings; where you save the movie and where you save the temporary screenshots that make up the final movie. The hotkey to start and stop the movie is already set up as F12.
---
I've been playing around with Dxtory with minimal luck tonight. I am not sure how to record/write in MP4 format yet, even with the H.264 codec, which I've seen others do. Plus the free download version does have a name branding across the screen which makes it pointless to use, for me. I do know you can use the same program to combine AVI files into an MP4, that might possibly be larger than 4GB, if that's helpful in some way, but my drive is FAT32 and I won't be doing any re-formating or partitioning any time soon, so I won't know. One good thing that I learned is using the program and codec together allowed me to make a 2+ minute long movie in 1080p@30 frames that is only 8MB, but it is an AVI and it did capture the GUI. My next task before sunrise is to use the built-in video capture in Space Engine using the various Ut codecs this time. Let me know what kind of luck you have if you decide to try any of these ideas.
BTW, recommended settings for HD videos for uploading to the web (in 2014)
Format MP4 (file extension .mp4)
Video Codec: H264 (High profile)
Frame Rate: 24, 25 or 30 (Constant)
Frame Size / Resolution: 1280 x 720 (720p)
Fields: Progressive
Bitrate: 5,000 – 10,000 kbit/s
Audio Codec: AAC-LC
Bitrate: 320 kbit/s
Sample Rate: 44.100 OR 48,000 kHz (retain original sample rate)