But approaching the black hole behind and in front get somewhat smeared out, so would there rather be bands of red and blue?
Objects behind you will be aberrated into view ahead, but there won't be bands of color. Just a smooth transition from redshift behind to blueshift ahead, as illustrated
here.
The reason is that gravitation is a conservative force, even in general relativity with black holes. This means the change in energy, and thus color, of a photon is path independent, and only depends on the change in radius in the gravitational field. Since all starlight comes from very far away from the black hole, where its gravitation is negligible, the change in color of starlight due the gravity is the same regardless of the direction it came from. It won't even matter if a photon made multiple orbits near the photon sphere before reaching you vs. came in from directly behind you.
So, the effects of the black hole's gravitation and your extra velocity do not entangle. We can treat them separately.
If you fall in with some additional velocity, then you introduce the special relativistic aberration, but there's still no change in color across the sky from the gravitation because of the path independence.
The special relativistic aberration effect also appears in the above video. The camera didn't drop straight in, but started off with a little
sideways velocity, and spirals inward. That sideways velocity increases closer to the black hole's center, blueshifting and brightening the view in that sideways direction of motion. (Another way to think of this effect is that the camera is plowing sideways into a rain of photons -- causing them to appear to come more from that direction and with greater numbers and energy, just like driving in rain and having the raindrops hit your front windshield harder.) You can really begin to notice it at around the 35 second mark, as an intensifying glow directly ahead and above the black disk.
Here's another useful visual (
source, again Andrew Hamilton's work) showing what it would look like to be orbiting at close to the speed of light, just above the photon sphere. Special relativistic aberration concentrates, brightens, and blueshifts the view ahead. The aberration also warps the black disk, making it appear larger and take up almost half the sky. (Aberration preserves the shape of a circle, but not its size.)