I originally brought this up in another forum, but wanted to drop it here now that I've isolated the problem to the Polar Warp effect.
Take any layer and apply the Polar Warp effect. If needed, adjust the effect's End Radius parameter so that you can see the outer edge of the warp as a circle (not cropped into a square/rectangle by the layer's original dimensions). That outer circular edge is not smoothed when "Antialiased" is on in the Viewer panel, nor is it antialiased during export. If you increase the effect's Start Radius to turn the warp into a ring, you'll notice that the inner edge is also not antialiased.
I'm seeing this behavior in HitFilm 3 Express 3.1.0132 (update 8) on a Mac Mini with 8GB of RAM.