I assume what you did is:
if (dist < -absDist)
return OUTSIDE;
break; // this doesn't end up doing what you think it will
// and not something super terrible like:
if (dist < -absDist)
break; // if did this, slap yourself
No. Because the entire box could be on the backside of 1 plane without any other corners on the backside of any other plane.
In that case the box would be out of view off to one side, or behind the nearplane, etc. Your change fixes your problem because you’ve broken the test to return false-positives very liberally, so virtually everything will pass containment tests.
Which type of light is having the shadow flicker? For instance directional light’s have an extrusion (GetShadowMaxExtrusion
SetShadowMaxExtrusion
) move back the light from it’s bounded fit so that offscreen things cast shadows. Default for that is 1000, if you’re using a tiny unit (like 1.0 == 1cm) then you only have 10m of peeling back.
If other lights are doing it then there’s a containment test problem somewhere either in selecting the light or selecting the shadowcasters to render (more likely the former). You’ll have to step through passing and failing frames. I’d use Renderdoc to capture both cases to see what the shadowmap passes are doing (ie. are the shadowmaps differing between the two) first to make some determinations of where to go next.
If you need more help then post scene files or at least pictures so we can see what’s going on.