I think I know what it is on my door panel I put a 8 in circle diameter with a 5 inch hole in the middle half inch thick Baltic Birch it's my speaker spacer.
I think when the panel goes back on it's hitting part of that speaker spacer because the way the speakers were attached originally they were short of on a bracket that looks sort of like a triangle they didn't have a full circle it only had the three mounting points and the rest of it was cut away I think that's what's happening.
I may have to pull the speaker off remove the spacer take a jigsaw and cut part of that outer circle off the spacer and put it back on and see if that's what stopping the panel from going on.