Both inputs to the handicap are dynamic, so it can shift without you taking any action:
Your percentile can shift. The influence factor percentile is a ranking relative to all other tracked accounts on the platform. As new users join or other users' data is updated, the distribution shifts. Even if your data stays the same, your percentile can move and crossing a tier boundary changes the penalty.
Your underlying score can cross a tier boundary. The handicap is calculated on your score before the penalty is applied. If other score factors (reviews, vouches, etc.) push your pre-penalty score into a higher tier, the gap between your score tier and influence tier widens, increasing the penalty. This can happen even while your visible score barely changes, because the higher penalty absorbs most of the gain.