Particular entities start with a large positive empathy account value. The value is drawn down through my repeated exposure to them (due to the associated emotional toil I experience).
Absent an intentional deposit by that individual, only a lack of exposure is able to make the necessary improvements to the balance.