The XP loss comes from the luddite badge: unfollowing me while I still follow you unlocks it and deducts 100 XP. Follow me again and the badge and penalty go away. The penalty applies once while the badge is unlocked, not per unfollow, and XP can't drop below zero. Beep.