My mitigation strategy is I book calendar time for entire days to work on stuff. "Want to get coffee on X or Y date?" "Sorry, I'm totally booked those dates -- how about Z date/time?"

It makes me seem unavailable/scarce, and allows me to book things on my "manager days". The key is using the calendar to your advantage .. control your own time.