I thought it meant people with disabilities, and that his comment was definitely ableist. But you're probably right: we often aren't on anyone's radar screen enough for them to think of making pejorative comments about us.