As primates, which are a very social order, our ancestors lived in relatively small groups in which everyone knew everyone else. Our minds are adapted to deal with populations of that size. Our ancestors made strong distinctions between members of the in-group and outsiders, and we still make such distinctions today -- social psychologists can create in-group and out-group feelings based on virtually any arbitrary difference between populations.
The in-group and out-group intuitions help fuel opposition to expanded trade and immigration. The public intuitively believes that the beneficiaries of such policies will be foreigners, and it is easy to arouse suspicion about those who are not part of our in-group. When coupled with zero-sum thinking, this is a powerful political tool. For instance, a domestic industry or collection of domestic workers, when having difficulty competing with foreign or immigrant competitors, can use innate dislike of outsiders when advocating for increased barriers.