Actually, and I don't know why, brazilians have no trouble understanding Spanish but the inverse is not true.
I went to Argentina and Colombia and could had no trouble from my side. Needed to speak Spanish tough because no way they get Portuguese
Probably at least two major factors:
1) Brazilians get more exposure to Spanish in school, television, culture, ect. than Spanish speakers do to Portuguese. Furthermore, Brazilians are more likely to travel to Latin American countries than the other way around.
2) Portuguese (and this is especially true for the Brazilian variant) has quite a bit more complex intonation and pronunciation than Spanish does. Comprehension of spoken Portuguese is more difficult than Spanish is. Put a book written in Portuguese in front of Spanish speakers and they will understand 15-20% of it. Speak Portuguese to a Spanish speaker and they will understand much less.
A lot of people overestimate the similarities between Portuguese and Spanish, though. Somewhat similar structure, and some significant similarities in verb conjugation, but there's a lot less overlap in words than people think there is.