Hold up, you're actually in the wrong here. That chimney debris is your mess to clean up, not his problem. It can wreck his yard and cause real damage if it builds up over time. He has every right to be upset if you're not handling your own stuff. Cleaning your chimney sometimes is just part of owning a home, so maybe do it.