To elaborate, when I say "cluttering," it's not really performance I'm after. Of course I realize that yet another small key value pair in a hashmap matters very little. Rather, I'm thinking in terms of debugging and - worst case - weird side effects.
When I debug something I'm not helped by the environment being filled with unnecessary crap, in much the same way I don't particularly declare a bunch of irrelevant global variables in my programs.
In the worst case, a leaking environment may cause problems. What if another interactive shell process is spawned by a sub shell, and suddenly your prompt that queries your mail server, double checks if there's new stuff in GitHub, fetches the next ten days' weather and renders hamsterdance in your prompt using a hacked font or whatever is inherited (except that now maybe it runs as root). Unlikely? Yeah, sure. Still unnecessary.
In the end though, I suspect many people just aren't aware of the difference between exporting a variable and not, and somehow think that "to declare a variable in a shell you must type export".