When I and a friend use E2EE between us neither of us are helping to hide a criminal's conversation regarding their criminal activity. The same can not be said about a mixer. They are not at all equivalent.
1. You are using and supporting the same protocol that they are also using. As the application grows and improves because more people are using and supporting it, the criminals are also being helped.
2. You are creating cover. The more people that use Tor and Matrix, the more secure it becomes for all users within the network.
If you and a criminal are both using the same Matrix server, neither you nor the host would know. Your plain text messages are going into the protocol, getting mixed and encrypted, and then spit out the other end.
You seem to be arguing as if this is an all or nothing proposition. Instead there are definite degrees of intent and interaction here. In the E2EE case the degree of interaction and intent is quite low. In the mixer case the degree is quite high. The line I believe should be drawn somewhere in the middle. But the Mixer case definitely is on the wrong side for me.
E2EE is all-or-nothing. You don't see Matrix adding a special case in the encryption technique to notify authorities if the phrase "nuclear" or "minor" appears in a message. The intent of E2EE and Tornado Cash is exactly the same: privacy. The line you draw here is arbitrary based on your own subjective view of the space.