Zeolites replace phosphates in detergent. Is that bad? Aren’t phosphates more harmful in wastewater than zeolites?

As I understand it, some zeolites can be aluminum based, and there’s some research to show a connection between breast cancer and some aluminum products (deodorant being the largest source). But from what I’ve read these connections are tenuous, and mostly related to aluminum in deodorant being absorbed and having estrogen-like effects in the body, and the relationship of estrogen to breast cancer.

Why, pray, do I not want to put zeolites in my washer?