you were never part of the Church. You were the flock the Church oversaw.
The clergy is the Church, a small group of mostly men that live among themselves and protect and serve their group to the exclusion of everyone else. They protect the priests over the congregation.
They protect child rapists rather than the children.
You are not leaving the Church. They don't consider you the Church. You are choosing another venue to follow your faith.