You have to look what exactly is being replaced:

Je veux venir à la piscine. Here, à la piscine is being replaced by y. Therefore the sentence becomes: Je veux y venir.

Je veux venir à la piscine. -- I want to come to the pool.
Je le veux. -- I want it.
Je veux y venir. -- I want to come there.

Sam, I was a little confused initially but it occured to me that when you use "y" to replace a phrase beginning with "à" it is because you are talking about an action, wanting "to go" there the emphasis is on the action of going, when you replace a phrase beginning with "à" with "le" or "l'" it is because you are talking about an idea, the emphasis is upon "wanting" to go there, an idea not an action. Also the lesson suggests that "le" or "l'" mean "it" or "that" in English, not "there" which is the suggestion in other lessons with "y".  Would welcome the views of the experts on this perception.