Mining enough to alter the orbit of the moon would require a pretty ridiculous amount of time and effort. Much more than our global mining efforts combined and multiplied and on a timescale of thousands of years.

And we only have to launch a few rockets, enough to set up a self-sufficient base which can then produce more rockets and fuel from resources on site. Not to mention it's much easier, and even feasible with existing materials, to build a space elevator on the moon.

Would it not be super cool to have all those minerals until we have extracted that much from the moon that it's orbit becomes unstable and then spirals into earth?

First, figure out how much the Moon weighs. The find out how much we mine form the Earth each year.

Second, the impact of dozens of flights a day will be much less than the impact of mining the Earth