If you advertise and it is oversubscribed, that is an opportunity to get the very best brains. One does not need to re-advertise with higher requirements (though subsequent ads should take that into account).
One can for example insist on higher academic qualifications and treat the advertised qualifications as the minimum. Where one asked for a Bachelors degree in a specific area and several Masters degree holders apply, these should have an advantage as long as the said degrees are relevant to the job.
I know for a fact that "relevant experience" does reduce the short list dramatically. Relevance should however be thoroughly worked out.
Most organizations celebrate when an advertised position is over subscribed and of course panic when the number of applicants declines. Otherwise people applying for jobs they aren't qualified for is a very normal act. They just have to be treated fairly by their applications being individually handled.