Paraguay striker Roque Santa Cruz converted the decisive spot kick to give the home side a 5-4 win in the shootout, which settled a dull tie that was goalless after extra time.
Schalke's captain, Denmark striker Ebbe Sand, contributed to Bayern's qualification for the quarter-finals when his opening penalty bounced off the top of the bar.
Bundesliga leaders Bayern, reduced to chasing domestic glory after bowing out of Europe, took revenge for their semi-final defeat by the same opponents last season.
The first half was a tight, tactical affair with few highlights after Bayern goalkeeper Oliver Kahn brought down Belgium striker Emile Mpenza inside the penalty area a few seconds into the game but escaped unpunished.
Bayern stepped up a gear after the break but kept bumping into a compact defensive wall and hardly created a clear chance.
Substitute striker Elber came close to scoring with a superb volley from inside the box five minutes from the end of extra time in the most spectacular moment during a poor game.
Mpenza fired just over the bar from a counter-attackseconds later when Santa Cruz wasted another great chance for Bayern with an effort from close range that Schalke goalkeeper Frank Rost did well to save.
Bundesliga strugglers Nuremberg, whose Macedonian striker Sasa Ciric missed a penalty just before the break, went out after a 2-0 home defeat by second division leaders Cologne, striker Matthias Scherz scoring both goals in the second half.
Four-times Cup winners Werder Bremen needed a strike one minute from time by substitute Ivan Klasnic to snatch a 2-1 win at second division LR Ahlen.
Greek striker Angelos Charisteas had put Werder ahead on 16 minutes, but the home side responded with a headed goal from midfielder Marc Arnold in the 74th minute.
Crisis-hit Kaiserslautern also went through with second-half goals from defender Hany Ramzy and midfielder Lincoln for a 2-0 home victory over second division SC Freiburg, who had knocked out German champions Borussia Dortmund in the previous round.
Bayer Leverkusen, last season's losing finalists, advanced on Tuesday after needing a penalty after the break for a 2-1 victory over second division Waldhhof Mannheim.
SpVgg Unterhaching, the only amateur side left in the competition, reached the last eight for the first time with a 3-2 win over first division Hansa Rostock, also on Tuesday.