Eric Dier scored the winning kick after Jordan Pickford's brilliant save from Carlos Bacca's attempt to set up a game with Sweden in Samara on Saturday (7).
Gareth Southgate's side had been seconds away from the last eight during normal time, with Harry Kane's penalty after he was fouled by Carlos Sanchez in the box - his sixth goal of the tournament - looking as though it was going to send them through.
But deep into injury time, Colombia threw everyone forward for a Juan Cuadrado corner, including goal-keeper David Ospina, and Yerry Mina headed a dramatic equaliser at the Spartak Stadium.
Kieran Trippier was standing on the line but could not keep out the Barcelona centre-back's third headed goal of the World Cup.
The Three Lions had been on top for most of the 90 minutes but appeared understandably deflated in extra time - although Danny Rose and Dier missed late chances to win the game and avoid a shootout.