One order, two axes
A B-tree is an ordered thing. It answers ranges fast because everything in the range sits together in the leaves, so the database seeks once and reads forward.
Notice what that property depends on. There being exactly one order.
Coordinates have two axes, and no ordering of pairs keeps neighbours together. Sort by latitude and two points at the same latitude sit side by side in your index even when one is in Portugal and the other in Kazakhstan. Sort by longitude and you get the identical problem, rotated.
A composite index on both does not rescue you either. The leading column decides the scan, so the database seeks to the latitude band and then reads every row in it, checking longitude one row at a time.
What it costs on real data
Run the numbers on a real box. A 5 kilometre radius near London is about 0.045 degrees of latitude and 0.072 of longitude.
Over a table of 8 million points, the latitude band alone picks up roughly 40,000 rows. Your database fetches and tests every one of them to throw away the 39,000 at the right latitude and the wrong longitude. That is 40,000 reads to return 1,000 results, and it gets linearly worse as the table grows.
State the requirement precisely, because everything that follows is an answer to it. You want one ordered key, computable from a pair of coordinates, where keys that sort near each other describe places that are near each other.
Worked example
Marta ships a store locator for a Portuguese pharmacy chain, 3,100 branches, on Postgres, a relational database, with a composite index on (lat, lon). It returns in 8 ms and nobody thinks about it again. Two years later the same service backs a delivery app covering 4.2 million listed businesses across Iberia, and p99 for the nearby query, the number the slowest one request in a hundred comes in under, has climbed to 1.9 seconds. EXPLAIN shows the cause plainly: an index scan on the latitude range returning 61,000 rows, then a filter discarding 59,700 of them. The index is doing its job, just in one dimension. Switching the column to a geohash prefix and querying 9 cells drops the rows examined from 61,000 to about 1,400, and p99 to 22 ms, with no change to the hardware.