As we have seen before, the pillar worship might likely have been associated with Lord Shiva at some point in time. Then for cleanliness purposes, a round or a rectangle with a spout might have been made around the pillar and it got its present shape. Then because of its shape and the word linga, it could have been misunderstood as sexual organs of the god and the goddess and considered a symbol of fertility. The same can be true with phallic worship also.
Also, some swayambhu lingams look like mountains. Hindus believe that Lord Shiva resides on Mount Kailash. Therefore, it is also possible that people might have started worshiping those mountain-like structures as the miniature form of Kailash as a symbol of Lord Shiva. It explains why the Shiva lingam at Kedarnath Temple is worshiped even if it is in a pyramid shape.
Shiva and Shakti are considered inseparable; therefore to represent Goddess Shakti, the pedestal could have been added to the lingam afterward.
Most probably, both pillar and phallic worship were associated with Lord Shiva at some point in time. Therefore, there are two main groups that consider the Shiva lingam a pillar and/or phallus.
It explains why there are stories associating it with both sexual and non-sexual things.
Based on the logical evidence, it could be a pillar, phallus, or Mount Kailash.