Because IP does not provide any mechanism to ensure that data packets are delivered in sequential order, or provide any Quality of Service guarantees, implementations of VoIP face problems dealing with latency and possibly data integrity.

One of the central challenges for VoIP implementers is restructuring streams of received IP packets, which can come in any order and have packets missing, to ensure that the ensuing audio stream maintains a proper time consistency. To help with this, the network provider can ensure that there is enough bandwidth end-to-end to guarantee low-latency, high quality voice. This is easy to do in private networks, but much more difficult to do with less than 256 kbit/s without a fragmentation mechanism.

The other main challenge is getting VoIP traffic to traverse firewalls and NAT. Intermediary devices called Session Border Controllers are often used to achieve this, though some proprietary systems such as Skype do firewall and NAT traversal without the need for an SBC.

Keeping packet latency down to acceptable levels on satellite circuits can also be a problem, simply due to the distance of the transmissions.