There are a large number of really crappy sound card implementations in the consumer space. If you're sure it's not your own fault, then you'll have to work around it. For example, you can detect when you receive too much data, and drop a packet now and then on the receiving side (don't let the queue exceed X milliseconds). For the reverse problem, you will also detect underrun when you get too little data, which will automatically add one buffer's worth of buffering through a silence/repeat glitch.
The real solution is to use good hardware, with ASIO drivers, which is defined and implemented for true sample accurate record/playback. That limits the amount of hardware you can use, though, as most cheap crap consumer hardware doesn't have ASIO drivers.
Jon Watte, Direct3D MVP
kW X-port 3ds Max .X exporter
kW Animation source code