When using the content manager I noticed that building my project was incredibly slow. I couldn't put my finger on what it was until I noticed that the files extracted from the XACT project file were being extracted every single time I compiled my project regardless of any changes I made. Hence, the slow build.

Is there something I can set to fix this or a way to avoid this mess? I'm using the same content manager for both the Xbox and Windows version so I'm looking for a solution that will work with both.

Go into preferences -> build in Visual Studio, and turn up build output verbosity to "diagnostic."
Then look at the build log, and see why the system decides to re-build those files.
It may be that some property has accidentally been set in the project to always build them.
Or it may be that they are dated in the future, for example, so that the output files are marked as older than the input files.
The build log ought to help narrow down the possible culprits.

I still haven't quite narrowed it down but this seems to be the error.

Building target "_CopyOutOfDateNestedContentItemsToOutputDirectory" partially, because some output files are out of date with respect to their input files.

Somehow removing the project and readding it fixed the problem. I readded it as an existing link but I'm not sure how much that helped or even if it helped at all.
In any case, problem solved!