The Bluetooth codec your headphones use significantly impacts sound quality. Unfortunately, macOS sometimes uses the subpar SBC codec instead of the superior aptX or AAC codecs. Here's how to force macOS to use one of those instead.
What is a Bluetooth Codec?
The word "codec" is short for "coder-decoder," which is precisely what a codec does. In this case, the codec is used to compress the audio file on one end (code) and decompress it on the other (decode). This allows the file to be transmitted quickly. Some codecs are better than others at compressing audio without reducing its quality, which is why choosing the right one is essential.
SBC is the codec that macOS sometimes defaults to, whereas aptX and AAC offer better sound quality. All three codecs compress the audio before sending it to your headphones, but aptX and AAC do so without a dramatic impact on the overall audio quality. SBC was designed to be efficient and ensure compatibility rather than to offer the best possible sound quality. Latency is also a consideration with SBC, and audio lag is noticeable when watching video content. Neither aptX or AAC exhibit this behavior.
If you want to learn more about how aptX and AAC work, we have a good primer on the subject.
In short, aptX promises "CD-like" sound quality, which is something SBC cannot offer. Likewise, AAC's improved compression algorithms allow for better sound quality, despite its lower maximum bitrate of 250Kbps when compared to the 328Kbps of SBC. The aptX codec, however, bests both at 352 Kbps.
Unfortunately, not all Bluetooth headphones support aptX or AAC, although most of the more recent models do. If you do force your Mac to use either codec, but your headphones don't support them, don't worry---it will automatically switch to SBC instead.
How to Check Which Codec is In Use
To see which codec is in use, connect your Bluetooth headphones to your Mac and play some audio. Hold down the Option key and click the "Bluetooth" icon in the menu bar.
Highlight your headphones and note the Active Codec.
This is the codec that is currently in use. You're fine if it's aptX or AAC; otherwise, read on.
How to Force aptX or AAC
Download Additional Tools for Xcode from the Apple Developer site. You will need to create a free Apple Developer account to do so. Once it's downloaded, double-click the "Additional Tools" DMG file to mount it.
Double-click "Additional Tools" on your desktop.
Double-click the "Hardware" folder.
Drag "Bluetooth Explorer.app" to your Applications folder.
Double-click the "Bluetooth Explorer" app and click "Tools > Audio Options" in the menu bar.
Check the "Force use of aptX" and "Enable AAC" boxes. Make sure the "Disable AAC" and "Disable aptX" boxes are unchecked. (Yeah, for some reason the app will allow you to check both "enable" and "disable" boxes.)
Click the "Close" button to save the changes.
Disconnect and reconnect your headphones, and then confirm that they are now using the aptX or AAC codecs as described earlier. If they are, you should notice an improvement in sound quality. If not, it might be time to treat yourself to some new headphones.