My Media Library permalinks are bringing up the source items directly and seem to be ignoring my attachment.php file.

This is happening for both attached and unattached files.

I'm trying to use attachment.php to show meta information and custom taxonomies. Should I do something with single.php to try to get it to recognize the existence of attachment.php, or is there something else I should do?