Once you have created a Project and selected the "Specify Contacts for Project" and therefore "Show Only Specified Contacts on Add/Edit Forms" you can no longer change that setting (and you can't even see how it is set).
So if you missed setting that up as required on initial creation or the situation changed on an existing project you are stuck with how it was done at project creation.
Could this setting be visible and editable where if you turn the setting on all contacts from existing Project Directory Companies would be shown and you can delete whos not required. Or if you need to go the other way and turn setting off then it will just be the companies from the contacts previously selected.
Plus if you have "Specify Contacts for Project" you could also turn the "Show Only Specified Contacts on Add/Edit Forms" on/off as required.