This was blocking SSH connections from being established if a dir that wasn't created yet is set.
codersdk
agentsdk