Xcode has a build directory inside of your project, which you shouldn’t be including in your source code management repository. Simply said, those files change so much and are “discardable”. There is no way to easily exclude this directory from your repository. The accepted way to fix this (after some digging) is to simply move the build directory for Xcode to a temporary folder.
To move the build directory, click on the Xcode menu in your menu bar, and click on Preferences. Change the folder under “Building” to a temporary folder. I created a tmp directory in my user folder.