I just upgraded to macOS Sierra (10.12) and have been finding some odd things happening with the search path that is being used to find installed libraries. Unfortunately there isn’t a very good solution at this time but I’m sharing what I’ve found in case it helps someone else.
Apple includes Python 2.7 with macOS and it is used by the system for various things. The general wisdom is to not mess with the system-provided install (and whatever you do don’t remove it unless you like reinstalling the operating system). Because I need to install other libraries that aren’t provided with the operating system I’ve installed the Python 2.7 distribution (at this writing version 2.7.12) from python.org and use that for my development work. That is where the problems start to show up.
There are two ways to install new libraries (called packages) using the pip install tool:
1. The global (global for the Python install) site-packages folder.
2. The user’s own site-packages folder.
Since I’m the only user on my system I prefer to just put everything I need into the global site-packages folder since in my case it is equivalent to the user-specific one.
When the Python interpreter starts up it, through various means, creates a search path that it uses to find other packages that you’ve installed when you try to use them with the “import” statement. You can see this yourself by opening up a Python shell and typing:
1. import sys
You should see something like:
Notice the paths with prefixed with “/System”? There is our problem. Through the standard path extension system our supposed-to-be-standalone Python environment is now seeing one Apple-provided library we might not be interested in using (PyObjC) and a folder where other libraries might be installed.1
The thing to note here is that those paths are at the end of the full sys.path so they will be searched last. If there is a version of the same library (PyObjC) for instance in one of those earlier paths it will be used instead of the system one. So theoretically I shouldn’t have to do anything but I use PyCharm from JetBrains for my Python development and I don’t like seeing this in my projects:
This can be fixed however and here is how to do it:
1. Ignore the problem. If I wanted to do that I wouldn’t be writing this article.
2. Modify a file called site.py that is responsible for generating the Python sys.path
3. Update the file Apple put in place to add those folders to sys.path.
Let’s look at #2 first because that’s the solution I’ve chosen. site.py is installed here by default:
and can be easily edited. The key is to filter out any lines that start with “/System” once the path has been built. sys.path is just a list and can be easily modified by any Python script. I’ve created a gist here with the code. The change was made at line 548 to remove the system paths from sys.path.
One thing to note about doing this is that this file can be overwritten by another Python install and these changes might need to be made again.
What about #3 above? I don’t recommend doing it because it’s an Apple provided file but it’s an easy change to make as well. The file itself is called Extras.pth and just contains two lines: the two paths we want to remove. It lives here:
The only thing required is to comment out those two lines.
I hope at some point there is a better way to address this problem but for now there is an easy solution for it. If you’re not like me and don’t care about having the system-provided stuff viewable by your other Python install none of this is an issue. As noted earlier if you have the same version of a library in one of the folders that’s earlier in sys.path it will be used instead.
Here is a list of some other reading on the subject I came across while looking into this issue. Some very interesting background information here.
- I have to assume that Apple did this to enable people to add packages to the system-provided Python without breaking anything but I don’t like that it’s there polluting my own Python install. ↩