Python 2.7 on the Mac: site-package weirdness

I just upgraded to macOS Sierra (10.12) and have been finding some odd things happening with the search path that is being used to find installed libraries. Unfortunately there isn’t a very good solution at this time but I’m sharing what I’ve found in case it helps someone else.

Apple includes Python 2.7 with macOS and it is used by the system for various things. The general wisdom is to not mess with the system-provided install (and whatever you do don’t remove it unless you like reinstalling the operating system). Because I need to install other libraries that aren’t provided with the operating system I’ve installed the Python 2.7 distribution (at this writing version 2.7.12) from python.org and use that for my development work. That is where the problems start to show up.

There are two ways to install new libraries (called packages) using the pip install tool:
1. The global (global for the Python install) site-packages folder.
2. The user’s own site-packages folder.

Since I’m the only user on my system I prefer to just put everything I need into the global site-packages folder since in my case it is equivalent to the user-specific one.

When the Python interpreter starts up it, through various means, creates a search path that it uses to find other packages that you’ve installed when you try to use them with the “import” statement. You can see this yourself by opening up a Python shell and typing:
1. import sys
2. sys.path

You should see something like:

Python Path
Shows the path Python uses for finding libraries after importing sys and typing sys.path

Notice the paths with prefixed with “/System”? There is our problem. Through the standard path extension system our supposed-to-be-standalone Python environment is now seeing one Apple-provided library we might not be interested in using (PyObjC) and a folder where other libraries might be installed.1

The thing to note here is that those paths are at the end of the full sys.path so they will be searched last. If there is a version of the same library (PyObjC) for instance in one of those earlier paths it will be used instead of the system one. So theoretically I shouldn’t have to do anything but I use PyCharm from JetBrains for my Python development and I don’t like seeing this in my projects:
PyCharm package display

This can be fixed however and here is how to do it:
1. Ignore the problem.
If I wanted to do that I wouldn’t be writing this article.
2. Modify a file called site.py that is responsible for generating the Python sys.path
3. Update the file Apple put in place to add those folders to sys.path.

Let’s look at #2 first because that’s the solution I’ve chosen. site.py is installed here by default:

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7

and can be easily edited. The key is to filter out any lines that start with “/System” once the path has been built. sys.path is just a list and can be easily modified by any Python script. I’ve created a gist here with the code. The change was made at line 548 to remove the system paths from sys.path.

One thing to note about doing this is that this file can be overwritten by another Python install and these changes might need to be made again.

What about #3 above? I don’t recommend doing it because it’s an Apple provided file but it’s an easy change to make as well. The file itself is called Extras.pth and just contains two lines: the two paths we want to remove. It lives here:

/Library/Python/2.7/site-packages

The only thing required is to comment out those two lines.

I hope at some point there is a better way to address this problem but for now there is an easy solution for it. If you’re not like me and don’t care about having the system-provided stuff viewable by your other Python install none of this is an issue. As noted earlier if you have the same version of a library in one of the folders that’s earlier in sys.path it will be used instead.

Here is a list of some other reading on the subject I came across while looking into this issue. Some very interesting background information here.

  1. https://mail.python.org/pipermail/pythonmac-sig/2016-September/thread.html#24152
  2. http://stackoverflow.com/questions/14881205/what-is-the-difference-between-pythons-extras-and-site-packages-directories
  3. https://github.com/pypa/pip/issues/2468
  4. http://stackoverflow.com/questions/4271494/what-sets-up-sys-path-with-python-and-when

  1. I have to assume that Apple did this to enable people to add packages to the system-provided Python without breaking anything but I don’t like that it’s there polluting my own Python install. 

4 thoughts on “Python 2.7 on the Mac: site-package weirdness

  1. Hi, Just FYI (I’m on the pythonmac-sig thread that you point to above): I originally chose option #3. I renamed extras.pth so it would be ignored. Unfortunately, when you do so, any app that uses the pyobjc framework modules (in my case it was TextMate’s LaTeX bundle) fails. I suspect that your option #2 solution will have a similar problem, but I haven’t thought it through (nor actually tried it).

    Like

    1. Hey Andrew….yeah I think I tried sending you an email about this after I found the thread on the python sig but not sure it made it through. I didn’t do #3 for exactly that type of reason since it fools with Apple’s stuff. #2 I think should avoid that type of problem because it’s not the system Python install. I assume it would be similar if you were pointing everything at that Python vs. the Apple-provided one but then again you could then use pip to install stuff like PyObjC to site-packages for that Python install.

      Like

  2. Don’t use system python for your projects; make a virtualenv. Once you’ve done this, you won’t be inheriting system baggage either from Python.org or from macOS.

    Like

    1. Thanks for the reply but as I explained I am not using the system Python install, I’m using a fresh install of python.org which doesn’t have any baggage. I probably should have discussed using virtualenv as well but for various reasons I don’t want to do that so never thought about writing about it.

      Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: