close Comments you submit will be routed for moderation. If you have an account, please log in first.
Modify

Opened 2 years ago

Last modified 8 months ago

#502 assigned defect

support for namespace packages

Reported by: anonymous Owned by: htgoebel
Priority: high Milestone: PyInstaller 3.0
Component: PyInstaller Version: 2.0
Severity: major Keywords:
Cc:

Description (last modified by htgoebel)

PyInstaller fails to package zope.interface.

The workaround is to put an empty __init__.py in the zope/ directory.

Attachments (0)

Change History (5)

comment:1 Changed 2 years ago by htgoebel

This means we should fake zope/__init__.pyc if i is missing.

comment:2 Changed 2 years ago by htgoebel

  • Description modified (diff)

comment:3 Changed 2 years ago by matysek

For zope.interface there exists file

/usr/lib64/python2.7/site-packages/zope.interface-3.6.1-py2.7-nspkg.pth

This file seems to get processed by python when importing zope. This file extends the PYTHONPATH. By default .pth files contain only paths for extending PYTHONPATH. But zope.interface is exception.

I think the proper implementation would be:

  1. Look for .pth files site-package directory before analyzing imports in python code.
  2. If .pth contains a python code (like zope) then process this code so the PYTHONPATH could be properly initialized.
  3. And maybe we should fake init.pyc in the final binary, created by pyinstaller.

Links:

comment:4 Changed 8 months ago by htgoebel

  • Milestone set to PyInstaller 2.1
  • Owner changed from matysek to htgoebel
  • Priority changed from normal to high
  • Severity changed from normal to major
  • Status changed from new to assigned
  • Summary changed from PyInstaller fails to package zope.interface; workaround is adding __init__.py to support for namespace packages
  • Version changed from develop to 2.0

Basically there are two types (and some sub-types) of namepace-packages:

  1. without an __init__.py file
    1. Just a directory, not even a .pth-file. This is the future direction, implemented in Python 3.3, see PEP 420
    2. There is a xxx-nspkg.pth file. This file is read on Python startup (like every .pth-file in site-packages), instantiates an empty module (using types.ModuleType) and add it to sys.modules.
  2. with a an __init__.py file
    1. using pkg_resources.declare_namespace()
    2. using pkgutil.()extend_path

Different ways of installation

  1. If setuptools are used, setup.py install just runs easy_install on the current directory. So the same as below apllies. In this case, if pkg_resources.declare_namespace() is not used in the __init__-file, setuptools will issue a warning.
  1. If plain distutils (w/o setuputils) are not used, setup.py install just copies the lib to the destination directory (typically site-packages) -- overwriting any existing namespace-__init__-file. The developer himself has to take care about all namespace-packages are on sys.path.
  1. pip is calling setuptool's command install with --single-version-externally-managed, see pip.req.Requirement.install(), which of course calls install_lib and install_egg_info.
  1. easy_install is not using --single-version-externally-managed, so the __init__.py-file is kept, see setuptools.commands.easy_install -> build_and_install(). It is calling bdist_egg and then installing the build egg. bdist_egg, is running 'egg_info' and install_lib, but not install (see setuptools.commands.bdist_egg -> run()). Thus the __init__.py-files are kept and no -nspkg.pth-file is generated.

Conclusion:

  1. If setuptools, easy_install or pip are used, there is always an egg-info which we can ask for the namespace.
  2. The only case this does not match is if pkgutil.extend_path() and plain distutils (w/o setuptools) are used for installation.

Links:

Implementation strategy

For 1: There is no need for reading and scanning the nspkg.pth file: The file is processed by site.py and the namespace-package is instantiates automatically. Try
python -c 'import zope ; print zope'

  • Append a NamespaceImportDirector to ImportTracker.metapath, so it gets called after all other import directors.
  • The NamespaceImportDirector test whether the module is already in sys.modules, but imp.find_module fails. In this case, assume a namespace-package and create a NamespaceModule.
  • Add an empty module for faking __init__-modules for namespace packages.
  • Add module-type NamespaceModule, faking the existence of the namespace- package by including the empty fake module.

This has been implemented in my git-branch at https://github.com/htgoebel/pyinstaller/tree/namespace-modules

For 2: For each __init__.py file, check if it is a namespace-package. Is so, replace it by a NamespaceModule.

Will not be implemented: Support for pain distutils and pkgutil.extend_path().

Rationale: extend_path also reads a .pkg-file in the package-directory (it it exists), which extends sys.path. So PyInstaller would need to detect if pkgutil.extend_path() is used in the __init__-file, or if pkgutil is used for other stuff. This would be hard if not impossible to decide without executing the __init__-file. As this method is deprecated anyway (see PEP 420) and there wo onone asking for such a feature up to now, I decide to not implement it.

Last edited 8 months ago by htgoebel (previous) (diff)

comment:5 Changed 8 months ago by htgoebel

  • Milestone changed from PyInstaller 2.1 to PyInstaller 3.0

Pushing to milestone PyInstaller? 3.0. Implementing the second way is complex. Since modulegraph says to support it, wait solving issue #439 should give it for free.

Add Comment

Modify Ticket

Action
as assigned .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.