Multiprocessing code fails when using a onefile executable on win32

[jkp]

I've had a problem for a long-while in some code that uses the multiprocessing module to co-oridinate multiple subprocesses. The code works fine without PyInstaller? involved, but when I package it as a PyInstaller? onefile executable I find that all kinds of errors are thrown when using cross-process constructs such as semaphores etc.

I spent some time tracking down the issue today have boiled it down to the following:

• I've implemented a simple test script that when packaged as a onefile executable illustrates the problem.
• The issue seems to be solveable by commenting out the line in launch.c that resets the _MEIPASS2 environment variable before running the packaged scripts. With this environment variable present PyInstaller? does not try to extract things again for subprocesses since this is already done (which seems like the correct behaviour).

The offending line was originally added here: http://www.pyinstaller.org/changeset/571. I'm not sure what the reasoning behind this existing is but it certainly breaks multiprocessing. Is there a better solution that I don't know of? Why is this line required?

I've attached the testcase code so that someone else can reproduce the issue and perhaps suggest what should be done.

http://www.pyinstaller.org/changeset/571

[jkp]

Test case that illustrates the issue

[matysek]

Thanks for the report. When this is fixed we should create a buildtest from the attached script.

[matysek]

Does it work when using the function multiprocessing.freeze_support()?

[matysek]

Another multiprocess example

[matysek]

This issue is still present.

We should find a way how to allow running pyinstaller apps by pyinstaller apps and at the same time not breaking multiprocess module.

[htgoebel]

This problem should only occur in Windows, where the process is re-started for each Process. On other systems, fork() is used.

_MEIPASS2 is used as an indicator whether the files have already been unpacked.

r571 seams to solve an issue occurring when running the program again for cases where it has to be unpacked again. (I'm unsure about which cases this really requires, but this is another issue).

I'm afraid, this can not be solved within PyInstaller?, but has to be solved in multiprocessing. Any try to solve it in PyInstaller? would in fact undo r571.

multiprocess need to set _MEIPASS2 before spawning and needs to unset just after.

[htgoebel]

test_multiprocess.py with an idea of an work-around

[matysek]

workaround that should work on winxp

[htgoebel]

Martin solved this issue, resolution is documented in http://www.pyinstaller.org/wiki/Recipe/Multiprocessing

[matysek]

Reopening since there is still an issue. The main process is not waiting for child (multiprocess) processes.

[htgoebel]

Solution is found: It requieres more code in main, see Recipe/Multiprocessing.

http://www.pyinstaller.org/wiki/Recipe/Multiprocessing

But still some work to be done:

Getting the value from PYTHONPATH will fail if sys.path was changed in the meantime. I propose changing the loader to store the value somewhere else. E.e. in sys._MEIPASS.

[matysek]

We could unset the _MEIPASS during bootstrap process instead doing directly in bootloader. Then when unsetting _MEIPASS we could set sys._MEIPASS afterwards.

[htgoebel]

Replying to matysek:

We could unset the _MEIPASS during bootstrap process instead doing directly in bootloader. Then when unsetting _MEIPASS we could set sys._MEIPASS afterwards.

Good idea. Moving as much logic into Python code as possible eases maintenance.

[matysek]

sys._MEIPASS is implemented and reseted in python code.

[htgoebel]

see r1680 and r1683