cosmicaug

Scenario:

The scenario involves the installation of different, coexisting versions of Python under Windows (in my case, it's Windows 8.1 but it should apply to other Windows versions). The versions of Python I had installed were 2.7.6 and 3.3.4 (but the issue described will occur with any combination of 2.x versions with 3.3.x or higher). The 2.x Python interpreter was installed first and this was followed by the installation of the 3.x Python interpreter. The path was set to include the directory containing the Python 3.x executable so as to allow the command, "Python", to invoke the Python3.x interpreter from any arbitrary directory.

With the above described setup, a 'Hello World' introductory exercise and some variations was tried. Unexpected results followed.

Observed behavior:

The initial 'Hello World!' program does nothing but illustrate a very simple use of the Python print function. Additional variations were tried illustrating slightly more complex uses of the print function.

Namely, version 1 of Hello World is as follows:

print("Hello, world!")

And it produces the following result:

C:\Users\Username\Documents\Python>helloworld.py Hello, world!

As expected, invoking the python interpreter directly and passing the program name as an argument, produces the same output:

C:\Users\Username\Documents\Python>python helloworld.py Hello, world!

Whereas version 2 of Hello World is:

print("Hello,",end='')

print(" world!")

It produces the following result when executed:

C:\Users\Username\Documents\Python>helloworld.py File "C:\Users\Username\Documents\Python\helloworld.py", line 1 print("Hello,",end='') ^ SyntaxError: invalid syntax

That is, a syntax error is produced and the interpreter is telling us that it was produced when it reached the '=' following the 'end' keyword.

And yet, when invoking the python interpreter directly and passing the script name as an argument, the output is exactly as expected:

C:\Users\Username\Documents\Python>python helloworld.py
Hello, world!

Interpretation:

All information about this issue points to the error originating in the 2.x Python interpreter being invoked by Windows against our expectation that the 3.x interpreter be invoked instead.

This can be easily confirmed (your results will differ depending on your installed versions). A script named 'what.py' was created as follows:

import sys
print(sys.version.split()[0]) # first part of string

Running the script directly gives:

C:\Users\Username\Documents\Python>what.py
2.7.6

Whereas, invoking Python with the script as an argument gives:

C:\Users\Username\Documents\Python>python what.py
3.3.4

So this absolutely confirms what we suspected. Some of the time, when we were intending to run the 3.x interpreter we were running the 2.x interpreter instead and that led to an error due to the differences in syntax between the two.

However, it is well known that one of the differences between the 2.x Python syntax and the 3.x Python syntax is that Python 2.x has a print statement and not a print function and Python 3.x has a print function and not a print statement. How is it possible that, without loading any special modules, the Python 2.x interpreter was able to correctly interpret the Python 3.x print function? For that matter, how come whatever black magic that appears to have allowed the Python 2.x interpreter to process a Python 3.x function did not also allow it to correctly process the 'end' keyword?

Explanation:

Why does the print() function form work at all in Python 2.x?

The explanation behind how the Python 2.x interpreter is able to process the Python 3.x print function is quite simple: it does not. This is addressed by the answers to the question "Using print() in Python2.x" at Stackoverflow.

Namely, when the Python 2.x reads something of the form «print("Hello, world!")», it processes it as print statement with the contents of the parenthesis as its argument rather than as a print function. As alluded by in the Stackoverflow thread, it is being treated exactly as the right side of a variable assignment expression would be treated. To see this, you may choose to create the following script:

a="Hello, nurse!"

b=("Hello, nurse!")

c=("Hello, nurse!",)

d=("Hello,","nurse!")

print a

print b

print c

print d

When run by the 2.x interpreter, we get the following output:

C:\Users\Username\Documents\Python>hellonurse.py
Hello, nurse!
Hello, nurse!
('Hello, nurse!',)
('Hello,', 'nurse!')

The first and second results show two instances where the right side of the assignment is evaluated as the same whether or not parenthesis are used. The third and fourth results show the right side of the assignment being interpreted as Python tuples.

Of course, attempting to use the 3.x interpreter on the previous script simply results in a syntax error because Python 3.x only uses the print function and does not know how to interpret "print a".

Why are we seeing both the Python 3.x and 2.x intepreters being used when we only expect the 3.x interpreter to be in use by default?

The short answer is that it's a "feature", not a bug.

A potential command will have an executable extension ('.com', '.exe', '.bat' or '.cmd' –I might be forgetting some) or it will have no extension. What happens, if what was entered did not specify a path, when you type a potential command into the command line is as follows:

The operating system will check to see if it is an internal command (like the command, "type") and, if so, execute it.
The operating system will check for an executable file matching the entered command within the current directory. If the command was entered without an extension, the operating system will look for files matching the different executable extensions. When such is found, it will be executed.
The operating system will traverse the list of directories stored in the %PATH% environment variable and check for an executable file matching the entered command within each directory. If the command was entered without an extension, the operating system will look for files matching the different executable extensions. When such is found, it will be executed.

If the file name entered has an extension which is not one of the operating system executable types, the operating system will check to see if the extension has a file association telling it that it is meant to be opened by a particular executable. As with a directly executable command, the operating system will look for the file in the current directory first and then within the directories listed by the %PATH% environment variable. When a match is found, the program corresponding to that file association will run using the found file name as its argument.

What this means is that the normal way to ensure that (no matter what the current directory) when a command is typed it will be executed is to add the directory which contains its executable to the %PATH% environmental variable. What this also means is that the normal way to ensure that a file corresponding to a given file type will be opened by the appropriate program is by registering the file type in the operating system so that the appropriate executable will be used to open it.

As such, a Python user might be inclined to think that the directory containing the Python executable which one desires to use should be included in the %PATH% environment variable. A python user in Windows, might also be inclined to think that the Python installation program would associate the .py extension with the latest installed Python executable.

And here lies the source of our issue. The Python installer, for what are allegedly good reasons, violates our assumptions.

As of Python 3.3, instead of creating file a file association for the '.py' extension to our preferred executable, the Python installer creates associations to two programs, "py.exe" and "pyw.exe" (the former for command prompt invocations and the latter for GUI invocations) which reside in the %SystemRoot% directory (usually, "C:\WINDOWS") and which serve as launchers for the actual Python interpreter executables. This is done so that with more than one installation coexisting one may have the flexibility to easily call one or the other by various means. The way this is achieved is by command line switches in the case of directly using "py" as a command, or by means of a script file shebang line argument in the case of attempting to execute a file with a '.py' extension.

The way the launcher violates our assumptions is that it is configured, by default, to launch the most recent 2.x version of the Python interpreter (rather than either the latest installed version or the latest version altogether). Given that so much of the Python code out there is 2.x code (including system functions in non-Windows operating systems), this is done under the assumption that 2.x should be the default to ensure maximum compatibility and to minimize the chances of breaking important scripts. In other words, because we have installed the 3.x version last and because we may have placed the 3.x executable (but not the 2.x executable) in our path we may be lead to believe that our default Python interpreter is 3.x but this is not the case. We are further confused, if we have put the 3.x executable in our path, by the fact that "python" does, indeed, invoke the 3.x interpreter. In other words, our "default" interpreter may be a different executable depending on how we invoke Python (if directly by typing "python" or if indirectly by typing a script file name and relying on the Windows file associations system).

Corrective actions:

Including a python binary directories in the path is no longer recommended as of Python 3.3. Direct invocation of the executables is now supposed to be done through the Windows Python Launcher by use of the "py" command rather than by typing "python". Command line switches for this command can be used to select the Python interpreter executable to be invoked (including choice of 64 or 32 bit interpreters).

Shebang line arguments can be placed in the script file itself to specify which interpreter is to be used. Thus, to ensure that the latest 3.x executable is used, the original example which returned an error under Python 2.x could be changed to the following:

#! python3
print("Hello,",end='')

print(" world!")

Alternatively, if you are certain that you have no need to have Python 2.x set up as a default interpreter (which I am guessing is probably the case with many Windows installations), you can change the default interpreter to be invoked by the Windows Python Launcher. This can be accomplished two ways:

By means of a settings file.

This file is named "py.ini" and can govern the launcher's default behavior on a system wide basis or on a per user basis.
In the former case, it is to be found in the same directory as the launcher executables.
In the latter case, it is to be found in the %LOCALAPPDATA% directory. The defaults set here will override the defaults set in the application directory settings file.
As an example settings file, the following would set the default interpreter as the latest 3.x executable:
[defaults] python=3

By means of an environmental variable.

The variable is %PY_PYTHON%
If configured on startup, %PY_PYTHON% can govern the launcher's default behavior on a system wide basis or on a per user basis (depending on whether it is set as a global variable or as a user variable).
The content of %PY_PYTHON% will override the settings in the py.ini configuration files.

References:

Python 3.3.5 Documentation: Python Setup and Usage, 3.4. Python Launcher for Windows.

Retrieved from https://docs.python.org/3.3/using/windows.html#python-launcher-for-windows

The New Windows Launcher in Python 3.3 by Mark Lutz (author of Learning Python by O'Reilly).

Retrieved from http://www.rmi.net/~lutz/py33-windows-launcher.html

cosmicaug

Sunday, December 7, 2014

Google+ sharer bookmarklet with a scroll bar

Tuesday, July 8, 2014

How many social scientists does it take to change a light bulb?

Question: How many social scientists does it take to change a light bulb?

Answer: Social scientists do not change light bulbs.

Thursday, May 8, 2014

Windows and Python: On unexpected version invocation with multiple, coexisting Python installs that include version 3.3 or higher