Learnings from python2 to python3 migration

Recently I have worked on couple of projects to migrate from python2 to python3. In this article I will discuss some of the learnings during that process.

1. Make code compatible with python2 and python3

The first step is to make code compatible with python2 and python3. We used futurize tool to achieve this. The benefits of making the code compatible is that we can keep merging our changes without causing regression. This is important in projects where many developers are contributing frequently. The other benefit is that we can switch between python2 and python3 seamlessly. Furturize tool has two stages and each stage has many fixtures. Each fixture is responsible for making certain aspect of code compatible with python2 and python3 .e.g lib2to3.fixes.fix_dict fixture is responsible for making changes in dictionary data structure usage in code to make it compatible with python2 and python3. More details can be found here https://python-future.org/futurize.html .

2. Make incremental futurize fixture changes

Futurize tool allow us to apply the fixtures independently. After making the changes run the integration and unit tests in both python2 and python3 to make sure everything is fine. If we apply all the fixtures at the same time, it may result in changes in many files and this may make it difficult to debug any issue that is caused by these changes.

3. String and number comparison

One of the surprise finding during migration was that python2 allows comparison between string and integer but python3 won’t. Futurize does not take care of this change we need to run integration tests and manual tests to catch such scenarios.

~$ python
Python 2.7.17 (default, Apr 15 2020, 17:20:14) 
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s ='10'
>>> s > 0
True
~$ python3
Python 3.6.9 (default, Jul 17 2020, 12:50:27) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> s='10'
>>> s > 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '>' not supported between instances of 'str' and 'int'

4. String handling

This is a major change between python2 and python3. In python2 “str” type represents both text and “bytes” or bytes string where as in python3 “str” type represents “unicode” type. This poses a challenge if we are using packages that has methods which either accepts parameter of type “bytes” and returns of type “bytes”. In python2 we can apply string slice operation on the return values but in python3 we can not apply string operations on object of type “bytes”. Following is an example of how we handled request in a TCP and UDP server

class UDPRequestHandler(socketserver.BaseRequestHandler):
    """Process a UDP request."""
    def handle(self):
        try:
            data = self.request[0].strip()
            if PY3:
                data = data.decode('utf-8', 'ignore')

class ThreadedTCPRequestHandler(socketserver.BaseRequestHandler):
    """Handle incoming TCP connection"""
    def handle(self):
        data = self.request.recv(4, socket.MSG_PEEK).strip()
        if PY3:
            data = data.decode('utf-8')

        if data[0:3] == 'GET' or data[0:4] == 'POST':
            CmdServerHTTPRequestHandler(self.request, self.client_address, self.server)
        else:
            CmdServerTelnetRequestHandler(self.request, self.client_address, self.server)

Please note that in case of python3 we have to explicitly change bytes to string before applying string operations. We need to import PY3 or PY2.

from future.utils import PY2
// or
from future.utils import PY3

5. Refactor third party library usages

If you are using third party libraries, check their documentation of how they have changed their API to handle bytes and unicode. In our project we are using msgpack library. Also We had to upgrade several third party libraries as they were not compatible with py3. We had to revisit libraries like Kafka, Thrift etc.

else:
    if PY2:
        services = msgpack.unpackb(packed_services)
    else:
        services = msgpack.unpackb(packed_services, raw=False)

6. Guard your changes with PY2 or PY3 flag

During the migration if we need to make changes that are not supported by futurize tool, use PY2 or PY3 flag to make the change python2 and python3 compatible.

7. String type check

In python2 str and unicode extends from basestring but in python3 there is no basestring and unicode type as the str represents unicode. We used following code to check type.

from builtins import str
from past.builtins import basestring
string_types = basestring if PY2 else str
...
if isinstance(graphs, string_types)):
    // do something

8. Unit test always helps

The unit tests were helpful in verifying that code was working with both python2 and python3. If possible add as many unit tests as possible before migration starts.