Unicodedecodeerror When Using Python 2.7 Code On Python 3.7 With Cpickle

I am trying to use cPickle on a .pkl file constructed from a 'parsed' .csv file. The parsing is undertaken using a pre-constructed python toolbox, which has recently been ported to

Solution 1:

The code is opening the file in text mode ('r'), but it should be binary mode ('rb').

From the documentation for pickle.load (emphasis mine):

[The] file can be an on-disk file opened for binary reading, an io.BytesIO object, or any other custom object that meets this interface.

Since the file is being opened in binary mode there is no need to provide an encoding argument to open. It may be necessary to provide an encoding argument to pickle.load. From the same documentation:

Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2. If fix_imports is true, pickle will try to map the old Python 2 names to the new names used in Python 3. The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The encoding can be ‘bytes’ to read these 8-bit string instances as bytes objects. Using encoding='latin1' is required for unpickling NumPy arrays and instances of datetime, date and time pickled by Python 2.

This ought to prevent the UnicodeDecodeError:

sm_database = cPickle.load(open("C:/Python37/TestX10/metadatafile.pkl","rb"))