Serialize A Group Of Integers Using Cython

I saw this sample code from the Pyrobuf page for serializing an integer ~3 times faster than via struct.pack: def ser2(): cdef int x = 42 return (&x)[:s

Solution 1:

I assume you want to speed up the following (Python3):

importstruct 
lst=list(range(100))  #any other size
struct.pack('i'*len(lst), *lst)

without struct and cython you could achieve it as following in python:

import array
bytes(array.array('i', lst))

this is however somewhat slower than the struct-module:

>>>%timeit struct.pack('i'*len(lst), *lst)
2.38 µs ± 9.48 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
>>>%timeit bytes(array.array('i',lst))
3.94 µs ± 92 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

However, cython can be used to speed up the creation of the array, for the documentation see here(arrays) and here(str/bytes):

 %%cython
 import arrayfrom cpython cimport array
 def ser_int_list(lst):
    cdef Py_ssize_t n=len(lst)
    cdef array.array res=array.array('i')
    array.resize(res, n)  #preallocate memoryfor i in range(n):
       res.data.as_ints[i]=lst[i]            #lst.__get__() needs Python-Integer, so let i be a python-integer (not cdef)return res.data.as_chars[:n*sizeof(int)] # str for python2, bytes for python3

The timings show the following performance:

#list_size    struct-code    cython-code    speed-up
     1343 ns         238 ns        1.510619 ns         283 ns         21002.38 µs        2.38 µs        3.5100021.6 µs        5.11 µs         410000266 µs        47.5 µs        5.5

i.e. cython provides some speed-up, from 1.5 for small list up to 5.5 for large lists.

Probably this could be tweaked even further, but I hope you get the idea.

---

Testing code:

import struct
for n in [1, 10,10**2, 10**3, 10**4]:
    print ("N=",n)
    lst=list(range(n))
    print("struct:")
    %timeit struct.pack('i'*len(lst), *lst)
    print("cython:")
    %timeit ser_int_list(lst)