Python Slots Performance
Python can’t just allocate a static amount of memory at object creation to store all the attributes. In 3 used 0.2305 MiB RAM in 0.12s, peaked 0.00 MiB above current, total RAM usage 15.80 MiB In 4: from slots import. In 4 used 9.3008 MiB RAM in 0.72s, peaked 0.00 MiB above current, total RAM usage 25.10 MiB In 5:%cat noslots.py. Python comes with the cProfile module to help evaluate performance. It not only gives the total running time, it also times each function separately. It then tells you how many times each function was called, which makes it easy to determine where you should make optimizations. Here's what a sample analysis by cProfile looks like.
This post describes the CPython implementation of the list object. CPython is the most used Python implementation.
Lists in Python are powerful and it is interesting to see how they are implemented internally.
Following is a simple Python script appending some integers to a list and printing them.
As you can see, lists are iterable.
List object C structure
A list object in CPython is represented by the following C structure. ob_item is a list of pointers to the list elements. allocated is the number of slots allocated in memory.
List initialization
Let’s look at what happens when we initialize an empty list. e.g. l = [].
It is important to notice the difference between allocated slots and the size of the list. The size of a list is the same as len(l). The number of allocated slots is what has been allocated in memory. Often, you will see that allocated can be greater than size. This is to avoid needing calling realloc each time a new elements is appended to the list. We will see more about that later.
Append
We append an integer to the list: l.append(1). What happens? The internal C function app1() is called:
Let’s look at list_resize(). It over-allocates memory to avoid calling list_resize too many time. The growth pattern of the list is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, …
Monty Python Slot Machine App
4 slots are now allocated to contain elements and the first one is the integer 1. You can see on the following diagram that l[0] points to the integer object that we just appended. The dashed squares represent the slots allocated but not used yet.
Append operation amortized complexity is O(1).
We continue by adding one more element: l.append(2). list_resize is called with n+1 = 2 but because the allocated size is 4, there is no need to allocate more memory. Same thing happens when we add 2 more integers: l.append(3), l.append(4). The following diagram shows what we have so far.
Insert
Let’s insert a new integer (5) at position 1: l.insert(1,5) and look at what happens internally. ins1() is called:
The dashed squares represent the slots allocated but not used yet. Here, 8 slots are allocated but the size or length of the list is only 5.
Insert operation complexity is O(n).
Pop
When you pop the last element: l.pop(), listpop() is called. list_resize is called inside listpop() and if the new size is less than half of the allocated size then the list is shrunk.
Pop operation complexity is O(1).
You can observe that slot 4 still points to the integer but the important thing is the size of the list which is now 4.
Let’s pop one more element. In list_resize(), size – 1 = 4 – 1 = 3 is less than half of the allocated slots so the list is shrunk to 6 slots and the new size of the list is now 3.
You can observe that slot 3 and 4 still point to some integers but the important thing is the size of the list which is now 3.
Remove
Python list object has a method to remove a specific element: l.remove(5). listremove() is called.
To slice the list and remove the element, list_ass_slice() is called and it is interesting to see how it works. Here, low offset is 1 and high offset is 2 as we are removing the element 5 at position 1.
Remove operation complexity is O(n).
That’s it for now. I hope you enjoyed the article. Please write a comment if you have any feedback. If you need help with a project written in Python or with building a new web service, I am available as a freelancer: LinkedIn profile. Follow me on Twitter @laurentluce.
Slot Machine In Python
In Python every class can have instance attributes. By default Pythonuses a dict to store an object’s instance attributes. This is reallyhelpful as it allows setting arbitrary new attributes at runtime.
However, for small classes with known attributes it might be abottleneck. The dict wastes a lot of RAM. Python can’t just allocatea static amount of memory at object creation to store all theattributes. Therefore it sucks a lot of RAM if you create a lot ofobjects (I am talking in thousands and millions). Still there is a wayto circumvent this issue. It involves the usage of __slots__ totell Python not to use a dict, and only allocate space for a fixed setof attributes. Here is an example with and without __slots__:
Without__slots__:
With__slots__:
The second piece of code will reduce the burden on your RAM. Some peoplehave seen almost 40 to 50% reduction in RAM usage by using thistechnique.
On a sidenote, you might want to give PyPy a try. It does all of theseoptimizations by default.
Below you can see an example showing exact memory usage with and without __slots__ done in IPython thanks to https://github.com/ianozsvald/ipython_memory_usage