The NumPy array

In short: A NumPy array (ndarray) holds numbers of one type (dtype) with a shape, and computes with them much faster than a Python list.

Problem

“Add VAT to every price”

The shop has 25 products. The manager wants the prices with VAT: every price times 1.2. Then they want a year of sales by day: 25 products over 365 days, that is 9,125 numbers.

So far numbers in Python have lived in a list. But a list does not do maths:

Pythondata: the shop tables

prices = [1.2, 1.9, 0.8]
print(prices * 2)
prices * 1.2

prices = [1.2, 1.9, 0.8] print(prices * 2) prices * 1.2

▶ RunRestore the code

* 2 does not double the prices, it repeats the list twice. * 1.2 is an error. What is left is a for loop for every calculation, and with 9,125 numbers, then a million, the code gets long and slow.

The idea

An array: one block of numbers

A list is a collection of separate objects. Each of its slots holds an address, and the number itself is elsewhere in memory, with its own Python “wrapper”. That is why a list can hold anything: a number, text, another list.

Data analysis does not need that. It needs many numbers of the same type. NumPy is the library for this, and its core is the array: the numbers sit next to each other in one block of memory, all of one type. Here is a million prices as a list and as an array:

Pythondata: the shop tables

import sys
import numpy as np

# a million different prices: 0.00, 0.01, 0.02 …
prices = [i / 100 for i in range(1_000_000)]
list_bytes = sys.getsizeof(prices) + sum(sys.getsizeof(p) for p in prices)
array = np.array(prices)
print('list: ', round(list_bytes / 1_000_000, 1), 'MB')
print('array:', round(array.nbytes / 1_000_000, 1), 'MB')

import sys import numpy as np # a million different prices: 0.00, 0.01, 0.02 … prices = [i / 100 for i in range(1_000_000)] list_bytes = sys.getsizeof(prices) + sum(sys.getsizeof(p) for p in prices) array = np.array(prices) print('list: ', round(list_bytes / 1_000_000, 1), 'MB') print('array:', round(array.nbytes / 1_000_000, 1), 'MB')

▶ RunRestore the code

The list takes 20.2 MB and the array 8.0 MB: 8 bytes per price and nothing more. The sandbox is 32-bit. On an ordinary 64-bit computer the addresses and wrappers are bigger and the list grows to about 32 MB, while the array stays at 8 MB.

A block of same-typed numbers is also faster to work through: NumPy runs over it with ready-made C code, without asking for the type of every number. Here is VAT on a million prices:

Pythondata: the shop tables

import time
import numpy as np

prices = [i / 100 for i in range(1_000_000)]
array = np.array(prices)

start = time.perf_counter()
with_vat = [p * 1.2 for p in prices]
list_time = time.perf_counter() - start

start = time.perf_counter()
array_vat = array * 1.2
array_time = time.perf_counter() - start

print('the array is faster:', array_time < list_time)

import time import numpy as np prices = [i / 100 for i in range(1_000_000)] array = np.array(prices) start = time.perf_counter() with_vat = [p * 1.2 for p in prices] list_time = time.perf_counter() - start start = time.perf_counter() array_vat = array * 1.2 array_time = time.perf_counter() - start print('the array is faster:', array_time < list_time)

▶ RunRestore the code

array * 1.2 multiplies all the prices at once. In the sandbox the array is several times faster than the loop. On an ordinary computer the gap is tens of times. The exact time depends on the machine, so the code only says which is faster.

More precisely, if you are curious

import numpy as np is everyone’s habit, like pd for pandas. pandas itself is built on NumPy: a table column keeps its numbers in an array. So what you learn here also holds for tables.

The idea

An array from the table: dtype

The prices are already in products.csv. to_numpy() takes a column of the table as an array. Every array knows the type of its numbers: that is the dtype.

Pythondata: the shop tables

import numpy as np
import pandas as pd

products = pd.read_csv('products.csv')
prices = products['price'].to_numpy()
stock = products['stock'].to_numpy()
print(prices[:5], prices.dtype)
print(stock[:5], stock.dtype)

import numpy as np import pandas as pd products = pd.read_csv('products.csv') prices = products['price'].to_numpy() stock = products['stock'].to_numpy() print(prices[:5], prices.dtype) print(stock[:5], stock.dtype)

▶ RunRestore the code

prices[:5] gives the first five, as with a list. An array prints without commas: [1.2 1.9 0.8 2.5 1.6].

• float64: decimal numbers, 8 bytes each (64 bits). The prices are like that.
• int64: whole numbers, also 8 bytes each. The stock is like that.

See it

When types mix

There is one type for the whole array. What happens if you put different things in it?

Pythondata: the shop tables

import numpy as np

print(repr(np.array([40, 25, 1.5])))
print(repr(np.array([1.2, 1.9, '0.8'])))

import numpy as np print(repr(np.array([40, 25, 1.5]))) print(repr(np.array([1.2, 1.9, '0.8'])))

▶ RunRestore the code

• Whole and decimal: all become decimal. 40 becomes 40.0 without loss, while the other way round would cut 1.5.
• One piece of text: everything becomes text. <U32 means a string of up to 32 characters. The numbers are no longer numbers.

1. What does np.array([1.2, 1.9, '0.8']) * 2 give?

An error: the array is text, and NumPy does not multiply text by a numberarray([2.4, 3.8, 1.6])array([2.4, 3.8, '0.80.8'])

Check

More precisely, if you are curious

If you write np.array([40, 25, 60]) in the sandbox, the type is int32, not int64. The default whole number follows the machine, and the sandbox is 32-bit. On your computer it is int64. Columns read with pandas are int64 here too. For quantities in a shop both types are enough: int32 holds numbers up to about 2.1 billion.

The idea

Shape: shape and ndim

The prices are one row of numbers. Sales by week are a table: a row for every product, a column for every week. An array can be both.

• shape is the shape: how many numbers there are in each direction.
• ndim is the number of directions (dimensions).

Pythondata: the shop tables

import numpy as np
import pandas as pd

prices = pd.read_csv('products.csv')['price'].to_numpy()
print(prices.shape, prices.ndim)

# units of 3 products over 4 weeks: row = product, column = week
qty = np.array([
    [4, 2, 5, 3],
    [1, 0, 2, 2],
    [6, 7, 5, 8],
])
print(qty.shape, qty.ndim)
qty

import numpy as np import pandas as pd prices = pd.read_csv('products.csv')['price'].to_numpy() print(prices.shape, prices.ndim) # units of 3 products over 4 weeks: row = product, column = week qty = np.array([ [4, 2, 5, 3], [1, 0, 2, 2], [6, 7, 5, 8], ]) print(qty.shape, qty.ndim) qty

▶ RunRestore the code

The prices have the shape (25,): one dimension with 25 numbers. The comma is there because a shape is always a pair, a triple and so on (a tuple), even with one number in it. The table is (3, 4): 3 rows (the products) and 4 columns (the weeks). Rows first, then columns, as with shape in pandas.

Try it

Products and weeks

qty holds units per product (rows) and per week (columns). Put the number of products in n_products and the number of weeks in n_weeks, taken from the shape of the array.

Pythondata: the shop tables

import numpy as np

qty = np.array([
    [4, 2, 5, 3],
    [1, 0, 2, 2],
    [6, 7, 5, 8],
])
n_products = ...
n_weeks = ...

import numpy as np qty = np.array([ [4, 2, 5, 3], [1, 0, 2, 2], [6, 7, 5, 8], ]) n_products = ... n_weeks = ...

Check▶ Run

The idea

Ready-made arrays: arange and zeros

Often an array does not come from a file, you make it yourself:

• np.arange(start, end) gives consecutive numbers, like range. The end is not included.
• np.zeros(shape) gives an array of zeros. It is handy for a table you will fill in. The shape of a table is a pair: np.zeros((rows, columns)), with double brackets.

Pythondata: the shop tables

import numpy as np

days = np.arange(1, 366)
print(days[:3], days[-3:], days.shape)

sales = np.zeros((25, 365))
print(sales.shape, sales.dtype, sales.size)

import numpy as np days = np.arange(1, 366) print(days[:3], days[-3:], days.shape) sales = np.zeros((25, 365)) print(sales.shape, sales.dtype, sales.size)

▶ RunRestore the code

The days run from 1 to 365, so the end is 366. The table for a year of sales is (25, 365): 9,125 zeros of type float64, ready for the numbers.

Exercise

The price list and the year

Make days with the days from 1 to 365, and sales: a table of zeros with a row for each of the 25 products and a column for every day.

Pythondata: the shop tables

import numpy as np

days = ...
sales = ...

import numpy as np days = ... sales = ...

Check▶ Run

Put the prices and the stock into one array table with two rows: the first with the prices, the second with the stock. Put the type of table in kind.

Pythondata: the shop tables

import numpy as np
import pandas as pd

products = pd.read_csv('products.csv')
prices = products['price'].to_numpy()
stock = products['stock'].to_numpy()
table = ...
kind = ...

import numpy as np import pandas as pd products = pd.read_csv('products.csv') prices = products['price'].to_numpy() stock = products['stock'].to_numpy() table = ... kind = ...

Check▶ Run

Test

Check yourself

1. Why does an array of a million prices take less memory than a list of the same prices?

The array keeps the numbers themselves in one block, the list keeps addresses of separate objectsThe array rounds the pricesThe array keeps only the distinct prices

2. What is the dtype of np.array([40, 25, 1.5])?

float64Both: int64 for the first two and float64 for the lastint64

3. The sales of 25 products over 7 days are in an array with a row per product. What is sales.shape?

(25, 7)(7, 25)175

4. Which days does np.arange(1, 8) give?

1, 2, 3, 4, 5, 6, 7, 80, 1, 2, 3, 4, 5, 6, 71, 2, 3, 4, 5, 6, 7

Check

Start the lesson over

Related lessons

• Builds on: The first table
• Continues in: Vectorization · Summaries and missing values · Broadcasting: the number, the row and the column · Series and DataFrame
• Tasks on this topic: NumPy