Sliding Window Iterator Using Rolling In Pandas

If it's single row, I can get the iterator as following import pandas as pd import numpy as np a = np.zeros((100,40)) X = pd.DataFrame(a) for index, row in X.iterrows(): prin

Solution 1:

I'll experiment with the following dataframe.

Setup

import pandas as pd
import numpy as np
from string import uppercase

defgeneric_portfolio_df(start, end, freq, num_port, num_sec, seed=314):
    np.random.seed(seed)
    portfolios = pd.Index(['Portfolio {}'.format(i) for i in uppercase[:num_port]],
                          name='Portfolio')
    securities = ['s{:02d}'.format(i) for i inrange(num_sec)]
    dates = pd.date_range(start, end, freq=freq)
    return pd.DataFrame(np.random.rand(len(dates) * num_sec, num_port),
                        index=pd.MultiIndex.from_product([dates, securities],
                                                         names=['Date', 'Id']),
                        columns=portfolios
                       ).groupby(level=0).apply(lambda x: x / x.sum())    


df = generic_portfolio_df('2014-12-31', '2015-05-30', 'BM', 3, 5)

df.head(10)

I'll now introduce a function to roll a number of rows and concatenate into a single dataframe where I'll add a top level to the column index that indicates the location in the roll.

Solution Step-1

def rolled(df, n):
    k = range(df.columns.nlevels)
    _k = [i - len(k) for i in k]
    myroll = pd.concat([df.shift(i).stack(level=k) for i in range(n)],
                       axis=1, keys=range(n)).unstack(level=_k)
    return [(i, row.unstack(0)) for i, row in myroll.iterrows()]

Though its hidden in the function, myroll would look like this

Now we can use it just like an iterator.

Solution Step-2

fori, rollinrolled(df.head(5), 3):
    printrollprint012PortfolioPortfolioA0.326164NaNNaNPortfolioB0.201597NaNNaNPortfolioC0.085340NaNNaN012PortfolioPortfolioA0.2786140.326164NaNPortfolioB0.3144480.201597NaNPortfolioC0.2663920.085340NaN012PortfolioPortfolioA0.2589580.2786140.326164PortfolioB0.0892240.3144480.201597PortfolioC0.2935700.2663920.085340012PortfolioPortfolioA0.0927600.2589580.278614PortfolioB0.2625110.0892240.314448PortfolioC0.0842080.2935700.266392012PortfolioPortfolioA0.0435030.0927600.258958PortfolioB0.1322210.2625110.089224PortfolioC0.2704900.0842080.293570

Solution 2:

That's not how rolling works. It "provides rolling transformations" (from the docs).

You can loop and use pandas indexing?

for i inrange((X.shape[0] + 9) // 10):
    X_subset = X.iloc[i * 10: (i + 1) * 10])