Python Panda Append Dataframe In Loop

I am trying to append many data frames into one empty data frame but It is not working. For this, I am using this tutorial my code is like this: I am generating a frame inside a lo

Solution 1:

As I mentioned in my comment, appending to pandas dataframes is not considered a very good approach. Instead, I suggest that you use something more appropriate to store the data, such as a file or a database if you want scalability.

Then you can use pandas for what it's built, i.e. data analysis by just reading the contents of the database or the file into a dataframe.

Now, if you really want to stick with this approach, I suggest either join or concat to grow your dataframe as you get more data

[EDIT]

Example (from one of my scripts):

results = pd.DataFrame()
for result_file in result_files:
    df = parse_results(result_file)
    results = pd.concat([results, df], axis=0).reset_index(drop=True)

parse_results is a function that takes a filename and returns a dataframe formatted in the right way, up to you to make it fit your needs.

Solution 2:

As the comments stated, your original error is that you didn't assign the df.append call to a variable - it returns the appended (new) DataFrame.

For anyone else looking to "extend" your DataFrame in-place (without an intermediate DB, List or Dictionary), here is a hint showing how to do this simply:

Pandas adding rows to df in loop

Basically, start with your empty DataFrame, already setup with the correct columns,

then use df.loc[ ] indexing to assign the new Row of data to the end of the dataframe, where len(df) will point just past the end of the DataFrame. It looks like this:

   df.loc[  len(df)  ] = ["my", "new", "data", "row"]

More detail in the linked hint.