Say I have a list of animals with their counts:

`import numpy as np import pandas as pd from random import randint table = np.zeros((5,1), dtype=int) for i in range(5): table[i]=randint(10, 20) df1 = pd.DataFrame(columns=['Animal', 'Count']) df1['Animal'] = animal_list df1['Count'] = table df1 `

And I have a matrix of how many times they appear together:

`table = np.zeros((5,5), dtype=int) animal_list = ['Monkey', 'Tiger', 'Cat', 'Dog', 'Lion'] for i in range(5): for j in range(5): table[i][j]=randint(0, 9) df2 = pd.DataFrame(table, columns=animal_list, index=animal_list) df2 `

I want to find the animals’ association strength, which is defined like so – if `Lion`

and `Cat`

appear together 5 times, and `Lion`

‘s count is 10 and `Cat`

‘s count is 15, then `Lion -> Cat`

association strength is `5/10=0.5`

, and `Cat -> Lion`

association strength is `5/15=0.33`

.

I do it like so:

`assoc_df = pd.DataFrame(columns=['Animal 1', 'Animal 2', 'Association Strength']) for row_word in df2: for col_word in df2: if row_word!=col_word: assoc_df = assoc_df.append({'Animal 1': row_word, 'Animal 2': col_word, 'Association Strength': df2[col_word][row_word]/df1[df1.Animal==row_word]['Count'].values[0]}, ignore_index=True) assoc_df `

The problem is, since there are 2 for loops, the complexity is `O(n^2)`

. This is a problem when (in my real dataset) I have ~1000 animals to loop on, which takes hours to finish computing the association strength table.

So, how to do I better optimize the creation/generation process of this association table?

**P.S.:** In most practical use cases, `df2`

is a symmetric matrix, as “`X`

appears together with `Y`

” generally also means the same as “`Y`

appears together with `X`

. So, I am ok with solutions that assume that `df2`

is symmetric, and cut down the running time by half. In the above example, `df2`

is not a symmetric matrix, which is applicable for situations where we want to express meanings such as “`X`

appears after `Y`

” and “`Y`

appears after `X`

“, which are not the same.