I am trying to iterate over dataframe which is from Indicators.csv from this kaggle link to collect random countries , year and random indicator which are common with respect to each other using Pandas but using while loops is taking a lot of time
I know itertuples() is better way and saves a lot of time but don't know how to implement it
Also I came to know vectorization should be used to loop over large datasets, if someone explain how to use that it will be appreciated.
data = pd.read_csv('Indicators.csv')
countries = data['CountryName'].unique().tolist()
indicators = data['IndicatorName'].unique().tolist()
yearsFilter = [2010, 2011, 2012, 2013, 2014]
filteredData1 = []
filteredData2 = []
while(len(filteredData1) < len(yearsFilter)-1):
# pick new indicator
indicatorsFilter = random.sample(indicators, 1)
countryFilter = random.sample(countries, 2)
# how many rows are there that have this country name, this indicator, and this year. Mesh gives bool vector
filterMesh = (data['CountryName'] == countryFilter[0]) & (data['IndicatorName'].isin(indicatorsFilter)) & (data['Year'].isin(yearsFilter))
# which rows have this condition to be true?
filteredData1 = data.loc[filterMesh]
filteredData1 = filteredData1[['CountryName','IndicatorName','Year','Value']]
# need to print this only when our while condition is true
if(len(filteredData1) < len(yearsFilter)-1):
print('Skipping ... %s since very few rows (%d) found' % (indicatorsFilter, len(filteredData1)))
Aucun commentaire:
Enregistrer un commentaire