dimanche 9 juillet 2017

Looping over large dataset in pandas

I am trying to iterate over dataframe which is from Indicators.csv from this kaggle link to collect random countries , year and random indicator which are common with respect to each other using Pandas but using while loops is taking a lot of time

I know itertuples() is better way and saves a lot of time but don't know how to implement it

Also I came to know vectorization should be used to loop over large datasets, if someone explain how to use that it will be appreciated.

data = pd.read_csv('Indicators.csv')
countries = data['CountryName'].unique().tolist()
indicators = data['IndicatorName'].unique().tolist()
yearsFilter = [2010, 2011, 2012, 2013, 2014]

filteredData1 = []
filteredData2 = []

while(len(filteredData1) < len(yearsFilter)-1):
    # pick new indicator
    indicatorsFilter = random.sample(indicators, 1)
    countryFilter    = random.sample(countries, 2)
    # how many rows are there that have this country name, this indicator, and this year.  Mesh gives bool vector
    filterMesh = (data['CountryName'] == countryFilter[0]) & (data['IndicatorName'].isin(indicatorsFilter)) & (data['Year'].isin(yearsFilter))
    # which rows have this condition to be true?
    filteredData1 = data.loc[filterMesh]
    filteredData1 = filteredData1[['CountryName','IndicatorName','Year','Value']]

    # need to print this only when our while condition is true
    if(len(filteredData1) < len(yearsFilter)-1):
        print('Skipping ... %s since very few rows (%d) found' % (indicatorsFilter, len(filteredData1)))




Aucun commentaire:

Enregistrer un commentaire