I Bought Hundreds Of Funds & It Didn't Diworsify

I have been hearing this absurd logic for years.

“If you buy too many mutual funds, you just end up with an expensive index fund.”

I never thought much about it because I always knew it was more of a rhetoric than a fact.

But rescently I am seeing people who has some knowledge in finace berate people who are not financially savvy for unintentionally/unknowingly buy too many funds than they need.

And interestingly, the people who are supposed to be financially savvy, couldnt comprehend how it couldnt create an index.

I guess if you repeat a lie often enough and it becomes the truth.

Heck even the AI, which is supposed to be voice of obective reason kept reverting back to statements like "It will cancel each out" to  create an index.

How can I blame the less savvy, when they are fed this idiocracy?

And so, I built an anti-rhetoric to fight it,  and mine goes like this - Even buying hundreds of active funds, randomly picked -will not re-create the index - the people who has too many funds are not holding an expensive index fund.😊

Yep - You heard me right!

I bought exactly 516 funds - well actually simulated.

It was quite literally almost everything they had, except for the the few funds I filtered out using very broad logic. 

I used filteration for 2 purpose - so that there is some order in what is bought and  because even with this absurd strategy, I wanted to beat the index.😂

Here were my filtering criteria :  

  • Direct Plans & Growth Options Only: To reduce the fee drag and eliminate dividend payouts. You can't beat an index, if they are charging too much.
  • No Pure Large Cap Funds: Since active large caps are known to not beat the index anyway, these were not included. But yes, flexicap, ELSS etc are part of it, and so there are large cap companies in the mix
  • The 2018 Starting Line: The funds had to exist from January 1, 2018, to August 4th - This filter allowed  me to categorize along with SEBI's Guidelines of Mutual fund categorization.
  • Dynamic Rebalancing: If a fund died or merged during the period, the balance from that fund was split between the rest.
  • Error in data : Any funds, which showed  returns that couldnt possibly exist was filtered out.

And the result - -  


=====================================================
                 FINAL RESULTS
=====================================================
Timeframe        : 2018-01-01 to 2026-08-03 (8.59 Years)
Funds Evaluated  : 516
Democracy Portfolio : 13.63% CAGR
BSE 500 Actual Index (Broad Market) : 11.07% CAGR
=====================================================

A note to the people who are not financially savvy:

The point of the experment is merely to show you that if you accidentally purchased too many funds - there is no reason to feel bad about it.

You did great with what information you had.

However,  As you can see above,  even a simple strategy can serve you good -  I am pretty confident that if I had used regular or involved Large Caps, I wouldnt have been able to beat the index - but it is debatable, depending on the number of Large caps that is available to purchase.

With a bit of luck, even a portfolio which was not built up using a strategy can beat the index - without it, you are still doing better than without investing.

The problem is not returns or diworsification or whatever.

The problem is that it will become too much to track - But if you can track it, I see no problem with anyone holding an extremely bloated portfolio.

If this collosally bloated portfolio had failed to beat the index, I wouldnt know what to tweak.

I dont know whether the portfolio as a whole is value, growth, quality or momentum oriented.

That is the problem with a bloated portfolio - the inability to track.

If you can track your 20 funds using an excel sheet and if you are comfortable with it - there is no need to consolidate  just because someone else told you it's bad.

Personal finance is afterall personal.

Below is the script I used to create my Democracy Of Fund Managers, if anyone is interested : 




import sys
import time
import requests
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import yfinance as yf
import warnings
import sqlite3
import os

# Suppress pandas warnings for cleaner terminal output
warnings.simplefilter(action='ignore', category=pd.errors.PerformanceWarning)
warnings.simplefilter(action='ignore', category=FutureWarning)

START_DATE_STR = '2018-01-01'
DB_FILE = 'active_funds.db'

# Strict blacklist to eliminate anything that isn't domestic pure equity
exclude_keywords = [
    "IDCW", "DIVIDEND", "INDEX", "ETF", "NIFTY", "SENSEX", "BSE", 
    "DEBT", "LIQUID", "GILT", "BOND", "FMP", "FIXED", "ARBITRAGE", 
    "HYBRID", "CONSERVATIVE", "MIP", "INCOME", "MONEY MARKET", "INTERVAL", 
    "FTP", "ASSET ALLOCATION", "SHORT TERM", "ULTRA", "CREDIT", 
    "TREASURY", "REGULAR", "INSTITUTIONAL", "FOF", "FUND OF", "GOLD", 
    "COMMODITY", "SAVINGS", "DYNAMIC", "MONTHLY", "CASH", "TAX PLAN -",
    "MATURITY", "FLOATING RATE", "PSU", "BALANCED", "ADVANTAGE", "DURATION",
    "MULTI-ASSET", "MULTI ASSET", "PLAN B", "PLAN C", "BONUS", "SILVER", 
    "GLOBAL", "INTERNATIONAL", "OVERSEAS", "CHILDREN", "RETIREMENT"
]

def is_pure_equity_growth(name):
    name_upper = name.upper()
    if "DIRECT" not in name_upper: return False
    if "GROWTH" not in name_upper: return False
    for word in exclude_keywords:
        if word in name_upper:
            return False
    return True

def get_candidates():
    conn = sqlite3.connect(DB_FILE)
    c = conn.cursor()
    c.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='candidates'")
    if c.fetchone():
        c.execute("SELECT code, name FROM candidates")
        rows = c.fetchall()
        if len(rows) > 0:
            conn.close()
            return rows

    print("-> Local DB not found. Downloading the AMFI Master Directory...")
    c.execute("CREATE TABLE candidates (code TEXT PRIMARY KEY, name TEXT)")
    try:
        response = requests.get("https://api.mfapi.in/mf", timeout=20)
        all_funds = response.json()
        candidates = [(str(f['schemeCode']), f['schemeName']) for f in all_funds if is_pure_equity_growth(f['schemeName'])]
        c.executemany("INSERT OR IGNORE INTO candidates (code, name) VALUES (?, ?)", candidates)
        conn.commit()
    except Exception as e:
        candidates = []
    conn.close()
    return candidates

print('=====================================================')
print(' Democracy of Fundmanagers 11.0: UNIFIED DATABASE CACHE')
print('=====================================================\n')

# --- CACHE LOGIC ---
use_cache = False
if os.path.exists(DB_FILE):
    # Check if the cache database is less than 24 hours old
    file_age = time.time() - os.path.getmtime(DB_FILE)
    if file_age < 86400: 
        conn = sqlite3.connect(DB_FILE)
        c = conn.cursor()
        c.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='portfolio_cache'")
        if c.fetchone():
            use_cache = True
        conn.close()

if use_cache:
    print(f"-> Found fresh local database cache! Bypassing API and loading historical data instantly...")
    conn = sqlite3.connect(DB_FILE)
    master_df = pd.read_sql('SELECT * FROM portfolio_cache', conn, index_col='date')
    master_df.index = pd.to_datetime(master_df.index)
    successful_funds = len(master_df.columns)
    conn.close()
else:
    unique_candidates = get_candidates()
    series_list = []
    successful_funds = 0

    print("\nCommencing Historical Data Validation (This will take a few minutes, but will cache afterwards)...")

    for code, name in unique_candidates:
        sys.stdout.write(f'\rTesting {name[:50].ljust(50)}...')
        sys.stdout.flush()

        try:
            response = requests.get(f'https://api.mfapi.in/mf/{code}', timeout=10)
            data = response.json()
            if 'data' not in data or not data['data']:
                continue

            df = pd.DataFrame(data['data'])
            df['date'] = pd.to_datetime(df['date'], format='%d-%m-%Y')
            df['nav'] = pd.to_numeric(df['nav'], errors='coerce')
            df = df[df['nav'] > 0].dropna().sort_values('date').set_index('date')

            if df.index[0] > pd.to_datetime(START_DATE_STR):
                continue

            df = df[df.index >= START_DATE_STR]
            if df.empty:
                continue
                
            # THE CIRCUIT BREAKER: Reject funds with corrupt data (e.g. > 50% move in a single day)
            daily_pct = df['nav'].pct_change()
            if (daily_pct > 0.5).any() or (daily_pct < -0.5).any():
                continue

            # Normalize NAV
            df['normalized'] = (df['nav'] / df['nav'].iloc[0]) * 100
            df['normalized'] = df['normalized'].replace([np.inf, -np.inf], np.nan)

            series_list.append(df['normalized'].rename(name))
            successful_funds += 1
            sys.stdout.write(f'\r[{successful_funds}] APPROVED: {name[:50].ljust(50)}\n')

        except Exception:
            pass
            
        time.sleep(0.02)

    print('\n\nCompiling and caching data for future runs...')
    master_df = pd.concat(series_list, axis=1).sort_index()
    
    # Save the massive matrix directly into a new table in the same SQLite DB
    conn = sqlite3.connect(DB_FILE)
    master_df.to_sql('portfolio_cache', conn, if_exists='replace', index=True)
    conn.close()

print('Simulating Dynamic Portfolio Reallocation (Equal-Weighting)...')

# Forward-fill minor gaps up to 15 days (dead funds turn to NaN)
master_df = master_df.ffill(limit=15)

# Calculate daily percentage returns
daily_returns = master_df.pct_change(fill_method=None)

# Average daily return of ALIVE funds (NaNs are ignored, simulating re-allocation)
portfolio_daily_returns = daily_returns.mean(axis=1)

# Reconstruct portfolio value starting at 100
democracy_portfolio = 100 * (1 + portfolio_daily_returns.fillna(0)).cumprod()

start_date = democracy_portfolio.index[0]
end_date = democracy_portfolio.index[-1]
years_lapsed = (end_date - start_date).days / 365.25

democracy_final_val = democracy_portfolio.iloc[-1]
democracy_cagr = (democracy_final_val / 100) ** (1 / years_lapsed) - 1

# ==========================================
# ROBUST YFINANCE INDEX FETCHING
# ==========================================
print("\nFetching Index Data from Yahoo Finance...")

nifty500_data = None
index_label = ""
tickers_to_try = [('BSE-500.BO', 'BSE 500 Actual Index (Broad Market)'), ('^NSEI', 'Nifty 50 (Fallback)')]

for ticker, label in tickers_to_try:
    print(f" -> Attempting to download {ticker}...")
    for attempt in range(3): 
        try:
            temp_data = yf.download(ticker, start=start_date.strftime('%Y-%m-%d'), progress=False)['Close']
            if not temp_data.empty:
                nifty500_data = temp_data
                index_label = label
                break
        except Exception:
            time.sleep(1) 
    if nifty500_data is not None and not nifty500_data.empty:
        print(f" -> SUCCESS: Retrieved {label}")
        break

if nifty500_data is not None and not nifty500_data.empty:
    nifty500_data.index = nifty500_data.index.tz_localize(None)
    nifty_benchmark = (nifty500_data.squeeze() / nifty500_data.iloc[0].squeeze()) * 100 
    nifty_cagr = (nifty_benchmark.iloc[-1] / 100) ** (1 / years_lapsed) - 1
else:
    print(" -> All yfinance fetches failed. Using historical CAGR baseline.")
    nifty_cagr = 0.155
    days_array = (democracy_portfolio.index - start_date).days.values
    nifty_benchmark = pd.Series(100 * ((1 + ((1 + nifty_cagr) ** (1 / 365.25) - 1)) ** days_array), index=democracy_portfolio.index)
    index_label = "BSE 500 TRI (Synthesized ~15.5%)"

print('\n=====================================================')
print('                 FINAL RESULTS')
print('=====================================================')
print(f'Timeframe        : {start_date.date()} to {end_date.date()} ({years_lapsed:.2f} Years)')
print(f'Funds Evaluated  : {successful_funds}')
print(f'Democracy Portfolio : {democracy_cagr * 100:.2f}% CAGR')
print(f'{index_label} : {nifty_cagr * 100:.2f}% CAGR')
print('=====================================================\n')

# --- Plot Graph ---
plt.style.use('ggplot')
plt.figure(figsize=(12, 6))

for col in master_df.columns:
    plt.plot(master_df.index, master_df[col], color='gray', alpha=0.10, linewidth=0.8)

plt.plot(nifty_benchmark.index, nifty_benchmark, label=f'{index_label} ({nifty_cagr * 100:.2f}% CAGR)', color='blue', linewidth=2, linestyle='--')
plt.plot(democracy_portfolio.index, democracy_portfolio, label=f'All {successful_funds} Active Funds Average ({democracy_cagr * 100:.2f}% CAGR)', color='red', linewidth=3.5)

# Keep the Y-axis focused on the real data, just in case a minor glitch slips through
max_y = max(democracy_portfolio.max(), nifty_benchmark.max()) * 1.3
plt.ylim(0, max_y)

plt.title(f"Democracy Of Fund Managers (2018-Present)", fontweight='bold')
plt.xlabel('Year')
plt.ylabel('Portfolio Value (Starting at ₹100)')
plt.legend(loc='upper left')
plt.grid(True, linestyle=':', alpha=0.6)
plt.tight_layout()

print('Opening graph...')
plt.show()

Comments

Post a Comment

Popular posts from this blog

How to start Investing

Image
Lurking in reddit, one of the most common questions I find is how to start investing. Some come with randomly picked mutual funds and others directly ask it. Unfortunately,  there is no structured way to teach it and I too started the same way -  simply  picking random funds, understanding things and then improving. Now that I have obsessed over the topic for nearly a decade, I think I am qualified to give an advice -  or atleast attempt to. However before I get into the actual process, I highly recommend doing the following two things :  Adequate Emergency Fund Adequate Medical Cover The reason I call this mandatory is because Equity is lumpy. It can go on for years without any returns whatsoever. Please refer to the below chart of nifty 500. As you can see from the chart, the market spends a long time doing nothing or at a loss. This is to be expected. For multiple years, the market does nothing, and then it in one shot, it will reward your patience before tes...
Read more

4% Safe Withdrawal Rate Is Valid For India

Image
I have been thinking about retirement for the past decade. “If you don’t find a way to make money while you sleep, you will work until you die.” _ Warren Buffet That is the quote that got me obsessed with it — I didn't want to work till I die. I suppose nobody does. Even though I have the dream to retire early and obsess over it, the standard advice I hear everywhere is to work till you die. Isn't that a bummer? They defend the idea by putting a positive spin on it, claiming it is your fault for not choosing work that you like. I suppose the argument has some merit, since life is known to be a bitch! I concede that safety lies in having an active income, but the most valuable asset you can ever own is time—the ability to do whatever you want, whenever you want to do it. Now, conquering time is not possible if you are dependent on an active income, because an active income requires your performance, your productivity, your politics etc. Don't get me wro...
Read more