Steps to get API Params

• Ctrl+Shift+I (Developer Tools)
• Network Tab
• Fetch/XHR
• Click on a request object
• Headers
• Request URL = API URL and User-Agent = user-agent
• Payload = Body of the Request
• Response = Server's Response to the request -> The actual data we need

Fig. 1 Open Devtools Fig. 2 Headers Fig. 3 User-Agent in Headers Fig. 4 Payload Fig. 5 Response

3️⃣ Data Cleaning & Structuring

• Removed source tags and newline artifacts
• Unified NSE and BSE datasets
• Ensured:
  • No missing values
  • Proper date parsing
  • Clean text for NLP models

4️⃣ Sentiment Scoring

Batch inference using FinBERT Extracted:

• Sentiment label
• Confidence score Generated compound sentiment metric

📊 Key Visualizations (Core Insights)

These graphs form the analytical backbone of the project:

📌 Market Sentiment Distribution

Donut / pie chart showing:

• Positive
• Neutral
• Negative Helps gauge overall market mood

📌 Percentage-wise Stacked Bar Chart

• Top NSE stocks
• Shows sentiment composition (%) per stock
• Ideal for relative comparison

📌 Sentiment Over Time

Mean compound sentiment per stock per day Highlights:

• Market reactions
• News-driven volatility
• Trend reversals

📌 Normalized Sentiment (Z-Score)

• Standardizes sentiment across stocks
• Includes error bars (volatility)
• Answers: “Which stocks are relatively most positive or negative?”

📌 Heatmap (Stock × Date)

• Visualizes sentiment evolution
• Identifies sentiment clusters and spikes

📌 Treemap (NSE & BSE)

• Size represents relative sentiment strength
• Fast visual ranking of sentiment dominance

📈 Additional Visualizations (Deeper NLP Insights)

🟦 Word Clouds

• Positive Word Cloud
• Negative Word Cloud

Highlights frequently used finance-related terms

🎭 Emotion Analysis (NRC Lexicon)

Using the NRC Emotion Lexicon (14,000+ words), emotions are mapped into finance-relevant categories:

• Greed
• Fear
• Panic
• Optimism
• Pessimism
• Confidence
• Uncertainty

🎯 Emotion-Based Visuals

• Stacked Emotion Bar Charts
• Emotion Radar Chart (per stock)
• Emotion Composition Comparison

These visuals explain how the market talks about a stock, not just whether sentiment is positive or negative.

📚 What I Learned

🧠 API Reverse Engineering

• Inspecting network traffic
• Understanding payloads, headers, and responses
• Handling pagination and dynamic feeds

🔧 Data Engineering

• Cleaning unstructured financial text
• Merging multi-exchange datasets
• GroupBy + Unstack workflows
• 🤖 NLP for Finance
• Differences between VADER and FinBERT
• Why domain-specific models matter
• Mapping transformer outputs into interpretable metrics

📊 Advanced Visualization

• Sentiment normalization
• Error bars and volatility
• Multi-dimensional visual storytelling

🚀 How It Can Be Improved

• Add real-time ingestion (cron / Airflow)
• Extend beyond Groww (NewsAPI, Twitter, Reddit)
• Build an interactive dashboard (Streamlit / Dash)
• Store historical data in SQL/NoSQL
• Predict price movement using sentiment trends
• Sector-wise and index-wise sentiment aggregation

#Graphs

Graph 1 Market Sentiments Graph 2 Percentage of Sentiments Per Stock Graph 3 Sentiment Over Time Per Stock Per Date Graph 4 Normalized average sentimens per stock Graph 5 Heatmap per stock per date of normalized data Graph 6 TreeMap of NSE Top 10 Graph 7 TreeMap of BSE Top 10 Graph 8 Emotion Stack Bar Graph Graph 9 Radar Chart of Emotions Graph 10 Positive Wordcloud Graph 11 Negative Wordcloud