teachNumpy

This repository contains Jupyter Notebook Files of some state of the art projects that I completed during my internship program in deeplearning.ai. The project files are divided into 5 main categories or respective courses that the deeplearning.ai provides.

I'm attempting the NYC Taxi Duration prediction Kaggle challenge. I'll by using a combination of Pandas, Matplotlib, and XGBoost as python libraries to help me understand and analyze the taxi dataset that Kaggle provides. The goal will be to build a predictive model for taxi duration time. I'll also be using Google Colab as my jupyter notebook. i will also predict without Google colab on normal system.

India GDP Analysis Problem Description - I NITI Aayog: Background NITI Aayog (National Institution for Transforming India) is a policy think tank of the Government of India; it provides strategic inputs to the central and the state governments to achieve various development goals. In the past, NITI Aayog has played an important role in initiatives such as Digital India, Atal Innovation Mission and various agricultural reforms and have designed various policies in education, skill development, water management, healthcare, etc. NITI Aayog was established to replace the Planning Commission of India, which used to follow a top-down model for policy making, i.e., it typically designed policies at the central level (such as the 5-year plans). On the other hand, NITI Aayog designs policies specific to the different states or segments of the economy. Finance Minister, Arun Jaitley, made the following observation on the necessity of creating NITI Aayog, "The 65-year-old Planning Commission had become a redundant organisation. It was relevant in a command economy structure, but not any longer. India is a diversified country and its states are in various phases of economic development along with their own strengths and weaknesses. In this context, a ‘one size fits all’ approach to economic planning is obsolete...". Project Brief We are working as the chief data scientist at NITI Aayog, reporting to the CEO. The CEO has initiated a project wherein the NITI Aayog will provide top-level recommendations to the Chief Ministers (CMs) of various states, which will help them prioritise areas of development for their respective states. Since different states are in different phases of development, the recommendations should be specific to the states. The overall goal of this project is to help the CMs focus on areas that will foster economic development for their respective states. Since the most common measure of economic development is the GDP, we will analyse the GDP of the various states of India and suggest ways to improve it. Understanding GDP Gross domestic product (GDP) at current prices is the GDP at the market value of goods and services produced in a country during a year. In other words, GDP measures the 'monetary value of final goods and services produced by a country/state in a given period of time'. GDP can be broadly divided into goods and services produced by three sectors: the primary sector (agriculture), the secondary sector (industry), and the tertiary sector (services). It is also known as nominal GDP. More technically, (real) GDP takes into account the price change that may have occurred due to inflation. This means that the real GDP is nominal GDP adjusted for inflation. We will use the nominal GDP for this exercise. Also, we will consider the financial year 2015-16 as the base year, as most of the data required for this exercise is available for the aforementioned period. Per Capita GDP and Income Total GDP divided by the population gives the per capita GDP, which roughly measures the average value of goods and services produced per person. The per capita income is closely related to the per capita GDP (though they are not the same). In general, the per capita income increases when the per capita GDP increases, and vice-versa. For instance, in the financial year 2015-16, the per capita income of India was ₹93,293, whereas the per capita GDP of India was $1717, which roughly amounts to ₹1,11,605. Problem Description - II Data The data is sourced from https://data.gov.in/, an Open Government Data (OGD) platform of India. The download instructions are provided in the next segment. The data for GDP analysis of the Indian states is divided into two parts: Data I-A: This dataset consists of the GSDP (Gross State Domestic Product) data for the states and union territories. Data I-B: This dataset contains the distribution of GSDP among three sectors: the primary sector (agriculture), the secondary sector (industry) and the tertiary sector (services) along with taxes and subsidies. There is separate dataset for each of the states. We are expected to read the dataset for the available states and join these (in Python) if needed. There are two parts to this project. In the first part, we will analyse and compare the GDPs of various Indian states (both total and per capita). The GDP of a state is referred to as the GSDP (Gross State Domestic Product). Then, we will divide the states into four categories based on the GDP per capita, and for each of these four categories, we will analyse the sectors that contribute the most to the GDP (such as agriculture, real estate, manufacturing, etc.). In the second part, we will analyse whether GDP per capita is related to dropout rates in schools and colleges. Part-I: GDP Analysis of the Indian States For each of the following steps of analysis, choose an appropriate type of plot for comparing the data. Also, ensure that the plots are in increasing or decreasing order for better comparison. For example, if we make a bar plot to compare the GDPs of the states, ensure that the bars are in either increasing or decreasing order of GDP. Part I-A: For the analysis below, use the Data I-A. First, we need to load the data in Python properly and then clean it. This also involves the treatment of missing values, we can choose to drop the row or column as well. Remember this will affect our next analysis and results drastically. Plot a graph for rows " % Growth over previous year" for all the states (not union territories) whose data is available, use as much data as possible for this exercise. Use the best fit line to represent the growth for each state. Draw a similar line graph for the nation as well. How will we compare the growth rates of any two states? Which states have been growing consistently fast, and which ones have been struggling? Rank top 3 fastest and 3 slowest-growing states. What is the Nation's growth rate? What has been the growth rate of my home state, and how does it compare to the national growth rate? Plot the total GDP of the states for the year 2015-16: Which Plot will we use for this? Why? (Remeber to plot the graph in a way such as it is easier to read and compare) Identify the top 5 and the bottom 5 states based on total GDP. What insights can we draw from this graph? What states are performing poorly? (Remember: this will not be solely based on total GDP) Part I-B: For the analysis below, use Data I-B. We can also use Data I-B along with Data I-A if required. Also, perform the analysis only for the duration 2014-15. Filter out the union territories (Delhi, Chandigarh, Andaman and Nicobar Islands, etc.) for further analysis, as they are governed directly by the central, not state governments. Plot the GDP per capita for all the states. Identify the top 5 and the bottom 5 states based on the GDP per capita. Find the ratio of the highest per capita GDP to the lowest per capita GDP. Plot the percentage contribution of the primary, secondary and tertiary sectors as a percentage of the total GDP for all the states. Which plot will we use here? Why? Why is (Primary + Secondary + Tertiary) not equal to total GDP? Can we draw any insight from this? Find correlation of percentile of the state (% of states with lower per capita GDP) and %contribution of Primary sector to total GDP. Categorise the states into four groups based on the GDP per capita (C1, C2, C3, C4, where C1 would have the highest per capita GDP and C4, the lowest). The quantile values are (0.20,0.5, 0.85, 1), i.e., the states lying between the 85th and the 100th percentile are in C1; those between the 50th and the 85th percentiles are in C2, and so on. Note: Categorisation into four groups will simplify the subsequent analysis, as otherwise, comparing the data of all the states would become quite exhaustive. For each category (C1, C2, C3, C4): Find the top 3/4/5 sub-sectors (such as agriculture, forestry and fishing, crops, manufacturing etc., not primary, secondary and tertiary) that contribute to approximately 80% of the GSDP of each category. Note-I: The nomenclature for this project is as follows: primary, secondary and tertiary are named 'sectors', while agriculture, manufacturing etc. are named 'sub-sectors'. Note-II: If the top 3 sub-sectors contribute to, say, 79% of the GDP of some category, we can report "These top 3 sub-sectors contribute to approximately 80% of the GDP". This is to simplify the analysis and make the results consumable. (Remember, the CEO has to present the report to the CMs, and CMs have limited time; so, the analysis needs to be sharp and concise.) Plot the contribution of the sub-sectors as a percentage of the GSDP of each category. Now that we have summarised the data in the form of plots, tables, etc., try to draw non-obvious insights from it. Think about questions such as: How does the GDP distribution of the top states (C1) differ from the others? Which sub-sectors seem to be correlated with high GDP? Which sub-sectors do the various categories need to focus on? Ask other such relevant questions, which we think are important, and note we insights for category separately. More insights are welcome and will be awarded accordingly. Finally, provide at least two recommendations for each category to improve the per capita GDP. Part-II: GDP and Education Dropout Rates In Part-I, we would have noticed that (one) way to increase per capita GDP is by shifting the distribution of GDP towards the secondary and tertiary sectors, i.e., the manufacturing and services industries. But these industries can thrive only when there is an availability of educated and skilled labour. In this part of the analysis, we will investigate whether there is any relationship between per capita GDP with dropout rates in education. Data Data II: This section will require the dropout rate dataset apart from the dataset that we used in Part-1 of the case study. Download instructions are provided in the next segment. Part-II: GDP and Education Analyse if there is any correlation of GDP per capita with dropout rates in education (primary, upper primary and secondary) for the year 2014-2015 for each state. Choose an appropriate plot to conduct this analysis. Is there any correlation between dropout rate and %contribution of each sector (Primary, Secondary and Tertiary) to the total GDP? We have the total population of each state from the data in part I. Is there any correlation between dropout rates and population? What is the expected trend and what is the observation? Write down the key insights we draw from this data: Form at least one reasonable hypothesis for the observations from the data About GDP analysis for India in the year for 2015-16 and recommendation for the individual states for increasing the GDP by focusing on various factor. Topics python statistical-analysis data-analysis gdp-analysis Resources Readme Stars 0 stars Watchers 1 watching Forks 0 forks Releases No releases published Packages No packages published Languages Jupyter Notebook 100.0% Footer © 2022 GitHub, Inc. Footer navigation Terms Privacy Security Status Docs Contact GitHub Pricing API Training Blog About

ANALYZING ROAD SAFETY & TRAFFIC DEMOGRAPHICS IN THE UK (Multi-class Classification) SUMMARY Here, I am aim to analyze the Road Safety and Traffic Demographics dataset (UK), containing accidents reported by the police between the years of 2004 - 2017. PROJECT GOALS: Identify factors responsible for most of the reported accidents. Build a machine learning model that is capable of accurately predicting the severity of an accident. Provide recommendations to the Department of Transport (UK Government), to improve road safety policies and prevent recurrences of severe accidents where possible. PACKAGES USED: Scikit-learn, numpy, pandas, imblearn (imbalanced-learn), seaborn, Matplotlib MOTIVATION World Health Organization (WHO) reported that more than 1.25 million people die each year while 50 million are injured as a result of road accidents worldwide. Road accidents are the 10th leading cause of death globally. On current trends, road traffic accidents are to become the 7th leading cause of death by 2030 making it a major public health concern. Between the years 2005 and 2016, there were roughly 2 million road accidents reported in the United Kingdom (UK) alone of which 16,000 were fatal. As a big data project, I wanted to explore the traffic demographics data in greater detail using machine learning! CONTEXT The UK government amassed traffic data from 2004 to 2017, recording over 2 million accidents in the process and making this one of the most comprehensive traffic data sets out there. It's a huge picture of a country undergoing change. Note that all the contained accident data comes from police reports, so this data does not include minor incidents. For steps undertaken to pre-process and clean the data, please view the "Data Cleansing & Descriptive Analysis_UK Traffic Demographics.ipynb" file DESCRIPTIVE ANALYTICS (EDA) Tools used include Python, Tableau, MS PowerBI Percent (%) distribution of target classes Percent dist of Accident Severity As seen above, the data is highly imbalanced. For detailed steps undertaken to deal with the imbalanced data, please view the "Modelling_Predictive Analytics_UK Traffic Demographics.ipynb" file. This article provides some great tips on utilizing the correct performance metrics when analyzing a models performance trained on an imbalanced dataset. This article describes several strategies that can help combat the case of a severly imbalanced dataset. Methods include: Resampling strategies (under - Tomek Links, Cluster Centroids, over sampling - SMOTE) Using Decision Tree based models Using Cost-Sensitive training (Penalize algorithms) Number of accidents by Year and Accident Severity Total accidents by year and severity It can be seen above that the trend seems to be increasing as the years go. In addition, the spike between 2008 - 2009 was because of a enhancement in the reporting system introduced in the UK in 2009, where all accident including minor accidents needed to be reported by the police so as to match the counts represented by hospitals, insurance claims etc. Accidents density by Location geomap Most accidents took place in major cities - Birmingham, London, leeds, Newcastle Accidents by Gender and Age Accidents by gender and age Accidents by Day of the week and Year Accidents by year and weekday Most accidents take place on a Friday Vehicle Manoever at time of accident Vehicle Manoever at time of accident Most accidents take place as a result of overtaking For more findings, please go to the "Images" folder. For steps undertaken to carry out some predictive modeling and hyper-parameter tuning, please view the "Modelling_Predictive Analytics_UK Traffic Demographics.ipynb" file. RECOMMENDATIONS TO THE DEPARTMENT OF TRANSPORT (UK) Decrease emergency response times during afternoon rush-hours (15-19) especially on Fridays. Allocate resources to investigate high density traffic points and identify new infrastructure needs to divert traffic from dual-carriage ways. Explore conditions of vehicles and casualties such as vehicle type, age of vehicles registered, pedestrian movements, etc. for policy makers. Adopt comprehensive distracted driving laws that increase penalties for drivers who commit traffic violations like aggressive overtaking. ACKNOWLEDGEMENTS The license for this dataset is the Open Givernment Licence used by all data on data.gov.uk. The raw datasets are available from the UK Department of Transport website. I had a lot of fun working on this dataset and learned a lot in the process. I plan to further my research in the area of predictive modeling using imabalanced data and how to effectively build a highly robust model for future projects. About Here, I analyze the Road Safety and Traffic Demographics dataset (UK), containing accidents reported by the police between the years of 2004 - 2017. Topics accident-rate accident-severity imbalanced-data imbalanced-learning road-accident reported-accidents road-safety uk-government transport traffic-demographics severe-accidents pca classification Resources Readme Releases No releases published Packages No packages published Languages Jupyter Notebook 100.0% © 2020 GitHub, Inc.

In this repo, I will share my Jupyter Notebooks of python Libraries like NumPy, Pandas, Matplotlib, TensorFlow, PyTorch, Scikit-learn, Requests, Keras, Seaborn and beyond. They will carry notes and practice questions. I will also provide hands on project on them. Anyone can Learn them with me. Till then Keep Learning Keep Growing

This repository contains my favourite examples of Image Denoising for microscopy/biology images by using Python and in particular Jupyter notebook. Full credit goes to Dr. Sreenivas Bhattiprolu, who created this examples in the first place; you can find his .py files at https://github.com/bnsreenu/python_for_microscopists. This repository is just a selection of my favourite denoising algorithms illustrated by Dr. Bhattiprolu. I provide Jupyter notebook files, original images to upload and PDF previews so that any person can have fun using these algorithms. Installation guidelines of Jupyter notebook (via Anaconda installation) are also provided.