justmarkham / pandas-videos Goto Github PK

Jupyter notebook and datasets from the pandas video series

Home Page: https://courses.dataschool.io/pandas-in-30-days

Jupyter Notebook 100.00%

data-science jupyter-notebook python pandas tutorial data-analysis data-cleaning

pandas-videos's Introduction

Python pandas video series

The series is also available as a free online course that includes updated content, exercises, and a certificate of completion.

📺 Videos (playlist)

What is pandas? (Introduction to the Q&A series) (6:24)
How do I read a tabular data file into pandas? (8:54)
How do I select a pandas Series from a DataFrame? (11:10)
Why do some pandas commands end with parentheses (and others don't)? (8:45)
How do I rename columns in a pandas DataFrame? (9:36)
How do I remove columns from a pandas DataFrame? (6:35)
How do I sort a pandas DataFrame or a Series? (8:56)
How do I filter rows of a pandas DataFrame by column value? (13:44)
How do I apply multiple filter criteria to a pandas DataFrame? (9:51)
Your pandas questions answered! (9:06)
How do I use the "axis" parameter in pandas? (8:33)
How do I use string methods in pandas? (6:16)
How do I change the data type of a pandas Series? (7:28)
When should I use a "groupby" in pandas? (8:24)
How do I explore a pandas Series? (9:50)
How do I handle missing values in pandas? (14:27)
What do I need to know about the pandas index? (Part 1) (13:36)
What do I need to know about the pandas index? (Part 2) (10:38)
How do I select multiple rows and columns from a pandas DataFrame? (21:46)
When should I use the "inplace" parameter in pandas? (10:18)
How do I make my pandas DataFrame smaller and faster? (19:05)
How do I use pandas with scikit-learn to create Kaggle submissions? (13:25)
More of your pandas questions answered! (19:23)
How do I create dummy variables in pandas? (13:13)
How do I work with dates and times in pandas? (10:20)
How do I find and remove duplicate rows in pandas? (9:47)
How do I avoid a SettingWithCopyWarning in pandas? (13:29)
How do I change display options in pandas? (14:55)
How do I create a pandas DataFrame from another object? (14:25)
How do I apply a function to a pandas Series or DataFrame? (17:57)
Bonus: How do I use the MultiIndex in pandas? (25:00)
Bonus: How do I merge DataFrames in pandas? (21:48)
Bonus: 4 new time-saving tricks in pandas (14:50)
Bonus: 5 new changes in pandas you need to know about (20:54)
Bonus: My top 25 pandas tricks (27:37)
Bonus: 21 more pandas tricks (24:39)
Bonus: Data Science Best Practices with pandas (PyCon 2019) (1:44:16)
Bonus: Your pandas questions answered! (webcast) (1:56:01)

📓 Jupyter Notebooks

Python pandas Q&A series (videos 1 to 30)
How do I use the MultiIndex in pandas? (video 31)
How do I merge DataFrames in pandas? (video 32)
4 new time-saving tricks in pandas (video 33)
5 new changes in pandas you need to know about (video 34)
My top 25 pandas tricks (video 35)
21 more pandas tricks (video 36)
Data Science Best Practices with pandas (PyCon 2019) (video 37)

📊 Datasets

Filename	Description	Raw File	Original Source	Other
chipotle.tsv	Online orders from the Chipotle restaurant chain	bit.ly/chiporders	The Upshot	Upshot article
drinks.csv	Alcohol consumption by country	bit.ly/drinksbycountry	FiveThirtyEight	FiveThirtyEight article
imdb_1000.csv	Top rated movies from IMDb	bit.ly/imdbratings	IMDb	Web scraping script
stocks.csv	Small dataset of stock prices	bit.ly/smallstocks	DataCamp
titanic_test.csv	Testing set from Kaggle's Titanic competition	bit.ly/kaggletest	Kaggle	Data dictionary
titanic_train.csv	Training set from Kaggle's Titanic competition	bit.ly/kaggletrain	Kaggle	Data dictionary
u.data	Movie ratings by MovieLens users	bit.ly/movielensdata	GroupLens	Data dictionary
u.item	Movie information from MovieLens	bit.ly/movieitems	GroupLens	Data dictionary
u.user	Demographic information about MovieLens users	bit.ly/movieusers	GroupLens	Data dictionary
ufo.csv	Reports of UFO sightings from 1930-2000	bit.ly/uforeports	National UFO Reporting Center	Web scraping script

pandas-videos's People

Contributors

Stargazers

Watchers

Forkers

rahulsingh50 rlugojr nanoresearch elinok ln1267 anhmike isaac1989 arkoneogy fernanccervone rayed-therap kuanhoong olinero tsudhahar redwa vinodhkrishnaraju pwaila wanggene memebrain danielcjlee aitatanit mitchshack niteeshkanungo akash-ansari syzdemonhunter marcds littletiger311 atrevas mismz feststelltaste mnalevanko phdkiran amit-dingare tammachari pineapplechiu mtchem ratulghosh pepper1709 leezqcst melakbet aiamateur anhlbt mimirui123 mshekha2 ranjankumarpatel sukanyamandal punyaslokamuduli popsonebz pgnepal drbaguiar githubsniper andy-buv sacridini subhraj07 joshuacourse fajk zhenjunz sandianyang jzhang45 matkal93 oscarrondoni atulsingh0 autred magenti fulkay true-developer mayanksingh09 pengyan-sophia willkriski pandeyaah samirpoonawala chenguanghn tonio145 paisely65 pawanvirsingh sgrim5235 mudragada roosbeh-nowrouzian shyamsy brandeddavid jibinjohney ist256 codetradr imanerezzag madhan1m udayoo7 iamamar venky14 wdillon2 zakved ridzuan05 terratenney phpmind daeil-kwon maitreyeepalghosh andyuan1618 zafrinminoti jessewei sluo6 christfan868 mboge

pandas-videos's Issues

How to add a new time column with new calculated time data?

Hi Just Markham,

I want to add +8 hours to all the existing times and save it to a new column in the DataFrame. I only figured out how to create a new column. But I don't know how to add my delta time to the new time data.

Electronically yours,
Mr. Biggles

from datetime import timedelta
import pandas as pd

ufo = pd.read_csv('http://bit.ly/uforeports')

print("")
ufo['Time'] = pd.to_datetime(ufo.Time)
print(ufo)

print("")
print("_______________________________________________________________________")
print("")

print("ufo.dtypes")
print(ufo.dtypes)

print("")
print("_______________________________________________________________________")
print("")

# Add a new column with new data.

delta = timedelta(hours=-8)
# calc_utc_time = data_cell + delta

ufo['utc_time'] = ufo.Time
print(ufo)

print("")
print("_______________________________________________________________________")
print("")

print("ufo.dtypes")
print(ufo.dtypes)

print("")
print("_______________________________________________________________________")
print("")

How to remove the text Timestamp when saving DataFrame as Dictionary?

Hi Sensei Markham, @justmarkham

I need to save my DataFrame as Dictionary in order to push my data into my database using Flask SQLAlchemy. But even when I change the DataType for column Time from Datetime to Object. My dictionary variable still contains the text Timestamp, how can I remove this text and parenthesis characters from my dictionary?

ufo_dict_2

{0: {'City': 'Ithaca', 'Colors Reported': nan, 'Shape Reported': 'TRIANGLE', 'State': 'NY', 'Time': Timestamp('1930-06-01 22:00:00')}}

x13_change_DataType_for_Series.py

import pandas as pd

#_______________________________________________________________________________

ufo = pd.read_csv('http://bit.ly/uforeports')

print(ufo.head())
print("")
print(type(ufo))
print("")
print("ufo.dtypes")
print(ufo.dtypes)

print("")
print("_______________________________________________________________________")
print("")

#_______________________________________________________________________________

# Convert Time datatype from Object to Datetime.

ufo['Time'] = pd.to_datetime(ufo.Time)

print(ufo.head())
print("")
print(type(ufo))
print("")
print("ufo.dtypes")
print(ufo.dtypes)

print("")
print("_______________________________________________________________________")
print("")

#_______________________________________________________________________________

# Convert DataFrame to Dictionary.

ufo_dict_1 = ufo.to_dict('index')

print("ufo_dict_1")
print("")
print(dict(list(ufo_dict_1.items())[0:3]))
print("")
print(type(ufo_dict_1))

print("")
print("_______________________________________________________________________")
print("")

#_______________________________________________________________________________

# Convert Time datatype from Datetime to Object.

ufo['Time'] = ufo.Time.astype(object)

print(ufo.head())
print("")
print(type(ufo))
print("")
print("ufo.dtypes")
print(ufo.dtypes)

print("")
print("_______________________________________________________________________")
print("")

#_______________________________________________________________________________

# Convert DataFrame to Dictionary.

ufo_dict_2 = ufo.to_dict('index')

print("ufo_dict_2")
print("")
print(dict(list(ufo_dict_2.items())[0:3]))
print("")
print(type(ufo_dict_2))

print("")
print("_______________________________________________________________________")
print("")

Converting pandas.Timestamp into date only by removing timestamp

You've made a good video but you missed this one thing to explain. Actually the situation is in below code.

class NumpyEncoder(DjangoJSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.ndarray):
            return obj.tolist()
        elif isinstance(obj, pd.Timestamp):
            return str(obj)

    return super().default(obj)

with pd.ExcelFile(default_storage.open(path)) as xls:
    sheet_names = xls.sheet_names
    sheets_df = [pd.read_excel(xls, sheet_name, header=None)
                 for sheet_name in sheet_names]
    worksheets_columns = [list(sheet_df.columns) for sheet_df in sheets_df]
    worksheets_data = [json.loads(json.dumps(
        sheet_df.fillna('').applymap(str).to_numpy(), cls=NumpyEncoder))
                       for sheet_df in sheets_df]

I need to convert pandas.Timestamp in the situation above to date only. Currently shows timestamp also.

Thanks in advance.

The use of DataFrame.Ix

Thanks for your videos! In the 19 video,the use of DataFrame.Ix has been deprecated now.

Drop duplicates from multiple columns

How do I find and remove duplicate rows in multiple columns in pandas?

df.col_1.drop_duplicates()

I have 5 columns how can I do that?

Hi, I have the same problem as you ,need some help

Hi,I had the same problem as you, then followed the steps in this comment and modified it, but it still doesn't work, may I ask how did you finally modify it to solve the problem?

Notebook in text format

Hi,

I just created my account in Github, pardon the simple question.

Is it possible to download the notebooks that you have provided in text format?

Here's the link of the one I'm looking at:
http://nbviewer.jupyter.org/github/justmarkham/pandas-videos/blob/master/pandas.ipynb#25.-How-do-I-work-with-dates-and-times-in-pandas%3F-%28video%29

I'm planning to work with the notebooks without having to type the code one by one.
If it's possible to do this please teach me how! Thank you!