Intro to Pandas¶
Pandas is a key package for working with tabular (table-like) data in Python. There is also a geospatial version which we will explore in the next lab.
Pandas introduces two key classes (Python objects) for working with data:
- Series: a list (or array) of data of some type.
- DataFrame: a table-like data struction with rows and columns. The data within a DataFrame is sorted as Series.
import pandas as pd # import and save it as an abbreviation
s = pd.Series([1, 2, 3, 4, 5]) # create a basic series
s # print out the series
s[0] # you can *mostly* treat a series like a list, for example accessing by index
There are a LOT of ways to create a DataFrame by passing in data and a key benefit of using a well-established open-source package like this is that there is plenty of documentation.
One way to do this is to create a dictionary of the column labels and data you want to include. For example:
data = { # the brackets make a "dictionary" that saves key and value pairs
'City': ["New York", "Dallas", "Los Angeles"],
'State': ["New York", "Texas", "California"],
'# of UTDs': [0, 1, 0]
}
data
We can access particular parts of the dictionary by giving the key (e.g., 'City').
data["City"]
We can also use pandas to transform the data into a DataFrame (basically a table) using the "DataFrame" function from Pandas (which we saved as pd):
df = pd.DataFrame(data)
df
You'll notice that the DataFrame labeled each row with a number. This is called the index which is kind of the like the ID of the row. It allows us to refer to individual rows using the iloc function.
print(type(df.iloc[1])) # this returns a Series
df.iloc[1]
We can similarly refer to columns using their name within brackets, like so:
df["State"]
If you want to view the whole DataFrame, you can just print it, but it is often more useful to only print the first few rows (especially for LONG dataframes). We can do this by calling the head function.
Here, we use <object>.<function name>() syntax because the head function is a method of the DataFrame object and not just a generic Python function. You sometimes need to keep track of:
- What are generic Python functions like
len(that get called bylen(<arguments>) - What are functions from certain packages like
pd.DataFrame(that get called by using the package name, dot, function name) - What are methods that come from certain Objects (that get called by using
<object>.<method>).
df.head()
You can also view how long a DataFrame is using len just like with lists.
len(df) # we can get the length using `len` just like with lists!
For more information on Pandas, check out the 10 minutes to pandas guide.
Data Wrangling Basics¶
To work with real-life data, we can also load SpreadSheet data like Comma-Separated Values (CSV) files. If you look in the File Browser (right) for this folder, you should see a "TEXAS_COUNTY.csv" file. This file is the Texas county-level Social Vulnerability Index (SVI) data for 2022, calculated by the Centers for Disease Control (CDC). This data is used to assess how vulnerable communities are to environmental hazards like natural disasters or pandemics.
We can load the data using the pandas read_csv function. Note that this is a pandas function, so we need to use the following syntax:
svi = pd.read_csv("TEXAS_COUNTY.csv") # load the CSV and save it using the svi variable
svi.head() # let's see what we are working with!
Quick Overviews¶
That's a lot of data and the ellipses (...) in the middle means we aren't seeing it all! Let's explore a few ways to get more info quickly:
print(list(svi.columns)) # we can print the column as a list to get all of them!
svi.info()
Another fun functionality is the describe method that provides quick statistics on the columns.
If you aren't entirely sure of what that outputs mean, remember you always refer to the documentation. Another option is Google or even AI, but it is always better to verify anything you get from random sites or AI just in case.
svi.describe()
Pandas also has some basic visualization functionalities built-in including histograms:
svi["E_TOTPOP"].plot.hist() # plotting a histogram of the svi dataframe's "E_TOTPOP" (estimated population) column
Basic Data Cleaning/Filtering¶
We will walk through a few basic steps to clean and filter data using Pandas. This will be useful when we move onto spatial data (GeoDataFrames) as it all uses DataFrames as a baseline.
We can create new columns or overwrite columns by using that bracket syntax we saw earlier to refer to columns. For example, let's say we want to work with population as thousands. The following code will create a new column called PopDensity and save the E_TOTPOP variable divided by AREA_SQMI variable as the result. It will automatically do this for every row and assign the correct values to each row.
svi["PopDensity"] = svi["E_TOTPOP"] / svi["AREA_SQMI"]
svi["PopDensity"].plot.hist()
If we want to delete that column, we can use the drop function and save the result. Note that just calling svi.drop() doesn't change the svi object, you need to assign svi to the result like so:
svi = svi.drop(columns=["PopDensity"])
svi["PopDensity"].plot.hist() # this won't work because the column was dropped. You'll get a "KeyError" meaning that column name doesn't exist.
Another common task is filtering data! We can use the loc method to filter our data down based on some criteria.
For example, below we filter the svi data to look for counties with a population over 1,000,000 (1 million, note you don't use the commas in the code). This will save the result in a new variable, so our svi dataset remains unaffected.
bigCounties = svi.loc[svi["E_TOTPOP"] > 1000000]
# this uses a fancy trick called "f-strings" to insert variables into the text we want to print!
print(f"There are {len(bigCounties)} counties with a population over 1,000,000! The svi dataset still has {len(svi)} rows though!")
Self-Guided Exercises¶
A few exercises to help you test your knowledge and help you explore the documentation!
- Can you filter the svi dataframe by population (
E_TOTPOP) above 1,000,000 but below 3,000,000? - What county has the highest overall social vulnerability (refer to the documentation in this folder to see the variable name).
- What does the histogram look like for
AREA_SQMI? - How can you sort a dataframe using a column?