data analysis Archives • Ashley Gingeleski

Georgia Tech’s OMSA: Halfway Point Reflections

In fall 2021, I started Georgia Tech’s Online Master’s of Science in Analytics (OMSA). Here are some thoughts so far on the courses I’ve taken and overall experience as I head into my sixth class of the program.

Lettie Pate Whitehead Evans Administration Building at Night — Lettie Pate Whitehead Evans Administration Building at Georgia Tech in Atlanta, December 2022.

OMSA Program Background

Georgia Tech’s OMSA program is one of a few well-known online graduate programs in the data community. As data science and analytics become more mainstream and academia leans further into online curriculums, I assume similar program offerings will continue to grow. I heard about Georgia Tech’s Master’s in Analytics initially through my brother who knew about their online cybersecurity program. My main deciding factors were the online format and price tag of about $10,000 USD. I completely support building data skills through free MOOCs (shoutout to Codecademy and O’Reilly, both of which I still find as great resources). However, I figured the structure, schedule, and breadth of what Georgia Tech offered would keep me accountable in my studies. There are 3 different tracks for coursework: Analytical Tools, Business Analytics, and Computational Data Analytics. I am in the Analytical Tools track.

Class Reviews

CSE 6040: Computing for Data Analysis – Fall Semester 2021

This class works primarily in Python to illustrate computing concepts. The homework assignments and tests were auto-graded from JuPyter notebooks and were open book and open internet. The instant feedback was helpful but overall I found the time restrictions for the midterm and final to be challenging. I found creating a comprehensive Python file with code snippets from the whole class helpful for quick searches on the final. There were some optional course items like a project that could be submitted for extra credit. Aside from some difficulties with overthinking, the class was fairly enjoyable for the range of information it covered.

ISYE 6501: Introduction to Analytics Modeling – Fall Semester 2021

ISYE 6501 is essentially a whirlwind tour of analytics modeling concepts through R. Homework assignments are peer-graded and quizzes allow for a restricted number of note sheets (1 or 2 pages). This was a solid refresher in R and I found the content helpful for other classes like Regression Analysis. The course provided a decent background to the theory and troubleshooting involved in real world analytics problems.

MGT 8803: Business Fundamentals for Analytics – Spring Semester 2022

The class covered as many business concepts as possible including marketing, accounting, and supply chain optimization. There were different professors who helped guide the modules so it was interesting to have that mix for course. The level of straight memorization required to succeed with graded assignments was a bit much for me. Unfortunately this was a required course so whether or not the evaluation style was in my lane, I had to deal with it.

ISYE 6414: Regression Analysis – Summer Semester 2022

Regression Analysis gave a breadth of model building and illustrated underlying concepts behind each. The course uses R and material included cleaning and transforming data, variable selection, and linear and logistic regression. The homework assignments were fairly simple and I found having code snippets prepared in one file for the open book portion of exams to be helpful.

CSE 6242: Data and Visual Analytics – Fall Semester 2022

As one of the advanced requirements for the OMSA program, I was a little hesitant on the learning curve this course was rumored to have. The class was a grand tour of dabbling in different languages and programs like Python, SQL, Spark, and D3. A course project is a huge component of this class and it was nice to collaborate with classmates and translate what we learned into something tangible. Overall the homework assignments were only a major bummer because you could have a solution that looked exactly like the answer, but still get zero or minimal points from the auto-grader. There are no explicit homework solutions and I struggled with understanding what exactly needed to be corrected when I did not get full credit. Luckily, there were ample opportunities for extra credit to make up for any missed points from the homework.

ISYE 7406: Data Mining and Statistical Learning – Spring Semester 2023

I’m currently a couple weeks into the course for the Spring 2023 semester but so far the blend of theoretical background for statistics and practical R analysis has been manageable.

Balancing Professional, Academic, and Social Obligations

I attended a meetup in December 2022 at the main Atlanta campus for the Analytics and Cybersecurity programs and reflected on my personal experience after speaking with fellow students and alumni. There seemed to be a decent mix of professional backgrounds and mostly everyone I spoke with also worked full time for the duration of the program.

Personally, I have found that one course a semester has worked best for me. The only semester I doubled up was my first semester for two of the required core classes: ISYE 6501 and CSE 6040. In hindsight I think I would have been better off just taking one course instead of constantly feeling like I was flipping back and forth between the two.

For time spent on homework and studying, I have found that chipping away a little bit everyday has given me the best results thus far. It tends to keep concepts relevant as opposed to taking couple-day breaks between learning material. There have still been times where I have taken breaks for travel or vacation but a lot of the courses allow for some leniency for frontloading assignments and learning with the schedules for module releases.

Overall OMSA Experience

I’ve been pleasantly surprised in the program so far but still have plenty of days where I have the same frustrations that anyone else might with learning new material. The practical translation of concepts from the courses seems to be the main draw and highlight for folks in the OMSA program. I look forward to continuing to build skills in future classes. I am likely taking MGT 6203 (Data Analytics in Business) for summer 2023 and then ISYE 6669 (Deterministic Optimization) for fall 2023.

In Data Science

Species Richness and Distribution in the National Parks with pandas

AshleyMay 23, 2021August 29, 2022Leave a comment

Here’s how to use Python and pandas to explore species data for the United States National Parks to find the average species richness and the distribution of species categories. This goes over some of the built-in functions in pandas and how to use those for exploratory data analysis. The source data is available via Kaggle, or the National Parks Species website. The associated Github repository for a more in-depth look at the two Jupyter Notebooks for the code below is available here.

An elk sits off a trail at Yellowstone National Park as visitors walk by. — An elk sits near a trail in Yellowstone National Park, May 2017.

Setup

Import the necessary packages, including pandas, matplotlib, seaborn, and math.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb
import math

Load the species dataset and the parks dataset with pandas. Given the size and that I used Jupyter Notebook for this, I give the low memory argument a value of False for the species data so it loads without too much trouble. The dataframes can be given any name.

species_data = pd.read_csv('species.csv', low_memory=False)
parks_data = pd.read_csv('parks.csv')

Preview the datasets with the head() and info() pandas functions. The columns of focus will be park name and category for the species dataframe, and park name and acres for the parks dataframe.

species_data.head()

A pandas dataframe shows columns and rows for species data.

species_data.info()

A preview of the species dataframe info shows all the columns, their counts, and data types.

parks_data.head()

A pandas dataframe shows columns and rows for parks data.

Species Richness

Species richness is the quantification of species in a given area and can be a helpful metric for biodiversity.

For this, we’ll want to merge the species data and the parks data. The species data will need to be re-formatted beforehand.

Out of the box, the species data has one record per type of species per row. This will need to be turned into individual counts per column to get the number of species in each park. Use the groupby() function in pandas specifying the park name column, and then use count().

all_species_data = species_data.groupby(['Park Name']).count()

A pandas dataframe shows columns per each national park and their respective counts.

Next, I narrow down to just the Species ID column and change the name of it to reflect just ‘Species’.

species_counts = all_species_data[['Species ID']].copy()
species_counts = species_counts.rename(columns={'Species ID' : 'Species'}

The parks data will need the names set as the index as well. Then, the two dataframes will be in good shape to merge. Specify the two dataframes, and give the left_index and right_index arguments values of true because they are the same between each.

parks_data = parks_data.set_index('Park Name')
richness_data = species_counts.merge(parks_data, left_index=True, right_index=True)

Preview the newly merged dataframe to confirm it looks correct.

richness_data.head()

A pandas dataframe shows merged species and park data columns.

To take this one step further, estimate the number of species per acre using the species counts. Create a function to take the species count column and acres column and divide them, and normalize the result to an integer with math’s floor() function.

def species_abundance(df):
   return df.apply(
       lambda row:
         math.floor(
           row['Acres'] / row['Species']),
       axis=1
   )

Create a new column in the dataframe and apply the species abundance function.

richness_data['Species Abundance'] = species_abundance(richness_data)
richness_data.head()

A dataframe show the number of species and abundance of species per acre in individual national parks.

Calculate the mean of species per acre in all the parks. There are a lot of variables to consider that might affect this number, but it serves its purpose as a quick summary statistic to give us about 525 species per acre.

print(richness_data['Species Abundance'].mean())

Species Type Distribution

Species distribution in this setting will be the ratio of species types throughout each park. The different species categories in this data set are: ‘Mammal’, ‘Bird’, ‘Reptile’, ‘Amphibian’, ‘Fish’, ‘Vascular Plant’, ‘Spider/Scorpion’, ‘Insect’, ‘Invertebrate’, ‘Fungi’, ‘Nonvascular Plant’, ‘Crab/Lobster/Shrimp’, ‘Slug/Snail’, ‘Algae’.

To extract just the park names and categories, create a new dataframe with just these columns.

types = species_data[['Park Name', 'Category']].copy()

A pandas dataframe shows one row per species record of a given park and category.

Use pandas’ groupby() function to group by park name and category. Then, use the size function to specify the number of rows for the unstack function, which will create a column for each of the unique row values. I give the argument of fill_value for the unstack function a value of 0, to keep anything with NaN values consistent for math operations.

df = types.groupby(['Park Name','Category']).size().unstack(fill_value=0)
df.head()

A pandas dataframe shows the number of species in each category per park.

Next, take the counts from each and put all on the same scale of out of 100. Use pandas’ div() function to divide based on the sum of the dataframe per each row. Then, multiply each value by 100 for better readability and translation for visuals.

ratios = df.div(df.sum(axis=1), axis=0).multiply(100)
ratios.head()

A pandas dataframe shows raw values for each park's species category ratios.

To clean the dataframe values further, round each variable to 2 decimal places using Python’s round() function.

rounded = ratios.round(2)
rounded.head()

A pandas dataframe shows rounded values for ratios in each park's species category.

Create a box plot using matplotlib and seaborn to show the breakdown of species categories across all parks.

f, ax = plt.subplots(figsize=(12, 8))
sb.boxplot(data=rounded, orient='h')
ax.xaxis.grid(True)
ax.set(ylabel="")
ax.set(xlim=(0,100))
sb.despine(trim=True, left=True)
plt.title('Species Category Distribution in the National Parks', fontsize=16)
plt.show()

A boxplot shows the species category types and their distributions.

Optionally, export the species categories dataframe as a CSV for further use.

Park Name	Algae	Amphibian	Bird	Crab/Lobster/Shrimp	Fish	Fungi	Insect	Invertebrate	Mammal	Nonvascular Plant	Reptile	Slug/Snail	Spider/Scorpion	Vascular Plant
Acadia National Park	0.0	0.88	21.3	0.0	2.22	0.0	0.0	0.0	3.22	0.0	0.64	0.0	0.0	71.74
Arches National Park	0.0	0.76	19.56	0.0	1.05	0.0	0.0	0.0	5.63	0.0	1.91	0.0	0.0	71.09
Badlands National Park	0.0	0.72	17.21	0.0	1.73	12.46	17.21	0.07	4.61	0.0	0.94	0.0	0.07	45.0
Big Bend National Park	0.0	0.57	18.29	0.0	2.34	0.0	0.0	0.0	3.92	2.12	2.73	0.0	0.0	70.03
Biscayne National Park	0.0	0.46	13.5	0.0	47.39	0.0	0.64	1.97	1.62	0.0	2.32	0.0	0.0	32.1
Black Canyon of the Gunnison National Park	0.0	0.18	15.82	0.0	1.45	0.0	0.0	0.0	6.06	0.0	0.99	0.0	0.0	75.5
Bryce Canyon National Park	0.0	0.31	16.87	0.0	0.08	0.0	0.0	0.0	5.91	0.0	1.01	0.0	0.0	75.82
Canyonlands National Park	0.0	0.57	17.99	0.0	2.7	0.0	0.0	0.0	6.21	0.0	1.8	0.0	0.0	70.73
Capitol Reef National Park	0.0	0.38	15.84	0.0	0.96	0.0	0.0	0.0	4.66	0.0	1.34	0.0	0.0	76.82
Carlsbad Caverns National Park	0.0	0.98	23.89	0.0	0.33	0.0	0.0	0.0	5.99	0.0	4.04	0.0	0.0	64.78
Channel Islands National Park	3.24	0.21	18.94	0.58	14.48	0.0	0.11	10.4	2.33	3.61	0.58	1.75	0.05	43.71
Congaree National Park	3.19	1.85	8.62	0.26	2.8	12.02	26.58	0.65	1.68	0.3	2.15	0.9	0.9	38.09
Crater Lake National Park	5.8	0.53	6.7	1.06	0.35	5.11	26.44	1.81	2.55	5.13	0.53	0.24	0.45	43.3
Cuyahoga Valley National Park	0.0	1.24	12.67	0.41	4.38	0.0	11.7	1.29	2.42	0.0	1.18	0.77	0.1	63.83
Death Valley National Park	1.35	1.6	11.96	0.86	0.2	2.77	20.34	0.41	4.78	0.5	2.03	1.62	0.72	50.87
Denali National Park and Preserve	0.0	0.08	13.56	0.0	1.06	2.2	3.86	0.23	3.26	12.05	0.0	0.0	0.0	63.71
Dry Tortugas National Park	0.0	0.0	33.37	0.0	33.14	0.0	0.0	4.95	0.71	0.0	0.59	0.0	0.0	27.24
Everglades National Park	0.0	0.82	17.75	0.0	20.44	0.0	0.0	0.0	2.02	0.0	2.93	0.0	0.0	56.05
Gates Of The Arctic National Park and Preserve	0.0	0.07	9.9	0.0	1.26	35.03	0.0	0.0	2.88	0.0	0.0	0.0	0.0	50.85
Glacier Bay National Park and Preserve	3.27	0.26	13.18	4.91	18.34	0.1	1.79	5.98	2.96	5.01	0.15	1.89	0.15	42.0
Glacier National Park	0.08	0.23	10.84	0.23	1.06	10.8	7.71	0.08	2.7	15.81	0.16	0.78	0.0	49.53
Grand Canyon National Park	0.0	0.57	17.39	0.0	1.11	0.0	2.17	0.04	4.04	0.0	2.9	0.08	5.42	66.29
Grand Teton National Park	0.05	0.34	13.1	0.05	1.03	1.38	7.59	0.25	3.65	0.0	0.25	0.2	0.05	72.07
Great Basin National Park	0.0	0.83	12.44	0.23	0.79	0.57	17.19	1.09	3.88	0.6	2.19	1.13	0.68	58.39
Great Sand Dunes National Park and Preserve	0.0	0.63	25.21	0.0	0.63	0.0	0.11	0.0	7.14	0.11	0.84	0.0	0.0	65.34
Great Smoky Mountains National Park	0.0	0.92	4.11	0.15	1.62	9.54	36.45	1.27	1.42	7.97	0.77	1.39	1.57	32.83
Guadalupe Mountains National Park	0.0	0.69	15.58	0.29	0.17	3.78	6.59	0.4	4.35	0.29	3.21	3.32	0.11	61.23
Haleakala National Park	0.0	0.12	1.71	0.7	0.23	2.56	41.12	1.43	0.58	8.8	0.39	1.82	1.78	38.76
Hawaii Volcanoes National Park	0.0	0.12	2.37	1.7	0.12	0.21	43.33	5.25	0.45	4.09	0.39	2.15	3.06	36.75
Hot Springs National Park	1.23	1.38	19.85	0.46	4.62	0.0	0.77	1.13	2.67	0.92	2.67	0.1	0.0	64.21
Isle Royale National Park	0.0	0.93	18.68	1.0	4.51	0.0	0.0	1.5	1.86	0.07	0.36	0.0	0.0	71.08
Joshua Tree National Park	0.22	0.22	13.12	0.0	0.04	1.66	14.91	0.44	2.92	0.26	2.27	0.0	6.67	57.28
Katmai National Park and Preserve	0.0	0.08	18.12	0.33	3.59	8.57	0.57	2.37	4.41	1.63	0.0	0.57	0.0	59.76
Kenai Fjords National Park	0.0	0.09	22.23	0.0	3.97	0.19	0.28	3.97	5.2	0.0	0.0	0.38	0.0	63.67
Kobuk Valley National Park	0.0	0.1	12.2	0.0	2.54	32.49	0.0	0.0	3.71	2.54	0.0	0.0	0.0	46.44
Lake Clark National Park and Preserve	0.0	0.05	9.62	0.05	2.74	10.96	0.15	0.3	2.49	14.0	0.0	0.0	0.0	59.64
Lassen Volcanic National Park	0.11	0.95	13.63	0.83	1.11	2.84	5.29	1.0	5.56	8.9	1.22	0.33	0.0	58.21
Mammoth Cave National Park	0.0	1.32	8.4	0.04	4.88	0.0	10.8	2.16	2.24	0.04	1.68	0.0	0.08	68.35
Mesa Verde National Park	0.0	0.64	19.07	0.0	0.0	0.0	0.0	0.0	6.92	0.0	1.69	0.0	0.0	71.68
Mount Rainier National Park	0.0	0.92	10.73	0.0	1.32	1.78	2.7	0.0	3.96	20.48	0.29	0.0	0.0	57.83
North Cascades National Park	0.0	0.36	6.72	0.0	0.98	16.03	16.62	0.0	2.35	11.45	0.3	0.0	1.25	43.95
Olympic National Park	0.0	0.82	15.91	0.0	4.98	0.0	4.47	0.0	4.11	0.0	0.31	0.0	0.0	69.4
Petrified Forest National Park	0.0	0.94	28.6	0.0	0.0	0.12	0.0	0.0	7.27	0.12	2.46	0.0	0.0	60.49
Pinnacles National Park	0.0	0.71	12.01	0.56	0.42	1.91	22.81	2.19	4.24	0.0	2.05	0.78	0.71	51.62
Redwood National Park	1.76	0.52	7.94	1.93	3.91	21.6	11.79	5.29	2.44	3.98	0.62	2.33	0.11	35.77
Rocky Mountain National Park	4.76	0.16	8.79	1.24	0.38	9.71	21.45	1.52	2.35	13.2	0.1	0.32	0.7	35.34
Saguaro National Park	0.0	0.55	13.41	0.05	0.0	0.11	0.0	0.0	5.56	0.0	3.38	0.0	0.0	76.94
Sequoia and Kings Canyon National Parks	0.0	0.65	11.03	0.0	0.95	0.0	0.0	0.0	4.46	0.0	1.2	0.0	0.0	81.7
Shenandoah National Park	0.0	0.86	5.76	0.06	0.88	16.43	6.83	0.04	1.35	7.52	0.82	0.04	0.09	59.31
Theodore Roosevelt National Park	0.0	0.69	19.14	0.09	2.75	0.09	6.27	0.26	5.67	5.24	1.12	0.17	0.17	58.37
Voyageurs National Park	0.0	1.03	16.38	0.0	3.99	0.21	2.27	0.48	4.34	0.76	0.41	0.0	0.0	70.13
Wind Cave National Park	0.0	0.5	16.85	0.0	0.57	3.08	7.53	0.0	6.38	0.0	0.86	1.79	0.0	62.44
Wrangell - St Elias National Park and Preserve	0.0	0.11	11.75	0.0	5.18	0.0	3.23	0.22	3.45	0.39	0.06	0.06	0.0	75.56
Yellowstone National Park	5.14	0.23	8.32	1.59	0.48	0.28	41.05	1.97	1.97	0.38	0.23	1.49	1.08	35.8
Yosemite National Park	0.0	0.72	12.93	0.0	0.48	0.0	0.0	0.0	4.21	0.0	1.05	0.0	0.0	80.6
Zion National Park	0.0	0.39	16.76	0.0	0.84	0.0	0.0	0.0	4.45	0.0	1.67	0.0	0.0	75.89

Summary

Further exploration of the species category and count outputs might involve comparing a select number of parks against each other. Unique factors such as location, park size, and biomes provide opportunities for further analysis and insights.

In Data Science

Analysis of USDA Plant Families Data Using Pandas

AshleySeptember 26, 2019September 26, 20191 Comment

The United States Department of Agriculture PLANTS database provides general information about plant species across the country. Given 3 states, I wanted to visualize which plant families are present in each and which state(s) hold the most species in each family. To accomplish this task, I used Python’s pandas, matplotlib, and seaborn libraries for analysis.

Different groups of plants, including wildflowers, are arranged in a rock bed in the desert. — Various plant species in Death Valley National Park, California.

Initial Setup

Before beginning, I import pandas, matplotlib, and seaborn.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb

Gathering Data

I pulled data sets from the USDA website for New York, Idaho, and California. The default encoding is in Latin-1 for exported text files. When importing into pandas, the encoding must be specified to work properly.

ny_list = pd.read_csv('ny_list.txt', encoding='latin-1')
ca_list = pd.read_csv('ca_list.txt', encoding='latin-1')
id_list = pd.read_csv('id_list.txt', encoding='latin-1')

I double check the files have been loaded correctly into dataframe format using head().

A preview of a table shows 5 entries of various plants that are present in New York state, including their USDA symbol, synonym symbol, scientific name, common name, and plant family. — Initial import of New York state plants list categorized by USDA symbol, synonym symbol, scientific name with author, national common name, and family.

Cleaning Data

The major point of interest in the imported dataframe is the ‘Family’ column. I create a new dataframe organized by this column and returning the count from each row.

ny_fam = ny_list.groupby('Family').count()

A table shows plants grouped by taxonomic family for New York state. — Initial dataframe for New York plant data grouped by taxonomic family.

Next, I remove the unwanted columns. I’ve chosen only to keep the ‘Symbol’ column as a representation of count because this variable is required for every plant instance.

ny_fam_1 = ny_fam.drop(['Synonym Symbol', 'Scientific Name with Author', 'National Common Name'], axis=1)

Then, I change the column name from ‘Symbol’ to ‘{State} Count’ to lend itself for merging the dataframes without confusion.

ny_fam_1 = ny_fam_1.rename(columns = {"Symbol":"NY Count"})

Two tables show a before and after image of a table where the column named 'Symbol' changes to 'NY Count.' — Column before (left) and after (right) a name change.

I complete the same process for the California and Idaho data.

ca_fam = ca_list.groupby('Family').count()
ca_fam_1 = ca_fam.drop(['Synonym Symbol', 'Scientific Name with Author', 'National Common Name'], axis=1)
ca_fam_1 = ca_fam_1.rename(columns = {"Symbol":"CA Count"})
id_fam = id_list.groupby('Family').count()
id_fam_1 = id_fam.drop(['Synonym Symbol', 'Scientific Name with Author', 'National Common Name'], axis=1)
id_fam_1 = id_fam_1.rename(columns = {"Symbol":"ID Count"})

Reset the index to prepare the data frames for outer merges based on column names. The index is set to ‘Family’ as default, from the initial data frame creation using the count() function. To discourage any unwanted changes, I create a copy of each data frame as I go.

ny_fam_2 = ny_fam_1.copy()
ny_fam_2 = ny_fam_2.reset_index()
ca_fam_2 = ca_fam_1.copy()
ca_fam_2 = ca_fam_2.reset_index()
id_fam_2 = id_fam_1.copy()
id_fam_2 = id_fam_2.reset_index()

Two tables show a before and after image of New York state plant family data. The first table shows plant families as the index and the second table shows plant families as a column, with a new numerical index. — New York dataframe before (left) and after (right) the index was reset to make the plant families a column.

Merging Data

To preserve all the plant species regardless of presence in each individual state, I perform outer merges. This will allow for areas without data to be filled with zeros after the family counts are combined.

combo1 = pd.merge(ny_fam_2, ca_fam_2, how='outer')
combo2 = pd.merge(combo1, id_fam_2, how='outer')

A table shows the combined plant family data for New York, California, and Idaho. — Plant family table with counts from each state, before formatting.

pd.options.display.float_format = '{:,.0f}'.format
combo2 = combo2.fillna(0)

A table shows combined data for plant families in New York, California, and Idaho where the numbers are formatted to show zero decimal places and any instances of no data are replaced by zeroes. — Plant family table with counts from each state, formatted to drop decimals and replace NaNs with zeros.

Creating a New Column

I added a column to aid in visualizations. I created a function to return the state with the highest presence of each plant family based on the existing columns.

def presence(row):
    if row['NY Count'] > row['CA Count'] and row['NY Count'] > row['ID Count']:
        return 'NY'
    elif row['CA Count'] > row['NY Count'] and row['CA Count'] > row['ID Count']:
        return 'CA'
    elif row['ID Count'] > row['NY Count'] and row['ID Count'] > row['CA Count']:
        return 'ID'
    elif row['NY Count'] == row['CA Count'] and row['NY Count'] > row['ID Count']:
        return 'CA/NY'
    elif row['CA Count'] == row['ID Count'] and row['CA Count'] > row['NY Count']:
        return 'CA/ID'
    elif row['ID Count'] == row['NY Count'] and row['ID Count'] > row['CA Count']:
        return 'ID/NY'
    else:
        return 'Same'
    
combo2['Highest Presence'] = combo2.apply(presence, axis=1)

A table of plant families from New York, California, and Idaho shows counts for each family and a new column that names which state has the highest presence of each plant family. — Table with added column to indicate highest presence of species count within each plant family.

Below is the full table of all plant families in the dataframe.

	Family	NY Count	CA Count	ID Count	Highest Presence
0	Acanthaceae	7	7	0	CA/NY
1	Acarosporaceae	1	1	18	ID
2	Aceraceae	46	29	24	NY
3	Acoraceae	5	2	4	NY
4	Actinidiaceae	3	0	0	NY
5	Adoxaceae	2	0	0	NY
6	Agavaceae	4	48	0	CA
7	Aizoaceae	5	42	0	CA
8	Alismataceae	64	53	33	NY
9	Amaranthaceae	78	82	38	CA
10	Amblystegiaceae	64	0	0	NY
11	Anacardiaceae	46	47	22	CA
12	Andreaeaceae	3	0	0	NY
13	Annonaceae	3	0	0	NY
14	Anomodontaceae	8	0	0	NY
15	Apiaceae	190	372	257	CA
16	Apocynaceae	48	60	42	CA
17	Aquifoliaceae	21	3	0	NY
18	Araceae	26	18	3	NY
19	Araliaceae	26	6	8	NY
20	Aristolochiaceae	28	9	3	NY
21	Asclepiadaceae	50	67	20	CA
22	Aspleniaceae	23	7	6	NY
23	Asteraceae	2,057	3,858	2,260	CA
24	Aulacomniaceae	7	0	0	NY
25	Azollaceae	4	4	0	CA/NY
26	Bacidiaceae	1	1	0	CA/NY
27	Balsaminaceae	10	6	11	ID
28	Bartramiaceae	11	0	0	NY
29	Berberidaceae	21	59	25	CA
30	Betulaceae	86	43	53	NY
31	Bignoniaceae	9	11	0	CA
32	Blechnaceae	5	11	7	CA
33	Boraginaceae	127	478	263	CA
34	Brachytheciaceae	59	0	0	NY
35	Brassicaceae	468	1,123	774	CA
36	Bruchiaceae	4	0	0	NY
37	Bryaceae	22	2	0	NY
38	Buddlejaceae	4	5	0	CA
39	Butomaceae	2	0	2	ID/NY
40	Buxaceae	5	0	0	NY
41	Buxbaumiaceae	5	0	0	NY
42	Cabombaceae	6	6	3	CA/NY
43	Cactaceae	10	238	78	CA
44	Callitrichaceae	16	18	11	CA
45	Calycanthaceae	7	2	0	NY
46	Campanulaceae	76	146	57	CA
47	Cannabaceae	13	8	10	NY
48	Cannaceae	4	0	0	NY
49	Capparaceae	18	49	26	CA
50	Caprifoliaceae	182	107	81	NY
51	Caryophyllaceae	338	506	414	CA
52	Celastraceae	24	18	4	NY
53	Ceratophyllaceae	8	5	5	NY
54	Cercidiphyllaceae	2	0	0	NY
55	Chenopodiaceae	245	404	245	CA
56	Cistaceae	44	27	0	NY
57	Cladoniaceae	5	2	2	NY
58	Clethraceae	4	0	0	NY
59	Climaciaceae	5	0	0	NY
60	Clusiaceae	52	15	12	NY
61	Commelinaceae	37	14	0	NY
62	Convolvulaceae	68	130	17	CA
63	Cornaceae	55	34	36	NY
64	Crassulaceae	62	230	57	CA
65	Cucurbitaceae	41	49	6	CA
66	Cupressaceae	39	130	30	CA
67	Cuscutaceae	31	56	31	CA
68	Cyperaceae	1,016	733	663	NY
69	Dennstaedtiaceae	8	5	5	NY
70	Diapensiaceae	8	0	0	NY
71	Dicranaceae	20	0	0	NY
72	Dioscoreaceae	6	0	0	NY
73	Dipsacaceae	17	10	8	NY
74	Ditrichaceae	9	2	0	NY
75	Droseraceae	8	11	6	CA
76	Dryopteridaceae	121	65	71	NY
77	Ebenaceae	7	7	0	CA/NY
78	Elaeagnaceae	16	10	12	NY
79	Elatinaceae	6	14	4	CA
80	Empetraceae	13	6	0	NY
81	Entodontaceae	8	0	0	NY
82	Ephemeraceae	5	0	0	NY
83	Equisetaceae	48	36	51	ID
84	Ericaceae	236	310	110	CA
85	Eriocaulaceae	7	3	0	NY
86	Euphorbiaceae	111	233	66	CA
87	Fabaceae	604	1,855	871	CA
88	Fagaceae	96	91	0	NY
89	Fissidentaceae	22	1	1	NY
90	Flacourtiaceae	2	0	0	NY
91	Fontinalaceae	9	0	0	NY
92	Fumariaceae	31	28	20	NY
93	Funariaceae	21	0	0	NY
94	Gentianaceae	72	162	122	CA
95	Geraniaceae	34	80	25	CA
96	Ginkgoaceae	2	0	0	NY
97	Grimmiaceae	2	3	3	CA/ID
98	Grossulariaceae	46	150	72	CA
99	Haemodoraceae	7	0	0	NY
100	Haloragaceae	37	28	20	NY
101	Hamamelidaceae	8	2	0	NY
102	Hippocastanaceae	14	2	0	NY
103	Hippuridaceae	2	2	2	Same
104	Hydrangeaceae	24	53	14	CA
105	Hydrocharitaceae	45	44	30	NY
106	Hydrophyllaceae	15	336	89	CA
107	Hylocomiaceae	8	0	0	NY
108	Hymeneliaceae	1	1	4	ID
109	Hymenophyllaceae	2	0	0	NY
110	Hypnaceae	20	0	0	NY
111	Iridaceae	46	114	26	CA
112	Isoetaceae	39	30	28	NY
113	Juglandaceae	52	10	2	NY
114	Juncaceae	162	177	143	CA
115	Juncaginaceae	9	22	13	CA
116	Lamiaceae	399	413	182	CA
117	Lardizabalaceae	2	0	0	NY
118	Lauraceae	12	11	0	NY
119	Lecanoraceae	3	1	1	NY
120	Lemnaceae	27	45	17	CA
121	Lentibulariaceae	37	23	17	NY
122	Leucobryaceae	2	0	0	NY
123	Liliaceae	263	741	243	CA
124	Limnanthaceae	3	36	3	CA
125	Linaceae	36	52	20	CA
126	Lycopodiaceae	114	9	47	NY
127	Lygodiaceae	2	0	0	NY
128	Lythraceae	25	26	16	CA
129	Magnoliaceae	20	0	0	NY
130	Malvaceae	66	284	62	CA
131	Marsileaceae	2	15	12	CA
132	Melastomataceae	12	0	0	NY
133	Meliaceae	3	3	0	CA/NY
134	Menispermaceae	2	0	0	NY
135	Menyanthaceae	10	7	3	NY
136	Mniaceae	10	0	0	NY
137	Molluginaceae	4	9	3	CA
138	Monotropaceae	19	31	21	CA
139	Moraceae	21	16	5	NY
140	Myricaceae	22	5	0	NY
141	Najadaceae	25	21	10	NY
142	Nelumbonaceae	9	5	0	NY
143	Nyctaginaceae	30	157	8	CA
144	Nymphaeaceae	50	23	30	NY
145	Oleaceae	39	49	0	CA
146	Onagraceae	237	661	314	CA
147	Ophioglossaceae	60	53	48	NY
148	Orchidaceae	285	175	173	NY
149	Orobanchaceae	22	74	51	CA
150	Orthotrichaceae	11	0	0	NY
151	Osmundaceae	10	0	0	NY
152	Oxalidaceae	70	58	47	NY
153	Paeoniaceae	2	4	2	CA
154	Papaveraceae	38	98	26	CA
155	Parmeliaceae	1	1	1	Same
156	Pedaliaceae	11	25	8	CA
157	Phytolaccaceae	4	7	0	CA
158	Pinaceae	52	113	65	CA
159	Plantaginaceae	64	78	33	CA
160	Platanaceae	9	8	0	NY
161	Plumbaginaceae	16	22	0	CA
162	Poaceae	1,927	2,347	1,507	CA
163	Podostemaceae	3	0	0	NY
164	Polemoniaceae	35	637	279	CA
165	Polygalaceae	29	17	0	NY
166	Polygonaceae	331	917	435	CA
167	Polypodiaceae	9	14	9	CA
168	Polytrichaceae	23	0	0	NY
169	Pontederiaceae	17	17	6	CA/NY
170	Portulacaceae	24	203	120	CA
171	Potamogetonaceae	115	83	103	NY
172	Pottiaceae	34	2	1	NY
173	Primulaceae	67	104	102	CA
174	Pteridaceae	16	111	39	CA
175	Pyrolaceae	54	65	67	ID
176	Ranunculaceae	288	434	412	CA
177	Resedaceae	7	8	0	CA
178	Rhamnaceae	18	202	20	CA
179	Rosaceae	1,305	803	609	NY
180	Rubiaceae	157	230	72	CA
181	Ruppiaceae	11	17	7	CA
182	Rutaceae	18	12	0	NY
183	Salicaceae	284	352	383	ID
184	Salviniaceae	5	3	0	NY
185	Santalaceae	9	5	9	ID/NY
186	Sapindaceae	5	37	3	CA
187	Sarraceniaceae	11	16	0	CA
188	Saururaceae	2	3	0	CA
189	Saxifragaceae	55	219	234	ID
190	Scheuchzeriaceae	5	5	5	Same
191	Schistostegaceae	2	0	2	ID/NY
192	Schizaeaceae	2	0	0	NY
193	Scrophulariaceae	430	1,146	556	CA
194	Selaginellaceae	6	15	14	CA
195	Sematophyllaceae	6	0	0	NY
196	Simaroubaceae	4	7	4	CA
197	Smilacaceae	22	3	0	NY
198	Solanaceae	118	244	63	CA
199	Sparganiaceae	24	22	24	ID/NY
200	Sphagnaceae	42	3	1	NY
201	Staphyleaceae	2	2	0	CA/NY
202	Sterculiaceae	2	30	0	CA
203	Styracaceae	7	9	0	CA
204	Symplocaceae	5	0	0	NY
205	Taxaceae	10	4	2	NY
206	Teloschistaceae	1	1	1	Same
207	Tetraphidaceae	2	0	0	NY
208	Theliaceae	3	0	0	NY
209	Thelypteridaceae	26	9	13	NY
210	Thuidiaceae	3	0	0	NY
211	Thymelaeaceae	6	2	0	NY
212	Tiliaceae	22	0	0	NY
213	Trapaceae	5	0	0	NY
214	Tropaeolaceae	2	2	0	CA/NY
215	Typhaceae	6	10	5	CA
216	Ulmaceae	23	19	10	NY
217	Urticaceae	52	61	32	CA
218	Valerianaceae	13	59	27	CA
219	Verbenaceae	53	126	7	CA
220	Violaceae	176	120	79	NY
221	Viscaceae	12	44	6	CA
222	Vitaceae	55	18	4	NY
223	Vittariaceae	2	0	0	NY
224	Xyridaceae	15	0	0	NY
225	Zannichelliaceae	5	5	5	Same
226	Zosteraceae	5	8	0	CA
227	Zygophyllaceae	5	31	5	CA
228	Aloaceae	0	2	0	CA
229	Aponogetonaceae	0	2	0	CA
230	Arecaceae	0	15	0	CA
231	Basellaceae	0	4	0	CA
232	Bataceae	0	2	0	CA
233	Burseraceae	0	2	0	CA
234	Caulerpaceae	0	2	0	CA
235	Crossosomataceae	0	18	9	CA
236	Cyatheaceae	0	3	0	CA
237	Cymodoceaceae	0	5	0	CA
238	Datiscaceae	0	2	0	CA
239	Elaeocarpaceae	0	4	0	CA
240	Ephedraceae	0	16	0	CA
241	Fouquieriaceae	0	3	0	CA
242	Frankeniaceae	0	6	0	CA
243	Garryaceae	0	11	0	CA
244	Gracilariaceae	0	2	0	CA
245	Gunneraceae	0	3	0	CA
246	Halymeniaceae	0	2	0	CA
247	Krameriaceae	0	8	0	CA
248	Lennoaceae	0	6	0	CA
249	Loasaceae	0	84	29	CA
250	Melianthaceae	0	2	0	CA
251	Myoporaceae	0	2	0	CA
252	Myrtaceae	0	32	0	CA
253	Parkeriaceae	0	4	0	CA
254	Passifloraceae	0	6	0	CA
255	Pittosporaceae	0	8	0	CA
256	Punicaceae	0	2	0	CA
257	Rafflesiaceae	0	2	0	CA
258	Scouleriaceae	0	3	3	CA/ID
259	Simmondsiaceae	0	4	0	CA
260	Stereocaulaceae	0	2	0	CA
261	Tamaricaceae	0	12	3	CA
262	Ulvaceae	0	3	0	CA
263	Verrucariaceae	0	0	2	ID

Visualizing the Data

I created a count plot using seaborn to show which states, or state combinations, have the highest variety within each plant family.

base_color = sb.color_palette()[2]
sb.countplot(data=combo4, x='Highest Presence', color="#B6D1BE", order=combo4['Highest Presence'].value_counts().index)
n_points = combo4.shape[0]
cat_counts = combo4['Highest Presence'].value_counts()
locs, labels = plt.xticks()
for loc, label in zip(locs, labels):
    count = cat_counts[label.get_text()]
    pct_string = count
    plt.text(loc, count+5, pct_string, ha='center', color='black', fontsize=12)
plt.xticks(rotation=25)
plt.xlabel('')
plt.ylabel('')
plt.title('Highest Concentration of Plant Families by State', fontsize=14, y=1.05)
plt.ylim(0, 140)
sb.despine();

A vertical bar graph show the highest concentration of plant families organized by state, where the concentrations are high to low as follows: California, New York, California/New York, Idaho, all the states tie, Idaho/New York, and California/Idaho. — Shows the count of plant families with the highest concentration in each state.

Further Considerations

There are many factors that play into plant family diversity. The comparison of plant families in New York, California, and Idaho was purely out of curiosity. Further investigations should take into account each state’s ecosystem types and land usage and ownership that may influence species diversity.

In Data Science

Statistics Principles for Data Analysis

AshleyMay 24, 2019May 27, 2019Leave a comment

Recently, I had to brush up on statistics terms for a data analyst exam. I had trouble pulling together old course notes to create a quick, cohesive study guide. Below, overarching concepts are from the test’s public posting and my notes are derived from a quantitative statistics textbook.

Central Tendency
- Mean = the average of a distribution
- Median = a distribution’s midpoint
- Mode = the variable which occurs most often in a distribution
Variability
- The distribution of data, also known as spread
- Five-number summary: Minimum, Q1, Median (Q2), Q3, Maximum
  - Represented through boxplots graphically
- Summarized through quartiles:
  - Q1: median of all values to left of Q2
  - Q2: median (50^th percentile) of all values in distribution
  - Q3: median of all values to right of Q2
- Variance: s^2 = SUM((value minus mean)^2 for all values)) / (number of values-1)
- Standard deviation: square root of the variance (s^2)
Normal Distribution: bell-curve distribution of data
Hypothesis Testing: Examine evidence against a null hypothesis, hypotheses referring to populations or models and not a certain outcome
- Compare claims
- Null hypothesis: statement challenged in significance testing
  - Example: There is not a difference between means.
- Alternative hypothesis: statement suspected as true instead of the null hypothesis
  - Example: The means are not the same.
- Accept or reject null hypothesis based on a certain p-value.
- p-value: the likelihood that the test statistic would be a value equal or higher than what is observed
- Smaller p-values signify stronger evidence against the null hypothesis in question. Often, an alpha value of 0.05 is used. Evidence would be so strong that something outside the p-value should only occur 5 out of every 100 times.
Statistical Significance Testing: Achieved at the level where the p-value is equal or less than alpha.
Probability: The proportion of times an outcome would occur given many repeated tests.
Correlation
- A measure of the linear relationship between two quantitative variables, based on direction and strength.
- Examples: strong, weak, or no correlation; positive or negative
- Represented by r
- r = (1/n-1)*SUM((all x-values minus mean summed/standard deviation of all x-values),(all y-values minus mean summed/standard deviation of all y-values))
Regression
- Simple linear: statistical model where the means of y occur on a line when plotted against x for one explanatory variable
- Multiple linear: statistical model with more than one explanatory variable
Parametric Statistics: Use numerical data because this assumes data has a normal distribution.
Nonparametric statistics: Use ordinal or categorical data because this does not assume a normal distribution.
Analysis of Variance (ANOVAs)
- One-way: Compare population means based on 1 independent variable
- Two-way: Compare population means classified based on 2 independent variables

Source:

Moore, D. S., McCabe, G. P., & Craig, B. A. (2012). Introduction to the practice of statistics. Seventh edition/Student edition. New York: W.H. Freeman and Company, a Macmillan Higher Education Company.

See here for an updated version of textbook

In Data Science

Data Analysis and UFO Reports

AshleyJune 17, 2018June 24, 2018Leave a comment

Data analysis and unidentified flying object (UFO) reports go hand-in-hand. I attended a talk by author Cheryl Costa who analyzes records of UFO sightings and explores their patterns. Cheryl and her wife Linda Miller Costa co-authored a book that compiles UFO reports called UFO Sightings Desk Reference: United States of America 2001-2015.

Records of UFO sightings are considered citizen science because people voluntarily report their experiences. This is similar to wildlife sightings recorded on websites like eBird that help illustrate bird distributions across the world. People report information about UFO sighting events including date, time, and location.

A dark night sky with the moon barely visible and trees below. — Night sky along the roadside outside Wayquecha Biological Field Station in Peru, taken April 2015.

Cheryl spoke about gathering data from two main online databases, MUFON (Mutual UFO Network) and NUFORC (National UFO Reporting Network). NUFORC’s database is public and reports can be sorted by date, UFO shape, and state. MUFON’s database requires a paid membership to access the majority of their data. This talk was not a session to discuss conspiracy theories, but a chance to look at trends in citizen science reports.

The use of data analysis on UFO reports requires careful consideration of potential bias and reasonable explanations for numbers in question. For example, a high volume of reports in the summer could be because more people are spending time outside and would be more likely to notice something strange in the sky.

This talk showed me that conclusions may be temptingly easy to draw when looking at UFO data as a whole, but speculations should be met with careful criticism. The use of the scientific method when approaching ufology, or the study of UFO sightings, seems key for a field often met with overwhelming skepticism.

I have yet to work with any open-source data on UFO reports, but this talk reminded me of the importance of a methodical approach to data analysis. Data visualization for any field of study starts with asking questions, being mindful of outside factors, and being able to communicate messages within large data sets to any audience.