Analysis of USDA Plant Families Data Using Pandas

The United States Department of Agriculture PLANTS database provides general information about plant species across the country. Given 3 states, I wanted to visualize which plant families are present in each and which state(s) hold the most species in each family. To accomplish this task, I used Python’s pandas, matplotlib, and seaborn libraries for analysis.

Different groups of plants, including wildflowers, are arranged in a rock bed in the desert.
Various plant species in Death Valley National Park, California.

Initial Setup

Before beginning, I import pandas, matplotlib, and seaborn.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb

Gathering Data

I pulled data sets from the USDA website for New York, Idaho, and California. The default encoding is in Latin-1 for exported text files. When importing into pandas, the encoding must be specified to work properly.

ny_list = pd.read_csv('ny_list.txt', encoding='latin-1')
ca_list = pd.read_csv('ca_list.txt', encoding='latin-1')
id_list = pd.read_csv('id_list.txt', encoding='latin-1')

I double check the files have been loaded correctly into dataframe format using head().

A preview of a table shows 5 entries of various plants that are present in New York state, including their USDA symbol, synonym symbol, scientific name, common name, and plant family.
Initial import of New York state plants list categorized by USDA symbol, synonym symbol, scientific name with author, national common name, and family.

Cleaning Data

The major point of interest in the imported dataframe is the ‘Family’ column. I create a new dataframe organized by this column and returning the count from each row.

ny_fam = ny_list.groupby('Family').count()
A table shows plants grouped by taxonomic family for New York state.
Initial dataframe for New York plant data grouped by taxonomic family.

Next, I remove the unwanted columns. I’ve chosen only to keep the ‘Symbol’ column as a representation of count because this variable is required for every plant instance.

ny_fam_1 = ny_fam.drop(['Synonym Symbol', 'Scientific Name with Author', 'National Common Name'], axis=1)

Then, I change the column name from ‘Symbol’ to ‘{State} Count’ to lend itself for merging the dataframes without confusion.

ny_fam_1 = ny_fam_1.rename(columns = {"Symbol":"NY Count"})
Two tables show a before and after image of a table where the column named 'Symbol' changes to 'NY Count.'
Column before (left) and after (right) a name change.

I complete the same process for the California and Idaho data.

ca_fam = ca_list.groupby('Family').count()
ca_fam_1 = ca_fam.drop(['Synonym Symbol', 'Scientific Name with Author', 'National Common Name'], axis=1)
ca_fam_1 = ca_fam_1.rename(columns = {"Symbol":"CA Count"})
id_fam = id_list.groupby('Family').count()
id_fam_1 = id_fam.drop(['Synonym Symbol', 'Scientific Name with Author', 'National Common Name'], axis=1)
id_fam_1 = id_fam_1.rename(columns = {"Symbol":"ID Count"})

Reset the index to prepare the data frames for outer merges based on column names. The index is set to ‘Family’ as default, from the initial data frame creation using the count() function. To discourage any unwanted changes, I create a copy of each data frame as I go.

ny_fam_2 = ny_fam_1.copy()
ny_fam_2 = ny_fam_2.reset_index()
ca_fam_2 = ca_fam_1.copy()
ca_fam_2 = ca_fam_2.reset_index()
id_fam_2 = id_fam_1.copy()
id_fam_2 = id_fam_2.reset_index()
Two tables show a before and after image of New York state plant family data. The first table shows plant families as the index and the second table shows plant families as a column, with a new numerical index.
New York dataframe before (left) and after (right) the index was reset to make the plant families a column.

Merging Data

To preserve all the plant species regardless of presence in each individual state, I perform outer merges. This will allow for areas without data to be filled with zeros after the family counts are combined.

combo1 = pd.merge(ny_fam_2, ca_fam_2, how='outer')
combo2 = pd.merge(combo1, id_fam_2, how='outer')
A table shows the combined plant family data for New York, California, and Idaho.
Plant family table with counts from each state, before formatting.
pd.options.display.float_format = '{:,.0f}'.format
combo2 = combo2.fillna(0)
A table shows combined data for plant families in New York, California, and Idaho where the numbers are formatted to show zero decimal places and any instances of no data are replaced by zeroes.
Plant family table with counts from each state, formatted to drop decimals and replace NaNs with zeros.

Creating a New Column

I added a column to aid in visualizations. I created a function to return the state with the highest presence of each plant family based on the existing columns.

def presence(row):
    if row['NY Count'] > row['CA Count'] and row['NY Count'] > row['ID Count']:
        return 'NY'
    elif row['CA Count'] > row['NY Count'] and row['CA Count'] > row['ID Count']:
        return 'CA'
    elif row['ID Count'] > row['NY Count'] and row['ID Count'] > row['CA Count']:
        return 'ID'
    elif row['NY Count'] == row['CA Count'] and row['NY Count'] > row['ID Count']:
        return 'CA/NY'
    elif row['CA Count'] == row['ID Count'] and row['CA Count'] > row['NY Count']:
        return 'CA/ID'
    elif row['ID Count'] == row['NY Count'] and row['ID Count'] > row['CA Count']:
        return 'ID/NY'
    else:
        return 'Same'
    
combo2['Highest Presence'] = combo2.apply(presence, axis=1)
A table of plant families from New York, California, and Idaho shows counts for each family and a new column that names which state has the highest presence of each plant family.
Table with added column to indicate highest presence of species count within each plant family.

Below is the full table of all plant families in the dataframe.

 FamilyNY CountCA CountID CountHighest Presence
0Acanthaceae770CA/NY
1Acarosporaceae1118ID
2Aceraceae462924NY
3Acoraceae524NY
4Actinidiaceae300NY
5Adoxaceae200NY
6Agavaceae4480CA
7Aizoaceae5420CA
8Alismataceae645333NY
9Amaranthaceae788238CA
10Amblystegiaceae6400NY
11Anacardiaceae464722CA
12Andreaeaceae300NY
13Annonaceae300NY
14Anomodontaceae800NY
15Apiaceae190372257CA
16Apocynaceae486042CA
17Aquifoliaceae2130NY
18Araceae26183NY
19Araliaceae2668NY
20Aristolochiaceae2893NY
21Asclepiadaceae506720CA
22Aspleniaceae2376NY
23Asteraceae2,0573,8582,260CA
24Aulacomniaceae700NY
25Azollaceae440CA/NY
26Bacidiaceae110CA/NY
27Balsaminaceae10611ID
28Bartramiaceae1100NY
29Berberidaceae215925CA
30Betulaceae864353NY
31Bignoniaceae9110CA
32Blechnaceae5117CA
33Boraginaceae127478263CA
34Brachytheciaceae5900NY
35Brassicaceae4681,123774CA
36Bruchiaceae400NY
37Bryaceae2220NY
38Buddlejaceae450CA
39Butomaceae202ID/NY
40Buxaceae500NY
41Buxbaumiaceae500NY
42Cabombaceae663CA/NY
43Cactaceae1023878CA
44Callitrichaceae161811CA
45Calycanthaceae720NY
46Campanulaceae7614657CA
47Cannabaceae13810NY
48Cannaceae400NY
49Capparaceae184926CA
50Caprifoliaceae18210781NY
51Caryophyllaceae338506414CA
52Celastraceae24184NY
53Ceratophyllaceae855NY
54Cercidiphyllaceae200NY
55Chenopodiaceae245404245CA
56Cistaceae44270NY
57Cladoniaceae522NY
58Clethraceae400NY
59Climaciaceae500NY
60Clusiaceae521512NY
61Commelinaceae37140NY
62Convolvulaceae6813017CA
63Cornaceae553436NY
64Crassulaceae6223057CA
65Cucurbitaceae41496CA
66Cupressaceae3913030CA
67Cuscutaceae315631CA
68Cyperaceae1,016733663NY
69Dennstaedtiaceae855NY
70Diapensiaceae800NY
71Dicranaceae2000NY
72Dioscoreaceae600NY
73Dipsacaceae17108NY
74Ditrichaceae920NY
75Droseraceae8116CA
76Dryopteridaceae1216571NY
77Ebenaceae770CA/NY
78Elaeagnaceae161012NY
79Elatinaceae6144CA
80Empetraceae1360NY
81Entodontaceae800NY
82Ephemeraceae500NY
83Equisetaceae483651ID
84Ericaceae236310110CA
85Eriocaulaceae730NY
86Euphorbiaceae11123366CA
87Fabaceae6041,855871CA
88Fagaceae96910NY
89Fissidentaceae2211NY
90Flacourtiaceae200NY
91Fontinalaceae900NY
92Fumariaceae312820NY
93Funariaceae2100NY
94Gentianaceae72162122CA
95Geraniaceae348025CA
96Ginkgoaceae200NY
97Grimmiaceae233CA/ID
98Grossulariaceae4615072CA
99Haemodoraceae700NY
100Haloragaceae372820NY
101Hamamelidaceae820NY
102Hippocastanaceae1420NY
103Hippuridaceae222Same
104Hydrangeaceae245314CA
105Hydrocharitaceae454430NY
106Hydrophyllaceae1533689CA
107Hylocomiaceae800NY
108Hymeneliaceae114ID
109Hymenophyllaceae200NY
110Hypnaceae2000NY
111Iridaceae4611426CA
112Isoetaceae393028NY
113Juglandaceae52102NY
114Juncaceae162177143CA
115Juncaginaceae92213CA
116Lamiaceae399413182CA
117Lardizabalaceae200NY
118Lauraceae12110NY
119Lecanoraceae311NY
120Lemnaceae274517CA
121Lentibulariaceae372317NY
122Leucobryaceae200NY
123Liliaceae263741243CA
124Limnanthaceae3363CA
125Linaceae365220CA
126Lycopodiaceae114947NY
127Lygodiaceae200NY
128Lythraceae252616CA
129Magnoliaceae2000NY
130Malvaceae6628462CA
131Marsileaceae21512CA
132Melastomataceae1200NY
133Meliaceae330CA/NY
134Menispermaceae200NY
135Menyanthaceae1073NY
136Mniaceae1000NY
137Molluginaceae493CA
138Monotropaceae193121CA
139Moraceae21165NY
140Myricaceae2250NY
141Najadaceae252110NY
142Nelumbonaceae950NY
143Nyctaginaceae301578CA
144Nymphaeaceae502330NY
145Oleaceae39490CA
146Onagraceae237661314CA
147Ophioglossaceae605348NY
148Orchidaceae285175173NY
149Orobanchaceae227451CA
150Orthotrichaceae1100NY
151Osmundaceae1000NY
152Oxalidaceae705847NY
153Paeoniaceae242CA
154Papaveraceae389826CA
155Parmeliaceae111Same
156Pedaliaceae11258CA
157Phytolaccaceae470CA
158Pinaceae5211365CA
159Plantaginaceae647833CA
160Platanaceae980NY
161Plumbaginaceae16220CA
162Poaceae1,9272,3471,507CA
163Podostemaceae300NY
164Polemoniaceae35637279CA
165Polygalaceae29170NY
166Polygonaceae331917435CA
167Polypodiaceae9149CA
168Polytrichaceae2300NY
169Pontederiaceae17176CA/NY
170Portulacaceae24203120CA
171Potamogetonaceae11583103NY
172Pottiaceae3421NY
173Primulaceae67104102CA
174Pteridaceae1611139CA
175Pyrolaceae546567ID
176Ranunculaceae288434412CA
177Resedaceae780CA
178Rhamnaceae1820220CA
179Rosaceae1,305803609NY
180Rubiaceae15723072CA
181Ruppiaceae11177CA
182Rutaceae18120NY
183Salicaceae284352383ID
184Salviniaceae530NY
185Santalaceae959ID/NY
186Sapindaceae5373CA
187Sarraceniaceae11160CA
188Saururaceae230CA
189Saxifragaceae55219234ID
190Scheuchzeriaceae555Same
191Schistostegaceae202ID/NY
192Schizaeaceae200NY
193Scrophulariaceae4301,146556CA
194Selaginellaceae61514CA
195Sematophyllaceae600NY
196Simaroubaceae474CA
197Smilacaceae2230NY
198Solanaceae11824463CA
199Sparganiaceae242224ID/NY
200Sphagnaceae4231NY
201Staphyleaceae220CA/NY
202Sterculiaceae2300CA
203Styracaceae790CA
204Symplocaceae500NY
205Taxaceae1042NY
206Teloschistaceae111Same
207Tetraphidaceae200NY
208Theliaceae300NY
209Thelypteridaceae26913NY
210Thuidiaceae300NY
211Thymelaeaceae620NY
212Tiliaceae2200NY
213Trapaceae500NY
214Tropaeolaceae220CA/NY
215Typhaceae6105CA
216Ulmaceae231910NY
217Urticaceae526132CA
218Valerianaceae135927CA
219Verbenaceae531267CA
220Violaceae17612079NY
221Viscaceae12446CA
222Vitaceae55184NY
223Vittariaceae200NY
224Xyridaceae1500NY
225Zannichelliaceae555Same
226Zosteraceae580CA
227Zygophyllaceae5315CA
228Aloaceae020CA
229Aponogetonaceae020CA
230Arecaceae0150CA
231Basellaceae040CA
232Bataceae020CA
233Burseraceae020CA
234Caulerpaceae020CA
235Crossosomataceae0189CA
236Cyatheaceae030CA
237Cymodoceaceae050CA
238Datiscaceae020CA
239Elaeocarpaceae040CA
240Ephedraceae0160CA
241Fouquieriaceae030CA
242Frankeniaceae060CA
243Garryaceae0110CA
244Gracilariaceae020CA
245Gunneraceae030CA
246Halymeniaceae020CA
247Krameriaceae080CA
248Lennoaceae060CA
249Loasaceae08429CA
250Melianthaceae020CA
251Myoporaceae020CA
252Myrtaceae0320CA
253Parkeriaceae040CA
254Passifloraceae060CA
255Pittosporaceae080CA
256Punicaceae020CA
257Rafflesiaceae020CA
258Scouleriaceae033CA/ID
259Simmondsiaceae040CA
260Stereocaulaceae020CA
261Tamaricaceae0123CA
262Ulvaceae030CA
263Verrucariaceae002ID

Visualizing the Data

I created a count plot using seaborn to show which states, or state combinations, have the highest variety within each plant family.

base_color = sb.color_palette()[2]
sb.countplot(data=combo4, x='Highest Presence', color="#B6D1BE", order=combo4['Highest Presence'].value_counts().index)
n_points = combo4.shape[0]
cat_counts = combo4['Highest Presence'].value_counts()
locs, labels = plt.xticks()
for loc, label in zip(locs, labels):
    count = cat_counts[label.get_text()]
    pct_string = count
    plt.text(loc, count+5, pct_string, ha='center', color='black', fontsize=12)
plt.xticks(rotation=25)
plt.xlabel('')
plt.ylabel('')
plt.title('Highest Concentration of Plant Families by State', fontsize=14, y=1.05)
plt.ylim(0, 140)
sb.despine();
A vertical bar graph show the highest concentration of plant families organized by state, where the concentrations are high to low as follows: California, New York, California/New York, Idaho, all the states tie, Idaho/New York, and California/Idaho.
Shows the count of plant families with the highest concentration in each state.

Further Considerations

There are many factors that play into plant family diversity. The comparison of plant families in New York, California, and Idaho was purely out of curiosity. Further investigations should take into account each state’s ecosystem types and land usage and ownership that may influence species diversity.

Statistics Principles for Data Analysis

Statistics Principles for Data Analysis
Four traditional dice on a game board

Recently, I had to brush up on statistics terms for a data analyst exam. I had trouble pulling together old course notes to create a quick, cohesive study guide. Below, overarching concepts are from the test’s public posting and my notes are derived from a quantitative statistics textbook.

  • Central Tendency
    • Mean = the average of a distribution
    • Median = a distribution’s midpoint
    • Mode = the variable which occurs most often in a distribution
  • Variability
    • The distribution of data, also known as spread
    • Five-number summary: Minimum, Q1, Median (Q2), Q3, Maximum
      • Represented through boxplots graphically
    • Summarized through quartiles:
      • Q1: median of all values to left of Q2
      • Q2: median (50th percentile) of all values in distribution
      • Q3: median of all values to right of Q2
    • Variance: s^2 = SUM((value minus mean)^2 for all values)) / (number of values-1)
    • Standard deviation: square root of the variance (s^2)
  • Normal Distribution: bell-curve distribution of data
  • Hypothesis Testing: Examine evidence against a null hypothesis, hypotheses referring to populations or models and not a certain outcome
    • Compare claims
    • Null hypothesis: statement challenged in significance testing
      • Example: There is not a difference between means.
    • Alternative hypothesis: statement suspected as true instead of the null hypothesis
      • Example: The means are not the same.
    • Accept or reject null hypothesis based on a certain p-value.
    • p-value: the likelihood that the test statistic would be a value equal or higher than what is observed
    • Smaller p-values signify stronger evidence against the null hypothesis in question. Often, an alpha value of 0.05 is used. Evidence would be so strong that something outside the p-value should only occur 5 out of every 100 times.
  • Statistical Significance Testing: Achieved at the level where the p-value is equal or less than alpha.
  • Probability: The proportion of times an outcome would occur given many repeated tests.
  • Correlation
    • A measure of the linear relationship between two quantitative variables, based on direction and strength.
    • Examples: strong, weak, or no correlation; positive or negative
    • Represented by r
    • r = (1/n-1)*SUM((all x-values minus mean summed/standard deviation of all x-values),(all y-values minus mean summed/standard deviation of all y-values))
  • Regression
    • Simple linear: statistical model where the means of y occur on a line when plotted against x for one explanatory variable
    • Multiple linear: statistical model with more than one explanatory variable
  • Parametric Statistics: Use numerical data because this assumes data has a normal distribution.
  • Nonparametric statistics: Use ordinal or categorical data because this does not assume a normal distribution.
  • Analysis of Variance (ANOVAs)
    • One-way: Compare population means based on 1 independent variable
    • Two-way: Compare population means classified based on 2 independent variables

Source:

Moore, D. S., McCabe, G. P., & Craig, B. A. (2012). Introduction to the practice of statistics. Seventh edition/Student edition. New York: W.H. Freeman and Company, a Macmillan Higher Education Company.

See here for an updated version of textbook

Data Analysis and UFO Reports

Data Analysis and UFO Reports

Data analysis and unidentified flying object (UFO) reports go hand-in-hand. I attended a talk by author Cheryl Costa who analyzes records of UFO sightings and explores their patterns. Cheryl and her wife Linda Miller Costa co-authored a book that compiles UFO reports called UFO Sightings Desk Reference: United States of America 2001-2015.

Records of UFO sightings are considered citizen science because people voluntarily report their experiences. This is similar to wildlife sightings recorded on websites like eBird that help illustrate bird distributions across the world. People report information about UFO sighting events including date, time, and location.

A dark night sky with the moon barely visible and trees below.
Night sky along the roadside outside Wayquecha Biological Field Station in Peru, taken April 2015.

Cheryl spoke about gathering data from two main online databases, MUFON (Mutual UFO Network) and NUFORC (National UFO Reporting Network). NUFORC’s database is public and reports can be sorted by date, UFO shape, and state. MUFON’s database requires a paid membership to access the majority of their data. This talk was not a session to discuss conspiracy theories, but a chance to look at trends in citizen science reports.

The use of data analysis on UFO reports requires careful consideration of potential bias and reasonable explanations for numbers in question. For example, a high volume of reports in the summer could be because more people are spending time outside and would be more likely to notice something strange in the sky.

This talk showed me that conclusions may be temptingly easy to draw when looking at UFO data as a whole, but speculations should be met with careful criticism. The use of the scientific method when approaching ufology, or the study of UFO sightings, seems key for a field often met with overwhelming skepticism.

I have yet to work with any open-source data on UFO reports, but this talk reminded me of the importance of a methodical approach to data analysis. Data visualization for any field of study starts with asking questions, being mindful of outside factors, and being able to communicate messages within large data sets to any audience.