Looking at the UK Greenhouse Gas Data

If you downloaded the full dataset from the previous link above you would see that it has 533,016 rows. The first step, let's just take a look at the data - the first 1000 rows are shown below:

NOTE: Depending on your screen size and resolution you will only see the first few columns initially but you can scroll right, or use the arrow keys to see the full set of variables

From a quick inspection visually of the first few hundred rows of this data we can actually learn quite a lot and start to answer some of the questions we may have.

The first thing you might notice is that there seems to be a lot of data from Hartlepool, and not a lot of data from anywhere else! You might be worried that data has been incorrectly duplicated.

Closer inspection of the data shows why the data looks like this - we can see that data is stored as one row per gas per sub-sector per year per local authority. And we have three different gases (CO2,CH4 and N20), 32 different sub-sectors, and 18 years of data. So that's potentially up to 1,728 rows per local authority.

So the first 1000 rows will indeed all come from Hartlepool, since the Local Authority code "E06000001" happens to be the first one alphabetically. Having data with this structure also tells us that we will need to be comfortable with manipulating the data between different levels - if we wanted to look at the total emissions per year, or within a sector, or correlate different sources/gases against each other then we would need to be able to manipulate our dataset in various different ways to allow us to do this.

The column names and units in the data are relatively transparent as to what they mean. The acronyms used are all relatively standard - some of the less familiar ones may be "LA"="Local Authority"; "GHG"="Greenhouse Gas"; "kt"="Kilo-tonnes";"CO2e"="CO2 equivalent". And, the UK government contains extensive metadata and documentation that would be able to help us if we got stuck here.

Something that is very clear is that manually checking through 533,016 rows of data sounds like a terrible idea!

Within R, and within most equivalent software, there will be commands to explore the structure, or summary of the data, and this can help us to identify other potentially important points about the data as a quick initial snapshot.

##              Country          Country.Code                Region      
##  England         :437065            :   414   South East     : 93988  
##  Northern Ireland: 16368   E92000001:437065   East of England: 65897  
##  Scotland        : 46538   N92000002: 16368   North West     : 53434  
##  Unallocated     :   414   S92000003: 46538   East Midlands  : 51598  
##  Wales           : 32631   W92000004: 32631   London         : 46863  
##                                               Scotland       : 46538  
##                                               (Other)        :174698  
##     Region.Code          Second.Tier.Authority                Local.Authority  
##  E12000008: 93988   Scotland        : 46538    Antrim and Newtownabbey:  1564  
##  E12000006: 65897   Wales           : 32631    Belfast                :  1564  
##  E12000002: 53434   Lancashire      : 18260    Lisburn and Castlereagh:  1564  
##  E12000004: 51598   Kent            : 17605    Mid and East Antrim    :  1564  
##  E12000007: 46863   Essex           : 17341    Oldham                 :  1551  
##  S92000003: 46538   Northern Ireland: 16368    Somerset               :  1551  
##  (Other)  :174698   (Other)         :384273    (Other)                :523658  
##  Local.Authority.Code Calendar.Year      LA.GHG.Sector  
##  N09000001:  1564     Min.   :2005   Transport  :88239  
##  N09000003:  1564     1st Qu.:2009   LULUCF     :85816  
##  N09000007:  1564     Median :2013   Agriculture:83706  
##  N09000008:  1564     Mean   :2013   Industry   :74459  
##  E06000066:  1551     3rd Qu.:2018   Domestic   :58380  
##  E08000004:  1551     Max.   :2022   Commercial :58203  
##  (Other)  :523658                    (Other)    :84213  
##                LA.GHG.Sub.sector  Greenhouse.gas
##  Industry Electricity   : 19656   CH4:164044    
##  Domestic Electricity   : 19602   CO2:185898    
##  Agriculture 'Other'    : 19494   N2O:183074    
##  Agriculture Electricity: 19494                 
##  Commercial 'Other'     : 19494                 
##  Commercial Electricity : 19494                 
##  (Other)                :415782                 
##  Territorial.emissions..kt.CO2e.
##  Min.   :-2854.921              
##  1st Qu.:    0.040              
##  Median :    0.534              
##  Mean   :   17.121              
##  3rd Qu.:    5.269              
##  Max.   :10542.349              
##                                 
##  CO2.emissions.within.the.scope.of.influence.of.LAs..kt.CO2.
##  Min.   :   0.00                                            
##  1st Qu.:   0.00                                            
##  Median :   0.00                                            
##  Mean   :  12.01                                            
##  3rd Qu.:   0.00                                            
##  Max.   :4091.07                                            
##                                                             
##  Mid.year.Population..thousands.   Area..km2.      
##  Min.   :   2.21                 Min.   :    3.15  
##  1st Qu.: 102.72                 1st Qu.:   95.09  
##  Median : 139.51                 Median :  269.24  
##  Mean   : 179.28                 Mean   :  693.70  
##  3rd Qu.: 225.23                 3rd Qu.:  641.18  
##  Max.   :1157.60                 Max.   :26473.95  
##  NA's   :414                     NA's   :414

There are a lot of numbers in there - but there is actually a lot of really useful insights we can make, that could help us to understand the data:

• There are 414 rows with missing values for population and area of the county.
• There are also 414 rows in the data where country is "Unallocated" to any of the countries of the UK. It would seem logical that these may be the same rows - but let us check to make sure!

• If we look more at data restricted to show only the unallocated rows the "local authority column" we can see that these are definitely not local authorities! This column provides a very short description of the context - "Large elec users" or "Unallocated consumption". Nearly all the rows for these unallocated entries are for the Industry or Domestic sectors although there are some which relate to Waste or LULUCF (https://unfccc.int/topics/land-use/workstreams/land-use--land-use-change-and-forestry-lulucf). The existence of these rows, attributing emissions to unknown sources, will mean we have to think carefully about what to do with them when producing summaries of our data as there may be a number of different approaches we could consider.

• There are negative values for the column "Territorial.emissions..kt.CO2e". We may not have been expecting that! In these cases we will have estimates linked to carbon offsetting schemes. So this column is dealing with net emissions, and this is an important point for us to understand. And it is good that we have identified these before we did something which would have been impossible with negative data values, like taking a log transformation for example.

• There are not the same number of rows for each of the sub-sectors. Although some of this could be explained by the unallocated sources the disparity definitely accounts for a lot more than 414 rows! So we might then start to explore whether we have data for each sub-sector within each year, for each of the three greenhouse gases and for each country.

Extension Exercise Download the raw data from the link in the previous section if you have not done so already, and see if you can explore the data to understand why some sub-sectors have more rows than other.

Data Aggregation & Restriction

To give ourself a slightly smaller task to take on in this workbook, we will focus on an aggregated subset of the data designed to meet the needs of a specific research question.

For the remainder of this module we will focus on the relationships that exist between the CO2 emissions and the geographic characteristics of each local authority from the most recently available data.

This will require three (or four) separate steps: - Subset the data so that it only includes data from the most recent year - 2022 - Subset the data so that it only contains data from the greenhouse gas of interest - CO2 - Aggregate the data so that for each local authority we calculate the sum of emissions across all of the sources accounted for - Being careful with the population and area variables in our aggregation process! We want to retain this variables in our new data set, but we do not want to sum these across all of the different sources, as the values of area and population are fixed for each local authority within each year.

Below you can explore the aggregated of the data showing the total CO2 emissions for each county from 2022 (Annual.CO2.Total), which is the one I used throughout the video for this module.

Summarising

Summary statistics help us to get a better idea of the variability, the central tendency, the extreme values, and the distribution and relative frequency of the values. Let's start with the CO2 emissions variable, which is a continuous numeric variable and consider as many different ways we can think of for producing summaries:

From the table you are hopefully fairly familiar with most of the quantities mentioned; although I expect there will be some you will have heard of but perhaps don't really understand, and maybe some with which you are less familiar with.

The interactive table below provides a summary of how you might intepret each of these numbers, and what the statistics can be used for. In general these statistics will fall into one of 4 categories -

• frequency - these statistics will tell us how much data is available, and how frequent specific events may be

• central tendency - often referred to as "average", these statistics will tell us something about the 'middle' of our data, or the most likely values to occur in the data.

• variability - these statistics will quantify how much deviation there is in our values away from the 'middle' of the data, and help us to the variation that exists within the data

• distribution - these statistics will help us to understand the shape of the data. Are values equally spread above or below the average? Are there outliers and how to way degree do these fall outside of expected ranges? Is there a single peak around the "average" or are there multiple peaks?

Visualisation

All of those numbers can feel a bit overwhelming, and to be honest most of them are not especially helpful in terms of actually trying to understand what is going on in my data!

Visual methods for exploration of data are almost always going to be more useful in uncovering what is going on in our data. As an example of this there is a famous set of simulated data, the 'Datasaurus dozen', https://en.wikipedia.org/wiki/Datasaurus_dozen, in which the following 13 pairs of variables which all have identical number of observations, identical means, identical standard deviations and identical correlations between the sets of variables. And yet they each show extremely different patterns that would not have been at all apparent from tables of statistics:

But, much like with the list of statistics, there are a whole lot of different options we could use here to visualise our data, and there are a lot of pros and cons to each of these methods.

In general there is a trade-off we need to make between 'displaying' the data and 'summarising' the data - methods that only 'display' the data will be difficult to make inferences from, methods which summarise the data too much will focus too much on one particular aspect (usually the average value), and may overlook other key aspects of the distribution and variability of the variable being plotted.

Take a look at some of the more common different approaches using the interactive plot below, and with each option think through what these plots are telling you about the CO2 variable. If there are methods you are unfamiliar with, then the table in the next section will provide a brief summary of what they are doing and how they can be interpreted.

Further explanations about exactly what the plots are presenting, and why, and some pros and cons can also be found in the table below.

Outliers

Within some of our plots we could potentially identify three points that perhaps look a little different to others.

• There are two points with CO2 emissions that are substantially higher than any other local authority - Birmingham and North Yorkshire. This can be seen in nearly all of the plots.

• Within some of the plots, the histogram and the caterpillar plot in particular, we may also be able to identify that there is one point with very low CO2 emissions relative to the other local authorities - the Isles of Scilly

We should take action against outliers if we believe the data to be incorrect, or the outlier to represent something which comes from a different process to the rest of our data such that it would be incomparable. But this should not be a purely data-driven our automated process - we should not be removing outliers from our data solely because they look different and their existence is mildly inconvenient to our analysis. This is likely to influence and bias our results as we are artificially reducing the variability that exists in the parameters we are trying to understand. We should be validating our outliers as genuine observations, and looking for factors that can help to explain why the results are different.

When considering the presence of outliers there are four key questions that we need to ask:

• Is this a real observation or is this an error? If it is an error then we should be trying to correct the value, and if that is not possible then we should exclude the value. Very commonly data entry errors may only become apparent when they appear as outliers - it is very easy to get decimal places in the wrong position, or add too many zeros at the end of large numbers.

• If it is real does this observation come from the same underlying population as the rest of the observations, and collected using the same methodology? It is possible for data to be 'real' but be incomparable to the rest of our observations. We may be "comparing apples to oranges" by including them side by side. In the video I highlighted the case of the "City of London", where the actual population is very low but the number of people who work within this area is very low. So considering a "per-capita" adjustment for the City of London based on population is going to be extremely misleading, and potential unhelpful to include within formal analysis.

• If it is real are there other variables that may explain why it is so different? The previous plot also helps illustrate a separate point - note the massive drop for the emissions in 2010 and 2011? The sudden drop may make us question if this is a mistake, but as is outlined in the wiki page - in 2010 and 2011 the Steelworks temporarily closed, before being taken over by new owners in 2012. It may be that the explanations can be linked to variables within our dataset that should then be used within our models. This is indeed the case for the three points highlighted in our data - Birmingham has an extremely large population, North Yorkshire has a large area, The Isles of Scilly has both a low population and a small area. These variables will need to be accounted for in our final analysis process.

• What is the impact of leaving the outlier in our data? At this point in our exploration we should definitely note down the potential issue as one to explore later in our analysis to assess whether we see the same results with and without the outlier included. We would consider this to be an example of a 'sensitivity analysis'; if we do see very different results we may choose to present both - with caveats included around the inclusion or exclusion of potential outliers. An outlier is generally not a problem unless it has "high leverage"; we will come back to that concept later on in this course.

Other key characteristics from visual exploration

There is a lot more than just outlier identification that we can see from this univariate exploration of our CO2 emissions variable!

Some of the key characteristics we should be interpreting from these plots would be: - The 'average' value of the variable is around 600-700 kt of CO2; although this is the one area when we can see that more clearly from the summary statistics rather than any of the plots

• The distribution is 'unimodal'; i.e. in the histogram, density and violin points there is one clear peak around which the "average" or most common level of CO2 emissions falls

• The values in the data are truly continuous variables, i.e. they are all distinct and not spaced at regular intervals. If data had been rounded off, or fell into an ordinal distribution, you would expect to see spikes in the shape of the density and violin plots, and possibly regularly spaced gaps in the histogram.

• There is a 'positive' skew in the data - there are an increasingly small number of points with increasingly high amount of CO2 emissions. This is very clearly seen in the histogram, density, caterpillar, boxplot & violin plots. In these cases the mean will be substantially higher than the median, as we saw in the summary statistics table.

• As a result of this the data is definitely not 'symmetrical' and could not be considered to be "normally" distributed. If it was the case then the histogram/density plots would fit into a "bell shaped curve" and the points in the QQ plot would follow the line

• There is a large range in the values, which can be seen in most of the plots, but the majority of data points fall in a relatively narrow range between 200-800 kt of CO2. This aspect can be best seen from the relatively small width of the box in the boxplot; the sharp peaks in the violin/density/histogram plots; the lack of variability in the caterpillar plot; and the rapid vertical ascent in the cumulative density plot.

When highlighting some of these key aspects of the plots, you may notice that most of the different types of plots have been referred to - except for bar charts, 'dynamite plots' and means with error bars.

As tools for exploratory data visualisation then these are all extremely limited - bar charts when used for numeric variables generalise a complex set of data down to a single point and then visually represent them in a way which is meaningless. There is no reason for a bar chart representing the emissions to start at 0 - we could choose to draw the bar starting anywhere, thus making the size of the bar meaningless and easy to provide misleading plots when we start bringing in additional variables to compare across.

The use of error bars within exploratory data analysis is also generally not a good practice - at this stage in the analysis process any calculation of standard errors or confidence intervals would be based on naive generic assumptions, that may be wildly incorrect. Using errorbars is a powerful visual tool to display and communicate the uncertainty in estimates that we have made after we have concluded our analysis and modelling which we will cover in later sessions. But it is not a good tool to help us understanding the data within this exploratory phase.

Visualisation

Similarly there are fewer options when considering visualisation of categorical data. All of them fundamentally distil down to the same idea of presenting some sort of object that has a size proportional to the frequency or the percentage.

Although there are perhaps a surprising amount of different options here:

With this particular variable, showing the reduction in CO2 since 2005 for each local authority, there is nothing that we should be too worried about. There are no issues with labelling of the categories; although the 40-50% category is the most frequent it is not approaching a level of dominance that would indicate it might cause problems for our analysis; and there are at least 20 counties in each of our four categories.

That final point matches perfectly a somewhat arbitrary rule of thumb that is used when determining if a category is 'too rare' - where we would be looking for at least 20 observations per group to be wanting to include it a statistical model. In reality, setting a minimum number of observations per group required for use in modelling is a little more complex than this, depending on the relationships between this variable and the rest of the data, and what modelling techniques are to be used, but in general it is not a bad rule of thumb to have in mind at this stage.

Comparing Numeric vs Categorical

All of the summary statistics covered previously would work in exactly the same way as we saw already, except we now have multiple groups to compare our statistics across. Even having some degree of comparison across groups defined by categorical variables can help to make many of these statistics a bit more useful! Let's consider splitting them by country:

But of course, the same limitations apply - and we will nearly always get a better understanding from comparing groups visually.

Instead of looking at one single box in a boxplot we are now looking at two (or more) boxes side by side.

Many of the methods covered previously for visualising one variable in fact become a lot more useful in this context, of comparing one variable across multiple groups. But not all of the methods covered are so suitable for comparison - qq plots for example may be produced for different groups, and we may be interested in understanding the results within each group, but there is no real benefit in visually comparing the relative 'normal-ness' of different groups. In these cases we may choose to plot different groups in different panels, rather than overlay the groups within the same plot.

Categorical vs. Categorical

When exploring the relationship between two (or more) categorical variables there is just one additional source of complexity to consider when summarising the results - what percentages are the most meaningful or useful to calculate?

In the table above which sort of percentage is likely to make the most sense?

• (within) row percentages: '92% of the counties who reduced CO2 by 50% or more were in England'

• (within) column percentages : '11% of the counties in England reduced CO2 by 50%'

• overall percentages : 9% of all counties reduced CO2 by 50% or more and were in England

Sometimes multiple versions of the percentages may be of interest; although it is usually the case that there will be one which is clearly the most appropriate option to be using. In this case, and in cases were there is a clear distinction between an 'outcome' variable (like CO2 emissions) and an 'explanatory' variable (like country), there is a clear answer as to which would be the most sensible to consider.

So the 'column' percentages from the table above will be the most useful here - being able to compare the % of counties falling into each of the CO2 reduction categories by country tells us some interesting things about differences between the countries of the UK.

Looking at the 'row' percentages here does not tell us much at all. Out of all of the counties in the UK 296/361 are in England. Therefore it makes it close to impossible to work out if the distribution across the four counties is really that different when comparing the percentages within each reduction category.

The overall percentages do not make sense at all here. They would only really make sense to compute where we were interested in combining these two variables together to form a new composite variable of the interaction between the categories of these variables.

Remember that the "row" and the "column" percentages refer only to the orientation of the variables in the table - whether the percentages within each row add up to 100%, or whether the percentages within each column add up to 100% . The 'useful' percentages would become the "row" percentages should we have decided to put the countries as the variable making up the rows, and the CO2 reduction categories as the variable making up the "columns".

A similar choice is required when producing visualisations, previously it did not matter whether we plotted frequencies or percentages as visually this would end in the same result.

When dealing with two (or more) variables then plotting the frequencies is more or less equivalent to plotting the "overall" percentages - where it may help to partition the overall dataset into subgroups based on the two variables, but it does not help us understand the relationship between those variables.

The waffle plot or tree map approach is one that only really makes sense when dealing with the "overall" percentages, or frequencies, as it will be visually misleading to present within group percentages. You can see an example of this in the interactive menu below where the presentation of each country taking the same size does not intuitively make sense in the same way as the other visualisation options.

Stacked bar charts come into their own here in this case where we are comparing an ordinal categorical variable, as it lets us accumulate between adjacent categories from the top or the bottom of the stack so that we can start visually comparing, for example, those with "40% or lower" reduction emissions - not just those with "30% or lower". Making visual comparisons between pie charts is something which is a challenge - there is no way to do this intuitively.

Correlation

You are probably also familiar with the concept of a 'correlation coefficient' which is a summary statistic to assess if two numeric variables are positively correlated (i.e. as one variable increases the other also increases; a positive value for the correlation coefficient), negatively correlated (i.e. as one variable increases the other decreases; a negative value for the correlation coefficient), or uncorrelated (no relationship between the variables a value close to zero).

The further away the coefficient is from zero, the stronger relationship is - it is often common to see a correlation matrix, which will assess all pairwise correlations between numeric variables in your dataset:

This matrix shows us that:

• All pairwise correlations between our three variables (area, population, and CO2 emissions) are positive
• The correlation between CO2 emissions and population is very strong - generally this would be considered a correlation of 0.7 or higher
• The correlation between CO2 emissions and area is moderate - generally as a rule of thumb this could be considered a correlation of 0.25-0.5
• The correlation between population and area is moderate - generally as a rule of thumb this could be considered a correlation of 0.25-0.5

Also notice that the correlation between a variable and itself is equal to 1; area is of course perfectly correlated to area! And that the "top right" and "bottom left" corners of the matrix are identical - the correlation between area and population, is identical to the correlation between population and area. So despite being a table of 9 numbers, there are only 3 that are of any interest to us.

Like with the scatter plot, this is the immediately obvious choice for a summary statistic when considering two numeric variables, but therere are lots of different variations of the correlation coefficient. The "default" correlation method that you are likely familiar with, and has been applied in the plot above, is the "Pearson" method. Some of the most common alternatives, "Spearman" & "Kendall", are summarised in the table below with the rationale for when they might be applied.

In statistics we often see this trade-off in choice of methodologies between 'power' and 'robust-ness' - the "Kendall" method for correlation is perhaps the most robust, but the least powerful - it will always be the smallest of the 3 measures. Whilst the "Pearson" method is the most powerful but least robust, it is highly sensitive to outliers in small data sets or to non linear trends. It is often incorrectly claimed that "Pearson" correlation also requires the variables to be normally distributed; but this is not actually true.

If the conditions for the Pearson method are satisfied (a consistent linear relationship, with no, or very few, data points sharing the exact same value, then it is likely that this version of the correlation coefficient will be the largest, providing the strongest evidence of correlation, but that all three methods will give broadly similar results.

You can explore how the choice of correlation method affects the values in the interactive table below:

Remember that these pairwise correlations may be hiding a more complex relationship - either one that is not monotonic or one that may be moderated, or influenced, by other variables. Take the relationship between area and population. The Spearman and Kendall coefficients for this relationship are both very close to zero, and in fact negative.

And again remember that all of these methods of correlation are useless if the relationship between the two variables does not strictly increase or decrease - you can go back to the previous section, and remind yourself of the "datasaurus" dozen, where all 13 of the plots showed the same lack of correlation between the x and y variables, despite the two variables being very clearly related in many of the plots.

So in practice, looking at the relationship through the scatter plot is a vastly superior method of understanding your data. And this comes with an additional advantage of being easy to extend scatter plots further to show beyond just a simple x/y relationship.

Going further with scatter plots

Beyond just establishing the potential existence and shape of trends, there are many further questions we can visually explore with scatter plots:

• Is the variability away from the trend consistent or does it vary (e.g. is there higher variation when we have higher values?)
• Is the relationship consistent across different groups?
• Are there values which do not fit into the trend? Are these mistakes, or are there other factors which may explain why they are different?
• If there are other factors, can we start to bring in additional variables to explore the relationship in 3 (or more) dimensions?

From the plots we have made so far there are some questions or next steps that we might take from our original plot, once we have acknowledged the clear strong trend that exists:

• The density of points is very high in the bottom left of the plot - there are lots of counties with relatively small populations and only a small number of counties with larger populations than others. Birmingham, North Yorkshire and Leeds all have points which are very distinct from the others. At the same time the deviation from the trend looks like it seems to get larger as we increase the population. Look at the point for North Yorkshire which appears to be something of an outlier compared to the trend. If you were to compare it to Glasgow, for example, both have very similar populations, but the emissions for North Yorkshire are around 3500 kt/CO2 and for Glasgow around 1950 kt/CO2. The emissions from North Yorkshire are similar to that of Birmingham, where the population is almost twice as large.

We saw this sort of trend a little with the CO2 and population variable but it is extremely pronounced with the area variable.

For the area variable this is the exact sort of pattern where we would want to consider a log transformation for population, or potentially for both of the variables. This will help us to see inside the dense cloud of points currently at the bottom left rather than only focusing on the small number of points with high values. There are other transformations beyond log transformations that we could choose to explore - and there will be particular scientific contexts where different transformations will be scientifically coherent - but it is usually a good idea to consider whether that transformation can be interpreted coherently rather than just being a 'good fit' for the data. Log transformations are nice in this way as they can, with a little bit of additional thought, usually be interpreted.

Considering a log transformation for population only, the question the plot is assessing changes from whether "as the size of the population by increases by (x) people does the CO2 emissions increase by (y) kt?" on the original scale, to a new question of "as the size of the population doubles in size does the CO2 emissions increase by (y) kt?". Remember though that this sort of transformation will only work when the variable is always larger than zero, as we cannot think about logarithms and rates of change when the values can plausibly be negative or equal to zero.

For area, it is of clear benefit to perform a transformation, but for population or CO2 emissions it is less clear cut. So it may be something to consider in our formal analysis where it may or may not be a useful transformation to help us model the relationship better.

• Having uniformly sized and styled points for all counties may be a little limited. We could bring in additional variables to map to the size and colour of the points - perhaps different colours for the four nations of the UK, and different sizes based on the area of the county?

You can explore these options, and more, by adjusting the selections in the interactive plot below.

By trying different options in our exploration, this will help us to understand even more about our data!

• The large deviation in the trend for North Yorkshire does indeed look be linked to its very large area as we suspected earlier
• The biggest outlier when assessed on the log scale, may in fact be the City of London, which has a very low population but a high level of emissions. This itself is likely to have some clear explanations - the number of people working within the City of London is extremely high, but the number of permanent residents is low. There may exist some variable to quantify this, based on economic activity, but this is not included within the dataset that we have. Sometimes deviations we see from the trends would require additional data to confirm.

Exploring Trends Over Time

'Time' is perhaps a special case of a numeric variable - and in fact in different sections of this tutorial we have in fact considered "year" as a numeric or sometimes as a categorical variable. In general if we are dealing with more than a small number of numeric observations we would want to consider time broadly more like a numeric variable. Beyond this there are also a few additional things to consider when dealing with time, and particularly when dealing with data recorded on a more regular basis than once per year:

• Conventional wisdom would suggest always placing time under the y axis and plotting trends over time as a line connecting the observations, rather than plotting points for each observation
• When dealing with data recorded at very frequent intervals plotting every single recording is likely to give a lot of noise; some decisions would have to be made about what level to aggregate the data to - every minute? every hour? every day?. It is worth exploring the data and producing summaries at all of these levels, and any other relevant ones, to better understand the periodicity and auto-correlation in the trends over time. Periodicity refers to consistently repeating patterns - e.g. temperatures getting warmer in the day and then colder in the night; while auto-correlation refers to how correlated adjacent observations would be - e.g. the temperature now is likely to be extremely highly correlated to the temperature 10 seconds ago; somewhat correlated to the temperature in 24 hours time; and, after accounting for the seasonal effects (which are part of the periodicity), very weakly or not at all correlated to the temperature in one months time. Plotting the data at different time resolutions lets us explore these components of time series, which are very important to understand before moving to formal time series modelling.
• When joining together data to form a line over time there is a visual question of whether to join each data point directly together, or whether to create steps, like a staircase. Often this will depend on whether you have regularly spaced data or 'event' data. If the change in the variable happen gradually between the time points, like may be expected where there are regular intervals, then a straight line would make more sense. If the change in the value happens at the exact time point being recorded then it would not make sense to interpolate between the points, so drawing 'steps' may make more sense.
• The 'area under the curve' is often a useful summary statistic to consider when dealing with trends over time. Within different contexts for different variables being plotted against time it can have different direct interpretations, but in general it is a measure of intensity. A variable which quickly rises to a high peak value, and then quickly falls back down, will have a lower "AUC" compared a variable which may reach a lower peak but remains around that peak for a longer period of time.

Let's take a look at a slightly different extract of the data we have seen already - this time looking at the total CO2 emissions per country over the full time period of the data 2005-2022.

Derivations

As well as the choice of visual method, there are further data manipulations, derivations and transformations which will help us to understand the trends better here. Specifically to time series we could consider plotting the change over time since baseline, in 'real terms' or in 'percentage terms' instead of the actual values on the y axis. Plotting this in percentage terms is a good option here to make a fair comparison in the trend across the four countries, at the trade-off of being able to easily read off from the plot the actual CO2 values.

The log or square root transformations are not especially useful in this plot! But adjusting the emissions per capita, or per area, are perhaps better options to help make a much fairer comparison across the 4 groups, which have vastly different areas and populations. This requires us to derive new variables, in this case by dividing the emissions by the population or area.

And when looking "per capita" we can see very similar trends across the four countries, with the line almost following in parallel to each other, all generally decreasing over time. When looking at the percentage change over time the trend is very similar for three of the four countries - with the lines not just parallel but in fact almost identical over time. The difference is from Northern Ireland - where the percentage reduction over time was lower, with emissions in fact increasing for the first few years of this data.